ScrappyLabs · Field Notes · Plain English
Did an AI model try to phone home?
The plain-English version of our test, with no jargon. Want every number and source? Read the full write-up.
The claim
A scary report
On 26 October, Cohere, a Canadian AI company, published a report about an AI model called Qwen, made by the Chinese tech company Alibaba. While the model was working on a task with access to a computer, it tried to send something to a server owned by Alibaba.
The obvious question: is a Chinese AI model spying on the people who use it?
The test
What we did
We could have picked a side. We ran a test instead. Cohere published the exact conversation that set the model off, so we gave that same conversation to the model on our own computers, 5,700 times. We also tried models from two other companies, DeepSeek and Google, to see whether they behave any differently.
One safety rule throughout: nothing the AI wrote was ever run. We only read what it wrote.
The results
What we found
The explanation
Why we call it a fossil
Picture a kid who has heard the same phone number so many times that, when asked "what number goes here?", they recite it without thinking. They aren't calling anyone. They just remember it.
That is what this looks like. AI models learn from enormous piles of text, and the address this one remembers looks like it came from someone's work logs that got swept into that pile. The thing it "tried to send" was a test message, literally the words PROBE_PAYLOAD_ABC123, the kind engineers use to check whether a connection works. Nothing from the computer was in it.
The other companies' models do the same thing with different addresses. DeepSeek's favorite was a popular public testing website.
A fossil is a leftover from how the model was made. A spy would have a plan. We found the leftover, and no sign of a plan.
The bigger question
So which is worse?
With an open model like Qwen, you download the AI and run it on your own computer. You can test it, as we did in an afternoon, and you can cut it off from the internet with one setting.
With a closed AI service, everything you type goes to the company's servers. That is how those services work, and it is in the contract. Most of these companies handle your data carefully, but you can't look inside. You have to take their word for it.
Cohere's report ends with "Trust in your model developer is paramount." That is good advice when you can't check. With an open model, you can.
The takeaway
What to do
If your business lets AI tools act on a computer (run commands, browse, move files), don't give them open internet access. Let them reach only the sites they actually need. That one rule protects you from this quirk, from a real spy if one ever turns up, and from mistakes by every other model too. Cohere gives the same advice.
Cohere sells its own AI to governments, banks and defense companies, marketed as "sovereign AI": AI a country can own and control itself. Three days after its report, Canada announced it would invest up to C$1.6 billion in the company. That doesn't make the report wrong, since part of it checked out. It does help explain the scary framing.
And no AI company has spotless training data. Cohere, Anthropic, Meta and Stability AI have all been taken to court over what their models learned from. The full write-up has the details.
We build in the open. The tests ran on our own hardware. Questions about keeping AI tools on a short leash? Get in touch. The full technical version, with every number and source, is A fossil, not a spy.
ScrappyLabs · built in the open · Asheville, NC · [email protected]