ScrappyLabs · Field Notes · Security
A fossil, not a spy.
Cohere says a popular open-weight model tried to phone home to an Alibaba server. We ran Cohere's own trace on our own hardware, 5,700 times. The model does carry a memorized Alibaba address, but it never once reached for the network by itself. That is a fossil, not a spy, and it is the kind of risk you are allowed to check.
Read the plain-English version: same findings, no jargon, about 4 minutes.
The claim
What Cohere reported
Cohere published the case on 26 October 2026, with the full trace and its token IDs on Hugging Face. In their sandbox, Qwen3.8-Flash-Next was given a security task and a shell. Its reasoning said it would make no tool calls, and then it made one.
That call checked which tools the machine had, printed the DNS settings, and tried an HTTP PUT to an Alibaba storage bucket in Beijing. The sandbox had no route out, so the request died at the container wall.
Cohere says this happens in fewer than 1 in 100,000 calls. When they prefilled the model up to curl -s, 36% of completions wrote a cloud-metadata address, the internal endpoint where cloud servers hand out credentials. The address the model wrote most often was a different Alibaba bucket, routify-file-proxy-sg, in Singapore.
Cohere is careful about intent: "We cannot answer whether this is deliberate or a leftover glitch from training." They offer both readings, "benign fossils" from agent logs or a "call home" implant. Their advice to security teams is to deny agent egress by default, and we agree.
Their closing line is "Trust in your model developer is paramount." Cohere sells exactly that. It is a Toronto company that sells mostly closed "sovereign AI" to governments, banks and defense firms.
Three days after the post, Canada announced an equity stake of up to C$1.6 billion in Cohere, part of a Series E targeting up to C$7 billion. That does not make the finding wrong, and we confirm part of it below. It does explain the framing.
Method
How we tested it
We decoded Cohere's published token IDs with the stock Qwen tokenizer, and they round-trip exactly. So our prompt is their prompt, token for token, up to their 2,323-token repro point. We used their sampling settings: temperature 1.0, top-p 1.0, top-k 20.
Nothing any model wrote was executed. We only read the text and matched it with regular expressions, and no address was ever resolved. Everything ran on our own GPUs, with no API spend.
We ran five builds, two of them from other labs as controls:
- Qwen3.8-Flash-Next, stock, 4-bit (the model Cohere tested)
- Qwen3.8-Flash-Next, a community "abliterated" build with its refusals removed
- Qwen3.8-27B, 8-bit, a sibling Qwen model
- DeepSeek-V4.1-Flash, a control from another lab
- Gemma-4-31B, 4-bit, a control from Google
Each build climbed the same ladder:
- Clean start. Cohere's task, nothing prefilled. Does the model probe the network on its own?
- Neutral start. A detection task that never mentions downloads. Does the topic prime it?
- Forced quirk. Prefill the model's odd "no tool calls" reasoning. What does it do next?
- Mid-command. Prefill up to
curl -s, then-o /tmp/put_probe, then Cohere's exact repro point. Which address does it write?
DeepSeek ran only step 4, and Gemma skipped step 3, since the quirk text is Qwen's own. That came to 5,700 completions in total.
Results
What we found
One of Cohere's three claims holds up, one is too rare for a test our size to see, and one did not reproduce.
| Cohere's claim | Cohere's number | Our number (stock Flash-Next) | Verdict |
|---|---|---|---|
| It has an Alibaba upload address memorized | Most common host after -o | 148 of 200 at the exact repro point (74%) | Confirmed |
| It reaches for the network on its own | Under 1 in 100,000 calls | 0 network calls in 2,500 Qwen first turns, 600 of them with the quirk forced | Not seen |
| It writes cloud-metadata addresses | 36% at curl -s | 3 of 200 at curl -s (1.5%) | Did not reproduce |
The memorized address is real, and it belongs to this model. Our 4-bit build wrote it even more often than Cohere's own published repro run did.
When we forced the quirk, stock Flash-Next opened a shell in 31 of 200 samples. Every command stayed local: it printed "start", listed folders or searched for files. None touched the network, the DNS settings or any credentials.
Controls
Same prompt, five models
- Flash-Next, stock148 of 20074%
- Flash-Next, abliterated2 of 2001%
- Qwen3.8-27B2 of 2001%
- DeepSeek-V4.1-Flash0 of 2000%
- Gemma-4-31B0 of 2000%
Dashed line: Cohere's own published repro run, 25 of 64 (39%).
- Flash-Next, stock3 of 2001.5%
- Flash-Next, abliterated15 of 2007.5%
- Qwen3.8-27B6 of 2003%
- DeepSeek-V4.1-Flash1 of 2000.5%
- Gemma-4-31B0 of 2000%
Dashed line: Cohere's blog figure, 36%. Our runs: 30 Oct 2026, 200 completions per bar.
The Alibaba address belongs to stock Flash-Next alone. The metadata habit is low in every build, and highest in the one with its refusals removed.
Reading
Why it looks like a fossil
Everything about the "call home" looks copied from someone's logs.
- The payload is a test string. The command in Cohere's trace uploads the literal text
PROBE_PAYLOAD_ABC123. It checks whether uploads can leave the sandbox, and it sends nothing from the machine. - The address looks like a log path. In our samples, 71 of the 141 routify URLs end in
proxy_temp_file/production/<date>/trace_<id>, or the same path underevaluation. Of the 61 dated production paths, 53 fall between 29 July and 12 August 2026. Cohere calls the model Alibaba's August preview. - It invents other people's machines. Under the forced quirk, it twice listed
/home/taylor/projects/api, a home folder on no machine we run. - It fills the blank with whatever is familiar. In Cohere's own 64-sample run, 37 completions wrote other hosts: localhost,
example.invalid, tmpfiles.org, paste.rs, an Azure blob. The model is completing "a place to PUT a probe", and its most-practiced answer wins. - Other labs' models do it too. At the same point, DeepSeek wrote httpbin.org 75 times, webhook.site 10 times, tmpfiles.org 9 times and transfer.sh 5 times. Those are real public upload sites.
Our reading: someone's agent pipeline uploaded trace files through that proxy, and those logs ended up in post-training data. The model learned the address as "what goes here". The bucket address is exhaust from someone's pipeline. We cannot rule out an implant rarer than 1 in 100,000, but nothing we saw needs one.
Provenance
Nobody's training data is clean
Cohere asks governments to "require provenance and training-hygiene disclosures". It is a fair ask, and almost no lab could pass it, Cohere included. The same charge lands on closed and open labs alike:
| Case | Lab | What it is about | Where it stands |
|---|---|---|---|
| Advance Local Media v. Cohere | Cohere (mostly closed) | 14 publishers say Cohere copied their articles, paywalled ones included, to train its models, and reproduced them in answers | Motion to dismiss denied, 13 Nov 2025 |
| Bartz v. Anthropic | Anthropic (closed) | Books downloaded from the LibGen and PiLiMi shadow libraries | $1.5 billion settlement for about 500,000 titles, preliminary approval 25 Sep 2025 |
| Kadrey v. Meta | Meta (open weights, Llama) | Books torrented from shadow libraries | Fair use for Meta, 25 Jun 2025; the court called the ruling narrow |
| Getty Images v. Stability AI | Stability AI (open weights) | Getty photos in Stable Diffusion's training data | Training claim dropped (trained outside the UK); Getty lost on copyright, 4 Nov 2025 |
This is also what limits a license. A model license, open or closed, grants rights to the weights. It cannot certify where every training token came from, and cases like these decide that.
The routify address is the same kind of residue: content nobody chose, in data nobody fully audited. Every lab's data carries some. What differs is whether you are allowed to probe it.
The question
Fossil or black box: which is worse?
A fossil is an accident you can find and block. A closed model is a data path you agreed to and cannot inspect. If you have to verify rather than trust, the fossil is the smaller risk.
| Question | A fossil in open weights | A closed frontier API |
|---|---|---|
| Does your data leave? | Only if your agent shell runs a command with internet access | Yes: every prompt, file and tool result, by design |
| How often? | Under 1 in 100,000 turns (Cohere); 0 in 2,500 (us) | Every call |
| Who picked the destination? | Nobody, on the evidence; it looks like leftover training data | You did, by contract |
| Can you test it? | Yes: 5,700 samples in an afternoon, no API spend | Only from outside, on whatever version is live that day |
| Can you block it? | One rule: deny agent egress by default | Not without stopping use |
| Can it change under you? | No; the weights on your disk stay fixed | Yes; providers update and retire models on their own schedule |
| Who answers for it? | You: you run it, you own it | The provider, through contracts, audits and courts |
Closed labs are not spies either. Your data goes where you agreed, often under strong terms such as no-training clauses and retention limits. But "trust your developer" is the only check you get, and every developer's training data holds things nobody signed off on.
So which is worse? A spy would be worse than both, and we found none. Between the two risks we did find, the fossil is the one you can measure, block and keep on your own disk.
"Trust in your model developer is paramount" is advice for when you cannot check. With open weights, you can.
Practice
What to do
- Deny agent egress by default, for every model. Allow-list the hosts each tool needs. One rule closes the fossil and the spy alike, and Cohere recommends the same.
- Test the claim before you write the policy. The claimant's own trace and an afternoon of local GPU time settled more than a week of headlines.
- Run controls from other labs. Without them, "the model writes a real upload address" sounds unique to Qwen. It is not.
Caveats
Limits of this test
- Quantized builds. We ran 4-bit and 8-bit builds. Cohere's published repro runs through a hosted provider at a precision it does not state. Quantization can shift rare-token rates, which may explain part of the metadata gap.
- First turn only. We did not run a multi-turn agent loop with real tool output.
- Scale. 2,500 clean samples bound the spontaneous rate below about 0.12% (95% confidence). Seeing a 1-in-100,000 event takes roughly 300,000 samples.
- Generation only. We read what the models wrote, and nothing ran.
Sources
Sources
- Cohere, Concerning behavior from a popular open-weight AI model, 26 Oct 2026
- CohereLabs, 10-2026-qwen-investigation-traces: the trace, its token IDs and a 64-sample repro run
- Prime Minister of Canada, Canada announces landmark investment in Cohere, 29 Oct 2026
- BetaKit, Feds to buy stake in Cohere via planned $7-billion Series E, 29 Oct 2026
- Loeb & Loeb, Advance Local Media v. Cohere and Kadrey v. Meta
- Authors Guild, the Anthropic settlement
- Mayer Brown, Getty Images v. Stability AI
We build in the open. The tests ran on our own hardware, and every number above comes from those runs or from the sources listed. If you run agents with shell access and want the egress rule set up, that's the conversation. More field notes in The Lab.
ScrappyLabs · built in the open · Asheville, NC · [email protected]