What Is Model Distillation — and Why Is Anthropic Banning Accounts Over It?
This week Anthropic announced that new safeguards have cut so-called "distillation attacks" against Claude by about 90% to 99%. Tens of thousands of accounts have been banned, and two Chinese AI labs were named in the announcement. It sounds dramatic — and it is — but the story is less about one company's crackdown and more about the economics of how AI models get built. Here's what distillation actually is, why the reasoning trace is the prize, and what Anthropic's number really means.
Model distillation is a legitimate technique: train a small, cheap model by learning from a big, expensive one. The attack version is doing it to someone else's model through their API, against their rules — bombarding it with questions and harvesting its reasoning to build a cheaper copy. Anthropic says new defenses (a feature called "preserved thinking," a two-stage classifier, and mass account bans) cut what attackers can harvest by 90–99%. The caveat: that's Anthropic's own measurement, not independently verified, and determined attackers will adapt.
Distillation is a legitimate technique — the attack is stealing it
First, the non-scandalous version. Distillation is a standard, openly published machine-learning method: you take a large "teacher" model, have it answer lots of questions, and train a smaller "student" model on those answers. The student ends up cheaper to run while keeping much of the teacher's ability. Plenty of respectable open models were built this way, with the teacher's permission or using openly licensed models as teachers.
The attack version skips the permission. A rival (or a proxy network acting for one) signs up for API access to a frontier model like Claude, fires hundreds of thousands of carefully chosen questions at it, and collects the outputs as training data for its own model. It's not hacking in the break-in sense — it's using the front door at industrial scale, for a purpose the terms of service forbid. That's what Anthropic means by a "distillation attack," and it's why accounts get banned rather than prosecuted: the enforcement lever is the API contract.
How a distillation attack works
The playbook, as described in Anthropic's announcement and the reporting around it, goes like this. Attackers create many API accounts and route traffic through proxy networks so the volume doesn't look like one user. They probe the model with questions designed to elicit long, detailed reasoning — not just the final answer, but the step-by-step thinking that produced it. The harvested material is then used to train a competing model at a fraction of the original training cost.
Two details from this week's announcement show how industrial this got. Anthropic named the Chinese startup Zhipu (also known as Z.ai), saying it attempted to distill cyber capabilities from Claude while developing its GLM-5.3 model, and said Moonshot's Kimi models "secretly forwarded" roughly 300,000 user requests to Claude in ten days — including, Anthropic claims, surveillance-analysis requests potentially linked to China's military. (Z.ai did not immediately respond to requests for comment.) Whether or not every allegation holds up, the scale is the point: this isn't a hobbyist copying homework. It's state-adjacent industrial espionage dressed up as API usage.
Why the reasoning trace is the prize
The most revealing number in Anthropic's announcement isn't the 90–99%. It's this: a model distilled from Claude's full reasoning transcripts scored 80.4% on SWE-bench Verified, a standard coding benchmark, versus 71.8% when distilled from only summarized reasoning.
That's a big gap, and it explains the whole conflict. The final answer to a hard question is worth something. The full chain of thought that produced it — the false starts, the verification steps, the way a strong model checks its own work — is worth far more as training data. It's the difference between copying someone's exam answers and copying their study method. Attackers weren't after Claude's answers. They were after how Claude thinks.
This is also why the current AI moat conversation has shifted. If the valuable part were just the model weights, stealing outputs wouldn't help much. But if the moat is the training environments, the reinforcement-learning pipelines, and the reasoning behavior they produce — then harvesting reasoning traces is harvesting the moat itself. Copying the answers only gets you so far; copying the reasoning gets you most of the way.
What Anthropic just did about it
The defense bundle, rolled into Claude Fable 5.1 and Opus 5.5, has four main parts:
Preserved thinking. New API accounts — created after August 31, 2026 — can no longer edit Claude's prior context in a conversation. That editing trick was reportedly the technique attackers used to pull long reasoning transcripts out of the model. Closing it doesn't stop questions; it stops the transcript-extraction maneuver.
A two-stage classifier. It now covers Sonnet 5.5, Opus 5 and 5.5, and Fable 5 and 5.1, flagging suspected extraction campaigns. Anthropic claims a false-positive rate below 0.1% and says the system is "roughly as robust as those that prevent Claude from helping build bioweapons" — a striking comparison that tells you how seriously they take this. Repeat attempters get stricter rate limits, and associated accounts are banned, Anthropic says, often within two hours of a campaign being detected.
Mass bans. Tens of thousands of accounts linked to distillation campaigns have been banned outright.
Making attacks expensive. Since September 24, requests the classifier blocks with high confidence have been billed — so attackers pay for the privilege of being stopped. Combined with proxy-network countermeasures, the goal isn't perfect prevention; it's raising the cost of extraction until it stops being economical.
What the 90–99% number actually means
Here's the honest reading. The figure comes from Anthropic's own measurements: after the new classifier went in, "the number of reasoning tokens extracted by distillers" fell to 10 to 100 times lower than before. A 10x drop is 90%; a 100x drop is 99%. That's where the range comes from.
What it doesn't establish: that all unauthorized distillation has stopped, that competitors were prevented from learning from Claude's outputs in every case, or that the same reduction applies to extraction attempts run through other providers. No third party has verified the number — it's Anthropic's own report on its own platform. And Anthropic itself concedes it hasn't yet observed a highly sophisticated, persistent attempt to get around the new defenses, and expects its systems will need continuous updates as attackers adapt. Determined attackers can still create new accounts, change their prompts, or spread traffic across providers.
So treat it as evidence of improved resistance under Anthropic's measurements, not proof of a permanent fix. The realistic outcome isn't the end of distillation attacks — it's an arms race where extraction gets more expensive and the easiest, laziest campaigns die first.
What this means for you
If you use Claude normally, almost nothing changes — except a small chance of tripping the classifier. Anthropic acknowledges occasional false positives for legitimate users, with a claimed rate under 0.1%. If your API usage ever gets flagged or rate-limited unexpectedly, that's the system most likely responsible, and it's worth knowing it exists before you assume something's broken.
The bigger takeaway is about where AI value actually lives now. Training a frontier model costs a fortune; using one through an API costs pennies per question. That gap is the economic engine of distillation attacks — and it's why every leading lab (Anthropic, OpenAI, Google) is now treating output-harvesting as a competitive and national-security issue, not just a terms-of-service violation. The models are getting cheaper to copy and more expensive to build. Expect the defenses — and the accusations — to keep escalating.
Frequently asked questions
Is model distillation illegal?
Not inherently — it's a standard ML technique. The "attack" framing is about doing it against a provider's terms of service: using their API to harvest training data for a competing model. Enforcement so far is contractual (account bans), not criminal, though Anthropic has framed some campaigns as national-security risks.
Could my normal Claude usage get flagged as a distillation attack?
Unlikely but not impossible. Anthropic claims a false-positive rate below 0.1% and acknowledges legitimate users are occasionally affected. If you suddenly hit unexplained rate limits or a ban on API usage, contact support rather than creating new accounts to dodge it — that pattern looks exactly like what the classifier hunts.
Do other AI labs face the same problem?
Yes. OpenAI and Google have made the same argument: that rivals use their models' outputs to reproduce advanced capabilities faster and cheaper. The US–China competition angle means this is now an industry-wide posture, not an Anthropic-only concern.
Will this stop cheap AI copycats?
No — it raises their costs. Anthropic's own framing is about making extraction uneconomical, not impossible. Determined attackers adapt: new accounts, different prompts, distributed traffic. Expect a continuing arms race rather than a final victory.