Hyrum's law for models
A proposal for deciding when a new model is safe to put under a product that people already depend on.
The problem
In August 2025 OpenAI launched GPT-5 and, on the same day, removed the older models from ChatGPT. By the company’s own measures GPT-5 was better. Within days, after a wave of complaints and cancelled subscriptions, it restored GPT-4o for paying users. Sam Altman said that suddenly deprecating models people depended on had been a mistake, and that the company had underestimated how much some of the things people liked in the old model mattered to them.
Every team that builds on a language model meets a smaller version of this several times a year. A new model arrives. It scores higher on the evaluation set. The old one will be retired on a published date, so staying put is not an option for long. The team upgrades, and support tickets start arriving from customers who say the product has got worse.
Software engineers have a name for what is happening. Hyrum Wright, an engineer at Google, observed it while migrating code across the company’s entire codebase, and it is now known as Hyrum’s law:
With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.
A conventional API has a few dozen observable behaviours. A language model has an almost unlimited number: how long its answers are, how it formats a list, when it asks a clarifying question, how readily it refuses, how warm it sounds. None of these were promised. All of them, given enough users, are depended on by somebody.
Why the usual answer fails
The usual launch gate is a comparison of scores. If the new model does better on the eval set, ship it.
An average score throws away the information that matters most for an upgrade, which is which cases changed. Consider two possible upgrades. Both take accuracy from 80% to 84%.
Two upgrades with the same score
What happened to each of 100 test cases
Upgrade A fixes six cases and breaks two. Upgrade B fixes fourteen and breaks ten. A dashboard shows them as identical. Users will not experience them that way. Each broken case is something a customer had built a habit or a workflow around.
This has been studied directly. In 2019 a team from Microsoft Research and the University of Washington ran experiments on people working alongside an AI system and published a result that should be better known among product managers: an update that makes the AI more accurate can make the human-AI team less accurate. People learn when to trust a system and when to check it. An update that introduces errors in places where the old system was reliable breaks that learned trust, and the team does worse even though the model does better. The authors proposed measuring an update’s compatibility: of the cases the old model got right, what share does the new one also get right?
Two years later researchers at Amazon gave the breakages a name, negative flips, and showed that they can be numerous even when the overall error rate falls.
There is a second reason the two upgrades feel different. Losses weigh more than gains. Daniel Kahneman and Amos Tversky’s work on loss aversion found that people typically feel a loss about twice as strongly as an equal gain. A customer does not net the fourteen things that improved against the ten that broke. They notice the ten.
The proposal: a gate with four checks
I propose replacing the score comparison with four checks, each borrowed from a field that has faced the same question.
A launch gate for a model upgrade
| Check | Question it answers | Passes when |
|---|---|---|
| Flip count | How many cases that worked before are now broken? | Broken cases stay under a budget set in advance, overall and for each customer segment |
| Non-inferiority | Could the new model be meaningfully worse on any slice? | The confidence interval on the paired difference stays above the agreed margin on every slice |
| Behaviour diff | What changed that no eval scores? | Length, tone, refusals, format, tool use and speed have been compared on the same traffic and written into a changelog |
| Routing plan | What happens to the slices that fail? | Those requests stay on the old model, with an owner and a date for moving them |
1. Count flips
Run the old and new models on the same cases and sort every case into one of four groups: still right, fixed, broken, still wrong. Then set a compatibility budget before looking at the results: the largest share of previously working cases that may break, overall and within each customer segment.
The segment view matters more than the total. An upgrade that breaks 2% of cases overall may break 15% for the customers who use one particular feature. They will churn, and the total would never have shown why.
2. Test for non-inferiority
Medicine has a well-developed method for approving something that does not need to be better, only safely no worse. It is used when a new treatment is tested against an existing one. The method is called a non-inferiority trial. The margin of acceptable difference is fixed in advance, and the new treatment passes only if the statistical evidence rules out its being worse by more than that margin.
For a model upgrade the equivalent is to set a margin for each slice of the eval set, compare the two models case by case, and require that the confidence interval on the difference stays above the margin everywhere. Comparing the same cases across both models makes the test far more sensitive than comparing two averages, as Anthropic’s researchers have shown in detail.
The change of question is the important part. Asking whether the new model is better invites cherry-picking, because it is easy to find a slice where it wins. Asking whether it could be worse anywhere puts the burden of proof on the upgrade.
3. Diff the behaviours that no eval scores
Hyrum’s law is about everything that can be observed, and an eval set scores only a narrow part of that. So run both models on the same sample of real traffic and compare everything that can be counted: length of response, reading level, how often it refuses, how often it asks a question back, formatting, number of tool calls, speed.
Then write down what changed, in plain language, the way engineers write release notes. “Answers are about 30% shorter. The model now asks for clarification before starting long tasks. It uses tables less often.” Publish it to customers before the change goes live.
Hardly anyone does this, and I think it is the cheapest of the four. A customer who has been told that answers will be shorter tends to adjust. One who finds out by surprise assumes something has broken.
4. Route before you switch
When a slice fails the first two checks, the usual choices are to delay the whole upgrade or to ship and absorb the complaints. There is a third. Send that slice of requests to the old model, and everything else to the new one, while the failing slice is fixed with prompt changes or raised with the model provider.
This makes an upgrade a gradual migration, with a list of slices still to move, an owner for each, and a deadline set by the old model’s retirement date.
How to tell if it is working
- Broken-case rate at each upgrade, overall and by segment. The trend across upgrades shows whether the team is getting better at this.
- Complaints mentioning change. Tickets that say “it used to” in the two weeks after an upgrade.
- Rollbacks. The number of upgrades reversed after launch. The target is zero.
- Time to adopt. Days from a model’s release to carrying most traffic. A good gate should shorten this, because the team stops arguing about whether it is safe.
Where it could go wrong
The budget can freeze the product. Some old behaviours were wrong. A model that agreed too readily with users, or padded its answers, should change, and some customers will have depended on exactly that. The compatibility budget has to allow deliberate breaks. They should be chosen on purpose and announced in the changelog.
Routing costs money and attention. Running two models means two sets of prompts and two sets of failure modes. Without a deadline, the old route becomes permanent.
The eval set may not contain what users depend on. This is the deepest limit. The behaviour diff in the third check is a partial answer. A better one is to add every “it used to” complaint to the eval set as a new case, so that each surprise can only happen once.
A score comparison tells a team whether the new model is better on average. It says nothing about what broke for the customers who relied on the old one, and that is what decides whether an upgrade is safe to ship.
Sources
- The Register, ‘Suddenly deprecating old models’ users depended on a ‘mistake,’ admits OpenAI’s Altman, 2025.
- Hyrum Wright, Hyrum’s Law.
- Bansal et al., Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff, AAAI 2019.
- Yan et al., Positive-Congruent Training: Towards Regression-Free Model Updates, CVPR 2021.
- Kahneman and Tversky, Prospect Theory: An Analysis of Decision under Risk, Econometrica, 1979.
- US Food and Drug Administration, Non-Inferiority Clinical Trials to Establish Effectiveness, 2016.
- Evan Miller, Adding Error Bars to Evals, Anthropic, 2024.