The half-life of an agent
Why an agent that looks finished in a demo fails in production, worked out in arithmetic, and what the arithmetic says about what to ship.
In the early years of rocketry, engineers kept running into the same puzzle. Every part of a missile had been tested, and every part worked. The missiles still failed, one after another.
The explanation is credited to Robert Lusser, a German engineer who worked on flying bombs during the war and on American missiles after it. A machine made of parts in a chain works only if every part works. So the chance that the whole machine works is the chance of each part working, multiplied together. Two parts that each work 95% of the time give you a machine that works about 90% of the time. Twenty such parts give you a machine that fails more often than it flies.
Reliability engineers call this Lusser’s law. I think it is the most useful single idea for anyone building products on AI agents, because an agent doing a long task is also a chain. It reads a message, looks something up, picks a tool, fills in the arguments, reads the result, and decides what to do next. Each step can go wrong, and one wrong step can sink the task.
The arithmetic of a long task
Suppose an agent gets each step right 95% of the time. That sounds like an A grade. Over twenty steps, the chance of getting all of them right is 0.95 multiplied by itself twenty times, which is 36%.
Small per-step errors compound quickly
Share of tasks that finish with every step correct, by number of steps
This is why a demo can take a week and the product can take a year. A demo is one run, on a path the builder picked, repeated until it worked. A product is every run, on paths that users pick. The demo shows that success is possible. The product has to make it routine, and Figure 1 shows how far apart those are.
It also explains a pattern that confuses teams. The agent gets a little better, from 95% to 99% per step, and the product goes from unusable to good. Nothing dramatic happened to the model. The chain simply stopped breaking.
Agents have a half-life
The research group METR measures AI progress in an unusual way. They ask how long a task, measured by the time a skilled person would need, a model can complete half the time. They call it the model’s time horizon. Their 2025 paper found that this horizon had doubled about every seven months for six years, and their later measurements suggest it has been doubling faster since.
The philosopher Toby Ord looked at the same data and noticed something. The results fit a very simple model: for every minute of work, the agent has the same small chance of making a fatal mistake. That is the same maths as radioactive decay. It means each agent has a half-life, a task length at which it succeeds half the time, and the success rate falls off exponentially from there.
Once you accept that model, you can ask a question that matters much more to a product than “what can it do half the time?” You can ask what it can do nine times in ten, or ninety-nine times in a hundred.
What an 8-hour half-life buys at higher reliability
Longest task an agent can be trusted with, if its 50% time horizon is 8 hours
The answer is sobering. An agent that can finish an eight-hour task half the time can be trusted at the 99% level with about seven minutes of work. The headline number and the number you can ship are a factor of seventy apart.
I find this the clearest way to explain the gap between what people read about AI and what they experience in products. Both are true. The headline describes the half-life. The product has to live far to the left of it.
Failures are not spread evenly
There is a second finding, and it is better news.
Lusser’s law assumes that failures are independent, like dice. If that were true of agents, running the same task again would be a fresh roll each time. A benchmark called τ-bench tested this. It gives an agent customer-service tasks in a simulated retail store and runs each task many times. In the original paper, GPT-4o solved about 61% of tasks on a single try. If failures were independent, it would solve the same task eight times in a row about 2% of the time. The measured figure was about 25%.
Failures cluster on particular tasks
GPT-4o on τ-bench retail tasks (Yao et al., 2024)
Twenty-five percent is still a poor number for a product. But the gap between 2% and 25% tells you something important. Failures are concentrated. Some tasks are solved nearly every time. Others are close to a coin flip. Reliability belongs to a model and a task together, and it varies a great deal from one kind of task to the next.
That turns reliability from a property you wait for into a boundary you can find. Somewhere inside the range of things your users ask for is a region the agent handles almost every time. The product question is where the edge of that region is.
Four ways to move the numbers
Put the two findings together and you get a short list of things a product team controls.
Shorten the chain. Every step removed raises the success rate of everything after it. Give the agent one tool that does a whole job in place of three that must be called in order. Cut the task where a person naturally wants to look anyway. A human approval in the middle of a forty-step task turns it into two twenty-step tasks, and the second one starts from a state someone has checked.
Make steps checkable. Checking an answer is usually much cheaper than producing it. If a step can be verified, by a test, a schema, a lookup, or a second model, then a failure becomes a retry and stops being fatal. A step that works 80% of the time, with a reliable check and up to two retries, works more than 99% of the time. Try it below.
Try the arithmetic
Move the sliders to see how step reliability, task length and checked retries combine
of tasks finish without a fatal error
Measure consistency, then draw the boundary. An average pass rate hides the clustering in Figure 3. Run every task in your evaluation set many times and record how often each task passes every time. The tasks that pass ten out of ten are your product. The ones that pass six out of ten are your roadmap. I would rather ship an agent that does five kinds of request every time than one that does fifty kinds most of the time.
Make failure cheap. The cost of an error depends on what the agent was allowed to do. An agent that drafts a reply for a person to send can be wrong one time in five and still save time. An agent that sends the reply itself needs to be right far more often, and an agent that issues refunds more often again. So autonomy should be granted one task type at a time: first drafting, then acting with approval, then acting alone, each promotion earned by a measured pass rate on that task type. When I worked on agents that handle guest messages for property managers, this ladder was the backbone of the design: drafting first, supervised action next, and autonomous resolution only where the numbers supported it.
The question to ask in a review
When someone shows me an agent now, I ask three things. How many steps does a typical task take? How reliable is each one? Where does the chain reset, through a check or a person?
The answers usually say more than the demo does. They also go out of date. If horizons keep doubling every few months, the reliable region grows with each model release, and a boundary drawn in the spring is too cautious by the autumn. The arithmetic stays the same. The inputs need measuring again each time the model changes.
Sources
- Lusser’s law, the product law of reliability for systems in series.
- Kwa et al., Measuring AI Ability to Complete Long Tasks, METR, 2025, and METR’s time horizons page.
- Toby Ord, Is there a half-life for the success rates of AI agents?, 2025.
- Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, 2024.
- METR, notes on the limits of time-horizon measurements, 2026.