Samyak Jain
Reliability

The half-life of an agent

Why an agent that looks finished in a demo fails in production, worked out in arithmetic, and what the arithmetic says about what to ship.

Samyak JainSeptember 29, 20269 min read

In the early years of rocketry, engineers kept running into the same puzzle. Every part of a missile had been tested, and every part worked. The missiles still failed, one after another.

The explanation is credited to Robert Lusser, a German engineer who worked on flying bombs during the war and on American missiles after it. A machine made of parts in a chain works only if every part works. So the chance that the whole machine works is the chance of each part working, multiplied together. Two parts that each work 95% of the time give you a machine that works about 90% of the time. Twenty such parts give you a machine that fails more often than it flies.

Reliability engineers call this Lusser’s law. I think it is the most useful single idea for anyone building products on AI agents, because an agent doing a long task is also a chain. It reads a message, looks something up, picks a tool, fills in the arguments, reads the result, and decides what to do next. Each step can go wrong, and one wrong step can sink the task.

The arithmetic of a long task

Suppose an agent gets each step right 95% of the time. That sounds like an A grade. Over twenty steps, the chance of getting all of them right is 0.95 multiplied by itself twenty times, which is 36%.

Small per-step errors compound quickly

Share of tasks that finish with every step correct, by number of steps

0%25%50%75%100%01020304050Steps in the task99.9% per step99% per step95% per step90% per step20 steps at 95% each: 36%
Figure 1. Each line assumes every step succeeds with the same probability and that one failed step fails the task. Hover over a line to read values.

This is why a demo can take a week and the product can take a year. A demo is one run, on a path the builder picked, repeated until it worked. A product is every run, on paths that users pick. The demo shows that success is possible. The product has to make it routine, and Figure 1 shows how far apart those are.

It also explains a pattern that confuses teams. The agent gets a little better, from 95% to 99% per step, and the product goes from unusable to good. Nothing dramatic happened to the model. The chain simply stopped breaking.

Agents have a half-life

The research group METR measures AI progress in an unusual way. They ask how long a task, measured by the time a skilled person would need, a model can complete half the time. They call it the model’s time horizon. Their 2025 paper found that this horizon had doubled about every seven months for six years, and their later measurements suggest it has been doubling faster since.

The philosopher Toby Ord looked at the same data and noticed something. The results fit a very simple model: for every minute of work, the agent has the same small chance of making a fatal mistake. That is the same maths as radioactive decay. It means each agent has a half-life, a task length at which it succeeds half the time, and the success rate falls off exponentially from there.

Once you accept that model, you can ask a question that matters much more to a product than “what can it do half the time?” You can ask what it can do nine times in ten, or ninety-nine times in a hundred.

What an 8-hour half-life buys at higher reliability

Longest task an agent can be trusted with, if its 50% time horizon is 8 hours

1 min10 min1 hour8 hoursLongest task it can be trusted with (log scale)Half the time8 hours4 times in 52 h 35 min9 times in 1073 min99 times in 1007 min999 times in 1,00042 seconds
Figure 2. Calculated from the constant-failure-rate model in Ord (2025). METR’s own curve fits are a little harsher, putting the 80% horizon at roughly one fifth of the 50% horizon.

The answer is sobering. An agent that can finish an eight-hour task half the time can be trusted at the 99% level with about seven minutes of work. The headline number and the number you can ship are a factor of seventy apart.

I find this the clearest way to explain the gap between what people read about AI and what they experience in products. Both are true. The headline describes the half-life. The product has to live far to the left of it.

Failures are not spread evenly

There is a second finding, and it is better news.

Lusser’s law assumes that failures are independent, like dice. If that were true of agents, running the same task again would be a fresh roll each time. A benchmark called τ-bench tested this. It gives an agent customer-service tasks in a simulated retail store and runs each task many times. In the original paper, GPT-4o solved about 61% of tasks on a single try. If failures were independent, it would solve the same task eight times in a row about 2% of the time. The measured figure was about 25%.

Failures cluster on particular tasks

GPT-4o on τ-bench retail tasks (Yao et al., 2024)

Solved on a single try61%Solved 8 times out of 8, if failures were independent2%Solved 8 times out of 8, as measuredabout 25%
Figure 3. If each attempt were an independent coin flip, 0.61 to the eighth power would give 2%. The measured rate is more than ten times higher, which means some tasks pass almost every time and others fail almost every time.

Twenty-five percent is still a poor number for a product. But the gap between 2% and 25% tells you something important. Failures are concentrated. Some tasks are solved nearly every time. Others are close to a coin flip. Reliability belongs to a model and a task together, and it varies a great deal from one kind of task to the next.

That turns reliability from a property you wait for into a boundary you can find. Somewhere inside the range of things your users ask for is a region the agent handles almost every time. The product question is where the edge of that region is.

Four ways to move the numbers

Put the two findings together and you get a short list of things a product team controls.

Shorten the chain. Every step removed raises the success rate of everything after it. Give the agent one tool that does a whole job in place of three that must be called in order. Cut the task where a person naturally wants to look anyway. A human approval in the middle of a forty-step task turns it into two twenty-step tasks, and the second one starts from a state someone has checked.

Make steps checkable. Checking an answer is usually much cheaper than producing it. If a step can be verified, by a test, a schema, a lookup, or a second model, then a failure becomes a retry and stops being fatal. A step that works 80% of the time, with a reliable check and up to two retries, works more than 99% of the time. Try it below.

Try the arithmetic

Move the sliders to see how step reliability, task length and checked retries combine

36%

of tasks finish without a fatal error

Figure 4. Assumes independent steps and a check that always catches a failed step. Real checks miss some failures, so treat the retry gain as an upper bound.

Measure consistency, then draw the boundary. An average pass rate hides the clustering in Figure 3. Run every task in your evaluation set many times and record how often each task passes every time. The tasks that pass ten out of ten are your product. The ones that pass six out of ten are your roadmap. I would rather ship an agent that does five kinds of request every time than one that does fifty kinds most of the time.

Make failure cheap. The cost of an error depends on what the agent was allowed to do. An agent that drafts a reply for a person to send can be wrong one time in five and still save time. An agent that sends the reply itself needs to be right far more often, and an agent that issues refunds more often again. So autonomy should be granted one task type at a time: first drafting, then acting with approval, then acting alone, each promotion earned by a measured pass rate on that task type. When I worked on agents that handle guest messages for property managers, this ladder was the backbone of the design: drafting first, supervised action next, and autonomous resolution only where the numbers supported it.

The question to ask in a review

When someone shows me an agent now, I ask three things. How many steps does a typical task take? How reliable is each one? Where does the chain reset, through a check or a person?

The answers usually say more than the demo does. They also go out of date. If horizons keep doubling every few months, the reliable region grows with each model release, and a boundary drawn in the spring is too cautious by the autumn. The arithmetic stays the same. The inputs need measuring again each time the model changes.