The smallest failing trace
A proposal for what an agent platform should hand a developer when a run goes wrong, borrowed from a twenty-year-old technique for finding bugs.
The problem
A developer has built a support agent on a model platform. A customer reports that it refunded the wrong order. The developer opens the trace of that run. It is eighty steps long: messages, searches, tool results, the model’s reasoning, more tool results. Somewhere in there is the cause.
The developer runs the same request again to watch it fail, and this time it works.
Agent failures differ from ordinary software bugs in three ways, and together they make the usual debugging habits useless.
They do not repeat. The same input can succeed four times and fail the fifth. A developer cannot fix what they cannot reproduce, and cannot tell whether a fix worked if the failure was intermittent to begin with.
The cause is far from the symptom. The wrong refund happened at step 61. The decision that led to it may have been made at step 12, when the agent misread a search result and carried the mistake forward.
Blame is spread across parts. The fault might lie in the instructions, a tool’s description, a document that was retrieved, the model, or the way these combined. Each has a different owner and a different fix.
Why the usual answer fails
The standard answer from platforms is observability. Record everything, and show it on a timeline with every step, its inputs and outputs, and how long it took.
This is necessary, and it leaves the developer where they started. They now have all the data and still have to read eighty steps and guess. A timeline shows what happened. It does not show which of the things that happened mattered.
The developer needs a much shorter answer: which few things, out of everything in that run, were enough to cause the failure?
What software testers worked out
Software engineers faced this in the 1990s with a different kind of program. A web browser would crash on a particular page, and the page was thousands of lines long. Somewhere in it were the few characters responsible.
Andreas Zeller and Ralf Hildebrandt described a way to find them automatically, in a 2002 paper called Simplifying and Isolating Failure-Inducing Input. The method, known as delta debugging, is simple to state. Cut the input in half and test each half. If one half still fails, keep it and throw the other away. If neither does, try removing smaller pieces. Repeat until nothing more can be removed without the failure going away.
What delta debugging did to two real bug reports
Size of the failing input before and after automatic shrinking (Zeller and Hildebrandt, 2002)
In their tests on the Mozilla browser, a crash that took 95 user actions to trigger was reduced to three. A page of 896 lines that broke printing was reduced to one. The method needs no understanding of why the program fails. It needs only the ability to run the program again and see whether it still fails.
The same idea is now standard in software testing. Tools in the QuickCheck family generate random test inputs, and when one fails, they shrink it to the smallest example before showing it to the developer.
The proposal: shrink every failure before anyone reads it
I propose that an agent platform do this for agent runs. When a run is marked as failed, the platform should produce the smallest version of it that still fails, and say which part is responsible. The process has four steps.
Four steps
1. Record
Save every input the model saw and every result each tool returned.So thatAny run can be replayed from any step, with the outside world held still.2. Repeat
Replay the failing run twenty times from the start.So thatThe failure becomes a rate, such as 14 in 20, which later steps can compare against.3. Shrink
Remove parts of the context and replay. Keep each removal that leaves the failure rate unchanged.So thatWhat remains is the small set of things the failure needs.4. Swap
Replace one remaining piece at a time with its version from a run that passed.So thatThe piece whose replacement cures the failure is the cause.1. Record enough to replay
Save what the model was shown at each step and what each tool returned. With that, any run can be started again from any point, with the tools’ answers played back from the recording. The customer database is not touched and no refund is issued a second time. Engineers call this record-and-replay, and it has long been used to debug ordinary programs whose behaviour depends on timing.
2. Turn the failure into a rate
Because the model’s output varies, replaying once proves nothing. Replay the run twenty times and count. “Fails in 14 of 20” is a measurement, and every later step is a comparison against it.
This changes what a bug report is. It stops being an anecdote about one bad run and becomes a number that a fix either moves or does not.
3. Shrink the context
Now remove things and replay. Drop the first half of the conversation history. Drop tools the agent never called. Drop retrieved documents one group at a time. Whenever the failure rate holds, the removed material was irrelevant, and it stays out.
What survives is the minimal failing context: the handful of items that are together enough to produce the failure. Anthropic’s engineers have written about context rot, the way a model’s accuracy degrades as its context fills up. Shrinking gives a direct test of whether that is what happened. If the failure disappears as soon as the context is cut down, the cause was the volume of material, and no single item is to blame.
4. Swap to find the cause
Take a run of the same task that passed. For each item in the minimal failing context, substitute the version from the passing run, or an obvious alternative: a different model, a revised tool description, a narrower search. Replay twenty times each. The substitution that makes the failure rate collapse identifies the cause.
The output is a report a developer can read in a minute.
What the developer receives
An illustrative failure report
| Field | Example |
|---|---|
| Failure | Refund issued for the wrong order |
| How often | Fails in 14 of 20 replays of the original run |
| Smallest failing context | 3 of 81 items: the customer's message, the result of the order search, the description of the refund tool |
| Where it goes wrong | Step 12. The order search returned 40 orders and the agent took the first |
| What cures it | With the search limited to the last 30 days: fails in 1 of 20. With a different model: fails in 12 of 20 |
| Likely cause | The search tool, not the model or the prompt |
| Test case | The three-item context, saved to the evaluation set |
The final row is easy to overlook. A minimal failing context is exactly what an evaluation set needs: small, cheap to run, and aimed at one specific weakness. A platform that shrinks failures is also building its customers’ test suites for them.
The obvious objection: cost
This uses a lot of model calls. Shrinking eighty steps at twenty replays per test could mean thousands of runs for one failure.
Three things bring it within reach. Most of each replay repeats a prefix the platform has already processed, and cached input is billed at a fraction of the normal price. Nobody is waiting for the result, so the work can run through batch processing at half price. And shrinking can work coarsely, removing whole steps before it considers anything finer, and stop testing a candidate as soon as the answer is clear.
The comparison that matters is with the cost of the alternative, which is an engineer’s afternoon. I would also not run it on every failure. Group similar failures first and shrink one from each group.
How to tell if it is working
- Time to diagnosis. From a failure being reported to a cause being named.
- Shrink ratio. The size of the minimal context relative to the original. Large ratios mean the method is finding real structure.
- Share of failures with a named cause. Some will not shrink. Track how many do.
- Failures converted to tests. How many shrunk cases are added to customers’ evaluation sets.
- Repeat failures. The same cause appearing again after a fix. This should fall.
Where it could go wrong
Some failures need everything. If a failure depends on the whole run, the method will report that nothing can be removed. That is still information, since it points to the length of the context as the problem.
Some tools cannot be replayed. A tool that returns the current time, or that depends on what an earlier call changed, will not behave the same from a recording. The platform has to record results faithfully and flag the steps it cannot reproduce.
Recorded runs contain customer data. Replaying them means keeping them. Retention limits, redaction and access controls have to be designed in from the beginning.
A cause in one run may not be the cause in general. One shrunk trace explains one failure. It takes a cluster of them pointing at the same tool before anyone should rewrite it.
A trace viewer shows the developer everything that happened in a run. What would save them the most time is the reverse, a report of the few things that had to be present for the run to fail.
Sources
- Zeller and Hildebrandt, Simplifying and Isolating Failure-Inducing Input, IEEE Transactions on Software Engineering, 2002.
- Claessen and Hughes, QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs, ICFP 2000.
- Anthropic, Effective context engineering for AI agents, 2025.
- Yao et al., τ-bench, 2024, on measuring agent consistency across repeated trials.