Samyak Jain
Developer platform

The smallest failing trace

A proposal for what an agent platform should hand a developer when a run goes wrong, borrowed from a twenty-year-old technique for finding bugs.

Samyak JainNovember 7, 20268 min read

The problem

A developer has built a support agent on a model platform. A customer reports that it refunded the wrong order. The developer opens the trace of that run. It is eighty steps long: messages, searches, tool results, the model’s reasoning, more tool results. Somewhere in there is the cause.

The developer runs the same request again to watch it fail, and this time it works.

Agent failures differ from ordinary software bugs in three ways, and together they make the usual debugging habits useless.

They do not repeat. The same input can succeed four times and fail the fifth. A developer cannot fix what they cannot reproduce, and cannot tell whether a fix worked if the failure was intermittent to begin with.

The cause is far from the symptom. The wrong refund happened at step 61. The decision that led to it may have been made at step 12, when the agent misread a search result and carried the mistake forward.

Blame is spread across parts. The fault might lie in the instructions, a tool’s description, a document that was retrieved, the model, or the way these combined. Each has a different owner and a different fix.

Why the usual answer fails

The standard answer from platforms is observability. Record everything, and show it on a timeline with every step, its inputs and outputs, and how long it took.

This is necessary, and it leaves the developer where they started. They now have all the data and still have to read eighty steps and guess. A timeline shows what happened. It does not show which of the things that happened mattered.

The developer needs a much shorter answer: which few things, out of everything in that run, were enough to cause the failure?

What software testers worked out

Software engineers faced this in the 1990s with a different kind of program. A web browser would crash on a particular page, and the page was thousands of lines long. Somewhere in it were the few characters responsible.

Andreas Zeller and Ralf Hildebrandt described a way to find them automatically, in a 2002 paper called Simplifying and Isolating Failure-Inducing Input. The method, known as delta debugging, is simple to state. Cut the input in half and test each half. If one half still fails, keep it and throw the other away. If neither does, try removing smaller pieces. Repeat until nothing more can be removed without the failure going away.

What delta debugging did to two real bug reports

Size of the failing input before and after automatic shrinking (Zeller and Hildebrandt, 2002)

A browser crash95 user actions3 still reproduce the failureA page that broke printing896 lines of HTML1 still reproduces the failure
Figure 1. Both examples are from the Mozilla browser. In the second, an 896-line page was reduced to a single tag.

In their tests on the Mozilla browser, a crash that took 95 user actions to trigger was reduced to three. A page of 896 lines that broke printing was reduced to one. The method needs no understanding of why the program fails. It needs only the ability to run the program again and see whether it still fails.

The same idea is now standard in software testing. Tools in the QuickCheck family generate random test inputs, and when one fails, they shrink it to the smallest example before showing it to the developer.

The proposal: shrink every failure before anyone reads it

I propose that an agent platform do this for agent runs. When a run is marked as failed, the platform should produce the smallest version of it that still fails, and say which part is responsible. The process has four steps.

Four steps

1. Record

Save every input the model saw and every result each tool returned.So thatAny run can be replayed from any step, with the outside world held still.

2. Repeat

Replay the failing run twenty times from the start.So thatThe failure becomes a rate, such as 14 in 20, which later steps can compare against.

3. Shrink

Remove parts of the context and replay. Keep each removal that leaves the failure rate unchanged.So thatWhat remains is the small set of things the failure needs.

4. Swap

Replace one remaining piece at a time with its version from a run that passed.So thatThe piece whose replacement cures the failure is the cause.
From an 80-step trace to a named cause
Figure 2. Steps 3 and 4 are the two halves of the original delta debugging method: shrink what fails, then isolate the difference from what passes.

1. Record enough to replay

Save what the model was shown at each step and what each tool returned. With that, any run can be started again from any point, with the tools’ answers played back from the recording. The customer database is not touched and no refund is issued a second time. Engineers call this record-and-replay, and it has long been used to debug ordinary programs whose behaviour depends on timing.

2. Turn the failure into a rate

Because the model’s output varies, replaying once proves nothing. Replay the run twenty times and count. “Fails in 14 of 20” is a measurement, and every later step is a comparison against it.

This changes what a bug report is. It stops being an anecdote about one bad run and becomes a number that a fix either moves or does not.

3. Shrink the context

Now remove things and replay. Drop the first half of the conversation history. Drop tools the agent never called. Drop retrieved documents one group at a time. Whenever the failure rate holds, the removed material was irrelevant, and it stays out.

What survives is the minimal failing context: the handful of items that are together enough to produce the failure. Anthropic’s engineers have written about context rot, the way a model’s accuracy degrades as its context fills up. Shrinking gives a direct test of whether that is what happened. If the failure disappears as soon as the context is cut down, the cause was the volume of material, and no single item is to blame.

4. Swap to find the cause

Take a run of the same task that passed. For each item in the minimal failing context, substitute the version from the passing run, or an obvious alternative: a different model, a revised tool description, a narrower search. Replay twenty times each. The substitution that makes the failure rate collapse identifies the cause.

The output is a report a developer can read in a minute.

What the developer receives

An illustrative failure report

FieldExample
FailureRefund issued for the wrong order
How oftenFails in 14 of 20 replays of the original run
Smallest failing context3 of 81 items: the customer's message, the result of the order search, the description of the refund tool
Where it goes wrongStep 12. The order search returned 40 orders and the agent took the first
What cures itWith the search limited to the last 30 days: fails in 1 of 20. With a different model: fails in 12 of 20
Likely causeThe search tool, not the model or the prompt
Test caseThe three-item context, saved to the evaluation set
Figure 3. The last row matters as much as the diagnosis. Every shrunk failure is a ready-made test case.

The final row is easy to overlook. A minimal failing context is exactly what an evaluation set needs: small, cheap to run, and aimed at one specific weakness. A platform that shrinks failures is also building its customers’ test suites for them.

The obvious objection: cost

This uses a lot of model calls. Shrinking eighty steps at twenty replays per test could mean thousands of runs for one failure.

Three things bring it within reach. Most of each replay repeats a prefix the platform has already processed, and cached input is billed at a fraction of the normal price. Nobody is waiting for the result, so the work can run through batch processing at half price. And shrinking can work coarsely, removing whole steps before it considers anything finer, and stop testing a candidate as soon as the answer is clear.

The comparison that matters is with the cost of the alternative, which is an engineer’s afternoon. I would also not run it on every failure. Group similar failures first and shrink one from each group.

How to tell if it is working

  • Time to diagnosis. From a failure being reported to a cause being named.
  • Shrink ratio. The size of the minimal context relative to the original. Large ratios mean the method is finding real structure.
  • Share of failures with a named cause. Some will not shrink. Track how many do.
  • Failures converted to tests. How many shrunk cases are added to customers’ evaluation sets.
  • Repeat failures. The same cause appearing again after a fix. This should fall.

Where it could go wrong

Some failures need everything. If a failure depends on the whole run, the method will report that nothing can be removed. That is still information, since it points to the length of the context as the problem.

Some tools cannot be replayed. A tool that returns the current time, or that depends on what an earlier call changed, will not behave the same from a recording. The platform has to record results faithfully and flag the steps it cannot reproduce.

Recorded runs contain customer data. Replaying them means keeping them. Retention limits, redaction and access controls have to be designed in from the beginning.

A cause in one run may not be the cause in general. One shrunk trace explains one failure. It takes a cluster of them pointing at the same tool before anyone should rewrite it.

A trace viewer shows the developer everything that happened in a run. What would save them the most time is the reverse, a report of the few things that had to be present for the run to fail.

Sources