Samyak Jain
Lab · No. 01 · Simulator

Agent system designer

Design an agent stage by stage, then watch what happens to ten thousand tasks: how many finish, how many fail quietly, what it costs, and how much it asks of the people supervising it.

An agent is a chain. Every step that can fail multiplies the chance that the whole task fails.

Checks, retries and people change that arithmetic. Each one costs money, time or someone’s attention.

Build a design below. It runs thousands of simulated tasks through it and shows you the trade, live.

Scroll
People
Checks

Where tasks end up

Share of tasks still on track

Where errors get in

Money per task

The person in the loop

What to change next

Optimise for

· Runs entirely in your browser.

Design notes

How the simulation works

Each simulated task walks the stages in order. Every try at a stage succeeds with the probability you set. The model selector moves all of those probabilities together on the log-odds scale: the small model is weaker at everything and the frontier model stronger. Its prices are illustrative and per million tokens.

A stage that fails without anyone noticing leaves the task corrupted. Later stages can still succeed at their own job, but they are now working on wrong material, which is how a mistake at step 3 becomes a wrong refund at step 7.

Hard tasks are hard everywhere

Each task draws a difficulty, and that difficulty shifts every stage and every retry for that task. This is why failures cluster on particular tasks instead of falling like independent dice. The “pass 8 times in 8” tile measures it directly by re-running a sample of tasks eight times each, the way τ-bench does. Turn task variation down to zero and the clustering disappears.

Checks and retries

An automated check catches a real failure with the recall you set, and flags good work by mistake at its false-positive rate. A flagged step is retried. When the retries run out, the task is handed to a person, which is a safe outcome but a costly one. A check that “also tests earlier work” can notice corruption carried in from upstream, but it cannot repair a stage it does not own, so it hands the task off.

People

A stage with a person approves its output before the task continues. How often that person misses a bad request depends on how often bad requests appear, fitted to Wolfe, Horowitz and Kenner (2005): about 7% missed when half of requests are bad, 16% at one in ten, 30% at one in a hundred. The reviewer keeps a running sense of how often things go wrong, and fatigue builds over a shift of two hundred reviews. This is the mechanism behind The 93% problem: putting a person on every step makes each review less likely to catch anything.

Harm and money

An irreversible stage acting on corrupted work counts as harm. Running cost is model usage, reviewer time and hand-offs. Margin also subtracts the failure costs you set for wrong answers and harmful actions. Under per-completion pricing, every task that is not handed to a person is billed, including those that went wrong without anyone noticing, which is how most outcome contracts work in practice and why the definition of an outcome matters.

The advisor

The advisor tries every single change it knows about, one at a time, and re-runs the simulation for each. All runs use the same random tasks, a technique called common random numbers, so a difference between two designs reflects the change itself rather than a luckier sample.

Checked against the arithmetic

Where a result can be worked out by hand, the simulator should agree with it. Your browser ran these checks when this page loaded. A longer suite of fourteen checks runs with the site’s source code.

Running…

What it leaves out

  • Checks that share the model’s blind spots. Here a check’s misses are independent of the step’s failures; in practice a model grading its own work tends to miss the same things.
  • Long-context decay, tool outages, rate limits and learning over time.
  • Real prices and real success rates. Every number is an input. The tool shows the shape of a trade-off; your own measurements belong in the sliders.

Related reading: The half-life of an agent, The 93% problem, The diagonal, Four cents a copy.