Agent system designer
Design an agent stage by stage, then watch what happens to ten thousand tasks: how many finish, how many fail quietly, what it costs, and how much it asks of the people supervising it.
An agent is a chain. Every step that can fail multiplies the chance that the whole task fails.
Checks, retries and people change that arithmetic. Each one costs money, time or someone’s attention.
Build a design below. It runs thousands of simulated tasks through it and shows you the trade, live.
Where tasks end up
Share of tasks still on track
Where errors get in
Money per task
The person in the loop
What to change next
Pinned designs
· Runs entirely in your browser.
Design notes
How the simulation works
Each simulated task walks the stages in order. Every try at a stage succeeds with the probability you set. The model selector moves all of those probabilities together on the log-odds scale: the small model is weaker at everything and the frontier model stronger. Its prices are illustrative and per million tokens.
A stage that fails without anyone noticing leaves the task corrupted. Later stages can still succeed at their own job, but they are now working on wrong material, which is how a mistake at step 3 becomes a wrong refund at step 7.
Hard tasks are hard everywhere
Each task draws a difficulty, and that difficulty shifts every stage and every retry for that task. This is why failures cluster on particular tasks instead of falling like independent dice. The “pass 8 times in 8” tile measures it directly by re-running a sample of tasks eight times each, the way τ-bench does. Turn task variation down to zero and the clustering disappears.
Checks and retries
An automated check catches a real failure with the recall you set, and flags good work by mistake at its false-positive rate. A flagged step is retried. When the retries run out, the task is handed to a person, which is a safe outcome but a costly one. A check that “also tests earlier work” can notice corruption carried in from upstream, but it cannot repair a stage it does not own, so it hands the task off.
People
A stage with a person approves its output before the task continues. How often that person misses a bad request depends on how often bad requests appear, fitted to Wolfe, Horowitz and Kenner (2005): about 7% missed when half of requests are bad, 16% at one in ten, 30% at one in a hundred. The reviewer keeps a running sense of how often things go wrong, and fatigue builds over a shift of two hundred reviews. This is the mechanism behind The 93% problem: putting a person on every step makes each review less likely to catch anything.
Harm and money
An irreversible stage acting on corrupted work counts as harm. Running cost is model usage, reviewer time and hand-offs. Margin also subtracts the failure costs you set for wrong answers and harmful actions. Under per-completion pricing, every task that is not handed to a person is billed, including those that went wrong without anyone noticing, which is how most outcome contracts work in practice and why the definition of an outcome matters.
The advisor
The advisor tries every single change it knows about, one at a time, and re-runs the simulation for each. All runs use the same random tasks, a technique called common random numbers, so a difference between two designs reflects the change itself rather than a luckier sample.
Checked against the arithmetic
Where a result can be worked out by hand, the simulator should agree with it. Your browser ran these checks when this page loaded. A longer suite of fourteen checks runs with the site’s source code.
Running…
What it leaves out
- Checks that share the model’s blind spots. Here a check’s misses are independent of the step’s failures; in practice a model grading its own work tends to miss the same things.
- Long-context decay, tool outages, rate limits and learning over time.
- Real prices and real success rates. Every number is an input. The tool shows the shape of a trade-off; your own measurements belong in the sliders.
Related reading: The half-life of an agent, The 93% problem, The diagonal, Four cents a copy.