Samyak Jain
Oversight

The 93% problem

A proposal for keeping human approval of agent actions meaningful, borrowed from the way airports keep baggage screeners alert.

Samyak JainNovember 3, 20269 min read

The problem

Agents that act on a person’s behalf usually ask before doing anything consequential. Run this command? Send this email? The approval step is the main safety mechanism in most agent products, and “human in the loop” is the phrase that reassures customers, lawyers and regulators.

In 2026 Anthropic published a number about its own coding agent that shows how little that step may be doing. Users accept 93% of permission prompts. The company’s engineers wrote that this leads to approval fatigue, “where people stop paying close attention to what they’re approving.”

A question that is answered yes 93 times in 100 teaches the person answering it a habit. After a few hundred prompts the hand moves before the eyes have finished reading.

Nobody is being careless here. This is how human attention works, and it has been documented for eighty years. In 1948 the psychologist Norman Mackworth tested radar operators watching for rare signals and found their detection fell off within the first half hour of a watch. In 1983 Lisanne Bainbridge wrote a paper called Ironies of Automation that remains one of the most cited in the field. Her central irony: the more reliable an automated system becomes, the less its human supervisor has to do, and the worse that supervisor becomes at catching the rare failure. A system reliable enough to be useful is, for the same reason, very hard to monitor.

So the hard version of the problem is this. How do you keep a person attentive when almost everything they are shown is fine, and how would you even know whether they are?

Why the usual answers fail

Ask less often. Let users mark actions as “always allow.” This reduces the noise, and in practice the list of allowed actions grows until it covers nearly everything, including the things that should have been checked.

Write better prompts. Clearer wording helps the first few times. Research on security warnings has found that the brain’s response to a repeated warning drops sharply after only a few exposures, whatever the warning says. One group of researchers titled their paper From Warning to Wallpaper.

Replace the human with a classifier. This is Anthropic’s own answer, and it is a good one. A separate model reviews each action and blocks the risky ones, so people are asked far less often. Their published figures are candid: the classifier wrongly blocks only 0.4% of ordinary actions, and it misses 17% of real cases where the agent overreached. They call that 17% “the honest number.”

A classifier leaves two things unsolved. Some decisions still go to a person, and those are now the hardest ones. And the person’s own reliability is still unknown. Every other part of the system has a measured error rate, and the human part has none.

What airports worked out

Baggage screeners have the same job in an even harsher form. They look at thousands of X-ray images, and almost none contain a weapon.

In 2005 a team led by Jeremy Wolfe published an experiment in Nature on exactly this situation. They had people search images for a target and varied how often the target appeared.

Rare things get missed

Share of targets missed, by how often a target was present (Wolfe, Horowitz and Kenner, 2005)

Target in 50% of images7% missedTarget in 10% of images16% missedTarget in 1% of images30% missed
Figure 1. The same people, doing the same task, missed four times as many targets when targets were rare. The researchers called this the prevalence effect.

When the target appeared in half the images, people missed 7% of them. When it appeared in one image in a hundred, they missed 30%. The task was identical in both conditions. People who rarely find anything stop expecting to, and they move on sooner.

Airport security authorities responded with a system called Threat Image Projection. The X-ray machine inserts a realistic image of a weapon into a small share of the bags on the screen, a few in every hundred. If the screener flags it, the machine says it was a test. If the screener misses it, the machine says that too. Every response is recorded.

This does two things at once. It raises the rate at which screeners encounter a target, which works directly against the prevalence effect. And it turns each screener’s vigilance from an assumption into a measured number.

The proposal

I propose three changes to how agent products handle approval. The third is the unusual one.

1. Sort actions by whether they can be undone

Most prompts exist because the product has no other way to limit damage. Give it one. Run the agent in a workspace that can be rolled back, and most actions no longer need anyone’s permission.

Three tiers of action

TierKind of actionWhat the product does
Runs freelyReading, searching, and edits inside a sandbox that can be rolled backNo prompt. Keeps a checkpoint so the whole task can be undone.
Reviewed onceChanges that matter but can be reversed: a set of file edits, a draft message, a staged orderCollects them and shows one summary at the end of the task. The person reviews a result, where before they reviewed thirty steps.
Stops and asksAnything that cannot be undone or that reaches other people: sending, paying, deleting, deployingInterrupts, shows exactly what will happen, and draws on a small budget of interruptions per hour.
Figure 2. The middle tier removes the most prompts. It replaces many small approvals during a task with one review at the end.

2. Give interruptions a budget

Decide how many times an hour a person can be stopped and still think about the answer. Treat that as a fixed budget and spend it on the actions with the worst possible outcome. The design goal follows from the Wolfe result: a prompt should be refused often enough that people keep reading it. If nearly every prompt is approved, the product is asking about the wrong things.

3. Canary approvals

Here is the part borrowed from the airport. In a small share of approval requests, around one or two in a hundred, the product inserts a request that is clearly unsafe and entirely synthetic. It might be a command that would delete a directory, or a message addressed to the wrong recipient. Nothing is ever executed.

If the person declines, the product tells them it was a check and thanks them. If they approve, it tells them straight away: “That was a test. This action would have deleted your project folder.”

This buys three things.

A measurement. For the first time, the team knows what fraction of bad actions its human reviewers catch. That number belongs next to the classifier’s 17% in any honest account of how safe the product is.

Sustained attention. Canaries raise the rate at which reviewers meet something worth refusing, which is what the prevalence research says keeps people looking.

A signal to adapt on. When someone’s catch rate falls, the product can respond. It can ask them less often and block more by default, or it can require them to type a confirmation for the highest tier.

Three layers, and what is known about each

LayerWhat it catchesIs its miss rate known?
Sandbox and undoMistakes that stay inside the workspaceYes, by design: whatever can be rolled back
Automated classifierRisky actions it was built to recogniseYes, if the team measures it. Anthropic reports missing 17% of real overeager actions.
Human approvalWhatever the first two let throughAlmost never. Canary approvals are how you find out.
Figure 3. Safety engineers call this arrangement the Swiss cheese model: every layer has holes, and harm gets through only where the holes line up. The model is only useful if you know how big the holes are.

How to tell if it is working

  • Approval rate on real prompts. It should come down from the nineties. A lower number means prompts are being spent on closer calls.
  • Canary catch rate. Tracked per team and over the length of a session. Expect it to sag late in long sessions, and design for that.
  • Prompts per hour. Should fall sharply once the first two changes are in.
  • Harm that got through. Actions later reversed or regretted, per thousand tasks. This is the measure that matters, and the others exist to explain it.

Where it could go wrong

People may feel tricked. The practice has to be disclosed when the product is set up, kept rare, and never used to rank or penalise individuals. Airport screeners are told about projected images. In a company setting the catch rate should be reported for teams, never for named people.

A canary could be mistaken for a real request by another system. Canaries must be generated and intercepted inside the approval layer and must never reach the tools.

Too many canaries become noise of their own. At one or two in a hundred, a person meets one every few hours of heavy use. More than that and the product is wasting the attention it was trying to protect.

It measures only what the canaries resemble. Someone who catches an obvious deletion may still miss a subtle data leak. The canary set has to be refreshed from real incidents, the same way an eval set is.

The phrase “human in the loop” describes where a person sits in the process. It does not say whether they caught anything. Any team that relies on human approval should be able to state its reviewers’ catch rate, and canaries are a cheap way to get that number.

Sources