The 93% problem
A proposal for keeping human approval of agent actions meaningful, borrowed from the way airports keep baggage screeners alert.
The problem
Agents that act on a person’s behalf usually ask before doing anything consequential. Run this command? Send this email? The approval step is the main safety mechanism in most agent products, and “human in the loop” is the phrase that reassures customers, lawyers and regulators.
In 2026 Anthropic published a number about its own coding agent that shows how little that step may be doing. Users accept 93% of permission prompts. The company’s engineers wrote that this leads to approval fatigue, “where people stop paying close attention to what they’re approving.”
A question that is answered yes 93 times in 100 teaches the person answering it a habit. After a few hundred prompts the hand moves before the eyes have finished reading.
Nobody is being careless here. This is how human attention works, and it has been documented for eighty years. In 1948 the psychologist Norman Mackworth tested radar operators watching for rare signals and found their detection fell off within the first half hour of a watch. In 1983 Lisanne Bainbridge wrote a paper called Ironies of Automation that remains one of the most cited in the field. Her central irony: the more reliable an automated system becomes, the less its human supervisor has to do, and the worse that supervisor becomes at catching the rare failure. A system reliable enough to be useful is, for the same reason, very hard to monitor.
So the hard version of the problem is this. How do you keep a person attentive when almost everything they are shown is fine, and how would you even know whether they are?
Why the usual answers fail
Ask less often. Let users mark actions as “always allow.” This reduces the noise, and in practice the list of allowed actions grows until it covers nearly everything, including the things that should have been checked.
Write better prompts. Clearer wording helps the first few times. Research on security warnings has found that the brain’s response to a repeated warning drops sharply after only a few exposures, whatever the warning says. One group of researchers titled their paper From Warning to Wallpaper.
Replace the human with a classifier. This is Anthropic’s own answer, and it is a good one. A separate model reviews each action and blocks the risky ones, so people are asked far less often. Their published figures are candid: the classifier wrongly blocks only 0.4% of ordinary actions, and it misses 17% of real cases where the agent overreached. They call that 17% “the honest number.”
A classifier leaves two things unsolved. Some decisions still go to a person, and those are now the hardest ones. And the person’s own reliability is still unknown. Every other part of the system has a measured error rate, and the human part has none.
What airports worked out
Baggage screeners have the same job in an even harsher form. They look at thousands of X-ray images, and almost none contain a weapon.
In 2005 a team led by Jeremy Wolfe published an experiment in Nature on exactly this situation. They had people search images for a target and varied how often the target appeared.
Rare things get missed
Share of targets missed, by how often a target was present (Wolfe, Horowitz and Kenner, 2005)
When the target appeared in half the images, people missed 7% of them. When it appeared in one image in a hundred, they missed 30%. The task was identical in both conditions. People who rarely find anything stop expecting to, and they move on sooner.
Airport security authorities responded with a system called Threat Image Projection. The X-ray machine inserts a realistic image of a weapon into a small share of the bags on the screen, a few in every hundred. If the screener flags it, the machine says it was a test. If the screener misses it, the machine says that too. Every response is recorded.
This does two things at once. It raises the rate at which screeners encounter a target, which works directly against the prevalence effect. And it turns each screener’s vigilance from an assumption into a measured number.
The proposal
I propose three changes to how agent products handle approval. The third is the unusual one.
1. Sort actions by whether they can be undone
Most prompts exist because the product has no other way to limit damage. Give it one. Run the agent in a workspace that can be rolled back, and most actions no longer need anyone’s permission.
Three tiers of action
| Tier | Kind of action | What the product does |
|---|---|---|
| Runs freely | Reading, searching, and edits inside a sandbox that can be rolled back | No prompt. Keeps a checkpoint so the whole task can be undone. |
| Reviewed once | Changes that matter but can be reversed: a set of file edits, a draft message, a staged order | Collects them and shows one summary at the end of the task. The person reviews a result, where before they reviewed thirty steps. |
| Stops and asks | Anything that cannot be undone or that reaches other people: sending, paying, deleting, deploying | Interrupts, shows exactly what will happen, and draws on a small budget of interruptions per hour. |
2. Give interruptions a budget
Decide how many times an hour a person can be stopped and still think about the answer. Treat that as a fixed budget and spend it on the actions with the worst possible outcome. The design goal follows from the Wolfe result: a prompt should be refused often enough that people keep reading it. If nearly every prompt is approved, the product is asking about the wrong things.
3. Canary approvals
Here is the part borrowed from the airport. In a small share of approval requests, around one or two in a hundred, the product inserts a request that is clearly unsafe and entirely synthetic. It might be a command that would delete a directory, or a message addressed to the wrong recipient. Nothing is ever executed.
If the person declines, the product tells them it was a check and thanks them. If they approve, it tells them straight away: “That was a test. This action would have deleted your project folder.”
This buys three things.
A measurement. For the first time, the team knows what fraction of bad actions its human reviewers catch. That number belongs next to the classifier’s 17% in any honest account of how safe the product is.
Sustained attention. Canaries raise the rate at which reviewers meet something worth refusing, which is what the prevalence research says keeps people looking.
A signal to adapt on. When someone’s catch rate falls, the product can respond. It can ask them less often and block more by default, or it can require them to type a confirmation for the highest tier.
Three layers, and what is known about each
| Layer | What it catches | Is its miss rate known? |
|---|---|---|
| Sandbox and undo | Mistakes that stay inside the workspace | Yes, by design: whatever can be rolled back |
| Automated classifier | Risky actions it was built to recognise | Yes, if the team measures it. Anthropic reports missing 17% of real overeager actions. |
| Human approval | Whatever the first two let through | Almost never. Canary approvals are how you find out. |
How to tell if it is working
- Approval rate on real prompts. It should come down from the nineties. A lower number means prompts are being spent on closer calls.
- Canary catch rate. Tracked per team and over the length of a session. Expect it to sag late in long sessions, and design for that.
- Prompts per hour. Should fall sharply once the first two changes are in.
- Harm that got through. Actions later reversed or regretted, per thousand tasks. This is the measure that matters, and the others exist to explain it.
Where it could go wrong
People may feel tricked. The practice has to be disclosed when the product is set up, kept rare, and never used to rank or penalise individuals. Airport screeners are told about projected images. In a company setting the catch rate should be reported for teams, never for named people.
A canary could be mistaken for a real request by another system. Canaries must be generated and intercepted inside the approval layer and must never reach the tools.
Too many canaries become noise of their own. At one or two in a hundred, a person meets one every few hours of heavy use. More than that and the product is wasting the attention it was trying to protect.
It measures only what the canaries resemble. Someone who catches an obvious deletion may still miss a subtle data leak. The canary set has to be refreshed from real incidents, the same way an eval set is.
The phrase “human in the loop” describes where a person sits in the process. It does not say whether they caught anything. Any team that relies on human approval should be able to state its reviewers’ catch rate, and canaries are a cheap way to get that number.
Sources
- Anthropic, Claude Code auto mode: a safer way to skip permissions, 2026.
- N. H. Mackworth, The breakdown of vigilance during prolonged visual search, Quarterly Journal of Experimental Psychology, 1948.
- Lisanne Bainbridge, Ironies of Automation, Automatica, 1983.
- Anderson et al., From Warning to Wallpaper: Why the Brain Habituates to Security Warnings and What Can Be Done About It, Journal of Management Information Systems, 2016.
- Wolfe, Horowitz and Kenner, Rare items often missed in visual search, Nature, 2005.
- Hofer and Schwaninger, Using threat image projection data for assessing individual screener performance, 2005.
- James Reason, Human Error, Cambridge University Press, 1990.