Solving AI product problems
Each post takes one problem that AI product teams are stuck on, explains why the usual answers fail, and proposes a specific design, with the research behind it and a way to check whether it worked.
Case law for model behaviour
A research team asks you to make the model better at science. Before anyone can build an eval, someone has to say what better means, and the people who know best disagree. A method that treats their disagreement as the raw material for the spec.
Busy signals
Flat subscriptions get wrecked by a small number of very heavy users. Meters and caps punish the best customers. A pricing design built on one observation, that the runaway usage comes from work nobody is watching.
The diagonal
Acceptance rates reward rubber-stamping, time-in-product falls when an agent works, and surveys measure feelings. A metric that compares how much people rely on an agent with how much it has earned, task by task.
The 93% problem
People approve almost everything an agent asks permission for, so the approval step protects very little. How to design human oversight whose miss rate you can measure, using a technique from airport security.
The smallest failing trace
An agent fails one run in twenty. The trace is eighty steps long, and when the developer runs it again it works. Why tracing dashboards do not solve this, and how a platform could shrink every failure to the few things that cause it.
Hyrum's law for models
The new model scores higher on every eval, and users say the product got worse. Why average scores are the wrong test for an upgrade, and a launch gate built on counting what broke.