Case law for model behaviour
A proposal for specifying what "better" means when the experts cannot agree and nobody can write the rubric in advance.
The problem
A product manager working with a research team gets a request that sounds simple: make the model better at science. Or at legal drafting, or at customer support. The job, as the hiring pages of the model labs describe it, is to define the target behaviours and build evaluations grounded in real work.
The trouble starts with the first step. Three things stand in the way, and each one alone would be enough to stall the project.
Nobody can write the criteria in advance. In 2024 a group of researchers at Berkeley studied how people evaluate model outputs and found what they called criteria drift. People need criteria in order to grade outputs. But it is grading outputs that teaches them what their criteria are. Participants changed their standards as they went, and went back to re-grade earlier answers. The authors concluded that it is impossible to fully fix evaluation criteria before people have judged real outputs.
The experts disagree. Ask five chemists whether a model should state a likely reaction yield with confidence or hedge it, and you will get a split. The split is widest on exactly the cases that matter most, because the easy cases were never in question.
The failures you can see are a biased sample. Users report the errors they notice. The dangerous errors are the ones that look right.
Why the usual answer fails
The usual answer is a rubric. Someone writes five criteria, experts score outputs from one to five on each, and the scores are averaged into a number.
Each step loses something. The rubric is written before the outputs are seen, so by the finding above it is wrong. Scores on a five-point scale vary from person to person and from morning to afternoon. And the average does the most damage, because it hides the disagreement. A response that three experts rate five and two rate one gets the same score as a response everyone rates a mild 3.4. Those are very different situations. The first one contains a decision that somebody has to make, and the average has just buried it.
The proposal: build the spec the way courts build law
Legal systems faced this problem long ago. Nobody can write a statute that anticipates every situation. Common-law systems handle this by letting rules accumulate from decided cases. A judge rules on a specific dispute and writes down the reasoning. Later cases are decided by analogy to earlier ones. General principles are distilled afterwards, once enough cases point the same way.
I propose that a behaviour spec for a model be built the same way: as a body of decided cases, with the general principles extracted from them later. The method has four steps.
Four steps
1. Sample
Draw cases from real usage, spread across every kind of request.WhyComplaints show only the failures users noticed and bothered to report.2. Compare
Show experts two answers to the same request. They pick one and give a one-line reason.WhyPeople judge pairs far more consistently than they assign scores.3. Sort
Ask again a week later. Sort cases into settled, contested and noise.WhyDisagreement between experts means something only if each expert is consistent.4. Rule
A named owner decides each contested case and writes down the reason.WhyThese rulings are the behaviour spec, and the examples a model grader learns from.Step 1: Sample from usage
Start with what people actually ask, grouped by topic, and sample from every group. Anthropic has described a system called Clio that clusters real conversations in a privacy-preserving way for this kind of analysis. Sampling across clusters is how the quiet failures come into view. A complaint queue will never show you the answer that was wrong and went unnoticed.
Step 2: Ask for comparisons
Show each expert two answers to the same request. Ask which is better, and for one line on why.
This rests on one of the oldest results in the measurement of judgment. In 1927 the psychologist Louis Thurstone published his law of comparative judgment, built on the observation that people are far more reliable at saying which of two things is better than at placing one thing on a scale. Chess ratings work this way, and so do the public leaderboards that rank language models from head-to-head votes. The one-line reason matters as much as the choice. It is the raw material for the next step.
Step 3: Sort the cases by the kind of disagreement
This is the step that departs most from normal practice. A week later, show each expert some of the same pairs again without telling them. You now know two things about every case: whether the experts agree with each other, and whether each expert agrees with their own earlier answer.
Three kinds of case
Illustrative; each dot is one case
That gives three piles.
Settled. Experts agree. These become eval items with a known right answer.
Noise. Experts disagree, and they also disagree with themselves a week later. The request was ambiguous or the two answers were too close to call. Rewrite the case or drop it. Leaving these in an eval set adds randomness to every score that comes out of it.
Contested. Experts disagree with each other, and each one holds their position when asked again. These are real differences of opinion about what the model should do. They are product decisions.
Most evaluation pipelines throw the contested pile away, or average it into the middle of the scale. I would argue it is the most valuable thing the exercise produces. Each contested case is a question about your product that nobody had asked out loud.
Step 4: Rule on the contested cases, in writing
Someone with authority over the product decides each contested case and writes two or three sentences on why. For example: “When the evidence for a claim comes from a single preprint, the model states the finding and says in the same sentence that it has not been replicated. Reason: our users are researchers who will act on it.”
The collection of these rulings is the behaviour spec. It is short, concrete, and grounded in real cases. It also solves a practical problem. Model-based graders, which most teams now use to score outputs at scale, tend to be more accurate when given worked examples than when given abstract instructions, and the rulings are worked examples.
When several rulings point the same way, write the principle that covers them. The published documents that describe how models should behave, such as Anthropic’s constitution and OpenAI’s Model Spec, are statutes of this kind. A product team working in one domain needs the layer below: the cases.
Keeping the eval honest afterwards
An eval set decays. Models improve until every model passes an item, and from then on that item says nothing. Education researchers have had tools for this since the 1950s, under the name item response theory. Every question on a test has a difficulty and a discrimination, which is how sharply it separates stronger test-takers from weaker ones.
Not every eval item is worth keeping
Three items, by how the chance of passing changes with model ability
The same analysis works on evals. A 2024 paper, tinyBenchmarks, used it to show that about a hundred well-chosen items could estimate a model’s score on a full benchmark to within about two percentage points. For a product team, that means a few hundred settled cases, pruned regularly, can do the work of thousands. And any comparison between two models should carry error bars. Anthropic’s researchers have published a practical guide to computing them.
How to tell if it is working
- Coverage. When a new failure is reported, how often is there already a ruling that covers it? This should rise over time.
- Grader agreement. On settled cases, how often does the model-based grader match the expert majority? Track it every release.
- Contested rate. The share of new cases that land in the contested pile should fall as rulings accumulate. If it does not, the rulings are too narrow.
- Rulings reversed. Some will be. A rising count means the product’s direction is unclear, which is worth knowing early.
Where it could go wrong
Rulings can conflict with each other as they pile up, so someone has to consolidate them every few months. A panel of experts who all trained in the same place will agree too easily, and the settled pile will carry their shared blind spots. And the method costs expert time, which is scarce. I think the cost is justified by what the alternative costs: a research team spending a quarter improving a number that turned out to measure the wrong thing.
Sources
- Shankar et al., Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences, UIST 2024.
- L. L. Thurstone, A law of comparative judgment, Psychological Review, 1927.
- Anthropic, Clio: privacy-preserving insights into real-world AI use, 2024.
- Polo et al., tinyBenchmarks: evaluating LLMs with fewer examples, 2024.
- Evan Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, Anthropic, 2024.