Model-graded evals are still evals

The objection to model-graded evaluation is that you are using the thing you are trying to measure to do the measuring. It is a real objection and it is not fatal. Compilers are written in the languages they compile. The discipline is in what you do about it.

Here is what we ask of any model-graded eval before we let it gate a deploy.

The grader is pinned

A grader that follows the latest model version is not an eval, it is a weather report. When the underlying model changes, your historical scores stop being comparable, and the first you hear about it is a regression that is actually a re-grading.

Pin the grader model, pin the prompt, version both together. When you change either, re-grade history rather than letting the discontinuity sit in your charts unlabelled. This is expensive. Do it anyway, or accept that your trend lines mean nothing across the change.

The rubric produces a label, not a number

Asking a model to score output from one to ten produces numbers that are stable within a run and meaningless across runs. Asking it to pick a labelled category with written criteria produces something you can audit, argue with, and compare against a human.

We write rubrics as small decision trees. Did the response cite a source that appears in the retrieved context? If no, label unsupported. If yes, does the citation support the claim it is attached to? If no, label misattributed. Each leaf is a label a human can adjudicate in about four seconds.

The scores you report are then proportions of labels, which have honest error bars, rather than averages of a made-up scale, which do not.

Every rubric has a human-agreement number, and it is published

Before a rubric gates anything, we sample 200 traces, have two people label them independently, and compute agreement between the humans and between each human and the grader.

If the humans agree with each other less than the grader agrees with them, the rubric is underspecified and the problem is the writing, not the model. That happens perhaps a third of the time on the first draft, and it is the single most useful signal in this whole process — a rubric that people cannot apply consistently was never going to measure anything.

We publish the agreement number alongside every eval result in the product. A score of 0.91 from a rubric with 0.62 human agreement should not be read the same way as 0.91 from a rubric at 0.94, and putting both numbers next to each other is the cheapest possible guard against over-trusting the first one.

The grader runs on production traces, not a curated set

A fixed evaluation set drifts away from production the moment you stop curating it, and you will stop curating it. Every team does. The set becomes a museum of the failure modes you had eighteen months ago.

Running the same rubrics against sampled production traces costs more and tells you what is happening now. Keep the fixed set for CI, where determinism is the point, and understand that a green CI eval means “no regression against known cases”, not “the system is good”.

Those two claims get conflated constantly, usually in a slide deck, usually right before the incident.

Where this leaves us

Model-graded evals are a legitimate instrument with a wide error bar and a specific set of failure modes. Pin the grader, label instead of scoring, measure human agreement and publish it, run against reality. Do those four things and you have a measurement. Skip them and you have a vibe with a decimal point.