AI Engineering
Judging
the judge
Calibrating an LLM-as-judge against human raters
The moment you let a model grade another model's work, you have created a number nobody has checked. A judge that has never been compared to a person is not a measurement — it is an opinion with a number attached.
In brief At E23 I calibrated an LLM-as-judge against human raters: model and person score the same work blind, their agreement is measured statistically, and a threshold fixed in advance decides whether the model's score may influence anything.
The challenge
Model-generated quality scores were shaping which work got attention, and nothing established that they agreed with the person whose standard mattered.
The approach
Score blind on both sides, measure the agreement with a statistic built for it, and fix the usable threshold before seeing the answer.
The result
The quality metric became measurable in its own right, with an explicit rule for when it must be ignored — a loop most systems never close.
Hover any step to read what it doesTap any step to read what it does