What is a model as judge?
Some checks are easy for code: does the answer mention Saturday, is it under two sentences, is it valid JSON. Many are not. Whether advice is safe, or whether a reply is polite, cannot be found by matching words.
For those, a second model does the grading. It is given the question, the answer and a rubric that says exactly what passes and what fails, and it returns a verdict with a reason.
A judge is only as good as its rubric, and it can be wrong. Teams check a sample of its verdicts against a person's from time to time, the same way you would with a new human grader.
Why does a model as judge matter?
Many qualities of a good answer cannot be checked by code: whether it is polite, whether it answers what was asked, whether it is supported by its source. Having people read thousands of answers is too slow. A second model, given clear criteria, can score them in minutes, which makes it possible to test such qualities on every change.
How does a model as judge work?
- Write a checklist of specific, yes-or-no criteria for a good answer.
- Give a model the answer, the checklist and any source material.
- It returns a verdict for each criterion, with a reason.
- Before trusting it, compare its verdicts with a careful person's on a sample.
- Where they disagree, fix the checklist and compare again.
When should you use a model as judge?
Use it when
- Quality cannot be checked by code
- There are too many answers for people to read
- Checking whether claims are supported by a source
Think twice when
- Code can check it exactly. Use code
- The criteria are vague. A judge asked to rate out of ten is nearly useless
Where does a model as judge sit in a real system?
The judge is a separate model call, usually in the eval pipeline, and sometimes live as a check on a reply before it is sent. It should see the material, not the first model's reasoning.
What goes wrong with a model as judge?
- Trusting a judge that was never compared with a person.
- Asking for one overall score in place of specific criteria.
- Using it where a simple check in code would be exact.
What does a model as judge look like in practice?
A judge scores replies against four criteria. Compared with a person on 50 cases, it agrees on 44. Four of the six disagreements trace to a vague line in the checklist. After that line is rewritten, they agree on 48.