StoryboardAI Engineering

A model as judge

Using a second model, with a written rubric, to grade answers that simple code cannot check.

What is a model as judge?

Some checks are easy for code: does the answer mention Saturday, is it under two sentences, is it valid JSON. Many are not. Whether advice is safe, or whether a reply is polite, cannot be found by matching words.

For those, a second model does the grading. It is given the question, the answer and a rubric that says exactly what passes and what fails, and it returns a verdict with a reason.

A judge is only as good as its rubric, and it can be wrong. Teams check a sample of its verdicts against a person's from time to time, the same way you would with a new human grader.

A judge is only as good as its agreement with a careful person.Step 1: Write a specific checklist. Step 2: A second model scores each answer. Step 3: Compare with a person's scores. Step 4: Fix the checklist where they differ.1Write a specificchecklist2A second modelscores each answer3Compare with aperson's scores4Fix the checklistwhere they differ
A judge is only as good as its agreement with a careful person.

Why does a model as judge matter?

Many qualities of a good answer cannot be checked by code: whether it is polite, whether it answers what was asked, whether it is supported by its source. Having people read thousands of answers is too slow. A second model, given clear criteria, can score them in minutes, which makes it possible to test such qualities on every change.

How does a model as judge work?

  1. Write a checklist of specific, yes-or-no criteria for a good answer.
  2. Give a model the answer, the checklist and any source material.
  3. It returns a verdict for each criterion, with a reason.
  4. Before trusting it, compare its verdicts with a careful person's on a sample.
  5. Where they disagree, fix the checklist and compare again.

When should you use a model as judge?

Use it when

  • Quality cannot be checked by code
  • There are too many answers for people to read
  • Checking whether claims are supported by a source

Think twice when

  • Code can check it exactly. Use code
  • The criteria are vague. A judge asked to rate out of ten is nearly useless

Where does a model as judge sit in a real system?

The judge is a separate model call, usually in the eval pipeline, and sometimes live as a check on a reply before it is sent. It should see the material, not the first model's reasoning.

What goes wrong with a model as judge?

  • Trusting a judge that was never compared with a person.
  • Asking for one overall score in place of specific criteria.
  • Using it where a simple check in code would be exact.

What does a model as judge look like in practice?

A judge scores replies against four criteria. Compared with a person on 50 cases, it agrees on 44. Four of the six disagreements trace to a vague line in the checklist. After that line is rewritten, they agree on 48.

Watch it happen

These stories show a model as judge at work, one step at a time.