AI Engineering, story 21Grading an agent

The answer was right. Was the work?

1 of 14

Grading an agent: AI Engineering, story 21

An agent says the refund is done. Was it? Follow how an agent is tested on what it did, step by step, and not on what it said.

An agent is graded on the end state and on its path, over many runs, in a practice world.

The story, step by step

  1. The answer was right. Was the work?

    An earlier story tested a prompt by comparing answers with known ones. That works for a chatbot. An agent is different, because it acts. Here the agent tells a customer that the refund has been issued. The sentence is perfect. HM Nayem opens the payments record, and there is no refund in it. An agent has to be graded on what it did.

  2. A task, not a question

    So a test for an agent has three parts. A starting situation: this customer, this order, delivered three days ago. A task: the customer's message. And a check on the world afterwards: one refund of $38 exists against that order. The expected reply is not part of the test at all.

  3. A practice shop

    The agent cannot be tested against real customers and real money. HM Nayem builds a practice copy of the shop's systems with invented orders and pretend payments. The agent's tools point at it, and the agent cannot tell the difference. Before each test the copy is reset to the task's starting situation, so one test never spoils the next.

  4. Check the end state, with code

    The agent runs. When it stops, code looks at the practice shop and asks plain questions with yes-or-no answers. Does the refund exist? For the right amount? Only once? Was anything else changed that should not have been? No model is involved in this check, so it gives the same verdict every time.

  5. Then check the path

    A right result can be reached the wrong way. In this run the refund is correct, and on the way the agent looked up an order belonging to somebody else. The harness records every step an agent takes. A second set of checks runs over that list, looking for steps that must never happen, whatever the outcome.

  6. Run it more than once

    A model does not choose the same words, or the same steps, every time. HM Nayem runs the same task ten times. It passes seven. An agent that passes once has not passed. Every task is run several times and the score is a rate. A task at seven in ten is a customer who is let down three times in ten.

  7. What code cannot judge

    Some things have no yes-or-no check. Was the reply kind to someone whose stove arrived broken? Did it explain the refund clearly? For these, a second model is given the conversation and a written checklist, and returns a score for each line with a reason. The checklist does the real work. A judge told only to rate the reply out of ten is close to useless.

  8. Checking the judge

    A judge is a model too, and can be wrong. Before trusting it, HM Nayem scores fifty conversations himself, without looking at the judge's marks, then compares. They agree on forty-four. He reads the six disagreements. In four, the checklist was vague, and he rewrites those lines. A judge is only as good as the agreement it has with a careful person.

  9. Credit for getting part way

    Pass or fail hides where an agent breaks. Each task is split into milestones, checked in order. This agent nearly always finds the order and checks the rules. It loses the thread at the last step. That tells HM Nayem to look at the refund tool, and not at how the agent searches.

  10. Steps, time and money are scores too

    Two agents can both pass a task, one in four steps and one in nineteen. The second costs five times as much and makes the customer wait. So every run also records how many steps it took, how long, and what it cost. An improvement that raises the pass rate by a point and doubles the cost is not obviously an improvement.

  11. Hard cases, on purpose

    A task set made of ordinary requests gives a flattering score. A quarter of HM Nayem's tasks are chosen to be awkward: a tool that fails half way, a customer who lies, a request with two possible meanings, an order that was already refunded. These are where agents do real damage, so they are tested most.

  12. A score for each kind of task

    One overall number hides everything useful. Broken down by kind, the picture is clear. The agent is excellent at lookups, good at refunds, and poor when a request is unclear, where it guesses and does not ask. That one row sets the next week's work.

  13. Before every change

    The task set earns its keep by being run constantly. Every change to a prompt, a tool or the model runs all eighty tasks, three times each, before it is allowed out. Because scores wobble from run to run, a change is only called worse if it drops by more than the wobble. Anything beyond that blocks the change until someone has read the failures.

  14. One task, graded, at real speed

    A single task from start to finish. The practice shop is reset. The agent works. Code checks the end state and the path. The judge scores the reply against its checklist. Steps and cost are written down. Then the next task begins. Eighty of these, three times over, take about six minutes.

Grade an agent on what it did, not on what it said.

Ideas in this story