One line of the prompt changes: AI Engineering, story 20
HM Nayem adds a single sentence to an assistant's instructions. It looks better. Follow the check that shows it quietly got worse.
How a team knows a change made things better, not just different.
The story, step by step
One small change
HM Nayem built a little assistant for the hiking club, called Trail Buddy. Members like it but say the answers ramble. So HM Nayem adds one line to its instructions. What could go wrong with one line?
Try it once
A quick test. HM Nayem asks the question members ask most. The answer is short and correct. It looks like a clear improvement, and it is very tempting to stop here.
The trouble with trying it once
Members ask hundreds of kinds of question, and HM Nayem tried one. A line in a prompt touches every answer the assistant will ever give. And because models pick words by chance, even this one answer could differ next time.
A set of test questions
So HM Nayem keeps a list: forty real questions from members, each with a note on what a good answer must include. This is called an eval set. It is the most valuable thing an AI product owns after its prompt.
Checks that code can do
Every answer has to be graded, forty at a time, again and again. Some checks are easy for ordinary code. Does the answer contain the word Saturday? Is it under two sentences? Those take no time at all.
Checks that need a judge
Whether gear advice is safe cannot be found by matching words. So a second model does that grading. It gets the question, the answer and a written rubric, and returns pass or fail with a reason. It is used only as a judge.
First, the score before
Before judging the new line, HM Nayem needs something to compare it with. So all forty questions are run with the old instructions. This is the baseline: 36 of 40.
Then, the score after
Now the same forty questions with the new line added. Same questions, same graders, one thing changed. The score drops to 31. The line that looked like an obvious improvement made the assistant worse.
Which ones broke
A score alone is not much use. The list of failures is. The new line did exactly what it said: shorter answers. And in doing so it squeezed the safety details out of five of them.
A better line
Now HM Nayem knows what to fix. Not “be short”, but “be short without cutting safety”. The failures wrote the new instruction.
Run it again
Forty questions once more. This time 38 pass. Shorter answers, and better than the baseline. That is what an improvement looks like when it is measured rather than hoped for.
Every change, every time
HM Nayem makes it automatic. Any change to the instructions, and any move to a newer model, runs all forty questions first. If the score falls, the change does not go live.
The list keeps growing
One last habit. Next month a member reports a bad answer about dogs on trails. That question goes straight into the list, as number 41. Every real mistake becomes a test, so it can never quietly come back.
You cannot tell if a prompt is better by trying it once.
- One line in a prompt changes every answer, in ways one test will not show.
- An eval set is a fixed list of real questions with what a good answer needs.
- Simple checks are done by code, harder ones by a second model with a rubric.
- Compare the score before and after, then read the failures.
- Every real mistake joins the list.
Ideas in this story
- Evals: A fixed set of test questions with known good answers, run after every change to see if things got better or worse.
- A model as judge: Using a second model, with a written rubric, to grade answers that simple code cannot check.