Why did it say that?: AI Engineering, story 32
HM Nayem is told something that is not true. Trace the reply back through every step to find the one that went wrong. It is not the one you expect.
How every step of a request is recorded so you can find what went wrong.
The story, step by step
A wrong answer
HM Nayem asks whether worn boots can be returned and is told yes, within 30 days. That is not the shop's policy. HM Nayem taps the thumbs-down. Somewhere inside the shop's system, something went wrong.
Where would you even start?
That one reply was the result of six separate steps, each handing its result to the next. Any of them could be at fault. Without a record of what each one did, all anybody can do is guess.
Write down every step
So the shop's code keeps a record. Every request gets its own ID. Each step, as it runs, notes what it was given, what it produced, how long it took and what it cost. This is called tracing.
A trace
Put the records for one request in order and you get a trace: the whole journey, step by step, with times. This is HM Nayem's request. Nothing in it looks slow or broken at a glance.
An engineer opens it
The thumbs-down carried the request's ID with it. An engineer at the shop uses that ID to pull up this exact request, out of the millions stored.
First guess: the model made it up
The obvious suspect is the model. You have seen it invent a 30-day policy before, in the story about made-up answers. Most people would stop here and start rewriting the prompt. The engineer looks at the step first.
Read it backwards
Start from the reply and look at what the model was given. And there it is, in the sources: the 30-day rule, word for word. The model did not invent anything. It answered faithfully from what it was handed.
One step further back
So where did that source come from? The search step. Its results show two returns pages: the current one, and an old one from 2023 that nobody removed. The old one scored a little higher and went first.
The real cause
When the shop changed its policy, it added the new page to the search index and forgot to take the old one out. The model was fine. The prompt was fine. The fault was a stale document, three steps upstream.
Without the trace
Think about what would have happened otherwise. The team blames the model, rewrites a prompt that was working, and the wrong answer keeps coming back. A trace does not just save time. It stops people fixing the wrong thing.
Make sure it stays fixed
One more step, from the story about test questions. HM Nayem's question goes into the eval set. If an outdated page ever sneaks back into the index, the score will drop and someone will know before a customer does.
Watching all of them at once
A trace explains one request. The same records, added up, describe the health of the whole product. The shop watches a handful of numbers, and pays most attention to the slow ones, because that is where customers feel pain first.
An alarm goes off
A week later the slowest replies jump from five seconds to nine, and an alarm fires. The engineer opens a few slow traces. The same step stands out in every one. Found in minutes, not days.
Careful what you record
A last caution. Traces hold customers' own words, so everything from the privacy story applies. Sensitive details are masked before they are stored, records are deleted after a set time, and only the people who need them can look.
You cannot fix what you cannot see.
- One reply is the result of many steps, and any of them can be the cause.
- A trace records each step: what went in, what came out, how long it took.
- Read a bad answer backwards from the reply until something looks wrong.
- The model is the usual suspect, and often not the culprit.
- The same records, added up, show the health of the whole product.
Ideas in this story
- Tracing: Recording every step of a request, with what went in, what came out and how long it took.
- Monitoring: Watching speed, cost, errors and quality across all requests, with alarms when something shifts.
- Evals: A fixed set of test questions with known good answers, run after every change to see if things got better or worse.