AI Engineering, story 32Why did it say that?

A wrong answer

1 of 14

Why did it say that?: AI Engineering, story 32

HM Nayem is told something that is not true. Trace the reply back through every step to find the one that went wrong. It is not the one you expect.

How every step of a request is recorded so you can find what went wrong.

The story, step by step

  1. A wrong answer

    HM Nayem asks whether worn boots can be returned and is told yes, within 30 days. That is not the shop's policy. HM Nayem taps the thumbs-down. Somewhere inside the shop's system, something went wrong.

  2. Where would you even start?

    That one reply was the result of six separate steps, each handing its result to the next. Any of them could be at fault. Without a record of what each one did, all anybody can do is guess.

  3. Write down every step

    So the shop's code keeps a record. Every request gets its own ID. Each step, as it runs, notes what it was given, what it produced, how long it took and what it cost. This is called tracing.

  4. A trace

    Put the records for one request in order and you get a trace: the whole journey, step by step, with times. This is HM Nayem's request. Nothing in it looks slow or broken at a glance.

  5. An engineer opens it

    The thumbs-down carried the request's ID with it. An engineer at the shop uses that ID to pull up this exact request, out of the millions stored.

  6. First guess: the model made it up

    The obvious suspect is the model. You have seen it invent a 30-day policy before, in the story about made-up answers. Most people would stop here and start rewriting the prompt. The engineer looks at the step first.

  7. Read it backwards

    Start from the reply and look at what the model was given. And there it is, in the sources: the 30-day rule, word for word. The model did not invent anything. It answered faithfully from what it was handed.

  8. One step further back

    So where did that source come from? The search step. Its results show two returns pages: the current one, and an old one from 2023 that nobody removed. The old one scored a little higher and went first.

  9. The real cause

    When the shop changed its policy, it added the new page to the search index and forgot to take the old one out. The model was fine. The prompt was fine. The fault was a stale document, three steps upstream.

  10. Without the trace

    Think about what would have happened otherwise. The team blames the model, rewrites a prompt that was working, and the wrong answer keeps coming back. A trace does not just save time. It stops people fixing the wrong thing.

  11. Make sure it stays fixed

    One more step, from the story about test questions. HM Nayem's question goes into the eval set. If an outdated page ever sneaks back into the index, the score will drop and someone will know before a customer does.

  12. Watching all of them at once

    A trace explains one request. The same records, added up, describe the health of the whole product. The shop watches a handful of numbers, and pays most attention to the slow ones, because that is where customers feel pain first.

  13. An alarm goes off

    A week later the slowest replies jump from five seconds to nine, and an alarm fires. The engineer opens a few slow traces. The same step stands out in every one. Found in minutes, not days.

  14. Careful what you record

    A last caution. Traces hold customers' own words, so everything from the privacy story applies. Sensitive details are masked before they are stored, records are deleted after a set time, and only the people who need them can look.

You cannot fix what you cannot see.

Ideas in this story