AI Engineering, story 5It made that up

HM Nayem asks about a return

1 of 10

It made that up: AI Engineering, story 5

The assistant confidently quotes a returns policy the shop never had. See why a model invents things, and what stops it.

Why models invent answers, and how the real document fixes it, until there are too many documents.

The story, step by step

  1. HM Nayem asks about a return

    HM Nayem wore a new pair of boots once, and they pinch. The question for the shop's chat is simple. Watch how confidently it gets answered.

  2. A confident answer

    Clear, friendly, specific. It even gives a number of days. There is just one problem.

  3. The shop never said that

    This is the shop's real policy. Fourteen days, and never for worn footwear. The assistant's answer was invented, and nothing in the way it was written gave that away.

  4. Why it invents

    A model does not look things up. It predicts the next token. Across everything it read, return policies most often say 30 days, so that is what it writes. To a model, sounding right and being right are the same thing.

  5. What a model knows, and does not

    A model learned from a huge pile of public text, up to a certain date. This shop's returns page was never a real part of that. Worse, a model has no sense of where its knowledge ends. It cannot feel the gap, so it fills it.

  6. Hand it the real page

    The fix is simple. Before calling the model, the server fetches the real returns page and puts it in the prompt, with one firm instruction: answer only from this.

  7. Now it is right

    Same model, same question. But now the true answer is sitting in front of it, so predicting the next token leads to the truth. This is called grounding.

  8. And when the page does not say

    Just as important: asked something the page does not cover, it now says so instead of guessing. An honest “I don't know” is worth more than a confident invention.

  9. The catch

    One page fits easily. But the shop has two thousand help pages, and the server cannot know in advance which one a question needs. All of them together are twelve times too big for the model's window.

  10. Now at real speed

    Question in, page fetched, answer grounded in it. The next story shows how the server finds the right page among two thousand, in a fraction of a second.

A model predicts. It does not look things up.

Ideas in this story