It made that up: AI Engineering, story 5
The assistant confidently quotes a returns policy the shop never had. See why a model invents things, and what stops it.
Why models invent answers, and how the real document fixes it, until there are too many documents.
The story, step by step
HM Nayem asks about a return
HM Nayem wore a new pair of boots once, and they pinch. The question for the shop's chat is simple. Watch how confidently it gets answered.
A confident answer
Clear, friendly, specific. It even gives a number of days. There is just one problem.
The shop never said that
This is the shop's real policy. Fourteen days, and never for worn footwear. The assistant's answer was invented, and nothing in the way it was written gave that away.
Why it invents
A model does not look things up. It predicts the next token. Across everything it read, return policies most often say 30 days, so that is what it writes. To a model, sounding right and being right are the same thing.
What a model knows, and does not
A model learned from a huge pile of public text, up to a certain date. This shop's returns page was never a real part of that. Worse, a model has no sense of where its knowledge ends. It cannot feel the gap, so it fills it.
Hand it the real page
The fix is simple. Before calling the model, the server fetches the real returns page and puts it in the prompt, with one firm instruction: answer only from this.
Now it is right
Same model, same question. But now the true answer is sitting in front of it, so predicting the next token leads to the truth. This is called grounding.
And when the page does not say
Just as important: asked something the page does not cover, it now says so instead of guessing. An honest “I don't know” is worth more than a confident invention.
The catch
One page fits easily. But the shop has two thousand help pages, and the server cannot know in advance which one a question needs. All of them together are twelve times too big for the model's window.
Now at real speed
Question in, page fetched, answer grounded in it. The next story shows how the server finds the right page among two thousand, in a fraction of a second.
A model predicts. It does not look things up.
- A model writes what is most likely, and a wrong answer sounds just as sure as a right one.
- Putting the real text in the prompt makes the likely answer the true one.
- Told to answer only from that text, it can also say “I don't know”.
- Everything will not fit, so the right text has to be found first.
Ideas in this story
- Made-up answers: A model writes what is likely, so a wrong answer sounds exactly as sure as a right one.
- Grounding: Putting the real text in the prompt and telling the model to answer only from it.