StoryboardAI Engineering

Made-up answers

A model writes what is likely, so a wrong answer sounds exactly as sure as a right one.

What are made-up answers?

A model does not look anything up. It predicts the next token from patterns in everything it read during training. When it has the facts, that prediction is usually right. When it does not, it predicts something that merely sounds right.

This is often called hallucination. The dangerous part is not that it happens, but that nothing in the wording gives it away. An invented policy is written in the same confident voice as a real one.

You cannot instruct this away. The reliable fix is to put the real information in front of the model and tell it to answer only from that.

A right answer and an invented one are produced the same way.The model knows: Likely words are true words; Fluent and confident; Correct. The model does not know: Likely words fill the gap; Fluent and confident; Wrong, and looks the same.The model knowsLikely words are true wordsFluent and confidentCorrectThe model does not knowLikely words fill the gapFluent and confidentWrong, and looks the same
A right answer and an invented one are produced the same way.

Why do made-up answers matter?

A model that does not know will often answer anyway, fluently and with complete confidence. Nothing in its tone marks the invented parts. This is the single biggest obstacle to trusting AI products, and much of AI engineering exists to prevent it or contain it.

How do made-up answers work?

  1. A model writes the most likely next token given what came before. It has no separate store of facts to check against.
  2. When the text it needs was common in its training, the likely words are also the true ones.
  3. When it was rare or absent, the likely words are whatever usually fills that kind of gap.
  4. The result reads the same either way: a plausible sentence, right or wrong.

When are made-up answers most likely?

More likely when

  • The question is about your own company, recent events or anything private
  • It asks for specifics: a number, a date, a name, a reference
  • The prompt pushes for an answer and offers no way to say "I don't know"

Less likely when

  • The facts are placed in the prompt and the model is told to use only those
  • The model is allowed, and shown how, to say it does not know
  • A check compares each claim with its source

Where do made-up answers sit in a real system?

It originates inside the model and cannot be removed there. The defences sit around it: fetching real text before the call, asking for sources in the reply, and checking claims afterwards.

What goes wrong with made-up answers?

  • Treating confidence as a sign of correctness.
  • Trying to fix it with a line in the prompt saying "do not make things up".
  • Assuming a newer model has solved it. It becomes rarer, not absent.

What does made-up answers look like in practice?

Asked for a shop's returns policy, a model replied that items could be returned within 14 days with a receipt. The shop's real policy was 30 days. The model had never seen it. Fourteen days with a receipt is simply what returns policies often say.

Watch it happen

These stories show made-up answers at work, one step at a time.