AI Engineering, story 7When search misses

A follow-up question

1 of 10

When search misses: AI Engineering, story 7

The answer is in the help pages, but the search brings back the wrong ones. Follow the three fixes that find it.

Why the first search often fails, and the fixes: rewrite the question, combine search types, re-sort results.

The story, step by step

  1. A follow-up question

    HM Nayem is mid-conversation, paying with a gift card, and hits an error. The fix is written down in the shop's help pages. Watch the search from the last story try to find it.

  2. The search comes back wrong

    Three pages about errors in general. None about this one. Meaning search is good at subjects and poor at exact codes, and “It says E-417” hardly has a subject at all.

  3. Why it missed

    On the map of meaning, the message lands in a vague middle. Nothing in “It says E-417. What now?” mentions gift cards, and to an embedding model one error code looks much like another. The page that holds the fix is far away.

  4. Wrong pages in, wrong answer out

    The model does exactly what it was told: it answers from what it was given. The reply is grounded, polite and useless. When the search misses, nothing later can save the answer.

  5. Fix one: rewrite the question

    People write follow-ups that only make sense in context. So before searching, the server asks a model to rewrite the message as a complete question, using the conversation so far.

  6. Fix two: search two ways

    Meaning search still struggles with the code itself. An old-fashioned keyword search does not: it finds the exact text E-417. Running both and merging the results is called hybrid search.

  7. Fix three: sort them properly

    The merged list has the right page, but not at the top. A reranker, a small model that reads each candidate next to the question, scores them carefully. Only the best three go forward.

  8. Right pages, right answer

    Same model as before. Nothing about it changed. It simply has the right text in front of it this time, so the answer is the one written in the help pages.

  9. What the fixes cost

    Each fix adds a little time. Together, about half a second. That is a trade most teams make gladly, because a fast wrong answer is worth nothing.

  10. Now at real speed

    Rewrite, search two ways, re-sort, then answer. When an AI product gets something wrong about its own documents, the cause is almost always in these steps, not in the model.

Finding the right text is most of the work.

Ideas in this story