When search misses: AI Engineering, story 7
The answer is in the help pages, but the search brings back the wrong ones. Follow the three fixes that find it.
Why the first search often fails, and the fixes: rewrite the question, combine search types, re-sort results.
The story, step by step
A follow-up question
HM Nayem is mid-conversation, paying with a gift card, and hits an error. The fix is written down in the shop's help pages. Watch the search from the last story try to find it.
The search comes back wrong
Three pages about errors in general. None about this one. Meaning search is good at subjects and poor at exact codes, and “It says E-417” hardly has a subject at all.
Why it missed
On the map of meaning, the message lands in a vague middle. Nothing in “It says E-417. What now?” mentions gift cards, and to an embedding model one error code looks much like another. The page that holds the fix is far away.
Wrong pages in, wrong answer out
The model does exactly what it was told: it answers from what it was given. The reply is grounded, polite and useless. When the search misses, nothing later can save the answer.
Fix one: rewrite the question
People write follow-ups that only make sense in context. So before searching, the server asks a model to rewrite the message as a complete question, using the conversation so far.
Fix two: search two ways
Meaning search still struggles with the code itself. An old-fashioned keyword search does not: it finds the exact text E-417. Running both and merging the results is called hybrid search.
Fix three: sort them properly
The merged list has the right page, but not at the top. A reranker, a small model that reads each candidate next to the question, scores them carefully. Only the best three go forward.
Right pages, right answer
Same model as before. Nothing about it changed. It simply has the right text in front of it this time, so the answer is the one written in the help pages.
What the fixes cost
Each fix adds a little time. Together, about half a second. That is a trade most teams make gladly, because a fast wrong answer is worth nothing.
Now at real speed
Rewrite, search two ways, re-sort, then answer. When an AI product gets something wrong about its own documents, the cause is almost always in these steps, not in the model.
Finding the right text is most of the work.
- If the search brings back the wrong text, the model answers wrongly from it.
- Rewriting a vague follow-up into a full question gives the search something to find.
- Meaning search and keyword search miss different things, so products run both.
- A reranker re-sorts the candidates so the best few reach the model.
Ideas in this story
- Rewriting the question: Turning what the person typed into a complete question that a search can actually find.
- Hybrid search: Searching by meaning and by exact words at the same time, then merging the results.
- Reranking: A second, more careful model re-sorts the search results so the best few come first.