StoryboardAI Engineering

Reranking

A second, more careful model re-sorts the search results so the best few come first.

What is reranking?

The first search is built for speed. It compares positions or counts words across the whole collection in milliseconds, which makes it fast and a little rough. The right text is usually somewhere in its top twenty, but not reliably in its top three.

A reranker fixes the order. It is a small model that reads the question and one candidate together and scores how well that candidate answers it. That is too slow to run on everything, but fine for twenty.

Only the top few after reranking go into the prompt. The model gets less text, and more of it is the text that matters.

A careful second look at a short list.Step 1: Search returns 40 chunks. Step 2: A reranker reads each with the question. Step 3: Chunks are sorted by score. Step 4: The top 3 go to the model.1Search returns 40chunks2A reranker readseach with thequestion3Chunks are sortedby score4The top 3 go tothe model
A careful second look at a short list.

Why does reranking matter?

Fast search is rough. It compares a question with each chunk separately and never reads the two together, so its top results are often close but not best. Reranking adds a slower, more careful second look at a short list. It is one of the most reliable ways to improve retrieval without changing anything else.

How does reranking work?

  1. Search returns a generous short list, say forty chunks, not just the top three.
  2. Each chunk is paired with the question and given to a reranking model.
  3. The model reads question and chunk together and scores how well the chunk answers.
  4. The chunks are sorted by that score, and only the best few go into the prompt.

When should you use reranking?

Use it when

  • Search returns the right chunk, but not near the top
  • The collection is large and many chunks look alike
  • You can afford a fifth of a second for better answers

Think twice when

  • The right chunk is not in the short list at all. Fix the search first
  • Speed matters more than a small gain in quality

Where does reranking sit in a real system?

It sits between search and the model call. It works on a short list only, because reading every chunk in the collection this carefully would be far too slow.

What goes wrong with reranking?

  • Reranking the top three. There is nothing to gain unless the list is wider than what you will use.
  • Expecting it to rescue a search that never found the right chunk.
  • Ignoring its cost at scale. Forty readings per question add up.

What does reranking look like in practice?

For a question about form T-14, search returns forty chunks. The reranker scores the one that describes the form at 0.97, another mention at 0.81 and a general chunk about forms at 0.12. The first two go to the model. The reranking took 0.18 seconds.

Watch it happen

These stories show reranking at work, one step at a time.