Finding the right page: AI Engineering, story 6
The shop has two thousand help pages and HM Nayem has one question. Watch the right three paragraphs get found, by meaning, in a blink.
Documents are cut up and found by meaning; the model answers only from those pieces and shows its sources.
The story, step by step
One question, two thousand pages
HM Nayem wants to know about washing a down jacket. The answer exists, somewhere in the shop's two thousand help pages. The server has about a tenth of a second to find it.
Long before the question
The work starts long before anyone asks. Every help page is cut into small pieces called chunks, a paragraph or two each. Small pieces are easier to find and cheaper to hand to the model.
Each chunk gets a position
Next, a second kind of model, an embedding model, turns each chunk into a list of numbers. Think of the numbers as a position on a map. Chunks that mean similar things land close together.
What a position really is
Here is one chunk and its position: a list of 1,536 numbers. No person can read them, and nobody needs to. All that matters is that two texts with similar meaning get similar numbers. The map on screen is that idea, flattened to two directions.
Stored, ready to search
All 8,400 positions are saved in a search index, also called a vector database. This preparation happens once, and again for any page that changes. From here on, every question can use it.
The question gets a position too
Now the question arrives. The server runs it through the same embedding model, so it lands on the same map, right beside the chunks about the same subject.
The nearest chunks win
Finding the answer is now just finding the nearest points. Notice that the question says “machine” and the best chunk says “front-loading washer”. No shared words. It matched on meaning.
Build the prompt
The server does what it did in the last story: it puts the text in the prompt and says to answer only from it. The difference is that these three chunks were chosen automatically, out of 8,400.
An answer with sources
The model answers from the chunks and points to where each fact came from. HM Nayem can check the source. So can the shop, when something looks wrong.
It has a name
This pattern is called retrieval-augmented generation, or RAG. Retrieve the right text, add it to the prompt, generate the answer. It is how most AI products know things their model was never taught.
Now at real speed
At real speed, the search is the fastest part. Two thousand pages, three paragraphs, one grounded answer, in about the time it took to read this sentence.
The model answers from text it was handed a moment ago.
- Help pages are cut into small chunks ahead of time.
- An embedding model gives each chunk a position, so similar meanings sit together.
- The question gets a position too, and the nearest chunks are the ones used.
- The model answers only from those chunks and shows its sources.
Ideas in this story
- Chunking: Cutting documents into small pieces so the right piece can be found and handed to the model.
- Embeddings: Turning text into a list of numbers so that similar meanings end up close together.
- Retrieval-augmented generation: Retrieve the right text, add it to the prompt, then let the model answer from it. RAG for short.