What is retrieval-augmented generation?
A model only knows what it was trained on, which never includes your company's documents or anything recent. RAG is the standard way around that.
It has two phases. Ahead of time, documents are cut into chunks and stored by meaning. At question time, the nearest chunks to the question are retrieved, added to the prompt, and the model generates an answer from them, usually with references to its sources.
The model itself is not changed at all. That is why RAG is popular: updating what the product knows is as simple as updating the documents.
Why does retrieval-augmented generation matter?
A model knows only what it was trained on, which never includes your documents or anything recent. Retrieval-augmented generation, RAG for short, is the standard way to give it that knowledge without training anything. It is behind nearly every assistant that answers from a company's own material.
How does retrieval-augmented generation work?
- Ahead of time, documents are cut into chunks and each chunk is stored by meaning.
- A question arrives and is used to search for the few most relevant chunks.
- Those chunks are placed in the prompt with the question and an instruction to answer only from them.
- The model writes an answer and marks which chunk each claim came from.
When should you use retrieval-augmented generation?
Use it when
- Answers live in documents you own
- The material changes, or is too large for a prompt
- People need sources they can check
Think twice when
- The answer needs every document at once, such as a total across all files
- The data is in tables better reached with a query
Where does retrieval-augmented generation sit in a real system?
It has two halves. The preparation runs in the background whenever documents change. The answering runs on each question, in the step before the model call.
What goes wrong with retrieval-augmented generation?
- Blaming the model when the search returned the wrong chunks. Check what was retrieved first.
- Sending twenty chunks to be safe. More text lowers answer quality.
- Skipping the instruction to say when the chunks do not hold the answer.
What does retrieval-augmented generation look like in practice?
A member asks how late a trip can be cancelled. Search returns three chunks out of 3,900. The prompt holds those three and the question, 400 tokens in all. The answer comes back in a second, costs a fifth of a cent, and cites page 14.