StoryboardAI Engineering

Retrieval-augmented generation

Retrieve the right text, add it to the prompt, then let the model answer from it. RAG for short.

What is retrieval-augmented generation?

A model only knows what it was trained on, which never includes your company's documents or anything recent. RAG is the standard way around that.

It has two phases. Ahead of time, documents are cut into chunks and stored by meaning. At question time, the nearest chunks to the question are retrieved, added to the prompt, and the model generates an answer from them, usually with references to its sources.

The model itself is not changed at all. That is why RAG is popular: updating what the product knows is as simple as updating the documents.

RAG: find the right text, then answer from it.Step 1: Documents are chunked and stored. Step 2: A question searches the store. Step 3: The best chunks join the prompt. Step 4: The model answers with sources.1Documents arechunked and stored2A questionsearches the store3The best chunksjoin the prompt4The model answerswith sources
RAG: find the right text, then answer from it.

Why does retrieval-augmented generation matter?

A model knows only what it was trained on, which never includes your documents or anything recent. Retrieval-augmented generation, RAG for short, is the standard way to give it that knowledge without training anything. It is behind nearly every assistant that answers from a company's own material.

How does retrieval-augmented generation work?

  1. Ahead of time, documents are cut into chunks and each chunk is stored by meaning.
  2. A question arrives and is used to search for the few most relevant chunks.
  3. Those chunks are placed in the prompt with the question and an instruction to answer only from them.
  4. The model writes an answer and marks which chunk each claim came from.

When should you use retrieval-augmented generation?

Use it when

  • Answers live in documents you own
  • The material changes, or is too large for a prompt
  • People need sources they can check

Think twice when

  • The answer needs every document at once, such as a total across all files
  • The data is in tables better reached with a query

Where does retrieval-augmented generation sit in a real system?

It has two halves. The preparation runs in the background whenever documents change. The answering runs on each question, in the step before the model call.

What goes wrong with retrieval-augmented generation?

  • Blaming the model when the search returned the wrong chunks. Check what was retrieved first.
  • Sending twenty chunks to be safe. More text lowers answer quality.
  • Skipping the instruction to say when the chunks do not hold the answer.

What does retrieval-augmented generation look like in practice?

A member asks how late a trip can be cancelled. Search returns three chunks out of 3,900. The prompt holds those three and the question, 400 tokens in all. The answer comes back in a second, costs a fifth of a cent, and cites page 14.

Watch it happen

These stories show retrieval-augmented generation at work, one step at a time.