StoryboardAI Engineering

Chunking

Cutting documents into small pieces so the right piece can be found and handed to the model.

What is chunking?

Whole documents are a poor unit for search. A ten-page guide covers many subjects, so it matches many questions weakly and none strongly, and handing all of it to the model wastes space.

So documents are cut into chunks, usually a paragraph or a few. Each chunk is about one thing, which makes it easier to match to a question and cheaper to include in a prompt.

Where you cut matters. A chunk that ends mid-sentence, or loses the heading that says what it is about, is hard to find and hard to use. Good chunking follows the document's own structure.

Chunking turns a document into small, labelled pieces.Step 1: A long document. Step 2: Cut along its own structure. Step 3: Each piece keeps its address. Step 4: Pieces are stored for search.1A long document2Cut along its ownstructure3Each piece keepsits address4Pieces are storedfor search
Chunking turns a document into small, labelled pieces.

Why does chunking matter?

A model does not need a whole handbook to answer one question. It needs the two paragraphs that matter. Cutting documents into small pieces is what makes it possible to find and send only those. How the cutting is done sets a ceiling on everything after it: a fact split across two pieces may never be found whole.

How does chunking work?

  1. Each document is turned into plain text.
  2. The text is cut into pieces of a paragraph or two, following its own structure: headings, paragraphs, table rows.
  3. Each piece keeps a note of where it came from: the file, the page, the heading above it.
  4. Neighbouring pieces overlap by a sentence, so a fact that straddles a cut survives in one of them.
  5. The pieces are stored so that the right ones can be searched for later.

When should you use chunking?

Use it when

  • Documents are longer than a page
  • You have more text than fits in a prompt
  • Answers should point to a specific place in a source

Think twice when

  • Documents are short enough to send whole
  • The meaning depends on the entire document, such as a contract read end to end

Where does chunking sit in a real system?

Chunking is part of the preparation done ahead of time, before any question is asked. It runs once per document and again whenever a document changes.

What goes wrong with chunking?

  • Cutting at a fixed number of characters, through the middle of sentences and tables.
  • Splitting a table so that the numbers lose their column headings.
  • Storing a piece without its heading, so nobody can tell what it refers to.

What does chunking look like in practice?

A page on cancelling trips is cut into three pieces. One says "A full refund is given if you cancel 7 days or more before the trip". It is stored with the label "Handbook, page 14, Trips, Cancelling a trip". A question about refunds finds that piece and nothing else.

Watch it happen

These stories show chunking at work, one step at a time.