Build: ask the handbook: AI Engineering, story 36
HM Nayem's hiking club has 400 files and the same ten questions every week. Build the page that answers them, one part at a time, and watch each version fail in a new way.
A document assistant is built in versions: ask, paste, read scans, cut up, search two ways, cite, refuse, refresh, restrict, test, and list what is still missing.
The story, step by step
The job
HM Nayem runs a hiking club. Its rules and history sit in 400 files, and members ask the same questions every week. So HM Nayem will build a page where anyone can ask, and get an answer with the place it came from. We will build it one part at a time.
Version 1: just ask a model
The first version is twenty lines of code. The app takes the question, sends it to a model and shows what comes back. It works, and the answer sounds certain. It is also wrong: the club's rule is seven days, not 24 hours.
Version 2: hand it the handbook
The model guessed because it has never read the club's files. So version 2 reads the handbook and places all 60 pages in the prompt, above the question. Now the model answers from the real text, and the answer is right.
Why version 2 will not last
Two problems. Every question makes the model read all 60 pages again, so each answer is slow and costs real money. And the handbook is one file of 400. All of them together are far more than a model can read at once.
Version 3: cut the files up
The model does not need 400 files. It needs the two or three paragraphs that answer this question. So the next part of the app cuts every file into small pieces, a paragraph or two each. These pieces are called chunks.
Sixty files come out empty
The cutting finds no text at all in 60 of the files. They are scans: old trip reports and paper forms that somebody photographed. To a computer, a scan is a picture, not words. So before cutting, each scanned page goes to a model that can read pictures and comes back as text. This step is often called OCR. Now all 400 files can be searched, though handwriting is often misread.
A chunk carries its address
A bare paragraph is not enough to store. Cut out of its page, "After that, the fee is kept" no longer says what it is about. So each chunk is saved with the heading it sat under, its file and page, and the date the file last changed. Each chunk also repeats the last sentence of the one before it, so a fact that straddles a cut is still whole in one of them.
A way to find the right chunks
Two new parts go in. An embedding model gives every chunk a position, so that chunks about the same thing sit close together. An index stores them all. We will call it the meaning index. This is done once, ahead of time, and takes about a minute.
The question takes the same road
Now a member asks. The app sends the question to the same embedding model, so the question gets a position too. The index then returns the chunks that sit nearest to it. Out of 3,900 chunks, three come back.
A much smaller prompt
The app builds the prompt again, with one change. Where the whole handbook used to be, there are now the best two chunks, each with a number. The model reads about four hundred tokens instead of thirty-six thousand.
An answer you can check
The answer comes back with numbers in it. Each number points to a chunk, and each chunk knows its file and page, so the app can show the source beside the answer. A member no longer has to trust it. They can look.
Version 4: when the files do not know
A member asks about dogs. No file mentions them. The index still returns its three nearest chunks, because something is always nearest. So the app checks the scores first. When nothing is close enough, the model is told to say that it does not know.
Version 5: a question with a code in it
A member asks about form T-14. Search by meaning does badly here. To the embedding model, T-14 is just a name with no meaning, so the chunks about forms in general come back and the one that mentions T-14 is not among them.
A second index, for exact words
Exact words need a different kind of index, with no embedding model involved. It is a plain list of every word, and beside each word, the chunks that contain it. The index at the back of a book works the same way. It is built from the same chunks, and sits beside the first one.
Two searches at the same moment
Now every question is searched twice, at once. One copy goes through the embedding model to the meaning index, as before. The other goes straight to the word index, as plain words. Each index sends back its own list, and the two lists do not agree.
One list out of two
The app merges them with a simple rule: a chunk near the top of either list goes near the top of the merged one. The best few go into the prompt. Meaning finds the right topic, exact words find names, codes and numbers. Using both is called hybrid search.
A closer look at the short list
Both searches are fast because they are rough. They compare a question with a chunk without ever reading the two together. So the app now asks each index for twenty chunks, not three, and hands the merged list to a reranker. This small model reads the question beside each chunk and scores how well it answers. Forty careful readings take a fifth of a second. The best three go to the prompt.
A question that leans on the last one
A member asks about cancelling, reads the answer, and then types: and for children? Searched alone, those three words find nothing useful. So before any search, a quick model call rewrites the newest message into a question that stands by itself, using the conversation so far. The search runs on the rewritten question. The member never sees it.
Version 6: a file changes
The committee changes the rule from 7 days to 10. The handbook is updated, but both indexes still hold the old chunks, so the app would keep giving the old answer. The fix: whenever a file is saved, cut it up again and replace its chunks in both indexes. Only that one file is redone.
Version 7: not everyone may read everything
Some files are private, such as the committee's meeting notes. So every chunk is stored with a label saying who may read it. Both indexes only search chunks the person asking is allowed to see. The model never receives the rest, so it cannot leak them.
A budget for the prompt
The prompt has grown: instructions, the chat so far, chunks, and the question. Each part is given a ceiling, and the app trims to it in a fixed order. Old chat turns go first, then the weakest chunk. The instructions and the question are never cut. The best chunk is placed last, next to the question, because models attend best to what sits at the start and the end.
Where the second goes
Six things now happen between a question and its answer. HM Nayem times every one of them on every request, and keeps the record. The model is by far the slowest stage, so its reply is streamed: the first words appear after half a second, while the rest is still being written. When a member says it felt slow, this list shows which stage to blame.
Version 8: write down what right looks like
HM Nayem will not share the link on a feeling that it works. So first comes a test sheet: 20 real questions, each with the answer a careful person would give and the page it should come from. Some have no answer in the files on purpose, to check that the app admits it.
Run all twenty
The app answers every question on the sheet. Each answer is then marked twice: does it say the same thing as the expected answer, and does it point to the right page? A second model call does the comparing, because the wording is never identical. Eighteen pass and two fail.
Read the failures, not the score
The score says something is wrong. The failures say what. For the weekend fee, the app said the files do not say. Look at what the search handed the model: the top row of a table, with the prices cut off into another chunk. The other failure is a table too. The model was fine. The cutting was the fault.
Listening after launch
The test sheet holds the questions HM Nayem thought of. Members will ask others. So every real question is logged with the chunks it found and the answer it got, and each answer has a thumbs-up and a thumbs-down. Once a week HM Nayem reads two lists. Questions the files could not answer tell the club which page to write next. Answers marked down become new rows on the test sheet.
What it still cannot do
The link goes out, with an honest list beside it. This version looks up facts, and that is all. It cannot do arithmetic, count across many files, read a map or settle two files that disagree. Each gap is a part we have not built yet. Earlier stories showed most of the fixes.
The finished product, at real speed
Here is one question through everything we built. Two indexes are searched at once, by meaning and by exact word, for chunks the member may read. The lists are merged and re-sorted by a closer reading, the app builds a small prompt, and the model answers with its sources. The model is one part of nine. The rest is HM Nayem's own code.
You built a document assistant, one part at a time.
- Start with the simplest version that runs, and let each failure tell you what to build next.
- Turn every file into text first, scans included, then cut it into chunks and store them by meaning.
- Store each chunk with its heading, file, page and date, and let neighbours overlap by a sentence.
- Keep two indexes, one for meaning and one for exact words. Search both, merge, then rerank by a closer reading.
- Rewrite a follow-up into a question that stands alone before searching for it.
- Give every part of the prompt a ceiling, time every stage, and stream the answer.
- Number the chunks so answers show their source, and check the scores so the app can say it does not know.
- Keep a sheet of questions with known answers. Read the failures, fix the cause, and run it again.
- After launch, log every question. What the files could not answer, and what was marked down, is your next work.
- Tell people what it cannot do yet. Every gap is a part still to build.
Ideas in this story
- Made-up answers: A model writes what is likely, so a wrong answer sounds exactly as sure as a right one.
- The context window: The fixed amount of text, counted in tokens, a model can take in at once.
- Chunking: Cutting documents into small pieces so the right piece can be found and handed to the model.
- Reading scanned pages (OCR): Turning a picture of a page into text, so that it can be searched and handed to a model.
- Embeddings: Turning text into a list of numbers so that similar meanings end up close together.
- Retrieval-augmented generation: Retrieve the right text, add it to the prompt, then let the model answer from it. RAG for short.
- Grounding: Putting the real text in the prompt and telling the model to answer only from it.
- Hybrid search: Searching by meaning and by exact words at the same time, then merging the results.
- Reranking: A second, more careful model re-sorts the search results so the best few come first.
- Rewriting the question: Turning what the person typed into a complete question that a search can actually find.
- Access control: Deciding in code, not in the prompt, whose data a request is allowed to touch.
- Context engineering: Deciding what goes into the model's window at each step, and what stays out.
- Tracing: Recording every step of a request, with what went in, what came out and how long it took.
- Evals: A fixed set of test questions with known good answers, run after every change to see if things got better or worse.
- A model as judge: Using a second model, with a written rubric, to grade answers that simple code cannot check.
- Monitoring: Watching speed, cost, errors and quality across all requests, with alarms when something shifts.