AI Engineering, story 36Build: ask the handbook

The job

The problem1 of 28

Build: ask the handbook: AI Engineering, story 36

HM Nayem's hiking club has 400 files and the same ten questions every week. Build the page that answers them, one part at a time, and watch each version fail in a new way.

A document assistant is built in versions: ask, paste, read scans, cut up, search two ways, cite, refuse, refresh, restrict, test, and list what is still missing.

The story, step by step

  1. The job

    HM Nayem runs a hiking club. Its rules and history sit in 400 files, and members ask the same questions every week. So HM Nayem will build a page where anyone can ask, and get an answer with the place it came from. We will build it one part at a time.

  2. Version 1: just ask a model

    The first version is twenty lines of code. The app takes the question, sends it to a model and shows what comes back. It works, and the answer sounds certain. It is also wrong: the club's rule is seven days, not 24 hours.

  3. Version 2: hand it the handbook

    The model guessed because it has never read the club's files. So version 2 reads the handbook and places all 60 pages in the prompt, above the question. Now the model answers from the real text, and the answer is right.

  4. Why version 2 will not last

    Two problems. Every question makes the model read all 60 pages again, so each answer is slow and costs real money. And the handbook is one file of 400. All of them together are far more than a model can read at once.

  5. Version 3: cut the files up

    The model does not need 400 files. It needs the two or three paragraphs that answer this question. So the next part of the app cuts every file into small pieces, a paragraph or two each. These pieces are called chunks.

  6. Sixty files come out empty

    The cutting finds no text at all in 60 of the files. They are scans: old trip reports and paper forms that somebody photographed. To a computer, a scan is a picture, not words. So before cutting, each scanned page goes to a model that can read pictures and comes back as text. This step is often called OCR. Now all 400 files can be searched, though handwriting is often misread.

  7. A chunk carries its address

    A bare paragraph is not enough to store. Cut out of its page, "After that, the fee is kept" no longer says what it is about. So each chunk is saved with the heading it sat under, its file and page, and the date the file last changed. Each chunk also repeats the last sentence of the one before it, so a fact that straddles a cut is still whole in one of them.

  8. A way to find the right chunks

    Two new parts go in. An embedding model gives every chunk a position, so that chunks about the same thing sit close together. An index stores them all. We will call it the meaning index. This is done once, ahead of time, and takes about a minute.

  9. The question takes the same road

    Now a member asks. The app sends the question to the same embedding model, so the question gets a position too. The index then returns the chunks that sit nearest to it. Out of 3,900 chunks, three come back.

  10. A much smaller prompt

    The app builds the prompt again, with one change. Where the whole handbook used to be, there are now the best two chunks, each with a number. The model reads about four hundred tokens instead of thirty-six thousand.

  11. An answer you can check

    The answer comes back with numbers in it. Each number points to a chunk, and each chunk knows its file and page, so the app can show the source beside the answer. A member no longer has to trust it. They can look.

  12. Version 4: when the files do not know

    A member asks about dogs. No file mentions them. The index still returns its three nearest chunks, because something is always nearest. So the app checks the scores first. When nothing is close enough, the model is told to say that it does not know.

  13. Version 5: a question with a code in it

    A member asks about form T-14. Search by meaning does badly here. To the embedding model, T-14 is just a name with no meaning, so the chunks about forms in general come back and the one that mentions T-14 is not among them.

  14. A second index, for exact words

    Exact words need a different kind of index, with no embedding model involved. It is a plain list of every word, and beside each word, the chunks that contain it. The index at the back of a book works the same way. It is built from the same chunks, and sits beside the first one.

  15. Two searches at the same moment

    Now every question is searched twice, at once. One copy goes through the embedding model to the meaning index, as before. The other goes straight to the word index, as plain words. Each index sends back its own list, and the two lists do not agree.

  16. One list out of two

    The app merges them with a simple rule: a chunk near the top of either list goes near the top of the merged one. The best few go into the prompt. Meaning finds the right topic, exact words find names, codes and numbers. Using both is called hybrid search.

  17. A closer look at the short list

    Both searches are fast because they are rough. They compare a question with a chunk without ever reading the two together. So the app now asks each index for twenty chunks, not three, and hands the merged list to a reranker. This small model reads the question beside each chunk and scores how well it answers. Forty careful readings take a fifth of a second. The best three go to the prompt.

  18. A question that leans on the last one

    A member asks about cancelling, reads the answer, and then types: and for children? Searched alone, those three words find nothing useful. So before any search, a quick model call rewrites the newest message into a question that stands by itself, using the conversation so far. The search runs on the rewritten question. The member never sees it.

  19. Version 6: a file changes

    The committee changes the rule from 7 days to 10. The handbook is updated, but both indexes still hold the old chunks, so the app would keep giving the old answer. The fix: whenever a file is saved, cut it up again and replace its chunks in both indexes. Only that one file is redone.

  20. Version 7: not everyone may read everything

    Some files are private, such as the committee's meeting notes. So every chunk is stored with a label saying who may read it. Both indexes only search chunks the person asking is allowed to see. The model never receives the rest, so it cannot leak them.

  21. A budget for the prompt

    The prompt has grown: instructions, the chat so far, chunks, and the question. Each part is given a ceiling, and the app trims to it in a fixed order. Old chat turns go first, then the weakest chunk. The instructions and the question are never cut. The best chunk is placed last, next to the question, because models attend best to what sits at the start and the end.

  22. Where the second goes

    Six things now happen between a question and its answer. HM Nayem times every one of them on every request, and keeps the record. The model is by far the slowest stage, so its reply is streamed: the first words appear after half a second, while the rest is still being written. When a member says it felt slow, this list shows which stage to blame.

  23. Version 8: write down what right looks like

    HM Nayem will not share the link on a feeling that it works. So first comes a test sheet: 20 real questions, each with the answer a careful person would give and the page it should come from. Some have no answer in the files on purpose, to check that the app admits it.

  24. Run all twenty

    The app answers every question on the sheet. Each answer is then marked twice: does it say the same thing as the expected answer, and does it point to the right page? A second model call does the comparing, because the wording is never identical. Eighteen pass and two fail.

  25. Read the failures, not the score

    The score says something is wrong. The failures say what. For the weekend fee, the app said the files do not say. Look at what the search handed the model: the top row of a table, with the prices cut off into another chunk. The other failure is a table too. The model was fine. The cutting was the fault.

  26. Listening after launch

    The test sheet holds the questions HM Nayem thought of. Members will ask others. So every real question is logged with the chunks it found and the answer it got, and each answer has a thumbs-up and a thumbs-down. Once a week HM Nayem reads two lists. Questions the files could not answer tell the club which page to write next. Answers marked down become new rows on the test sheet.

  27. What it still cannot do

    The link goes out, with an honest list beside it. This version looks up facts, and that is all. It cannot do arithmetic, count across many files, read a map or settle two files that disagree. Each gap is a part we have not built yet. Earlier stories showed most of the fixes.

  28. The finished product, at real speed

    Here is one question through everything we built. Two indexes are searched at once, by meaning and by exact word, for chunks the member may read. The lists are merged and re-sorted by a closer reading, the app builds a small prompt, and the model answers with its sources. The model is one part of nine. The rest is HM Nayem's own code.

You built a document assistant, one part at a time.

Ideas in this story