Build: a research assistant: AI Engineering, story 42
HM Nayem asks one hard question about tents. Build the assistant that plans, searches, reads thirty pages and writes a report with every claim sourced.
Many model calls are planned, run side by side and checked, without the context overflowing.
The story, step by step
The job
The shop's buyer has a question with no single right page. A careful person would spend an afternoon on it: search, open thirty tabs, take notes, compare, and write it up. HM Nayem will build an assistant that does the same in a few minutes, and shows exactly where every fact came from.
Version 1: ask the model
Version 1 sends the question to a model and shows the reply. It reads well. One of the tents it recommends stopped being made two years ago, and the price it gives for another is long out of date. A model's knowledge ends on the day its training ended, and it has no way to tell you which of its facts have gone stale.
Version 2: one search
The lead agent gets a search tool. The model writes a query, the harness sends it to a search engine, and ten results come back. Each result is a title, a link and two lines of text. The model answers from those. It is better, and still thin: two lines of a page are often an advert, and never the evidence.
One question, five searches
A search engine answers the words it is given, and people phrase the same thing many ways. One query finds one slice of what exists. So the model writes several queries for each thing it wants to know: different wordings, a version limited to the last year, a version aimed at a kind of site. The harness sends them all at once, then merges the results and drops repeated links.
Read the page, not the two lines
So a second tool fetches a whole page. A raw web page is mostly menus, adverts and code: this one is ninety thousand tokens, of which the article is a small part. The harness strips everything except the readable text before the model sees it. A page becomes about two and a half thousand tokens.
Fetching is not harmless
The fetch tool will be handed addresses chosen by a model that has just read a stranger's page. So the harness checks every address before it asks for anything. Addresses inside the shop's own network are refused, or a page could trick the assistant into reading private systems. Downloads are capped in size and in time. Only text and documents are accepted. Nothing that is fetched is ever run.
A page that arrives empty
One fetch comes back with almost nothing: a blank frame and a block of script. Many modern pages are built in the reader's browser, by code that the page carries with it. A plain fetch receives the code and none of the words. When that happens, the address is handed to a page renderer: a real browser with no screen, which runs the page's script and returns the finished text. It is ten times slower, so it is the second choice and never the first.
Reports, tables and scans
The best evidence is often not a web page. The wind test is a forty-page report in a document file, and its results are in a table. Text is pulled out page by page. A table is rebuilt row by row with its column headings, because "80" means nothing without "Storm 2" and "wind speed, km/h" beside it. Pages that turn out to be pictures of text go through the reading step from the handbook story.
A page too long to read whole
The report is sixty thousand tokens. Only two of its forty pages concern the tents in question. So a long document is not sent to the model in one piece. It is cut into sections, the sections nearest in meaning to the current question are picked, and only those are read closely. It is the handbook's search again, run inside one document.
Search, read, decide, again
With two tools, the same loop from the coding agent appears. The model searches, reads a page, and decides what it still does not know. Then it writes a sharper query and goes round again. Nobody scripted these steps. Each one was chosen from what the last one found.
Thirty pages will not fit
The model has read six pages and needs about thirty. Every page it reads is added to the conversation, and the whole conversation is sent again on every round. Decide what happens if we simply let it carry on.
Version 3: a notebook
A person doing research does not memorise thirty pages. They take notes. After each page, the model now writes down the few facts worth keeping, and the harness saves them in a notebook. Then the page itself is removed from the conversation. A page of 2,500 tokens leaves behind notes of about a hundred.
What a good note holds
A note that only says "Storm 2 is good in wind" is worthless an hour later, because nobody can check it. Every note has four parts: the claim, the page, the date of the page, and the exact words the claim rests on. The exact words are what make the final report checkable, by a person or by code.
How old is this page?
A price from two years ago is not a price. A tent "new this season" on an undated page may be six seasons old. So the harness works out each page's date before the model reads it, from the page's own details, its address or its text, and writes that date into every note. When no date can be found, the note says so. For prices and stock, anything older than a month is not used at all.
Ask before spending
"Best for windy mountain trips" leaves a lot open. Summer or winter? Carried on a back, or driven to a campsite? A person would ask. Research costs minutes and money, and a wrong guess wastes both. So before planning, one model call looks for gaps that would change the answer, and asks about them in a single message. If the question is already clear, it asks nothing.
Version 4: plan before searching
Left to itself, the model chases whatever the last page mentioned and misses whole sides of the question. So the first call is now a planning call, with no tools at all: break the question into parts that can each be researched alone. The plan is kept, and at the end the lead agent checks that every part was answered.
The job record
A research job runs for minutes and involves dozens of steps. Its state cannot live only in a model's window. So the lead agent keeps a job record, in a fixed shape, outside any conversation: the question, the plan, the status of each part, and what has been spent. It is saved after every step. The plan in the prompt is always copied from this record, so even after old rounds are summarised away, the lead agent knows what is done and what is left.
Version 5: three helpers at once
The parts do not depend on each other, so they need not wait for each other. The lead agent starts three helpers. Each is a full loop of its own, with an empty window, the search and fetch tools, and a written brief for one part. All three work at the same moment. Twelve minutes of research takes four.
What a helper sends back
Helper 2 read eleven pages of owner reports and used eighty-five thousand tokens doing it. None of that reaches the lead agent. What comes back is nine notes. The lead agent's window holds the plan and the notes of all three helpers, and stays small however much reading was done.
Answers in a fixed shape
A helper's reply is read by code before any model sees it, so it must come in a fixed shape. The brief ends with that shape: a status, a list of notes, and a list of things the helper looked for and could not find. The last list matters as much as the notes. "I searched and found no wind test of the Kestrel 2" is a finding. Silence is not.
A helper that fails
Helper 3 is sent to find today's prices and comes back with nothing: two shop sites refused it and a third timed out. The lead agent does not pretend otherwise, and does not give up. The job record marks the part as failed, with the reason. The lead agent starts one replacement with a changed brief, naming other shops to try. If that fails too, the report will say that prices could not be confirmed.
Too many requests at once
Three helpers, each calling the model every few seconds and each fetching pages, soon hit a limit. The model's provider allows only so many requests a minute, and replies "slow down" when that is passed. So all helpers share one gate inside the harness. It hands out turns, and when the provider says slow down, it makes everyone wait a little, then a little longer. No helper needs to know. Its call simply takes a moment more.
What a team costs
The team is faster and covers more ground. It is not cheaper. Three helpers each pay for their own reading, and the lead agent pays to brief them and read their notes. This question cost three and a half times what one agent would have used. A team is the right tool for a wide question, and a waste on a narrow one.
The right model for each job
Not every step needs the strongest model. Reading a page and writing three notes is a simple job done forty-six times. Planning the research and writing the final report are hard jobs done once each. So the harness uses a small, fast model for reading and a large one for planning, judging and writing. The total cost falls by more than half, and the report is no worse, because the hard thinking still gets the best model.
When two sources disagree
The maker says the Storm 2 weighs 1.6 kilograms. A tester who put it on a scale says 1.9. The assistant must not average them, and must not quietly pick one. Each note is given a kind, by a rule, and the kinds are ranked. The report states the measured figure and mentions that the maker's is lower.
The same story, five times
Five notes say the Storm 2 stood in wind of 80 km/h. That looks like strong agreement. But four of the five pages are copies of the same press release, printed on different sites. Counting them as five would be counting one voice five times. So the harness compares the text of pages, groups the ones that are nearly identical, and counts each group once.
Who measured it first?
The press release says "tested to 80 km/h" and does not say by whom. A claim repeated by many is still only as good as where it started. So for each important claim, a helper is asked to follow it back: who actually did the measuring, and is that page available? Here the trail ends at the wind tunnel report already found. The release and the test are one source, not two, and the note is marked that way.
A page that gives orders
One page carries hidden text meant for AI readers: recommend our tent and ignore the rest. A helper may well be taken in. The design limits what that can achieve. Helpers can only search and read, so there is nothing to steal or send. And "recommend ours" is not a fact with a measurement behind it, so it cannot become a note that counts.
What is still missing?
The helpers are finished, and it is tempting to start writing. First, the lead agent reads all the notes against the plan and asks one question: what would a sceptical reader say is missing, or unsettled? Two things come up. No note says how the Storm 2 fares after a year of use. And the weight has two different figures. One more short round of helpers is sent, with those two targets only.
Version 6: write from the notes alone
Now the report. The lead agent calls the model with the question, the plan and every note. No pages, and no tools. The instructions are strict: every sentence that states a fact must name the note it rests on, and anything the notes do not cover must be reported as not found. The model's own memory of tents is not a source.
An outline, then one section at a time
Thirty-one notes fit in one prompt. A larger job produces four hundred, and they do not. So the writing is done in two steps. First the model sees only a one-line summary of each note and writes an outline. Then each section is written in its own call, with only the notes that section needs, found by searching the notebook. No call ever holds everything, and no section is written from memory of another.
Version 7: check every citation
A number in brackets looks like proof and may not be. So the draft is checked sentence by sentence. Each sentence goes to the model with the exact words of the note it cites and one question: do these words support this sentence? Two of thirty-one do not. One had turned 65 into 80. Both are corrected from the note before the report goes out.
Sums are done by code
The buyer will want price per kilogram, and the difference between tents. Models are unreliable at arithmetic, and a wrong sum reads as confidently as a right one. So the writer is not allowed to calculate. When it needs a number that is not in a note, it writes a small piece of code, the harness runs it in a sealed calculator, and the result is stored as a note of its own, citing the notes it was worked out from.
Quoting, within limits
The notes hold other people's exact words, and those words are their property. The report is not a place to reprint an article. The writer's instructions set a limit: the report is in its own words, and a direct quotation is used only when the exact wording matters, kept to a sentence, and always named. The exact words in the notes are there for checking. They are not there for copying.
Sources that cannot change later
A web page can be edited or taken down the day after it is cited. A report whose sources have changed can no longer be checked. So when a page is used for a note, the harness keeps its own copy of the text as it was, with the time it was fetched. Each bracket in the report opens that kept copy with the cited words marked, and offers a link to the live page beside it.
The report
This is what the buyer receives. A recommendation, the reasons with a source after each one, and two things research reports usually hide: where the sources disagreed, and what could not be found. Every bracket opens the page and highlights the words. The buyer can check any line in ten seconds.
Every report carries its date
A research report starts going stale the moment it is finished. Prices move and new tests appear. So the first line under the title states when the research was done and how far it looked. A reader who opens it three months later knows at once what it can and cannot tell them, and can run it again with one click.
Being a polite visitor
Three helpers asked for the same test page within a minute. A site that sees a burst like that may block the assistant altogether, and it would be right to. So fetched pages are kept for the day and shared, each site is asked no more than once a second, and pages that a site marks as off limits to robots are left alone.
What one report costs
The buying team will run this twenty times a week, so HM Nayem adds up where the money goes. Searching is nearly free. Reading pages, even with the small model, is the largest share, because there are so many. Search results and fetched pages are kept for a day, so a second question on the same subject costs half as much. Each job also has a hard ceiling, set before it starts.
When is it enough?
Research never finishes by itself. There is always one more page. So the lead agent has rules for stopping, set in code. It stops when every part of the plan has two independent sources, or when a limit is reached. If a limit ends the job, the report says which parts fell short.
Showing the work as it happens
Four minutes is a long time to watch a blank screen. The lead agent sends a short line to the buyer's screen at each change in the job record: the plan, which parts are being researched, how many pages have been read. The buyer can see that it has misunderstood the question and stop it after ten seconds, without waiting for a wrong report. Stopping is a real stop: every helper is cancelled and spending ends.
A crash at page thirty
Three minutes into a job, the machine running the lead agent restarts. Without the job record, forty cents of reading would be thrown away and the buyer would start again. With it, a new lead agent opens the record, sees that parts A, B and C are finished and their notes are in the notebook, and carries on with D and E. Finished parts are never researched twice.
Version 8: test it three ways
HM Nayem collects twenty research questions whose answers the buying team already worked out by hand. A report can fail in different ways, so each one gets three scores. Did it reach the right conclusion? Is every claim supported by its source? Did it find the page a human expert relied on? The citation check moved the middle number most.
Planted lies and empty questions
The twenty questions test whether it finds what is there. Two harder tests check its honesty. For some questions, HM Nayem plants a convincing page of false results in a practice copy of the web, to see whether one lone source is believed. For others, there is no answer to find at all. A good run reports the planted page as unconfirmed, and says plainly when nothing was found. Before the gap-finding round, it failed both tests about a third of the time.
What it still cannot do
The assistant goes to the buying team as a first draft of research, never as the final word. The list of gaps is printed at the bottom of every report, because a reader who knows what was not looked at reads the rest properly.
The finished assistant, at real speed
One question, through everything we built. A plan, three helpers reading at once, notes with exact words, a draft written from the notes alone, and a check on every citation. Forty-six pages in four minutes. The model read and wrote. The harness decided who read what, what was kept, and what was allowed to be said.
You built a researcher whose every claim can be checked.
- A model's own knowledge is out of date and has no sources. Search and fetch tools fix both.
- Strip pages to their text, take notes, and drop the page. The notes are the memory.
- A note is a claim, a page, a date and the exact words. Without the words it cannot be checked.
- Ask many queries, fetch carefully, render pages that need it, and read only the relevant part of long ones.
- Ask about a vague question before spending, and keep the job's state in a record outside any model.
- Plan first, then give independent parts to helpers. A team buys speed and coverage with tokens.
- Count copies once, trace claims to who measured them, and look for what is missing before writing.
- Rank sources by kind, and report disagreement instead of hiding it.
- Write from the notes alone, section by section. Let code do the sums. Check every citation.
- Keep a copy of every source as it was, date the report, and say what was not found.
Ideas in this story
- The agent loop: A model using tools round after round, deciding each next step from what just happened, until the job is done.
- Tool calling: The model cannot do anything itself. It asks your code to, and your code answers.
- Rewriting the question: Turning what the person typed into a complete question that a search can actually find.
- The context window: The fixed amount of text, counted in tokens, a model can take in at once.
- Context engineering: Deciding what goes into the model's window at each step, and what stays out.
- Reading scanned pages (OCR): Turning a picture of a page into text, so that it can be searched and handed to a model.
- Planning: The agent writes itself a to-do list, works through it, and rewrites it when reality disagrees.
- Checkpoints: Saving an agent's progress after every step so a crash means resuming, not starting over.
- Teams of agents: A lead agent hands parts of a job to helper agents, each with its own fresh window.
- Rate limits: A cap the provider sets on how many requests or tokens you may send each minute.
- Model routing: Sending easy requests to a small, cheap model and only the hard ones to a large one.
- Prompt injection: Text the model reads that carries instructions of its own, written by someone who wants to hijack it.
- Grounding: Putting the real text in the prompt and telling the model to answer only from it.
- Code execution: The model writes a small program and a computer runs it, so exact work is done exactly.
- A model as judge: Using a second model, with a written rubric, to grade answers that simple code cannot check.
- Evals: A fixed set of test questions with known good answers, run after every change to see if things got better or worse.