What the model actually sees: AI Engineering, story 17
A customer gets a perfect answer. Open up the single block of text the model read to write it, and see why each part is there and in that order.
Context engineering: the model knows only what is in its window, so what goes in, in what order and how much decides the answer.
The story, step by step
One reply, and everything behind it
HM Nayem asks the shop's assistant about returning some boots, and gets an answer that is exactly right. It even knows when the boots were bought. In this story we will not follow the message. We will open up the single block of text the model read to write that reply, and ask why each part is there.
The window, opened
This is all of it. The model was sent one long text made of six parts, joined end to end. The assistant's code built every part. HM Nayem typed only the last line. Deciding what goes into this text, and what stays out, has a name: context engineering.
Nothing else exists
Two facts about a window shape everything that follows. What is not in it does not exist for the model: it cannot go and look. And everything in it is read together, so every extra line competes with the lines that matter. A good window holds what this one reply needs, and little else.
Every part takes up room
Here is the same window, measured. The question is twenty tokens. Everything the assistant added around it is over seven thousand. The window has a limit, every token in it is paid for, and a fuller window is a slower one. So each part has to earn its place.
More is not safer
A tempting idea: put in twenty help pages, to be safe. HM Nayem tries it on two hundred test questions. The answers get worse. With twenty pages in front of it, the model more often answers from the wrong one, or blends two policies into a rule that does not exist. Extra text is not harmless padding. It is noise.
The middle gets lost
Where a fact sits matters as well. HM Nayem hides one sentence, the one that answers the question, at different places in a long window. At the start or the end, the model nearly always uses it. Buried in the middle, it is missed far more often. Models attend best to the beginning and the end.
An order chosen on purpose
So the order is a decision. Things that never change go first: instructions and tools. That also lets the model's provider keep that opening ready and charge less for it. Reference material comes next. The chat follows. The question goes last, where attention is strongest, with the one rule that matters most repeated just before it.
Old turns fade into a summary
By the fortieth message, the chat alone would be most of the window, and its early turns are no longer needed word for word. The assistant keeps the last six turns exactly as they were said. Everything before them is replaced by a short summary, written by a model and saved. The thread is kept. The bulk is not.
The loudest guest is a tool result
The model asks the orders system for this customer's orders. The system answers the way systems do: every order ever placed, with every field it stores. Nine thousand tokens, of which the model needs four lines. So tool results never go into the window raw. Code keeps the fields this question needs, and says how many rows were left out.
Fetch it when it is needed
The customer record holds four years of history. Loading all of it into every window, in case it is useful, would bury the question. The better pattern is the opposite: put in a one-line pointer, and give the model a tool to fetch more. Most questions never need it. The ones that do, ask, and pay for it only then.
Labels around everything
To the model, the window is one stream of text. It cannot see who wrote which part. So the assistant wraps each part in a plain label saying what it is and where it came from. The model can then tell an instruction from a help page, and a help page from something a customer typed. It is also the first defence against text that pretends to be an instruction.
Say each thing once
Windows grow by accretion. A rule is added to the instructions, a slightly different one sits in a help page, and a third arrived in a tool's description. Faced with three versions, the model picks one, and not always the same one. Each instruction should live in exactly one place. When HM Nayem removed the duplicates, the assistant's instructions shrank by a third and its answers became steadier.
A budget, kept by code
None of this is left to chance. Each part of the window has a ceiling, and code enforces it on every request. When the total would go over, parts are trimmed in a fixed order, starting with what matters least. Two things are never trimmed: the instructions and the question.
When an answer is wrong, read the window
The assistant saves the exact window behind every reply. When a reply is wrong, that is the first thing HM Nayem opens, with one question in mind: was the right information in there? For fifty wrong answers, he sorts the causes. The model itself was to blame in seven. In the rest, it was given the wrong window.
The same question, at real speed
One more time, as it happens. The question arrives. The assistant gathers the customer's record, three help pages and one trimmed order, labels each, puts them in order, and sends one block of text. The model's whole job is to read that block. Most of the skill in an AI product is in building it.
The model reads one block of text. You write it.
- The model knows only what is in its window, and uses whatever is there.
- More text is not safer. Extra pages lower the quality of the answer.
- Order matters: fixed parts first, the question last, key facts away from the middle.
- Summarise old turns, trim tool results, and fetch detail only when it is asked for.
- Label every part, say each rule once, and keep a budget in code.
- When an answer is wrong, look first at what the model was shown.
Ideas in this story
- The context window: The fixed amount of text, counted in tokens, a model can take in at once.
- Context engineering: Deciding what goes into the model's window at each step, and what stays out.
- The system prompt: Instructions the product's builders write, placed ahead of every user message.
- Retrieval-augmented generation: Retrieve the right text, add it to the prompt, then let the model answer from it. RAG for short.
- Compaction: Replacing a long record of work with a short summary so the agent can keep going.
- Tool calling: The model cannot do anything itself. It asks your code to, and your code answers.
- Prompt caching: The provider remembers the unchanging start of a prompt, so repeats of it cost about a tenth and arrive faster.
- Tracing: Recording every step of a request, with what went in, what came out and how long it took.