AI Engineering, story 17What the model actually sees

One reply, and everything behind it

1 of 15

What the model actually sees: AI Engineering, story 17

A customer gets a perfect answer. Open up the single block of text the model read to write it, and see why each part is there and in that order.

Context engineering: the model knows only what is in its window, so what goes in, in what order and how much decides the answer.

The story, step by step

  1. One reply, and everything behind it

    HM Nayem asks the shop's assistant about returning some boots, and gets an answer that is exactly right. It even knows when the boots were bought. In this story we will not follow the message. We will open up the single block of text the model read to write that reply, and ask why each part is there.

  2. The window, opened

    This is all of it. The model was sent one long text made of six parts, joined end to end. The assistant's code built every part. HM Nayem typed only the last line. Deciding what goes into this text, and what stays out, has a name: context engineering.

  3. Nothing else exists

    Two facts about a window shape everything that follows. What is not in it does not exist for the model: it cannot go and look. And everything in it is read together, so every extra line competes with the lines that matter. A good window holds what this one reply needs, and little else.

  4. Every part takes up room

    Here is the same window, measured. The question is twenty tokens. Everything the assistant added around it is over seven thousand. The window has a limit, every token in it is paid for, and a fuller window is a slower one. So each part has to earn its place.

  5. More is not safer

    A tempting idea: put in twenty help pages, to be safe. HM Nayem tries it on two hundred test questions. The answers get worse. With twenty pages in front of it, the model more often answers from the wrong one, or blends two policies into a rule that does not exist. Extra text is not harmless padding. It is noise.

  6. The middle gets lost

    Where a fact sits matters as well. HM Nayem hides one sentence, the one that answers the question, at different places in a long window. At the start or the end, the model nearly always uses it. Buried in the middle, it is missed far more often. Models attend best to the beginning and the end.

  7. An order chosen on purpose

    So the order is a decision. Things that never change go first: instructions and tools. That also lets the model's provider keep that opening ready and charge less for it. Reference material comes next. The chat follows. The question goes last, where attention is strongest, with the one rule that matters most repeated just before it.

  8. Old turns fade into a summary

    By the fortieth message, the chat alone would be most of the window, and its early turns are no longer needed word for word. The assistant keeps the last six turns exactly as they were said. Everything before them is replaced by a short summary, written by a model and saved. The thread is kept. The bulk is not.

  9. The loudest guest is a tool result

    The model asks the orders system for this customer's orders. The system answers the way systems do: every order ever placed, with every field it stores. Nine thousand tokens, of which the model needs four lines. So tool results never go into the window raw. Code keeps the fields this question needs, and says how many rows were left out.

  10. Fetch it when it is needed

    The customer record holds four years of history. Loading all of it into every window, in case it is useful, would bury the question. The better pattern is the opposite: put in a one-line pointer, and give the model a tool to fetch more. Most questions never need it. The ones that do, ask, and pay for it only then.

  11. Labels around everything

    To the model, the window is one stream of text. It cannot see who wrote which part. So the assistant wraps each part in a plain label saying what it is and where it came from. The model can then tell an instruction from a help page, and a help page from something a customer typed. It is also the first defence against text that pretends to be an instruction.

  12. Say each thing once

    Windows grow by accretion. A rule is added to the instructions, a slightly different one sits in a help page, and a third arrived in a tool's description. Faced with three versions, the model picks one, and not always the same one. Each instruction should live in exactly one place. When HM Nayem removed the duplicates, the assistant's instructions shrank by a third and its answers became steadier.

  13. A budget, kept by code

    None of this is left to chance. Each part of the window has a ceiling, and code enforces it on every request. When the total would go over, parts are trimmed in a fixed order, starting with what matters least. Two things are never trimmed: the instructions and the question.

  14. When an answer is wrong, read the window

    The assistant saves the exact window behind every reply. When a reply is wrong, that is the first thing HM Nayem opens, with one question in mind: was the right information in there? For fifty wrong answers, he sorts the causes. The model itself was to blame in seven. In the rest, it was given the wrong window.

  15. The same question, at real speed

    One more time, as it happens. The question arrives. The assistant gathers the customer's record, three help pages and one trimmed order, labels each, puts them in order, and sends one block of text. The model's whole job is to read that block. Most of the skill in an AI product is in building it.

The model reads one block of text. You write it.

Ideas in this story