StoryboardAI Engineering

The context window

The fixed amount of text, counted in tokens, a model can take in at once.

What is the context window?

Everything a model reads for one request has to fit in its context window: the instructions, the whole conversation, any documents, and the reply it is writing. The size is fixed for each model.

Whatever is outside the window does not exist for the model. If it is not in the request, the model does not know it.

A fuller window costs more, takes longer, and makes it easier for the model to overlook a detail. So products trim and summarise rather than send everything they have.

A window is shared between everything sent and the reply still to be written.1. Instructions and tools: 2,300 tokens. 2. Fetched text: 2,800 tokens. 3. The conversation so far: 2,200 tokens. 4. The new message: 20 tokens. 5. Room kept for the reply: 1,000 tokens.Instructions and tools2,300 tokensFetched text2,800 tokensThe conversation so far2,200 tokensThe new message20 tokensRoom kept for the reply1,000 tokens
A window is shared between everything sent and the reply still to be written.

Why does the context window matter?

The window is the hard edge of what a model can work with. Anything outside it does not exist for the model, and everything inside it is paid for on every call. Most of the craft of building AI products is deciding what earns a place in that limited space.

How does the context window work?

  1. Each model has a fixed limit on the tokens it can take in one call, counting the prompt and the reply together.
  2. Your code assembles the prompt: instructions, tools, fetched text, the conversation, the new message.
  3. If the total is over the limit, the call fails. If it is close, there is no room left for the reply.
  4. So your code measures the prompt before sending, and trims or summarises to stay well inside.

When do you need to manage the window?

Manage it actively when

  • Conversations run long
  • Documents or search results are added to the prompt
  • An agent works through dozens of steps

You can ignore it when

  • Requests are short and stand alone
  • The total is a small fraction of the limit

Where does the context window sit in a real system?

The limit belongs to the model. Managing it is the job of your code: the part that builds the prompt counts tokens, sets a budget for each section and cuts in a fixed order when the total is too high.

What goes wrong with the context window?

  • Filling the window because it is large. Answers get worse, slower and dearer as it fills.
  • Forgetting to leave room for the reply.
  • Burying the one fact that matters in the middle of a long prompt, where it is most likely to be missed.

What does the context window look like in practice?

A model with a window of 200,000 tokens is sent 20 help pages to be safe. It answers 81 questions in 100 correctly. Sent the 3 best pages only, it answers 94. The window had room for all 20. Using it made things worse.

Watch it happen

These stories show the context window at work, one step at a time.