What is the context window?
Everything a model reads for one request has to fit in its context window: the instructions, the whole conversation, any documents, and the reply it is writing. The size is fixed for each model.
Whatever is outside the window does not exist for the model. If it is not in the request, the model does not know it.
A fuller window costs more, takes longer, and makes it easier for the model to overlook a detail. So products trim and summarise rather than send everything they have.
Why does the context window matter?
The window is the hard edge of what a model can work with. Anything outside it does not exist for the model, and everything inside it is paid for on every call. Most of the craft of building AI products is deciding what earns a place in that limited space.
How does the context window work?
- Each model has a fixed limit on the tokens it can take in one call, counting the prompt and the reply together.
- Your code assembles the prompt: instructions, tools, fetched text, the conversation, the new message.
- If the total is over the limit, the call fails. If it is close, there is no room left for the reply.
- So your code measures the prompt before sending, and trims or summarises to stay well inside.
When do you need to manage the window?
Manage it actively when
- Conversations run long
- Documents or search results are added to the prompt
- An agent works through dozens of steps
You can ignore it when
- Requests are short and stand alone
- The total is a small fraction of the limit
Where does the context window sit in a real system?
The limit belongs to the model. Managing it is the job of your code: the part that builds the prompt counts tokens, sets a budget for each section and cuts in a fixed order when the total is too high.
What goes wrong with the context window?
- Filling the window because it is large. Answers get worse, slower and dearer as it fills.
- Forgetting to leave room for the reply.
- Burying the one fact that matters in the middle of a long prompt, where it is most likely to be missed.
What does the context window look like in practice?
A model with a window of 200,000 tokens is sent 20 help pages to be safe. It answers 81 questions in 100 correctly. Sent the 3 best pages only, it answers 94. The window had room for all 20. Using it made things worse.