The context window, explained without the jargon
Why AI chats forget things, why long documents get skimmed, and what you can do about it. The context window in plain English.
, 2 min read
Ever had a long chat where the AI suddenly forgot something you told it an hour ago? Or pasted a long report and got an answer that ignored the middle? Both come down to one idea: the context window.
The model's desk
Think of the context window as the model's desk. Everything it can work with right now has to fit on the desk: your messages, its replies, any documents you pasted, and its own standing instructions. Anything not on the desk does not exist for it.
The size of the desk is measured in tokens, which are chunks of words. A rough rule: 100 tokens is about 75 English words. Current models have large desks, often hundreds of thousands of tokens, which is several long books. But large is not the same as unlimited, and a full desk is a messy desk.
Why chats forget
In a long chat, every message stays on the desk until it fills up. Then the app has to make room, usually by dropping or summarising the oldest parts of the conversation. That is when the AI forgets your early instructions.
What to do: for long projects, start a fresh chat now and then, and open it with a short summary of what matters. Or put the standing instructions somewhere permanent, like a project's instructions or a skill, so they never fall off the desk.
Why long documents get skimmed
Even when everything fits, models pay most attention to the start and the end of what they are given. Details buried in the middle of a long document are easier to miss.
What to do:
- Put your question before and after the document, not just before.
- Ask for quotes from the document, so you can see what it actually read. The answer from documents prompt does this.
- For very long material, work in sections, then combine the results.
Why it matters for cost and speed
If you use AI through an API, you pay for every token on the desk, every time you send a message. A long chat gets slower and more expensive with each turn. Techniques like prompt caching and RAG, which fetches only the relevant pieces of a large library, exist to keep the desk small.
The short version
- The context window is everything the model can see at once.
- When it fills up, the oldest material goes first.
- Even within it, the middle gets the least attention.
- Keep the desk tidy: fresh chats for new topics, standing instructions in settings, and only the material you need.
For the full definition, see the glossary entry on the context window.