A context window is the amount of information, counted in tokens, that a language model can take into account in a single request. It holds the instructions, the conversation so far, any documents or tool results, and the model's own reply.
Anthropic's documentation describes the context window as the model's "working memory", separate from the much larger body of data it was trained on. Whatever falls outside the window, the model cannot see on that turn.
What fills the context window on a customer call
Instructions. The system prompt: the agent's role, tone, policies, and what it may and may not do.
Tool definitions and results. Descriptions of the APIs the agent can call, and what they returned, such as an account record.
Retrieved knowledge. Passages from policy documents or help articles, added for the current question.
The conversation. Every customer message and agent reply so far. On a voice call, these are usually transcripts.
The reply. The output being generated counts too, including any reasoning steps, per Anthropic's documentation.
Each turn usually resends all of this, so the window fills up as a conversation goes on. A 40-minute call about a disputed bill can carry many times the history of a two-minute balance check.
Why a bigger window is not automatically better
Anthropic's documentation says "more context isn't automatically better." It names the effect context rot: as the token count grows, accuracy and recall degrade.
Position matters as well. In "Lost in the Middle" (2023), Nelson F. Liu and six co-authors found that models often did best when the relevant information sat at the start or end of the input. Performance dropped significantly when it sat in the middle, even for models built for long contexts.
In a contact center, it helps to send only what the current step needs. A full policy manual in every request costs tokens and may bury the one clause that applies to the caller.
Long calls and long documents
Summarizing earlier turns. Older parts of the conversation are condensed into a short summary, which Anthropic's API calls compaction. The summary has to keep reference numbers, amounts agreed, and promises made.
Retrieving instead of pasting. Retrieval-augmented generation (RAG) brings in only the passages relevant to the question, which also helps grounding.
Trimming tool results. An API that returns the three fields the agent needs uses far less space than a full customer record.
Keeping critical facts outside the model. Amounts, due dates, and disclosure wording can be looked up or computed at the moment they are needed. Then they do not depend on what survived in the window.
When the input exceeds the limit, the request fails or older content is dropped, depending on the system. Anthropic's API, for example, returns an error when the input alone is too long.
Frequently asked questions
How big is a context window?
It depends on the model, and sizes change often. On November 14, 2026, Anthropic's documentation listed 1 million tokens for its current models and 200,000 for some older ones. Google's documentation listed 1,048,576 tokens for its Gemini models.
What happens when a conversation exceeds the context window?
The system has to reject, drop, or summarize something. Some APIs return an error. Others, or the application around them, remove the oldest turns or replace them with a summary. On long calls, it helps to test that key facts such as agreed amounts survive the summary.
Does a larger context window reduce hallucinations?
Not by itself. A larger window lets more source material in, which can help grounding. Very long inputs can also lower accuracy, as context rot and the "Lost in the Middle" results show. Relevant, well-placed content tends to matter more than volume.
Is the context window the same as memory?
No. The context window is what the model can see in one request. AI agent memory is information stored outside the model, such as notes from earlier conversations. The agent loads it into the window when it is relevant.
How we approach this at Zowie
Zowie's AI agents work on voice, chat, and email, with over 100 million conversations a year across our clients. The language model speaks. A separate, deterministic Decision Engine returns every amount, rate, offer, disclosure, and escalation from your rules, with an audit log. To see it on your own long calls and policy documents, you are welcome to book a demo.
Explore: AI tokens, AI agent memory, Large language model (LLM), Retrieval-augmented generation (RAG), AI grounding, Agentic workflow