Context window

How a context window works

Models read and write in tokens, which are chunks of words. Everything in a request takes up room, from the system prompt and earlier messages to pasted documents and tool results. Anthropic's documentation notes that the reply the model writes counts too.

The model keeps nothing between requests beyond what you send. A chat app that seems to remember is resending the history each turn, or a summary of it.

For the model itself, see large language model. What you put into the window, and how you word it, is the subject of prompt engineering.

How big are context windows today?

These are the sizes each provider published for its current models when we checked on 12 November 2026. They change often, and each provider also offers other models with different limits, so check the linked page before you plan around a number.

Sources: Anthropic models overview, OpenAI models and Google Gemini models. Anthropic's page also lists earlier Claude models with 200K-token windows, which is a reminder that the limit belongs to a model, not to a company.

A token is not a word. Anthropic's page puts 1M tokens at roughly 555,000 words on its current tokenizer, so the same document can take a different number of tokens on a different model.

What takes up space in a context window

It helps to think of the window as a set of slots that all draw on the same limit:

• Instructions. The system prompt that tells the model its role, rules and format. It is sent with every request.
• Tool definitions. If the model can call tools, the description of each one is part of the request too.
• The conversation. Every earlier message, the model's own earlier replies included, unless the app trims or summarises them.
• Documents and tool results. Files you pass in, search results, database rows. This is usually the largest and most variable part.
• The reply. Room has to be left for the answer. A request that fills the window leaves the model nowhere to write.

Before you design a feature, add up the parts you control. Most providers offer a way to count tokens for a request before you send it, which turns the estimate into a number.

Why a bigger context window isn't automatically better

Larger windows let you pass long contracts or whole codebases in one go. But Anthropic's docs warn that accuracy and recall degrade as the token count grows, a problem they call context rot.

Every token also costs money and time. Stuffing a request with every document just in case makes answers slower and pricier. They're often worse, too, than if you'd passed the few pages that matter.

Context window vs RAG vs memory

The context window is the model's working space for one request. Retrieval-augmented generation (RAG) is a way to choose what goes into that space, by searching your documents and passing in only the relevant parts.

Memory is a product feature built on top. The app stores facts or summaries somewhere else and adds the useful ones back into later requests. All three meet in the same place: whatever you send is all the model sees.

Long windows haven't made RAG obsolete. Google's Gemini docs encourage passing all relevant information upfront, while saying techniques like RAG remain valuable in specific scenarios. With cost and context rot in the picture, most products still select.

For how retrieval chooses what goes in, see retrieval-augmented generation.

How to tell whether your content will fit

Start with the largest realistic input, not the average one: the longest contract, the busiest customer account, the conversation that has been running for a month. Count its tokens with the provider's own tool, add your instructions and the space the reply needs, and compare the total with the model's limit.

If it fits with room to spare, you may not need retrieval at all for a first version. If it fits only just, plan for growth, because customer data rarely gets smaller. If it does not fit, you need to select: retrieve the relevant parts, summarise the older ones, or split the job into steps.

Then test quality, not only size. Ask the same questions with a small context and a large one. If answers get worse as you add material, the limit that matters for your feature is lower than the one on the model card.

What happens when the context window fills up

Depending on the API, the request fails or the output gets cut short. Chat apps usually drop or summarise old turns, which is why a long conversation can forget early details.

For long-running agents, Anthropic describes compaction. The system summarises a conversation near the limit and starts a fresh window with that summary. It works, but anything the summary leaves out is gone.

How Gr8r manages the context window in AI products

We treat context as a budget. For each feature we decide what the model must always see, what it can look up when needed, and what it never needs.

We measure token use and test answer quality as the context grows. That way a feature doesn't quietly get worse as customers add more data.

We do this as part of product engineering. Tokens are also the unit you are billed in, so the same budget decides the monthly bill. Our post on what drives the cost to build an AI app shows how to estimate it from published prices.