Definition

The context window, also called context length, is the maximum number of tokens a large language model can take in and refer to in a single pass. Think of it as the model’s working memory: anything inside the window, the model can see and use; anything outside it might as well not exist.

The window is usually shared by input and output. System prompts, conversation history, uploaded documents, tool results and the answer being generated all count toward it. Many services also set a separate cap on how long a single response can be.

How It Works

Why there’s a limit. LLMs are built on the Transformer architecture, whose attention mechanism computes relationships between positions in the sequence. In standard implementations, compute and memory grow roughly with the square of sequence length. During inference, the model also caches intermediate results for processed content (the KV cache), which grows linearly with length. Longer means more expensive and slower.

Limits from position information. Models rely on positional encoding to know where each token sits. If a model only saw sequences up to a certain length during training, it often does worse on longer input. Researchers have developed ways to extrapolate and interpolate positional encodings, and add long-text data late in training, to extend the length a model can use effectively.

Engineering optimizations. Memory-efficient attention implementations and compression and reuse of the KV cache have made longer windows practical.

A big window isn’t the same as good use of it. Research and practice both show that models don’t find and use information evenly across very long input. They sometimes handle the beginning and end well and the middle less well. Fitting content in and using it well are two different things.

What happens past the limit. It depends on the app: some return an error, some silently drop the oldest messages, some compress earlier content into a summary. That’s a common reason long chats seem to “forget” what came first.

Typical Applications

Larger context windows help with:

  • Long documents: reading a contract, report or paper in one go for summaries or Q&A.
  • Code understanding: referencing many source files at once to grasp a project before editing.
  • Long conversations: keeping more history so you repeat yourself less.
  • Agent tasks: holding tool-call logs and intermediate results across many steps.

Practical tips:

  • Put the most important instructions where they’re easy to find, and restate them after long material if needed.
  • Leave out content unrelated to the question. It saves money and improves accuracy.
  • With large amounts of material, favor retrieval-augmented generation (RAG) and include only relevant passages.
  • In long chats, summarize periodically and replace full history with the summary.
  • Token: the unit the context window is measured in.
  • Attention mechanism: its cost is a major reason windows are limited.
  • RAG: supplies relevant material on demand when the window is limited.
  • Hallucination: very long input can lead models to miss or confuse details.
  • AI agent: needs careful context management so execution logs don’t fill the window.

Paper: Lost in the Middle: How Language Models Use Long Contexts