Context window — how much hell fits in a prompt

Golden rule: Context is not memory. It is a workbench. Everything you put on it costs three times: money, latency and model attention stolen from what matters. Curate. Do not dump.

When you need this

Any time you hit “it does not fit the prompt”, “the model forgot what I said an hour ago”, “why is this so expensive”, or “should I use RAG or just paste the whole document?”. The context window is the budget every LLM app spends. Most quality and cost problems trace back to how that budget is managed.

For agents the window is a battlefield: system prompt, tools, history, diffs, logs, web pages. Without curation the closed loop becomes an expensive warehouse of noise.

How it works

A context window is the maximum tokens a model sees in one run — all at once: system instructions, tool schemas, conversation history, documents, tool results and the current request. A token is not a word. It is a text piece. Count conservatively.

Three properties that matter:

1. Attention is uneven. Models read the start and end more reliably; the middle of a long window gets lost (“lost in the middle”). A million-token window is not a million tokens of attention.
2. You pay for every token, repeatedly. In chat, the whole history is resent each turn. Long context makes every round more expensive — hence prompt caching for a stable prefix.
3. More ballast, worse answers. A relevant detail drowned in an archive dump loses to the same detail presented cleanly. Garbage in becomes elegantly processed garbage out.

How to manage the window, step by step

1. Measure what is in it. Count system prompt, tool descriptions and typical task context. People are often shocked that half the budget is unread boilerplate.

2. Slim the stable part. Keep system prompt and tool schemas dense. Every sentence pays rent. Keep them stable — an unchanged prefix is the condition for useful caching.

3. Choose what you insert. Sections relevant to the question, not whole documents; last N log lines plus the error, not the full dump. When the corpus outgrows reason, that is RAG time — relevance selection is the whole point.

4. Put critical information at the edges. Instructions up top, current question at the bottom, ballast nowhere. The middle of a long context is where facts go to die.

5. Manage conversation history. Summarize long chats and drop older turns. When the thread fills with dead ends, start clean with a sharper brief. Contaminated history drags quality down.

6. Verify with evals. Measure answer quality against context length and composition on your task. Needle-in-a-haystack graphs look great and say little about your app. For agents also track success rate and cost per completed task, not only “it fit”.

Context vs memory vs RAG

Context is what the model sees right now. Memory chooses what survives between runs (summaries, notes, structured store). RAG finds relevant pieces and adds them to context. Long windows can replace RAG for small corpora. More often they complement it: RAG selects, the window holds.

Practical 2025–26 decision: small stable docs → long context + cache. Large live corpus → RAG (often hybrid). Multi-hop over selected docs → RAG candidates + a large enough window. Tool-using agents → keep only what the next step needs; let tools fetch the rest.

Common mistakes

• “It fits, so dump it all” → fits ≠ belongs.
• Critical instruction buried mid-100k tokens → edges only.
• Mutating the system prompt every turn → you break cache and pay full price; put dynamic bits at the end.
• Treating the window as long-term memory → it dies when the run ends; persist deliberately.
• Endless conversations → summarize or restart.
• Trusting marketing numbers → “1M tokens” is capacity, not quality.
• Agents with unfiltered tool dumps → API payloads and full logs kill budget and attention; summarize and filter.

When long context is enough (and RAG is waste)

Small stable corpora (tens of pages), one-shot large-document analysis, synthesis across a whole file rather than detail lookup. With prompt caching, “paste once, ask many times” can beat a fragile RAG pipeline on cost and quality. Find the boundary by measuring both on the same eval set.

Sources

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)
• Effective context engineering for AI agents (Anthropic)
• Anthropic docs: Context windows
• AI Engineering (Chip Huyen)

What to remember

A context window is the model’s workbench. A larger desk helps, but it will not sort the papers for you. Measure what sits on it; keep stable parts stable for cache; put important facts at the edges; summarize history; insert a selection, not an archive. Good systems combine context curation, RAG, memory and evals — not brute-force million-token dumping.