A 1M-token window is an attention budget, not a hard drive — and the model reads the middle of it like a tired commuter skims the middle of a long email.
A 1M-token context window is an attention budget, not a hard drive. Every token the model emits must, in principle, weigh its relationship against every token it can see — an n² problem — so as you pour in more tokens, the share of attention any single fact can claim shrinks. The empirical fingerprint of this is "lost in the middle" (Liu et al., 2023): accuracy on retrieval-style tasks traces a U-shaped curve — high when the needed fact sits at the very start or very end of the context, sagging 20–30% when it sits in the middle. This is not a bug in positional embeddings you can patch; it is a budget and recency effect that survives even on million-token windows. The senior move is therefore to treat context as something you engineer — retrieve the right facts, place them at the edges, rerank and compress — rather than something you fill. The model with the biggest window does not win; the team with the best context discipline does.
The words first.
Step by step.
Remember this: a bigger window lets you fit more, not attend to more — placement and retrieval beat stuffing.
The transformer's core operation is scaled dot-product attention: for a sequence of n tokens, each output position computes a weighted sum over all n input positions, giving an n × n matrix of pairwise interactions. Two consequences follow directly. First, compute and memory scale as n² (FlashAttention and friends reduce the constant and the memory footprint, but the fundamental pairwise structure remains). Second, and more important for this lesson, the softmax that produces attention weights is a normalized distribution — the weights over all positions sum to 1. So attention is a zero-sum resource: every token competes for a fixed total of attention mass. Add 900K tokens of distractors around your one important sentence and, all else equal, that sentence's share of the budget collapses. This is the first-principles reason "more context" is not "more knowledge."
The answer the model needs sits at position 10 of 20 — retrieval accuracy there is ≈56%. Drag it to the edges (recency + primacy) and accuracy climbs; bury it in the middle and the model often misses it. The dip deepens with longer context — which is why you rerank and put the best evidence first or last, not why you stuff everything in.
Liu et al. ran a clean experiment: multi-document question answering with 20 documents, where exactly one document contains the answer and the rest are relevant-looking distractors. They swept the position of the gold document from 1 to 20 and measured accuracy across GPT-3.5, GPT-4, Claude, and open models. The result was strikingly consistent — a U-shaped curve:
Name the symbols in plain words:
pos — where the gold document sits among 20 docs (1 = first, 20 = last).acc(pos) — task accuracy when the gold doc is at that position.Concrete (illustrative numbers in the spirit of Liu et al.'s shape):
acc(1) = 0.75 — gold at the front, model nails it (primacy).acc(20) = 0.73 — gold at the back, almost as good (recency).acc(12) = 0.50 — gold buried in the middle.Compute the middle penalty: 0.75 - 0.50 = 0.25, i.e. a 25% absolute drop just from moving the same fact from the edge to the middle. Nothing about the fact changed — only its position. That is the entire phenomenon in one subtraction: position is a first-class lever on accuracy, so where you put a fact is part of your prompt's correctness, not cosmetics.
A tempting hypothesis: "the model's positional encoding (RoPE, ALiBi, etc.) just degrades in the middle." Liu et al. and the follow-up TACL analysis argue against this. The U-shape is asymmetric in a way pure position can't explain — recency (end) is favored partly by the causal attention mask and the training objective (next-token prediction makes recent tokens highly predictive), while primacy (start) is favored because the system prompt and earliest tokens are attended to by every downstream position and often anchored by training. The middle has neither advantage. So the mechanism is (a) a finite attention budget the middle must compete hardest for, plus (b) a recency bias baked into causal LM training, not a coordinate bug. This matters for interviews because it tells you the fix is placement and pruning, not "wait for better positional embeddings."
In June 2026, Opus 4.8, Sonnet 4.6, and Haiku 4.5 ship 1M-token windows (Sonnet 4.5 and earlier: 200K; max single-request output: 128K). But Anthropic's own framing is blunt: usable performance depends on context engineering — placement, compression, retrieval — not window size alone. As tokens accumulate, accuracy degrades non-linearly and gradually ("context rot") rather than failing at a hard limit. The practical reading: you should not plan to operate at 95% of the window and expect 95% of the quality. The window is a ceiling on what's possible, with a quality gradient underneath it.
These are not rivals; they are layers. Long context is a capability (the model can hold 1M tokens); RAG is a discipline (you choose to hold only the right ~10K). Use raw long-context when the task genuinely needs global reasoning over a coherent body (one 300-page contract, a full codebase diff) where chunking would sever cross-references. Use RAG when the corpus is large, mostly irrelevant per query, and changes often. Most production systems are hybrid: retrieve to narrow the field, then let the long window hold the retrieved set plus working state — and measure on a long-context benchmark like MRCR v2 (which replaced the now-saturated Needle-in-a-Haystack) rather than trusting that "it fit."
A runnable demo that reproduces the U-curve locally: we build a context of n filler documents, slot a "needle" fact at a chosen position, ask the model to retrieve it, and sweep position to plot accuracy. This is the spine of the lost-in-middle demo and a real eval harness you could run in CI.
import os
from anthropic import Anthropic
client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
MODEL = "claude-opus-4-8" # 1M-token window, June 2026
# A distinctive needle the model can't guess from priors.
NEEDLE = "The access code for vault Theta-9 is QUARTZ-4417."
QUESTION = "What is the access code for vault Theta-9? Answer with only the code."
# Cheap, semantically-neutral filler so we measure position, not content difficulty.
def filler(i: int) -> str:
return f"Log entry {i}: routine telemetry nominal; no anomalies recorded; sensor {i} ok."
def build_context(n_docs: int, needle_pos: int) -> str:
docs = [filler(i) for i in range(n_docs)]
docs[needle_pos] = NEEDLE # overwrite one slot with the needle
return "\n".join(docs)
def ask(context: str) -> str:
resp = client.messages.create(
model=MODEL,
max_tokens=32,
# Stable instruction first -> cache-friendly + primacy slot for the task.
system="You answer retrieval questions from the provided log. Be exact.",
messages=[{
"role": "user",
"content": (
f"{QUESTION}\n\n<logs>\n{context}\n</logs>\n\n{QUESTION}"
), # question repeated at start AND end = both edge slots
}],
)
return resp.content[0].text.strip()
def sweep(n_docs: int = 400, positions=(0, 0.25, 0.5, 0.75, 0.99), trials: int = 5):
for frac in positions:
pos = min(int(frac * n_docs), n_docs - 1)
hits = 0
for _ in range(trials):
ctx = build_context(n_docs, pos)
if "QUARTZ-4417" in ask(ctx):
hits += 1
print(f"needle at {frac:>4.0%} of context -> accuracy {hits/trials:.2f}")
if __name__ == "__main__":
sweep()What to notice, and why each line is load-bearing:
needle_pos and expect to see the U-curve: high at 0% and 99%, lowest near 50%. With trials > 1 you get a noisy but real accuracy estimate per bin — the same design MRCR v2 formalizes with token-based bins up to 1M.Run it once at n_docs=400 (cheap) and once near the window limit; the gap between the two is your context-rot budget.
| Strategy | Cost | Latency | Quality effect | Primary failure mode |
|---|---|---|---|---|
| Stuff everything in 1M window | High (you pay for every input token) | High (prefill scales with tokens) | Strong global reasoning; U-curve hurts mid-context facts | Middle facts silently dropped; context rot at depth |
| RAG (retrieve top-k) | Low (small prompt) | Low | High if retrieval recall is good | A missed retrieval = the fact is simply absent |
| Rerank then place at edges | + reranker call | + tens of ms | Recovers much of the U-curve penalty | Reranker is itself a model with its own errors |
| Compression / summarization | + a summarize call | + one model pass | Fits more turns; lossy | Drops a fact the user later needs; precision < full context |
| Prompt-cache the stable prefix | 0.1× on hits, 1.25–2× on writes | Big prefill savings on hits | Neutral on quality | Cache miss if any earlier block changes (prefix invariant) |
Prose on what changes at scale:
Flashcard. Lost in the middle = U-shaped accuracy vs. fact position (high at start/end, ~20–30% lower in the middle); it's an attention-budget + recency effect, so the cure is retrieve → rerank → place at the edges, not a bigger window.
Next: /context-engineering/prompt-caching — how to make a big stable context nearly free, so retrieval and placement are the only costs left to optimize.