Memory & Compaction: Managing Context That Outgrows the Window

1. Quick anchor

A context window is RAM, not a hard drive. It is finite (1M tokens on Opus 4.8, Sonnet 4.6, Haiku 4.5; 200K on older Sonnet), it is volatile (gone the instant the request ends), and — critically — every token in it is paid for on every subsequent turn, because the model re-attends over the whole prefix each step. A long-running agent therefore has three jobs it must do continuously: keep the live working set small enough to fit and stay accurate, push durable facts to an external store (files, a database) it can re-retrieve on demand, and evict the stale junk (old tool outputs, dead reasoning) before it crowds out signal. Compaction (lossy summarization of the whole conversation) and context editing (surgical deletion of specific blocks) are the two server-side levers for that eviction. Get the policy wrong and you either truncate mid-task or pay 1M-token prices to carry a transcript that is 90% noise.

2. Why interviewers probe this

• IC5 — Can you reason about the running cost of an agent, not just a single call? Do you know that history is re-billed every turn (so a 40-step agent is quadratic in token spend), that compaction costs two model passes, and that the wrong edit order silently destroys your prompt cache? Can you implement a working set policy without hand-waving?
• IC6 — Can you architect memory as a system: the boundary between window and store, the retrieval path back in, the consistency and staleness model across sessions, and the failure modes (lost-in-the-middle, lossy compaction dropping the one fact that mattered, cache thrash). Can you defend build-vs-buy (server-side compaction vs. your own summarizer) and quantify the tradeoff?

3. Concept build-up

3.1 Short-term vs long-term: the window is not your memory

The single most common architecture mistake is treating the context window as the agent's memory. It is not. It is the attention surface for one request. Anything you want the agent to "remember" beyond this request must live somewhere durable: a file, a key-value store, a vector index, a scratchpad. The window is then reconstructed each turn from (a) a stable preamble, (b) the recent live working set, and (c) whatever you chose to retrieve back in.

This split matters because the two have opposite cost curves. The store is cheap to hold (bytes on disk) but costs a retrieval hop to use. The window is instant to use but is re-billed on every single turn. Anthropic's engineering guidance is explicit here: maintain lightweight identifiers (file paths, IDs, glob/grep handles) in-window and load the heavy content just-in-time, rather than pre-stuffing everything. A coding agent that keeps src/auth/session.py as a 6-token path and reads it only when needed is dramatically cheaper than one that pastes the whole file into history and carries it for 30 turns.

3.2 The real cost of carrying history

Here is the load-bearing fact most candidates miss. A transformer re-attends over the entire prefix at each generation step, and the API bills you for the full input on each turn. So if turn k carries a history of roughly k × m tokens (where m is the average tokens added per turn), the cumulative input billed across an n-turn agent is the sum m + 2m + 3m + ... + nm = m·n(n+1)/2 — quadratic in the number of turns. Doubling an agent's length roughly quadruples its history-carrying spend. This is why eviction is not a nicety; it is the difference between a $0.40 task and a $4 task.

So a senior answer to "what does it cost to run this agent" is never the single-call price. It's the integral of the window size over the run, minus what caching and eviction claw back.

3.3 Compaction: summarize the whole thing

When a conversation genuinely needs its narrative preserved — the agent has been reasoning across many steps and the thread of decisions matters — you compact: replace the transcript with a model-written summary and continue. Anthropic's server-side compaction (beta, Jan 2026; supported on Opus 4.8/4.7/4.6, Sonnet 4.6, Fable 5, Mythos 5) automates this. It triggers at a configurable input-token threshold (default 150K, minimum 50K), runs a compaction sampling pass to produce a summary, then the API auto-drops the pre-compaction message blocks; the client appends the compaction block and continues.

The cost is the thing to internalize: compaction is two model evaluations, not one. From the docs' worked example, a single compacted step costs roughly (180K input + 3.5K summary output) for the compaction pass plus (23K input + 1K output) for the continued message — about 207.5K tokens total for what feels like one turn. You pay full freight to read the long history one last time in order to shrink it. Compaction is therefore something you amortize across many subsequent cheap turns, not something you do every turn.

And it is lossy. Summarization of fact-dense content (exact error strings, line numbers, a specific config value) cannot match full context — precision is the first casualty. The mitigation is preservation rules: instruct the summarizer to verbatim-preserve code, technical decisions, file paths, and live state, and to drop only redundant narrative. You can also pause after compaction for custom handling (e.g. snapshot the dropped blocks to your store before they vanish).

3.4 Context editing: prune, don't summarize

Often you don't need a narrative summary — you need to throw out the 80K tokens of stale tool output while keeping the reasoning intact. That's context editing: surgical, structural deletion.

Tool-result clearing (beta, Sept 2025) removes old tool outputs once input crosses a threshold (default 100K). You set a keep policy (retain the N most recent tool uses, default 3), an exclude list (tools whose results are never cleared — e.g. web_search, whose findings you can't re-derive), and whether to also clear the tool call parameters. Cleared content is replaced with a placeholder so the model knows something was removed rather than silently hallucinating around a gap.

Thinking-block clearing (beta, Oct 2025) manages extended-thinking blocks: keep "all" or {type: "thinking_turns", value: N}. Defaults are model-specific (Opus 4.5+ keep all by default; earlier keep only the last turn).

The two compose, and order matters: put thinking clearing first in the edits array, then tool-result clearing. This is the surgical alternative to compaction — finer-grained, no extra model pass, and it preserves the exact wording of what you keep instead of paraphrasing it.

3.5 Lost in the middle: why placement is a memory decision

Even when everything fits, where a fact sits in the window changes whether the model uses it. Liu et al. (2023) showed a U-shaped accuracy curve on multi-document QA: models retrieve best when the answer is at position 1 or the last position, and worst — a ~20-30% accuracy drop — when it's in the middle (positions ~10-15). This persists on 100K+ windows and on key-value retrieval, and the mechanism is attention-budget and recency bias, not positional embeddings. The practical consequence for memory design: after you retrieve or compact, reorder so the most relevant material sits at the edges. A compaction summary should generally go near the boundary, and retrieved documents should be position-mapped highest-relevance → start/end. This is why memory is not just "what to keep" but "where to put it."

3.6 Cross-session memory: the store is the agent

For an agent that must persist across weeks, none of the in-window machinery survives — the window is rebuilt every session. The durable memory is the external store. The pattern (per Anthropic's effective-context guidance) is structured note-taking: the agent writes salient facts (user preferences, prior decisions, project state) to files/records via a tool, and on a new session retrieves only the relevant slice — not the whole history. Sub-agent architectures extend this: a focused sub-agent does heavy work in its own isolated window and returns a condensed 1-2K-token summary to the parent, keeping the parent's window clean. The store is queryable, versionable, and inspectable; the window is none of those. Treat persistence as a database problem, and retrieval back into the window as a RAG problem — see /rag for reranking and the retrieval-before-stuffing discipline.

4. Minimal implementation

A working set manager that caps in-window history, spills overflow to a durable store, and configures server-side eviction. This is the shape of the loop in a real agent harness.

import json, os, time
from anthropic import Anthropic
 
client = Anthropic()
STORE = "memory_store.jsonl"      # durable long-term memory (the "hard drive")
WORKING_SET_TOKEN_CAP = 40_000     # keep the live window dense
 
def remember(fact: dict) -> None:
    """Persist a durable fact outside the window. Cheap to hold, retrievable later."""
    with open(STORE, "a") as f:
        f.write(json.dumps({"ts": time.time(), **fact}) + "\n")
 
def recall(query: str, k: int = 3) -> list[dict]:
    """Just-in-time retrieval: load only what's relevant back into the window.
    A real system uses a vector index + reranker; substring match keeps this runnable."""
    if not os.path.exists(STORE):
        return []
    hits = [json.loads(l) for l in open(STORE) if query.lower() in l.lower()]
    return hits[-k:]  # most-recent k; place at window edges to dodge lost-in-the-middle
 
def run_turn(messages: list[dict], tools: list[dict], user_query: str):
    # 1. Pull durable facts back in, JUST-IN-TIME, not pre-stuffed.
    recalled = recall(user_query)
    preamble = ("Relevant memory:\n" + json.dumps(recalled)) if recalled else ""
 
    resp = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=4096,
        system=[
            # Stable preamble -> cache it. cache_control on the LAST stable block.
            {"type": "text", "text": "You are a long-running research agent.",
             "cache_control": {"type": "ephemeral"}},
        ],
        messages=messages + [{"role": "user", "content": user_query + "\n" + preamble}],
        tools=tools,
        # Server-side eviction: prune stale tool results, keep recent + protected ones.
        # Order: thinking-clearing FIRST, then tool-result clearing.
        extra_body={
            "context_management": {
                "edits": [
                    {"type": "clear_thinking", "keep": "all"},
                    {"type": "clear_tool_uses",
                     "trigger": {"type": "input_tokens", "value": 100_000},
                     "keep": {"type": "tool_uses", "value": 3},
                     "exclude_tools": ["web_search"],   # never clear unrecoverable results
                     "clear_at_least": {"type": "input_tokens", "value": 20_000}},
                ]
            }
        },
    )
    return resp
 
def enforce_working_set(messages, count_tokens) -> list[dict]:
    """If local history exceeds the cap, spill the oldest turns to the store
    and drop them from the window. Turns O(n^2) carry-cost into O(n)."""
    while count_tokens(messages) > WORKING_SET_TOKEN_CAP and len(messages) > 4:
        evicted = messages.pop(0)
        remember({"type": "evicted_turn", "content": str(evicted)[:2000]})
    return messages

Three load-bearing choices. (1) recall() is just-in-time — facts come back only when the query touches them, so the window stays small. (2) exclude_tools=["web_search"] protects results the model cannot re-derive; clearing them would force expensive re-fetches. (3) enforce_working_set() is the cheap local backstop that keeps history-carry linear; server-side editing handles the in-flight request, but your harness owns the durable spill. Note the cache_control sits on the last stable block of the preamble — put it on dynamic content and you get zero cache hits.

5. Production tradeoffs

In prose: cost is dominated by the quadratic carry term, so the highest-leverage move at scale is capping the working set and caching the stable prefix — that converts the dominant O(n²) into O(n) plus a 0.1× preamble. Latency has a sharp asymmetry: context editing is free-ish, but compaction inserts a visible stall because it must read the entire history one last time before shrinking it; budget for that spike or pre-empt it off the critical path. Quality is where compaction and editing diverge most — editing is lossless on what it keeps, while compaction is lossy on everything, so prefer editing (prune stale tool results) and reserve compaction for when the narrative genuinely must be preserved. The nastiest cross-cutting failure mode is the cache interaction: compaction rewrites the prefix, so it invalidates the prompt cache — the turn after a compaction pays full write price again. This is why preserved thinking blocks (which keep the cache warm) and aggressive editing (which doesn't rewrite the prefix wholesale) often beat blanket compaction economically. At scale, the store stops being a file and becomes a vector index + reranker with its own eviction and consistency model; "memory" becomes a retrieval system you must evaluate (see /evals), not a buffer you trust.

6. How it's asked

7. Pitfalls & flashcards

• Treating the window as memory. It vanishes when the request ends. Durable facts must go to a store; the window is rebuilt each turn.
• Pricing the single call, not the integral. History is re-billed every turn — cost is quadratic in turns unless you cap the working set.
• Compacting every turn. It's two model passes; amortize it across many cheap follow-ups, don't trigger it constantly.
• Compaction invalidating the cache. Rewriting the prefix kills your 0.1× cache hits; prefer surgical editing where possible.
• Clearing unrecoverable tool results. Always exclude things like web_search you can't cheaply re-derive.
• Ignoring placement. A retrieved fact stranded in the middle of a 200K window can be ~20-30% less likely to be used — reorder to the edges.
• Pre-stuffing instead of just-in-time. Carry lightweight IDs/paths in-window; load heavy content only when the task touches it.
• Cache breakpoint on dynamic content. Put cache_control on the last stable block, or you get zero hits.

Flashcard. Compaction = lossy summary of everything, costs two model passes, invalidates cache; context editing = lossless prune of specific blocks, near-free, cache-friendly. Reach for editing first.

8. Further reading

• Anthropic — Compaction — triggers, the two-pass cost model, and the pause-for-custom-handling hook.
• Anthropic — Context editing — tool-result and thinking-block clearing, keep policies, exclude lists, and edit ordering.
• Anthropic — Context windows — 1M-token specs, context-awareness budgeting, and context rot.
• Effective context engineering for AI agents — note-taking, just-in-time retrieval, sub-agent architecture.
• Liu et al., 2023 — Lost in the Middle — the U-shaped curve and the attention-budget mechanism.
• Anthropic — Prompt caching — the 0.1× hit / 1.25-2× write economics and the prefix-invariant rule.

Next: /context-engineering/lost-in-the-middle — how placement and reranking turn a full window into one the model can actually use.