From Prompt Engineering to Context Engineering

1. Quick anchor

A transformer is a function from a token sequence to a probability distribution over the next token. It has no memory, no state, no awareness of your database — it sees exactly the tokens you assemble into its context window and nothing else. Prompt engineering asked: how do I word the instruction so the model does the thing? Context engineering asks the harder, more general question: of everything the model could see — instructions, tools, retrieved documents, conversation history, scratchpad notes, prior reasoning — what subset do I place in the window, in what order, at what point in the task, and which agent gets to see it? The shift happened because frontier models in 2026 are good enough at following clear instructions that cleverness in wording yields diminishing returns; the dominant failure mode is no longer "the model misunderstood me" but "the model didn't have the right information, or had too much of the wrong information, at the moment it had to act." Context is now the scarce, expensive, latency-bearing resource you budget — and engineering it is a curation problem, not a wording problem.

2. Why interviewers probe this

• IC4 — Can you articulate that the model is stateless and that everything it knows on a given call lives in the window you built? Do you know the four levers (what / when / how much / visibility) by name and reach for them instead of just rewording the prompt?
• IC5 — Can you quantify the tradeoffs? When you add a 50K-token document, do you instinctively think about cost (input tokens), latency (prefill time), quality (context rot), and cache invalidation — and can you put rough numbers on each? Can you choose between retrieval, compaction, and context-editing for a concrete scenario?
• IC6 — Can you design a context architecture across multiple agents and turns, reason about per-agent visibility boundaries, place cache breakpoints to survive a long-horizon run, and defend the failure modes (cache thrash, lost-in-the-middle, compaction information loss) under follow-up pressure? Do you treat context as a system-design discipline with explicit budgets, not a bag of tricks?

3. Concept build-up

3.1 The model is a stateless function — that is the whole foundation

Start from first principles. A decoder-only transformer computes P(next_token | token_1, ..., token_n). It carries no hidden state between API calls; the API is stateless. So when an agent "remembers" your name across turns, that is not memory inside the model — it is your harness re-sending the prior turns in the context window every single call. This single fact is why context engineering exists: the model's entire epistemic state on call k is the token sequence you assembled for call k. Nothing else.

Prompt engineering treated that sequence as mostly fixed — a system prompt plus the user's question — and optimized the wording. That was the right bottleneck when models were unreliable instruction-followers (GPT-3.5 era): a better-phrased prompt was the difference between working and not. But as models crossed a capability threshold, wording stopped being the binding constraint. The binding constraint became composition: which of the millions of possible tokens you assemble, given a finite window, finite attention, real dollars per token, and real milliseconds of prefill latency.

3.2 The four dimensions

Context engineering decomposes into four orthogonal decisions. Memorize these — they are the spine of the discipline and the spine of the interview.

1. What — knowledge, tools, and history. Which instructions, which retrieved documents, which tool definitions, which slice of conversation history. The core insight from Anthropic's engineering guidance: find the smallest set of high-signal tokens that maximizes the likelihood of the desired behavior. More is not better. A bloated tool set with overlapping functions makes tool selection worse, not better; ten carefully chosen documents beat a hundred mediocre ones. "What" is a curation and ranking problem, which is exactly why retrieval and reranking (see /rag) are part of context engineering.

2. When — timing. Does a piece of knowledge belong in the window from token zero, or should it be fetched just-in-time when the task demands it? Pre-loading everything is the naive default and it is usually wrong: it fills the window with tokens that are irrelevant to 90% of requests. The alternative is just-in-time retrieval — keep lightweight identifiers (file paths, IDs, glob patterns) in context, and load the actual content via a tool call only when needed. This trades a round-trip of latency for a dramatically smaller, higher-signal window. Claude Code works this way: it doesn't pre-read your repo, it greps and reads on demand.

3. How much — the budget. Every token has three costs: dollars (input-token price), latency (prefill scales with prompt length), and quality (context rot — section 3.4). The window is a budget you spend, not a container you fill. Context-aware models now expose this explicitly: Claude Sonnet 4.6 and Haiku 4.5 track a running token budget (Token usage: X/1M remaining) updated after each tool call, so the model itself can pace a long task and avoid premature shutdown. At the harness level, "how much" is enforced by compaction, context editing, and summarization — mechanisms that actively remove tokens to stay within budget.

4. Visibility — per-agent scoping. In a single-agent system there is one window and everyone sees everything. In a multi-agent system you choose what each agent sees. A sub-agent doing a focused search task should get a minimal window — its instructions and the search tools — and return a condensed 1–2K-token summary to the orchestrator, not its entire noisy 50K-token exploration transcript. Visibility is the most advanced lever and the one IC6 candidates are expected to design with: it lets you parallelize, isolate failure, and keep each window small, at the cost of coordination complexity and the risk that an agent lacks context it needed.

These four are not independent knobs you can max out individually — they trade against each other. Loading more (what) raises cost (how much). Just-in-time loading (when) adds latency but shrinks the budget. Isolating agents (visibility) shrinks each window but risks dropping needed context. The discipline is navigating these tradeoffs deliberately.

3.3 Why caching makes order a first-class concern

Context engineering is not only about what tokens — it is about their order, because of prefix caching. Providers cache the KV tensors for the front of your prompt. On Anthropic's API a cache hit costs 0.1× the base input price (a 90% reduction) and a cache write costs 1.25× (5-minute TTL) or 2.0× (1-hour TTL). The render order is fixed: tools → system → messages. The invariant is brutal and absolute: caching is a prefix match — any byte change anywhere in the prefix invalidates everything after it.

This converts a design principle into an economic one. Put stable content first (frozen system prompt, deterministic tool list, sorted JSON) and volatile content last (the user's varying question, timestamps, per-request IDs). A datetime.now() interpolated into your system prompt header silently destroys caching for the entire prompt — you pay full price on every call and never see an error, just cache_read_input_tokens: 0. For agents this also dictates architecture: don't change tools or swap models mid-conversation (both invalidate everything); if you need a cheaper model for a sub-task, spawn a sub-agent rather than switching the main loop's model.

3.4 Capability made cleverness yield to curation — and context rot is why curation matters

Two empirical facts explain why the paradigm flipped.

First, models got good at instructions. When the model reliably does what a clear instruction says, the marginal return on rewording collapses. You no longer need the "you are an expert, take a deep breath, I'll tip you $200" incantations of 2023. A plain, well-structured instruction works. So the lever with leverage moved elsewhere — to what surrounds the instruction.

Second, bigger windows did not make curation optional — they made it more important. This is the counterintuitive part that interviewers love. With 1M-token windows you might think "just stuff everything in." But attention is an n² operation over pairwise token relationships, and the model has a finite attention budget. Two robust findings:

• Lost in the middle (Liu et al., 2023): on multi-document QA with 20 documents, accuracy is highest when the answer sits at position 1 or position 20 and drops ~20–30% when the answer is in the middle (positions ~10–15). The curve is U-shaped. It is not a positional-embedding artifact — it persists on 100K+ token models and reflects attention-budget limits and recency bias.
• Context rot: accuracy degrades non-linearly as the token count rises, well before the hard limit. There is no clean cliff, just a sagging gradient. A model that scores 95% at 10K tokens may score meaningfully lower at 800K, on the same task.

The mitigations are pure context engineering: retrieve before you stuff (use RAG + reranking to bring only the top documents in), place critical information at the boundaries (start or end, never buried in the middle), and reorder retrieved documents so the highest-relevance ones land at the highest-attention positions. The benchmark that exposes models that only have a big window but no engineering discipline is MRCR v2 (multi-round co-reference resolution) — it embeds duplicate prompts among distractors and asks the model to retrieve a specific instance; at 1M tokens with two needles, even strong models score well below their single-needle numbers. Window size alone does not buy you long-context performance; placement, compression, and retrieval strategy do.

3.5 The mechanisms, mapped to the four dimensions

The discipline is operationalized by named mechanisms. Mapping them to the levers makes the framework concrete:

• What → retrieval + reranking (/rag), minimal non-overlapping tool sets, skills (load full instructions only when relevant).
• When → just-in-time retrieval (glob/grep/read on demand), tool search (load tool schemas only when relevant), structured note-taking (persist state to a file outside the window, retrieve via tool).
• How much → server-side compaction (summarize history near the limit; beta, default trigger ~150K input tokens), context editing (clear old tool results past a threshold, keep the N most recent; clear thinking blocks), token-budget awareness.
• Visibility → sub-agent architecture (isolated windows returning condensed summaries), per-agent tool scoping.

And a fifth, cross-cutting lever that the four dimensions all interact with: reasoning-time compute. Extended/adaptive thinking lets the model spend tokens reasoning before answering — quality improves with budget but flattens above roughly 10–32K thinking tokens, with linear cost. This is "what" (the reasoning is context the model produces for itself) and "how much" (you budget it via effort levels) at once. Crucially, thinking blocks interact with caching: preserved thinking enables cache hits across turns; clearing thinking invalidates the cache. That coupling is exactly the kind of cross-dimension interaction a senior engineer is expected to reason about.

4. Minimal implementation

The cleanest way to feel context engineering is to build the same prefix two ways — one that caches and one that silently doesn't — and watch the usage numbers. This is runnable against the Anthropic API and is the single most common "why is my bill high" bug in production.

import anthropic
 
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from env
 
# A large, STABLE prefix: instructions + a loaded document. This is the
# "what" you've decided the model needs, and it must stay byte-identical
# across calls for the cache to hit.
STABLE_SYSTEM = (
    "You are a support assistant for Acme Corp.\n\n"
    "=== PRODUCT MANUAL (excerpt) ===\n"
    + ("Section policy text ... " * 4000)  # ~stand-in for a real ~20K-token doc
)
 
def ask(question: str) -> anthropic.types.Message:
    return client.messages.create(
        model="claude-opus-4-8",
        max_tokens=1024,
        # cache_control on the system block: render order is tools -> system
        # -> messages, so a breakpoint here caches the whole stable front.
        system=[{
            "type": "text",
            "text": STABLE_SYSTEM,
            "cache_control": {"type": "ephemeral"},  # 5-min TTL, 1.25x write / 0.1x read
        }],
        # The volatile part — the varying question — goes LAST, after the
        # cached prefix, so it never invalidates the cache.
        messages=[{"role": "user", "content": question}],
    )
 
questions = [
    "What is the refund window?",
    "How do I reset my password?",
    "Is there an enterprise SLA?",
]
 
for i, q in enumerate(questions):
    r = ask(q)
    u = r.usage
    print(
        f"[{i}] write={u.cache_creation_input_tokens:>6} "
        f"read={u.cache_read_input_tokens:>6} "
        f"uncached={u.input_tokens:>4}"
    )

Run it and you'll see request [0] pay the write (cache_creation_input_tokens is large, cache_read is 0), then requests [1] and [2] flip to large cache_read and near-zero cache_creation — the 90%-off path. Now introduce the classic bug: change STABLE_SYSTEM to interpolate f"Current time: {datetime.now()}\n" at the top. The byte at the front changes every call, the prefix match fails, and cache_read_input_tokens stays 0 forever — you've silently turned off caching and 5×'d the cost of this prefix, with no error to alert you. That is context engineering in miniature: a single placement decision (stable-first vs. volatile-first) is worth ~79% of your input bill. To verify a real prompt, assert usage.cache_read_input_tokens > 0 across repeated calls in a test; if it's zero, diff the rendered prompt bytes between two requests to find the invalidator.

5. Production tradeoffs

The prose that matters: compaction is lossy and that is unavoidable — summarization cannot match full-context precision for fact-dense content (legal clauses, code, numeric tables). Prefer extraction with preservation rules (keep code blocks, technical decisions, and current state verbatim; summarize only narrative) over blanket summarization, and prefer context-editing (delete stale tool results, keep the structure) where you can, because deletion is honest about what's gone while a summary can confidently hallucinate completeness. What changes at scale: at low volume, just pick a big model and stuff the window — engineering effort isn't worth it. At high volume or long horizons, the window becomes the dominant cost and quality driver: caching swings the bill by 5×, context rot swings accuracy by tens of points, and a 200K-token agent loop that never compacts will eventually hit the context limit mid-task and fail. The discipline is not optional past a certain scale; it is the system.

6. How it's asked

7. Pitfalls & flashcards

• Confusing a big window with no need to curate. 1M tokens makes curation more important, not less — context rot and lost-in-the-middle bite hardest when the window is full.
• Silent cache invalidation. A timestamp, UUID, or unsorted json.dumps() in the prefix kills caching with no error. Always verify cache_read_input_tokens > 0 across repeated calls.
• Treating compaction as lossless. Summaries drop fact-dense detail and can hallucinate completeness. Use preservation rules; prefer deletion (context editing) when you can.
• Pre-loading everything. The naive default fills the window with tokens irrelevant to most requests. Default to just-in-time retrieval; pre-load only what's needed on every call.
• Bloated tool sets. Overlapping tools make selection worse. Curate a minimal, non-overlapping set.
• Dumping sub-agent transcripts into the coordinator. Isolation only pays off if sub-agents return condensed summaries, not their raw exploration.

Flashcard. Context engineering = choosing what (knowledge/tools/history), when (timing), how much (budget), and who sees it (per-agent visibility) — because the model is a stateless function over the window you build, and a capable model rewards curation over clever wording.

8. Further reading

• Effective context engineering for AI agents — Anthropic — the canonical framing of the four levers, just-in-time retrieval, structured note-taking, and sub-agent architecture.
• Prompt caching — Claude docs — the prefix invariant, cache economics, and breakpoint placement.
• Lost in the Middle (Liu et al., 2023) — the U-shaped accuracy curve that justifies boundary placement and reranking.
• Context windows — Claude docs — window sizes and token-budget awareness.
• Extended thinking — Claude docs — reasoning-time compute, budgets, and how thinking interacts with caching.
• Compaction — Claude docs — server-side summarization triggers and the cost of staying within budget.

Next: Retrieval-augmented generation — the "what" dimension in depth: sparse vs. dense retrieval, reranking, and document ordering against lost-in-the-middle.