Prompt Caching: The Prefix-Match Invariant

1. Quick anchor

Prompt caching reuses the computed key-value (KV) attention tensors for a prompt prefix across requests, so you pay full price to process that prefix once and roughly a tenth of the price every time after. The entire mechanism rests on one invariant: the cache key is a hash of the exact bytes of the rendered prompt up to each breakpoint, so any byte change anywhere in the prefix invalidates everything downstream. The render order is fixed — tools, then system, then messages — and you place a cache_control breakpoint on the last stable block, never on volatile content. Done right, repeated large-context calls drop ~90% in cost and reuse the prefix's prefill for a large latency win; done wrong, a single datetime.now() in your system prompt means you pay the write premium on every request and never read a thing. This is the highest-leverage, lowest-effort cost optimization in the entire context-engineering stack, and it is also the one most often silently broken in production.

2. Why interviewers probe this

• IC4 — Can you explain the prefix-match model from first principles? Do you know that caching keys on bytes, not "meaning," and that breakpoints go on stable content? Can you read usage.cache_read_input_tokens to confirm a hit?
• IC5 — Can you debug a cache that isn't hitting? This separates people who've shipped from people who've read docs. Interviewers want the silent-invalidator audit (timestamps, unsorted JSON, per-user IDs) and the instinct to diff rendered prompt bytes between two requests.
• IC6 — Can you architect a prompt-assembly layer where caching is a property of the system, not an afterthought? The hard version is multi-tenant agents: tool-set changes, model switches, mid-session operator instructions, sub-agent forks, and the 20-block lookback all conspire against you. Interviewers probe whether you understand the cache-tier hierarchy and can reason about cost/latency tradeoffs at scale.

3. Concept build-up

3.1 Why a prefix, and only a prefix?

A transformer computes attention causally: each token attends only to tokens before it. So the KV tensors for token N depend on tokens 0…N and nothing after. That is the deep reason caching is prefix-only — the moment one token changes at position N, every KV tensor from N onward is computed against different upstream context and is no longer reusable. There is no "cache the middle" because the middle's representations are entangled with everything before it.

This is also why the render order is a hard, fixed contract: tools → system → messages. Tools render at position 0. If you add, remove, or reorder a single tool, the byte at position 0 shifts and the entire cache — tools, system, and every cached message turn — is invalidated. The hierarchy is the thing you design around: put the most stable content (frozen system instructions, a deterministically-serialized tool list) first, and let volatility increase monotonically toward the end of messages.

3.2 The economics, derived

Let the base input price be 1×. The published cost structure is: a cache read costs 0.1× (a 90% reduction), and a cache write costs 1.25× for the default 5-minute TTL or 2.0× for the 1-hour TTL. From these three numbers you can derive the break-even point — the question every cost review asks.

With the 5-minute TTL: the first request writes at 1.25×; each subsequent identical-prefix request reads at 0.1×. Two requests total 1.25 + 0.1 = 1.35× versus 2.0× uncached — caching wins on the second request. With the 1-hour TTL: 2.0 + 0.1 = 2.1× for two requests versus 2.0× uncached, so two requests lose; you need a third (2.0 + 0.2 = 2.2× vs 3.0×) to come out ahead. The 1-hour TTL buys you survival across idle gaps in bursty traffic at the cost of needing more reads to amortize the doubled write.

The latency story is parallel but not identical: a cache read skips the prefill of the cached prefix (the compute-heavy first pass), so time-to-first-token drops sharply on cached calls. The notes peg this at roughly an 85% latency cut on a large cached prompt. Output generation is unaffected — caching speeds up the read of context, not the writing of the answer.

3.3 Placement: the last stable block

The breakpoint goes on the last block that is byte-identical across the requests you want to share a cache. Three canonical placements:

• Large shared system prompt. Put the breakpoint on the last system text block. Because tools render before system, that one marker caches tools + system together.
• Multi-turn conversation. Put the breakpoint on the last content block of the most-recently-appended turn. Each new request reuses the entire prior conversation as its prefix, and hits accrue incrementally as the conversation grows.
• Shared preamble, varying question. Put the breakpoint at the end of the shared preamble — not at the end of the whole prompt. If you mark the end of the whole prompt, the varying question is inside the cached span, every request writes a distinct entry, and nothing is ever read.

You get a maximum of 4 breakpoints per request, and each breakpoint walks backward at most 20 content blocks to find a prior entry — a constraint that bites agentic loops, covered in §5. Minimum cacheable prefix is model-dependent: Opus 4.8 requires 4,096 tokens; Sonnet 4.6 and Fable 5 cache at 2,048; older Sonnets at 1,024. A 3K-token prompt caches on Sonnet 4.5 but silently will not on Opus 4.8 — no error, just cache_creation_input_tokens: 0.

3.4 The silent-invalidator audit

This is the heart of the IC5 question. A cache that "doesn't work" almost never throws an error — it just quietly writes and never reads. The culprit is always dynamic content that has leaked into the prefix. Grep your prompt-assembly path for these:

The fix is always one of three moves: move the dynamic piece after the last breakpoint, make it deterministic (sort the keys), or delete it if it isn't load-bearing. The classic example: "Current date: {today}" at the top of the system prompt. It feels harmless. It invalidates the entire prompt on every single request. Move it into a message turn — a message at turn 5 invalidates nothing before turn 5. (See /context-engineering for where date/context injection belongs in the broader pipeline.)

3.5 Mid-conversation operator instructions — the cache-preserving channel

A subtle production trap: an operator instruction arrives mid-session ("terse mode on", "the user just enabled auto-approve"). The naive fix is to edit the top-level system prompt. That changes the prefix ahead of the entire conversation history, so every cached turn is re-processed uncached — you just nuked hours of accumulated cache. The correct move (on supporting models, beta header mid-conversation-system-2026-04-07) is to append a {"role": "system", "content": "..."} message to messages[]. It sits after the cached history, leaves the prefix byte-identical, and carries operator authority — which also makes it the prompt-injection-safe alternative to embedding instructions as forgeable user text. This is the cache-aware way to be dynamic.

4. Minimal implementation

The smallest honest example: a large frozen system prompt cached across two requests, verified by reading usage. Note the breakpoint on the system block (stable), the question in messages (volatile), and the date deliberately placed in the message turn rather than the system prompt.

import os
from anthropic import Anthropic
 
client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment
 
# Frozen, deterministic prefix. No timestamps, no per-request IDs, no unsorted JSON.
SYSTEM_PROMPT = open("playbook.md").read()  # imagine ~20K tokens of stable instructions
 
def ask(question: str, today: str) -> dict:
    resp = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=1024,
        system=[
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                # Breakpoint on the LAST stable block: caches tools (none here) + system.
                "cache_control": {"type": "ephemeral"},  # 5-min TTL; use {"ttl": "1h"} for bursty traffic
            }
        ],
        messages=[
            {
                "role": "user",
                # Volatile content lives AFTER the breakpoint, in messages.
                # The date goes here, NOT in the system prompt — putting it in
                # system would invalidate the cache on every call.
                "content": f"Today is {today}.\n\n{question}",
            }
        ],
    )
    u = resp.usage
    return {
        "write": u.cache_creation_input_tokens,  # tokens written to cache (~1.25x)
        "read": u.cache_read_input_tokens,       # tokens served from cache (~0.1x)
        "uncached": u.input_tokens,              # the remainder, at full price
    }
 
# First call: cold. Expect write > 0, read == 0.
print(ask("Summarize section 3.", "2026-06-17"))
# Second call within 5 min: warm. Expect write == 0, read > 0 (the whole prefix).
print(ask("Now summarize section 4.", "2026-06-17"))

Three things make this correct. (1) SYSTEM_PROMPT is read once and never has dynamic content interpolated in — it's a frozen byte string. (2) The cache_control marker is on the system block, the stable boundary; the varying question and the date are in messages, after it. (3) The verification is explicit: if the second call shows read == 0, you have a silent invalidator and should diff the two rendered prompts byte-for-byte. Note also that input_tokens is the uncached remainder only — total prompt size is input_tokens + cache_creation_input_tokens + cache_read_input_tokens, so an agent that ran for hours showing input_tokens: 4000 is fine if the sum is large; the rest came from cache.

For a chat UI, prefer top-level cache_control={"type": "ephemeral"} on the request, which auto-places the marker on the last cacheable block — the simplest option when you don't need fine-grained placement.

5. Production tradeoffs

What changes at scale. Three constraints dominate agent workloads. First, the 20-block lookback: a single agentic turn that appends more than 20 content blocks (common with many tool_use/tool_result pairs) pushes the previous cache entry out of the lookback window, and the next breakpoint silently misses. Fix: place an intermediate breakpoint every ~15 blocks in long turns. Second, tool and model changes mid-session force a full rebuild — tools at position 0 invalidate everything, and caches are model-scoped so a model switch starts cold. The agent-design workarounds: use tool search (it appends schemas rather than swapping them, preserving the prefix), and for a cheaper sub-task spawn a sub-agent on the smaller model rather than switching the main loop's model. Third, fork operations (summarization, compaction, sub-agents) that rebuild system/tools/model with any difference miss the parent's cache entirely — copy the parent's prefix verbatim, then append fork-specific content at the end.

The cache-tier hierarchy is the nuance that separates IC5 from IC6: not every parameter change invalidates everything. Changes invalidate their own tier and below. Tool-definition or model changes blow away all three tiers (tools, system, messages). A system-content change keeps the tools cache. Toggling tool_choice, thinking, or images keeps both tools and system caches. So you can flip tool_choice per request without losing your expensive tools+system cache — don't over-worry about those; only tool-definition and model changes force a full rebuild. One more scale gotcha: concurrent fan-out. A cache entry only becomes readable after the first response begins streaming. Fire N identical-prefix requests in parallel and all N pay the write — none can read what the others are still writing. The pattern is to send one, await its first streamed token, then fire the remaining N−1.

Pair caching with the rest of the context-management toolkit: context editing (clearing stale tool results) interacts with caching because cleared thinking blocks invalidate the cache from that point, while preserved thinking enables hits — so when combining edits, clear thinking before clearing tool results. Caching also doesn't fix the "lost in the middle" accuracy degradation — it makes a bloated context cheaper, not better, which is its own failure mode if it lulls you into stuffing the window.

6. How it's asked

7. Pitfalls & flashcards

• Caching matches bytes, not meaning. "Logically identical" prompts with reordered JSON keys do not share a cache.
• The date in the system prompt is the #1 killer. It feels harmless; it invalidates everything on every call. Inject dynamic context as a message turn.
• A breakpoint on the wrong side of the varying content caches nothing. Mark the end of the shared preamble, not the end of the whole prompt.
• input_tokens is the uncached remainder, not the total. Sum all three usage fields before concluding the cache failed.
• Below the model minimum, caching silently no-ops. 4,096 tokens on Opus 4.8 — a 3K prompt writes nothing, no error.
• Concurrent identical requests all pay the write. Warm one, await its first token, then fan out.
• 1-hour TTL doubles the write cost. Only worth it when traffic has gaps longer than 5 minutes; otherwise it loses money.

Flashcard. Prompt caching is a prefix hash of exact bytes; freeze the prefix (tools → system → messages), breakpoint the last stable block, push everything dynamic past it, and verify with cache_read_input_tokens.

8. Further reading

• Anthropic — Prompt caching — the authoritative cost structure, breakpoint rules, and minimums.
• Anthropic — Context windows — context-awareness and token budgeting that pair with caching.
• Anthropic — Context editing — how clearing tool results and thinking blocks interacts with cache validity.
• Anthropic Engineering — Effective context engineering for AI agents — the architectural patterns (just-in-time retrieval, sub-agents) that caching must survive.
• Liu et al. — Lost in the Middle — why a cheaper big context is not automatically a better one.

Next: /context-engineering/long-context — the accuracy side of the context window, where caching makes a stuffed prompt cheap but not correct.