Inference Economics: Prefill, Decode, and the Memory Wall

1. Quick anchor

LLM inference is not one workload — it is two, glued together. Prefill reads your whole prompt at once and is compute-bound: it does dense matrix-matrix multiplies and pins the GPU's math units at 90-95% utilization. Decode then emits one token at a time, each step a thin matrix-vector multiply that must re-read the entire KV cache from HBM with essentially zero data reuse — so it is memory-bandwidth-bound and the GPU's math units sit 60-80% idle. The single most important sentence in this pillar: decode is starved for memory bandwidth, not compute, and almost every serving trick (batching, paging, quantizing the KV cache, speculative decoding, disaggregation) is a maneuver to feed that starved phase. Once you internalize that prefill fills latency-to-first-token and decode fills the gap between tokens, the cost and latency math falls out almost mechanically.

2. Why interviewers probe this

• IC4 — Can you name the two phases and correctly classify each as compute- vs memory-bound? Do you know why batching helps decode but barely touches prefill? This is the table-stakes signal that you've actually run a serving stack, not just called an API.
• IC5 — Can you reason about TTFT vs TPOT vs throughput as separate, often-conflicting objectives, and pick the right lever for a given SLA? Can you do a back-of-envelope arithmetic-intensity calculation to predict whether a workload will be compute- or memory-bound before you profile it?
• IC6 — Can you reason about fleet economics: tokens/sec/dollar, when disaggregated prefill/decode pays off, how quantization and hardware choice (H100 vs H200 vs B200) shift the cost curve, and where the next bottleneck appears at scale? Staff candidates are expected to argue tradeoffs with numbers and name the failure modes.

3. Concept build-up

3.1 The two phases, from first principles

Take a transformer with hidden size d and a weight matrix W of shape [d, d]. In prefill, you have L prompt tokens stacked into an activation matrix X of shape [L, d]. The projection X @ W is a matrix-matrix multiply: it costs roughly 2 · L · d · d FLOPs but reads W (size d · d) only once to serve all L tokens. So arithmetic intensity — FLOPs per byte read — scales with L. With a typical prompt this lands at 200-400 FLOP/byte on an H100, comfortably above the hardware's compute-to-bandwidth ratio (the "ridge point" of the roofline, ~295 FLOP/byte for H100 BF16). Prefill is therefore compute-bound: the GPU's tensor cores are the bottleneck and run at 90-95% utilization.

In decode, you generate one token at a time. Now X is shape [1, d] — a single vector. The same projection x @ W is a matrix-vector multiply: it still reads all of W from HBM, but does only 2 · d · d FLOPs to serve one token. Arithmetic intensity collapses by roughly the prefill batch factor — the notes put it at 60-80 FLOP/byte, ~5x lower than prefill — landing you well below the roofline ridge. Decode is therefore memory-bound: the tensor cores starve waiting for weights and KV cache to arrive over the memory bus, and GPU utilization falls to 20-40%.

This is not an implementation defect you can engineer away; it is arithmetic. A single autoregressive step cannot reuse a weight across multiple tokens, because the next token doesn't exist yet. The only reuse available is across concurrent requests — which is exactly why batching is the central lever for decode and nearly irrelevant for prefill.

3.2 The memory-bandwidth wall

Now add the KV cache, which makes decode worse. At step t, attention must read the K and V vectors of all t previous tokens. That's pure streaming with zero reuse: every byte of KV cache is read once per token and never again within the step. As the sequence grows, KV reads dominate the memory traffic and the decode step gets slower per token the longer the conversation runs.

The notes are blunt about how bad this is in practice: production attention kernels like XFormers achieve only ~23% compute-bandwidth utilization and ~47% memory-bandwidth utilization — the GPU "stalls 10-50x longer on memory reads than on arithmetic execution." This is the memory wall: HBM bandwidth, not FLOPs, sets your decode speed. Every modern serving optimization is best understood as an attack on this wall:

• Larger batches amortize each weight read across more tokens (raises intensity).
• KV-cache quantization (FP8/INT8) halves the bytes you must stream per token — see /inference/quantization.
• PagedAttention / RadixAttention pack the cache densely so you can fit bigger batches in the same HBM — see /inference/kv-cache.
• Speculative decoding verifies several draft tokens in one weight read, converting a memory-bound step into a slightly-more-compute-bound one.
• Faster/bigger HBM (H200's 4.8 TB/s, B200's 8 TB/s) literally widens the bus.

3.3 TTFT, TPOT, and throughput — three numbers that fight

There is no single "latency." Interactive serving has three first-class metrics, and they trade off:

• TTFT (Time To First Token) = queue delay + prefill time. Dominated by prompt length and how busy the prefill engine is. This is what a user feels as "responsiveness."
• TPOT (Time Per Output Token) = the steady-state decode step time. Dominated by memory bandwidth, KV-cache size, and batch size. This is what a user feels as "streaming speed."
• Throughput = total tokens/sec across all requests. This is what finance feels — it sets cost per token.

End-to-end latency for one request is approximately TTFT + output_tokens × TPOT. The cruel part: pushing throughput (bigger batches) usually raises TPOT for any individual user, because more requests share the same memory bandwidth each step. And aggressively co-scheduling prefill with decode (to keep the tensor cores fed) can spike TPOT for in-flight users whenever a big new prompt lands — the classic "my stream stuttered" symptom. Senior candidates name which metric a given change helps and which it hurts; that's the whole game.

3.4 Why batching is decode's best friend (and prefill's afterthought)

In decode, the dominant cost is reading the model weights (and KV cache) from HBM. If you process one request, you read all the weights to produce one token. If you process 32 requests in a batch, you read the same weights once and produce 32 tokens. The memory cost is roughly fixed; the useful output scales with batch size. That's the entire reason decode throughput can improve 4-24x (PagedAttention) or 23x+ (continuous batching vs static batching) — you're climbing the roofline toward the compute-bound regime by raising effective arithmetic intensity.

But two things cap the batch:

1. HBM for KV cache. Each concurrent sequence needs its own growing KV cache. Run out of HBM and you can't add requests — which is precisely why dense KV management (paging) and KV quantization translate directly into bigger batches and lower cost.
2. Diminishing returns + TPOT. Once you're compute-bound, more batch doesn't add throughput, it just adds per-token latency.

Prefill, by contrast, is already compute-bound at batch size 1 for a normal prompt — there are no idle math units to fill — so batching prefill yields little and mostly just delays TTFT. The asymmetry between the phases is the deepest structural fact in this pillar, and it directly motivates continuous batching (mix new prefills with ongoing decodes at token granularity) and chunked prefill (slice a long prompt so it doesn't monopolize an iteration). See /inference/continuous-batching.

4. Minimal implementation

You don't need a cluster to feel the two regimes. The snippet below estimates, from first principles, the decode step time of a dense model purely from memory traffic, then compares it to the actual measured TPOT to expose how much of the wall you're hitting. This is the back-of-envelope every serving engineer should be able to do at a whiteboard.

def decode_step_time_ms(
    n_params: float,        # total model parameters
    bytes_per_param: float, # 2 = BF16/FP16, 1 = FP8/INT8
    kv_bytes_per_token: float,  # KV cache bytes per token in context
    context_len: int,       # tokens currently in the KV cache
    batch_size: int,        # concurrent sequences
    hbm_bw_TBps: float,     # GPU memory bandwidth, e.g. 3.35 for H100
) -> float:
    """Lower bound on a single decode step, set by memory reads.
 
    Decode is memory-bound, so step time >= (bytes read) / bandwidth.
    Weights are read ONCE per step regardless of batch (the win of batching).
    KV cache is read PER sequence in the batch (it does not amortize).
    """
    weight_bytes = n_params * bytes_per_param                 # amortized over the batch
    kv_bytes = kv_bytes_per_token * context_len * batch_size  # scales with batch AND length
    total_bytes = weight_bytes + kv_bytes
    bw_bytes_per_ms = hbm_bw_TBps * 1e12 / 1e3                # TB/s -> bytes/ms
    return total_bytes / bw_bytes_per_ms
 
 
def tokens_per_sec(step_ms: float, batch_size: int) -> float:
    """Whole-server decode throughput: one token per sequence per step."""
    return batch_size * 1000.0 / step_ms
 
 
if __name__ == "__main__":
    # Llama-3-70B-class model, FP8 weights, H100.
    # KV bytes/token (GQA, 8 KV heads, 80 layers, head_dim 128, FP8): ~163 KB.
    cfg = dict(
        n_params=70e9, bytes_per_param=1.0,
        kv_bytes_per_token=163_840, context_len=2048,
        hbm_bw_TBps=3.35,
    )
    for bs in (1, 8, 32, 64):
        ms = decode_step_time_ms(batch_size=bs, **cfg)
        tps = tokens_per_sec(ms, bs)
        print(f"batch={bs:3d}  step={ms:6.2f} ms  "
              f"TPOT~{ms:5.2f} ms/tok  server={tps:7.0f} tok/s")

What this teaches, line by line. Weight bytes are read once per step (n_params × bytes_per_param), so they're divided across the whole batch — that's the amortization that makes batching work. KV bytes are read per sequence (× batch_size), so they don't amortize and eventually dominate at long context or large batch, which is why KV quantization and dense paging matter. Dividing total bytes by bandwidth gives a lower bound on step time — real kernels hit only ~47% memory-BW utilization (notes), so double these numbers for a realistic TPOT. Run it: at batch 1 you're paying ~21 ms to read 70 GB of weights to make one token (the memory wall, naked); at batch 32 that weight read is amortized 32-fold and server throughput leaps, until KV traffic and the compute roofline cap you. Switching bytes_per_param to 2.0 (BF16) or doubling kv_bytes_per_token shows quantization's effect directly. This is the model you should be able to reproduce in an interview from memory.

5. Production tradeoffs

The prose that ties it together. At small scale, you optimize for latency: a single H100, modest batch, maybe speculative decoding (EAGLE-3 lands ~0.80-0.88 acceptance on coding/instruction tasks, cutting TPOT with no quality loss because accepted tokens follow the exact target distribution). At medium scale, throughput dominates cost, so you turn on continuous batching + chunked prefill + FP8 KV cache and push batch size to the HBM limit; this is where most teams live. At large scale, the asymmetry between phases becomes a hardware decision: prefill wants FLOPs (H100/B200, wide tensor parallelism), decode wants bandwidth and HBM (H200's 4.8 TB/s and 141 GB, or B200's 8 TB/s). Disaggregated serving runs each phase on the hardware it likes — the notes cite ~3-4x TCO benefit on prefill-heavy and 2.5-4x on decode-heavy workloads — at the cost of a sub-50 ms KV-cache transfer over the network and real operational complexity. A subtle trap to call out: weight-only INT4 often does not speed up a busy server. In a compute-bound, well-batched regime the bottleneck is FP16 tensor-core math, not weight bytes, so shrinking the weights mostly saves memory (enabling bigger batches) rather than cutting step time — and W4A16 can even cost ~1.7x more energy per FP-INT op than W8A8. Quantization choice must match your regime, not your intuition.

6. How it's asked

7. Pitfalls & flashcards

• Treating "latency" as one number. TTFT and TPOT have different physics and different fixes; conflating them leads to optimizing the wrong phase. Always ask "first token or steady-state?"
• Assuming INT4 weights speed up serving. In a batched, compute-bound server it often doesn't — it mainly saves memory (which indirectly helps via bigger batches). Speed gains need activation quantization (W8A8/FP8) or a memory-bound regime.
• Ignoring KV-cache reads in the decode budget. Weight reads amortize over the batch; KV reads do not. At long context or large batch, KV traffic — not weights — becomes the wall. This is why FP8 KV cache and dense paging matter.
• Maxing batch size blindly. Beyond the compute roofline you add TPOT (per-user latency) with no throughput gain, and you risk HBM exhaustion → preemption/thrash.
• Forgetting prefill–decode interference. Without chunked prefill, a single long prompt can stall every in-flight user's stream for an iteration.
• Quoting throughput without batch and context. "10k tok/s" is meaningless without the batch size, context length, and precision that produced it.

Flashcard. Prefill = compute-bound (matrix-matrix, high arithmetic intensity, fills TTFT); decode = memory-bandwidth-bound (matrix-vector + KV reads, low intensity, fills TPOT). Batching rescues decode by amortizing one weight read across many tokens — until HBM or the compute roofline caps it.

8. Further reading

• PagedAttention / vLLM (Kwon et al., 2023) — why dense KV management unlocks the batch sizes that make decode economical.
• SARATHI: Chunked Prefills (2023) — the mechanism for mixing prefill and decode without stuttering streams.
• LLM Inference Serving Survey (2024) — comprehensive map of the optimization landscape and where each lever applies.
• vLLM Documentation — continuous batching, chunked prefill, prefix caching in a production stack.
• Meta Engineering: Tensor / Context / Expert Parallelism (Oct 2025) — how the phase asymmetry shapes parallelism choices at scale.
• SmoothQuant (Xiao et al., 2022) — activation quantization that actually moves throughput, and why weight-only often doesn't.

Next: /inference/kv-cache — how PagedAttention and RadixAttention pack the cache so you can climb the batch-size ladder this lesson described.