A small model gambles on the next few tokens, the big model checks all of them in one cheap pass, and you pay for memory bandwidth you were already wasting.
Autoregressive decoding generates one token per forward pass, and each pass through a large model is memory-bound: you drag the entire weight matrix and KV cache through HBM just to compute a single matrix-vector product. The GPU's compute units sit ~60-80% idle. Speculative decoding exploits that slack. A cheap draft model proposes k tokens ahead; the expensive target model then runs one forward pass over all k proposed positions in parallel — a matrix-matrix multiply that costs almost the same wall-clock time as a single decode step. A clever acceptance rule keeps every token that the target "would have" produced anyway, so the output distribution is provably identical to plain sampling. You get multiple tokens for the price of one forward pass, and you pay only in wasted draft compute on rejected guesses.
Speculative decoding is the cleanest test of whether a candidate actually understands the roofline of LLM inference, not just the API.
The words first.
Step by step.
k candidate tokens, one after another (cheap, because it's small).k positions in parallel.Remember this: You're trading cheap, sometimes-wasted draft compute for the expensive thing you were already paying for — and getting several tokens per target pass instead of one.
Take a 70B model in BF16. Generating one token in plain decode reads ~140 GB of weights from HBM (plus the KV cache) to do a batch-of-1 matrix-vector multiply. On an H100 with 3.35 TB/s of bandwidth, just reading the weights takes ~42 ms — and the FLOPs to multiply them against a single token vector finish in a fraction of that time. The GPU stalls waiting on memory; compute utilization on decode typically lands at 20-40%. That idle compute is the resource speculative decoding monetizes.
The key asymmetry: a target forward pass over a single token and a forward pass over k=5 tokens cost nearly the same wall-clock time, because both are dominated by the same weight read. The k-token version turns the matrix-vector op into a matrix-matrix op that finally uses the idle FLOPs. So if you can guess what those k tokens should be and verify them in one pass, you get up to k+1 tokens for the latency of roughly one decode step.
The non-obvious part is that you can do this without changing the output distribution. The algorithm (Leviathan et al. 2023; Chen et al. 2023) works token-by-token over the draft. Let p(x) be the target's probability for token x at a position and q(x) the draft's probability for the token it sampled there.
r in [0, 1).x if r < min(1, p(x) / q(x)).q(x) >= p(x)) you accept with probability p(x)/q(x); if the target liked it more (p(x) >= q(x)) you always accept.p_resid(x) = norm(max(0, p(x) - q(x))) — the part of the target's mass the draft under-weighted. Then you discard the rest of the draft.The math guarantee: tokens emitted this way are distributed exactly as if you had sampled from p directly. The proof is a two-case probability decomposition; the residual-resampling step is precisely what corrects the bias the draft introduced. This is why you can swap in a sloppy draft model with zero quality risk — a bad draft only lowers your acceptance rate, never your accuracy. (At temperature 0 / greedy, the rule collapses to "accept iff the draft's argmax equals the target's argmax.")
Symbols: p = target's probability for a token, q = draft's probability for the token it proposed, r = a random draw in [0,1).
Say at one position the draft proposes the token "blue" with q(blue) = 0.6. The target, in its verification pass, assigns p(blue) = 0.3.
min(1, p/q) = min(1, 0.3/0.6) = 0.5.r = 0.42. Since 0.42 < 0.5, accept "blue". We keep it and move to the next drafted token.r = 0.71. Since 0.71 >= 0.5, reject. We now resample this position from the residual max(0, p - q) over the whole vocab, renormalized — favoring tokens the target liked but the draft under-rated (e.g. "green", where p=0.4, q=0.05). Then we throw away every drafted token after this one.What it did: it kept a cheap guess only as often as the math says it must to stay faithful to p, and on a miss it spent one corrected sample to stay exactly on the target's distribution.
Let alpha be the per-token acceptance probability (assume i.i.d. for intuition; real acceptance decays with position). With a draft of length k, the expected number of tokens accepted before the first rejection is a truncated geometric series. The expected tokens produced per target forward pass is:
E[tokens] = (1 - alpha^(k+1)) / (1 - alpha)
The +1 accounts for the bonus token the target emits for free after the last accepted position. The wall-clock speedup is roughly:
speedup ≈ E[tokens] / (1 + k·c)
where c is the cost of one draft forward pass relative to one target forward pass (e.g. a 1B draft against a 70B target gives c ≈ 0.015). The denominator is the overhead: one target verification pass plus k draft passes.
Concretely, with alpha = 0.7, k = 5, c = 0.02: E[tokens] = (1 - 0.7^6)/0.3 ≈ (1 - 0.118)/0.3 ≈ 2.94 tokens per target pass, and overhead 1 + 5·0.02 = 1.1, so speedup ≈ 2.94 / 1.1 ≈ 2.7x. Push alpha to 0.85 and you get E[tokens] ≈ 4.4, speedup ~3.9x. This is why acceptance rate dominates everything — it's inside an exponent. Draft length k has diminishing returns: each extra token is accepted with probability alpha^position, so beyond k ≈ 4-8 you're mostly paying draft cost for tokens that rarely survive.
Two practical corollaries: (1) the right k depends on alpha — high acceptance justifies longer drafts; (2) you want the cheapest draft that hits your acceptance target, because c and per-step latency scale with draft size. The 2025 scaling-laws work (arXiv:2505.07858) formalizes the draft-size-to-acceptance relationship so you can size a draft without grid-searching.
There are three families, and choosing among them is the real engineering decision.
Separate draft model. A genuinely smaller model from the same family (e.g. Llama-3.2-1B drafting for Llama-3.3-70B). Highest ceiling on acceptance because it's a real LM, but you pay extra HBM for its weights and KV cache, and the draft must share the target's tokenizer. Sweet spot: draft is roughly 5-25% of target params.
Self-speculation (Medusa, EAGLE). Instead of a separate model, you bolt lightweight prediction heads onto the target itself.
N-gram / prompt lookup. Training-free. You maintain a suffix automaton over recently generated and prompt tokens, and when the current suffix matches, you "draft" by copying the continuation that followed last time. Zero extra parameters, zero extra HBM. Astonishingly effective on repetitive or extractive workloads — code with repeated identifiers, JSON/structured output, RAG where the model quotes the context, edit/rewrite tasks. Useless on creative, low-repetition text.
A linear draft of k tokens commits to one continuation. A draft tree proposes several branches (e.g. top-2 at each of a few positions), then verifies the whole tree in a single target pass using a custom tree attention mask so each candidate path only attends to its own ancestors. You accept the longest valid root-to-node path. This raises expected accepted length per pass for a modest increase in verified-token count, and is the core trick behind EAGLE-2/3 and SpecInfer. The cost: more complex kernels and a larger verification batch dimension, which eats into the "verification is free" assumption as the tree grows.
Here is a correct, distribution-preserving single-sequence implementation. It's deliberately framework-free so the accept/reject logic is visible; production systems (vLLM, SGLang) fuse all of this into batched kernels with tree attention.
import torch
import torch.nn.functional as F
@torch.no_grad()
def speculative_generate(target, draft, input_ids, n_new, k=5, temperature=1.0):
"""Lossless speculative decoding for one sequence.
target, draft: callables returning logits [batch, seq, vocab].
Emits exactly the target's sampling distribution."""
device = input_ids.device
seq = input_ids
produced = 0
def sample(logits): # logits: [vocab]
if temperature == 0:
return logits.argmax(-1, keepdim=True)
probs = F.softmax(logits / temperature, dim=-1)
return torch.multinomial(probs, 1)
while produced < n_new:
# 1) Draft k tokens autoregressively (cheap, small model).
draft_ids, draft_probs = [], []
cur = seq
for _ in range(k):
logits = draft(cur)[0, -1] # [vocab]
p = F.softmax(logits / max(temperature, 1e-6), dim=-1)
tok = torch.multinomial(p, 1) if temperature > 0 else logits.argmax(-1, keepdim=True)
draft_ids.append(tok)
draft_probs.append(p)
cur = torch.cat([cur, tok.view(1, 1)], dim=1)
draft_ids = torch.cat(draft_ids) # [k]
# 2) ONE target pass scores all k drafted positions + the next-token slot.
t_logits = target(cur)[0, -(k + 1):] # [k+1, vocab]
t_probs = F.softmax(t_logits / max(temperature, 1e-6), dim=-1)
# 3) Walk the draft, accept/reject token by token.
n_accepted = 0
for i in range(k):
tok = draft_ids[i].item()
p_t = t_probs[i, tok] # target prob of drafted token
p_d = draft_probs[i][tok] # draft prob of drafted token
r = torch.rand(1, device=device)
if r < torch.clamp(p_t / (p_d + 1e-9), max=1.0):
n_accepted += 1
else:
# Reject: resample THIS position from residual (p_target - p_draft)+.
resid = torch.clamp(t_probs[i] - draft_probs[i], min=0)
resid = resid / resid.sum()
corrected = torch.multinomial(resid, 1)
seq = torch.cat([seq, draft_ids[:i].view(1, -1),
corrected.view(1, 1)], dim=1)
produced += n_accepted + 1
break
else:
# All k accepted -> append them + the FREE bonus token from t_probs[k].
bonus = (t_probs[k].argmax(-1, keepdim=True) if temperature == 0
else torch.multinomial(t_probs[k], 1))
seq = torch.cat([seq, draft_ids.view(1, -1), bonus.view(1, 1)], dim=1)
produced += k + 1
return seqThe load-bearing details: (1) the single target(cur) call over k+1 positions is what makes verification "free" — it's one weight read. (2) On rejection you resample from the clamped residual max(0, p_target - p_draft), not from p_target — that correction is what makes the whole thing exactly equal to sampling from the target. (3) The else branch on the for loop fires only when nothing was rejected, harvesting the bonus token. A real implementation caches KV for the accepted prefix so the next iteration doesn't recompute it, and batches across sequences.
| Approach | Acceptance ceiling | Extra HBM | Setup cost | Best workload | Failure mode |
|---|---|---|---|---|---|
| Separate draft model | High (real LM) | Draft weights + KV | Need matching tokenizer | General chat, low batch | Tokenizer mismatch; HBM pressure |
| Medusa heads | Medium | Small (heads only) | Train heads on target | Latency-critical, single stream | Acceptance decays with length |
| EAGLE-3 | High (~70-80% flat) | Small (one head) | Train head on features | General, production default | Training pipeline; feature coupling |
| N-gram / lookup | Variable (workload-dependent) | ~zero | None | Code, JSON, RAG, edits | Collapses on creative text |
The decisive variable is batch size / load, and it's where IC6 answers live. Speculative decoding spends extra compute (draft passes + a larger verification matmul) to buy back latency on memory-bound decode. At batch size 1, decode is deeply memory-bound, compute is idle, and you cash in — 2-3x TPOT improvement is routine. But continuous batching also fills idle compute, by stacking many independent sequences into one matmul. As batch size climbs, decode shifts from memory-bound toward compute-bound; the idle FLOPs speculative decoding was monetizing no longer exist. Now the extra draft and verification compute is pure overhead, and you can lose throughput.
The practical regime map: speculative decoding is a latency optimization (minimize TTFT-adjacent and TPOT for a single user), continuous batching is a throughput optimization (maximize tokens/sec/GPU across users). They trade off. High-end serving stacks gate speculation dynamically — enable it when the running batch is small (interactive, bursty, or latency-SLO traffic) and disable it under heavy batched load. The crossover is hardware- and model-specific but commonly lands somewhere in the low-tens of concurrent sequences; you measure it, you don't guess it. Other failure modes: tokenizer or vocab mismatch between draft and target (silent corruption or zero acceptance), KV cache for the draft eating your memory budget, and acceptance rate that looks great on your eval prompts but cratering on out-of-distribution production traffic.
k tokens both cost roughly one full weight read from HBM, which dominates wall-clock time; the extra FLOPs for k tokens run on otherwise-idle compute units. So verifying k drafted tokens in parallel costs about the same latency as generating one token sequentially. You're converting idle compute into accepted tokens; the "same total FLOPs" framing misses that FLOPs weren't the bottleneck.(1 - 0.7^6)/(1 - 0.7) ≈ 2.9. Net of draft overhead (cheap draft, ~10% added) you land near 2.5-2.7x on TPOT. What kills it: (1) acceptance falling on out-of-distribution traffic — it's inside an exponent, so a drop from 0.7 to 0.5 roughly halves the gain; (2) running under high batch load, where continuous batching already saturates compute and the extra draft work becomes overhead; (3) a draft that's too large, inflating c and per-step latency.x with probability min(1, p(x)/q(x)) and on rejection resamples from the normalized residual max(0, p(x) - q(x)). Decompose the probability of finally emitting any token y: the "accepted" mass is q(y)·min(1, p(y)/q(y)) = min(q(y), p(y)), and the "rejected then resampled" mass works out to p(y) - min(q(y), p(y)). Summed, that's exactly p(y). So the emitted token is distributed as p, identically to plain sampling — the draft only affects how often you accept, never what distribution you sample from.k as batch grows) is a softer version of the same idea.alpha on representative traffic.p instead of the residual on rejection. This silently biases the output — it's no longer lossless. The correction must be max(0, p - q) normalized.i survive with probability ~alpha^i; past k ≈ 4-8 you mostly pay draft cost for tokens that get rejected.Flashcard. Speculative decoding is lossless and helps only when decode is memory-bound; expected tokens per target pass =
(1 - alpha^(k+1))/(1 - alpha), so acceptance rate (inside an exponent) dominates draft length.
Next: /inference/continuous-batching — the throughput lever that trades off against speculation, and why the two have to be co-scheduled.