A 405B model does not fit on one GPU and a single phase does not deserve uniform hardware — this is how you split weights, KV, and experts across a fleet and bill it by the token.
A frontier model is too big for one GPU, and the two phases of inference want opposite hardware, so distributed serving is two separate decompositions stacked on top of each other. The intra-model decomposition splits one forward pass across GPUs: tensor parallelism (TP) splits each matrix, pipeline parallelism (PP) splits the layer stack, expert parallelism (EP) splits the experts of an MoE, and FSDP/ZeRO splits the parameters themselves (a training idiom). The inter-phase decomposition is disaggregation: prefill is compute-bound and decode is memory-bandwidth-bound, so you run them on different machines and ship the KV cache between them. On top of both sits the fleet layer — many model replicas, autoscaling, prefix-cache affinity, and a scheduler that bills by the token. The whole job of a serving engineer is to choose where each cut falls so that the comms each cut imposes stays cheaper than the parallelism it buys.
The words first.
Step by step.
Remember this: every cut you make in the model buys you memory or throughput and charges you a network collective — your job is to keep the trade favorable.
Start from the one number that governs everything: a forward pass is a stack of matrix multiplies, and the GPU you run it on has finite HBM and finite interconnect bandwidth. Every parallelism axis is a different answer to "what do I cut so it fits, and what do I send so it still computes the right thing?"
Tensor parallelism (TP) cuts inside each operation. A feed-forward block Y = GeLU(X·W1)·W2 is split by sharding W1 column-wise and W2 row-wise across N GPUs. Each GPU computes a partial Y, and a single all-reduce sums the partials back into the full activation before the next layer. Attention shards the same way — heads partition across GPUs. The key property: TP communicates activations, which are small (a few MB per token-batch), and every rank participates in every step so there are no idle GPUs. The cost is that all-reduce happens twice per transformer layer, so TP only pays off on very fast links — NVLink inside a node, ~900 GB/s on H100. Cross-node TP over InfiniBand usually loses. This is why the production default is "TP up to the node boundary (TP=8 on an 8-GPU box), no further."
Pipeline parallelism (PP) cuts between layers: GPU 0 holds layers 1–10, GPU 1 holds 11–20, and so on. Communication is a tiny point-to-point send of the activation at the layer boundary — cheap, and crosses node boundaries fine. The price is the pipeline bubble: while GPU 0 works on micro-batch 1, GPUs 1–N sit idle, and they only fill once the pipeline is primed. For training you hide the bubble with many micro-batches; for inference, especially decode where the batch is small and latency-sensitive, the bubble is mostly unhidable. That's why PP is a training workhorse and an inference last-resort — you use it only when the model is too big for TP-within-a-node and you've run out of NVLink.
Expert parallelism (EP) is specific to Mixture-of-Experts. An MoE layer has, say, 128 experts but routes each token to only 2 of them. EP places different experts on different GPUs, and routing becomes a two-shot all-to-all: an all-to-all to dispatch each token to the GPU holding its chosen experts, compute, then an all-to-all to gather results back. This is the hardest collective to make fast because it's irregular — token-to-expert assignment is data-dependent and unbalanced, so some GPUs get flooded ("hot experts") while others starve. MegaScale-Infer (SIGCOMM 2025) attacks exactly this for disaggregated MoE serving: mapping co-activated experts to separate GPUs and tuning the all-to-all to keep ranks balanced.
FSDP / ZeRO shards the parameters themselves. ZeRO-1 shards optimizer state, ZeRO-2 adds gradients, ZeRO-3 (= FSDP) adds parameters: each GPU holds 1/N of every weight and all-gathers the full layer just before computing it, then drops it. This is the right tool for training a model too big to replicate, because the optimizer/gradient state is what dominates training memory. For inference it's a poor fit — you'd all-gather full weights every forward pass with no gradient state to amortize against, paying TP-level comms for none of TP's evenness. The interview trap is conflating FSDP (parameter sharding, training) with TP (operation sharding, inference). They both "split the weights," but the collective, the timing, and the use case differ.
Name the symbols: d_model is the hidden width (4096), d_ff is the FFN width (14336 for a Llama-style 70B), N is the TP degree (8 GPUs), B is tokens in the batch (say 32). TP shards W1 (shape d_model × d_ff) by columns and W2 (d_ff × d_model) by rows across the 8 GPUs.
Work it: each GPU now holds W1 columns 4096 × (14336/8) = 4096 × 1792 and the matching W2 rows. It computes a partial output of shape B × d_model = 32 × 4096. To get the true output, all 8 partials must be summed — that's one all-reduce over a 32 × 4096 float16 tensor = 32 × 4096 × 2 bytes = 256 KB per all-reduce, and there are 2 per layer (one in attention, one in FFN). For an 80-layer model that's 80 × 2 × 256 KB ≈ 40 MB of all-reduce traffic per forward step.
What it did: it turned a matrix that needs 4096 × 14336 × 2 = ~117 MB of weights (won't dominate one GPU but stacks up over 80 layers and KV) into 8 slices of ~15 MB each, at the cost of 40 MB of NVLink traffic per step. On a 900 GB/s NVLink that 40 MB costs ~44 microseconds — negligible. Push the same all-reduce onto a 50 GB/s cross-node link and it costs ~800 microseconds per step, which is why TP stays inside the node.
No single axis serves a 400B+ model. You compose them, and the ordering is dictated by the bandwidth hierarchy: fastest collective on fastest link.
A worked sizing: DeepSeek-style 400B+ MoE, TP=8 (intra-node), EP across 4 nodes for the expert layers, PP=2 for the dense layers, then N replicas of that 8-node unit behind a load balancer. The thing an IC6 must articulate is why this ordering: if you put PP inside the node and TP across nodes, you'd push 40 MB all-reduces over InfiniBand every step and the model would run at a fraction of its FLOPs. The bandwidth hierarchy is the constraint; the parallelism plan is just a packing problem against it.
The second decomposition is orthogonal to all of the above and is where 2025–2026 serving made its biggest gains. Recall the phase asymmetry: prefill processes the whole prompt in parallel as matrix–matrix multiplies, hitting 200–400 FLOP/byte arithmetic intensity and 90–95% GPU utilization — compute-bound. Decode generates one token at a time as matrix–vector ops over the KV cache, arithmetic intensity collapsing ~5x to 60–80 FLOP/byte with 20–40% utilization — memory-bandwidth-bound. Co-locating them on one GPU means decode's latency-sensitive token stream gets stalled behind prefill's compute bursts, and you size the same expensive hardware for two opposite profiles.
Disaggregation runs them on separate clusters: prefill on compute-optimized hardware (H100, wide TP, modest HBM is fine), decode on bandwidth-optimized hardware (H200/B200, big HBM, high bandwidth). Prefill computes the KV cache for the prompt, ships it over the network to a decode worker, and decode streams tokens. The notes put the network hop at typically <50ms — acceptable, because it's a one-time cost folded into TTFT, not paid per token.
The economic payoff is real: per the research notes, B200-prefill + Gaudi3-decode lands ~3–4x TCO benefit on prefill-heavy workloads and ~2.5–4x on decode-heavy ones, because each phase runs on hardware it actually utilizes instead of overpaying for the wrong resource. The cost is operational: two pools to autoscale independently, a KV transfer path to keep fast, and a scheduler that has to match a freed decode slot to a just-finished prefill.
This is also where it composes with the batching story you should already know — continuous (iteration-level) batching mixes chunked-prefill work for new requests with ongoing decode for in-flight ones, and chunked prefill (SARATHI) splits a length-L prompt into size-K chunks to cut prefill's memory pressure from O(L²) toward O(LK) so it can be interleaved. Disaggregation pushes those two kinds of work onto different machines rather than interleaving them on one — a cleaner separation when traffic is high enough to keep both pools full.
Above one model copy sits everything that makes it a service: replicas, a router, an autoscaler, and prefix-cache affinity.
Multi-model serving means many models (or many fine-tunes/LoRA adapters) share a GPU fleet. The lever is that LoRA adapters are tiny relative to the base; you keep one base model resident and hot-swap adapters per request, so a single GPU pool serves dozens of fine-tunes without dozens of full copies.
Prefix-cache affinity is the highest-leverage routing decision. SGLang's RadixAttention stores KV in a radix tree and matches new prompts against cached prefixes automatically — multi-turn chat and agentic loops re-send the same system prompt and history every turn, and the notes cite up to 5x throughput over vLLM on high-prefix-reuse workloads. But the cache lives on a specific GPU. So your router must be cache-aware: send a continuing conversation back to the replica that already holds its prefix, or you re-prefill from scratch and throw the win away. This is the single subtlest fleet decision and a favorite IC6 probe.
Autoscaling must scale on the right signal. CPU and even GPU-utilization are misleading for LLM serving because a decode-bound box can be 30% "utilized" and still saturated on memory bandwidth. The signals that actually track SLOs are queue depth / waiting requests, TTFT, and TPOT — scale prefill pool on TTFT and queue depth, scale decode pool on TPOT and KV-cache occupancy. Because cold-starting a 400B model is slow (minutes to load weights), you scale on predictive signals and keep warm headroom rather than reacting after the SLO is already breached.
The realistic "implementation" at this level is a serving deployment, not a from-scratch kernel. Here's a production-shaped vLLM launch that exercises tensor parallelism, plus the disaggregation control plane sketch an IC6 should be able to write. This runs against vLLM as documented (June 2026).
# serve_tp.py — launch a 70B model with TP=8 inside one node, expose an OpenAI API.
# Run: python serve_tp.py (requires an 8x H100/H200 node, vllm>=0.8)
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Llama-3.3-70B-Instruct",
tensor_parallel_size=8, # shard each matmul across 8 NVLink-connected GPUs
pipeline_parallel_size=1, # no PP: model fits within one node's TP group
enable_prefix_caching=True, # block-level prefix cache for repeated system prompts
enable_chunked_prefill=True, # interleave long-prompt prefill with ongoing decode
max_num_batched_tokens=8192, # decode-maximal batching budget per iteration
kv_cache_dtype="fp8", # halve KV memory + traffic on Hopper/Blackwell
gpu_memory_utilization=0.92, # leave headroom; OOM here = dropped requests
)
# Continuous batching is automatic: the engine slots new requests into freed
# decode slots every iteration. We just fire concurrent requests.
prompts = ["Explain pipeline bubbles in one paragraph."] * 64
out = llm.generate(prompts, SamplingParams(max_tokens=256, temperature=0.7))
print(out[0].outputs[0].text)What each knob buys you, in the language of section 3: tensor_parallel_size=8 is the §3.1 all-reduce split kept inside the NVLink domain. enable_chunked_prefill + max_num_batched_tokens is the §3.3 batching story — they let one iteration mix prefill chunks with decodes so a long prompt doesn't stall the token stream. enable_prefix_caching is the §3.4 affinity win, but note: within one engine it's free; across replicas you need a cache-aware router the engine can't give you. kv_cache_dtype="fp8" halves decode's memory-bandwidth load, directly attacking the decode bottleneck.
For disaggregation, the control-plane shape (vLLM and SGLang both ship a version of this in 2026) is two pools and a connector:
# Prefill pool: compute-optimized, produces KV and pushes it to the connector.
vllm serve meta-llama/Llama-3.3-70B-Instruct \
--tensor-parallel-size 8 \
--kv-transfer-config '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer"}' \
--port 8100
# Decode pool: bandwidth-optimized, consumes KV and streams tokens.
vllm serve meta-llama/Llama-3.3-70B-Instruct \
--tensor-parallel-size 8 \
--kv-transfer-config '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer"}' \
--port 8200
# A front proxy routes: request -> prefill (8100) -> KV handoff -> decode (8200) -> stream.The load-bearing detail is kv_role: the producer computes and exports the KV cache; the consumer imports it and decodes. The proxy in front is where your cache-affinity and pool-balancing logic lives — that's the part no framework hands you, and the part an interviewer wants you to own.
| Axis | What splits | Collective on the wire | Where it lives | Inference fit | Main failure mode |
|---|---|---|---|---|---|
| Tensor (TP) | each matmul | all-reduce (2/layer), small activations | intra-node, NVLink | Default — no bubbles, every rank busy | dies on slow links; degree capped at node size |
| Pipeline (PP) | layer ranges | point-to-point, tiny | cross-node, IB | last resort | pipeline bubble, unhidable at decode |
| Expert (EP) | MoE experts | all-to-all, irregular | cross-node | MoE-only | hot experts, load imbalance |
| FSDP/ZeRO-3 | parameters | all-gather per layer | training | poor | no gradient state to amortize at inference |
| Disaggregation | phases | KV transfer (<50ms, once) | two pools | strong at scale | KV transfer path, two-pool scheduling |
Cost. The fleet bill is dominated by GPU-hours, and the two biggest levers are (1) matching hardware to phase via disaggregation (the notes cite 2.5–4x TCO improvements) and (2) raising effective batch size so each GPU amortizes its weight reads — paged/radix KV management is what lets you batch larger in the same HBM (4–24x throughput from larger batches, per the PagedAttention work). FP4 on B200 and FP8 KV cache shift the per-token economics further by cutting both compute and memory traffic.
Latency. TTFT is set by prefill + scheduling delay + (in disaggregation) the KV hop; TPOT is set by decode's memory bandwidth and how full the batch is. The cruel coupling: raising batch size lifts throughput-per-dollar but raises TPOT under load, so every fleet has an SLO-bounded max batch. Disaggregation helps because decode's TPOT no longer jitters behind prefill bursts.
Quality. Parallelism is mathematically lossless — TP all-reduce reconstructs the exact activation, speculative decoding accepts only target-distribution tokens. The quality risks come in adjacent to parallelism: aggressive KV quantization (W4/INT4 KV) and over-large speculation lengths trade accuracy for speed, and those choices interact with how you've sharded.
What changes at scale. At one node, you tune batch size and call it a day. At fleet scale the hard problems migrate to the control plane: cache-aware routing (or you re-prefill and lose 5x), predictive autoscaling on queue depth/TTFT/TPOT rather than utilization, MoE expert load-balancing, and warm-pool management because cold-starting a 400B model is minutes. The model-parallel plan becomes a solved, static config; the dynamic, money-losing failures are all in scheduling and routing.
Flashcard. TP = split each matmul, all-reduce activations, intra-node. PP = split layers, point-to-point, eats a bubble. EP = split experts, all-to-all, watch hot experts. FSDP = split parameters, all-gather, training only. Disaggregation = split phases, ship the KV.
Next: /inference/kv-cache-and-paged-attention — the memory layer that makes every batching and caching win in this lesson physically possible.