vLLM, PagedAttention, KV cache, speculative decoding, quantization (AWQ/FP8), FSDP & tensor/pipeline parallelism.
Every token your model emits is governed by two physics regimes — a compute-bound prefill and a memory-starved decode — and almost every serving decision is a fight over the second one.
The KV cache is the silent tenant of every GPU — derive its exact byte cost, then watch PagedAttention turn 40% wasted HBM into near-zero fragmentation.
Trading bits for throughput — how INT8, FP8, and INT4 reshape the memory wall, and why FP8 on Blackwell is the default you'll defend in a senior interview.
A small model gambles on the next few tokens, the big model checks all of them in one cheap pass, and you pay for memory bandwidth you were already wasting.
A 405B model does not fit on one GPU and a single phase does not deserve uniform hardware — this is how you split weights, KV, and experts across a fleet and bill it by the token.