Trading bits for throughput — how INT8, FP8, and INT4 reshape the memory wall, and why FP8 on Blackwell is the default you'll defend in a senior interview.
Quantization stores a model's numbers in fewer bits — 8 or 4 instead of 16 — so they take less memory and less memory bandwidth to read. That second part is the whole game: LLM decode is memory-bound, not compute-bound, so shrinking the bytes you drag across the bus per token is the most direct lever on latency and cost. But not all quantization is equal: weight-only (GPTQ, AWQ) shrinks the model on disk and helps memory-bound decode, while activation quantization (SmoothQuant, FP8) actually feeds smaller numbers into the matmul and unlocks the tensor cores for raw throughput. The hard part is quality: low-precision formats clip outliers, and a handful of outlier channels carry most of a transformer's signal. In June 2026 the default answer for a frontier serving stack is FP8 on Hopper/Blackwell for weights, activations, and KV cache — with INT4 weight-only reserved for memory-constrained, low-batch, or edge regimes where you knowingly trade some quality for fitting the model at all.
Quantization is where a candidate either parrots "INT4 makes it 4x faster" or demonstrates they've actually profiled a serving stack. The signal differs sharply by level.
The trap at both levels is conflating memory savings with speedup. They are different claims with different bottlenecks, and the interviewer is listening for whether you keep them separate.
The words first.
q = round(x / scale).Step by step.
Remember this: fewer bits saves memory for sure, but only buys speed when it removes your actual bottleneck — and only ships if quality holds.
Start from the roofline. A matmul's throughput is bounded by either compute (FLOP/s) or memory bandwidth (bytes/s), whichever you hit first. The crossover is arithmetic intensity — FLOP done per byte read.
Prefill processes the whole prompt as matrix-matrix multiplies: arithmetic intensity is high (200-400 FLOP/byte on H100), the GPU runs at 90-95% utilization, and it's compute-bound. Decode generates one token at a time as matrix-vector multiplies against the KV cache, with essentially zero data reuse — every generated token re-reads the entire KV cache from HBM. Intensity collapses ~5x to 60-80 FLOP/byte, utilization falls to 20-40%, and it's memory-bound. Real serving kernels confirm this: measured implementations achieve roughly 23% compute-bandwidth utilization but ~47% memory-bandwidth utilization — DRAM bandwidth is the saturated resource.
This split is the master key to the whole lesson:
Quantization picks a scale s (and sometimes a zero-point z) and maps a float x to an integer q = round(x / s) + z, recovering x_hat = (q - z) * s. The error is bounded by s/2 per element, so the entire art is choosing s to be as small as possible without clipping real values.
Granularity controls how many values share one scale:
INT vs FP formats differ in how the grid is spaced:
Symbols: x = a float weight, s = scale (largest magnitude / 127 for symmetric INT8), q = stored 8-bit integer, x_hat = recovered value.
Take one output channel's weights: [0.10, -0.42, 1.27, 0.03]. Max magnitude = 1.27, so s = 1.27 / 127 = 0.01.
Quantize: q = round(x / s) →
0.10 / 0.01 = 10-0.42 / 0.01 = -421.27 / 0.01 = 1270.03 / 0.01 = 3Stored INT8: [10, -42, 127, 3] — 4 bytes instead of 8 (these were FP16). Recover: x_hat = q * s → [0.10, -0.42, 1.27, 0.03]. Here error is ~0 because the values landed on grid; the worst-case error per weight is s/2 = 0.005.
What it did: replaced 16-bit floats with 8-bit integers plus one shared scale per channel, halving weight bytes — the read that decode is bottlenecked on.
If you try the same per-tensor INT8 trick on activations, you fail. A handful of channels in LLM activations have magnitudes 10-100x the rest. With one shared scale, those outliers force s huge, and every normal value collapses into a few grid steps — catastrophic error. This is the central obstacle that the named methods solve in different ways.
70B params × 2 bytes = 140 GB (FP16 — baseline quality). Add the KV cache and activations on top — and you need it to fit in GPU HBM (e.g. an H100 has 80 GB). Going FP16 → INT4 is a 4× memory cut.
SmoothQuant (Xiao et al., 2022) is the elegant fix for W8A8. Its insight: the difficulty is migratable. For a linear layer Y = X · W, you can insert a per-channel scaling vector s and rewrite it as Y = (X / s) · (s · W) — mathematically identical, but now you've divided the heavy-tailed activations by s (taming outliers) and multiplied the well-behaved weights by s (which absorb the difficulty easily, since weights are flat and uniform). The scaling is folded offline into the preceding layer norm, so there's no runtime cost. Result: INT8 on both activations and weights, ~1.56x speedup and 2x memory reduction on models up to 530B params, with negligible accuracy loss.
Why it matters for the interview: without SmoothQuant (or an FP8 format whose non-uniform grid tolerates outliers natively), W8A8 quality falls off a cliff. This is the difference between "quantized the weights, easy" and "quantized the activations, knew what I was doing."
When you only need to fit the model and you're memory-bound at low batch, INT4 weight-only wins. Two methods dominate:
GGUF is the format that confuses people in interviews: it is not a quantization method, it's a container format (successor to GGML) optimized for CPU and Apple Silicon inference via llama.cpp, supporting flexible precisions (Q4_K_M, Q5_K_M, etc.). If a candidate calls GGUF "a quantization algorithm," that's a tell. It's the file format your INT4 weights live in for local/edge deployment.
The KV cache is frequently the largest and fastest-growing memory consumer — and because decode re-reads it every token, quantizing it cuts both capacity pressure and the bandwidth bottleneck directly. Two choices:
The frontier is going lower: TurboQuant (ICLR 2026) reports a K=4-bit + V=2-bit combination with near-zero accuracy loss, exploiting that keys and values tolerate quantization differently. The takeaway: KV-cache quant is often the highest-leverage quant decision for long-context serving, because it attacks memory and bandwidth simultaneously.
Below is production-shaped vLLM serving with FP8 weights + activations and an FP8 KV cache (the 2026 Hopper/Blackwell default), plus a small standalone function that makes the per-channel INT8 math from section 3.2 concrete and runnable.
# serve_fp8.py — vLLM with FP8 weights+activations and FP8 KV cache.
# Runs on Hopper (H100) or Blackwell (B200). Requires vllm >= 0.8.
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Llama-3.1-70B-Instruct",
quantization="fp8", # weights + activations in FP8 (E4M3)
kv_cache_dtype="fp8", # halve KV memory + KV bandwidth
tensor_parallel_size=2, # shard across 2 GPUs; activations-only comms
gpu_memory_utilization=0.92, # leave headroom for the paged KV pool
max_model_len=32768,
enable_prefix_caching=True, # reuse shared system-prompt KV across requests
)
# Quick A/B harness: compare FP8 output to a BF16 reference on YOUR eval set,
# because quant quality loss is silent until a user trips it.
prompts = ["Explain the memory-bandwidth wall in LLM decode."]
out = llm.generate(prompts, SamplingParams(temperature=0.0, max_tokens=256))
print(out[0].outputs[0].text)# int8_channel.py — the actual per-channel symmetric INT8 quantizer.
# This is what "weight-only INT8" does under the hood; no framework needed.
import numpy as np
def quantize_per_channel_int8(W: np.ndarray):
"""W: [out_features, in_features] float weights.
Returns int8 weights + per-channel (per-row) scales."""
# one scale per output channel = per row; keepdims for broadcasting
absmax = np.abs(W).max(axis=1, keepdims=True) # [out, 1]
scale = absmax / 127.0 # symmetric INT8 range
scale = np.maximum(scale, 1e-8) # avoid div-by-zero
q = np.round(W / scale).clip(-127, 127).astype(np.int8)
return q, scale.astype(np.float32)
def dequantize(q: np.ndarray, scale: np.ndarray):
return q.astype(np.float32) * scale # x_hat = q * s
if __name__ == "__main__":
rng = np.random.default_rng(0)
W = rng.standard_normal((4, 8)).astype(np.float32) # toy weight matrix
q, s = quantize_per_channel_int8(W)
err = np.abs(W - dequantize(q, s)).max()
bytes_fp16, bytes_int8 = W.size * 2, q.size + s.size * 4
print(f"max abs error: {err:.4f}") # bounded by max(scale)/2
print(f"bytes FP16={bytes_fp16} INT8(+scales)={bytes_int8}")The first script is what you'd actually deploy; the second is what you'd whiteboard to prove you understand the mapping. Note the deliberate temperature=0.0 A/B comment — the most common production mistake is shipping a quant config without diffing it against a full-precision reference on a real eval set.
| Strategy | Bits (W/A/KV) | Memory saving | Speeds up… | Quality risk | When to reach for it |
|---|---|---|---|---|---|
| FP16/BF16 | 16/16/16 | baseline | — | none | Quality ceiling / debugging reference |
| W8A8 SmoothQuant (INT8) | 8/8/16 | ~2x weights | compute + memory | low (outliers handled) | Pre-Hopper hardware, throughput-max |
| FP8 (E4M3) | 8/8/8 | ~2x w+kv | compute + memory + decode | very low | 2026 default on Hopper/Blackwell |
| W4A16 AWQ/GPTQ | 4/16/16 | ~4x weights | decode only (memory-bound) | moderate | Memory-constrained, low batch, edge |
| GGUF Q4_K_M | ~4/16/16 | ~4x weights | CPU/Apple decode | moderate | Local/edge, llama.cpp |
| FP8 KV cache | —/—/8 | ~2x KV | decode (bandwidth) | very low | Long context, any modern GPU |
| FP4 (Blackwell) | 4/4/— | ~4x | compute + memory | model-dependent | B200, throughput-per-dollar at scale |
Cost & latency. The two metrics to keep separate: TTFT (time to first token, dominated by prefill = compute-bound) and TPOT (time per output token, dominated by decode = memory-bound). Weight-only INT4 improves TPOT in low-batch decode but barely touches TTFT, and can even raise it under compute pressure due to dequant overhead. FP8/INT8 activation quant improves both because it puts real low-precision math on the tensor cores. On Blackwell, native FP4 doubles theoretical compute versus FP8 and accelerates both phases — which is why B200 dominates throughput-per-dollar at FP4, despite higher TCO and ~1000W TDP.
Failure modes. (1) Silent quality regression — quant loss doesn't crash, it degrades reasoning, code, or long-context recall in ways unit tests miss; always gate on a held-out eval. (2) Outlier blow-up — naive activation quant without SmoothQuant/FP8 destroys accuracy. (3) Speedup mirage — measuring memory saved and claiming throughput; profile tokens/sec end-to-end. (4) Calibration drift — GPTQ/AWQ/SmoothQuant calibrate on a sample set; an unrepresentative calibration corpus bakes in error.
What changes at scale. With continuous batching, decode stops being purely memory-bound as batch size grows (you reuse weights across many sequences), which shrinks weight-only INT4's advantage and grows activation-quant's. In a disaggregated prefill/decode deployment you can even pick precision per phase — FP8/FP4 prefill on compute-dense B200, FP8 decode on high-bandwidth H200 — letting quant choice follow the bottleneck rather than forcing one config everywhere. See /inference/batching and /system-design.
X·W as (X/s)·(s·W) — mathematically identical — dividing the heavy-tailed activations by a per-channel s to tame outliers, and multiplying the flat, well-behaved weights by s, which absorb that difficulty easily. The scaling folds offline into the prior layer norm, so there's no runtime cost. Without it, W8A8 activation quant suffers catastrophic accuracy loss; you'd be forced back to weight-only INT8 and lose the activation-quant throughput gains. FP8's non-uniform grid is an alternative that tolerates outliers natively without the rewrite.Flashcard. Decode is memory-bound, so shrinking bytes-per-token is the master lever — but only activation/FP8 quant feeds small numbers into the matmul and speeds up both phases; weight-only INT4 only helps memory-bound decode and is a tool for fitting the model, not a default for throughput. In 2026, FP8 on Hopper/Blackwell is the baseline; INT4 is the exception.
kv_cache_dtype in practice.Next: /inference/speculative-decoding — the other lossless-ish throughput lever, and how it composes with quantization.