Every token a model emits is a deliberate bet placed on a reshaped probability distribution — this lesson teaches you to control that bet.
A transformer's forward pass ends with one vector of vocab_size real numbers — the logits — for the next position. Everything you call "the model's personality," "creativity," "determinism," or "hallucination at high temperature" happens after that vector, in the decoding step that turns logits into a single chosen token. Decoding is not the model; it is a tiny, cheap, fully-controllable post-processor you bolt on top. The whole field is a sequence of two operations applied to logits — reshape the distribution (temperature, top-k, top-p, min-p, penalties) and then pick from it (argmax, sample, or search) — and almost every production quality complaint traces back to a bad choice in those two operations rather than to the weights. Master this and you can debug "the model is repetitive / unhinged / off-format" without touching a single parameter.
openai.chat.completions.create(temperature=0.7) without knowing what 0.7 means.temperature=0 is still not bit-for-bit reproducible across hardware?The words first.
Step by step.
vocab_size for the next position.Remember this: decoding never changes the weights — it only reshapes one probability vector at a time and then makes a pick.
The forward pass gives you logits z = [z_1, ..., z_V]. Softmax maps them to probabilities: p_i = exp(z_i) / sum_j exp(z_j). Two properties matter for decoding. First, softmax is shift-invariant — adding a constant to every logit changes nothing — which is why implementations subtract max(z) before exponentiating (numerical stability, no semantic effect). Second, softmax is scale-sensitive: multiply all logits by a constant and the distribution sharpens or flattens. That scaling is exactly what temperature exploits.
Temperature tau rescales logits before softmax: p_i ∝ exp(z_i / tau). Define tau as the divisor applied to every logit. Lower tau makes the gaps between logits larger in exponent space, so the top token dominates (sharper, more deterministic). Higher tau shrinks the gaps, flattening toward uniform (more random). tau = 1 is the identity. The limit tau -> 0 collapses to greedy (argmax), and in practice APIs alias temperature=0 to greedy because dividing by literal zero is undefined — they short-circuit to argmax.
We have three candidate tokens with logits z = [2.0, 1.0, 0.0] (say tokens "cat", "dog", "fish").
[e^2, e^1, e^0] = [7.39, 2.72, 1.00], sum = 11.11. Probabilities = [0.665, 0.245, 0.090]. "cat" 66%, "dog" 25%, "fish" 9%.[4.0, 2.0, 0.0], exponentiate [54.6, 7.39, 1.0], sum = 62.99. Probabilities = [0.867, 0.117, 0.016]. "cat" jumps to 87%; "fish" nearly vanishes.[1.0, 0.5, 0.0], exponentiate [2.72, 1.65, 1.0], sum = 5.37. Probabilities = [0.506, 0.307, 0.186]. Now "fish" has a real 19% shot.What it did: the same logits produced a near-deterministic pick at tau=0.5 and a genuinely diverse three-way race at tau=2.0. The model never changed — only the temperature divisor did.
Temperature alone is dangerous at the high end: flattening the distribution gives every token — including the absurd long tail — non-trivial mass, so you eventually sample nonsense. Truncation methods solve this by discarding the tail before sampling.
Top-k keeps the k highest-probability tokens, masks the rest to probability zero, renormalizes, and samples. It is simple but rigidly fixed: when the model is supremely confident (one token at 0.99), k=40 still drags in 39 near-zero tokens; when the model is genuinely uncertain (a flat distribution over 200 plausible continuations), k=40 arbitrarily amputates the rest.
Top-p / nucleus (Holtzman et al., ICLR 2020) fixes the rigidity by being adaptive. Sort tokens by descending probability, accumulate until the cumulative sum first reaches p (e.g. 0.9), keep exactly that set (the "nucleus"), renormalize, sample. When the model is confident the nucleus might be 1-2 tokens; when uncertain it might be hundreds. This is the production default for open-ended generation and the single most important method in this lesson.
Min-p (Hewitt et al., 2024, "Turning Up the Heat") attacks a specific pathology: at high temperature, top-p can still admit very low-probability tokens because flattening inflates the tail's cumulative share. Min-p sets a relative floor: keep token i only if p_i >= min_p_ratio * p_max, where p_max is the top token's probability and min_p_ratio is typically 0.05–0.1. Because the threshold scales with the model's own confidence, min-p stays coherent even at tau = 2+, letting you crank temperature for creativity without the garbage. Define min_p_ratio as the fraction of the peak probability a token must clear to survive.
Autoregressive models fall into loops ("the the the", or restating a sentence verbatim) because a token, once emitted, raises its own conditional probability on the next step. Repetition penalty divides the logit of any already-seen token by a coefficient gamma > 1 (often gamma^count for frequency). Define gamma as the penalty divisor. Set it too high (>1.3) and the model avoids necessary repeated words ("the", "is") and degrades fluency. Newer methods — DRY (Don't Repeat Yourself) and XTC (Exclude Top Choices) — target phrase-level repetition and "boring" top choices respectively, which a per-token penalty misses. For structured output (JSON, code) repetition penalties are dangerous: braces, commas, and indentation must repeat.
Everything above is local — it picks one token at a time. Beam search searches over sequences: keep the b highest-scoring partial sequences (by cumulative log-probability), expand each by all tokens, re-rank, keep the top b. Define b as the beam width. It beats greedy's myopia for tasks with a single correct answer (translation, ASR, constrained extraction) where the highest-likelihood sequence really is the best one.
But for open-ended generation, beam search is worse than sampling — the likelihood trap. The most probable sequence under a well-trained LM is often degenerate: empty, repetitive, or generic ("I don't know. I don't know. I don't know."), because real human text is not the mode of the distribution — it lives in the high-entropy bulk. Beam search also suffers beam collapse, where all b beams share a prefix and explore nothing; Diverse Beam Search adds an inter-beam penalty to counter this. The staff-level takeaway: choosing a decoder is choosing a search objective — maximize likelihood (beam) vs. sample from a calibrated distribution (nucleus) — and the right objective depends entirely on whether your task has one answer or many.
When output must obey a grammar — valid JSON, a regex, a SQL dialect, a function-call schema — you don't sample and pray. You mask the logits at each step to only tokens that keep the output valid under a finite-state machine or context-free grammar compiled from the schema. Libraries like Outlines, guidance, and llama.cpp's GBNF do this; OpenAI's Structured Outputs and Anthropic's tool-use schemas enforce it server-side. The win is a guarantee: the output parses, every time, with zero retries. The cost is a per-step mask computation and the subtle risk that constraining the distribution can lower quality — if you force a token the model finds unlikely, you've pushed it off-manifold, sometimes producing valid-but-wrong content. Prefer schema-guided generation for format guarantees, but keep a semantic eval (see /evals) because "parses" is not "correct."
Sampling is sequential and memory-bound: each token needs a full forward pass. Speculative decoding (Leviathan et al., 2022) uses a small cheap draft model to propose several tokens, then verifies them in a single batched forward pass of the big target model, accepting the longest correct prefix via a rejection-sampling step that provably preserves the exact target distribution — same outputs in expectation, fewer expensive passes. It is a pure latency optimization that lives in the decoding loop, which is why it belongs here conceptually but is taught in depth in /inference/speculative-decoding.
Here is a real, dependency-light decoding loop showing the full pipeline — temperature, top-k, top-p, min-p, and repetition penalty — applied to logits, then sampling. This is the same logic vLLM and TGI implement (vectorized over a batch); the structure is identical.
import torch
import torch.nn.functional as F
def decode_step(logits: torch.Tensor,
generated: list[int],
temperature: float = 1.0,
top_k: int | None = None,
top_p: float | None = None,
min_p: float | None = None,
rep_penalty: float = 1.0) -> int:
"""logits: shape (vocab_size,), raw model output for the next position.
Returns a single sampled token id. Order of ops matters."""
logits = logits.clone().float()
# 1) Repetition penalty: push down logits of already-emitted tokens.
if rep_penalty != 1.0 and generated:
seen = torch.tensor(sorted(set(generated)))
# Divide positive logits, multiply negative ones (HF convention).
sel = logits[seen]
logits[seen] = torch.where(sel > 0, sel / rep_penalty, sel * rep_penalty)
# 2) Temperature: scale before softmax. temperature==0 means greedy.
if temperature == 0.0:
return int(logits.argmax())
logits = logits / temperature
# 3) Top-k: keep only the k largest logits.
if top_k is not None:
kth = torch.topk(logits, top_k).values[-1]
logits[logits < kth] = float("-inf")
probs = F.softmax(logits, dim=-1)
# 4) Top-p (nucleus): keep smallest set whose cumprob >= p.
if top_p is not None:
sorted_p, sorted_idx = torch.sort(probs, descending=True)
cum = torch.cumsum(sorted_p, dim=-1)
# Mask everything strictly after the token that crosses p.
keep = cum - sorted_p < top_p # token that crosses p stays
remove_idx = sorted_idx[~keep]
probs[remove_idx] = 0.0
# 5) Min-p: relative floor against the peak probability.
if min_p is not None:
thresh = min_p * probs.max()
probs[probs < thresh] = 0.0
probs = probs / probs.sum() # renormalize survivors
return int(torch.multinomial(probs, num_samples=1))A few things to internalize. The order is load-bearing: penalty and temperature reshape, truncation prunes, then you renormalize over survivors. Top-p's mask keeps the token that crosses the threshold (so a single 0.95 token still survives top_p=0.9). temperature=0 short-circuits to argmax rather than dividing by zero. And multinomial is the actual weighted die-roll — replace it with argmax and the whole stochastic apparatus collapses to greedy.
| Method | Latency cost | Determinism | Quality / failure mode | What changes at scale |
|---|---|---|---|---|
Greedy (temp=0) |
Lowest | High (not bit-exact) | Bland, repetitive; deterministic for tests | Cheapest; batch-friendly |
| Temperature | Negligible | Low (>0) |
Too high -> tail garbage; too low -> bland | Per-request knob, no infra cost |
| Top-k | One topk op |
Low | Rigid set; bad when confidence varies | Cheap; superseded by top-p |
| Top-p (nucleus) | One sort | Low | The default; mild over-truncation if p low |
Sort cost trivial vs. forward pass |
| Min-p | One max + compare | Low | Keeps coherence at high temp |
Cheap; pairs with high temp |
| Repetition penalty | Gather/scatter | Low | Too high breaks the/is/JSON braces |
Needs history buffer per sequence |
Beam search (b) |
~bx memory + compute |
High | Likelihood trap, beam collapse | Expensive; rare in LLM serving |
| Constrained decoding | Per-step mask (FSM/CFG) | High | Guarantees format, can lower semantic quality | Grammar compile + per-step mask |
| Speculative decoding | Saves passes | Preserves target dist. | Net latency win; draft-model overhead | Big win at scale; see /inference |
Prose. The dominant cost in serving is the forward pass, so every sampling method here is essentially free except beam search (which multiplies memory and compute by the beam width) and constrained decoding (which adds a per-step grammar mask). That is why production LLM serving almost never uses beam search: it costs bx for outputs that are often worse on open-ended tasks. The real scale concern is determinism: temperature=0 is algorithmically greedy but still not bit-for-bit reproducible across GPU types, batch sizes, or kernel versions, because floating-point reduction order changes which of two near-tied logits wins the argmax. If you need exact reproducibility for evals or compliance, pin the model version, seed, batch size, and hardware — and even then, treat it as best-effort. The dominant failure modes in the wild: (1) high temperature without min-p/top-p -> incoherent tail tokens; (2) repetition penalty applied to structured output -> broken JSON; (3) temp=0 on a creative task -> users complain it's "robotic"; (4) constrained decoding masking the model off-manifold -> valid JSON with hallucinated field values that pass parsing but fail eval.
tau divides every logit before softmax: p_i ∝ exp(z_i / tau). Below 1 it widens the relative gaps in exponent space, sharpening toward the top token; above 1 it shrinks them, flattening toward uniform. As tau approaches 0 the top logit's exponential dominates infinitely, so the distribution collapses onto argmax — pure greedy. Because dividing by literal zero is undefined, APIs short-circuit temperature=0 directly to argmax, which is why it's the deterministic special case.p — and renormalizing, so the tail is gone before you sample. Min-p is often better at high temperature because it sets a relative floor (p_i >= min_p_ratio * p_max): the threshold scales with the model's confidence, so even at tau=2 you stay coherent while preserving real diversity. The general fix is "never raise temperature without a truncation method underneath it."min(1, p_target / p_draft) and, on rejection, resamples from an adjusted residual distribution — which provably yields samples from the exact target distribution. So you amortize multiple cheap draft steps against one expensive verification pass, cutting wall-clock latency while remaining distributionally identical to sampling from the target alone. It's a decoding-loop optimization, covered in depth in /inference.top_p reopens the tail you thought you closed. If you crank temperature, pair it with min-p, not just top-p.temp=0 is reproducible. It's deterministic in algorithm, not in bits — floating-point reduction order across hardware/batch can flip near-tied argmaxes. Pin everything if you need exact evals.bx.Flashcard. Decoding = reshape then pick: temperature/penalties/truncation reshape one logit vector, then argmax (greedy), a weighted draw (sampling), or a sequence search (beam) picks — and for open-ended text, sampling from a calibrated nucleus beats maximizing likelihood.
Next: /inference/speculative-decoding — make the decoding loop fast without changing what it emits.