Two knobs on the decoder — a grammar that constrains the *shape* of every token, and a thinking budget that buys *quality* with latency — and the engineering judgment to know which one the problem actually needs.
There are two independent knobs sitting on the autoregressive decoder, and most engineers conflate them. The first is shape: at every generation step, a constrained decoder masks out tokens that would violate a grammar (JSON schema, regex, context-free grammar), so the output is syntactically guaranteed to parse. The second is depth: a reasoning model spends extra tokens thinking before it answers, trading latency and cost for accuracy on problems that genuinely need search. Shape is a decode-time filter — it changes which tokens are legal, not how smart the model is. Depth is a compute allocation — it changes how much the model deliberates, not whether the output parses. The senior skill is knowing that constraining shape can hurt quality if the schema fights the model's natural reasoning order, and that buying depth is pure waste on problems the model already one-shots.
response_format param"? Can you name the cost of a thinking budget and read a quality-vs-tokens curve? Do you know that "100% valid JSON" and "100% correct JSON" are different claims?The words first.
Step by step.
Remember this: constrained decoding controls the shape of the answer; a thinking budget controls the depth — they are different knobs and you tune them independently.
An autoregressive LLM generates one token at a time. At step t it produces a logit vector z over the whole vocabulary (≈100K+ tokens), and softmax(z) gives the probability of each next token. Constrained decoding intervenes between the logits and the softmax: a grammar engine tracks the current parse state and computes, for that state, the set of tokens that would keep the output valid. Every token not in that set has its logit set to -∞, so after softmax its probability is exactly 0. The model can only sample something legal.
The grammar engine is the interesting part. For a JSON schema, you compile the schema into a state machine. Simple shapes (a flat object with known keys) are a finite-state machine (FSM) — Outlines builds a regex-derived FSM. Nested/recursive structures (arbitrarily deep arrays, balanced braces) need a pushdown automaton because they require a stack to track nesting depth — this is what XGrammar does for full context-free grammars, and it's why XGrammar can handle grammars that a pure regex FSM cannot. At each step the engine maps "current state" → "allowed token set," which is precomputed and cached so the per-token overhead is small (a mask lookup, not a re-parse).
The guarantee is syntactic validity, not semantic correctness. Constrained decoding promises the output parses against the schema. It says nothing about whether the values are right. You can get a perfectly valid {"sentiment": "positive"} on a negative review — 100% schema-valid, 100% wrong. Internalize that gap; it drives the whole quality discussion in §5.
There's a subtler cost: naive masking distorts the probability distribution. When you zero out tokens and renormalize, you've changed the conditional distribution the model was trained to produce — you're sampling from a truncated, reweighted distribution that may not reflect the model's true belief. Most of the time this is harmless, but in fact-dense or ambiguous fields it can push the model toward a token it didn't actually "want." This is what methods like (G)I-DLE address: they minimize KL divergence between the constrained and unconstrained distributions, excluding banned tokens while perturbing the surviving probabilities as little as possible.
Symbols: z is the logit (raw score) per candidate token; p is the probability after softmax. The grammar's job is to decide which tokens are legal right now.
Suppose we're decoding a JSON value for a field whose schema says it must be the string "yes" or "no". The model is mid-string and just emitted "y. Legal next tokens per the grammar: only es". Illegal: everything else.
Raw logits the model produced for four candidates:
es" → z = 2.0ep" → z = 2.4 (model actually prefers this — "yep") (space) → z = 1.1123 → z = 0.5Unconstrained softmax would pick ep" (highest logit). But the grammar only allows es", so we set the other three logits to -∞:
es" → z = 2.0 → p ≈ 1.00z = -∞ → p = 0.00Output: es", completing "yes". What it did to the data: it overrode the model's top choice (ep") to keep the output schema-valid — buying a guaranteed parse at the cost of the model's preferred (here, junk) token. That override is the whole tradeoff in miniature.
Constrained decoding isn't the only path to structured output. Tool/function calling gives the model a tool whose input is a JSON schema, and the model fills it in. On frontier APIs (Anthropic, OpenAI) this is often the preferred way to get structured data, because the model was post-trained heavily on tool-shaped JSON — gpt-4o-2024-08-06 was trained for 100% reliability on complex JSON schemas, and Claude's tool-use path is similarly hardened. The practical difference: pure constrained decoding (vLLM + Outlines/XGrammar) is enforced by the runtime and works on any open model; tool-use schema conformance is trained-in and leans on the model's own competence, with the API layer adding validation. Direct prompting ("respond in JSON") sits at the bottom — roughly 70% reliable in the wild, fine for prototypes, unacceptable for a pipeline that parses the output downstream.
CoT started as a prompting trick: ask the model to "think step by step" and accuracy on multi-step problems jumps, because the intermediate tokens give the model scratch space to externalize computation it can't do in a single forward pass. The 2024–2025 shift was to bake reasoning into the weights. o1 (OpenAI, Sept 2024) is RL-trained to produce long internal chains-of-thought before answering. DeepSeek-R1 (Jan 2025) matched o1-class performance at a fraction of the cost using GRPO (group relative policy optimization) and — crucially — released the recipe openly, showing that long-CoT reasoning emerges from RL on verifiable rewards without needing a separate value network. o3 (early 2025) added adaptive compute: the model decides how much inference budget to spend based on perceived difficulty.
The mental model: test-time compute is a third scaling axis alongside training data and model size. Instead of a bigger model, you let a fixed model spend "orders of magnitude more tokens" deliberating. The scaling curve is real but bends — quality rises with the thinking budget and then hits diminishing returns above roughly 10–32K thinking tokens for most tasks. Past that, you're paying linearly more for marginal (or negative) quality.
How do you make those extra tokens count? Three verification strategies: outcome reward models judge only the final answer; process reward models (PRMs) score each reasoning step, catching a wrong turn early; best-of-N samples N independent reasoning paths and picks the best (by a verifier or by majority vote). Best-of-N is "parallel" test-time compute; a long single chain is "sequential." They trade differently — N short chains parallelize across hardware; one long chain has lower total token count but can't be sped up by adding GPUs.
Frontier APIs expose the budget directly. Anthropic's extended thinking emits thinking content blocks (with encrypted signatures for multi-turn continuity) and you set budget_tokens — e.g. 10K tokens of reasoning headroom, though the model may use less. You are billed for every thinking token regardless of whether you display it (summarized shows reasoning with minimal latency; omitted gives faster time-to-first-text but still bills the hidden thinking). On Opus 4.5+ thinking tokens don't consume the context window; on earlier models they did.
The 2026 evolution is adaptive thinking (Claude 4.6+): instead of a fixed budget you set an effort level — low, medium, high (default), xhigh, max — and the model decides how much to actually think, with effort as a soft preference. For Opus 4.7+ this type: "adaptive" form replaces the legacy fixed-budget type: "enabled". And interleaved thinking (Opus 4.8+) lets the model reason between tool calls — think, call pricing service, think about the result, query a database, think, then answer — with a total thinking budget that can exceed max_tokens because it's pooled across blocks. That's the bridge from "reasoning model" to "reasoning agent."
Two reference implementations: enforced constrained decoding on an open model, and an adaptive thinking budget on a frontier API.
# A. Enforced JSON via constrained decoding (vLLM + a JSON schema).
# The runtime MASKS illegal tokens — validity is guaranteed by construction.
from vllm import LLM, SamplingParams
from vllm.sampling_params import GuidedDecodingParams
schema = {
"type": "object",
"properties": {
"sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
"confidence": {"type": "number", "minimum": 0, "maximum": 1},
"evidence": {"type": "string"},
},
"required": ["sentiment", "confidence", "evidence"],
"additionalProperties": False,
}
llm = LLM(model="Qwen/Qwen3-8B")
# XGrammar backend -> pushdown automaton, handles nested schemas.
guided = GuidedDecodingParams(json=schema, backend="xgrammar")
params = SamplingParams(temperature=0.0, max_tokens=256, guided_decoding=guided)
review = "Shipped late and the API kept 500ing, but support was responsive."
out = llm.generate(f"Classify this review as JSON:\n{review}", params)
print(out[0].outputs[0].text) # ALWAYS parses against `schema`The grammar engine compiles schema into an automaton once, then per token it intersects "vocabulary" with "tokens legal in the current state" and applies the mask. temperature=0.0 plus the mask makes this fully deterministic. Note "evidence" is before the model is forced to commit — but it comes after sentiment in key order, which matters in §5.
# B. Adaptive thinking budget on a frontier model (Anthropic).
# Always read the live API skill/docs before shipping — params drift.
import anthropic
client = anthropic.Anthropic()
resp = client.messages.create(
model="claude-opus-4-8",
max_tokens=2048,
thinking={"type": "adaptive", "effort": "medium"}, # low|medium|high|xhigh|max
messages=[{
"role": "user",
"content": "A train leaves A at 60mph, another leaves B (180mi away) "
"at 40mph toward it. When do they meet? Show reasoning, "
"then give the answer.",
}],
)
for block in resp.content:
if block.type == "thinking":
print("[thinking tokens billed but here summarized]")
elif block.type == "text":
print(block.text)
print("usage:", resp.usage) # input/output tokens — thinking is billedeffort: "medium" is a soft preference; the model spends more on a genuinely hard step and less on an easy one. You pay for usage thinking tokens either way, so the eval question is whether medium beats low on your task by enough to justify the cost — measure, don't assume. Before relying on exact parameter names, model IDs, or pricing, consult the live claude-api reference — these change.
| Technique | Cost | Latency | Quality effect | Primary failure mode |
|---|---|---|---|---|
| Direct "respond in JSON" prompt | Baseline | Baseline | ~70% parse rate | Silent malformed JSON breaks the parser downstream |
| Constrained decoding (Outlines/XGrammar) | Negligible compute overhead | +small per-token mask cost | 100% valid, can lower accuracy | Schema-valid-but-wrong; distribution distortion |
| Tool-use schema (frontier API) | Standard token price | Baseline | High validity + trained correctness | Model declines / hallucinates a field value |
| Extended thinking (fixed budget) | + every thinking token | + thinking time (can be seconds) | +accuracy on hard reasoning | Overthinking easy tasks; wasted budget |
| Adaptive thinking (effort levels) | Variable, model-decided | Variable | Best cost/quality if calibrated | Effort misjudged on out-of-distribution inputs |
| Best-of-N reasoning | N× generation cost | Parallelizable | +accuracy, needs a verifier | Verifier becomes the bottleneck/oracle |
The structure-vs-quality trap is the one that gets shipped broken. Constraining shape can degrade accuracy through two mechanisms. First, distribution distortion (§3.1): masking + renormalize samples from a perturbed distribution. Second, and more important in practice, ordering: if your schema forces the model to emit the answer field first and the reasoning field last, you've effectively banned chain-of-thought — the model must commit to sentiment before it writes evidence, so the evidence is post-hoc rationalization of an answer it already locked in. The fix that keeps the guarantee: put a reasoning or evidence field first in key order, then the answer field. The grammar still enforces shape, but now the model reasons into the JSON before committing. This single reordering often recovers most of the accuracy gap versus free-form CoT.
Reasoning is not free quality. The diminishing-returns elbow (~10–32K thinking tokens) means a fixed budget_tokens=32000 on a classification task is mostly waste — you pay 32K tokens of latency for a task that needed 200. At fleet scale this dominates your bill. The discipline: default to no thinking (or low/adaptive), and promote to high effort only for task slices where an eval shows the lift. Reserve big budgets for verifiable, search-heavy problems (math, code, multi-hop planning) where the scaling curve actually pays.
What changes at scale. (1) Caching interacts with thinking — preserved thinking blocks let you hit the prompt cache on the next turn, but clearing thinking invalidates it; in long agent loops this is a real cost lever. (2) Constrained decoding + batching — grammar masks must be computed per-sequence, which complicates high-throughput batching; XGrammar's precomputed transition caches matter here. (3) Overthinking is a tail-latency problem — a model that occasionally burns its full budget blows your p99, so cap budgets even on reasoning slices. (4) Schema-valid-but-wrong is invisible to your monitoring unless your eval checks values, not just parses — a green "100% valid JSON" dashboard can sit on top of a 60%-correct pipeline.
-∞, so after softmax their probability is exactly zero — the model can only sample legal tokens, so the result always parses. The costs: a small per-token masking overhead, and — more importantly — it guarantees only syntactic validity, not correctness. It can even lower accuracy by distorting the sampling distribution and, if the schema forces the answer before any reasoning field, by suppressing chain-of-thought.medium/adaptive setting may capture most of the lift at half the tokens. Finally I'd consider a router — cheap model first, escalate to the reasoner only on low-confidence cases — which usually beats committing the whole fleet to either model.reasoning/evidence field comes first and the answer field comes after — the grammar still enforces shape, but the model now reasons into the JSON before committing. If distortion still bites in fact-dense fields, a KL-preserving masking method like (G)I-DLE perturbs surviving probabilities less.omitted/summarized display modes still charge every thinking token — display setting changes latency, not cost.Flashcard. Constrained decoding controls the shape of the output (logit masking → guaranteed parse, not guaranteed truth); a thinking budget controls the depth (test-time compute → better accuracy with diminishing returns past ~10–32K tokens). Tune them independently, and never let an answer-first schema silence the model's reasoning.
type: "adaptive" migration.Next: /context-engineering/memory-and-compaction — what to do when the answer plus its reasoning no longer fit in the window.