The unglamorous machinery — constrained decoding, relevance classifiers, and input/output guards — that turns a pile of LLM calls into a workflow you can put on-call for.
A workflow is a flowchart you could draw before runtime: fixed boxes, fixed arrows, LLM calls and tools wired together by code you control. The thing that makes those boxes connect reliably is not the prompts — it's three pieces of glue. Structured outputs turn free-text model responses into typed data your code can branch on without a regex. Routing classifiers decide which box runs next (and whether the input even belongs in your system). Guardrails are validation steps — wrapped around the call, not buried in the prompt — that catch bad input before it costs you a tool call and bad output before it reaches a user. Get this glue right and an unreliable component (the LLM) becomes a reliable subsystem; get it wrong and your "workflow" is just a pile of try/except and prayer.
json.loads(response.strip("```json")), that you validate at the boundary, and that you have a fallback when validation fails. The signal is: do you treat the model as a flaky network dependency or as an oracle?The words first.
Step by step.
Remember this: structured outputs make the connection typed, classifiers make it routed, guards make it safe — and every one of them is a workflow step you can test before you ship.
The naive way to get JSON from an LLM is to ask for it in the prompt and parse the string. This fails in production for boring, frequent reasons: the model wraps the JSON in a ```json fence, adds a "Here's the result:" preamble, hallucinates a field your code doesn't expect, emits a trailing comma, or — under load — truncates mid-object. Each is a different exception class, and you'll write a different hack for each.
Constrained decoding removes the entire failure class. The provider takes your JSON Schema, compiles it to a grammar, and at each decoding step masks the logits of any token that would make the output invalid against that grammar. The model literally cannot emit a closing brace where the schema demands more required fields, or a string where it demands an integer. Claude's Structured Outputs (GA Nov 2025+) does exactly this — you pass a JSON Schema with the request and get 99.9%+ schema-valid output, working even with streaming. The older, still-robust pattern is tool use with tool_choice forced: you define a tool whose input schema is your target shape and force the model to "call" it; the tool arguments come back schema-valid.
Why this matters beyond convenience: the schema becomes a contract between workflow boxes. Box A's output schema is box B's input schema. You can unit-test that contract without the model in the loop. You eliminate an entire category of runtime branches ("what if the field is missing"). And critically, an enum in your schema is a guarantee, not a hope — if your router must output "pricing" | "refund" | "returns", constrained decoding makes it impossible to get "refunds" or "pricing-related" back.
Name the symbols: the model produces a probability distribution over the vocabulary at each step; logit[t] is the raw score for token t before softmax. A schema constraint is a mask that sets disallowed tokens to negative infinity so their probability becomes ~0.
Concrete example. Router schema says the category field is enum: ["pricing","refund","returns"]. The model has just emitted {"category": " and now picks the next token. Suppose the unconstrained top tokens are:
pric → logit 8.1ref → logit 7.6bill → logit 7.2 (model "wants" to say "billing", which is NOT in the enum)return → logit 6.9Without constraints, softmax over these gives bill ≈ 18% probability — and "billing" is a category your code has no handler for. Crash, or silent misroute.
With constrained decoding, the grammar knows that after "category": " only tokens that can start a valid enum value are legal. bill cannot begin any of pricing/refund/returns, so its logit is set to −inf → probability 0. The renormalized distribution is now only over pric, ref, return. The model picks pric, then the grammar forces the remaining tokens to complete pricing exactly.
What it did to the data: turned a 3-way "hope" into a guaranteed member of a closed set. Your downstream match category is now total — no default: raise.
A caveat worth stating honestly: constrained decoding guarantees the shape is valid, never that the content is correct. A schema-valid {"refund_amount": 999999} is still wrong. Constraints kill parse errors and type errors; they do nothing for hallucinated values. That's what output guards are for (3.3).
Routing is the workflow pattern with the best cost/quality ratio in the entire Anthropic taxonomy. You spend one cheap classification call upstream to pick a specialized handler, and every downstream prompt gets to be focused and short instead of a 4,000-token mega-prompt trying to handle pricing, refunds, and returns simultaneously. Specialized handlers are more accurate and cheaper per call, and you can route simple categories to a small model (Haiku) and only hard ones to a frontier model.
Two distinct jobs hide under "routing," and conflating them is a common mistake:
Run relevance first, dispatch second. The blast radius of a routing error is the whole point: a misroute doesn't error — it silently runs the wrong correct-looking handler and returns a confident wrong answer. That's worse than a crash, because nothing pages you. So routers need their own eval set and their own confidence threshold. A strong production pattern: have the router emit not just a label but a confidence and a reasoning field (via structured output), and when confidence is below threshold, fall through to a more capable model or a human queue rather than committing to a guess.
The single biggest mindset shift here: a guardrail is code that wraps the model call, not a sentence inside the prompt. "Do not reveal PII" in your system prompt is a request; a regex-and-classifier pass over the output is an enforcement. Interviewers love watching candidates discover this distinction live.
The canonical architecture is a sandwich:
input guardrails → LLM call → output guardrailsInput guardrails (cheapest first): rule-based filters run in ~1ms — length caps, banned-keyword regex, schema validation of the incoming request, prompt-injection signatures. Survivors hit ML classifiers for the expensive checks: relevance (scope), safety (toxicity, jailbreak attempts), PII detection. The ordering is an explicit cost optimization — you never pay for a 50ms ML classifier on a request a 1ms regex would have killed.
Output guardrails: structured-schema validators (does the output satisfy business invariants, not just types — is refund_amount within policy?) plus safety classifiers on the generated text (did the model leak PII, make a hallucinated claim, violate policy?). Frameworks like Guardrails AI let you declare these with RAIL (an XML-based markup for format/type/validation rules); on failure you get automatic corrective actions — reask the model, filter the offending span, or repair the output to satisfy the schema.
The part juniors skip: what happens on failure. A guard that only detects is half a guard. Production guards have an explicit fallback ladder:
Pick deliberately per guard. A schema-repair on a malformed JSON field is fine to auto-fix; a safety violation should never be auto-repaired and re-shown — it should hard-stop to a safe default. The Anthropic framing of "evaluator-optimizer" is the same loop dressed up: generator produces, evaluator (a guard) critiques against criteria, generator refines — bounded, because loop depth is unpredictable and unbounded loops are how you get a $400 single request.
This whole lesson is about deterministic glue, and the honest staff-level answer is that you should keep as much of the system deterministic as you can. The Anthropic guidance is explicit: start with workflows, and most production systems shouldn't need more autonomy than the parallelization pattern. You add agentic autonomy only in bounded scopes where the solution path genuinely depends on runtime discoveries — open-ended research, multi-step debugging.
The mature architecture is hybrid: a deterministic supervisor (your router + guards) acts as the entry point and the safety envelope, and hands off to an autonomous sub-agent only inside a sandbox with the guards still wrapping its inputs and outputs. The handoff is a structured output: the supervisor produces a typed task spec (goal, constraints, token budget, allowed tools) that the agent consumes. And the agent's results pass back through the same output guardrails before re-entering deterministic flow. The structured-output contract and the guardrail sandwich are exactly what make a bounded agent safe to embed — without them, "add an agent here" means "add an unbounded, unauditable, non-deterministic hole in my workflow." Why it matters at scale: a 10-step agentic process at 99% per-step success compounds to ~90% overall; the guards and typed contracts are what stop that 10% from reaching production untouched.
A routing-then-handler workflow with a relevance guard, forced structured output, and an output guard with a bounded re-ask. This is the production shape: every model call is wrapped in validation, and failures have explicit fallbacks. (Uses the Anthropic SDK and a tool_choice-forced tool for schema-valid output; the same pattern works with native Structured Outputs.)
import json
from anthropic import Anthropic
client = Anthropic()
MODEL_FAST = "claude-haiku-4-5" # cheap router/guard
MODEL_MAIN = "claude-sonnet-4-5" # handler
# --- Structured output via a forced tool (schema = contract) ---
ROUTE_TOOL = {
"name": "route",
"description": "Classify a support request.",
"input_schema": {
"type": "object",
"properties": {
"category": {"type": "string", "enum": ["pricing", "refund", "returns"]},
"in_scope": {"type": "boolean"},
"confidence": {"type": "number"}, # 0..1
},
"required": ["category", "in_scope", "confidence"],
},
}
def _forced_tool_call(model, system, user, tool):
"""One call, output guaranteed valid against tool['input_schema']."""
msg = client.messages.create(
model=model, max_tokens=512, system=system,
tools=[tool],
tool_choice={"type": "tool", "name": tool["name"]}, # force schema-valid output
messages=[{"role": "user", "content": user}],
)
for block in msg.content:
if block.type == "tool_use":
return block.input # already parsed, already schema-valid
raise RuntimeError("model did not emit tool_use") # should be impossible when forced
# --- Input guard + dispatch routing in one structured call ---
def route(request: str) -> dict:
return _forced_tool_call(
MODEL_FAST,
"You are a router. Decide scope, category, and your confidence.",
request, ROUTE_TOOL,
)
# --- Per-handler schema; the handler's output is also typed ---
ANSWER_TOOL = {
"name": "answer",
"input_schema": {
"type": "object",
"properties": {
"reply": {"type": "string"},
"refund_amount": {"type": "number"}, # 0 unless a refund
},
"required": ["reply", "refund_amount"],
},
}
REFUND_CAP = 500.0
def output_guard(out: dict) -> tuple[bool, str]:
"""Business-rule validation. Returns (ok, error_for_reask)."""
if out["refund_amount"] < 0:
return False, "refund_amount must be >= 0"
if out["refund_amount"] > REFUND_CAP:
return False, f"refund_amount exceeds policy cap of {REFUND_CAP}"
return True, ""
def handle(category: str, request: str, max_reask: int = 1) -> dict:
system = f"You handle {category} requests. Set refund_amount=0 unless issuing a refund."
user = request
for attempt in range(max_reask + 1):
out = _forced_tool_call(MODEL_MAIN, system, user, ANSWER_TOOL)
ok, err = output_guard(out)
if ok:
return out
# bounded re-ask: feed the validation error back, regenerate once
user = f"{request}\n\n[Your previous answer was rejected: {err}. Fix it.]"
# fallback ladder: safe default + human handoff, never a wrong refund
return {"reply": "Let me connect you to a specialist.", "refund_amount": 0,
"_escalate": True}
def workflow(request: str) -> dict:
r = route(request)
if not r["in_scope"]:
return {"reply": "That's outside what I can help with here.", "_rejected": True}
if r["confidence"] < 0.6: # low-confidence -> human, don't guess
return {"reply": "Let me get a specialist for you.", "_escalate": True}
return handle(r["category"], request)
if __name__ == "__main__":
print(json.dumps(workflow("I returned a jacket 3 weeks ago, where's my $80?"), indent=2))What each piece earns its keep: tool_choice forcing means route() and handle() cannot return malformed JSON or an off-enum category — the parse step is gone. The confidence field lets the router abstain instead of guessing, converting silent misroutes into explicit escalations. output_guard enforces a business invariant (the refund cap) that constrained decoding can't — schema says "a number," policy says "≤ 500." The bounded re-ask gives the model exactly one chance to self-correct against the actual error message before the fallback ladder takes over with a safe default. Every branch here is testable without the model: feed output_guard a {"refund_amount": 999} and assert it rejects.
| Stage | Typical latency | Relative cost | Buys you | Failure mode if skipped |
|---|---|---|---|---|
| Rule-based input filter (regex/length/schema) | ~1 ms | ~$0 | Drops obvious junk before any model call | Pay model cost on garbage; injection slips through |
| Relevance/safety classifier (small model) | 50–300 ms | 1× small-model call | Scope + safety gate; cuts token spend & error propagation | Out-of-scope & harmful inputs reach handlers |
| Router (structured, small model) | 100–400 ms | 1× small-model call | Specialized, cheaper, more accurate handlers downstream | One 4k-token mega-prompt; lower quality, higher cost |
| Handler w/ constrained decoding | 0.5–4 s | 1× main-model call | Typed output, zero parse errors, contract between boxes | Regex parsing; ~0.1–1% malformed-output failures |
| Output guard + bounded re-ask | +0–1 extra call | 0–1× extra | Catches policy/safety/business-rule violations | Wrong-but-valid answers reach users; no audit trail |
Cost. The whole architecture is a cost optimizer, not a tax. Routing to Haiku for simple categories and reserving Sonnet for hard ones, plus killing junk with a 1ms filter, typically nets lower cost than one big model handling everything — while raising quality. The expensive failure is the unbounded re-ask loop: cap retries at 1–2, always. Workflows have a fixed, budgetable token cost per run; the moment you let any step loop without a bound you've reintroduced the 4–15× agent token spike you were trying to avoid.
Latency. Guards add real wall-clock time. The mitigation is ordering (cheap-first) and parallelism: independent input guards (relevance, PII, safety) can run concurrently and you fail fast on the first rejection. Don't serialize what you can fan out.
Quality / failure modes. Constrained decoding eliminates parse and type errors but not hallucinated values — output guards on business invariants are non-negotiable. Routers fail silently (confident wrong answer, no exception), so they need their own eval set and a confidence-abstain path. Safety guard failures are asymmetric: a false negative (harmful output shipped) is far costlier than a false positive (over-blocking), so tune thresholds accordingly and never auto-repair-and-reship a safety failure — hard-stop to a safe default.
What changes at scale. At 1,000s of runs/day you need three things the demo didn't: (1) structured logging — every prompt, output, model ID, token count, and guard verdict as machine-readable spans (OpenTelemetry), so you can query "all low-confidence routes last week"; (2) annotation queues — 89% of orgs have agent observability but only ~52% have a real evaluation loop; traces are worthless if no domain expert ever judges correctness, so route a sample of production traces to humans and convert their feedback into eval datasets; (3) durable execution — if a guard escalates to a human, the workflow may pause for hours, so state has to survive restarts (LangGraph checkpointing to Postgres/Redis, or Temporal event-history replay). InMemory defaults lose every paused workflow on the next deploy.
enum fields a guarantee: a router constrained to ["pricing","refund","returns"] cannot return "billing", so your downstream branch is total. The one thing it does not do is guarantee correctness — a schema-valid wrong value still needs an output guard.confidence and reasoning via structured output, and below a threshold abstain — fall through to a stronger model or a human queue instead of committing. Second, build a router eval set from logged traces (route + the human-judged correct route) so you can measure the 6% per-category and see which confusions dominate. Third, attack the top confusion pairs with better handler descriptions or a two-stage route (coarse then fine) for the ambiguous cluster, and consider a relevance guard upstream if some of the 6% is actually out-of-scope traffic. You've now converted silent misroutes into measurable, mostly-escalated events — and you did it with thresholds, evals, and decomposition, no training run.Flashcard. Structured outputs make the wire typed, classifiers make it routed, guards make it safe — and all three are workflow steps you can test before you ship; the prompt is not where reliability lives.
Next: Workflows vs. Agents — when to give up control · See also Evals for turning guard verdicts into datasets and Safety for classifier design.