A layered chain of cheap-to-expensive checks wrapped around the model — input validation, PII redaction, injection detection, Llama Guard, tool-call bounds, output filtering — that reduces risk to a number you can defend but never to zero.
ai-eng-wiki/examples/safety/guardrails.pyA guardrail stack is defense in depth wrapped around a stochastic core you do not control. You cannot patch the model's weights at request time, so you build a pipeline of independent checks — cheap deterministic ones first (regex PII, allowlists), expensive model-based ones later (Llama Guard, injection classifiers) — that each get to ALLOW, BLOCK, or REWRITE the request and the response. The mental model is a Swiss-cheese stack: every layer has holes, but the holes rarely line up, so stacking imperfect filters drives the joint failure rate down multiplicatively. The single most important architectural move is putting a bound on the model's actions (tool allowlists, confirmation gates, sandboxing) because input/output text filtering is fundamentally a soft, defeatable layer. Internalize one fact above all: guardrails reduce risk, they never eliminate it — alignment is proxy-based and every fix opens a new gaming vector, so you design assuming a layer will be bypassed.
The words first.
Step by step.
Remember this: order checks cheap-to-expensive, fail fast, fail closed, and never trust that any single layer caught everything.
You cannot design a guardrail without naming what you're guarding against. The 2025 OWASP LLM Top 10 is the canonical map: #1 is Prompt Injection, followed by Sensitive Information Disclosure, Supply Chain, Data/Model Poisoning, Improper Output Handling, Excessive Agency, System Prompt Leakage, and Vector/Embedding Weaknesses. Notice that most of these are not "the model said something toxic" — they are application-architecture failures. That reframing matters: a guardrail stack is 20% content moderation and 80% controlling the blast radius of a model you must assume can be manipulated.
The sharpest threat for agents is Simon Willison's lethal trifecta (June 2025): an agent is in danger of data theft when it simultaneously has (1) access to private data, (2) exposure to untrusted content, and (3) an exfiltration channel. Email assistants, coding agents, and Copilot-style tools routinely have all three. The insight is that you don't have to win the injection arms race if you architecturally remove one leg — deny the exfiltration path, isolate private data, or sandbox the untrusted content. Real exploits hit Microsoft 365 Copilot, ChatGPT plugins, Google Bard, and Slack precisely because all three legs were present.
Distinguish direct injection (the user's own prompt alters behavior) from indirect injection (malicious instructions ride in on a fetched web page, file, email, or another LLM's output). Indirect injection is the agentic nightmare because the attacker is not the user — a poisoned page in a RAG corpus or a booby-trapped email can hijack the model's reasoning while a benign user watches. Your guardrails must therefore moderate not just user input but every untrusted span the model ingests.
The architecture is a pipeline where each stage returns ALLOW, BLOCK, or REWRITE. Ordering is an optimization problem: put the cheapest, highest-recall checks first so you reject obvious garbage before spending a model call.
detect_pii uses Microsoft Presidio; Bedrock Guardrails identifies SSNs, card numbers, and emails natively. Regex is high-recall on structured PII but must be backed by an NER pass for names and addresses.The multiplicative intuition: if each independent layer lets through a fraction p of bad requests, two independent layers let through roughly p². A 10% individual bypass rate becomes ~1% stacked. The word independent is load-bearing — two regex filters that miss the same Base64 trick are not independent, so you deliberately mix modalities (pattern, semantic, model-based, architectural).
NVIDIA's NeMo Guardrails is the most complete open framework. It exposes five rail types — input, output, dialog, retrieval, and execution rails — and a domain-specific language called Colang for procedural conversation rules. It intercepts tool calls, converts them to resource hints, checks them against policies, and enforces decisions. Crucially, NeMo integrates Llama Guard as a drop-in input/output filter, which clarifies the relationship people confuse in interviews: Llama Guard is a single content-moderation model; NeMo Guardrails is the orchestration layer that decides when to call it and what else to run alongside (PII blocking pre-prompt, harmful/noncompliant blocking post-generation, dialog flow enforcement). One is a tool, the other is the stack that wields it.
Input/output text filtering is a soft layer that determined attackers defeat. The 2025 attack literature is sobering: FlipAttack flips character order for 81% black-box and ~98% GPT-4o success; encoding attacks hide instructions in Base64/Morse that input filters scan but decoded output reveals; multi-turn jailbreaks like ABC (98% attack success in ~10 queries) and Siren use swarm/learning-based decomposition. Most damning: early circuit-breaker defenses fail because multi-turn attacks push the model's hidden states deeper into benign regions, bypassing the harmful clusters the defense was trained on. This is the whack-a-mole dynamic — each fix opens a new gaming vector along a previously unmonitored dimension, mirroring regulatory arbitrage in finance.
The first-principles reason is that proxy-based alignment is inherently vulnerable: a filter is a proxy for "is this harmful," and once it's a target, attackers optimize against the proxy, not the true property. This is why architecture beats filtering: removing the exfiltration channel is a hard guarantee; a content classifier is a probabilistic speed bump. It's also why the durable fixes live in training — only explicit adversarial training ensures robustness (2025 finding: scaling alone does not), and mechanistic approaches like ReFAT (Refusal Feature Adversarial Training) ablate the refusal feature in the residual stream to detect attacks at the mechanism level rather than the token level.
Name the symbols: p_i is the fraction of bad requests that slip past layer i (its miss rate, or false-negative rate). If layers fail independently, the joint slip-through is the product p_1 × p_2 × ... × p_n.
Concrete example. Say a request is a genuine jailbreak attempt. Three layers:
p_1 = 0.40.p_2 = 0.20.p_3 = 0.30.Joint miss rate = 0.40 × 0.20 × 0.30 = 0.024 -> about 2.4% of jailbreaks get all the way through. One layer alone (the best, Llama Guard) leaves 20%. Stacking three independent imperfect filters cut residual risk from 20% to ~2.4%.
What it did to the data: it turned three mediocre filters into one decent system — but note the number is still 2.4%, not 0%. And the moment two layers share a blind spot (say both miss Base64), the independence assumption breaks and the real number is worse than the product. That gap between the optimistic product and reality is exactly the residual risk a staff engineer budgets for.
A staff-level answer knows the boundary. Sycophancy and factual drift are reward-model artifacts — you fix them in training (refusal calibration, better reward signals, GRPO-style verifiable rewards for verifiable domains), not with a regex. Dangerous-capability gaps are found by red-teaming (HarmBench's finding: no attack or defense is uniformly effective, robustness is independent of model size) and closed by adversarial training. And superhuman deception is a scalable-oversight problem — at a ~400 Elo capability gap, oversight success drops below 52%, and no output filter saves you from a model that out-reasons its evaluator. Guardrails are the runtime safety net; they are not where you fix the model.
The file examples/safety/guardrails.py is a dependency-light, runnable chain that wires every stage above. Each stage returns a Verdict (ALLOW/BLOCK/REWRITE/CONFIRM), and guard_request runs them in cheap-to-expensive order, failing fast and failing closed (unknown tools are denied). The model-based stages (llama_guard_classify, injection_classifier) are stubbed deterministically so it runs offline — swap their bodies for real Llama Guard / Bedrock / Presidio calls.
def guard_request(user_input, llm, confirm_cb=None, proposed_tool=None):
trace = []
# Stage 1 — deterministic input (cheapest first): redact PII, don't block.
clean, pii = redact_pii(user_input)
if pii:
trace.append(Verdict(Action.REWRITE, "input_pii", text=clean))
user_input = clean
inj = injection_classifier(user_input) # heuristic + classifier
trace.append(inj)
if inj.action == Action.BLOCK:
return GuardrailResult(False, None, trace, blocked_by=inj.stage)
# Stage 2 — model-based input safety (Llama Guard).
lg_in = llama_guard_classify(user_input, role="user")
trace.append(lg_in)
if lg_in.action == Action.BLOCK:
return GuardrailResult(False, None, trace, blocked_by=lg_in.stage)
# Stage 3 — tool-call bounds: allowlist + confirmation on exfiltrating actions.
if proposed_tool is not None:
tv = check_tool_call(proposed_tool)
trace.append(tv)
if tv.action == Action.BLOCK:
return GuardrailResult(False, None, trace, blocked_by=tv.stage)
if tv.action == Action.CONFIRM:
if not (confirm_cb and confirm_cb(proposed_tool)):
return GuardrailResult(False, None, trace, blocked_by="tool_confirm_denied")
raw = llm(user_input) # the expensive call
# Stage 4 — output filtering: re-run PII + safety on what the model emitted.
out = filter_output(raw)
trace.append(out)
if out.action == Action.BLOCK:
return GuardrailResult(False, None, trace, blocked_by=out.stage)
return GuardrailResult(True, out.text, trace)Three design choices are worth defending in an interview. First, PII is a REWRITE not a BLOCK — redact_pii substitutes <EMAIL_REDACTED> so the request still works; blocking on any email would make the assistant useless. Second, the tool gate is the trifecta-breaker: send_email, execute_shell, and transfer_funds require confirmation, and the allowlist fails closed so a hallucinated rm_rf tool is denied by default, not by enumeration of bad tools. Third, the output is re-filtered — the same PII and Llama Guard checks run on the model's response, because the model can emit memorized secrets or reconstruct redacted data. Running it prints a full verdict trace per case (benign allowed with email redacted, injection blocked, weapon request blocked, tool confirm denied, unknown tool blocked) so you have an audit log — the raw material for measuring your bypass rate.
| Layer | Added latency | Cost driver | Catches | Failure mode |
|---|---|---|---|---|
| Regex PII / injection heuristics | ~µs–low ms | CPU only | Structured PII, lazy attacks | Misses NER-only PII, any obfuscation (Base64, FlipAttack) |
| Presidio / Bedrock PII | ~10–50 ms | CPU/managed | Names, addresses, structured PII | Over-redaction kills utility; novel formats slip |
| Llama Guard (input + output) | ~tens–hundreds ms each | GPU inference (doubles model calls) | 6 harm categories | Trained clusters bypassed by multi-turn / encoded attacks |
| Injection classifier | ~tens ms | GPU/CPU | Direct + indirect injection | Whack-a-mole; new vectors per fix |
| Tool bounds + confirmation | ~µs (+ human time) | Eng + UX friction | Excessive agency, exfiltration | Confirmation fatigue → users rubber-stamp |
| Output grounding (RAG) | <200 ms (hybrid) | Retrieval + compute | Hallucination (~97% detect) | Unfaithful-but-plausible claims; entity ambiguity |
The dominant cost is that input + output Llama Guard roughly triples your inference calls (input classify, generate, output classify), and a serial chain adds its latencies. Mitigations: run independent classifiers in parallel, cache verdicts for repeated inputs, and use a small fast guard model rather than a frontier model as judge. At scale, the failure modes shift — confirmation fatigue is the silent killer: gate too many tools and users reflexively approve, defeating the gate, so you reserve confirmation for genuinely irreversible/exfiltrating actions and let the allowlist handle the rest.
The deepest tradeoff is the alignment tax: stronger refusal and filters reduce capability and raise over-refusal on benign requests. Refusal calibration is the explicit objective of balancing safety against helpfulness — testing refusal consistency across request variants so the model refuses the harmful ones without nuking the benign ones. Tune your stack on a held-out set of both attacks and benign-but-edgy requests, and report two numbers: attack-bypass rate and benign-over-refusal rate. Optimizing one without the other is how you ship a system that's either dangerous or unusable. For regulated deployments, note the EU AI Act (enforcement August 2026, penalties up to EUR 35M or 7% of global revenue) — your guardrail trace is your compliance evidence, which is another reason every verdict must be logged.
send_email and any outbound HTTP behind a confirmation step (or an allowlist of recipients), isolate the private inbox from the span of context that processed untrusted content, and sandbox tool execution. The tool-bounds layer is doing the real work here; the content filters are just reducing how often that gate gets stress-tested.p_i; under independence the joint bypass is the product, so three layers at 0.4/0.2/0.3 give ~2.4% residual — non-zero, and worse once layers share blind spots (correlated misses on Base64, multi-turn state-shifting that pushes hidden states into benign regions). So you budget for bypass: design for blast-radius minimization, not prevention. Remove exfiltration paths so a bypassed content filter can't cause data theft; sandbox execution with network allowlists and resource caps so a jailbroken code agent can't reach the host; require confirmation on irreversible actions so a single bad model output isn't catastrophic; and log everything to detect bypasses post-hoc. The staff move is recognizing that some risk classes (sycophancy, deception, dangerous capability) don't belong in runtime guardrails at all — they're training problems (adversarial training, refusal calibration, ReFAT) or oversight problems, and pretending a filter fixes them is the actual failure.Flashcard. Guardrails are cheap-to-expensive, fail-fast, fail-closed, defense-in-depth — and the residual risk is the product of layer miss-rates, never zero; the only hard guarantees come from architecture (remove the exfiltration path), not from any filter.
Next: /safety/red-teaming-and-dangerous-capability-evals — how you measure the residual risk your stack leaves behind.