The model is a stateless next-token oracle; the harness is the control plane that turns one good thought into a hundred reliable actions — and it's where production agents actually die.
The model is a stateless function: tokens in, a probability distribution over the next token out. It has no memory, no ability to act, and no notion of "the task" beyond what is currently in its context window. Everything that makes an agent reliable — remembering intent across hours, calling the right tool, noticing it was wrong, recovering without a human — lives in the harness: the runtime software wrapped around the model. Field data from 2025–2026 deployments attributes roughly 65% of enterprise agent failures to harness defects (context drift, schema misalignment, state degradation) rather than to the model's raw capability. The mental model to carry into any interview: a frontier model is a brilliant amnesiac contractor, and the harness is the project manager, the filing cabinet, and the QA gate that turns brilliance into shipped work.
The harness is the cleanest signal of systems maturity in an agent engineer, because it separates people who have only prompted from people who have operated agents at scale.
The words first.
Step by step.
Remember this: the model thinks; the harness remembers, checks, and acts — and that's where reliability is won or lost.
Start from what a transformer actually is. Attention assigns a softmax-weighted score to every token in the window when predicting the next one. As the window grows, the probability mass available to any single relevant token shrinks — the denominator of the softmax includes thousands of irrelevant tokens (old logs, stale diffs, dead-end tool calls). This is the mechanical basis of context rot: it is not that the model "forgets," it is that signal is drowned in accumulated noise. Empirically, the accumulating layer — raw tool outputs piled into the window — degrades reasoning quality gradually, so it is rarely caught until a run visibly goes off the rails.
Two siblings follow from the same root cause:
The harness exists to counter all three. None of these are model-quality problems; a smarter model rots more slowly but still rots.
The harness manages the window as a three-layer system:
A robust long-horizon pattern is the "Ralph Loop": periodically reinject the original intent into a clean window, rehydrating from external memory rather than from the polluted running transcript. You pay a re-read cost; you buy back attention and kill drift.
The single highest-leverage harness pattern: make a durable file, not the context window, the source of truth. A repo-local AGENTS.md (canonical instructions, conventions, gotchas) plus a running todo log (what's done, what's next, what was tried and failed) gives the agent a system of record that survives any window reset. When the window is compacted or the Ralph Loop fires, state is reconstructed from the file — the file is authoritative, the window is a cache.
This inverts the naive design where state lives implicitly in the transcript and dies whenever you summarize. Explicit read-write interfaces to memory let you version state, diff it, audit it, and hand it to a fresh agent instance. The same brevity discipline applies as to tools: a tight, curated AGENTS.md beats a sprawling one, for the same softmax-dilution reason.
The structural fix for premature victory and compounding error is Plan-Execute-Verify (PEV) with a model cascade:
[0.0, 1.0] and run deterministic recovery logic.Sub-threshold confidence triggers a retry that injects the error feedback into the next attempt; after exhausting retries, the harness escalates to full replanning with the failure as context. Every attempt is preserved in an audit trail, which is what prevents a single wrong step from silently propagating. The economic punchline: routing only the executor through the capable model yields roughly 60–70% per-run cost savings versus all-capable routing — a typical three-step task runs about $0.01 under PEV vs ~$0.027 all-Sonnet — while raising reliability, because the cheap validator catches what the executor would have shipped blind.
Name the symbols in plain words: p is the probability one step succeeds; n is the number of steps; end-to-end success of a naive chain is p multiplied by itself n times, written p**n.
Work it with real numbers. Say each step is 85% reliable (p = 0.85) and the task has 10 steps (n = 10). Naive end-to-end success is 0.85**10. Compute it: 0.85**2 = 0.7225, 0.85**5 ≈ 0.4437, 0.85**10 ≈ 0.4437 * 0.4437 ≈ 0.197. So about 0.20 — roughly 80% of runs fail, even though every individual step is "pretty good." That is the compounding tax, and it is why "the model is 85% accurate" tells you almost nothing about a 10-step agent.
Now add a verifier that catches and retries failed steps. Suppose verification catches 80% of step failures and the retry succeeds. A step now fails only when it errs (0.15) AND the verifier misses it (0.20) AND so on — effective per-step failure drops to roughly 0.15 * 0.20 = 0.03, i.e. effective p ≈ 0.97. Then end-to-end is 0.97**10 ≈ 0.74 — from 20% to ~74% with no smarter base model.
What it did to the data: it converted a multiplicative collapse into a near-flat success curve by inserting a cheap check between every step. The harness, not the model, bought 54 points of reliability.
More tools is not more capability. Past a small set, the model's understanding of which action to take degrades — same dilution problem, now over the action space. The empirical rule: ten well-chosen, non-overlapping tools outperform fifty overlapping ones. Two harness disciplines follow. First, progressive disclosure of tool schemas: expose tool definitions scoped to the current sub-task rather than the full catalog. Second, the ratchet principle: every tool, hook, and config line must trace to a documented past failure or an external requirement. If you can't name why it's there, it's bloat, and bloat is attention you've spent before the agent thinks. This is also why purpose-built tools that integrate with the permission system (Read, Grep, Edit) beat the model improvising bash grep | sed — typed tools are legible to both the model and the audit layer.
Recovery must be structured, not "try again and hope." The harness runs lifecycle hooks at defined points: block dangerous commands pre-execution, require human approval before risky actions, and validate immediately after a code change (run the linter/tests, not "looks done"). On failure, the harness has a fixed ladder — retry-with-feedback, then replan-with-failure-context, then escalate to a human — each rung preserved in the trajectory. This is the difference between an agent that fails loudly and recoverably and one that fails silently and confidently, which is the worst outcome in production.
A real, runnable PEV loop with external memory, a confidence gate, and structured retry. This is the skeleton of every reliable agent harness — model calls stubbed so it runs standalone, with the integration points marked.
import json, os, time
from dataclasses import dataclass, field
# --- External memory: the FILE is the system of record, the window is a cache ---
TODO_LOG = "agent_todo.md"
def memory_read() -> str:
return open(TODO_LOG).read() if os.path.exists(TODO_LOG) else "# Todo\n"
def memory_write(line: str) -> None:
with open(TODO_LOG, "a") as f:
f.write(line.rstrip() + "\n")
@dataclass
class Step:
desc: str
result: str = ""
confidence: float = 0.0
attempts: list = field(default_factory=list) # audit trail
# --- Model calls (swap these for real Anthropic API calls) ---
def plan(intent: str) -> list[Step]:
# Cheap model -> structured JSON decomposition. Stubbed:
raw = '["read the failing test", "edit the source", "run the test suite"]'
return [Step(desc=d) for d in json.loads(raw)]
def execute(step: Step, intent: str, feedback: str = "") -> str:
# CAPABLE model + tools. Intent reinjected EVERY call to fight lost-intent.
ctx = f"INTENT (reinjected): {intent}\nMEMORY:\n{memory_read()}\nFEEDBACK: {feedback}"
return f"did: {step.desc}" # stub; real version returns model output + tool effects
def verify(step: Step, intent: str) -> float:
# Cheap validator scores [0,1]. Real version: separate model call w/ rubric.
return 0.95 if "test" not in step.desc or "PASS" in step.result else 0.4
def run_agent(intent: str, max_retries: int = 2, threshold: float = 0.7) -> bool:
memory_write(f"## Run start: {intent} @ {time.strftime('%H:%M:%S')}")
steps = plan(intent)
for step in steps:
feedback = ""
for attempt in range(max_retries + 1):
step.result = execute(step, intent, feedback)
step.confidence = verify(step, intent)
step.attempts.append({"try": attempt, "conf": step.confidence, "fb": feedback})
if step.confidence >= threshold:
memory_write(f"- [x] {step.desc} (conf={step.confidence:.2f})")
break
feedback = f"Step '{step.desc}' scored {step.confidence:.2f} < {threshold}. Fix and retry."
else:
# retries exhausted -> escalate to replanning with failure context
memory_write(f"- [!] FAILED {step.desc}; replanning")
steps = plan(f"{intent}\nFAILURE: could not complete '{step.desc}'")
return run_agent_from_memory(intent) # bounded re-entry in real code
memory_write(f"## Run complete: {intent}")
return True
def run_agent_from_memory(intent: str) -> bool:
# Ralph Loop: fresh window, rehydrate from the durable log, not the polluted transcript
return run_agent(intent)
if __name__ == "__main__":
run_agent("fix the failing parser test")What to notice: intent is reinjected on every executor call (not left to rot at the top of a transcript); the todo log is appended to disk as the run proceeds so a crash or window reset loses nothing; the verifier gates progress and its score drives a for/else retry ladder that ends in replanning; every attempt is recorded for the audit trail. Swap the three stubbed model functions for real API calls — a cheap model for plan/verify, a capable one for execute — and you have the cost cascade from §3.4. For the production tool layer (Read/Grep/Edit/Bash semantics, sandboxing, batched calls), see /harness and the Claude Code tools reference.
| Dimension | Cheap/naive choice | Robust harness choice | What changes at scale |
|---|---|---|---|
| Context | Append everything to window | Compact + offload to disk + progressive disclosure | Rot is invisible at 3 turns, fatal at 300; offloading becomes mandatory |
| State | Implicit in transcript | External memory (AGENTS.md + todo log) as system of record |
Window resets and multi-day tasks need durable, versioned state |
| Model routing | One capable model for all steps | PEV cascade: cheap plan/verify, capable execute | 60–70% cost savings — but only if tasks are cleanly decomposed |
| Verification | Trust the model's "done" | Confidence gate + retry-with-feedback + replan | Compounding error (p**n) makes this the difference between 20% and 74% |
| Tools | Expose the full catalog | 10 curated tools, schema disclosed on demand, ratchet principle | 50→4,000 tools collapses action selection without routing |
| Recovery | Retry blindly | Structured ladder: retry → replan → human escalation, all audited | Silent failures compound; audit trail is required for incident review |
Failure modes to name in an interview: context rot (gradual quality decay, hard to detect); premature victory (confident "done" with failing tests); lost intent (drift toward recent activity); schema misalignment (model's mental model of a tool diverges from its actual contract); and capability leakage (a badly partitioned PEV plan dumps reasoning back onto the executor, erasing the cost win).
The core tensions. Context vs. reasoning: a bigger window preserves more state but costs more tokens and latency and rots faster — you are always trading completeness against attention. Capability vs. cost: PEV's savings are real but fragile; a sloppy decomposition leaks the executor's expense back in. Autonomy vs. control: more autonomy means less human attention required but more blast radius per mistake — the human-in-the-loop boundary (approve-before-risky-action) is a deliberate harness decision, not a default. Tool breadth vs. focus: the optimal set is found empirically, by removing tools until quality drops, not by adding them until coverage feels complete.
0.85**10 ≈ 0.197, so ~80% of runs fail even though each step is decent — this is multiplicative, so length is brutal. Plan-Execute-Verify breaks the cascade by inserting a cheap validator after each step: a failure now requires both the step to err and the verifier to miss it, pushing effective per-step success toward ~0.97, which makes 0.97**10 ≈ 0.74. Crucially this also lowers cost — only the executor runs on the capable model, so you get ~60–70% savings while raising reliability.AGENTS.md and todo log) is a durable system of record you can diff, audit, access-control, and hand to a fresh agent instance. The discipline is to make the file authoritative and the window disposable — on a long-horizon task you periodically rehydrate intent and state from the file into a clean window (the Ralph Loop), buying back attention you'd otherwise lose to accumulated noise.Flashcard. Naive multi-step success is
p**n— atp=0.85, n=10that's ~20%; a per-step verifier pushes effectiveptoward 0.97 and end-to-end to ~74%, which is why the harness, not the model, owns reliability.
Next: /context-engineering — how to fill the window the harness gives you, token by token.