Agentic Frontends & Harness Engineering
IC5IC6

Harness Engineering: The 80% That Isn't the Model

The model is a stateless next-token oracle; the harness is the control plane that turns one good thought into a hundred reliable actions — and it's where production agents actually die.

15 min read · 14 sections
0

1. Quick anchor

The model is a stateless function: tokens in, a probability distribution over the next token out. It has no memory, no ability to act, and no notion of "the task" beyond what is currently in its context window. Everything that makes an agent reliable — remembering intent across hours, calling the right tool, noticing it was wrong, recovering without a human — lives in the harness: the runtime software wrapped around the model. Field data from 2025–2026 deployments attributes roughly 65% of enterprise agent failures to harness defects (context drift, schema misalignment, state degradation) rather than to the model's raw capability. The mental model to carry into any interview: a frontier model is a brilliant amnesiac contractor, and the harness is the project manager, the filing cabinet, and the QA gate that turns brilliance into shipped work.

2. Why interviewers probe this

The harness is the cleanest signal of systems maturity in an agent engineer, because it separates people who have only prompted from people who have operated agents at scale.

  • IC5 (senior): Can you name the concrete failure modes — context rot, premature victory, lost intent — and map each to a specific mechanism (compaction, verification loop, external memory)? Can you reason about per-step vs. end-to-end reliability and the cost of a verification model? Can you write the control loop, not just describe it?
  • IC6 (staff): Can you design the control plane for many agents, many tools, and many teams? Do you treat configuration as something that must trace to a documented failure rather than vibes? Can you build the eval harness that proves a change helped, design the cost/latency budget for a model cascade, and articulate where autonomy must yield to a human-in-the-loop boundary? Staff candidates are expected to talk about the harness as a product with SLOs, not a script.

3. Concept build-up

Beginner explainerNew here? The words first

The words first.

  • Harness — the runtime code around the model: tool orchestration, memory, verification, and lifecycle rules. The "control plane."
  • Context window — the fixed token budget the model can attend to in one call (e.g., 200K tokens). Everything the model "knows" right now lives here.
  • Context rot — slow degradation of answer quality as the window fills with stale tool output, so the model's attention gets diluted.
  • Tool — a typed function the model can call (read a file, query a DB). The harness executes it and feeds the result back.
  • Trajectory — the full ordered transcript of one agent run: messages, tool calls, tool results.
  • External memory — durable storage outside the window (a file, a todo log) that survives even when the window is reset.
  • Verification loop — a step where a (often cheaper or different) model checks the previous step's output before the agent proceeds.
  • Premature victory — the agent declares the task done when it isn't, because nothing forced it to check.

Step by step.

  1. The harness assembles a context window: system prompt + intent + relevant memory + recent tool output.
  2. The model emits text and/or a tool call.
  3. The harness executes the tool inside a sandbox and captures the result.
  4. The harness decides what to keep, summarize, or offload to disk (managing rot).
  5. A verifier scores the step; low confidence triggers retry or replan.
  6. Durable facts get written to external memory so they survive a window reset.
  7. Loop until a verified done-condition is met — not just until the model says "done."

Remember this: the model thinks; the harness remembers, checks, and acts — and that's where reliability is won or lost.

3.1 The three failure modes, from first principles

Start from what a transformer actually is. Attention assigns a softmax-weighted score to every token in the window when predicting the next one. As the window grows, the probability mass available to any single relevant token shrinks — the denominator of the softmax includes thousands of irrelevant tokens (old logs, stale diffs, dead-end tool calls). This is the mechanical basis of context rot: it is not that the model "forgets," it is that signal is drowned in accumulated noise. Empirically, the accumulating layer — raw tool outputs piled into the window — degrades reasoning quality gradually, so it is rarely caught until a run visibly goes off the rails.

Two siblings follow from the same root cause:

  • Lost intent. The original task lives at the top of the window. After fifty turns it is far from the prediction point and competing with everything in between. The agent drifts toward whatever it was most recently doing, not what it was asked to do.
  • Premature victory. Nothing in next-token prediction inherently checks "is the task actually complete?" The model is trained to produce plausible, confident continuations. "I've fixed the bug." is a high-probability sentence regardless of whether the tests pass. Absent an external gate, agents declare success early.

The harness exists to counter all three. None of these are model-quality problems; a smarter model rots more slowly but still rots.

3.2 Context management: compaction, offloading, progressive disclosure

The harness manages the window as a three-layer system:

  1. Compaction — summarize older turns into a dense fact list, freeing tokens while retaining the load-bearing details. The trick is preserving decisions and constraints ("DB is read-replica only," "ticket says ship behind a flag") while discarding the verbose scaffolding that produced them.
  2. Offloading — move large artifacts (full logs, diffs, test output) to the filesystem or an external store, leaving a pointer in the window. The model retrieves on demand instead of carrying everything. This decouples storage from reasoning, which is what makes versioning, targeted edits, and access control possible.
  3. Progressive disclosure — load tool schemas, sub-prompts, and reference docs only when the current sub-task needs them. Injecting all fifty tool definitions up front fragments attention before the agent has even started.

A robust long-horizon pattern is the "Ralph Loop": periodically reinject the original intent into a clean window, rehydrating from external memory rather than from the polluted running transcript. You pay a re-read cost; you buy back attention and kill drift.

3.3 External memory as the system of record (AGENTS.md / todo log)

The single highest-leverage harness pattern: make a durable file, not the context window, the source of truth. A repo-local AGENTS.md (canonical instructions, conventions, gotchas) plus a running todo log (what's done, what's next, what was tried and failed) gives the agent a system of record that survives any window reset. When the window is compacted or the Ralph Loop fires, state is reconstructed from the file — the file is authoritative, the window is a cache.

This inverts the naive design where state lives implicitly in the transcript and dies whenever you summarize. Explicit read-write interfaces to memory let you version state, diff it, audit it, and hand it to a fresh agent instance. The same brevity discipline applies as to tools: a tight, curated AGENTS.md beats a sprawling one, for the same softmax-dilution reason.

3.4 Verification loops: the capable model checks the cheap one

The structural fix for premature victory and compounding error is Plan-Execute-Verify (PEV) with a model cascade:

  • Planner (cheap model, e.g. Haiku-class): decompose the task into a structured JSON list of steps. Decomposition is easy; cheap models do it well.
  • Executor (capable model, e.g. Sonnet-class): the reasoning and tool calls — this is where capability spend earns ROI.
  • Validator (cheap model): score each step's result with a confidence in [0.0, 1.0] and run deterministic recovery logic.

Sub-threshold confidence triggers a retry that injects the error feedback into the next attempt; after exhausting retries, the harness escalates to full replanning with the failure as context. Every attempt is preserved in an audit trail, which is what prevents a single wrong step from silently propagating. The economic punchline: routing only the executor through the capable model yields roughly 60–70% per-run cost savings versus all-capable routing — a typical three-step task runs about $0.01 under PEV vs ~$0.027 all-Sonnetwhile raising reliability, because the cheap validator catches what the executor would have shipped blind.

Compounding failure and why verification wins — on real numbers

Name the symbols in plain words: p is the probability one step succeeds; n is the number of steps; end-to-end success of a naive chain is p multiplied by itself n times, written p**n.

Work it with real numbers. Say each step is 85% reliable (p = 0.85) and the task has 10 steps (n = 10). Naive end-to-end success is 0.85**10. Compute it: 0.85**2 = 0.7225, 0.85**50.4437, 0.85**100.4437 * 0.44370.197. So about 0.20 — roughly 80% of runs fail, even though every individual step is "pretty good." That is the compounding tax, and it is why "the model is 85% accurate" tells you almost nothing about a 10-step agent.

Now add a verifier that catches and retries failed steps. Suppose verification catches 80% of step failures and the retry succeeds. A step now fails only when it errs (0.15) AND the verifier misses it (0.20) AND so on — effective per-step failure drops to roughly 0.15 * 0.20 = 0.03, i.e. effective p ≈ 0.97. Then end-to-end is 0.97**100.74 — from 20% to ~74% with no smarter base model.

What it did to the data: it converted a multiplicative collapse into a near-flat success curve by inserting a cheap check between every step. The harness, not the model, bought 54 points of reliability.

3.5 Tool routing: ten good tools beat fifty

More tools is not more capability. Past a small set, the model's understanding of which action to take degrades — same dilution problem, now over the action space. The empirical rule: ten well-chosen, non-overlapping tools outperform fifty overlapping ones. Two harness disciplines follow. First, progressive disclosure of tool schemas: expose tool definitions scoped to the current sub-task rather than the full catalog. Second, the ratchet principle: every tool, hook, and config line must trace to a documented past failure or an external requirement. If you can't name why it's there, it's bloat, and bloat is attention you've spent before the agent thinks. This is also why purpose-built tools that integrate with the permission system (Read, Grep, Edit) beat the model improvising bash grep | sed — typed tools are legible to both the model and the audit layer.

3.6 Structured error recovery and lifecycle hooks

Recovery must be structured, not "try again and hope." The harness runs lifecycle hooks at defined points: block dangerous commands pre-execution, require human approval before risky actions, and validate immediately after a code change (run the linter/tests, not "looks done"). On failure, the harness has a fixed ladder — retry-with-feedback, then replan-with-failure-context, then escalate to a human — each rung preserved in the trajectory. This is the difference between an agent that fails loudly and recoverably and one that fails silently and confidently, which is the worst outcome in production.

4. Minimal implementation

A real, runnable PEV loop with external memory, a confidence gate, and structured retry. This is the skeleton of every reliable agent harness — model calls stubbed so it runs standalone, with the integration points marked.

import json, os, time
from dataclasses import dataclass, field
 
# --- External memory: the FILE is the system of record, the window is a cache ---
TODO_LOG = "agent_todo.md"
 
def memory_read() -> str:
    return open(TODO_LOG).read() if os.path.exists(TODO_LOG) else "# Todo\n"
 
def memory_write(line: str) -> None:
    with open(TODO_LOG, "a") as f:
        f.write(line.rstrip() + "\n")
 
@dataclass
class Step:
    desc: str
    result: str = ""
    confidence: float = 0.0
    attempts: list = field(default_factory=list)  # audit trail
 
# --- Model calls (swap these for real Anthropic API calls) ---
def plan(intent: str) -> list[Step]:
    # Cheap model -> structured JSON decomposition. Stubbed:
    raw = '["read the failing test", "edit the source", "run the test suite"]'
    return [Step(desc=d) for d in json.loads(raw)]
 
def execute(step: Step, intent: str, feedback: str = "") -> str:
    # CAPABLE model + tools. Intent reinjected EVERY call to fight lost-intent.
    ctx = f"INTENT (reinjected): {intent}\nMEMORY:\n{memory_read()}\nFEEDBACK: {feedback}"
    return f"did: {step.desc}"  # stub; real version returns model output + tool effects
 
def verify(step: Step, intent: str) -> float:
    # Cheap validator scores [0,1]. Real version: separate model call w/ rubric.
    return 0.95 if "test" not in step.desc or "PASS" in step.result else 0.4
 
def run_agent(intent: str, max_retries: int = 2, threshold: float = 0.7) -> bool:
    memory_write(f"## Run start: {intent}  @ {time.strftime('%H:%M:%S')}")
    steps = plan(intent)
    for step in steps:
        feedback = ""
        for attempt in range(max_retries + 1):
            step.result = execute(step, intent, feedback)
            step.confidence = verify(step, intent)
            step.attempts.append({"try": attempt, "conf": step.confidence, "fb": feedback})
            if step.confidence >= threshold:
                memory_write(f"- [x] {step.desc} (conf={step.confidence:.2f})")
                break
            feedback = f"Step '{step.desc}' scored {step.confidence:.2f} < {threshold}. Fix and retry."
        else:
            # retries exhausted -> escalate to replanning with failure context
            memory_write(f"- [!] FAILED {step.desc}; replanning")
            steps = plan(f"{intent}\nFAILURE: could not complete '{step.desc}'")
            return run_agent_from_memory(intent)  # bounded re-entry in real code
    memory_write(f"## Run complete: {intent}")
    return True
 
def run_agent_from_memory(intent: str) -> bool:
    # Ralph Loop: fresh window, rehydrate from the durable log, not the polluted transcript
    return run_agent(intent)
 
if __name__ == "__main__":
    run_agent("fix the failing parser test")

What to notice: intent is reinjected on every executor call (not left to rot at the top of a transcript); the todo log is appended to disk as the run proceeds so a crash or window reset loses nothing; the verifier gates progress and its score drives a for/else retry ladder that ends in replanning; every attempt is recorded for the audit trail. Swap the three stubbed model functions for real API calls — a cheap model for plan/verify, a capable one for execute — and you have the cost cascade from §3.4. For the production tool layer (Read/Grep/Edit/Bash semantics, sandboxing, batched calls), see /harness and the Claude Code tools reference.

5. Production tradeoffs

Dimension Cheap/naive choice Robust harness choice What changes at scale
Context Append everything to window Compact + offload to disk + progressive disclosure Rot is invisible at 3 turns, fatal at 300; offloading becomes mandatory
State Implicit in transcript External memory (AGENTS.md + todo log) as system of record Window resets and multi-day tasks need durable, versioned state
Model routing One capable model for all steps PEV cascade: cheap plan/verify, capable execute 60–70% cost savings — but only if tasks are cleanly decomposed
Verification Trust the model's "done" Confidence gate + retry-with-feedback + replan Compounding error (p**n) makes this the difference between 20% and 74%
Tools Expose the full catalog 10 curated tools, schema disclosed on demand, ratchet principle 50→4,000 tools collapses action selection without routing
Recovery Retry blindly Structured ladder: retry → replan → human escalation, all audited Silent failures compound; audit trail is required for incident review

Failure modes to name in an interview: context rot (gradual quality decay, hard to detect); premature victory (confident "done" with failing tests); lost intent (drift toward recent activity); schema misalignment (model's mental model of a tool diverges from its actual contract); and capability leakage (a badly partitioned PEV plan dumps reasoning back onto the executor, erasing the cost win).

The core tensions. Context vs. reasoning: a bigger window preserves more state but costs more tokens and latency and rots faster — you are always trading completeness against attention. Capability vs. cost: PEV's savings are real but fragile; a sloppy decomposition leaks the executor's expense back in. Autonomy vs. control: more autonomy means less human attention required but more blast radius per mistake — the human-in-the-loop boundary (approve-before-risky-action) is a deliberate harness decision, not a default. Tool breadth vs. focus: the optimal set is found empirically, by removing tools until quality drops, not by adding them until coverage feels complete.

6. How it's asked

[IC4] What is context rot and what's the cheapest fix? Context rot is the gradual degradation of reasoning quality as the window fills with accumulated, mostly-stale tool output — mechanically, the softmax over attention dilutes the relevant tokens among thousands of irrelevant ones, so the symptom is a slow drift you usually notice only after the run derails. The cheapest high-leverage fix is offloading: write large artifacts (logs, diffs, full test output) to disk and keep only a pointer plus a short summary in the window, so the model retrieves on demand instead of carrying everything.
[IC5] Derive why a 10-step agent at 85% per-step reliability mostly fails, and how PEV fixes it. End-to-end success of a naive chain is the product of per-step success: 0.85**100.197, so ~80% of runs fail even though each step is decent — this is multiplicative, so length is brutal. Plan-Execute-Verify breaks the cascade by inserting a cheap validator after each step: a failure now requires both the step to err and the verifier to miss it, pushing effective per-step success toward ~0.97, which makes 0.97**100.74. Crucially this also lowers cost — only the executor runs on the capable model, so you get ~60–70% savings while raising reliability.
[IC5] Why is external memory better than just a long context window? A long window is a cache that dies on reset, compaction, or crash, and it rots as it grows; external memory (a versioned AGENTS.md and todo log) is a durable system of record you can diff, audit, access-control, and hand to a fresh agent instance. The discipline is to make the file authoritative and the window disposable — on a long-horizon task you periodically rehydrate intent and state from the file into a clean window (the Ralph Loop), buying back attention you'd otherwise lose to accumulated noise.
[IC6] 4,000 tools across teams, success degrading as the catalog grows. Design the harness and the eval. The degradation is action-space dilution: the model can't reliably select among 4,000 tool schemas, and the schemas alone may not fit the window. Harness design: a router that retrieves a small candidate set (10–20) per sub-task via semantic match over tool descriptions, progressive disclosure so only candidate schemas enter the window, a ratchet requiring each tool to justify its existence (owner, documented use, deprecation date), and namespacing to kill overlaps. For the eval: build a held-out task suite scored on a binary resolved rate (all required tool calls correct and the task's success assertions pass) plus a soft progress rate for partial credit, run it as an A/B against the current full-catalog harness, and confirm the router didn't regress recall on the right tool. Report cost and p95 latency alongside success so a reliability win isn't a latency loss.
[IC6] Where does the harness end and the model begin — and why does that boundary matter for hiring? The model owns judgment under uncertainty — reasoning, synthesis, tool-call formulation; the harness owns everything that must be deterministic and accountable — memory, verification, routing, recovery, and the human-in-the-loop gates. The boundary matters because ~65% of production failures are harness defects, so a model upgrade rarely fixes a reliability problem rooted in lost state or missing verification. A staff engineer is expected to instrument the boundary: when a run fails, can you attribute it to a model error (wrong judgment with correct context) versus a harness error (right model, wrong context/state/gate)? That attribution is what makes the system improvable.

7. Pitfalls & flashcards

  • Treating the window as state. It's a cache. If a fact must survive a reset, it lives in external memory, period.
  • Trusting "done." Next-token prediction emits confident completions regardless of truth; a done-condition must be verified (tests pass, assertions hold), never asserted.
  • Adding tools to feel complete. Each tool dilutes action selection. Remove until quality drops; that's your set.
  • Unstructured retry. "Try again" without injecting the specific failure feedback just re-rolls the same dice. Feedback-in-the-loop is the point.
  • Measuring per-step accuracy. It hides the multiplicative collapse. Always report end-to-end resolved rate over multi-step tasks (see SWE-EVO: GPT-5.4 hits 72.8% on isolated SWE-Bench Verified but only ~25% on long-horizon multi-file evolution).
  • PEV without clean decomposition. A bad plan dumps reasoning onto the executor and erases the cost win — partition tasks so cheap models genuinely handle plan/verify.

Flashcard. Naive multi-step success is p**n — at p=0.85, n=10 that's ~20%; a per-step verifier pushes effective p toward 0.97 and end-to-end to ~74%, which is why the harness, not the model, owns reliability.

8. Further reading

Next: /context-engineering — how to fill the window the harness gives you, token by token.

Primary sources
← More in Agentic Frontends & Harness Engineering