A coding agent is not a smart model — it is a tight loop where read/edit/grep/bash tools meet a sandbox, a staleness check, and a test runner that feeds failures back until the diff is green.
A coding agent is a loop, not a model. The model is a stateless next-token predictor; the harness is the software that turns its text into deterministic, accountable filesystem action — a small set of tools (read, grep, glob, edit, write, bash, run_tests), a sandbox that bounds what those tools can touch, a staleness check that refuses edits against a file the model hasn't seen, and a verification step that runs the tests and feeds failures back as the next observation. Research attributes ~65% of agent production failures to harness defects (context drift, schema misalignment, stale state) rather than model limits — so the engineering lives in the loop, not the prompt. The single highest-leverage decision is making the loop eval-driven: every edit is followed by run_tests, and a red result is not an error to swallow but the most valuable input the agent gets. Build the agent so the test runner — not the model's confidence — decides when the task is done.
Edit needs a prior Read (staleness), why exact-string replacement beats regex, why batching independent tool calls matters for latency, and how the test runner gates "done"? Can you write the agent loop from scratch, not just call a framework?0.85^10 ≈ 20% success across ten unverified steps, and how a test gate breaks the cascade?The words first.
read(path), bash(cmd)); the harness runs it and returns the result as the next message.bash and friends can do.Step by step.
edit, verify the file hasn't changed since the model read it.edit, run run_tests and capture the output.Remember this: the test runner, not the model, decides when the task is done.
A coding agent's capability ceiling is set by its tools long before its model. Claude Code's production surface is six core tools — and the smallness is the point. The "focused tool set" principle is empirical: ten well-chosen tools outperform fifty overlapping ones because the model keeps a sharper mental model of what's available, and decision overhead per step drops.
The canonical surface for a coding agent:
.gitignore, supports multiline matching, and can scope by file type. This is how the agent navigates a repo it has never seen.**/*.ts), sorted by mtime, capped (Claude Code caps at 100 results) so a giant repo doesn't blow the context window.old_string is unique (or replace_all=true). Exactness is a safety property: regex edits silently match the wrong span; an exact match either lands or fails loudly.On top of these, a coding agent adds one domain tool: run_tests — really a constrained bash wrapper that runs the project's test command, parses pass/fail, and returns a compact summary. Keeping it separate from raw bash lets you attach structured semantics (FAIL→PASS counts) the loop can branch on.
Two non-obvious design rules. First, built-in tools beat shelling out: a first-class Edit integrates with the permission system and the staleness check, whereas bash("sed -i …") bypasses both. Second, batch independent calls: if the model needs three files, emitting three read calls in one turn lets the harness run them concurrently — throughput, not just elegance.
The tools are dangerous by construction — bash can rm -rf, write can clobber a config. The harness contains this with two layers. Sandboxing is OS-level: filesystem scope bounded to the project, network restrictions, time limits, output caps. Permission routing is declarative: rules decide which tools run freely, which prompt the human, and which are blocked outright. Claude Code threads this through subagents too — a subagent inherits the parent's tools unless narrowed via tools/disallowedTools, and crucially a background subagent auto-denies any unpermitted call (no human is watching to approve), while a foreground one surfaces the prompt immediately.
This is where lifecycle hooks live — code that runs at fixed points in the loop: block a dangerous command before execution, require approval before a risky action, validate immediately after a code change. The discipline here is the ratchet principle: every hook, every config line, every tool must trace to a documented past failure or external requirement. No speculative guardrails — they bloat the surface and fragment the model's attention. You add the "block force-push" hook the day after a force-push incident, not before.
The hardest part of a coding agent is not editing one file — it's editing many, in dependency order, against a repo that changes under you (tests rewrite files, formatters run, a parallel process commits). Two mechanisms keep this sane.
Read-before-edit (the staleness contract). The Edit tool refuses unless the model has read the file in this session. The harness tracks the version the model saw; if the on-disk content has changed since, the edit is rejected with the fresh contents, forcing a re-read. This prevents the classic failure: the model proposes a patch against line 40, but a prior edit shifted everything by ten lines, so the old_string now matches the wrong span — or matches nothing and corrupts the file. Exact-string matching makes the check enforceable: there's a unique anchor to validate against.
Ordering and atomicity. A multi-file refactor (rename a function across 21 files — the kind of long-horizon change SWE-EVO measures) must apply edits so the repo is never half-broken in a way that makes the next read misleading. In practice the agent edits the definition, then greps for call sites, then edits each — re-reading between batches. The harness doesn't enforce a transaction; the loop enforces correctness by re-verifying with run_tests after the batch.
Name each piece in plain words:
seen_version — a hash of the file contents the model last read this session.disk_version — a hash of the file's current bytes on disk.old_string — the exact text the model wants to replace.Concrete run. The model reads utils.py (12 lines); the harness records seen_version = hash("…def parse(x):\n return int(x)…") = a1f3. The model then edits another file, which a formatter reflows — utils.py is untouched, still a1f3. Now the model emits edit("utils.py", old_string="return int(x)", new_string="return int(x.strip())").
The harness checks: disk_version = hash(current bytes) = a1f3. Since a1f3 == seen_version, the file is fresh -> proceed. It then searches for old_string: found exactly once at line 2 -> apply -> file becomes return int(x.strip()).
Counter-case: suppose a parallel process had appended a line to utils.py, so disk_version = b7e0 ≠ a1f3. The harness rejects the edit and returns the current contents, forcing a re-read. What it did: it refused to patch a version of reality the model never saw, trading one wasted turn for a prevented file corruption.
This is the heart. Naively, an agent edits code and declares victory on its own confidence. That fails because errors compound: if each step is 85% reliable, ten unverified steps give 0.85^10 ≈ 0.20 — an 80% chance the final result is wrong, with no signal about where it broke. The fix is to make the test runner the source of truth.
The loop: edit -> run_tests -> parse -> branch. A green result ends the task. A red result is not an error to swallow — it's the highest-value observation the agent receives. The harness feeds the failing test names and tracebacks back into context as the next message, and the model edits again with that concrete signal. This is the same idea as Plan-Execute-Verify (PEV): a verifier scores each step, and sub-threshold results trigger a retry loop that injects the error feedback, escalating to a re-plan only after retries are exhausted. Every attempt is preserved in an audit trail so a cascading error can be traced rather than silently propagated.
The economic version of this is model routing: a cheap model (e.g. Haiku at ~$0.25/MTok) plans the decomposition and validates results, while the capable model (Sonnet at ~$3/MTok) does the hard reasoning and edits. A three-step task can run ~$0.01 under PEV vs. ~$0.027 routing everything through the capable model — a 60–70% saving — but only if the decomposition is clean; a badly partitioned task leaks the hard work back into the executor and the savings evaporate.
SWE-Bench is the standard scoreboard: real GitHub issues where the agent must produce a patch that flips the repo's failing tests (FAIL→PASS) without breaking passing ones (PASS→PASS). Two metrics matter. Resolved rate is binary and brutal — all targeted FAIL→PASS tests pass and no PASS→PASS regressions; partial credit is zero. Soft fix rate captures partial progress (fraction of FAIL→PASS fixed under the regression constraint) and is the better diagnostic when you're tuning a harness, because it tells you whether you're moving.
The headline number for staff interviews is the cliff. GPT-5-class agents reach ~72.8% on SWE-Bench Verified (isolated bug fixes) but only ~25% on SWE-EVO — a 2025 benchmark for long-horizon software evolution, where the agent reads release notes, plans modifications across 21+ files, and validates against ~874 tests per instance (avg 81 FAIL→PASS). The model is the same; what collapses is sustained multi-file reasoning over a long horizon. That gap is the single most important fact for deciding where harness effort goes (see §6).
A real agent loop in ~70 lines. This is production-shaped: a tool dispatch table, a staleness map, exact-string edits, a test gate, and a turn cap. It uses the Anthropic Messages API tool-use protocol (the same shape OpenAI/others expose).
import hashlib, subprocess, anthropic
client = anthropic.Anthropic()
SEEN: dict[str, str] = {} # path -> hash of contents the model last read
def _hash(s: str) -> str:
return hashlib.sha256(s.encode()).hexdigest()[:8]
def read(path, **_):
text = open(path).read()
SEEN[path] = _hash(text)
return "\n".join(f"{i+1}\t{ln}" for i, ln in enumerate(text.splitlines()))
def edit(path, old_string, new_string, **_):
if path not in SEEN: # read-before-edit
return f"ERROR: read {path} before editing."
disk = open(path).read()
if _hash(disk) != SEEN[path]: # staleness check
SEEN.pop(path)
return f"ERROR: {path} changed on disk. Re-read it.\n{disk}"
if disk.count(old_string) != 1: # exact, unique match
return f"ERROR: old_string occurs {disk.count(old_string)}x; must be unique."
new = disk.replace(old_string, new_string)
open(path, "w").write(new)
SEEN[path] = _hash(new) # keep the contract fresh
return "OK"
def run_tests(cmd="pytest -q", **_):
p = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=120)
out = (p.stdout + p.stderr)[-30_000:] # output cap, like bash
return f"EXIT {p.returncode}\n{out}"
TOOLS = {"read": read, "edit": edit, "run_tests": run_tests}
SCHEMA = [ # abbreviated; each needs an input_schema in real code
{"name": "read", "description": "Read a file with line numbers.",
"input_schema": {"type": "object", "properties": {"path": {"type": "string"}},
"required": ["path"]}},
{"name": "edit", "description": "Exact-string replace; requires prior read.",
"input_schema": {"type": "object", "properties": {
"path": {"type": "string"}, "old_string": {"type": "string"},
"new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}},
{"name": "run_tests", "description": "Run the test suite.",
"input_schema": {"type": "object", "properties": {"cmd": {"type": "string"}}}},
]
def agent(task: str, max_turns: int = 25):
msgs = [{"role": "user", "content": task}]
for _ in range(max_turns):
r = client.messages.create(model="claude-sonnet-4-6", max_tokens=4096,
tools=SCHEMA, messages=msgs)
msgs.append({"role": "assistant", "content": r.content})
if r.stop_reason != "tool_use":
return r # model is done talking
results = []
for block in r.content: # batch all tool calls this turn
if block.type == "tool_use":
out = TOOLS[block.name](**block.input)
results.append({"type": "tool_result",
"tool_use_id": block.id, "content": out})
msgs.append({"role": "user", "content": results})
raise RuntimeError("hit max_turns without finishing")What to notice. The staleness map (SEEN) is the whole safety story for edits — three lines that prevent the most common file-corruption bug. edit returns the fresh disk contents inside the error so the model can recover in one turn. run_tests caps output exactly like bash so a verbose suite can't blow the window. The loop batches every tool_use block from a turn before replying, so independent reads run together. And max_turns is the backstop against a model that loops forever on a test it can't fix — in production you'd escalate to a re-plan (PEV) instead of crashing. The system prompt (omitted) must instruct: read before edit, run tests after every change, stop only on green.
| Axis | Cheap / simple end | Expensive / robust end | What changes at scale |
|---|---|---|---|
| Tool count | Few focused tools (6–10) | Many specialized tools | More tools fragment model attention; keep it small, add only on documented need (ratchet) |
| Model routing | One capable model everywhere | PEV cascade (Haiku plan/validate, Sonnet execute) | ~60–70% cost cut if decomposition is clean; bad partitioning leaks work back to the executor |
| Verification | Model self-reports "done" | Test gate + audit trail every step | 0.85^10≈20% unverified success; the gate is non-optional past ~3 steps |
| Sandbox | Trust the model | OS scope + permission routing + hooks | Background subagents must auto-deny unpermitted calls; foreground prompts the human |
| Context | One growing window | Compaction + offloading to files (AGENTS.md) | Layer-3 tool-output accumulation causes context rot; long-horizon tasks degrade silently |
| Test cost | Run full suite each edit | Targeted tests, then full suite at the end | Full-suite-per-edit dominates latency and $; scope to touched modules, gate final on full run |
Cost. The dominant line item is usually not the model — it's re-running the test suite and re-reading large files every turn. A 90-second test run executed on each of 15 edit attempts is 22 minutes of wall-clock and a tail of token spend re-ingesting output. Mitigations: run only tests for touched modules in the inner loop, full suite once at the end; offload long logs to the filesystem and let the model grep them on demand rather than holding them in context.
Latency. Each turn is a model round-trip plus tool execution. Batching independent tool calls collapses N reads into one round-trip. The other big lever is parallel test execution — but watch the sandbox: parallel tests writing temp files can trip the staleness check on shared fixtures.
Quality / failure modes. (1) Context rot — accumulated tool outputs degrade reasoning until someone notices; counter with compaction (summarize old turns) and the "Ralph Loop" (reinject the original intent into a clean window on long tasks). (2) Stale-edit corruption — the staleness check prevents it; never let the model edit via bash/sed, which bypasses the check. (3) Silent green — the model edits the test to pass instead of the code; defend with a frozen test set the agent can't modify (SWE-Bench does this by separating the gold test patch from the agent's editable surface). (4) Reward/confidence over trust — never let model confidence end the task; only a green test run does.
0.85^10 ≈ 20% unverified) by making the test runner, not model confidence, decide "done." It goes wrong three ways: running the full suite every edit (latency/cost blowup — scope to touched modules in the loop), the model editing the test to pass instead of the code (freeze the test set), and feeding back too much output so context rots (cap and summarize). The subtle failure is declaring success on a green inner loop while a regression breaks elsewhere — always gate the final answer on the full suite, PASS→PASS included.bash("sed -i …"). Why is that a problem and how do you stop it?
Shelling out bypasses both the permission system and the staleness check, so you lose the read-before-edit contract and risk patching stale or wrong spans with no validation, plus you can't audit or approve the change. Stop it by routing all mutations through the first-class Edit tool and using a permission rule/hook to block sed -i, > redirects, and in-place mutators in bash. The built-in tool is strictly better because it integrates with the harness's safety machinery; bash mutation is an escape hatch that defeats the harness's whole reason to exist.bash, and a permission policy where mutating tools and any push/merge require human approval (foreground) and auto-deny for background subagents. Verification: a frozen test set the agent cannot edit, full-suite gate (FAIL→PASS and PASS→PASS) before any merge proposal, and an audit trail of every attempt so a bad change is traceable, not silently propagated. Apply the ratchet principle — every hook traces to a real incident or compliance requirement — and never let the agent self-certify; the merge is gated on green tests plus a human approval, with the model's confidence carrying zero authority.bash. sed -i/redirects bypass the staleness check and permission system — route all mutations through Edit.Flashcard. A coding agent is a loop: read -> edit (staleness-checked, exact-match) -> run_tests -> feed red back -> stop on green. ~65% of failures are harness, not model; the test runner, not confidence, decides "done."
Next: /harness for the full harness pillar, or /evals for grading agent trajectories and keeping the test gate honest.