RL on Coding Trajectories: ART, GRPO, and RULER

1. Quick anchor

Your harness already records every rollout: a tree of system/user/assistant/tool messages ending in success or failure. That audit trail is not just for debugging — it is training data. RL on trajectories closes the loop: sample several rollouts per task, score them, and nudge the policy toward the high-scoring ones. The two pieces that make this practical in 2026 are GRPO (Group Relative Policy Optimization — PPO with the value network deleted, so reward is just "how good was this rollout relative to its siblings") and RULER (an LLM-as-judge that ranks sibling trajectories against each other, so you never hand-write a reward function). OpenPipe's ART framework wires both into a client-server loop with LoRA, and the headline result — a Qwen 2.5 14B model trained past OpenAI's o3 on email retrieval — is the whole pitch: on one narrow task, experience beats raw capability. This lesson is the harness-engineer's view of when that trade is real and when it is a trap.

2. Why interviewers probe this

This is an IC6 / staff topic because it sits at the seam between harness engineering and post-training — owning it means you can argue systems economics, not just plumbing.

• [IC6] Systems judgment under cost pressure. Can you decide, with numbers, when to train a small model versus keep paying frontier per-token? The wrong call burns either a GPU budget or an inference budget for a quarter.
• [IC6] Reward-signal design. RL is only as good as its reward. Can you reason about RULER's failure modes (judge bias, reward hacking, reward saturation) and the guardrails — held-out verifiable evals, anchor trajectories — that keep a learned signal honest?
• [IC6] Closing the data loop. Do you see that your production trajectory logs are a moat? The candidate who treats the harness's audit trail as a renewable training asset, versioned and replayable, is operating a level above the one who treats RL as someone else's job.
• [IC6] Knowing the ceiling. Can you name where this doesn't work — broad open-ended agents, sparse-reward long-horizon tasks, anything where the judge can't tell good from bad? Honesty about the boundary is the senior signal.

3. Concept build-up

3.1 The compounding-error problem RL is trying to fix

Start from why a capable model still fails as an agent. A multi-step workflow multiplies per-step reliability. If each step succeeds independently with probability 0.85, a 10-step task lands at 0.85^10 ≈ 0.197 — under 20% end-to-end success, even though every individual step "mostly works." Prompt engineering and PEV verification loops (see /harness) attack this from the harness side. RL attacks it from the weights side: it raises the per-step success probability on your distribution of tasks by burning in the patterns that actually worked, so the same chain compounds from a higher base.

The catch that defines the whole field: agentic reward is sparse and expensive. You usually only know if a coding trajectory was good at the very end (tests pass / don't), there's no labeled "correct trajectory" to imitate, and the search space of action sequences is enormous. Every design choice below — dropping the critic, judging instead of labeling, LoRA instead of full fine-tune — is a response to that scarcity.

3.2 GRPO: PPO with the critic deleted

PPO, the workhorse of RLHF, trains two networks: the policy (the model you want) and a critic / value model that estimates the expected future reward of a state, used to compute the advantage — how much better an action was than the baseline. The critic is a second model the same size as the policy: extra GPU memory, extra training instability, extra thing to get wrong.

GRPO's move is to replace the learned critic with the group mean. For one task, sample a group of G trajectories. Score each one, then define each trajectory's advantage as its reward minus the group's mean, divided by the group's standard deviation:

advantage_i = (reward_i − mean(rewards)) / std(rewards)

That's it — the baseline is just "the average of my siblings on this exact task." No value network to fit. The policy update then pushes probability mass toward trajectories with positive advantage and away from negative ones (clipped, PPO-style, so no single step moves too far). The deep consequence: only relative ranking matters, not absolute scores. If RULER scores a group [2, 7, 9, 4] or [0.2, 0.7, 0.9, 0.4], the normalized advantages are identical. This is exactly what frees you from calibrated, hand-engineered reward — a property RULER exploits directly.

3.3 RULER: the judge that ranks instead of scores

GRPO needs a reward per trajectory. The classic answer is a verifiable reward: run the unit tests, +1 if green. That's gold when it exists — unfakeable, cheap, deterministic — and you should always use it when you can. But most agentic tasks have no clean oracle: "did the agent answer the support ticket well," "is this refactor good," "did it use the right tools." Hand-writing a reward function for those is the bottleneck.

RULER (Relative Universal LLM-Elicited Rewards), built into ART, sidesteps it: feed the group of G sibling trajectories to a strong judge model and ask it to rank them relative to each other against a rubric you describe in plain English ("prefer fixes that pass tests, touch fewer files, and add no dead code"). The judge emits relative scores; GRPO consumes the ranking. Because GRPO only cares about ordering, the judge never has to be calibrated — it just has to get which-is-better right more often than chance, on average over many groups. The noise washes out across thousands of groups; the signal accumulates.

This is the key unlock: you replace a labeling project with a prompt. You can change the reward criterion by editing the rubric, version it like code, and bootstrap a domain where no labeled data exists.

Where it bites: the judge inherits its own biases (length, verbosity, surface polish over correctness), and once the policy figures out what the judge likes, it will reward-hack — producing trajectories the judge loves and reality hates. The defenses are non-negotiable: keep a held-out verifiable eval the judge can't see, anchor groups with a known-good reference trajectory so the judge has a fixed point, and watch for reward going up while the held-out metric flatlines (the signature of hacking).

3.4 ART: the client-server loop that runs it

ART (Agent Reinforcement Trainer, OpenPipe, ~9k+ GitHub stars, built on Unsloth's memory-efficient GRPO) packages this into a two-process architecture that maps cleanly onto how a harness already works.

• Client — an OpenAI-compatible endpoint that lives in your codebase. Your existing agent loop calls it for completions, exactly as it would call any LLM API. Each message lands in a Trajectory object; when a rollout finishes you attach a reward (a number, a test result, or hand it to RULER).
• Server — owns the GPU. It runs the model's current LoRA adapter inside vLLM for fast inference, and when you ship it a batch of trajectory groups it runs GRPO, writes a fresh LoRA checkpoint, and hot-loads it back into vLLM.

The cycle is inference → training → inference, initializing from an empty LoRA on iteration one and from the latest checkpoint thereafter. The two phases are deliberately decoupled: your agent code does rollouts at its own pace; the GPU server does the heavy gradient math. LoRA is what makes this affordable — you train a small adapter (millions of params) rather than the full 14B, so a training round fits on modest hardware and adapters are cheap to swap, A/B test, and roll back. Supported backbones cover most vLLM/HF causal LMs that Unsloth backs — Qwen variants, Llama, GPT-OSS — with Gemma 3 currently unsupported.

This ties straight into /finetuning: GRPO/RULER is online, preference-style RL over your own rollouts, the agentic cousin of RLHF. The difference is the data source — not human-labeled comparisons, but your harness's own audit trail, replayed and re-scored.

3.5 When the small model actually wins

The headline: ART's ART·E agent — a Qwen 2.5 14B trained on email-retrieval trajectories — beat OpenAI's o3 at that task. That is not "small models are better." It's narrower and more useful: on one task with a clean reward and enough rollouts, experience on the exact distribution beats general capability. Frontier models are generalists priced for breadth; if your production load is one repetitive shape — triage these tickets, retrieve from this email corpus, fix lint failures in this repo — you are paying generalist prices for a specialist job, and an RL-tuned 14B can dominate on quality, latency, and cost simultaneously. SWE-EVO is the cautionary mirror: even GPT-5.4 scores only ~25% on long-horizon, 21-file software evolution versus 72.8% on SWE-Bench Verified — sustained open-ended reasoning is exactly where narrow RL won't save you.

4. Minimal implementation

A realistic ART rollout-and-train loop for a coding agent, scored by RULER. This is the shape of real ART code — the rollout function is your agent harness; ART just observes it.

import art
from art.rewards import ruler_score_group
 
# 1. The model under training: a LoRA on a small open backbone, served by vLLM.
model = art.TrainableModel(
    name="swe-fixer-v1",
    project="coding-agent",
    base_model="Qwen/Qwen2.5-14B-Instruct",
)
 
async def rollout(task: dict) -> art.Trajectory:
    """One agent run on one task. This is your normal harness loop —
    Read/Grep/Edit/Bash tools, etc. — but completions go through ART's
    OpenAI-compatible client so every message is recorded."""
    traj = art.Trajectory(
        messages_and_choices=[{"role": "system", "content": SWE_SYSTEM_PROMPT}],
        reward=0.0,
    )
    traj.messages_and_choices.append({"role": "user", "content": task["issue"]})
 
    for _ in range(MAX_STEPS):
        client = model.openai_client()                  # points at vLLM + latest LoRA
        chat = await client.chat.completions.create(
            model=model.name,
            messages=traj.messages(),
            tools=TOOL_SCHEMAS,                          # Read, Grep, Edit, Bash...
        )
        choice = chat.choices[0]
        traj.messages_and_choices.append(choice)        # record the assistant turn
        if not choice.message.tool_calls:
            break
        for call in choice.message.tool_calls:          # execute in the sandbox
            result = run_tool(call, repo=task["repo"])
            traj.messages_and_choices.append(
                {"role": "tool", "tool_call_id": call.id, "content": result}
            )
 
    # Hard, verifiable signal when we have one — never throw this away.
    traj.metrics["tests_pass"] = run_test_suite(task["repo"])  # 1.0 / 0.0
    return traj
 
async def train():
    for step in range(NUM_STEPS):
        # GRPO needs a GROUP per task: G rollouts of the SAME task.
        tasks = sample_tasks(batch=8)
        groups = [
            art.TrajectoryGroup([await rollout(t) for _ in range(8)])  # G = 8
            for t in tasks
        ]
        # RULER ranks each group relative to itself -> rewards, no hand-written fn.
        scored = [await ruler_score_group(g, judge_model="o4-mini") for g in groups]
        # GRPO: advantage = (reward - group mean) / group std, then policy update.
        await model.train(scored)   # server runs GRPO, writes + hot-loads new LoRA

What to notice as a harness engineer:

• The rollout function is just your agent. ART is non-invasive — it sits behind an OpenAI-compatible client. If your harness already logs trajectories, you're 80% there.
• The group is the unit, not the trajectory. TrajectoryGroup of 8 rollouts of the same task is what makes the GRPO baseline (the group mean) meaningful. Eight rollouts of eight different tasks would be useless — there's nothing to compare.
• Two signals, layered. tests_pass is a verifiable anchor; RULER fills the gap where tests don't capture quality (style, file-count, no dead code). In production you'd blend them — never let a learned judge be your only reward.
• model.train() hides the GPU. That call ships groups to the server, runs GRPO under Unsloth, checkpoints the LoRA, and reloads vLLM. Your loop just keeps generating.

5. Production tradeoffs

Cost/latency. The economics flip on volume × narrowness. Below some monthly call volume, the GPU and engineering cost of ART never amortizes — keep calling the API. Above it, on a stable task, a self-hosted RL-tuned 14B wins on per-call cost and latency (smaller model, your own serving, no rate limits). The break-even is a real spreadsheet question, not a vibe — model trajectory volume, training GPU-hours, serving cost, and the frontier per-token price you'd otherwise pay.

Quality / failure modes. The dominant production failure is reward hacking: the policy learns to please RULER rather than solve the task. Guard with a held-out verifiable eval the judge never sees, anchor each group with a reference trajectory, and alarm on "RULER reward up, held-out pass rate flat." The second failure is distribution shift — the tuned model is sharp on its training distribution and dull everywhere else, so when product scope widens, quality silently rots. A third, subtler one: wasted compute from zero-variance groups — if every rollout in a group scores identically, advantage is zero and that group teaches nothing; low group diversity (temperature too low, task too easy/hard) quietly stalls training.

What changes at scale. The trajectory pipeline becomes the real product. You need versioned, replayable rollout logs (your harness audit trail, promoted to a data asset), reproducible eval sets, LoRA checkpoint management with rollback, and continuous monitoring for both the held-out metric and reward-hacking signatures. The model weights are almost the easy part; the data and eval loop is the moat.

6. How it's asked

7. Pitfalls & flashcards

• Mixing tasks within a group. GRPO's baseline is the group mean of one task. Putting different tasks in a group makes the baseline meaningless and the gradient noise — always group rollouts of the same instance.
• Zero-variance groups waste compute. If every rollout scores the same, advantage is 0 and the group teaches nothing. Tune temperature and task difficulty for spread.
• Trusting RULER as the only reward. A learned judge will get hacked. Keep a verifiable held-out eval and watch for reward-up / held-out-flat divergence.
• Forgetting the verifiable signal you already have. If unit tests exist, use them — an unfakeable +1/0 is worth more than any judge. Reserve RULER for the quality dimensions tests can't see.
• Over-specializing. The tuned 14B is brittle off-distribution. Track an off-distribution eval so you notice when product scope outgrows the model.
• Treating it as model work, not systems work. The hard, durable part is the trajectory pipeline, eval sets, and LoRA versioning — not the gradient step.

Flashcard. GRPO = PPO minus the critic: advantage is (reward − group_mean) / group_std, so only relative ranking within a same-task group matters — which is exactly why RULER (an LLM ranking sibling trajectories) can supply reward with no labels, and why ART can train a 14B past a frontier model on one narrow task.

8. Further reading

• OpenPipe ART (GitHub) — the framework, RULER, and the ART·E (Qwen 2.5 14B > o3) result.
• ART: Train Agents That Learn From Experience — Daily Dose of DS — trajectory abstraction and the inference/training cycle, explained.
• How Top AI Labs Are Building RL Agents (2026) — GRPO mechanics and where RL on trajectories fits the stack.
• Unsloth RL Guide — Training AI Agents with RL — the memory/compute-efficient GRPO implementation ART builds on.
• Scaling Coding Agents via Atomic Skills (arXiv) — tool-use patterns and skill decomposition for coding agents.
• SWE-EVO: Long-Horizon Benchmarking (arXiv) — the capability wall on open-ended, multi-file evolution that narrow RL won't fix.

Next: /finetuning — how online trajectory RL relates to RLHF, DPO, and the rest of the post-training toolkit.