Building an Eval Harness in CI

1. Quick anchor

An eval harness in CI is the machine that turns "we think this prompt is better" into a number that either lets a PR merge or blocks it. Think of it as a test suite where the assertions are statistical, not deterministic: instead of assertEqual, you have assert ndcg@10 >= baseline - 0.02. The hard part is almost never the metric math — it's the operational scaffolding around it: versioning the dataset so scores are comparable across commits, capping judge spend so a 5,000-row suite doesn't cost $40 per run, keeping the test set out of the training loop, and wiring online monitoring so the silent regressions (a provider model bump, a new user cohort) page you before users churn. The mental model: offline evals are a merge gate for changes you make; online evals are an alarm for changes that happen to you. Everything below is about building both so they're cheap, trustworthy, and hard to game.

2. Why interviewers probe this

This is the question that separates engineers who use evals from engineers who own the eval platform. The signal differs sharply by level:

• IC5 (senior IC): Can you stand up a regression gate that actually blocks a bad merge? Do you understand relative vs. absolute thresholds, why a single pass/fail on accuracy is naive, and how to keep judge cost and flakiness from making the gate useless? Can you debug "the eval passed but prod broke"?
• IC6 (staff): Can you design the eval org? Dataset governance (who curates golden sets, how they version, how production failures flow back in), contamination defense at scale, the offline/online handoff, alerting thresholds that don't drown the team in false pages, and the political reality that a gate the team routinely overrides is worse than no gate. Staff candidates are expected to name the failure modes — Goodhart on the eval, leakage via feedback loops, benchmark saturation — and design around them.

The tell: weak answers describe a metric. Strong answers describe a system with feedback loops, owners, and known ways it rots.

3. Concept build-up

3.1 The gate as the spec: eval-driven development

Eval-driven development (EDD) is test-driven development for probabilistic systems. In TDD, a failing test is the spec; you write code until it passes. In EDD, a failing eval is the spec, but "passing" is a distribution, not a boolean. Every prompt tweak, model swap, retrieval-chunking change, and tool-schema edit becomes a measurable experiment that runs through the same suite before it ships (Braintrust). The reason eyeball testing fails at scale is interaction: a prompt change that fixes three reported bugs silently breaks a fourth behavior nobody spot-checked. The gate's job is to make those interactions visible in the PR, not in an incident channel.

The non-negotiable architectural decision: the gate compares against a baseline, not an absolute floor. Models and data move; a hardcoded faithfulness >= 0.85 either rots into noise or blocks legitimate improvements. Relative thresholds — "must not degrade more than 2 points versus current production" — are the workhorse, with a few absolute floors for safety-critical dimensions (refusal correctness, PII leakage) (Latitude).

3.2 Dataset versioning is the load-bearing wall

A score is only meaningful relative to the exact dataset that produced it. If engineer A adds 40 rows to the golden set in the same PR that changes the prompt, the metric delta now conflates two things and the gate is lying. Three rules make this work:

1. The dataset is a versioned artifact, hashed and pinned per CI run, stored alongside code/prompt/model versions (CSV/JSON in the repo or a dataset registry like Braintrust). A score row is (commit, dataset_hash, model_id, prompt_hash, metric, value).
2. Dataset changes and system changes ship in separate PRs. Adding rows re-baselines; changing the prompt is measured against a frozen dataset. Never both.
3. Golden sets grow from production failures. Every escalation, thumbs-down, and incident becomes a labeled row. This is the single highest-leverage habit — the dataset that catches tomorrow's regression is built from today's outages (Arize).

3.3 The judge-cost budget and caching

If you grade with an LLM judge, cost and latency become first-class engineering constraints. A 5,000-row suite with a chain-of-thought judge at ~1,500 tokens in / 400 out per row is roughly 9.5M tokens per run; at frontier-judge pricing that's real money per PR, multiplied by every push. Four levers, in order of leverage:

1. Cache by content hash. Key the judge result on hash(system_output + reference + rubric_version). If the output didn't change, the score is cached — most PRs touch one feature and leave 95% of outputs byte-identical, so cache hit rates of 90%+ are normal. This alone is the biggest win.
2. Tier the suite. Run a fast ~200-row smoke set on every push (deterministic metrics + a cheap judge); run the full judge suite on PR-to-main and nightly. Don't pay for the full grade on every WIP commit.
3. Use deterministic metrics where you can. Retrieval (nDCG, recall@k, MRR) and schema/tool-validity checks need no judge at all. Reserve the expensive judge for faithfulness, relevance, and subjective quality where there's no programmatic oracle.
4. Cheaper judge model + spot-check correlation. A smaller judge model often correlates >0.9 with a frontier judge on a given rubric; validate that correlation on a held-out human-labeled slice, then run the cheap judge in CI and the expensive one nightly.

Budget it explicitly: a gate that costs $40 and 25 minutes per run will get disabled by the team within a month, and a disabled gate catches nothing.

3.4 Contamination, leakage, and saturation

Three forces quietly invalidate eval scores, and staff candidates are expected to name all three.

Test-set leakage into pretraining. Public benchmark items and solutions are all over the internet-scale training corpora, so a model may have memorized the answers. String-match detection fails because paraphrases evade it (arXiv 2505.18102). Defense: prefer production-derived golden sets over public benchmarks for anything gate-worthy, and rephrase items to detect concealed contamination (if a paraphrase tanks the score, the original was memorized).

Leakage via your own feedback loop. The subtler killer: you tune prompts/models iteratively against the offline set, then evaluate on that same set. Even a private test set overfits through this loop — you're doing gradient descent by hand on the eval (arXiv 2311.04850). Defense: keep a locked holdout that's only ever run at release, never during iteration, and rotate fresh items in from production.

Benchmark saturation. As a metric approaches its ceiling, it stops discriminating — the gap between two systems shrinks into the noise floor, and construction artifacts dominate. A saturated eval gives false confidence. Defense: retire saturated metrics, add harder production-grounded cases, and watch for the smell that "everything passes at 0.98."

3.5 The offline/online handoff

Offline and online evals are not redundant — they catch disjoint classes of failure:

• Offline catches what your team changes: prompt edits, model upgrades you initiate, retrieval changes. It's the merge gate.
• Online catches what happens to you: a provider silently bumps a model version, a new user cohort arrives with input patterns your golden set never saw, slow concept drift in what users expect (Deepchecks).

The handoff is bidirectional: online monitoring surfaces novel failures, which become golden rows, which strengthen the offline gate. A harness without the online half will pass every PR while quality erodes from forces no merge introduced.

4. Minimal implementation

A real, runnable CI gate. It loads a pinned, hashed dataset, runs the system, computes a deterministic retrieval metric plus a cached judge metric, compares against a stored baseline, and exits non-zero to block the merge. This is the shape of the thing — swap run_system and judge for your stack.

# eval_gate.py — run in CI; exit 1 blocks the merge.
import hashlib, json, os, sys
from pathlib import Path
 
TOL = {"ndcg@10": 0.02, "faithfulness": 0.03}  # max allowed drop vs baseline
ABS_FLOOR = {"faithfulness": 0.80}              # hard safety floor
 
def load_dataset(path="golden/v7.jsonl"):
    rows = [json.loads(l) for l in Path(path).read_text().splitlines()]
    h = hashlib.sha256(Path(path).read_bytes()).hexdigest()[:12]
    return rows, h
 
def ndcg_at_10(retrieved_ids, relevant_ids):
    import math
    dcg = sum(1.0 / math.log2(i + 2)
              for i, rid in enumerate(retrieved_ids[:10]) if rid in relevant_ids)
    ideal = sum(1.0 / math.log2(i + 2)
                for i in range(min(len(relevant_ids), 10)))
    return dcg / ideal if ideal else 0.0
 
# --- judge with a content-addressed cache so unchanged outputs cost $0 ---
CACHE = Path(".eval_cache/judge.json")
_cache = json.loads(CACHE.read_text()) if CACHE.exists() else {}
 
def judge_faithfulness(answer, context, rubric_version="r3"):
    key = hashlib.sha256(f"{answer}||{context}||{rubric_version}".encode()).hexdigest()
    if key in _cache:
        return _cache[key]                      # cache hit: no API call
    score = call_llm_judge(answer, context)     # your judge call (CoT -> 0..1)
    _cache[key] = score
    return score
 
def run_eval():
    rows, dhash = load_dataset()
    ndcgs, faiths = [], []
    for r in rows:
        out = run_system(r["query"])            # your pipeline under test
        ndcgs.append(ndcg_at_10(out["retrieved_ids"], r["relevant_ids"]))
        faiths.append(judge_faithfulness(out["answer"], out["context"]))
    CACHE.parent.mkdir(exist_ok=True)
    CACHE.write_text(json.dumps(_cache))
    return {"ndcg@10": sum(ndcgs)/len(ndcgs),
            "faithfulness": sum(faiths)/len(faiths)}, dhash
 
def gate(candidate, dhash):
    baseline = json.loads(Path("baselines/main.json").read_text())
    assert baseline["dataset_hash"] == dhash, \
        "Dataset changed vs baseline — re-baseline in a separate PR before gating."
    failed = []
    for m, val in candidate.items():
        drop = baseline["metrics"][m] - val
        if drop > TOL[m]:
            failed.append(f"{m}: {val:.3f} regressed {drop:.3f} (> {TOL[m]})")
        if m in ABS_FLOOR and val < ABS_FLOOR[m]:
            failed.append(f"{m}: {val:.3f} below floor {ABS_FLOOR[m]}")
    return failed
 
if __name__ == "__main__":
    cand, dhash = run_eval()
    print(json.dumps(cand, indent=2))
    fails = gate(cand, dhash)
    if fails:
        print("EVAL GATE FAILED:\n  " + "\n  ".join(fails)); sys.exit(1)
    print("EVAL GATE PASSED"); sys.exit(0)

The load-bearing details: the dataset hash assertion forbids comparing across different datasets — it forces dataset changes into their own re-baselining PR. The judge cache is content-addressed, so a PR that touches one prompt re-grades only the rows whose outputs actually changed. Both relative tolerances and absolute floors apply — faithfulness can't quietly slide and can't drop below the safety floor. In a real setup you'd run this under a framework that gives you tracing and a log viewer (Inspect AI's Dataset → Task → Solver → Scorer primitives, adopted by Anthropic and DeepMind, or DeepEval's pytest-native metrics for CI integration), but the control flow is exactly this (Inspect AI).

5. Production tradeoffs

Cost/latency. The gate must run in the time and money budget of a normal CI check — single-digit minutes, cents-to-low-dollars per PR — or engineers route around it. Caching and tiering are not optimizations; they're survival.

Quality / statistical honesty. A 1-point metric move on a 200-row suite is often noise. At scale you report confidence intervals and gate on whether the regression is significant, not just negative — otherwise the gate flaps red on random seeds and the team learns to retry until green, which is functionally a disabled gate.

Failure modes that bite in production:

• Goodhart / gaming the eval. Once a number gates merges, people optimize the number. A verbose answer scores higher with a verbosity-biased judge; a model can self-favor if it judges its own family. Rotate held-out cases and audit the judge against humans.
• The override spiral. A flaky or slow gate gets an "override" button, which becomes the default click. Track override rate as a health metric — a frequently-overridden gate is worse than none because it launders bad merges with a green checkmark.
• Stale baseline. If baselines aren't refreshed on every green main merge, candidates drift away from reality and the gate either over- or under-fires.
• Online blind spot. No production sampling means a silent provider model bump degrades quality for weeks with every offline run still green.

6. How it's asked

7. Pitfalls & flashcards

• Conflating dataset and system changes in one PR — the metric delta becomes uninterpretable. Hash the dataset and assert it's unchanged at gate time.
• No statistical significance — gating on a raw negative delta makes the suite flap on random seeds; report CIs and gate on significance.
• Hardcoded absolute thresholds that rot — prefer relative-to-baseline with a few safety floors.
• Uncapped judge cost — a $40, 25-minute gate gets disabled. Cache by content hash, tier the suite, use deterministic metrics where possible.
• Feedback-loop leakage — tuning against the same set you evaluate on. Keep a release-only locked holdout.
• No online half — offline can't see provider drift or new cohorts. Sample production and alert on drift.
• The override spiral — a frequently-overridden gate launders bad merges. Track override rate as a health metric.
• Saturated benchmarks giving false confidence — retire them, add harder production cases.

Flashcard. Offline evals are a merge gate for changes you make; online evals are an alarm for changes that happen to you. A harness needs both, plus a feedback loop that turns production failures into golden rows.

8. Further reading

• Braintrust — Eval-Driven Development: why evals are the spec and how they gate releases.
• Latitude — CI/CD LLM Evaluation Guide: relative thresholds, quality gates, versioned results.
• Inspect AI (UK AISI): Dataset→Task→Solver→Scorer primitives, sandboxed agent evals, log viewer.
• arXiv 2505.18102 — Data contamination and 2311.04850 — feedback-loop overfitting: why scores lie.
• VentureBeat — Monitoring drift, retries, refusals and AWS — Production monitoring & drift: online signals.

Next: /evals/llm-as-judge — calibrating the judge whose scores this whole gate depends on.