Why Evals, and the Loop That Keeps Them Honest

1. Quick anchor

For deterministic code, the source is the spec: read it, and you know what it does. For an LLM system, the source is a prompt plus weights you didn't train, and the same input can produce different outputs across calls and silently different outputs across provider updates. So the spec has to live outside the code, as a set of input/expected-behavior pairs you can re-run: an eval suite. Eval-driven development (EDD) makes that suite the working specification — every prompt tweak, model swap, retrieval change, and chunking decision becomes a measured experiment against it, the same way TDD makes tests the spec for deterministic systems. The loop has two halves that catch different failures: offline evals (curated golden data, run in CI) catch regressions you introduce; online evals (production traffic) catch changes that happen to you — provider drift, new user cohorts, shifting topics. And the single highest-leverage move in a retrieval system is to evaluate retrieval and generation separately, because a wrong final answer has two completely different root causes and one number can't tell you which.

2. Why interviewers probe this

• IC4 — Can you articulate why eyeballing fails for non-deterministic systems, and name the right metric for the right layer (retrieval vs. generation, balanced vs. imbalanced)? Do you reach for a measurement before a vibe?
• IC5 — Can you build the loop: source a golden dataset from real failures, wire a CI gate with a sane threshold, and decompose a quality problem into retrieval-vs-generation? Can you reason about dataset rot, leakage, and the cost of running judges on every PR?
• IC6 — Can you design the whole system — offline/online split, what each catches and what neither does, drift detection, the feedback loop that grows the golden set — and defend it against the failure mode where your evals look great while users suffer? Do you treat the eval suite itself as a system with bugs, contamination, and maintenance cost?

3. Concept build-up

3.1 Why "it looks better" is not evidence

A deterministic test passes or fails the same way every run. An LLM output is a sample from a distribution, so a single good demo is one draw from a coin you haven't characterized. Worse, changes interact: a prompt edit that fixes the case you were staring at can degrade ten cases you weren't. The reason eyeball testing feels productive and is actually dangerous is that you over-fit to the one example in front of you and have no visibility into the rest of the distribution. EDD replaces "I looked and it seemed fine" with "the suite of 300 cases moved from 0.81 to 0.84 with no individual case dropping more than 0.05." That is the difference between an anecdote and a measurement, and it's the same epistemic move TDD made twenty years ago (Braintrust; arXiv 2411.13768).

The corollary that trips up IC4s: if the eval doesn't capture quality, optimizing it makes things worse with full confidence. An eval is a proxy for the target construct ("is this a good support answer"). The whole discipline is keeping that proxy honest as the system and the world change.

3.2 Offline vs. online — two halves, two failure classes

These are not redundant; they catch disjoint failure modes.

Offline evaluation runs on a fixed, curated golden set with controlled inputs. Its job is CI/CD gating: catch the regressions the team introduces — a prompt change, a model upgrade, a new chunking strategy, a retriever config. Because the inputs are fixed, a score change means your code changed the behavior. Clean signal, fully reproducible, cheap to attribute.

Online evaluation runs on raw production traffic — messy, adversarial, novel, shifting. Its job is to catch changes that happen to the product: the provider silently ships a new model snapshot, a new customer cohort arrives with input patterns nobody anticipated, the topic mix drifts, refusal rates spike. None of these touch your golden set, so offline is blind to them (Arize; Deepchecks).

The synergy: offline gives you reproducible gates; online gives you a freshness signal and a source of new failures. The classic trap (a great IC6 question) is a team with only offline evals that look perfect while users complain — because the failures live entirely in the distribution shift that the frozen golden set can't see.

3.3 Building a golden dataset from production failures

This is the part candidates hand-wave and interviewers dig into. A golden dataset is human-annotated, sourced from production failures, edge cases, and user-reported issues — not random samples and definitely not LLM-generated "synthetic" cases as your only source (those encode the model's blind spots into your spec).

Practical shape for a real product, roughly 200–500 curated examples to start:

• Stratify, don't sample uniformly. Bucket by intent/topic, by difficulty, and explicitly by known failure mode. A flat random sample over-represents easy head traffic and hides the tail where you actually lose users.
• Mine failures. Production thumbs-down, escalations to a human, explicit user corrections, and bug reports are gold — each is a real input where the system was wrong, with a free signal that it was wrong.
• Annotate the expected behavior, not just the answer. For retrieval, mark which documents are relevant. For generation, mark the correct answer and the must-cite facts. For agents, mark the correct trajectory.
• Keep it evolving. New failure patterns surface in online monitoring; promote them into the golden set. A frozen golden set rots: the model and the traffic move past it, and your "regression gate" starts protecting behavior nobody cares about anymore.

Two contamination hazards to name in an interview: (1) leakage — if these exact items (or paraphrases) are in pretraining data, your score is inflated and string-matching won't detect the paraphrase; (2) feedback-loop overfitting — if you iteratively tune on the same offline set, even a private one, you overfit to it through the loop. Mitigations: rephrased/paraphrased held-out variants, fresh production-derived sets, and treating "production distribution shift" as a natural freshness signal (arXiv 2505.18102).

3.4 Evaluate retrieval and generation SEPARATELY

The single most important architectural rule for RAG/agent evals. A wrong final answer has two root causes that need opposite fixes:

1. Retrieval failed — the relevant context wasn't in the top-k. No prompt or model change fixes this; you fix the retriever, embeddings, or chunking.
2. Generation failed — the right context was retrieved but the model ignored it, hallucinated, or contradicted it. No retriever change fixes this; you fix the prompt, model, or grounding.

If you only measure end-to-end answer quality, you can't tell these apart and you'll burn weeks tuning the wrong layer. So you split:

• Retrieval layer — order-aware and coverage metrics: Recall@k (did we fetch the relevant docs at all), nDCG@k (are they ranked well), MRR (did the first relevant doc come early), plus RAGAS-style context precision (fraction of retrieved chunks that are relevant) and context recall (did we get all the necessary chunks).
• Generation layer — given the retrieved context as fixed, measure faithfulness (fraction of answer claims supported by context, typically via NLI or an LLM judge) and answer relevancy (does the answer address the query).

A powerful diagnostic move: run generation on gold context (the human-annotated correct documents). If the answer is now good, your bug is in retrieval. If it's still bad, it's generation. You've isolated the layer in one experiment (RAGAS docs).

3.5 The loop: dev → PR → regression gate → prod

The pipeline that operationalizes all of the above:

1. Dev — engineer iterates locally against a fast subset of the golden set (cheap scorers, maybe 30–50 cases) for tight feedback.
2. PR / CI — the full offline suite runs on every pull request. Results are stored per run (JSON/CSV, versioned alongside the prompt and model id) so changes are diffable.
3. Regression gate — the build fails if a metric breaches its threshold. Prefer relative thresholds ("must not drop more than 2% vs. the current production baseline") over brittle absolute ones, because absolute targets fight you as the suite evolves. Per-metric, per-business: e.g., faithfulness ≥ 0.85, nDCG@5 ≥ baseline − 0.02, p95 latency < 2s, cost/query < $0.01 (Latitude).
4. Prod — after promotion, online evals sample real traffic, score it, and watch for drift. New failures flow back into step 0 (the golden set), closing the loop.

This is the EDD discipline: prompt changes, model swaps, and code updates promote only if they pass the gates. Minor tweaks can't silently break a feature, because every change runs the same suite before it ships.

4. Minimal implementation

A runnable harness that does the two non-negotiables: separate retrieval from generation, and gate on a relative threshold vs. baseline. No framework needed to see the shape; the same structure is what DeepEval/Inspect/RAGAS formalize.

import json
import math
from dataclasses import dataclass
from pathlib import Path
 
@dataclass
class GoldenCase:
    query: str
    relevant_doc_ids: set[str]   # human-annotated: the docs that SHOULD be retrieved
    must_cite_facts: list[str]   # claims a faithful answer must be grounded in
 
# --- Retrieval-layer metrics (order-aware + coverage) ---
 
def recall_at_k(retrieved_ids: list[str], relevant: set[str], k: int) -> float:
    if not relevant:
        return 1.0
    top_k = set(retrieved_ids[:k])
    return len(top_k & relevant) / len(relevant)
 
def ndcg_at_k(retrieved_ids: list[str], relevant: set[str], k: int) -> float:
    dcg = sum(
        1.0 / math.log2(i + 2)                      # i+2 because positions are 1-indexed: log2(pos+1)
        for i, doc_id in enumerate(retrieved_ids[:k])
        if doc_id in relevant
    )
    ideal_hits = min(len(relevant), k)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(ideal_hits))
    return dcg / idcg if idcg > 0 else 0.0
 
# --- Generation-layer metric (faithfulness as claim support) ---
# In production this 'supports' call is an NLI model or an LLM judge.
# Stubbed here so the harness is deterministic and runnable.
 
def faithfulness(answer_claims: list[str], context: str, supports) -> float:
    if not answer_claims:
        return 0.0
    supported = sum(1 for c in answer_claims if supports(c, context))
    return supported / len(answer_claims)
 
def evaluate(suite: list[GoldenCase], system, supports, k: int = 5) -> dict:
    """system(query) -> (retrieved_ids, context_text, answer_claims)."""
    recalls, ndcgs, faiths = [], [], []
    for case in suite:
        retrieved_ids, context, claims = system(case.query)
        recalls.append(recall_at_k(retrieved_ids, case.relevant_doc_ids, k))
        ndcgs.append(ndcg_at_k(retrieved_ids, case.relevant_doc_ids, k))
        faiths.append(faithfulness(claims, context, supports))
    return {
        "recall@k": sum(recalls) / len(recalls),
        "ndcg@k":   sum(ndcgs) / len(ndcgs),
        "faithfulness": sum(faiths) / len(faiths),
    }
 
# --- The regression GATE: relative threshold vs. stored baseline ---
 
def gate(current: dict, baseline_path: str, max_drop: float = 0.02) -> bool:
    baseline = json.loads(Path(baseline_path).read_text())
    failed = False
    for metric, value in current.items():
        drop = baseline[metric] - value
        status = "OK"
        if drop > max_drop:                 # relative threshold, not absolute
            status, failed = "REGRESSION", True
        print(f"{metric:14s} {value:.3f}  (baseline {baseline[metric]:.3f}, "
              f"drop {drop:+.3f})  {status}")
    return not failed   # False -> CI exits non-zero -> PR blocked

What matters here, not the syntax:

• The metrics are computed at two layers. recall@k/ndcg@k judge the retriever in isolation; faithfulness judges the generator given whatever context it received. A failing run tells you which layer to fix.
• ndcg_at_k uses i + 2 because positions are 1-indexed and the discount is log2(position + 1) — the off-by-one here is a real bug source.
• The gate is relative. It compares to a stored baseline and blocks only on a significant drop, so normal sampling noise doesn't redflag every PR while a genuine 5-point regression does.
• supports is the swap point. In dev it's a stub; in CI it's an NLI classifier (cheap, deterministic) or an LLM judge (expensive, higher human-correlation, biased — see §5).

In a real stack you'd run this under a framework: Inspect AI (Dataset → Solver → Scorer, sandboxed, adopted by Anthropic/DeepMind) for agentic and multi-turn, DeepEval for pytest-native CI with 50+ metrics, RAGAS for the four-metric RAG core, and Braintrust to connect offline scores with production traces (Inspect AI).

5. Production tradeoffs

Prose on what changes at scale:

• Cost compounds on every PR. An LLM judge over a 400-case suite, run on every push by every engineer, is real money and real minutes. Standard pattern: a cheap subset (NLI/embedding scorers) gates every PR for fast feedback, and the expensive LLM-judge full suite runs nightly or on release branches. Don't put a slow judge in the inner dev loop.
• LLM judges carry systematic, correlated bias — which is the dangerous kind because it doesn't average out. Verbosity bias: judges prefer longer answers (more surface area to find supporting phrases). Position bias: in rubric or pairwise setups, judges favor options at certain positions independent of content. Self-enhancement bias: a judge favors outputs from its own model family. Mitigations: balanced-permutation rubrics (distribute each score option across all positions and aggregate), randomize candidate order and average, prefer pairwise over pointwise (lower calibration variance), and ensemble judges from different model families (arXiv 2602.02219).
• Non-determinism in the judge means your eval has noise. A flaky judge produces a flaky gate. Pin the judge model/version, set temperature low, and treat the judge itself as something you must calibrate against human labels — measure judge-human agreement on a held-out slice before you trust the number.
• Your golden set saturates and rots. As the system improves, easy cases all pass and the suite stops discriminating; as traffic drifts, the suite stops representing reality. At scale this means a standing process: continuously mine production failures into the set, paraphrase-refresh to fight leakage, and retire stale cases.
• Online eval can't be exhaustive. You can't human-label all of production. So you sample, you rely on reference-free metrics (faithfulness, schema conformity) plus cheap proxies (refusal rate, malformed-output rate, embedding KL-divergence vs. baseline) as drift alarms, and you escalate to human review only on the flagged slice.

6. How it's asked

7. Pitfalls & flashcards

• One end-to-end number for a RAG/agent system. It can't tell retrieval failure from generation failure; the two need opposite fixes. Always decompose.
• Accuracy on imbalanced data. A "never fraud" classifier hits 98% accuracy and catches zero fraud. Use F1, PR-AUC, ROC-AUC — never raw accuracy alone.
• Putting a slow LLM judge in the inner dev loop. Kills iteration speed and burns budget. Cheap scorers gate every PR; the judge runs nightly/on-release.
• Trusting an uncalibrated judge. Verbosity, position, and self-enhancement biases are correlated, so they don't average out. Measure judge-human agreement, use pairwise + balanced permutations + ensembles.
• A frozen golden set. It saturates and drifts out of relevance; your "regression gate" ends up protecting behavior nobody uses. Continuously mine production failures back in.
• Offline-only. Looks perfect while users suffer, because provider drift and new cohorts live entirely outside the curated set. Pair offline gates with online monitoring.
• Tuning against the gating set. Feedback-loop overfitting inflates the score; keep a fresh, never-optimized-against held-out slice.

Flashcard. Offline evals catch the regressions you introduce (prompt/model/code changes, on a fixed golden set); online evals catch the changes that happen to you (provider drift, new cohorts, topic shift, on production traffic). You need both, and in RAG you score retrieval and generation separately.

8. Further reading

• Braintrust — Eval-Driven Development — EDD as the TDD analogue; evals as the working spec.
• arXiv 2411.13768 — eval-driven development for LLM apps — the methodology framing.
• Arize — LLM evaluation and Deepchecks — online vs. offline — the two-paradigm split and what each catches.
• Inspect AI (UK AISI) — Dataset → Solver → Scorer primitives for agentic/multi-turn evals; adopted by Anthropic and DeepMind.
• Latitude — CI/CD LLM evaluation guide — regression gates, relative thresholds, versioned results.

Next: Retrieval metrics in depth — Recall@k, MRR, nDCG, MAP@k and LLM-as-judge: biases and calibration.