Agent & RAG Evals: Component-Level Scoring That Survives Production

1. Quick anchor

A RAG answer and an agent run are both pipelines of things that can fail independently, and the entire job of evaluation is to attribute a bad output to the specific stage that produced it. For RAG that means scoring retrieval (did the right chunks come back?) and generation (did the model use them honestly?) as separate numbers, because a hallucinated answer and a missing-context answer demand opposite fixes. For agents it means scoring the trajectory (right tools, right order, valid arguments) separately from the outcome (task completed), because a correct answer reached through the wrong path is a brittle answer waiting to break. A single end-to-end "was it good?" score tells you nothing actionable; component-level evals turn a regression into a pointer. Build these as a runnable harness with CI gates so every prompt tweak, model swap, and chunking change runs the same suite before it ships.

2. Why interviewers probe this

• IC5 — can you decompose? They want to see you refuse a single opaque "answer quality" metric and instead name the components (retrieval vs generation; trajectory vs outcome), pick the right metric per component, and explain what a low score on each one means operationally. They're checking whether you'd ship blind or with attribution.
• IC5 — do you know the failure modes of the judge? RAGAS and LLM-as-judge are not ground truth. If you can't name position bias, verbosity bias, and self-enhancement bias — and the fact that RAGAS cannot verify whether the source corpus is even true — you'll over-trust a number and ship a regression.
• IC6 — can you build the system, not the metric? Offline gates plus online monitoring, golden sets sourced from production failures, contamination defenses, and the specific signal that catches a silent provider update. They want the org-level eval strategy, the cost/latency budget of judges in CI, and how you keep the eval itself from rotting.
• All levels — honesty. The strongest signal is admitting where the eval lies: token-overlap faithfulness misfires on paraphrase, exact-match trajectory punishes valid alternate paths, and a saturated benchmark stops discriminating. Naming the lie is the senior move.

3. Concept build-up

3.1 RAG decomposes into a retrieval ceiling and a generation honesty check

The single most useful mental model: context recall is the ceiling, faithfulness is the floor. If the fact needed to answer never made it into the retrieved context (low context recall), the generator cannot be right no matter how good it is — you have a retrieval bug, and tuning the prompt is wasted effort. If the context was there but the answer asserts things the context doesn't support (low faithfulness), you have a generation bug — the model is hallucinating, and you fix it with grounding instructions, a smaller temperature, or a better model, not with better retrieval.

The four RAGAS metrics map cleanly onto the two stages (RAGAS docs):

All four are reference-free — they need no gold answer, only the question, the retrieved context, and the generated answer (context recall additionally needs a human-marked set of necessary chunks). That's what makes RAGAS cheap to run on production traffic.

The mechanism for faithfulness is worth understanding precisely because it's where interviewers push. You extract atomic factual claims from the answer (ideally with an LLM; cheaply with sentence-splitting), then for each claim run a Natural Language Inference (NLI) check against the context: is this claim entailed, contradicted, or neutral relative to the context? Faithfulness is the fraction entailed. NLI catches logical contradictions that pure embedding similarity misses — "the tower is 200m tall" and "the tower is 300m tall" are embedding-similar but NLI-contradictory (FutureAGI NLI).

The critical limitation, and a classic IC6 trap: RAGAS operates only at the inference layer. Faithfulness asks "is the answer grounded in the retrieved context" — it does not ask "is the retrieved context true." If your corpus contains a wrong fact, a perfectly faithful answer repeats it and scores 1.0. Independent benchmarking found no RAGAS-style tool reliably distinguished factually correct from incorrect contexts (Atlan). Source trustworthiness is a separate, upstream problem.

3.2 Alternative RAG framings: the TruLens triad and where to draw boundaries

TruLens reframes the same idea as the RAG triad: context relevance (query↔context), groundedness (answer↔context, ≈ faithfulness), and answer relevance (answer↔query) (Atlan). Same two-stage decomposition, different names. Its real edge is OpenTelemetry-based span-level tracing, which lets you localize failures inside a multi-step agentic pipeline rather than scoring only the final output. The cost is a steeper learning curve.

The boundary that matters in interviews: answer relevancy and faithfulness are orthogonal. An answer can be faithful (every claim grounded) but irrelevant (grounded in the wrong part of the context, or evasive). It can be relevant (clearly about the question) but unfaithful (confidently wrong). You need both numbers; neither subsumes the other. This orthogonality is the answer to the IC4 question below.

3.3 Agents: score the trajectory, not just the destination

Agent evaluation's central insight is that the final answer is necessary but not sufficient. An agent that returns the right answer by calling a payments API it should never have touched, or by lucking into the answer after three wrong tool calls, has a broken trajectory — and that brittleness shows up as a production incident the moment inputs shift. So you score three layers (Confident AI; Medium Vinod):

1. Trajectory accuracy — did the agent take a correct path? Not just which tools, but the order, the reasoning, and the decision sequence. Reaching the right answer via wrong tools is a trajectory failure despite a correct outcome.
2. Tool correctness — three sub-checks: selection (right tool chosen?), argument accuracy (parameters syntactically and semantically valid?), execution validation (did the tool output match expectations?). Tool correctness correlates strongly with final-answer accuracy and is the single most diagnostic agentic signal.
3. Task completion (a.k.a. goal accuracy / task success) — binary or scalar: did the agent accomplish the stated user goal? This is the outcome, measured as a success rate across trajectories, often with intermediate checkpoints for multi-step tasks.

Layer in the non-quality dimensions that gate production: system efficiency (token usage, latency, tool-call count, API cost) and behavioral signals (action advancement — is each step making progress; agent flow — is it following the intended workflow). These are measured at two granularities: session level (the whole trajectory) and node level (each individual step). When something regresses, node-level scores tell you which step failed.

The hard design choice — and a frequent IC6 question — is how strict to make trajectory scoring. Exact tool-sequence match is brittle: many valid paths reach the same goal, and a too-strict scorer fails good agents (false negatives that train you to chase noise). The harness below uses longest common subsequence (LCS) of called-vs-expected tools, normalized by expected length: it rewards the right tools in the right relative order while tolerating extra detours. That's a deliberate tradeoff — LCS forgives a redundant call but still penalizes a wrong order or a skipped step. For open-ended agents you often graduate to an LLM-as-judge over the trajectory ("given the goal and these steps, was this a reasonable path?"), accepting the judge's bias and cost in exchange for flexibility.

3.4 The judge is biased — design around it

Whenever a metric is computed by an LLM (faithfulness via NLI-prompting, G-Eval rubrics, trajectory judging), you inherit three systematic biases you must name in an interview (Arxiv 2602.02219):

• Position bias — in rubric/pairwise settings the judge prefers options at specific positions (start or end of the list), independent of content. Mitigation: balanced permutation — evenly distribute each score option across all positions and aggregate; this both reveals the latent bias and improves correlation with human judges. Or simpler: randomize candidate order and average across randomizations.
• Verbosity bias — judges equate longer with better, because more text gives more surface to find supporting phrases. Mitigation: explicit anti-length instructions, reference-based metrics that penalize padding, or ensembling judges with differing verbosity preferences.
• Self-enhancement bias — a judge favors outputs from its own model family (in-group preference). Mitigation: ensemble judges from different model families, anonymize the generation source, or use an external reference model as judge.

A senior tell: pairwise comparison has lower calibration variance than pointwise absolute scoring. Asking "is A better than B?" is more stable than asking "rate A from 1–10," because absolute scales drift run-to-run. When you can afford it, prefer pairwise for ranking and reserve pointwise for cheap pass/fail gates.

4. Minimal implementation

The harness in examples/evals/eval_harness.py implements all of the above with a pluggable judge: the default StubJudge is deterministic token-overlap so the suite runs offline in CI with no API key, and you swap in a real model (AnthropicJudge-shaped: claim extraction + NLI entailment) for production fidelity. The companion file examples/evals/metrics.py holds the from-scratch retrieval/classification primitives (ndcg_at_k, recall_at_k, roc_auc).

# RAG: retrieval ceiling vs generation honesty, scored separately.
def faithfulness(sample: RagSample, judge: Judge) -> float:
    """RAGAS faithfulness = supported_claims / total_claims. Reference-free."""
    claims = split_claims(sample.answer)              # 1 claim per sentence (cheap)
    if not claims:
        return 0.0
    context = "\n".join(sample.retrieved)
    supported = sum(1 for c in claims if judge.entails(c, context))  # NLI per claim
    return supported / len(claims)
 
def context_recall(sample: RagSample, judge: Judge) -> float:
    """Coverage: fraction of NECESSARY (gold) chunks actually retrieved.
    This is the upper bound on RAG quality — if a fact never reached context,
    no prompt fix can recover it."""
    if not sample.gold_contexts:
        return 1.0
    retrieved = "\n".join(sample.retrieved)
    covered = sum(1 for g in sample.gold_contexts if judge.entails(g, retrieved))
    return covered / len(sample.gold_contexts)
 
# Agent: order-aware path score, tolerant of detours, strict on order.
def trajectory_accuracy(case: AgentCase) -> float:
    """LCS(called tools, expected tools) / len(expected). Rewards right tools
    in right order; forgives extra calls; punishes skips and reorderings."""
    called, expected = [c.name for c in case.trace.calls], case.expected_tools
    if not expected:
        return 1.0
    m, n = len(called), len(expected)
    dp = [[0] * (n + 1) for _ in range(m + 1)]
    for i in range(1, m + 1):
        for j in range(1, n + 1):
            dp[i][j] = (dp[i-1][j-1] + 1 if called[i-1] == expected[j-1]
                        else max(dp[i-1][j], dp[i][j-1]))
    return dp[m][n] / len(expected)

The harness then runs each metric over a dataset, averages it, and applies CI regression gates (faithfulness >= 0.80, context_recall >= 0.90, task_completion >= 1.0), emitting a machine-readable ci_report.json for artifact storage and dashboards, and exiting non-zero if any gate fails. Running it on the bundled dataset — which deliberately plants one hallucinated claim ("tallest building in the world") in sample 2 — produces:

=== RAG component evals ===
  faithfulness          0.417  FAIL (gate >= 0.8)
  answer_relevancy      0.625  PASS (no gate)
  context_precision     0.583  PASS (no gate)
  context_recall        1.000  PASS (gate >= 0.9)
=== Agent evals ===
  trajectory_accuracy   1.000  PASS (no gate)
  tool_correctness      1.000  PASS (gate >= 0.9)
  task_completion       1.000  PASS (gate >= 1.0)
GATE FAILED: 1 metric(s) below threshold.

Read the attribution: context_recall is 1.0 (retrieval fetched the needed chunk) but faithfulness is 0.417 (the answer hallucinated) — the failing layer is the generator, not retrieval. That's the whole point of component-level scoring. Note also that the StubJudge's token-overlap is conservative: it rejects valid paraphrases, so 0.417 understates the true faithfulness. That's a feature in CI (no flaky API, deterministic) and a reminder for production: swap in a real NLI model before trusting the absolute number. In a real pipeline this main() is your pytest body — DeepEval and Inspect AI wrap exactly this loop with typed metric objects and richer scorers.

5. Production tradeoffs

The dominant production cost is the judge in CI. An LLM judge on a 500-case suite, run on every PR, is real money and 30–120s of latency; teams cache deterministic judgments keyed on (input, model, prompt) hash, run the full judged suite nightly, and gate PRs on a cheaper deterministic subset. Budget gates concretely — e.g. faithfulness > 0.85, latency < 2s, cost < $0.01/call (Latitude).

The dominant quality failure mode is the eval lying to you. Three flavors: (1) the judge bias of §3.4 inflating or deflating scores; (2) benchmark saturation — as a metric approaches its ceiling it stops discriminating, and further "gains" don't transfer to production; (3) data contamination — test items leak into pretraining, and paraphrase evades string-match detection, so a model "passes" by memorization. Defenses: rephrased/paraphrased benchmarks to expose concealed contamination, production-derived fresh golden sets (the distribution shift is itself a freshness signal), and treating any single offline number with suspicion (Arxiv 2505.18102).

At scale the architecture splits in two. Offline evals gate CI/CD on a curated golden set — they catch regressions the team introduces (a prompt edit, a model swap). Online evals run a sampled subset of the same metrics on live production traffic — they catch changes that happen to the product: a silent provider model update, a new user cohort with novel inputs, behavioral drift. The two are synergistic, not redundant: offline alone misses the silent provider update; online alone can't gate a deploy (Arize). Online drift is detected via input-embedding KL divergence / Population Stability Index (data drift) and behavioral signals — refusal-rate spikes, schema-violation rate, tool-call error rate (concept drift). When drift fires: increase offline eval frequency, fold the new production data into the golden set, prompt-tune or retrain on the shifted distribution, and re-baseline if it improves.

6. How it's asked

7. Pitfalls & flashcards

• Single end-to-end score with no attribution. "Answer quality 0.7" tells you nothing to fix. Always decompose: retrieval vs generation, trajectory vs outcome.
• Trusting faithfulness as a truth check. It only checks grounding in retrieved context, never whether the corpus is correct. A faithful answer can repeat a corpus lie. Source trust is a separate problem.
• Context recall ignored. If retrieval didn't fetch the needed fact, no generation fix can save you. Recall is the ceiling — check it first when an answer is wrong.
• Over-strict trajectory matching. Exact tool-sequence equality fails valid alternate paths and trains you to chase false negatives. Use LCS or a judge, and tolerate detours while penalizing skips/reorderings.
• Ignoring judge bias. Position, verbosity, and self-enhancement bias are systematic, not noise. Permute, ensemble, anonymize, prefer pairwise.
• Contaminated or saturated benchmarks. Memorized test items and ceiling-bound metrics stop measuring anything. Use paraphrased and production-fresh golden sets.
• Offline-only. Offline can't see a silent provider update or a new user cohort. You need online sampling of reference-free metrics plus drift signals.

Flashcard. RAG: context recall is the ceiling, faithfulness is the floor — recall says the fact made it into context, faithfulness says the answer didn't invent anything; check recall before you touch the prompt. Agents: score the trajectory and the outcome separately — a right answer via the wrong path is a brittle answer.

8. Further reading

• RAGAS metrics documentation — faithfulness, answer relevancy, context precision/recall, reference-free.
• Confident AI: LLM agent evaluation guide — trajectory accuracy, tool correctness, task completion, session vs node level.
• Atlan: LLM eval frameworks compared — RAGAS vs TruLens vs DeepEval, and RAGAS's source-truth limitation.
• Inspect AI (UK AISI) — Dataset → Task → Solver → Scorer; adopted by Anthropic and DeepMind for agentic evals.
• Position bias in rubric-based LLM-judge evaluation (arXiv 2602.02219) — balanced permutation as bias reveal + mitigation.
• Braintrust: eval-driven development — evals as the working spec, CI regression gating.

Next: Eval-driven development & CI regression gates for wiring this harness into the deploy pipeline, and /rag for the retrieval levers that move context precision and recall.