Design an LLM Eval Platform

1. Quick anchor

An eval platform is a compiler for model quality: it takes a frozen set of inputs (the golden set), runs your system against them, scores each output with deterministic checks plus LLM judges, and emits a single verdict — pass, regress, or improve — that a human or a CI gate can act on. The hard parts are not the judges; they are the boring distributed-systems parts wrapped around judges: making runs reproducible, caching judge calls so a 5,000-case suite costs cents instead of dollars, detecting a real regression versus judge noise, and isolating twelve teams on one shared budget. The mental model: treat every eval run as an experiment keyed by (dataset_version, system_version, judge_version), store the per-case results immutably, and compare versions as deltas — never absolute scores in a vacuum. If you can't answer "what changed and by how much, beyond noise" in one query, you have a dashboard, not an eval platform.

2. Why interviewers probe this

• IC5 signal — Can you build the pipeline, not just call a judge? They want golden-set versioning, deterministic-checks-before-LLM-judge ordering, judge caching keyed correctly, a regression gate with a defensible threshold, and explicit cost math. Red flag: "I'd use an LLM to score it" with no calibration, no caching, no noise model.
• IC5 signal — Do you know judges are measurement instruments that must themselves be validated? They probe judge-human agreement, calibration sets, pairwise-vs-pointwise tradeoffs, and what you do when the judge is the thing that's wrong.
• IC6 signal — Multi-tenancy and platform thinking. One eval service for many teams: quota isolation, schema governance, judge-version migration without breaking everyone, online-vs-offline separation, and cost attribution per team. They want to hear failure modes (Goodhart, judge drift, golden-set rot) named before they ask.
• IC6 signal — Org-level judgment: when is an eval platform worth building versus buying (Langfuse/Braintrust/Comet)? What's the build/buy line, and what do you never outsource (your golden sets, your gate policy)?

3. Concept build-up

3.1 Golden sets are the product; everything else is plumbing

The single highest-leverage artifact is the golden set, and it is the one thing you should never outsource or auto-generate carelessly. Per the Confident AI playbook and Comet's LLM-as-judge guide, the working recipe is 30–50 expert-annotated examples per judge dimension, segmented by use case, user segment, and failure mode, with a target of under 20% annotator disagreement on clear cases. That last number is your sanity floor: if humans can't agree on the easy cases, no judge will, and the dimension is underspecified.

Treat golden sets as versioned data, not config. Each case is {id, input, reference_output?, metadata, labels, dimension} and lives in a content-addressed store so a run can pin dataset_version = sha256(sorted cases). This is non-negotiable for reproducibility: a regression alarm is meaningless if you can't prove the dataset didn't move underneath you. Golden sets also rot — a policy document from 2021 contradicts its 2026 version, and a case whose "correct" answer changed becomes a silent false-positive regression. Build a review cadence and track per-case last-validated dates.

A subtle trap: golden sets must be balanced toward failure modes you've actually seen, not toward easy happy-path cases. A suite that's 95% trivial queries will report 96% pass and tell you nothing the day a real bug ships. Pull hard cases from production traces (online → offline pipeline) so the static set tracks the live distribution.

3.2 Judges are instruments — calibrate before you trust

There are three judge architectures and you'll use all three for different jobs:

• Pointwise is the backbone — score every trace against a rubric, no baseline needed, runs in production. A stronger secondary model scores against a strict rubric. This scales: thousands of responses in minutes at a fraction of a human team's cost.
• Pairwise asks "is A or B better?" — inherently easier than absolute scoring, so judge-human agreement is higher. The cost: it scales quadratically — 5 variants = 10 comparisons — so it's for offline model/prompt bake-offs, not per-trace production scoring.
• G-Eval (chain-of-thought) has the judge write evaluation steps before emitting a score, which improves correlation with human judgment over direct scoring. Use it for fuzzy dimensions (helpfulness, tone) where a single number from a cold judge is noisy.

The discipline most candidates skip: a judge is a model that can be wrong, so you measure its error. Build a calibration set where you have human labels, run the judge, and compute agreement. The rubric design rules that move agreement: 3–5 dimensions per judge call maximum (more and quality collapses), few-shot examples with scores (consistency jumps), separate specialized judges for distinct concerns (a hallucination judge and a tone judge should be different prompts), and start binary pass/fail before numeric scales — a judge that can't reliably split pass from fail has no business emitting a 7.3/10.

The honest caveat: agreement of 0.82 means the judge disagrees with humans on ~1 in 5 cases. For a deploy gate that's often fine because you care about aggregate movement across hundreds of cases, where per-case noise averages out — but it is not fine for auto-rejecting an individual user's output in production. Match the instrument's precision to the decision's blast radius.

3.3 The run pipeline: deterministic first, judge last

Order matters for both cost and correctness. For each case:

1. Deterministic checks — format/schema validation, regex must-contain/must-not-contain, JSON parseability, PII scan, exact-match where a reference exists. These are free, fast, and catch the dumb regressions (broken JSON, dropped citation) that a judge would waste tokens confirming.
2. Only survivors reach the LLM judge. A response that fails schema validation is already a fail; don't pay a judge to tell you a malformed blob is bad.
3. Aggregate per dimension and per segment, store immutable per-case results keyed by the experiment triple.

This is the 5-step regression workflow distilled: prepare datasets → instrument the app to log traces → deterministic checks → LLM-judge for semantic quality → compare versions via experiment tracking.

3.4 Caching: where the cost actually lives

The judge calls dominate cost, and they are highly cacheable because in CI you re-run the same golden set against systems that change one prompt at a time. Cache key = hash(judge_model, judge_version, judge_prompt_version, input, candidate_output). If the system output for a case is byte-identical to last run (common — most cases are unaffected by a one-line prompt tweak), the judge verdict is reused for free. In a typical PR that changes 8% of outputs, you pay for ~8% of judge calls. Two layers, mirroring semantic-cache practice: an exact K-V layer for identical (input, output) and optionally a semantic layer (cosine ≥ ~0.95) for near-identical outputs where the verdict is safe to reuse. Be conservative with the semantic layer on a judge — a 0.95-similar output can flip a borderline pass/fail, so high-precision threshold and only on non-gating dimensions.

3.5 Online vs offline, and multi-team isolation

Offline = the gate. Fixed golden set, before merge/deploy, blocks bad versions. Online = production monitoring: sample 10–20% of live traces for detailed LLM-judge scoring (logging cheap metrics — tokens, cost, latency — for 100%), route flagged responses to a manual-review queue, and feed the gnarliest ones back into the offline golden set. The two form a loop; online finds the failure modes offline didn't anticipate.

For multi-team (the IC6 core): one eval service, many tenants. Each team owns its golden sets, rubrics, and gate thresholds (namespaced, isolated — team A's judge-version bump must not move team B's baseline). The platform owns the shared runner, the judge-call cache, quota enforcement, and cost attribution. Per-team token budgets with rate limits at multiple levels (per-team, per-judge-model, system-wide) stop one team's 5,000-case suite from starving the rest. Cost is attributed per team/feature/model on every trace so the bill is legible and chargeable back.

4. Minimal implementation

A real, runnable offline eval runner with deterministic-checks-first ordering, judge caching, and a noise-band gate. This is the shape of the CI step.

import hashlib, json, sqlite3, statistics, re
from dataclasses import dataclass, asdict
 
# ---- content-addressed caches & result store (sqlite stands in for Redis/Postgres) ----
db = sqlite3.connect("eval.db")
db.execute("CREATE TABLE IF NOT EXISTS judge_cache(k TEXT PRIMARY KEY, verdict INT, reason TEXT)")
db.execute("CREATE TABLE IF NOT EXISTS runs(exp TEXT, case_id TEXT, dim TEXT, score INT)")
 
@dataclass(frozen=True)
class Case:
    id: str; input: str; dimension: str
    must_contain: tuple = ()        # deterministic check
    must_be_json: bool = False
 
def dataset_version(cases): 
    blob = json.dumps([asdict(c) for c in cases], sort_keys=True, default=list)
    return hashlib.sha256(blob.encode()).hexdigest()[:12]
 
# ---- LAYER 1: deterministic checks (free, run first, short-circuit) ----
def deterministic(case: Case, output: str):
    if case.must_be_json:
        try: json.loads(output)
        except Exception: return ("fail", "invalid_json")
    for token in case.must_contain:
        if token.lower() not in output.lower(): return ("fail", f"missing:{token}")
    if re.search(r"\b\d{3}-\d{2}-\d{4}\b", output): return ("fail", "pii_ssn")
    return ("pass", "deterministic_ok")           # not a final pass — just survived
 
# ---- LAYER 2: LLM judge (cached, only for survivors) ----
JUDGE_VERSION = "hallucination-judge-v3"
def judge(case: Case, output: str, call_model) -> tuple[int, str, bool]:
    key = hashlib.sha256(f"{JUDGE_VERSION}|{case.input}|{output}".encode()).hexdigest()
    hit = db.execute("SELECT verdict, reason FROM judge_cache WHERE k=?", (key,)).fetchone()
    if hit: return hit[0], hit[1], True           # cache hit -> zero cost
    # binary pass/fail, single dimension, few-shot rubric lives in call_model's prompt
    score, reason = call_model(case.dimension, case.input, output)
    db.execute("INSERT OR REPLACE INTO judge_cache VALUES(?,?,?)", (key, score, reason))
    return score, reason, False
 
def run_eval(cases, system, call_judge, exp_id):
    cache_hits = total_judged = 0
    for c in cases:
        out = system(c.input)
        verdict, reason = deterministic(c, out)
        if verdict == "fail":
            score = 0
        else:
            score, reason, hit = judge(c, out, call_judge)
            total_judged += 1; cache_hits += int(hit)
        db.execute("INSERT INTO runs VALUES(?,?,?,?)", (exp_id, c.id, c.dimension, score))
    db.commit()
    hit_rate = cache_hits / max(total_judged, 1)
    return hit_rate
 
def pass_rate(exp_id, dim):
    rows = db.execute("SELECT score FROM runs WHERE exp=? AND dim=?", (exp_id, dim)).fetchall()
    return statistics.mean(s for (s,) in rows) if rows else 0.0
 
# ---- the CI gate: compare to baseline against a measured noise band ----
def gate(new_exp, base_exp, dim, noise_band=0.012):
    new, base = pass_rate(new_exp, dim), pass_rate(base_exp, dim)
    delta = new - base
    if delta < -2 * noise_band:        # >2x noise = real regression
        return "FAIL", f"{dim}: {base:.3f} -> {new:.3f} (Δ{delta:+.3f}, regression)"
    if delta > 2 * noise_band:
        return "IMPROVE", f"{dim}: {base:.3f} -> {new:.3f} (Δ{delta:+.3f})"
    return "PASS", f"{dim}: Δ{delta:+.3f} within noise"

What's load-bearing: (1) dataset_version pins the ruler so a regression can't be a phantom of a moved dataset; (2) deterministic checks run before and short-circuit the judge — broken JSON never costs a token; (3) the judge cache key includes JUDGE_VERSION, so bumping the judge correctly invalidates everything (a judge change is a measurement change and must re-score); (4) the gate compares to a baseline against an empirically measured noise_band, not a hardcoded "must be >0.9". Swap the sqlite tables for Redis (judge cache) + Postgres (immutable run store) and call_judge for a real Claude/GPT call with a few-shot binary rubric, and this is production-shaped.

5. Production tradeoffs

Cost math you should say out loud. A 5,000-case suite, judge at ~$0.40/M tokens (current frontier-level pricing), ~1.5k tokens/judge-call ≈ 7.5M tokens ≈ $3 per cold full run. With an 88% cache hit rate on a typical PR, the marginal run is ~$0.36. Across 12 teams running CI dozens of times a day, the cache is the only thing standing between you and a four-figure daily judge bill — and remember the agentic-era trap: cheap per-token prices hide expensive per-task costs when a judge fans out across thousands of cases.

Latency. Offline runs are throughput-bound, not latency-bound — batch the judge calls, use continuous batching on a self-hosted judge if you own one (3–10x throughput), and the gate finishes in the time it takes to judge the ~8% of changed outputs. Online scoring is async and off the request path entirely; never put an LLM judge in the user's critical path.

Failure modes. (1) Goodhart — teams optimize the judge, not the product; the score climbs while users complain. Rotate held-out cases and keep human review in the loop. (2) Judge drift — the judge provider silently updates the model and your baseline shifts under you; pin judge model versions and re-run calibration on every judge bump. (3) Golden-set rot — covered above. (4) Averaged-away regressions — a 3-point drop on your highest-value segment hides inside a flat aggregate; gate per-segment. (5) Self-preference bias — a judge from the same family as the system under test tends to favor it; use a different judge family for bake-offs.

6. How it's asked

7. Pitfalls & flashcards

• Treating the judge as ground truth. It's an instrument with measured error; calibrate it on human labels before you trust a single gate decision.
• Absolute scores instead of deltas. "We're at 0.91" means nothing without last version's number and the noise band. Eval is differential.
• Skipping deterministic checks. Paying a judge to confirm broken JSON is bad is pure waste and adds noise; cheap checks first, always.
• Judge cache key missing judge_version. Bump the judge prompt and you silently mix two instruments' readings in one baseline.
• Averaged-away regressions. Gate per-segment; a flat aggregate hides a cliff on your most valuable cohort.
• Same-family judge. Self-preference bias inflates scores for systems from the judge's own model family; use a different family for bake-offs.
• Golden set of happy paths. A suite that's 95% trivial reports 96% pass and catches nothing real. Weight toward observed failure modes, mined from production.
• LLM judge on the request path. Online scoring is async sampling (10–20%), never blocking the user.

Flashcard. Eval platform = experiments keyed by (dataset_version, system_version, judge_version) → deterministic checks short-circuit before a calibrated, cached judge → gate on the delta vs baseline beyond a measured noise band, per segment. The judge is an instrument; the golden set is the product.

8. Further reading

• Comet: LLM-as-a-Judge — pointwise/pairwise/G-Eval architectures, golden sets, regression testing.
• Confident AI: LLM Evaluation Playbook — golden-set construction, calibration targets, regression pipelines.
• Langfuse: LLM Observability — online vs offline eval, tracing, sampling strategy.
• Redis: LLMOps Guide 2026 — semantic caching, model routing, unified infrastructure, cost attribution.
• Morph: LLM Inference Optimization — batching and caching economics that make large suites affordable.

Next: /evals for the eval-design depth behind these platform mechanics, then /system-design/design-a-rag-chatbot to see where online-mined failure cases come from.