Classification & Retrieval Metrics

1. Quick anchor

Every eval that produces a label or a ranked list reduces to two families of arithmetic, and they are the most common place senior candidates embarrass themselves. Classification metrics answer "is this prediction right?" by counting four buckets (TP, FP, FN, TN) and dividing — but the choice of which ratio you report is a product decision disguised as a math decision. Retrieval metrics answer "did the right documents come back, near the top?" and the subtlety is entirely about order. The single mental model: a metric is a lossy projection from a confusion matrix (or a ranked list) down to one number, and your job is to pick the projection that aligns with the cost of being wrong. Accuracy throws away the asymmetry between a missed fraud and a false alarm; recall@k throws away rank; nDCG keeps rank but needs graded labels. Know what each one discards, and you can answer almost any eval question by reasoning forward from the four buckets.

2. Why interviewers probe this

• IC3: Can you name the four confusion-matrix cells without hesitating, compute precision and recall, and articulate why accuracy lies on imbalanced data? This is table stakes — fumbling it caps the loop.
• IC4: Do you map metrics to product cost? They want to hear "for a cancer screen we maximize recall and eat false positives; for a content-takedown bot we protect precision because false removals are PR incidents." And: can you reason about the precision/recall tradeoff as a threshold moves?
• IC5: Can you design the metric layer for a real RAG or ranking system — choose nDCG vs MRR vs recall@k with explicit justification, explain why PR-AUC beats ROC-AUC under extreme imbalance, and connect the offline metric to a CI regression gate and online drift monitoring? They're testing whether your metric choices survive contact with production.

3. Concept build-up

3.1 The confusion matrix is the source of truth

Every classification number is a ratio of four counts. Lay them out as predicted (rows) vs actual (columns):

• TP — caught a real positive.
• FP — false alarm: flagged something that was fine (Type I error).
• FN — a miss: let a real positive slip through (Type II error).
• TN — correctly left a negative alone.

From these four numbers, everything follows. accuracy = (TP + TN) / total. precision = TP / (TP + FP) — the column of what you flagged. recall = TP / (TP + FN) — the column of what was actually positive. The reason interviewers start here is that a candidate who can derive any metric from the matrix never has to memorize formulas; one who memorizes formulas falls apart the moment you ask for something non-standard like specificity or balanced accuracy.

3.2 Why accuracy lies, and what to use instead

Imagine 1,000 emails, 50 of them spam. A model that labels everything "not spam" gets 950/1000 = 95% accuracy while catching zero spam. That's the accuracy trap: when one class dominates, accuracy is mostly measuring the majority class, and a degenerate model scores high. The same failure shows up in fraud (often <1% positive), safety classifiers, and rare-intent routing. The fix is to use metrics that ignore the giant TN bucket — precision and recall both exclude TN, so they can't be inflated by an ocean of easy negatives.

3.3 Precision vs recall — and when you care about each

Precision and recall pull in opposite directions, and which you favor is dictated by the cost of each error type:

• Maximize recall when a miss is catastrophic and a false alarm is cheap to triage: cancer screening, fraud blocking, CSAM detection, "did the retriever miss the one doc that answers the question." You accept FPs.
• Maximize precision when a false positive is the expensive event: auto-removing user content, auto-blocking transactions, auto-paging an on-call engineer. A wrong action erodes trust; you accept some FNs.
• F1 when both errors cost roughly the same and you want one comparable number across model versions. F1 is the harmonic (not arithmetic) mean because the harmonic mean is dragged down hard by the smaller of the two — you can't get a good F1 by acing one and tanking the other. Use F-beta to weight: beta > 1 favors recall, beta < 1 favors precision.

The lever connecting them is the decision threshold. Most classifiers (and LLM judges that emit a probability or score) output a continuous score; you call "positive" when score ≥ threshold. Lower the threshold and you flag more things — recall rises, precision falls. Raise it and you flag fewer, more confidently — precision rises, recall falls. This is why "what's your precision?" is an incomplete question without a threshold, and why the demo above lets you drag it.

3.4 Threshold-free quality: ROC-AUC and PR-AUC

To compare two models independent of where you'll eventually set the threshold, sweep all thresholds and summarize the curve.

• ROC curve plots recall (TPR) on the y-axis against the false positive rate FPR = FP/(FP+TN) on the x-axis, across every threshold. ROC-AUC is the area underneath, and it has a beautiful interpretation: it equals the probability that a randomly chosen positive scores higher than a randomly chosen negative. AUC of 1.0 is perfect ranking; 0.5 is a coin flip. It's threshold-independent and insensitive to class ratio because both axes are within-class rates.
• PR-AUC plots precision against recall across thresholds. On severe imbalance, prefer PR-AUC: ROC-AUC can look deceptively high because the FPR denominator FP+TN is enormous (lots of easy negatives), so even thousands of false positives barely move the x-axis. PR-AUC keeps the spotlight on the positive class, so it degrades honestly when the model floods you with false alarms. Rule of thumb: ROC-AUC for roughly balanced data and discrimination questions; PR-AUC when positives are rare and false positives are costly.

3.5 Retrieval metrics — now order matters

A retriever returns a ranked list, and a good answer buried at rank 9 is nearly as bad as missing. So retrieval metrics differ in how much they reward early placement:

• Recall@k = (# relevant in top-k) / (total relevant). Coverage, order-unaware. "Did we get the right docs into the window at all?" The first metric to check, because if the answer isn't in the top-k the generator cannot be faithful — recall@k is the ceiling on RAG quality.
• Hit-rate@k = fraction of queries with at least one relevant doc in top-k. Coarser than recall@k (presence, not quantity); good for "single-answer" lookups.
• MRR (Mean Reciprocal Rank) = average of 1 / (rank of first relevant). First relevant at rank 1 → 1.0, rank 4 → 0.25. Rewards getting one good result early; ignores everything after the first hit. Right when the user reads only the top result.
• nDCG@k keeps full graded order. DCG@k = Σ gain_i / log2(i+1); the log2(i+1) discount means rank-1 gets full credit, rank-2 gets /1.585, rank-3 gets /2, etc. Normalize by the ideal ordering's DCG (IDCG) so the score lands in [0, 1]. nDCG is the dominant RAG-retrieval metric because it's the only common one that uses graded relevance (a perfect doc beats a merely-okay one) and penalizes putting good results late.
• MAP@k (Mean Average Precision) averages precision@i at each relevant hit, then over queries. Binary relevance, precision-weighted, penalizes burying relevant items below the cutoff. Standard in ranking/recsys.

4. Minimal implementation

examples/evals/metrics.py implements all of these from the literal formulas — no sklearn — so the definition is readable straight off the code. Below is the core: a confusion matrix that derives every classification metric, plus a threshold-free AUC via the rank-sum identity (no curve sweep needed).

from dataclasses import dataclass
 
@dataclass(frozen=True)
class Confusion:
    tp: int; fp: int; fn: int; tn: int
 
    def precision(self):              # of flagged, how many were right
        d = self.tp + self.fp
        return self.tp / d if d else 0.0
 
    def recall(self):                 # of real positives, how many caught
        d = self.tp + self.fn
        return self.tp / d if d else 0.0
 
    def f_beta(self, beta=1.0):
        p, r = self.precision(), self.recall()
        if p == 0 and r == 0: return 0.0
        b2 = beta * beta
        return (1 + b2) * p * r / (b2 * p + r)   # beta>1 favors recall
 
def confusion_at_threshold(scores, labels, threshold):
    tp = fp = fn = tn = 0
    for s, y in zip(scores, labels):
        pred = 1 if s >= threshold else 0       # the lever
        if   pred and y:     tp += 1
        elif pred and not y: fp += 1
        elif not pred and y: fn += 1
        else:                tn += 1
    return Confusion(tp, fp, fn, tn)
 
def roc_auc(scores, labels):
    """AUC = P(random positive scores above random negative). Ties get 0.5."""
    pos = [s for s, y in zip(scores, labels) if y == 1]
    neg = [s for s, y in zip(scores, labels) if y == 0]
    if not pos or not neg: return float("nan")
    wins = sum((sp > sn) + 0.5 * (sp == sn) for sp in pos for sn in neg)
    return wins / (len(pos) * len(neg))

Running the file sweeps the threshold over an imbalanced (4-of-100) problem and prints the tradeoff directly. At threshold 0.6 you get recall 1.000 but precision 0.154 (26 flags, 22 of them false); push to 0.7 and precision jumps to 1.000 while recall drops to 0.750 — the same model, a different operating point. The "always-negative" baseline prints accuracy 0.960 with recall 0.000, the accuracy trap in one line, while ROC-AUC reports 0.982 because the ranking is actually good regardless of where you cut. The retrieval section computes recall@3 = 0.500, MRR = 0.333, nDCG@5 = 0.520, and AP@5 = 0.367 on one query so you can see how each metric scores the same ranked list differently.

5. Production tradeoffs

Cost & latency. Counting-based metrics (precision, recall, F1, recall@k, MRR, nDCG) are essentially free — pennies of CPU, run on every CI build over a golden set. The expensive part is the labels: judging "is this chunk relevant" or "is this email spam" at scale is what costs money, especially with graded relevance for nDCG (every doc needs a 0–3 grade, not just a yes/no). That annotation cost is why teams report recall@k (binary) on large sets and reserve nDCG for a smaller, carefully graded slice.

Quality & what changes at scale. A single-query metric is noise; you need enough queries that the mean is stable, and you should report confidence intervals, not point estimates, when comparing model versions in a CI gate (e.g., fail the build if nDCG@10 drops more than 2 points below the production baseline). At scale the dominant failure is distribution shift: your offline golden set was curated months ago, production traffic drifts, and your metric stays green while users suffer. Pair the offline gate with online monitoring — recompute the metric (or a proxy like click-through / thumbs-down) on sampled production traffic and alarm on KL-divergence of the input embedding distribution. The other scale failure is label leakage / saturation: once a model is iterated against the same golden set repeatedly, the score overfits and stops predicting downstream quality; refresh the set from new production failures. See /evals for the offline-vs-online split and /rag for where these metrics gate the retriever.

6. How it's asked

7. Pitfalls & flashcards

• Reporting accuracy on imbalanced data. The single most common eval mistake; it makes degenerate models look good. Default to precision/recall/F1 plus an AUC.
• Quoting precision without a threshold. Precision and recall are points on a curve; name the operating point or report the AUC.
• Confusing which class is "positive." Define positive as the thing you're catching before you compute anything; flip it and precision/recall swap meaning.
• Using recall@k for a ranking-sensitive product. Recall@k is order-blind — a relevant doc at rank 10 scores the same as rank 1. Use MRR or nDCG when position matters.
• Single-query retrieval scores. One query is noise. Average over a representative query set and report intervals before gating CI.
• Binary nDCG. nDCG's whole advantage is graded relevance; feeding it 0/1 labels collapses it toward MAP and wastes the annotation.
• Green offline metric, sad users. Offline gates catch regressions you introduce; they don't catch distribution shift. Monitor the metric (or a proxy) on production traffic too.

Flashcard. Precision = TP/(TP+FP) (of flagged, how many right — punishes false alarms). Recall = TP/(TP+FN) (of real positives, how many caught — punishes misses). F1 = their harmonic mean. ROC-AUC = P(random positive ranks above random negative). nDCG = graded, order-aware retrieval quality, Σ gain_i/log2(i+1) normalized by the ideal ordering.

8. Further reading

• Google ML Crash Course — accuracy, precision, recall: the canonical confusion-matrix derivations.
• Pinecone — offline evaluation & ranking metrics: recall@k, MRR, MAP with retrieval framing.
• Towards Data Science — DCG@k and nDCG@k for RAG: the log-discount intuition, worked.
• Towards Data Science — MAP@K clearly explained: precision-weighted ranking.
• Medium (Thinking Loop) — top 10 retrieval metrics for RAG: a tuning-oriented survey.

Next: LLM-as-Judge — bias, calibration, and rubrics — what to do when there's no clean label to count and your "metric" is itself a model.