LLM-as-Judge: Rubrics, Biases, and When to Trust a Model Grader

1. Quick anchor

An LLM judge is a measurement instrument, not an oracle. You prompt a model to score or compare outputs against a rubric, and you get a cheap, flexible signal that correlates with human judgment better than n-gram metrics like BLEU — but only after you treat it like an instrument with a known error profile. It has three systematic, reproducible biases: it prefers options at certain rubric positions (position bias), it prefers longer answers (verbosity bias), and it prefers its own family's outputs (self-enhancement bias). The single number that tells you whether a judge is usable is its agreement with human labels on a held-out set; everything else — pairwise vs pointwise, balanced permutation, ensembling — exists to push that number up. And the moment a task has a checkable ground truth (does the code compile, does the number match, is the JSON valid), you should prefer a verifiable reward over a judge, because a deterministic check has zero bias and zero per-call cost.

2. Why interviewers probe this

• IC4 — Can you name the biases and apply the standard fixes (swap order, anchor the rubric, parse robustly)? Do you reach for a judge only when cheaper deterministic checks won't work? Can you read a confusion matrix between judge and human?
• IC5 — Can you design a calibration loop: collect human labels, measure judge-human agreement (Cohen's kappa, not raw accuracy), iterate the rubric, and gate on the agreement threshold? Do you understand why pairwise reduces variance and what it costs in queries?
• IC6 — Can you reason about the judge as part of a larger optimization loop and its failure under pressure? When a judge scores a training signal (RLAIF, reward modeling, eval-driven dev), do you see the reward-hacking and contamination risks? Can you draw the line between "judge is good enough" and "this needs a verifier," and defend the cost/latency/quality tradeoff at scale?

3. Concept build-up

3.1 Why a model judge at all

Reference-based metrics break on open-ended text. BLEU and ROUGE measure n-gram overlap against a reference answer, so a correct paraphrase that shares no surface tokens scores near zero. Embedding metrics like BERTScore fix the paraphrase problem — it computes token-level cosine similarities between candidate and reference contextual embeddings and aggregates a greedy-matched precision/recall into an F1, correlating with human judgment around 59% versus roughly 47% for BLEU — but it still needs a reference and still can't tell you whether an answer is faithful, helpful, or safe. Those are properties of meaning, not overlap.

An LLM judge needs no reference for many criteria. You describe the quality dimension in the prompt and let the model reason about it. On faithfulness specifically, LLM judges show the highest correlation with human judgments among automatic metrics — they catch subtle entailment failures that token-overlap metrics miss. The price: the judge is non-deterministic, costs an API call per evaluation, and carries the biases below. Everything in this lesson is about paying that price intelligently.

3.2 Pointwise vs pairwise

Pointwise (absolute scoring): the judge sees one response and emits a scalar, e.g. "rate faithfulness 1–5." Simple to prompt, simple to aggregate (average the scores). But absolute scores have high calibration variance — what the judge means by "4" drifts across calls, across prompt phrasings, and across model versions. Two runs of the same judge on the same answer can disagree by a point.

Pairwise: the judge sees two responses and says which is better. This reduces calibration variance relative to absolute scoring because the judge only has to make a relative decision — "is A better than B" is an easier, more stable judgment than "is A worth exactly 4." You convert pairwise verdicts into a ranking (Elo, Bradley-Terry, or simple win-rate). The cost is combinatorial: ranking n candidates pairwise is O(n²) comparisons in the worst case, versus n for pointwise. For a leaderboard of 5 models you can afford all 10 pairs; for scoring 10,000 production traces you cannot, so you fall back to pointwise (often comparing each output against a single fixed baseline).

Rule of thumb: pairwise for model selection and A/B decisions (few candidates, you want a trustworthy ranking); pointwise for monitoring and CI gates (many items, you want a stable absolute threshold like faithfulness > 0.85).

3.3 The three biases

Position bias. Rubric-based and pairwise evaluation both exhibit systematic position bias: the judge prefers options at specific positions independent of content. In a pairwise setup it may favor whichever answer is presented first; in a rubric with a numbered scale it may favor score options at the start or end of the list. This is an artifact of the implicit multiple-choice structure, not a content judgment. It is large enough to flip verdicts.

Verbosity bias. The judge systematically prefers longer responses, mistaking length for quality. Longer text gives the judge more phrases that look like supporting evidence, so it rationalizes a higher score. A wordy, padded answer beats a crisp correct one.

Self-enhancement bias. A judge disproportionately favors responses generated by itself or its own model family — an in-group preference. If you evaluate GPT-family and Claude-family candidates with a Claude-family judge, the judge tilts toward the Claude outputs for reasons unrelated to quality. This is the most dangerous bias in competitive evaluation because it silently corrupts cross-vendor comparisons.

3.4 Calibration: the only number that matters

A judge is useful exactly to the degree it agrees with humans. So the calibration loop is:

1. Build a golden set of items you have labeled by hand — ideally sourced from production failures, edge cases, and user-reported issues, because those are the cases that actually matter and the cases where judges are weakest.
2. Run the judge on that set and build a confusion matrix of judge verdict vs human verdict.
3. Measure chance-corrected agreement, not raw accuracy. On an imbalanced set (say 90% of answers are "faithful"), a judge that blindly says "faithful" scores 90% raw accuracy and is worthless. Cohen's kappa corrects for chance agreement; report it. Treat the human label as the actual class and the judge as the predictor, then read precision and recall per class — a judge with high recall on "unfaithful" but low precision is over-flagging, which costs you in alert fatigue.
4. Iterate the rubric and the bias fixes until kappa clears your threshold. There is no universal threshold; pick one tied to the decision the judge gates (a CI merge gate needs higher agreement than an exploratory dashboard).
5. Keep auditing. Production drifts; a judge calibrated in March can decay by June. Periodically re-sample production, re-label by hand, and re-measure. A model-provider's silent version bump can shift judge behavior with no code change on your side.

The mitigations stack on top of calibration: balanced permutation for position bias, explicit anti-length instructions plus reference-based penalties for verbosity, and ensembling judges from different model families (and anonymizing the generation source) for self-enhancement. Ensembling also dampens the idiosyncratic variance of any single judge.

3.5 G-Eval: a structured rubric judge

G-Eval is the standard way to turn "score this for coherence" into a reproducible procedure. It (1) states the task and criteria, (2) has the LLM generate explicit chain-of-thought evaluation steps from those criteria, (3) uses a form-filling paradigm where the model answers yes/no/unsure per step, and (4) weights the final score by token-level log-probabilities so the score is a smooth expected value rather than a single sampled integer. It is reference-free and customizable for any dimension (coherence, fluency, relevance, safety), which is why frameworks like DeepEval ship it as a core metric. The log-prob weighting is the clever part: instead of the model emitting "4" (high variance), you read the probability mass over each score and compute the expectation, which is more stable across calls.

3.6 When the judge should be retired: verifiable rewards

A judge is the right tool when correctness is a matter of meaning with no closed-form check. It is the wrong tool when a deterministic verifier exists. If you can check the answer with a compiler, a unit test, a JSON-schema validator, an exact-match against a known key, a regex, or a calculator, do that instead. A verifier has:

• Zero bias — no position, verbosity, or self-enhancement effect.
• Zero marginal cost — no API call, no tokens.
• Determinism — same input, same verdict, every time, which is exactly what a CI gate needs.

The boundary is rarely all-or-nothing. For a product with mostly open-ended outputs, the move is to carve verifiable sub-claims out of the open-ended whole: in a RAG answer, faithfulness of individual factual claims can be checked by NLI entailment against the retrieved context (a classifier, near-deterministic) even though "is this answer helpful" stays a judge call. RAGAS scores faithfulness exactly this way — extract claims, verify each against context — turning part of a fuzzy judgment into a near-verifiable one. Push as much of the evaluation onto verifiers as the task allows, and reserve the judge for the irreducibly subjective remainder. This matters doubly when the judge feeds a training loop (RLAIF, reward modeling): a biased or hackable judge becomes a reward the policy will learn to exploit, so verifiable rewards are strongly preferred wherever the task admits them.

4. Minimal implementation

A production-shaped pairwise judge with swap-and-average position-bias correction, robust parsing, and a calibration harness. It uses Claude (Opus 4.8) as the judge.

import json
from anthropic import Anthropic
 
client = Anthropic()  # reads ANTHROPIC_API_KEY
JUDGE_MODEL = "claude-opus-4-8"
 
RUBRIC = """You are comparing two answers to the same question for FAITHFULNESS
to the provided context. An answer is faithful only if every factual claim it
makes is supported by the context. Penalize unsupported claims. Do NOT reward
length: a shorter answer that is fully supported beats a longer one that adds
unsupported claims.
 
Question: {question}
Context: {context}
 
Answer A: {a}
Answer B: {b}
 
Respond with ONLY a JSON object: {{"winner": "A" | "B" | "tie", "reason": "<one sentence>"}}"""
 
def _judge_once(question, context, ans_first, ans_second):
    """One pairwise call. Returns 'A', 'B', or 'tie' relative to (first, second)."""
    prompt = RUBRIC.format(question=question, context=context,
                           a=ans_first, b=ans_second)
    resp = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=512,
        thinking={"type": "adaptive"},          # let the judge reason before deciding
        output_config={                          # constrain output to valid JSON
            "format": {
                "type": "json_schema",
                "schema": {
                    "type": "object",
                    "properties": {
                        "winner": {"type": "string", "enum": ["A", "B", "tie"]},
                        "reason": {"type": "string"},
                    },
                    "required": ["winner", "reason"],
                    "additionalProperties": False,
                },
            }
        },
        messages=[{"role": "user", "content": prompt}],
    )
    text = next(b.text for b in resp.content if b.type == "text")
    return json.loads(text)["winner"]
 
def judge_pairwise(question, context, answer_x, answer_y):
    """Position-bias-corrected verdict for X vs Y. Runs both orders."""
    # Order 1: X is shown first (as 'A'), Y second (as 'B')
    v1 = _judge_once(question, context, answer_x, answer_y)
    # Order 2: Y is shown first (as 'A'), X second (as 'B')
    v2 = _judge_once(question, context, answer_y, answer_x)
 
    # Translate each verdict into a vote for X.
    def x_vote(verdict, x_is_a):
        if verdict == "tie":
            return 0.5
        x_label = "A" if x_is_a else "B"
        return 1.0 if verdict == x_label else 0.0
 
    x_score = (x_vote(v1, x_is_a=True) + x_vote(v2, x_is_a=False)) / 2
    # A verdict only "wins" if it survives order reversal.
    if x_score > 0.5:
        return "X"
    if x_score < 0.5:
        return "Y"
    return "tie"
 
def cohens_kappa(judge_labels, human_labels):
    """Chance-corrected agreement. The number that decides if the judge is usable."""
    n = len(judge_labels)
    classes = set(judge_labels) | set(human_labels)
    po = sum(j == h for j, h in zip(judge_labels, human_labels)) / n
    pe = sum(
        (judge_labels.count(c) / n) * (human_labels.count(c) / n)
        for c in classes
    )
    return (po - pe) / (1 - pe) if pe < 1 else 1.0
 
if __name__ == "__main__":
    # Calibration set: items YOU labeled by hand, ideally from production failures.
    golden = [
        {"question": "When was the contract signed?",
         "context": "The agreement was executed on March 3, 2024.",
         "answer_x": "It was signed on March 3, 2024.",
         "answer_y": "It was signed in early 2024, likely after lengthy negotiation.",
         "human": "X"},  # Y adds an unsupported claim about negotiation
        # ... dozens more, sourced from real failure cases
    ]
    judge = [judge_pairwise(g["question"], g["context"], g["answer_x"], g["answer_y"])
             for g in golden]
    human = [g["human"] for g in golden]
    print(f"judge-human kappa = {cohens_kappa(judge, human):.3f}")

The load-bearing parts: output_config.format forces schema-valid JSON so parsing never fails on a stray preamble; running both orders and counting a win only if it survives reversal kills position bias; and the cohens_kappa call is the gate — if that number is low, nothing else in your eval pipeline is trustworthy and you fix the rubric before shipping the judge. thinking: {type: "adaptive"} lets the judge reason (the G-Eval insight) without you hand-tuning a budget.

5. Production tradeoffs

Cost. At Opus 4.8 pricing ($5 / 1M input, $25 / 1M output), a judge call on a ~2K-token rubric+context plus a ~200-token verdict is roughly $0.015. Swap-correction doubles it; a 3-judge ensemble with swap is ~6x. For 10,000 daily production traces, a single pointwise pass is ~$150/day; a swap-corrected 3-judge ensemble is ~$900/day. Drive this down by routing the bulk of grading to a cheaper model (Haiku 4.5 at $1/$5 is ~5x cheaper) and reserving the expensive judge for disagreements or borderline scores, by caching the stable rubric/context prefix, and by batching (50% off, non-latency-sensitive).

Latency. A judge in a CI gate adds seconds per item; swap and ensemble add parallel calls, so wall-clock is the slowest single call if you fan out. A judge in the serving path (online guardrail) is usually a non-starter at the per-request budget — push it to async monitoring instead.

Quality. Calibrated pairwise > calibrated pointwise > uncalibrated anything > n-gram metrics. The biggest quality lever is not the method but the calibration: an uncalibrated state-of-the-art judge can be worse than a calibrated cheap one.

What changes at scale. Three things bite. (1) Cost scales linearly with traffic — you cannot afford pairwise-everything on millions of items, so pointwise-against-baseline plus sampling becomes mandatory. (2) The judge becomes a moving target — provider version bumps silently shift its behavior, so you need scheduled re-calibration against fresh human labels, treated as a recurring cost. (3) Reward hacking — if the judge gates training or prompt-optimization loops, the system under test learns to exploit the judge's biases (pad for the verbosity bias, phrase for the self-enhancement bias). At scale, prefer verifiable rewards wherever the task admits them, and hold out a human-labeled set the optimizer never sees to detect when your judge score and real quality have decoupled.

6. How it's asked

7. Pitfalls & flashcards

• Reporting raw accuracy instead of chance-corrected agreement. On imbalanced sets, raw accuracy makes a useless judge look great. Use Cohen's kappa.
• Calibrating once and trusting forever. Provider version bumps and production drift decay a judge silently. Schedule re-calibration on fresh human labels.
• Letting the judge see length. Verbosity bias means padded answers win. Add explicit anti-length instructions and penalize unnecessary verbosity with reference-based checks.
• Judging with a same-family model in a cross-vendor comparison. Self-enhancement bias corrupts the ranking. Ensemble across families and anonymize sources.
• Using a judge where a verifier exists. If the compiler or a unit test can answer it, the judge is strictly worse — biased, costly, non-deterministic.
• Fragile parsing. A judge that emits prose around its score breaks your pipeline. Constrain output to a schema and parse with a real JSON parser, never a regex on free text.
• Optimizing against an uncalibrated judge. You are then optimizing the judge's biases, not quality — classic reward hacking.

Flashcard. What single number decides whether an LLM judge is usable, and what makes it lie? Its chance-corrected agreement (Cohen's kappa) with human labels on a held-out set — raw accuracy lies on imbalanced data, so always correct for chance.

8. Further reading

• Position bias in rubric-based evaluation and the balanced-permutation fix (arXiv 2602.02219) — the measurement and the calibration technique.
• Large Language Models are not Fair Evaluators (arXiv 2310.01432) — position bias and randomize-and-average mitigation.
• Self-preference / self-enhancement bias in LLM judges (arXiv 2410.21819) — quantifying in-group preference and ensemble mitigation.
• G-Eval: the definitive guide (Confident AI) — CoT evaluation steps plus log-prob-weighted scoring.
• RAGAS faithfulness and the claim-extraction approach (RAGAS docs) — turning a fuzzy judgment into near-verifiable claim checks.
• LLM judges vs automatic metrics for faithfulness (arXiv 2501.00269) — why judges correlate best with humans, and where they fail.

Next: Classification metrics — precision, recall, F1, and why accuracy lies on imbalanced data.