8.3 — LLM-as-Judge and Human Evaluation

1. What is it?

LLM-as-judge is the technique of using a separate LLM call to score or critique another LLM's output against defined criteria — the workhorse technique behind most of the reference-free evaluation this book has referenced (Part 6.2, Part 8.1/8.2's faithfulness checking). Human evaluation is the older, higher-fidelity but slower and more expensive alternative: routing outputs to human reviewers for scoring or annotation. This chapter examines both techniques' actual mechanics, genuine strengths, and — critically — their specific, well-documented failure modes, since using either technique without understanding its limitations produces a false sense of evaluation rigor.

2. Why does it exist?

Part 8.1 established that many quality properties (helpfulness, faithfulness, tone) don't have a single correct string to exact-match against, ruling out purely deterministic evaluation for these properties. Human judgment is the gold standard for exactly these subjective/contextual properties, but doesn't scale to the volume real production evaluation needs (Part 6.3's continuous online monitoring, evaluating potentially thousands of interactions). LLM-as-judge exists as a scalable proxy for human judgment — genuinely useful, but a proxy, not a replacement, and understanding precisely where that proxy breaks down is essential to using it responsibly.

3. What problem does it solve?

It solves "how do I get reference-free quality scores at a volume and speed human review can't match, for properties that genuinely require judgment rather than mechanical checking" — while human evaluation solves "how do I get the highest-fidelity, most trustworthy assessment for the cases where an LLM judge's own limitations matter most (Part 8.1's most safety-critical or ambiguous cases, or as a calibration check on the LLM judge itself)."

4. How does it work internally?

How an LLM-as-judge call is actually structured

judge_prompt = """
You are evaluating an AI customer support response for helpfulness.
Question: {question}
Response: {response}
Score the response's helpfulness from 1-5, and explain your reasoning.
"""

This is, structurally, just another LLM call (Part 3.1) — with all of Part 3.1's prompt-engineering considerations applying directly (how the criteria are phrased, whether examples of "good" and "bad" scores are provided as few-shot examples, whether chain-of-thought reasoning is elicited before the final score) genuinely affecting the judge's actual scoring behavior and consistency, exactly as they would for any other generation task.

Known, well-documented LLM-as-judge failure modes — this is the core of this chapter

• Position bias: when asked to compare two outputs (pairwise comparison, Part 6.2), some judge models show a measurable tendency to favor whichever option is presented first (or second) regardless of actual quality — a real, documented effect, mitigated by running the comparison twice with the order swapped and checking for consistency.
• Verbosity bias: judges can systematically favor longer, more elaborate responses over more concise, equally (or more) correct ones — relevant to watch for if you're optimizing for conciseness (Part 7.10's cost-reduction pressure toward shorter responses) while using an LLM judge that might inadvertently penalize exactly the brevity you're trying to encourage.
• Self-preference bias: a judge model can show a measurable tendency to favor outputs generated by the same model family as itself over outputs from a different model — a real, documented consideration when using an LLM-as-judge to compare outputs across different model providers (Part 3.11's model-selection evaluation).
• Inconsistency across repeated calls: because generation isn't perfectly deterministic (Part 2.6), the same judge scoring the same output twice can produce different scores — mitigated by using a low or zero temperature for judge calls specifically (Part 2.6's temperature discussion) and, for high-stakes evaluation, averaging multiple judge calls rather than trusting a single one.

Before human labels can be "ground truth" — measuring inter-annotator agreement

Section 4's calibration step treats human-reviewed scores as ground truth to validate the LLM judge against — but a human label is only trustworthy as ground truth if independent human reviewers would actually agree on it. If two reviewers scoring the same responses disagree with each other as often as the LLM judge disagrees with either of them, "calibrate against human judgment" is comparing the judge to a coin flip dressed up as ground truth, not to a genuinely reliable reference. Measuring inter-annotator agreement is a prerequisite step this chapter's calibration process implicitly assumes has already happened, and it needs to be checked, not assumed.

Cohen's kappa measures agreement between exactly two annotators, correcting for the agreement you'd expect by pure chance: κ = (p_o − p_e) / (1 − p_e), where p_o is the observed proportion of items the two annotators labeled the same way, and p_e is the proportion they'd be expected to agree on by chance alone, given each annotator's own labeling frequencies. Fleiss' kappa generalizes the same idea to three or more annotators. A commonly used interpretation scale (Landis & Koch): < 0.40 poor/questionable agreement, 0.40–0.60 moderate, 0.60–0.80 substantial, > 0.80 almost perfect — a kappa in the "poor" or "moderate" range is a signal to not treat that dataset's labels as reliable ground truth yet, not a minor caveat to note in passing.

Worked example. Two reviewers independently label 10 support responses as "helpful" or "not helpful":

                          Reviewer B: helpful   Reviewer B: not helpful
Reviewer A: helpful              5                       1
Reviewer A: not helpful          1                       3

• Observed agreement: p_o = (5 + 3) / 10 = 0.8
• Reviewer A said "helpful" 6/10 times, "not helpful" 4/10; Reviewer B said "helpful" 6/10, "not helpful" 4/10.
• Chance agreement: p_e = (0.6 × 0.6) + (0.4 × 0.4) = 0.36 + 0.16 = 0.52
• κ = (0.8 − 0.52) / (1 − 0.52) = 0.28 / 0.48 ≈ 0.583 — moderate agreement by the Landis-Koch scale, despite 80% raw agreement looking superficially reassuring. This is exactly why raw percent-agreement alone is misleading: two annotators who both mostly say "helpful" will agree often purely by chance, and kappa is what corrects for that.

def cohens_kappa(labels_a: list[str], labels_b: list[str]) -> float:
    """
    Computes Cohen's kappa, the chance-corrected agreement between two
    annotators labeling the same set of items.

    Args:
        labels_a (list[str]): Reviewer A's label for each item, in order.
        labels_b (list[str]): Reviewer B's label for each item, in order,
            aligned index-for-index with labels_a.

    Returns:
        float: Kappa in roughly [-1, 1]; interpret via the Landis-Koch scale
            (below 0.40 poor, 0.40-0.60 moderate, 0.60-0.80 substantial, above
            0.80 almost perfect) before treating these labels as ground truth.
    """
    n = len(labels_a)
    observed_agreement = sum(a == b for a, b in zip(labels_a, labels_b)) / n

    categories = set(labels_a) | set(labels_b)
    chance_agreement = sum(
        (labels_a.count(cat) / n) * (labels_b.count(cat) / n) for cat in categories
    )
    return (observed_agreement - chance_agreement) / (1 - chance_agreement)


reviewer_a = ["helpful"] * 5 + ["not helpful"] + ["helpful"] + ["not helpful"] * 3
reviewer_b = ["helpful"] * 5 + ["helpful"] + ["not helpful"] + ["not helpful"] * 3
assert round(cohens_kappa(reviewer_a, reviewer_b), 3) == 0.583

What to do when agreement is too low to trust the labels. A moderate-or-worse kappa doesn't mean give up on human labels — it means the labels need resolution before they're used as calibration ground truth: (1) adjudication — a third, senior reviewer (or the person who defined the criteria, section 4) breaks ties on the specific items where the two reviewers disagreed; (2) discussion-based consensus — the disagreeing reviewers discuss the specific disputed items and converge on a single label, often surfacing that the evaluation criteria themselves were ambiguous (a finding worth feeding back into sharper criteria, not just a labeling fix); (3) for three-or-more-reviewer setups, majority vote per item, falling back to adjudication only for items with no majority. Only the resulting, agreement-checked (or disagreement-resolved) labels should feed into section 17's calibrate_judge_against_humans — using raw, unreconciled single-reviewer labels as if they were uncontested ground truth risks calibrating your LLM judge against one reviewer's idiosyncrasies rather than a genuinely reliable reference.

Calibrating an LLM judge against human judgment — the necessary validation step

Because of these known failure modes, a responsible LLM-as-judge deployment doesn't just trust the judge blindly — it periodically validates the judge's scores against a sample of human-reviewed ground truth, checking for correlation/agreement between the two. If the LLM judge's scores diverge significantly from human judgment on a validation sample, that's a signal the judge's prompt/criteria need refinement (or that this specific property genuinely isn't well-suited to LLM-as-judge and needs human review more centrally) — this calibration step is what separates a validated LLM-as-judge practice from an unvalidated one that might be silently producing systematically biased scores no one has checked.

Human evaluation — where it remains necessary

Human evaluation is genuinely irreplaceable for: calibrating and validating an LLM judge (as above); defining evaluation criteria in the first place (someone has to decide what "helpful" or "appropriate tone" actually means for a specific domain, before either a human reviewer or an LLM judge can apply that definition); and the highest-stakes, most ambiguous, or most novel cases where an automated judge's known limitations pose too much risk to rely on alone (a genuinely borderline safety/compliance decision, Part 5.4's human-in-the-loop territory, applied to evaluation rather than live execution).

5. Simple mental model

An LLM judge is like hiring a knowledgeable but occasionally biased assistant grader for a large stack of essays — genuinely useful and much faster than grading every single one yourself, generally competent at applying a rubric, but known to have specific quirks (perhaps favoring longer essays, or being slightly inconsistent grading the same essay twice) that you need to account for, and whose grading you should periodically spot-check against your own judgment (calibration) rather than trusting completely and unconditionally, especially for the essays that matter most (the highest-stakes cases still warrant your own direct review).

6. Real-world example

A team building an LLM-as-judge for evaluating support-response helpfulness initially trusted the judge's scores uncritically, and noticed their prompt-optimization efforts kept "improving" scores by making responses progressively longer and more elaborate — a real instance of verbosity bias (section 4) silently rewarding the wrong thing. Discovering this required a calibration step: sampling 50 judge-scored responses and having a human reviewer independently score the same ones, revealing the human reviewer consistently preferred several of the shorter responses the LLM judge had scored lower — prompting a redesign of the judge's prompt to explicitly instruct it to reward conciseness and directness, not penalize it, and a subsequent re-validation against human judgment confirming the fix actually corrected the bias rather than introducing a new one.

7. Architecture diagram

LLM Judgefast, scalable — scores most production traffic (Part 6.3 online evaluation)

Human reviewslow, expensive, highest-fidelity — validates judge scores agree with human judgment on a sample; also handles the highest-stakes/most ambiguous cases directly

Refine judge prompt/criteria, re-validateif divergence found

8. Production considerations

• Always calibrate an LLM judge against human judgment before trusting it at scale (section 4/6) — this is a required validation step, not optional extra rigor.
• Measure inter-annotator agreement (Cohen's/Fleiss' kappa) before treating human labels as calibration ground truth (section 4) — a calibration run against unreconciled, low-agreement human labels tells you nothing trustworthy about the judge, since you have no evidence the labels themselves are reliable.
• Use zero or low temperature for judge calls, and consider averaging multiple calls for high-stakes evaluation (section 4) — mitigates the inconsistency failure mode directly.
• Be explicit and specific in judge prompts about what NOT to reward (e.g., explicitly instructing against favoring verbosity) — the section 6 example shows this is often necessary to counteract a documented bias, not something a judge naturally avoids on its own.
• Re-calibrate periodically, not just once — as prompts, models, and the judge itself evolve (Part 3.11's "the model landscape shifts over time" applies to judge models too), a calibration that held six months ago may no longer.

9. Common mistakes

• Trusting an LLM judge's scores uncritically without ever validating against human judgment, potentially optimizing a system against the judge's specific biases rather than genuine quality (section 6's exact failure).
• Using a judge prompt with vague, underspecified criteria, producing inconsistent or unpredictable scoring behavior that's hard to interpret or trust.
• Never re-calibrating after a judge model or prompt change, missing a newly-introduced bias.
• Using pairwise comparison (Part 6.2) without swapping presentation order to check for and mitigate position bias.

10. Security considerations

• An LLM judge processing potentially sensitive content (the outputs being evaluated) carries the same third-party-data-handling considerations as any other LLM call touching that data (Part 9.6) — don't treat "it's just for evaluation" as exempting this call from the same data-sensitivity review as a production-facing call.

11. Performance considerations

• Averaging multiple judge calls for high-stakes evaluation (section 8) multiplies latency/cost for that specific evaluation — a deliberate trade-off reserved for cases where the added confidence justifies it, not a universal default.

12. Cost considerations

• Every LLM-as-judge call is a real, additional LLM call cost (Part 6.2, section 12) — human evaluation has a different but equally real cost (reviewer time) — the calibration-then-scale pattern (validate with humans on a sample, then run the cheaper/faster LLM judge at full production volume) is specifically designed to get most of human evaluation's trustworthiness at a fraction of its cost, by concentrating the expensive human effort where it has the highest leverage (calibration and the highest-stakes cases) rather than spreading it thin across all volume.

13. When to use it

LLM-as-judge: any reference-free evaluation need at real production volume (Part 6.3's online evaluation, Part 8.1/8.2's faithfulness/relevance checking), always paired with periodic human calibration. Human evaluation: defining evaluation criteria initially, calibrating the LLM judge, and reviewing the highest-stakes or most ambiguous individual cases.

14. When NOT to use it

Don't use LLM-as-judge for properties that have a genuine, mechanically-checkable correct answer (Part 3.2's schema conformance, an exact-match factual lookup) — a deterministic evaluator (Part 6.2) is cheaper, faster, and perfectly reproducible for these, and using an LLM judge instead is both wasteful and introduces unnecessary inconsistency risk for a property that didn't need judgment at all.

15. Alternatives and trade-offs

Approach	Good for	Weak point
LLM-as-judge	Scalable, fast, reference-free evaluation	Documented biases (position, verbosity, self-preference); needs calibration
Human evaluation	Highest fidelity, defines criteria, catches judge failures	Slow, expensive, doesn't scale to production volume
Deterministic evaluator (Part 6.2)	Objective, reproducible, cheap	Only fits mechanically-checkable properties

16. Practical Python/code example

A pairwise LLM-as-judge comparison with position-bias mitigation (section 4), by running the comparison twice with swapped order:

async def compare_responses_bias_mitigated(client, question: str, response_a: str, response_b: str) -> str:
    """
    Compares two responses via LLM-as-judge, running the comparison twice with
    swapped presentation order to detect and mitigate position bias.

    Returns:
        str: "a", "b", or "tie" — only trusted if both orderings agree.
    """
    async def judge_once(first: str, second: str, first_label: str, second_label: str) -> str:
        response = await client.messages.create(
            model="claude-sonnet-4-5", max_tokens=50,
            system="Compare which response better answers the question. Respond with only 'first' or 'second'.",
            messages=[{"role": "user", "content": f"Question: {question}\n\nFirst: {first}\n\nSecond: {second}"}],
        )
        choice = response.content[0].text.strip().lower()
        return first_label if choice == "first" else second_label

    result_1 = await judge_once(response_a, response_b, "a", "b")
    result_2 = await judge_once(response_b, response_a, "b", "a")  # swapped order

    if result_1 == result_2:
        return result_1
    return "tie"  # disagreement between orderings signals position bias, not a real preference

17. Production-quality example

A calibration harness comparing judge scores against a human-labeled sample, directly implementing section 8's required validation step:

from dataclasses import dataclass
from scipy.stats import pearsonr  # or an equivalent correlation measure


@dataclass
class CalibrationResult:
    """Result of validating an LLM judge against human-labeled ground truth."""
    correlation: float
    sample_size: int
    well_calibrated: bool


async def calibrate_judge_against_humans(
    client, judge_fn, human_labeled_sample: list[dict], min_correlation: float = 0.7
) -> CalibrationResult:
    """
    Runs the LLM judge against a human-labeled sample and checks agreement,
    required before trusting the judge at full production scale.

    Args:
        client: An async LLM client.
        judge_fn: The LLM-as-judge function being calibrated.
        human_labeled_sample (list[dict]): Examples with human-assigned scores.
        min_correlation (float): Minimum acceptable correlation to consider the
            judge well-calibrated.

    Returns:
        CalibrationResult: Whether the judge is sufficiently aligned with human judgment.
    """
    judge_scores = []
    human_scores = []
    for example in human_labeled_sample:
        judge_score = await judge_fn(client, example["question"], example["response"])
        judge_scores.append(judge_score)
        human_scores.append(example["human_score"])

    correlation, _ = pearsonr(judge_scores, human_scores)
    return CalibrationResult(
        correlation=correlation,
        sample_size=len(human_labeled_sample),
        well_calibrated=correlation >= min_correlation,
    )

18. Short exercise

A calibration run shows an LLM judge's scores correlate with human scores at only 0.45 (below a reasonable threshold). List three specific hypotheses (referencing this chapter's documented bias types) you'd investigate before concluding the judge simply "doesn't work" for this evaluation task.

19. Interview questions

• Explain verbosity bias in LLM-as-judge evaluation and describe a concrete way to detect it in your own evaluation pipeline.
• Why is calibrating an LLM judge against human judgment a required step, not optional extra rigor?
• Design a pairwise comparison evaluation that mitigates position bias — what specifically would you do differently from a naive single-pass comparison?

20. FDE/customer scenario

Customer: "We're using an LLM to automatically grade our AI's responses — how do we know the grader itself is any good?"

This is exactly the calibration question this chapter centers on: recommending a periodic human-calibration sample (comparing the LLM judge's scores against independent human review, section 8/17) is the concrete, credible answer — without it, the customer has no actual evidence their automated grading system isn't itself systematically biased in ways that could be silently rewarding the wrong behavior, exactly as happened in section 6's real-world example.

---

Key takeaways

• LLM-as-judge is a scalable proxy for human judgment with well-documented, specific failure modes (position bias, verbosity bias, self-preference bias, inconsistency) — not a neutral, unbiased ground truth.
• Calibrating an LLM judge against human judgment on a sample is a required validation step, not optional extra rigor.
• Human evaluation remains necessary for defining criteria, calibration, and the highest-stakes individual cases, even as LLM-as-judge handles production-scale volume.

Things you should be able to explain

• The specific documented biases in LLM-as-judge evaluation and how each manifests.
• Why calibration against human judgment is necessary before trusting an LLM judge at scale.

Things you should be able to build

• A position-bias-mitigated pairwise comparison and a judge-calibration harness measuring agreement with human-labeled ground truth.

Common mistakes

• Trusting LLM judge scores uncritically without human calibration.
• Not mitigating position bias in pairwise comparisons.
• Never re-calibrating after a judge model or prompt change.

Recommended next chapter

04-observability-tracing-monitoring.md