8.1 — Evaluating LLMs, RAG, and Agents

1. What is it?

This chapter is the conceptual foundation for the evaluation practice Part 6.2/6.3 implemented using LangSmith specifically — what actually makes a good evaluation, what distinct properties need distinct measurement approaches for LLM outputs, RAG pipelines, and agents respectively, and how to design an evaluation strategy that's genuinely trustworthy rather than superficially reassuring.

2. Why does it exist?

Part 1.9 drew a hard line between traditional software testing (exact-match assertions work fine) and evaluating non-deterministic LLM output (exact-match assertions don't). This chapter exists to give you the actual methodology for that second category — the specific techniques, metrics, and design principles for measuring quality when "correct" isn't a single string you can compare against, and where RAG and agent systems each introduce their own additional properties beyond plain LLM output quality that need their own measurement approach.

3. What problem does it solve?

It solves "how do I know, with actual evidence rather than vibes, whether my LLM/RAG/agent system is good, and specifically how good at which specific things" — the difference between a system that's been genuinely validated and one that merely seemed fine on the handful of examples someone happened to try.

4. How does it work internally?

The layered evaluation problem — different systems need different measured properties

Plain LLM outputcorrectness, relevance, coherence/fluency, format adherence (Part 3.2)

RAG systemretrieval relevance, faithfulness (Part 8.2) — does the answer reflect retrieved content, answer relevance to query

Agent systemtask completion, correct tool selection, efficient path (Part 3.7), safety — no excessive agency (Part 9.2)

Each layer down adds properties the layer above doesn't need to measure — a RAG system needs everything a plain LLM output needs to measure plus whether retrieval and faithfulness are working correctly; an agent system needs everything RAG needs plus whether the sequence of decisions and actions taken was itself correct and efficient. Evaluating an agent system only on its final answer's correctness, while ignoring whether it took an unnecessarily convoluted or unsafe path to get there (Part 3.7's efficiency and Part 9.2's safety concerns), is a genuinely incomplete evaluation, even if the final-answer metric looks good.

Measuring retrieval relevance concretely: precision@k, recall@k, MRR, NDCG

The diagram above lists "retrieval relevance" as a property a RAG system needs to measure, but that name alone isn't a metric — it's a category. Four standard information-retrieval metrics make it concrete, and each answers a genuinely different question. All four are computed against the same two inputs: the ranked list of chunks your retriever actually returned, and the set (or graded scores) of chunks that are actually relevant to the query — normally established from a labeled evaluation set (section 4's "Building a representative evaluation set", Part 6.2).

Worked example. Say a query's true relevant set (established by a human labeler or an existing labeled corpus) is {doc2, doc5, doc9} — 3 relevant documents out of the whole corpus. Your retriever returns this ranked top-5:

Rank:      1       2       3       4       5
Chunk:   doc5    doc1    doc9    doc3    doc2
Relevant?  yes     no      yes     no      yes

• Precision@k — of the k chunks retrieved, what fraction are actually relevant. precision@k = (# relevant in top k) / k. This answers "how much noise is the LLM being handed" — directly relevant to Part 3.9's context-budget concern, since every irrelevant chunk passed to generation wastes tokens and risks distracting or misleading the model.

  • precision@3 = 2/3 ≈ 0.667 (doc5, doc9 relevant; doc1 not)
  • precision@5 = 3/5 = 0.6

• Recall@k — of all the relevant chunks that exist, what fraction did the top k actually surface. recall@k = (# relevant in top k) / (total relevant). This answers "did we miss anything that mattered" — the right metric when a missed relevant document is more costly than a few irrelevant ones tagging along (e.g., compliance/legal-discovery-style RAG, where an omitted relevant clause is a much worse failure than an extra irrelevant paragraph in the context).

  • recall@3 = 2/3 ≈ 0.667
  • recall@5 = 3/3 = 1.0 (all 3 relevant docs appeared somewhere in the top 5)

• MRR (Mean Reciprocal Rank) — for each query, take the reciprocal of the rank position of the first relevant result (1/rank), then average across all queries. Formula: MRR = (1/|Q|) × Σ (1 / rank_i) for query set Q. This answers "is a relevant result near the top, quickly" — the right choice when there's typically one expected/best answer and users only read the top result (a search-suggestion box, "find me the one clause that answers this"), and it doesn't care how many other relevant results exist further down.

  • For this query, the first relevant result (doc5) is at rank 1, so RR = 1/1 = 1.0. If instead the first relevant chunk had appeared at rank 3, RR = 1/3 ≈ 0.333. MRR is the mean of RR across every query in the evaluation set, not a single-query number.

• NDCG (Normalized Discounted Cumulative Gain) — like precision/recall, but for graded (not just binary) relevance, and it rewards relevant results appearing earlier more than later. First compute DCG: DCG@k = Σ_{i=1}^{k} (rel_i / log2(i + 1)), where rel_i is the graded relevance score of the result at rank i. Then normalize by the IDCG (the DCG of the ideal ordering — the same relevance scores sorted best-first): NDCG@k = DCG@k / IDCG@k. This is the right choice when relevance genuinely isn't binary — e.g., some chunks fully answer the query (rel=3), some are partially useful background (rel=1), most are irrelevant (rel=0) — which is the typical real shape of retrieval quality, making NDCG the metric of choice when tuning a reranker (Part 3.6).

  • Suppose graded relevance scores are doc5=3, doc9=2, doc2=1 (others =0). Using the same ranking (doc5, doc1, doc9, doc3, doc2 at ranks 1-5): DCG@5 = 3/log2(2) + 0/log2(3) + 2/log2(4) + 0/log2(5) + 1/log2(6) = 3 + 0 + 1 + 0 + 0.387 = 4.387
  • The ideal ordering sorts by relevance descending (3, 2, 1, 0, 0): IDCG@5 = 3/log2(2) + 2/log2(3) + 1/log2(4) + 0 + 0 = 3 + 1.262 + 0.5 = 4.762
  • NDCG@5 = 4.387 / 4.762 ≈ 0.921 — close to the ideal ordering, penalized mainly for putting the non-relevant doc1 ahead of doc9.

import math


def precision_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    """
    Computes what fraction of the top-k retrieved chunks are actually relevant.

    Args:
        retrieved (list[str]): Ranked chunk/document IDs returned by the retriever.
        relevant (set[str]): IDs of chunks actually relevant to the query.
        k (int): Cutoff rank to evaluate.

    Returns:
        float: Precision at k, in [0, 1].
    """
    top_k = retrieved[:k]
    return sum(1 for doc_id in top_k if doc_id in relevant) / k


def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    """
    Computes what fraction of all relevant chunks were surfaced in the top k.

    Args:
        retrieved (list[str]): Ranked chunk/document IDs returned by the retriever.
        relevant (set[str]): IDs of chunks actually relevant to the query.
        k (int): Cutoff rank to evaluate.

    Returns:
        float: Recall at k, in [0, 1].
    """
    top_k = retrieved[:k]
    return sum(1 for doc_id in top_k if doc_id in relevant) / len(relevant)


def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
    """
    Computes the reciprocal rank of the first relevant result for one query.

    Args:
        retrieved (list[str]): Ranked chunk/document IDs returned by the retriever.
        relevant (set[str]): IDs of chunks actually relevant to the query.

    Returns:
        float: 1/rank of the first relevant hit, or 0.0 if none was retrieved.
    """
    for rank, doc_id in enumerate(retrieved, start=1):
        if doc_id in relevant:
            return 1.0 / rank
    return 0.0


def ndcg_at_k(retrieved: list[str], graded_relevance: dict[str, float], k: int) -> float:
    """
    Computes NDCG@k for graded (non-binary) relevance scores.

    Args:
        retrieved (list[str]): Ranked chunk/document IDs returned by the retriever.
        graded_relevance (dict[str, float]): Relevance score per document ID
            (0 for chunks with no score, i.e. not relevant at all).
        k (int): Cutoff rank to evaluate.

    Returns:
        float: NDCG at k, in [0, 1]. Returns 0.0 if the ideal DCG is 0.
    """
    def dcg(scores: list[float]) -> float:
        return sum(rel / math.log2(i + 2) for i, rel in enumerate(scores))

    top_k_scores = [graded_relevance.get(doc_id, 0.0) for doc_id in retrieved[:k]]
    ideal_scores = sorted(graded_relevance.values(), reverse=True)[:k]
    ideal_dcg = dcg(ideal_scores)
    return dcg(top_k_scores) / ideal_dcg if ideal_dcg > 0 else 0.0


# Reproduces the worked example above:
retrieved_ranking = ["doc5", "doc1", "doc9", "doc3", "doc2"]
relevant_set = {"doc2", "doc5", "doc9"}
graded = {"doc5": 3, "doc9": 2, "doc2": 1}

assert round(precision_at_k(retrieved_ranking, relevant_set, 5), 3) == 0.6
assert round(recall_at_k(retrieved_ranking, relevant_set, 5), 3) == 1.0
assert round(reciprocal_rank(retrieved_ranking, relevant_set), 3) == 1.0
assert round(ndcg_at_k(retrieved_ranking, graded, 5), 3) == 0.921

These are exactly the metrics Part 3.5's "evaluated against your actual corpus and real user questions" and Part 3.6's "should be evaluated, not assumed to be a strict improvement" are pointing at without naming — when either chapter tells you to evaluate chunking strategy or reranking quality, precision@k/recall@k/MRR/NDCG (computed against a labeled query→relevant-chunks evaluation set) are the concrete metrics you compute to do it.

Existing eval-harness tooling — build vs. buy

Nothing in this chapter requires hand-rolling evaluation infrastructure from scratch. RAGAS (RAG-specific: faithfulness, answer relevance, context precision/recall built in as ready-made metrics), DeepEval (broader LLM-app testing, pytest-style assertions, integrates the same retrieval/generation metrics into a CI-friendly test-runner), and promptfoo (config-driven prompt/model regression testing with a lightweight, declarative test-case format) are standard tools worth evaluating before building a custom harness. The buy-vs-build trade-off: these tools give you precision@k/recall@k/NDCG/faithfulness/answer-relevance scoring, dataset management, and CI integration out of the box, at the cost of adapting your data/pipeline to their expected interfaces and accepting their specific implementation of each metric (worth spot-checking against the formulas above rather than trusting blindly, per this chapter's general "understand what you're actually measuring" theme). Building your own (as this chapter's code examples do, for teaching clarity) gives full control over metric definitions and trajectory-evaluation logic specific to your agent's tool set, at the cost of building and maintaining that infrastructure yourself. A common, pragmatic middle path: use RAGAS/DeepEval/promptfoo for the standard retrieval/generation metrics this section covers, and hand-roll only the trajectory-specific evaluators (section 16/17's evaluate_trajectory_used_required_tool) that are inherently specific to your own agent's tools and domain.

Reference-based vs. reference-free evaluation

• Reference-based: comparing output against a known-correct expected answer (Part 6.2's dataset examples with expected outputs) — works well for tasks with a genuinely correct, checkable answer (does this extraction match the expected structured fields, Part 3.2).
• Reference-free: evaluating a property of the output without a pre-written correct answer to compare against (is this response helpful, is this response faithful to the provided context, Part 8.2/8.3) — necessary for open-ended generation tasks where writing an exhaustive "correct answer" for every possible valid phrasing isn't practical, and where the property being measured (helpfulness, faithfulness) is about a relationship between the output and something else (the context, the query), not a fixed target string.

Agent-specific evaluation: trajectory, not just outcome

An agent's full "trajectory" — the sequence of reasoning steps, tool calls, and intermediate decisions (Part 3.7's ReAct pattern, captured fully in a LangSmith trace, Part 6.1) — is itself an evaluable artifact, separate from whether the final answer happened to be correct. Outcome evaluation asks "was the final answer right." Trajectory evaluation asks "was the process that produced it sound" — did it call the right tools in a sensible order, did it avoid unnecessary steps, did it correctly recover from a tool failure rather than getting stuck. A system can produce a correct final answer via a genuinely bad, inefficient, or unsafe process (getting lucky), and trajectory evaluation is what catches this — directly relevant because a process that "got lucky" once is a much less reliable system going forward than one whose process was actually sound.

Building a representative evaluation set — the foundation everything else depends on

Every evaluation technique in this chapter and Part 8.2/8.3 is only as good as the dataset it's run against (Part 6.2's dataset-building discussion) — a dataset that doesn't represent the actual distribution of real inputs (missing edge cases, missing adversarial inputs, missing the specific failure patterns real users produce, Part 6.2's "build from production traces" argument) will give a falsely reassuring evaluation result regardless of how sophisticated the scoring methodology on top of it is. This is worth stating explicitly here because it's easy, once absorbed in the details of a specific scoring technique, to forget that the dataset's representativeness is the actual foundation the whole evaluation's validity rests on.

5. Simple mental model

Evaluating a plain LLM output is like grading a single essay answer — you check correctness, clarity, and whether it followed the assignment's format. Evaluating a RAG system is like grading the same essay, but also checking whether the student actually used and correctly cited the specific reference materials they were given, rather than writing from memory or making things up despite having the right sources available. Evaluating an agent is like grading not just the final essay, but the student's entire research and drafting process — did they look up the right sources in a sensible order, did they waste time on irrelevant tangents, did they recover sensibly when an initial source turned out to be unhelpful — because a good final essay produced through a chaotic, lucky process is a worse signal about the student's actual reliable capability than the same essay produced through a sound process.

6. Real-world example

An insurance claims-processing agent (Part 3.7's recurring example) is evaluated only on "did it reach the correct approve/deny recommendation" for a test set of historical claims — and scores well. A deeper trajectory evaluation reveals that in a meaningful fraction of "correct" cases, the agent skipped the fraud-check tool entirely (Part 3.7's real-world redesign example) and still happened to land on the right recommendation by chance, because the specific test claims in the dataset didn't happen to have fraud indicators that mattered. Outcome-only evaluation would have missed this significant reliability gap entirely — the system was one input away from a serious, systematic failure (approving genuinely fraudulent claims by skipping the check that would have caught them) that only trajectory evaluation surfaced.

7. Architecture diagram

Evaluation setPart 6.2 · must be representative of real inputs, including edge cases and known failure patterns

Outcome evaluationwas the final answer correct/relevant/well-formatted? (reference-based or reference-free, per task)

Trajectory evaluation (agents)was the PROCESS sound — right tools, sensible order, efficient, safe? (Part 3.7, Part 9.2)

Honest, complete quality picture

8. Production considerations

• Never evaluate an agent system on outcome alone — section 6's example is the concrete, recurring reason trajectory evaluation is necessary, not optional rigor, for any agent whose process has real safety or compliance implications.
• Continuously audit your evaluation dataset's representativeness (Part 6.2/6.3's feedback-loop discussion) — a static dataset, however good initially, degrades in representativeness as real usage patterns evolve.
• Match the evaluation technique to the property being measured — don't force reference-based exact-match evaluation onto genuinely open-ended generation tasks (Part 1.9's original warning), and don't skip reference-based evaluation for tasks that genuinely do have a checkable correct answer (Part 3.2's structured extraction) just because LLM-as-judge (Part 8.3) feels more sophisticated.

9. Common mistakes

• Evaluating an agent purely on final-outcome correctness, missing systematic process failures that happened not to matter for the specific test cases used (section 6's exact scenario).
• Using a non-representative evaluation dataset (hand-picked "nice" examples, or a synthetic set that doesn't reflect real usage) and treating a good score against it as strong evidence of real-world quality.
• Applying reference-based exact-match scoring to genuinely open-ended, multiple-valid-phrasing tasks, producing a misleadingly low score that reflects a rigid scoring method's limitation, not the system's actual quality.

10. Security considerations

• Trajectory evaluation is directly relevant to Part 9's security concerns: an agent that reaches a correct outcome via a path that included an unauthorized or excessively broad tool call (even if that specific call happened not to cause harm in this instance) represents real risk that outcome-only evaluation would completely miss.

11. Performance considerations

• Full trajectory evaluation (inspecting every step of every trace, Part 6.1) is more expensive and slower than outcome-only evaluation, especially with human review (Part 8.3) — a reasonable, common practice is applying full trajectory evaluation to a representative sample rather than every single evaluation run, reserving full-coverage outcome evaluation for the cheaper, faster check.

12. Cost considerations

• Building and maintaining a genuinely representative evaluation dataset, and running comprehensive (including trajectory-level) evaluation regularly, is a real, ongoing engineering investment — but the cost of not doing this (a serious, undetected reliability or safety gap reaching production, section 6) is typically far higher, an argument worth making explicitly when a customer questions the investment in evaluation infrastructure.

13. When to use it

Comprehensive (outcome plus trajectory, where relevant) evaluation should be standard practice for any production LLM/RAG/agent system, applied both before deployment (Part 6.2's offline gate) and continuously afterward (Part 6.3's online monitoring).

14. When NOT to use it

Full trajectory-level evaluation may be more rigor than a very low-stakes, simple, single-step LLM call needs — proportionality (Part 3.12's "scale rigor to actual stakes" principle) applies to evaluation depth just as it does to architecture.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Outcome-only evaluation	Simple, fast, sufficient for single-step tasks	Misses process-level failures in multi-step agents (section 6)
Outcome + trajectory evaluation	Catches both output and process quality issues	More expensive, more complex to implement and interpret
Reference-based scoring	Objective, reproducible for tasks with a genuine correct answer	Doesn't fit open-ended generation tasks
Reference-free scoring	Fits open-ended generation	Requires a trustworthy scoring method (Part 8.3) since there's no ground truth to check against

16. Practical Python/code example

A trajectory-aware evaluator checking whether an agent used a required tool, not just whether its final answer was correct — directly implementing section 6's lesson:

def evaluate_trajectory_used_required_tool(trace, required_tool_name: str) -> dict:
    """
    Checks whether an agent's trajectory included a required tool call, catching
    'correct by luck' outcomes that skipped a safety-critical step.

    Args:
        trace: The full execution trace (Part 6.1), including every tool call made.
        required_tool_name (str): A tool that should always be called for this task
            category (e.g., "fraud_check" for loan/claims review).

    Returns:
        dict: {"key": "used_required_tool", "score": 1.0 or 0.0}
    """
    tool_calls_made = [run.name for run in trace.child_runs if run.run_type == "tool"]
    used_required_tool = required_tool_name in tool_calls_made
    return {"key": f"used_{required_tool_name}", "score": 1.0 if used_required_tool else 0.0}

17. Production-quality example

A combined outcome-plus-trajectory evaluation report, giving an honest, complete quality picture rather than a single potentially-misleading number:

from dataclasses import dataclass


@dataclass
class ComprehensiveEvalResult:
    """An evaluation result distinguishing outcome correctness from process soundness."""
    example_id: str
    outcome_correct: bool
    used_required_tools: bool
    unnecessary_steps_count: int
    trajectory_sound: bool


def evaluate_comprehensively(trace, expected_outcome, required_tools: list[str]) -> ComprehensiveEvalResult:
    """
    Produces a combined outcome-and-trajectory evaluation, so a correct final
    answer reached via an unsound process is flagged rather than hidden inside
    a single passing score.

    Args:
        trace: The full execution trace.
        expected_outcome: The expected/reference final outcome for this example.
        required_tools (list[str]): Tools that must appear in the trajectory.

    Returns:
        ComprehensiveEvalResult: A structured, honest evaluation covering both
            what the system concluded and how it got there.
    """
    actual_outcome = trace.outputs.get("final_answer")
    tool_calls_made = {run.name for run in trace.child_runs if run.run_type == "tool"}
    used_required = all(tool in tool_calls_made for tool in required_tools)
    unnecessary_steps = max(0, len(trace.child_runs) - len(required_tools) - 2)

    return ComprehensiveEvalResult(
        example_id=trace.id,
        outcome_correct=(actual_outcome == expected_outcome),
        used_required_tools=used_required,
        unnecessary_steps_count=unnecessary_steps,
        trajectory_sound=used_required and unnecessary_steps < 3,
    )

18. Short exercise

An agent evaluation shows 95% outcome correctness but only 70% of trajectories used all required safety-check tools. Write a one-paragraph explanation, suitable for a non-technical stakeholder, of why the 95% number alone is not sufficient evidence the system is ready for production, using this chapter's outcome-vs-trajectory distinction.

19. Interview questions

• Explain the difference between outcome evaluation and trajectory evaluation for an agent, and give a concrete example of a system that would score well on one but poorly on the other.
• Why does an evaluation dataset's representativeness matter more than the sophistication of the scoring method applied to it?
• When would reference-based evaluation be the wrong choice, even though it's simpler and more objective than reference-free evaluation?

20. FDE/customer scenario

Customer: "Our agent gets the right answer 95% of the time in our testing — isn't that good enough to launch?"

This is precisely the section 6/18 scenario, and the FDE-correct response asks a follow-up before answering: has trajectory been evaluated, not just outcome, particularly for any safety- or compliance-critical steps the agent is supposed to take? A high outcome-correctness number can coexist with a serious, undetected process reliability gap — and surfacing this distinction, with a concrete example like section 6's fraud-check scenario, is exactly the kind of rigor that separates a credible pre-launch AI reliability assessment from a superficially reassuring one.

---

Key takeaways

• Plain LLM output, RAG systems, and agents each need progressively more evaluated properties — evaluating only what the simplest layer needs misses real reliability gaps in more complex systems.
• Agent evaluation must include trajectory (was the process sound), not just outcome (was the final answer correct) — a correct answer can be reached via an unsound, unsafe, or lucky process.
• Every evaluation technique's validity rests on the representativeness of its underlying dataset.

Things you should be able to explain

• Why outcome-only evaluation is insufficient for agent systems specifically.
• The difference between reference-based and reference-free evaluation and when each fits.

Things you should be able to build

• A combined outcome-and-trajectory evaluator that flags a correct answer reached via an unsound process.

Common mistakes

• Outcome-only evaluation for agents, missing systematic process failures.
• Non-representative evaluation datasets producing falsely reassuring results.

Recommended next chapter

02-hallucination-faithfulness-relevance.md