6.2 — LangSmith Datasets and Evaluations

1. What is it?

LangSmith's datasets and evaluations tooling is the concrete implementation of the evaluation discipline Part 3.1/1.9 have referenced repeatedly ("build a regression eval set before changing a prompt") — a dataset is a collection of examples (input plus expected/reference output), and an evaluation (LangSmith calls a run of an evaluation an "experiment") runs your application against every example in a dataset and scores each result using one or more evaluators. Part 8 of this book covers the conceptual foundations of LLM/RAG/agent evaluation in full depth; this chapter covers LangSmith specifically as the tool implementing that discipline for LangChain/LangGraph applications.

2. Why does it exist?

Part 3.1, section 8 argued that prompt changes need a regression eval set, not ad hoc "try it a few times" iteration. Building this evaluation infrastructure by hand — maintaining a dataset, running your application against every example, scoring the results, tracking scores across versions over time — is real, repetitive engineering work. LangSmith's datasets and evaluation tooling exists to make this a first-class, low-friction workflow, integrated directly with the tracing infrastructure from Part 6.1 (an evaluation run produces the same rich, inspectable traces as any other execution).

3. What problem does it solve?

It solves "how do I systematically, repeatably measure whether a change (a new prompt, a new model, a new retrieval strategy) makes my application better or worse, across a representative set of cases, rather than relying on spot-checking a few examples by hand" — directly operationalizing this book's repeated argument (Part 3.1, Part 1.9, Part 8) that AI systems need the same measurement discipline as any other engineering artifact.

4. How does it work internally?

Building a dataset — including directly from production traces

A dataset is a collection of examples, each with an input and (usually) a reference/expected output. LangSmith supports building datasets multiple ways: manually curated examples, or — a genuinely powerful, distinctive capability — built directly from production traces (Part 6.1) via the SDK, letting you pull real, representative production interactions (including specifically interesting ones: failures a user flagged, low-confidence responses, edge cases discovered in the wild) directly into a dataset, rather than needing to hand-author synthetic examples that may not represent real usage patterns well.

Production traces (Part 6.1) ──► curate/select interesting ones ──► Dataset
       │                                                                │
       │ (e.g., every trace where a user gave negative feedback,        │
       │  every trace that took unusually long, every trace flagged     │
       │  by an online evaluator as low-quality, Part 6.3)               │
                                                                         ▼
                                                          used as regression
                                                          test cases going forward

This directly closes a real gap: a synthetic, hand-written eval set risks missing the actual failure patterns real users produce; a dataset built from real production traces (especially ones that were flagged as problematic) is, by construction, testing against genuine, observed failure modes.

The four evaluator families

LangSmith organizes evaluators into four families, each suited to different evaluation needs (Part 8 covers the conceptual reasoning behind choosing among these in depth):

• LLM-as-judge: a separate LLM call scores or critiques your application's output against criteria you define (Part 8.3 covers this technique's mechanics, strengths, and pitfalls in full depth) — useful for qualities that are hard to check with a simple rule (tone, helpfulness, faithfulness to retrieved context).
• Deterministic/code-based evaluators: plain code checking exact-match, containment, format validity (does the output parse as valid JSON matching a schema, Part 3.2), or any other programmatically-checkable property — cheap, fast, perfectly reproducible, but limited to properties expressible as code.
• Human review: routing examples (or a sample of them) to a human reviewer for scoring/annotation — the highest-fidelity but slowest and most expensive option, reserved for cases where automated evaluation genuinely can't substitute (Part 8.3's human-evaluation discussion).
• Pairwise comparison: comparing two different outputs for the same input (e.g., from two different prompt versions, or two different models) and judging which is better, rather than scoring each in isolation — often more reliable than absolute scoring for subjective qualities, since it's frequently easier for a judge (human or LLM) to say "A is better than B" confidently than to assign a consistent absolute score to A and B independently.

Experiments — tying evaluation runs to traces over time

Running an evaluation (an "experiment," in LangSmith's terminology) against a dataset produces, for every example: your application's actual output, every configured evaluator's score, and the full trace (Part 6.1) of that specific execution — meaning a low score isn't just a number, it's immediately connected to the complete, inspectable record of exactly what happened, closing the loop between "this scored badly" and "here's precisely why" without needing separate debugging effort to connect the two.

Offline vs. online evaluation — a critical distinction

• Offline evaluation: running an experiment against a fixed dataset, typically before deploying a change (a regression gate, directly implementing Part 3.1's prompt-versioning discipline, and Part 1.7's CI-gate example).
• Online evaluation: running evaluators continuously against live production traffic/traces (Part 6.1), typically sampling some or all real production interactions, to monitor ongoing quality in real time rather than only at deployment checkpoints.

These serve genuinely different purposes and are both necessary in a mature system: offline evaluation catches regressions before they reach users (a pre-deployment gate); online evaluation catches quality degradation that only manifests under real production conditions and real user behavior that a fixed offline dataset couldn't fully anticipate (Part 8.4/6.3 cover online monitoring in more depth).

5. Simple mental model

A LangSmith dataset-and-evaluation setup is like a standardized exam with an answer key, re-administered to every new version of a student (your application) before letting them graduate to production — and building the exam's questions directly from real situations the student has actually encountered (production traces) rather than only hypothetical textbook problems makes the exam a much more honest predictor of real-world performance. Offline evaluation is the exam given before graduation (before deployment); online evaluation is more like ongoing performance reviews on the job (continuous monitoring of real, live work) — both matter, and neither substitutes for the other.

6. Real-world example

A team preparing a prompt change for their support agent (Part 3.1's exact scenario) builds a dataset from 200 production traces flagged with negative user feedback over the past month, then runs an offline experiment comparing the current prompt against the proposed new one, using an LLM-as-judge evaluator scoring "does this response actually address the user's stated problem" plus a deterministic evaluator checking response length stays within a reasonable bound. The experiment reveals the new prompt improves the judge's helpfulness score but occasionally produces responses exceeding the length bound — concrete, quantified evidence that lets the team decide (fix the length issue before shipping, or accept the trade-off deliberately) rather than shipping based on a few manually-reviewed examples looking better.

7. Architecture diagram

Production tracesPart 6.1

Datasetinput + reference examples, built from real observed cases including flagged ones

Experiment runactual output, evaluator scores (LLM-as-judge, code-based, human review, pairwise), full trace per example

Regression tracking / deployment gatecompare across experiments over time · Part 1.7

8. Production considerations

• Build datasets from real production traces (especially flagged/problematic ones), not only hand-written synthetic examples — the section 4 argument for representativeness applies with real force; a synthetic-only dataset risks systematically missing your application's actual failure patterns.
• Wire offline evaluation into your CI/CD pipeline as a deployment gate (Part 1.7's exact example, now concretely implementable with LangSmith's experiment/dataset APIs) — a prompt or model change shouldn't merge or deploy without passing the regression evaluation.
• Choose evaluator type deliberately per quality you're measuring — don't default to LLM-as-judge for everything when a deterministic check (does the output parse correctly, Part 3.2) would be cheaper, faster, and perfectly reproducible for that specific property.
• Run online evaluation continuously in production, not just offline gates before deployment — this is what catches quality drift that only manifests under real, evolving production conditions (Part 6.3, Part 8.4).

9. Common mistakes

• Building an eval dataset once, early in a project, and never updating it as real production usage reveals new failure patterns — an eval set that doesn't evolve with your actual observed traffic loses relevance over time.
• Using only LLM-as-judge evaluation for properties that a cheap, deterministic code check could verify more reliably and cheaply (Part 3.2's schema-conformance checks are a perfect fit for deterministic evaluators, not LLM judges).
• Treating offline evaluation as sufficient on its own, without any online/continuous monitoring, and missing quality degradation that only shows up under real production traffic patterns the offline dataset didn't anticipate.
• Not connecting evaluation results back to traces for debugging — running an evaluation and only looking at the aggregate score, missing the diagnostic value of drilling into exactly which examples failed and why (available directly via the trace, section 4).

10. Security considerations

• Datasets built from production traces inherit whatever sensitive data (Part 9.6) was present in those original interactions — apply the same data-handling discipline to dataset storage and access as to the trace data it's built from (Part 6.1, section 10).
• LLM-as-judge evaluators, being themselves LLM calls, process whatever content is in the examples being evaluated — if that content is sensitive, the same third-party-data-handling considerations apply as to any other LLM call touching that data (Part 9.6).

11. Performance considerations

• Running a large evaluation experiment (many examples, especially with LLM-as-judge evaluators making their own separate LLM calls per example) has real, non-trivial latency — plan CI/CD gate timing accordingly (Part 7's CI/CD discussion) rather than assuming evaluation is instantaneous.

12. Cost considerations

• Every example scored by an LLM-as-judge evaluator is an additional LLM call, on top of your application's own calls being evaluated — a large dataset with an LLM-judge evaluator run frequently (e.g., on every PR) has a real, additive cost worth accounting for, and is a legitimate reason to prefer deterministic evaluators wherever they suffice (section 9).
• Online evaluation running continuously against a sample (or all) of production traffic likewise adds ongoing LLM-judge cost proportional to sampling rate — a real trade-off between coverage and cost worth tuning deliberately.

13. When to use it

Before any meaningful change to a prompt, model, or retrieval strategy (offline evaluation, as a deployment gate) and continuously in production for ongoing quality monitoring (online evaluation) — essentially the standard, expected practice for any production LLM application, per this book's repeated argument that AI systems need the same measurement rigor as any other engineering artifact.

14. When NOT to use it

A one-off exploratory prototype with no production deployment plan doesn't need the full dataset/CI-gate infrastructure — though even here, at least a handful of representative test cases checked informally before iterating further is worth the minimal effort, per Part 1.9's general testing philosophy.

15. Alternatives and trade-offs

Approach	Good for	Weak point
LangSmith datasets/evaluations	Integrated with tracing, four evaluator families, deployment-gate-ready	Hosted-service dependency, own cost/data-sensitivity considerations
Hand-rolled eval scripts	Full control, no new dependency	More engineering effort to reach parity, especially for trace-linked debugging
Manual spot-checking (anti-pattern)	Zero setup	Not systematic, not reproducible, misses regressions (Part 3.1's core warning)

16. Practical Python/code example

from langsmith import Client
from langsmith.evaluation import evaluate

client = Client()

# Build a dataset from curated production traces (conceptually — actual API
# calls for pulling from traces should be verified against current docs)
dataset = client.create_dataset(dataset_name="support-agent-regression-v1")
client.create_examples(
    inputs=[{"question": "Where is my order #4521?"}],
    outputs=[{"expected_topic": "order_status"}],
    dataset_id=dataset.id,
)


def correctness_evaluator(run, example) -> dict:
    """A deterministic evaluator checking the response addresses the expected topic."""
    addressed = example.outputs["expected_topic"] in run.outputs.get("response", "").lower()
    return {"key": "addresses_topic", "score": 1.0 if addressed else 0.0}


results = evaluate(
    lambda inputs: run_support_agent(inputs["question"]),
    data="support-agent-regression-v1",
    evaluators=[correctness_evaluator],
)

Verify the exact current SDK function signatures (evaluate, create_examples, etc.) against current LangSmith documentation before shipping — this is an actively maintained part of the API surface.

17. Production-quality example

A CI-gate script wiring evaluation results into a deployment decision, directly implementing Part 1.7's CI-gate pattern with LangSmith as the evaluation engine:

import sys
import logging

logger = logging.getLogger("eval_gate")

MIN_PASS_RATE = 0.90


def run_regression_gate(dataset_name: str, evaluators: list) -> bool:
    """
    Runs the regression evaluation suite and returns whether the change passes
    the required threshold, for use as a CI/CD deployment gate (Part 1.7).

    Args:
        dataset_name (str): The LangSmith dataset to evaluate against.
        evaluators (list): Evaluator functions to score each example.

    Returns:
        bool: True if the pass rate meets or exceeds MIN_PASS_RATE.
    """
    results = evaluate(
        lambda inputs: run_support_agent(inputs["question"]),
        data=dataset_name,
        evaluators=evaluators,
    )
    scores = [r["evaluation_results"]["results"][0].score for r in results]
    pass_rate = sum(1 for s in scores if s >= 1.0) / len(scores)

    logger.info("regression gate: pass_rate=%.2f (threshold=%.2f)", pass_rate, MIN_PASS_RATE)
    return pass_rate >= MIN_PASS_RATE


if __name__ == "__main__":
    passed = run_regression_gate("support-agent-regression-v1", [correctness_evaluator])
    sys.exit(0 if passed else 1)  # non-zero exit fails the CI build, blocking merge/deploy

18. Short exercise

A team wants to evaluate whether their RAG system's responses are faithful to retrieved context (Part 3.5, section 4's faithfulness concern) versus whether the responses are simply well-formatted. Decide which of the four evaluator families (LLM-as-judge, deterministic, human review, pairwise) fits each of these two distinct properties best, and justify your choice for each.

19. Interview questions

• Why is building a dataset from real production traces generally more valuable than only hand-writing synthetic examples?
• Explain the difference between offline and online evaluation, and why a mature system needs both.
• When would you choose a deterministic evaluator over an LLM-as-judge evaluator, and vice versa?

20. FDE/customer scenario

Customer: "How do we know a prompt change our team makes won't accidentally make things worse somewhere we didn't think to check?"

This is precisely what an offline regression evaluation gate (section 8, 17), built from a dataset of real production cases (ideally including previously-flagged problem cases), is designed to catch — being able to describe (and ideally demonstrate) a concrete CI-integrated evaluation gate, rather than "we'll test it carefully by hand," is a substantially more credible and technically grounded answer to exactly the kind of quality-assurance question a careful enterprise customer will ask before trusting an AI system with real business processes.

---

Key takeaways

• LangSmith datasets, built ideally from real (especially flagged) production traces, plus its four evaluator families (LLM-as-judge, deterministic, human review, pairwise), operationalize this book's repeated argument for systematic, measured evaluation over ad hoc spot-checking.
• Experiments tie evaluation scores directly to full traces, closing the loop between "this scored badly" and "here's exactly why" without separate debugging effort.
• Offline evaluation (a pre-deployment gate) and online evaluation (continuous production monitoring) serve different purposes and are both necessary.

Things you should be able to explain

• The four evaluator families and when each is the appropriate choice.
• Why building datasets from production traces produces more representative evaluation than synthetic-only examples.

Things you should be able to build

• A CI-integrated regression evaluation gate using a dataset and a mix of deterministic and LLM-as-judge evaluators.

Common mistakes

• Static eval datasets never updated with real observed failure patterns.
• Using LLM-as-judge for properties a deterministic check could verify more reliably and cheaply.
• Offline evaluation with no complementary online/continuous monitoring.

Recommended next chapter

03-production-monitoring-regression.md