6.3 — Production Monitoring and Regression Testing with LangSmith

1. What is it?

This chapter synthesizes Parts 6.1 (tracing) and 6.2 (datasets/evaluation) into an ongoing production monitoring practice: using online evaluation (Part 6.2's distinction) continuously against live traffic, tracking quality/cost/latency trends over time, detecting regressions after deployment (not just before it), and feeding production learnings back into the offline evaluation datasets that gate future changes — closing the full loop this Part has been building toward.

2. Why does it exist?

Part 6.2 established that offline evaluation, however good, only tests against a fixed, necessarily-incomplete set of examples — it can't anticipate every real-world input pattern, and it says nothing about how the system is actually performing right now, in production, against real live traffic. Part 8 (upcoming) will establish the full conceptual case for continuous evaluation and monitoring in AI systems generally; this chapter covers how LangSmith specifically implements the "close the loop between production and evaluation" practice using the tracing and dataset/evaluation tooling from Parts 6.1–6.2 together.

3. What problem does it solve?

It solves "how do I know my AI system's quality, cost, and latency are staying healthy in production over time, catch degradation as it happens rather than only when a customer complains, and continuously improve my evaluation coverage using what production actually reveals" — the ongoing operational discipline that offline evaluation and one-time launch testing alone can't provide.

4. How does it work internally?

Online evaluation — evaluators running against live traces

Part 6.2 introduced online evaluation as running evaluators continuously against production traces rather than only a fixed offline dataset. Mechanically, this means configuring one or more of Part 6.2's evaluator types (most commonly LLM-as-judge or deterministic checks, since human review doesn't scale to full production volume, and pairwise comparison needs two variants to compare which isn't always available in a single production stream) to run automatically against some or all incoming production traces (Part 6.1), typically sampled at a configured rate to manage cost (Part 6.2, section 12).

Live production traffic

Trace capturedPart 6.1, sampled at configured rate

Online evaluator scores the tracee.g. LLM-as-judge: "was this helpful and faithful to context?"

Score attached, aggregated into dashboards/trendstriggers alerts if scores degrade below threshold

Regression detection over time — comparing experiments across versions

Because every evaluation run (Part 6.2's "experiment") is tied to a specific version of your application (a specific prompt version, model, or code commit — Part 1.7's versioning discipline made concrete here), you can compare experiment results across versions over time, not just as a single pass/fail gate at one point in time. This is what turns evaluation from a one-time deployment check into an ongoing regression-tracking practice: a gradual, creeping decline in a specific evaluator's score across several recent versions (even if each individual version passed its own deployment gate, Part 6.2's threshold) is a genuine early-warning signal that a fixed offline pass/fail gate alone would miss.

Closing the loop — production traces feeding back into evaluation datasets

This is the practice that ties this entire Part together: online evaluators flag interactions with low scores (or negative user feedback captured directly, another common LangSmith capability — collecting explicit user feedback like a thumbs-down and attaching it to the relevant trace); these flagged, real, production-observed failure cases are then curated into the offline regression dataset (Part 6.2's dataset-building-from-traces capability), so future prompt/model changes are automatically tested against exactly the kind of real failure the system has actually encountered — a continuously self-improving evaluation practice, rather than a static dataset that goes stale.

Production traffic → traces → online evaluationflags low-scoring / negatively-rated interactions (Part 6.1)

Curated into offline regression datasetPart 6.2

Future changes gated against this updated datasetPart 6.2's CI gate, before deployment

New version deployedgenerates new production traces

5. Simple mental model

If offline evaluation (Part 6.2) is a pre-flight checklist run once before takeoff, online monitoring is the aircraft's continuous in-flight instrumentation, plus a practice of adding any newly-discovered failure mode to next time's pre-flight checklist. A plane doesn't just get checked once at the factory and then trusted forever — it's continuously monitored in flight, and any anomaly discovered during a real flight becomes a new, permanent item future pre-flight checks specifically verify, so the checklist itself improves over the fleet's operational lifetime rather than staying frozen at its original design.

6. Real-world example

A support-agent system's online faithfulness evaluator (an LLM-as-judge checking whether responses are grounded in retrieved context, Part 3.5/8.2) begins flagging a slowly increasing rate of low-faithfulness scores over two weeks — not a single dramatic failure, but a gradual trend that a one-time launch evaluation would never have caught, since it only existed as a signal after deployment, accumulating over real production traffic. Investigating traces (Part 6.1) for the flagged interactions reveals a subtle regression in the retrieval pipeline (a recent vector-store reindexing, Part 3.4, that inadvertently changed chunk boundaries) — caught and fixed within days because of continuous online monitoring, rather than persisting silently for months until enough customers complained to trigger a manual investigation.

7. Architecture diagram

See section 4's closed-loop diagram — this chapter's "architecture" is that continuous cycle, built from Parts 6.1 and 6.2's components operating together on an ongoing basis rather than as one-time setup steps.

8. Production considerations

• Set explicit alerting thresholds on online evaluator scores, not just dashboards someone has to remember to check — a genuine production monitoring practice needs to proactively surface degradation, not passively wait for someone to notice a chart trending down.
• Establish a regular (e.g., weekly or biweekly) practice of curating newly-flagged production failures into the offline regression dataset — this feedback loop (section 4) only provides value if someone actually does it consistently, not as an occasional, easily-deprioritized task.
• Track cost and latency trends alongside quality scores — a regression isn't only "the answers got worse"; it's equally a regression if cost or latency creeps up for the same quality (Part 3.11, Part 7.10's cost discipline applies to ongoing monitoring, not just point-in-time optimization).
• Sample online evaluation at a rate that balances coverage against cost (Part 6.2, section 12) — 100% coverage with an LLM-as-judge evaluator on every production trace may be prohibitively expensive at high volume; a well-chosen sampling rate (or full coverage only for cheap deterministic evaluators, with LLM-judge sampling at a lower rate) is a legitimate, common trade-off.

9. Common mistakes

• Setting up online evaluation and dashboards but never establishing actual alerting thresholds, meaning degradation is only noticed if someone happens to look at the right time.
• Treating the launch-time evaluation dataset as permanent and complete, never feeding newly-discovered production failures back into it — missing the continuous-improvement loop this chapter's core practice depends on.
• Monitoring only quality metrics and missing cost/latency regressions that are equally real production problems.
• Sampling online evaluation at 100% coverage with expensive LLM-as-judge evaluators without considering the cost implications at real production volume.

10. Security considerations

• Continuous online evaluation processes production traffic on an ongoing basis, meaning every data-sensitivity consideration from Part 6.1/6.2 (sections 10) applies not as a one-time setup concern but as an ongoing, continuous data flow — worth periodic re-review as data volume and evaluator configuration evolve, not just verified once at initial setup.
• User feedback mechanisms (thumbs up/down) that attach to traces should themselves be authenticated/scoped appropriately — an unauthenticated feedback endpoint could be abused to pollute your evaluation signal or, in principle, to probe for information about specific traces.

11. Performance considerations

• Online evaluation should run asynchronously, out of the critical path of the actual user-facing response — a production system should never make a user wait for an evaluator's score before returning its actual answer; the evaluation happens on the trace after the fact, not gating the live response (a distinct, important behavior compared to an offline evaluation gate, Part 6.2, which appropriately does block deployment).

12. Cost considerations

• Continuous online evaluation cost compounds with production traffic volume (Part 6.2, section 12) — this is an ongoing, scaling cost line item (not a one-time expense like a pre-deployment gate) and should be modeled as part of the system's steady-state operating cost (Part 7.10, Part 15), with sampling rate as the primary lever for controlling it.

13. When to use it

Any production LLM system operating at meaningful, ongoing volume and serving real users over time — the continuous-monitoring discipline this chapter describes is what separates a system that was evaluated once at launch from one that's genuinely, ongoingly managed for quality, exactly the operational maturity Part 8 (AI Reliability/GenAIOps) will formalize as a discipline in its own right.

14. When NOT to use it

A short-lived, low-volume, or genuinely experimental deployment may not justify the full continuous-monitoring infrastructure — though even a lightweight version (periodic manual review of a sample of traces, rather than fully automated online evaluation) captures much of this chapter's core value at lower setup cost, and is a reasonable middle ground for smaller-scale deployments.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Continuous online evaluation + feedback loop (this chapter)	Catches gradual/production-only degradation, continuously improves eval coverage	Ongoing cost and setup/maintenance effort
Offline evaluation only (Part 6.2)	Catches regressions before deployment	Blind to degradation that only manifests under real production conditions over time
Manual periodic review (lightweight alternative)	Low setup cost, still catches gross issues	Slower, less systematic, easy to deprioritize over time

16. Practical Python/code example

from langsmith import Client

client = Client()


async def score_production_trace(trace_id: str, evaluator) -> None:
    """
    Runs an online evaluator against a specific production trace and attaches
    the score, for continuous quality monitoring.

    Args:
        trace_id (str): The trace to evaluate.
        evaluator: An evaluator function (Part 6.2) scoring the trace's run.
    """
    run = client.read_run(trace_id)
    score_result = evaluator(run, example=None)  # online eval typically has no fixed reference
    client.create_feedback(
        run_id=trace_id, key=score_result["key"], score=score_result["score"]
    )

# Verify exact current SDK methods (read_run, create_feedback) against current
# LangSmith documentation — this is an actively maintained API surface.

17. Production-quality example

A closed-loop pipeline that curates low-scoring production traces into the regression dataset automatically, directly implementing section 4's core practice:

import logging

logger = logging.getLogger("eval_feedback_loop")

FAITHFULNESS_THRESHOLD = 0.7


async def curate_flagged_traces_into_dataset(
    client, project_name: str, dataset_name: str, lookback_hours: int = 24
) -> int:
    """
    Finds recent production traces with low online-evaluator scores and adds
    them to the regression dataset, closing the production-to-evaluation loop.

    Args:
        client: LangSmith client.
        project_name (str): The production tracing project to scan.
        dataset_name (str): The regression dataset to add flagged examples to.
        lookback_hours (int): How far back to scan for newly-flagged traces.

    Returns:
        int: Number of traces newly added to the regression dataset.
    """
    flagged_runs = client.list_runs(
        project_name=project_name,
        filter=f'and(gte(feedback_score, "faithfulness", 0), lt(feedback_score, "faithfulness", {FAITHFULNESS_THRESHOLD}))',
        # Verify exact current filter syntax against current LangSmith documentation.
    )

    added_count = 0
    for run in flagged_runs:
        client.create_example(
            inputs=run.inputs,
            outputs={"note": "flagged from production — low faithfulness score", "actual_output": run.outputs},
            dataset_name=dataset_name,
        )
        added_count += 1

    logger.info("added %d newly-flagged production traces to %s", added_count, dataset_name)
    return added_count

Running this on a schedule (e.g., a nightly job) is what makes the feedback loop from section 4 an actual, ongoing operational practice rather than a one-time manual curation effort.

18. Short exercise

Design (in plain language) an alerting rule for a production support-agent system: what specific online-evaluator score pattern (a single bad score vs. a trend) would you alert on, who would you alert, and what would you want them to be able to see immediately (linking back to Part 6.1's tracing) to start investigating?

19. Interview questions

• Why can't offline evaluation alone catch every quality regression a production system might experience?
• Describe the full feedback loop connecting production traces, online evaluation, and offline regression datasets — why does each stage depend on the others?
• Why should online evaluation run asynchronously, out of the critical path of the user-facing response?

20. FDE/customer scenario

Customer's product lead: "We passed all our tests before launch, but three months in, users say answers have gotten noticeably worse — how could that happen if we tested thoroughly?"

This is precisely the gap between one-time offline evaluation and continuous online monitoring (section 2, 6) — a system can pass every pre-launch test and still degrade over time due to factors a fixed test set couldn't anticipate (data drift in retrieved content, subtle upstream changes, gradually shifting usage patterns). Proposing continuous online evaluation with a feedback loop back into the regression dataset is the concrete, technically-grounded fix — and explaining clearly why pre-launch testing alone was never going to catch this specific class of problem is itself valuable, credibility-building communication distinct from the technical fix itself.

---

Key takeaways

• Online evaluation runs continuously against live production traces, catching degradation that a fixed offline dataset (Part 6.2) can't anticipate.
• The full value of this practice comes from closing the loop: flagged production failures feed back into the offline regression dataset, continuously improving future evaluation coverage.
• Online evaluation must run asynchronously, never blocking the user-facing response, unlike an offline deployment gate which appropriately does block.

Things you should be able to explain

• Why offline evaluation alone is insufficient for catching all production quality regressions.
• The full closed-loop practice connecting production traces, online evaluation, and offline datasets.

Things you should be able to build

• An automated pipeline that curates low-scoring production traces into a regression dataset on a schedule.

Common mistakes

• Dashboards with no actual alerting thresholds.
• Never feeding production-discovered failures back into the offline evaluation dataset.
• Monitoring quality only, missing cost/latency regressions.

Recommended next chapter

Part 6 complete. Continue to handbook/07-production-ai/01-docker.md.