8.4 — Observability, Tracing, and Monitoring for AI Systems

1. What is it?

This chapter covers the general observability discipline for AI systems — the three pillars (logs, metrics, traces) as applied specifically to LLM-based systems, and how to design AI-specific SLIs (Service Level Indicators) and SLOs (Service Level Objectives) — framework-agnostic, building on Part 6's LangSmith-specific implementation with the broader conceptual and operational practice.

2. Why does it exist?

Part 6.1 covered LangSmith's tracing specifically for LangChain/LangGraph applications. But observability as an operational discipline is broader than any one tool: it's the practice of being able to answer "is my system healthy right now, and if not, precisely why" at any moment, for a production system — whether or not it's built with LangChain, and whether or not LangSmith specifically is part of the stack. This chapter exists to ground the discipline itself, so you can apply it correctly regardless of specific tooling choices.

3. What problem does it solve?

It solves "how do I know my AI system is healthy, catch degradation before customers report it, and diagnose the root cause quickly when something does go wrong" — the same fundamental operational need every production system has, with AI-specific additions (quality metrics, not just uptime/latency; cost as a first-class operational metric, Part 7.10) that traditional software observability practices don't natively include.

4. How does it work internally?

The three pillars, applied to AI systems

• Logs: discrete, timestamped records of events — for AI systems, this includes every LLM call's parameters and outcome (Part 4.2's response_metadata), every tool call and its result (Part 3.3), and application-level events (an auth failure, a validation error). Every trace and log record should be tagged with the exact prompt version and model version/snapshot that produced it (not just "gpt-4" or "the support prompt," but the specific versioned identifier, Part 7.2's commit-SHA tagging applied to prompts/models specifically) — without this tag, a quality-score dip on a dashboard can tell you that something degraded but not which recent prompt or model rollout caused it, turning a five-minute correlation into an open-ended investigation.
• Metrics: numeric measurements aggregated over time — traditional metrics (request rate, error rate, latency percentiles) plus AI-specific metrics: token usage per request, cost per request, cache hit rate (Part 7.8), evaluation scores over time (Part 6.3's online evaluation), and agent-specific metrics like average tool calls per run or loop-iteration counts (Part 3.7's bounded-loop discipline made observable).
• Traces: the connected, hierarchical record of one request's full journey through the system (Part 6.1's trace/run model) — essential for AI systems specifically because a single user-facing request can involve many internal steps (retrieval, multiple LLM calls, tool calls) whose individual contribution to overall latency/cost/quality is invisible without trace-level granularity.

AI-specific SLIs and SLOs — beyond uptime and latency

Traditional SLOs (99.9% uptime, p99 latency under 500ms) remain relevant but insufficient alone for AI systems. AI-specific SLIs worth defining explicitly:

Quality SLIs:
  - Faithfulness score (Part 8.2) staying above a defined threshold
  - Task completion rate for agents (Part 8.1's outcome evaluation)
  - User feedback signal (thumbs down rate, Part 6.3) staying below a threshold

Latency SLIs (beyond a single aggregate "request latency p99"):
  - Time-to-first-token (TTFT) — for any streamed response, the delay between
    the request starting and the first token reaching the client; this is the
    metric that actually governs perceived responsiveness for a streaming UX,
    and can look fine on an end-to-end-latency SLI while TTFT quietly degrades
    (e.g., a slow retrieval or tool-call stage delays the first generated token
    even though total request time hasn't changed much)
  - Retrieval latency — time spent in the retrieval step alone (Part 3.4/3.6),
    tracked as its own SLI, not folded into total request latency
  - Tool-call latency — time spent per tool call (Part 3.3), tracked separately
    per tool where tools have meaningfully different latency profiles
  - Generation latency — time spent in the LLM call(s) themselves, isolated
    from retrieval and tool-call time

Cost SLIs:
  - Cost per request/conversation staying within budget (Part 7.10)
  - Token usage per request not exceeding expected bounds (catching context
    bloat, Part 3.9, or unexpectedly verbose generation)

Reliability SLIs (AI-specific reliability, beyond generic uptime):
  - LLM provider error rate (distinct from your own application's error rate,
    Part 7.5's bottleneck-diagnosis distinction applied to monitoring)
  - Agent loop iteration count staying within expected bounds (catching a
    creeping tendency toward runaway loops before they hit the hard cap, Part 3.7)

Breaking latency into named, per-stage SLIs rather than one aggregate number matters for the same reason Part 7.5's bottleneck-diagnosis discipline matters for infrastructure: an aggregate "p99 latency is fine" can hide a retrieval-stage regression that's fully offset by a faster generation stage, or vice versa — per-stage SLIs (backed by Part 6.1's trace-level granularity, which already has the timing data needed) let you see exactly which stage degraded, not just that the total did. TTFT specifically deserves its own SLI because it's a distinct UX property from total latency: a response that streams its first token in 300ms but takes 8 seconds to finish reads as responsive to a user, while one that takes 4 seconds to produce a first token but finishes in 5 reads as sluggish, even though the second has lower total latency.

Defining explicit SLOs for these (not just "we have some dashboards") is what turns observability data into an actionable operational practice — an SLO violation is a defined, actionable trigger for investigation, versus a metric that exists on a dashboard but has no threshold anyone is actually accountable for watching.

Alerting design — actionable, not noisy

Effective alerting requires distinguishing genuinely actionable signals from noise: alerting on every single evaluation score dip (which, given natural variance, will happen routinely even in a healthy system) produces alert fatigue and trains people to ignore alerts — exactly Part 5.4's human-in-the-loop-fatigue warning, applied to operational alerting instead of approval gates. Effective AI-system alerting typically triggers on sustained trends (a metric degrading over a meaningful window, not a single noisy data point) or threshold breaches with statistical significance, and routes different severity levels to different response urgency (a cost-anomaly alert might warrant next-business-day investigation; a sudden faithfulness-score collapse might warrant immediate escalation).

Closing the loop — SLO breach triggering automatic rollback, not just an alert

Routing a severe SLO breach to a human page (section 8's severity routing) is necessary but, for the most severe category of breach, not sufficient on its own — a faithfulness or task-completion collapse that started the moment a new prompt or model version rolled out (Part 7.2's prompt/model-version tagging, section 4) shouldn't have to wait for a human to notice the page, diagnose the cause, and manually revert before damage stops accumulating. The same prompt/model-version tag that lets you correlate a quality dip to a specific rollout (section 4) is what makes an automatic rollback possible: if an IMMEDIATE-severity SLO breach is detected within the monitoring window immediately following a version change (Part 7.2's canary/progressive-rollout stage is the natural place this check runs), the system automatically reverts traffic to the last-known-good prompt/model version before paging a human, rather than only paging and waiting for manual action. This directly closes the loop this Part's chapters have built toward separately: Part 7.2 tags every deployment with a precise version and gates risky changes behind canary/progressive rollout; this chapter defines the quality/cost SLOs and severity-routed alerting; automatic rollback is the missing link connecting an SLO breach during a rollout to an automatic, immediate reversion, with the human page still firing afterward for confirmation and root-cause investigation rather than as the only response.

5. Simple mental model

Observability for an AI system is like a hospital's patient monitoring setup, not just a pulse check. A pulse check (basic uptime monitoring) tells you the patient is alive, but a real monitoring setup tracks multiple distinct vital signs simultaneously (heart rate, blood oxygen, blood pressure — analogous to latency, quality scores, cost) with defined normal ranges (SLOs) and alerts staff specifically when a vital sign moves outside its expected range in a sustained, meaningful way (not on every tiny natural fluctuation) — because a patient can have a perfectly normal pulse while another vital sign is quietly deteriorating in a way a pulse check alone would completely miss.

6. Real-world example

A support-agent platform's operations team initially monitored only uptime and latency (traditional web-service SLOs) — the system stayed "up" and "fast" throughout a two-week period during which a retrieval-pipeline regression (Part 3.4's stale-index scenario) was silently degrading faithfulness scores and increasing customer complaints. Because no faithfulness or user-feedback SLO existed, this degradation was invisible to their monitoring entirely, discovered only when enough customer complaints accumulated to trigger a manual investigation — a full illustration of Part 6.3's "passed uptime/latency checks, but users noticed the quality degrade" scenario, now framed as a monitoring/SLO design gap specifically, not just a "we should have used online evaluation" gap.

7. Architecture diagram

AI System Observability

LogsLLM calls, tool calls, errors

Metricslatency, quality, cost (Part 7.10)

Tracesfull request journey through retrieval/agent/tool steps (Part 6.1)

Explicit SLIs/SLOsquality, cost, reliability

Actionable alertingsustained trend/threshold breach, routed by severity

8. Production considerations

• Define explicit SLOs for quality and cost, not just traditional uptime/latency (section 4/6) — this is the specific, actionable fix for the section 6 failure mode, turning invisible degradation into a defined, monitored, alertable condition.
• Design alerts around sustained trends, not single data points (section 4) — this is what prevents the alert-fatigue trap that would otherwise undermine the entire monitoring investment.
• Route different severities to different urgency levels — not every SLO breach warrants the same response speed, and treating them uniformly either over-escalates minor issues or under-escalates serious ones.
• Wire the most severe SLO breaches to an automatic rollback, not only a page, for any breach detected during an active canary/progressive rollout (Part 7.2's canary stage, section 4's "closing the loop") — this is what turns "we found out the new prompt was bad" into "the system already reverted before most users were affected."
• Ensure logs/metrics/traces are correlated via a shared identifier (a request ID or trace ID, Part 4.3's practical example) — being able to jump from "this metric dashboard shows an anomaly" to "here's the specific trace that explains it" is what makes the three pillars work together rather than as three disconnected data sources.

9. Common mistakes

• Monitoring only traditional uptime/latency SLOs for an AI system, missing quality and cost degradation entirely (section 6's exact failure).
• Alerting on every noisy fluctuation rather than sustained trends, producing alert fatigue that eventually causes real alerts to be ignored.
• Logs, metrics, and traces existing as three disconnected systems with no shared correlation ID, making it hard to move from "something's wrong" (a metric) to "here's exactly why" (a trace) efficiently.
• Defining SLOs once at launch and never revisiting them as the system, its usage patterns, and business requirements evolve.

10. Security considerations

• Observability data itself (logs, traces) is a data store with its own sensitivity, exactly as Part 6.1's tracing discussion covered — apply the same access-control and retention discipline to whatever observability stack you use, LangSmith or otherwise.

11. Performance considerations

• Comprehensive logging/tracing has a real, if usually small, performance overhead (Part 6.1, section 11) — verify this overhead is genuinely negligible for your specific system's scale rather than assuming it universally is.

12. Cost considerations

• Cost as a first-class, monitored SLI (section 4) is precisely what enables Part 7.10's optimization practice — without it, cost anomalies are discovered only when a bill arrives, far too late for proactive management.
• Observability infrastructure itself (log storage, metrics retention, tracing service usage, Part 6.1's pricing) is a real, distinct infrastructure cost worth budgeting explicitly.

13. When to use it

Comprehensive, AI-specific observability (quality and cost SLOs, not just traditional uptime/latency) should be standard for any production AI system — this is not optional maturity to add later; the section 6 failure mode shows how costly its absence can be, in ways that compound silently until discovered by accident or customer complaint.

14. When NOT to use it

A very early, low-stakes prototype might reasonably defer the full SLO/alerting formality — but even here, at minimum tracking basic quality/cost metrics informally (not necessarily with formal SLOs and alerting) is worth the low effort involved, per this book's repeated argument that measurement discipline scales down cheaply even when full formal rigor doesn't yet make sense.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Full AI-specific observability (quality + cost + reliability SLOs)	Catches the full range of AI-specific degradation, not just infra issues	Real setup and ongoing maintenance investment
Traditional web-service observability only	Simpler, familiar, catches infra-level issues	Blind to quality/cost degradation (section 6's exact gap)
No formal observability (manual, reactive investigation only)	Zero setup cost	Degradation discovered late, often via customer complaints

16. Practical Python/code example

An SLO-checking function that evaluates whether a quality metric has sustained a concerning trend, avoiding the noisy-single-point-alert trap from section 4:

from datetime import datetime, timedelta


def check_faithfulness_slo(
    recent_scores: list[tuple[datetime, float]], threshold: float = 0.85, window: timedelta = timedelta(hours=6)
) -> bool:
    """
    Checks whether faithfulness has been sustained below threshold over a window,
    rather than alerting on a single noisy low score.

    Args:
        recent_scores (list[tuple[datetime, float]]): (timestamp, score) pairs.
        threshold (float): Minimum acceptable faithfulness score.
        window (timedelta): How long a sustained breach must persist to trigger.

    Returns:
        bool: True if the SLO has been breached in a sustained way, warranting an alert.
    """
    cutoff = datetime.now() - window
    recent = [(ts, score) for ts, score in recent_scores if ts >= cutoff]
    if not recent:
        return False
    return all(score < threshold for _, score in recent)

17. Production-quality example

A multi-SLI alerting router that assigns severity and routing based on which SLI breached, per section 8's differentiated-urgency recommendation:

import logging
from enum import Enum

logger = logging.getLogger("slo_alerting")


class AlertSeverity(str, Enum):
    IMMEDIATE = "immediate"       # page on-call now
    NEXT_BUSINESS_DAY = "next_business_day"
    INFORMATIONAL = "informational"


SLI_SEVERITY_MAP = {
    "faithfulness_collapse": AlertSeverity.IMMEDIATE,
    "cost_anomaly": AlertSeverity.NEXT_BUSINESS_DAY,
    "cache_hit_rate_drop": AlertSeverity.INFORMATIONAL,
}

# SLIs where an IMMEDIATE breach during/just-after a rollout should trigger an
# automatic rollback (section 4's "closing the loop"), not just a page.
ROLLBACK_ELIGIBLE_SLIS = {"faithfulness_collapse", "task_completion_collapse"}


def route_slo_breach_alert(
    sli_name: str, current_value: float, threshold: float, active_rollout: dict | None = None
) -> None:
    """
    Routes an SLO breach to the appropriate response urgency, and — for an
    IMMEDIATE-severity breach tied to a version currently mid-rollout — triggers
    an automatic rollback before paging, rather than paging and waiting on a
    human to revert manually.

    Args:
        sli_name (str): Which SLI breached (must be a key in SLI_SEVERITY_MAP).
        current_value (float): The current (breaching) value.
        threshold (float): The SLO threshold that was breached.
        active_rollout (dict | None): The in-progress canary/progressive rollout
            (Part 7.2), if any, e.g. {"prompt_version": "v14", "last_known_good":
            "v13", "traffic_pct": 25}. None if no rollout is currently active.
    """
    severity = SLI_SEVERITY_MAP.get(sli_name, AlertSeverity.INFORMATIONAL)
    logger.warning(
        "SLO breach: %s current=%.3f threshold=%.3f severity=%s",
        sli_name, current_value, threshold, severity.value,
    )
    if severity == AlertSeverity.IMMEDIATE:
        if active_rollout is not None and sli_name in ROLLBACK_ELIGIBLE_SLIS:
            rollback_to_last_known_good(active_rollout)
            logger.critical(
                "AUTOMATIC ROLLBACK: %s breached SLO during rollout of %s — reverted to %s",
                sli_name, active_rollout["prompt_version"], active_rollout["last_known_good"],
            )
        page_on_call(sli_name, current_value, threshold)
    elif severity == AlertSeverity.NEXT_BUSINESS_DAY:
        create_ticket(sli_name, current_value, threshold)
    # INFORMATIONAL: logged only, surfaced on a dashboard, no active notification


def rollback_to_last_known_good(active_rollout: dict) -> None:
    """
    Shifts 100% of traffic back to the last-known-good prompt/model version,
    halting an in-progress canary rollout (Part 7.2) immediately.

    Args:
        active_rollout (dict): The in-progress rollout state, including
            "last_known_good" — the version identifier to revert to.
    """
    set_traffic_split(version=active_rollout["last_known_good"], traffic_pct=100)

18. Short exercise

A team has uptime and latency SLOs but no quality or cost SLOs. Using this chapter's framework, propose two specific quality SLIs and one cost SLI they should add, including a reasonable threshold and alerting window for each, and justify why each specific choice matters for their (assumed) customer support agent use case.

19. Interview questions

• Why are traditional uptime/latency SLOs insufficient on their own for AI systems, and what categories of SLI need to be added?
• Explain why alerting on sustained trends rather than single data points matters for AI-specific metrics specifically, given their natural variance.
• Why does correlating logs, metrics, and traces via a shared identifier matter operationally, not just architecturally?

20. FDE/customer scenario

Customer's operations lead: "Our AI system has been 'green' on all our dashboards, but we just found out quality has been degrading for weeks based on customer complaints — how did we miss this?"

This is precisely section 6's failure mode, and the concrete, credible fix is proposing explicit quality and cost SLOs (faithfulness score thresholds, user-feedback-rate thresholds, cost-per-request bounds) alongside their existing uptime/latency monitoring — with alerting designed around sustained trends rather than noisy single points, so the next quality regression is caught by the monitoring system itself, rather than discovered after enough customers have already been affected to trigger a manual investigation.

---

Key takeaways

• AI systems need quality and cost SLIs/SLOs in addition to traditional uptime/latency ones — a system can be fully "green" on infrastructure health while quality silently degrades.
• Alert on sustained trends, not single noisy data points, to avoid the alert-fatigue trap that undermines monitoring's actionability.
• Logs, metrics, and traces should be correlated via a shared identifier so an anomaly on a dashboard can be traced to its specific root cause quickly.

Things you should be able to explain

• Why traditional web-service SLOs are insufficient for AI systems specifically.
• The distinction between alerting on a single data point versus a sustained trend, and why it matters.

Things you should be able to build

• A sustained-trend SLO checker and a severity-routed alerting system for multiple distinct SLIs.

Common mistakes

• Monitoring only uptime/latency, missing quality/cost degradation entirely.
• Alerting on every noisy fluctuation, causing alert fatigue.
• No shared correlation ID linking logs, metrics, and traces.

Recommended next chapter

05-guardrails-red-teaming-hitl.md