18.11 — Observability for AI Infrastructure

Relationship to Part 6/8.4: Part 6 taught LangSmith's trace/run model; Part 8.4 taught AI-specific observability (faithfulness SLOs, trace-based debugging). This chapter is the layer underneath both: the traditional infrastructure observability (logs, metrics, traces, SLIs/ SLOs) that CloudWatch (18.5) and an APM/tracing stack provide, and — critically — how it's genuinely different from, and complementary to, AI-specific observability, not a duplicate of it.

1. What is it?

The practice and tooling for knowing what a running system is actually doing: logs (discrete, timestamped records of events), metrics (numeric measurements over time — request count, latency, error rate), and traces (the path of one request through multiple services/ components) — plus the discipline of defining SLIs/SLOs/SLAs to turn raw signals into an actual, actionable definition of "healthy."

2. Why does it exist?

18.1 taught you to inspect a single host by hand; 18.9/18.10 taught you to inspect a single pod. Neither scales to "is our production AI system, running across dozens of pods and several AWS services, healthy right now, and if not, where exactly is the problem" — that question requires aggregated, centralized, correlated observability, not one-host-at-a-time inspection.

3. What problem does it solve?

It solves two genuinely distinct problems that are easy to conflate: (1) is the infrastructure healthy (are pods up, is latency acceptable, is the database reachable) and (2) is the AI system actually good (are responses faithful, is the agent completing tasks correctly, Part 8.1/8.2). A system can score perfectly on (1) while failing badly at (2) — every pod healthy, every API call returning 200, while the LLM is confidently hallucinating — which is exactly why this chapter treats them as two complementary layers, not one.

4. How does it work internally?

Logs, metrics, and traces — the three pillars, concretely

Logs are discrete events — "request X completed in 340ms," "failed to connect to Redis" — best for answering "what exactly happened at this moment." Metrics are numeric time series — p50/p95/p99 latency, requests-per-second, error rate — best for answering "how is the system behaving over time, and has it changed." Traces (Part 6.1's LangSmith trace model, and the general distributed-tracing concept it's an instance of) follow one request across every component it touches — best for answering "where, specifically, in a multi-hop request did the time go or the failure occur." A production incident typically uses all three in sequence: a metric alert (error rate spiked) tells you that something is wrong, a trace shows you where in the request path it went wrong, and logs at that specific point show you why.

SLI, SLO, and SLA — turning signals into a definition of "healthy"

A Service Level Indicator (SLI) is a specific, measured metric (e.g., "p95 API latency," "percentage of requests returning 5xx"). A Service Level Objective (SLO) is a target for that SLI over a time window (e.g., "p95 latency under 3 seconds, 99.5% of the time, over a rolling 30 days"). A Service Level Agreement (SLA) is an SLO turned into an external, often contractual, commitment to a customer, typically with a consequence for missing it. The relationship matters practically: SLOs should be set tighter than any SLA you commit to externally, so you have room to detect and respond to degradation internally before it becomes a customer-visible SLA breach — a real, deliberate margin, not an accidental gap.

Error budgets follow directly from an SLO: if your SLO allows 0.5% of requests to fail over 30 days, that 0.5% is your "budget" — spent faster than expected, it's a signal to slow down risky changes (a new feature rollout, an aggressive prompt change) until the budget recovers, directly connecting to 18.8's canary-and-rollback discipline as the practical mechanism for not overspending it.

Traditional observability vs. LLM/AI observability — stated precisely

TRADITIONAL (infrastructure) observability answers:
  - Is the process up? (18.1's process/health checks)
  - What's the request latency, error rate, throughput?
  - Is the database/cache reachable and performant?
  - Are pods healthy, is the cluster scheduling correctly? (18.9/18.10)

LLM/AI observability (Part 6/8.1/8.2/8.4) answers:
  - Is the RESPONSE actually correct/faithful/relevant?
  - Did the agent choose the right tools, in a sensible sequence?
  - Is hallucination rate trending up?
  - Did retrieval actually surface the relevant documents?

A system can score perfectly on the first list and terribly on the
second — HTTP 200, 200ms latency, confidently wrong answer. Neither
layer substitutes for the other; both are required, together.

For the Enterprise AI Assistant specifically: CloudWatch (18.5) and whatever APM/tracing stack you run answer the first list; LangSmith (Part 6) answers the second — and a mature production setup correlates them, so a CloudWatch-reported latency spike and a LangSmith trace for the same request can be viewed together, turning "latency is high" into "latency is high because the retrieval step specifically is slow for this request," a materially more actionable finding.

OpenTelemetry and GenAI semantic conventions

OpenTelemetry (OTel) is a vendor-neutral standard and set of libraries for producing logs, metrics, and traces in a consistent format, exportable to many different backends (CloudWatch, a self-hosted collector, a commercial APM tool) without rewriting instrumentation for each. OTel has an evolving set of semantic conventions for generative AI — standard attribute names for LLM-specific span data (model name, token counts, temperature, and similar call parameters) — the emerging, vendor-neutral answer to "how should an LLM call look in a trace," distinct from (and potentially complementary to) a framework-specific tracing model like LangSmith's own. Because this area is actively evolving, verify the current OTel GenAI semantic-convention specification directly before building instrumentation against a specific attribute name, rather than assuming an interface observed at one point in time is stable.

Alerting and dashboards

An alert fires when an SLI crosses a threshold for a sustained period — "sustained" is the operative word: alerting on a single noisy data point (Part 8.4's exact pitfall, revisited here at the infrastructure level) produces false pages and trains an on-call rotation to ignore alerts, a serious, self-defeating outcome. A dashboard presents the current and historical state of key SLIs for human review — the daily- use complement to alerting's exception-based model, and usually the first thing opened at the start of any incident investigation.

Log aggregation and cost-aware sampling

At real production volume, centralizing logs (rather than 18.1's per-host journalctl) into a searchable, aggregated system (CloudWatch Logs, or a dedicated log platform) is what makes cross-service log correlation practical at all. Full-fidelity logging and tracing at very high request volume has a real, direct cost (18.18) — sampling (recording only a percentage of traces/logs in detail, while still recording aggregate metrics for everything) is the standard mitigation, with the important caveat that a sampling strategy should preserve 100% of error traces even while sampling down successful ones, since the traces you need most for debugging are disproportionately the failures.

5. Simple mental model

If 18.1's tools were a doctor examining one patient by hand, this chapter's tooling is a hospital's full monitoring system — vital-sign monitors on every patient (metrics), a shared patient chart (logs), and a way to trace one patient's path through every department they visited (traces) — while LangSmith (Part 6) is the specialist reviewing whether the actual diagnosis and treatment plan (the AI's response) was medically sound, a genuinely different, complementary kind of review.

6. Real-world example

A CloudWatch alarm fires: p95 latency for the Enterprise AI Assistant's /chat endpoint has exceeded its SLO for 10 sustained minutes. The on-call engineer opens the correlated trace for a slow request (via a shared request ID propagated from the infrastructure trace into the LangSmith trace, Part 6.1) and sees the time is concentrated in a retrieval step, not the LLM call itself. kubectl top pod (18.10) shows normal CPU/memory — ruling out resource pressure. The vector database's own CloudWatch metrics (18.4/18.14) show elevated query latency, correlating with a recent, much larger-than-usual batch of document uploads (18.15) still being indexed. The fix (temporarily throttling ingestion, or scaling the vector database) follows directly from having both observability layers available and correlated, rather than either one alone.

7. Architecture diagram

Requests

Infrastructure layerCloudWatch: metrics, logs, alarms (ALB, ECS/EKS, RDS, ElastiCache, SQS)

AI layerLangSmith: LLM/agent/retrieval traces (Part 6) — is the RESPONSE good?

Correlated dashboard / incident viewboth layers, same request, side by side

8. Production considerations

• Propagate a single request/correlation ID from the infrastructure layer (an ALB or application-generated request ID) into the AI-tracing layer (LangSmith, Part 6.1) so the two can be correlated for one request, not investigated as two disconnected systems.
• Set SLOs with real margin below any external SLA, and treat error-budget burn as a concrete signal for pacing risky changes (18.8's canary discipline).
• Sample logs/traces at high volume, but never sample away error traces — preserve 100% fidelity for failures specifically.

9. Common mistakes

• Treating "all our infrastructure metrics are green" as evidence the AI system is working well — the single most consequential mistake this chapter addresses directly.
• Alerting on a single noisy data point instead of a sustained breach, producing alert fatigue (Part 8.4's exact pitfall, at the infra layer).
• No correlation between infrastructure traces and AI-specific traces, turning every cross-layer incident into two separate, disconnected investigations instead of one.
• Sampling away error traces along with successful ones "to save cost," losing exactly the data most needed during an incident.

10. Security considerations

Logs and traces routinely contain sensitive data (a user's actual question, a document's extracted content, Part 9.3's data-exposure concerns) — observability infrastructure itself needs access controls and retention policies, not an assumption that "it's just logs" makes it exempt from the same data-handling discipline as the application itself.

11. Performance considerations

Heavy, unsampled tracing/logging at high request volume adds real overhead (serialization, network calls to a collector) to the request path itself — sampling (section 4) is as much a performance consideration as a cost one.

12. Cost considerations

Log/trace/metric volume is a direct, scaling cost (18.18) — CloudWatch Logs ingestion and storage, and any third-party observability platform's per-GB or per-seat pricing, both grow with traffic; sampling and retention-period tuning are the two primary, concrete levers.

13. When to use it

Any production AI system needs both layers (infrastructure and AI- specific observability) from day one of production — retrofitting observability after an incident you couldn't diagnose is a common, avoidable position to be in.

14. When NOT to over-apply it

A local prototype (Part 13.2) doesn't need SLOs or a full tracing pipeline — basic logging is sufficient until the system is handling real production traffic with real availability expectations.

15. Alternatives and trade-offs

CloudWatch-native tooling is the lowest-friction default on AWS; a dedicated observability platform (built on OpenTelemetry for vendor neutrality) offers richer cross-service correlation and often better tracing UX, at additional cost and integration effort — a real trade-off between "good enough, already available" and "materially better, additional investment," not a universal answer either way.

16. Practical example — propagating a correlation ID

"""
Generate (or extract) a correlation ID at the API boundary, propagate it
into both structured logs AND the LangGraph invocation's metadata, so an
infrastructure trace and a LangSmith trace can be joined on the same ID.
"""
import uuid
from fastapi import Request

async def chat_endpoint(request: Request, body: ChatRequest):
    correlation_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
    logger.info("chat_request_received", extra={"correlation_id": correlation_id})

    result = await graph.ainvoke(
        {"messages": [body.message]},
        config={
            "configurable": {"thread_id": body.session_id},
            "metadata": {"correlation_id": correlation_id},  # surfaces in
                                                                # the LangSmith
                                                                # trace, Part 6.1
        },
    )
    return {"response": result, "correlation_id": correlation_id}

17. Production-quality example — an SLO definition with error-budget tracking

"""
A concrete SLO definition and error-budget calculation for the Enterprise
AI Assistant's /chat endpoint — the kind of artifact an SRE/platform
team, or an FDE communicating with one, actually maintains.
"""
from dataclasses import dataclass
from datetime import timedelta

@dataclass
class SLO:
    name: str
    sli_description: str
    target_percentage: float   # e.g. 99.5
    window: timedelta

CHAT_LATENCY_SLO = SLO(
    name="chat-p95-latency",
    sli_description="Percentage of /chat requests with p95 latency < 3s",
    target_percentage=99.5,
    window=timedelta(days=30),
)

def error_budget_remaining(slo: SLO, total_requests: int, breaching_requests: int) -> float:
    """
    Returns the fraction of the error budget remaining (0.0 to 1.0).
    A negative value means the SLO has already been breached for the window.
    """
    allowed_breach_fraction = 1 - (slo.target_percentage / 100)
    allowed_breaches = total_requests * allowed_breach_fraction
    if allowed_breaches == 0:
        return 0.0 if breaching_requests > 0 else 1.0
    return 1 - (breaching_requests / allowed_breaches)

Note the SLO is defined with an explicit target below 100% and an explicit window — an SLO of "100%, always" is not a real, achievable target for any distributed system and sets up a permanently "failing" metric that stops being useful as a signal (18.8's canary-pacing logic depends on the error budget being a meaningful, sometimes-positive number).

18. Short exercise

Define an SLI/SLO for the Enterprise AI Assistant's document-ingestion pipeline (18.4/18.15) — not the chat endpoint — and explain specifically why the right SLI here (e.g., "percentage of documents successfully ingested within N minutes of upload") is different in shape from the chat endpoint's latency-based SLI, given the asynchronous, queue-based nature of that pipeline.

19. Interview questions

• What's the difference between an SLI, an SLO, and an SLA?
• Why can a system be fully healthy by infrastructure metrics while producing bad AI output, and what does that imply about observability design?
• Why should you never sample away error traces even under a general sampling strategy?
• What does an error budget give you operationally that a simple pass/fail SLO check doesn't?

20. FDE/customer scenario

A customer asks: "How will we know if the AI is giving bad answers, separate from just knowing the servers are up?" A strong answer names this chapter's exact distinction explicitly — infrastructure observability (CloudWatch: is it up, is it fast) answers one question, and AI-specific observability (LangSmith-based faithfulness/quality monitoring, Part 8) answers a genuinely different one — and proposes correlating both via a shared request ID so an incident review sees "the system was technically healthy AND producing bad answers" as one diagnosable picture, not two disconnected data sources the customer has to reconcile themselves.

---

Key takeaways

• Infrastructure observability (is it up, is it fast) and AI observability (is the output actually good) are complementary, non-substitutable layers — a system can pass one completely while failing the other badly.
• SLOs should have real margin below any external SLA, and error budgets turn "are we healthy" into an actionable pacing signal for risky changes.
• Sampling is necessary at scale for cost, but must never sample away error traces — those are the ones most needed during an incident.

Things you should be able to explain

• The difference between logs, metrics, and traces, and when each answers the question you actually have.
• SLI vs. SLO vs. SLA, and why SLOs are set tighter than SLAs deliberately.
• Why infrastructure health and AI-response quality are genuinely different, both-required observability layers.

Things you should be able to build

• A correlation-ID propagation scheme joining infrastructure and AI-specific traces for the same request.
• A concrete SLO definition with an error-budget calculation.

Common mistakes

• Treating green infrastructure dashboards as proof the AI is working well.
• Alerting on single noisy data points instead of sustained breaches.
• Sampling away error traces along with successful ones.

Recommended next chapter

12-reliability-engineering.md