18.21 — Production Incidents Playbook

This chapter is a reference, not a tutorial — each incident uses the same structure (Symptoms → Possible Causes → Investigation → Commands/ Tools → Diagnosis → Immediate Mitigation → Permanent Fix → Prevention) and points back to the chapter that taught the underlying mechanism rather than re-deriving it. Read it once end to end, then use it as a lookup table during a real incident.

1. What is it?

Ten of the most common production incidents an AI FDE actually encounters operating a system like the Enterprise AI Assistant, each diagnosed with the same disciplined structure this track has used throughout (18.1, 18.10, 18.12).

2. Why does it exist?

Every individual chapter in this track taught mechanisms in isolation. Real incidents rarely announce which chapter they belong to — "latency is high" could be 18.1 (CPU throttling), 18.4 (database connections), 18.13 (the wrong bottleneck), or 18.16 (GPU saturation, if self-hosted). This chapter exists as the fast, cross-referenced lookup an on-call AI FDE actually needs at 2 AM.

3. What problem does it solve?

It solves "something is wrong in production, right now, and I need a systematic starting point" — for the ten failure shapes that recur most often across real AI systems.

4. How does it work internally?

Incident 1: API latency suddenly increases

SYMPTOMS: p95/p99 latency (18.11's SLI) climbs; users report slowness.
POSSIBLE CAUSES: CPU throttling (18.1/18.9); database connection
  exhaustion (18.13); LLM provider slowness (outside your control);
  vector DB query degradation; a noisy-neighbor workload sharing a
  resource (18.12's bulkhead gap).
INVESTIGATION / COMMANDS:
  kubectl top pods (18.10) — CPU/memory saturation?
  Check RDS connection count (18.4/18.13) — near max_connections?
  Check LangSmith traces (Part 6.1) — which SPECIFIC step is slow?
DIAGNOSIS: apply 18.13's bottleneck framework — do NOT assume it's the
  LLM call without checking the trace breakdown first.
IMMEDIATE MITIGATION: scale the actually-bottlenecked resource (18.13);
  if it's the LLM provider itself, verify the circuit breaker/fallback
  (18.12) is engaging correctly.
PERMANENT FIX: address the specific root cause found above.
PREVENTION: SLO-based alerting (18.11) on sustained (not single-sample)
  latency breach, with the trace-based root-cause step built into the
  runbook, not discovered fresh each time.

Incident 2: Kubernetes pods keep restarting

SYMPTOMS: kubectl get pods shows climbing RESTARTS / CrashLoopBackOff.
POSSIBLE CAUSES: OOMKilled (18.1/18.9's cgroup-OOM); liveness probe
  misconfigured for a slow-starting service (18.9/18.10); a genuine
  application crash; a missing/incorrect environment variable or secret.
INVESTIGATION / COMMANDS: kubectl describe pod (Events first, 18.10);
  kubectl logs --previous (18.10).
DIAGNOSIS: use 18.10's exact CrashLoopBackOff decision tree — distinguish
  OOMKilled, probe misconfiguration, and genuine crash from the evidence.
IMMEDIATE MITIGATION: roll back to the last known-good version (18.8) if
  this followed a recent deploy.
PERMANENT FIX: raise the memory limit (with real usage data, 18.1) for
  OOMKilled; add a startupProbe (18.9) for a slow-starting service;
  fix the underlying bug for a genuine crash.
PREVENTION: canary deployment (18.8) catches this on 5% of traffic
  before a full rollout.

Incident 3: LLM requests are timing out

SYMPTOMS: requests to the LLM provider exceed their configured timeout
  (18.12); users see errors or unusually long waits.
POSSIBLE CAUSES: genuine provider-side slowness/outage; a timeout set
  too aggressively relative to the model's normal generation time for
  the request's context/output length (Part 7.11, 18.16's token-
  throughput point); network path issue (NAT Gateway, 18.2/18.5) between
  your VPC and the provider.
INVESTIGATION / COMMANDS: check the provider's public status page;
  check LangSmith traces for the actual time-to-first-token vs. total
  generation time (Part 6.1); check NAT Gateway metrics (18.5/18.11).
DIAGNOSIS: distinguish "genuinely slow/down provider" from "our own
  timeout is miscalibrated for this request shape" from "a network path
  problem on our side."
IMMEDIATE MITIGATION: circuit breaker + fallback provider (18.12)
  should already be engaging — verify it is.
PERMANENT FIX: recalibrate timeout to the request shape (streaming,
  18.11's tracing helps identify realistic percentiles) or fix the
  network path.
PREVENTION: 18.12's full resilience stack (timeout, retry+backoff,
  circuit breaker, fallback) applied correctly from the start.

Incident 4: Redis goes down

SYMPTOMS: Redis-dependent operations fail or degrade.
POSSIBLE CAUSES: failover in progress (18.14, should be brief); out of
  memory with a `noeviction` policy misconfigured for a pure-cache use
  case (18.14's exact flagged mistake); network partition (18.2).
INVESTIGATION / COMMANDS: redis-cli INFO (18.14); check ElastiCache
  events/CloudWatch (18.4/18.11) for a failover event.
DIAGNOSIS: is the APPLICATION actually breaking (a violation of 18.14's
  cache-is-not-a-system-of-record principle) or just running slower
  (the correct, expected degraded behavior)?
IMMEDIATE MITIGATION: if the application is hard-failing on Redis
  unavailability, this is itself the incident to fix — not merely
  "wait for Redis to recover."
PERMANENT FIX: ensure every Redis-dependent code path treats a miss/
  error as a fallback path (recompute, call the LLM fresh), never a
  hard failure (18.14).
PREVENTION: Multi-AZ replication (18.14) for faster failover; explicit
  chaos/failure-injection testing (18.7's cross-cutting gap) verifying
  the application actually degrades gracefully, not just assuming it does.

Incident 5: Database reaches maximum connections

SYMPTOMS: new requests fail with a connection error; existing requests
  may succeed normally.
POSSIBLE CAUSES: no connection pooling, or a pool sized too large across
  too many replicas (18.13's exact flagged mistake); a connection leak
  (code not releasing connections back to the pool); a genuine traffic
  increase beyond provisioned capacity.
INVESTIGATION / COMMANDS: SELECT count(*) FROM pg_stat_activity (18.4/
  18.13); check for long-idle connections (a leak signature) vs. many
  short-lived active ones (genuine load).
DIAGNOSIS: pooling misconfiguration (fixable immediately) vs. a leak
  (needs a code fix) vs. genuine capacity need (needs a real scale-up).
IMMEDIATE MITIGATION: restart the leaking service to release stuck
  connections (a temporary relief, not a fix) if a leak is confirmed;
  reduce per-instance pool size if over-provisioned across replicas.
PERMANENT FIX: fix the leak in code; recompute correct per-instance
  pool sizing (18.13's exact formula) as replica count changes; consider
  RDS Proxy for connection multiplexing at real scale.
PREVENTION: connection pool metrics as a standing dashboard (18.11), not
  discovered only during an outage.

Incident 6: AWS service becomes unavailable

SYMPTOMS: a specific AWS service (not your own application) experiences
  a regional outage.
POSSIBLE CAUSES: a genuine AWS-side incident (check the AWS Health
  Dashboard/Service Health status directly — don't assume, verify).
INVESTIGATION / COMMANDS: AWS Health Dashboard; CloudWatch alarms for
  the specific affected service (18.11); check whether the outage is
  region-wide or service-specific.
DIAGNOSIS: which of YOUR components depend on the affected service, and
  do they have the redundancy (Multi-AZ, 18.4/18.5) or fallback (18.12)
  this track has built in?
IMMEDIATE MITIGATION: failover to a healthy AZ/replica if the outage is
  AZ-scoped (18.4/18.5's Multi-AZ design existing exactly for this); a
  region-wide outage requires a genuine DR plan (18.12's RTO/RPO) if one
  exists and this scenario was actually planned for.
PERMANENT FIX: none — this is AWS's incident to resolve; your
  "permanent fix" is ensuring your OWN redundancy design actually covers
  this class of event for next time.
PREVENTION: multi-AZ (18.5) for AZ-level events; a genuinely tested DR
  plan (18.12) for region-level events, sized to how much a given
  system's actual availability requirement justifies the added
  complexity and cost of that level of protection.

Incident 7: LLM provider rate limit is reached

SYMPTOMS: 429 responses from the LLM provider; requests fail or fall
  back (18.12) more often than expected.
POSSIBLE CAUSES: genuine traffic growth beyond provisioned quota; a
  retry storm (18.12's exact flagged risk) amplifying request volume
  during a transient issue; an inefficient calling pattern (e.g., one
  logical operation making many small calls instead of one batched call).
INVESTIGATION / COMMANDS: check provider rate-limit response headers/
  dashboard; check your own request-volume metrics (18.11) for a genuine
  spike vs. a retry-storm-driven artificial one.
DIAGNOSIS: distinguish "we need a higher quota" from "our own retry
  behavior is amplifying the problem" (18.12's backoff+jitter should
  prevent this if correctly implemented).
IMMEDIATE MITIGATION: model routing to a less-constrained model/provider
  (Part 7.9/7.11); ensure backoff+jitter (18.12) is actually functioning,
  not contributing to the spike.
PERMANENT FIX: request a quota increase from the provider if genuine
  growth; fix retry logic if it's amplifying the problem; consider
  request batching (18.18's batch-API point) for non-latency-sensitive
  volume.
PREVENTION: rate-limit-aware model routing and pre-emptive backpressure
  (18.12/18.13) before hitting the provider's own limit, not only
  reactive handling after a 429.

Incident 8: Cloud bill suddenly doubles

SYMPTOMS: a large, unexpected month-over-month cost increase.
POSSIBLE CAUSES: see 18.18's full investigation order — NAT Gateway
  volume, an inefficient external-API calling pattern, over-provisioned
  or newly-added compute, an observability-volume spike, a genuine
  traffic increase.
INVESTIGATION / COMMANDS: AWS Cost Explorer broken down by service
  (18.18) — do NOT guess; check the actual breakdown first.
DIAGNOSIS: 18.18's exact systematic order — LLM API, compute
  utilization, NAT/data transfer, managed-service sizing, observability
  volume, in that sequence.
IMMEDIATE MITIGATION: address the specific largest driver found — often
  NOT the intuitively-assumed one (18.18's explicit warning).
PERMANENT FIX: the specific fix depends on the diagnosed driver (caching
  improvement, right-sizing, batching an inefficient calling pattern).
PREVENTION: resource tagging (18.6/18.18) enabling routine cost-
  breakdown review as a standing practice, not only triggered by a shock.

Incident 9: Agent starts making incorrect tool calls

SYMPTOMS: an agent (Part 3.7/5) calls the wrong tool, with wrong
  arguments, or in a nonsensical sequence.
POSSIBLE CAUSES: this is PRIMARILY an AI-quality incident (Part 8.1/8.2),
  not an infrastructure one — but check infrastructure causes FIRST to
  rule them out cheaply: a recent, unreviewed prompt change (18.8's eval
  gate should have caught a regression — check whether it actually ran
  and passed); a tool's underlying API changed/degraded (a dependency
  issue, not an AI-reasoning issue at all); a model version change
  (Part 2.6/18.8's pinning point) if the provider defaults to "latest."
INVESTIGATION / COMMANDS: check LangSmith traces (Part 6.1) for the
  exact reasoning that led to the wrong call; check recent deploys
  (18.8) for a prompt/model change correlated with the onset; check the
  tool's own API/service health independently.
DIAGNOSIS: infrastructure-adjacent cause (unpinned model version, a
  tool's degraded API, an eval gate that didn't actually run) vs. a
  genuine AI-reasoning/prompt-quality issue requiring Part 8's evaluation
  and prompt-engineering tools, not an infrastructure fix.
IMMEDIATE MITIGATION: roll back the correlated deploy (18.8) if one
  exists; if a tool's own API is degraded, apply 18.12's circuit-
  breaker/fallback for that tool specifically.
PERMANENT FIX: pin the model version explicitly (18.8); strengthen the
  eval gate's coverage for this failure mode (Part 8.1/8.3); fix the
  underlying prompt/tool-definition issue (Part 3.1/3.3) if it's genuinely
  an AI-reasoning problem, not an infrastructure one.
PREVENTION: this incident is the clearest illustration in this whole
  chapter of WHY 18.11 insists infrastructure and AI observability are
  separate, both-required layers — infrastructure metrics alone would
  show this incident as invisible.

Incident 10: RAG latency becomes extremely high

SYMPTOMS: retrieval-augmented requests specifically (not all requests)
  become slow.
POSSIBLE CAUSES: vector database query degradation (index bloat from
  significant new ingestion volume without corresponding scaling,
  18.4/18.15); an embedding-model call itself slow (Part 3.4, or 18.16 if
  self-hosted); a reranking step (Part 3.6) adding unexpected latency
  under load; the ingestion pipeline (18.15) competing for the same
  vector database resources as the query path (18.13's shared-resource
  bottleneck).
INVESTIGATION / COMMANDS: LangSmith traces (Part 6.1) isolating time
  spent specifically in the retrieval step vs. the generation step;
  vector database's own query-latency metrics; check for a concurrent,
  large ingestion batch (18.15) running against the SAME vector database
  instance the query path uses.
DIAGNOSIS: is retrieval slow because of query-path load, or because a
  large concurrent ingestion job is competing for the same underlying
  resource (18.13's bottleneck-identification, applied specifically
  across the sync/async boundary 18.17's diagram drew)?
IMMEDIATE MITIGATION: throttle or pause the competing ingestion batch
  (18.15) if that's the cause; scale the vector database if it's genuine
  query-path load.
PERMANENT FIX: resource isolation (a bulkhead, 18.12) between the
  ingestion and query paths if they share infrastructure and this
  recurs; right-size vector database capacity to actual corpus growth.
PREVENTION: monitor vector database latency as its own explicit SLI
  (18.11), and treat ingestion-volume spikes as a known risk to query-
  path latency, not a surprising, disconnected coincidence.

5. Simple mental model

This playbook is a differential-diagnosis reference for a doctor — not a substitute for actually examining the patient (running the specific commands), but a fast way to recognize "this symptom cluster usually means one of these three things" so the actual investigation starts in the right place instead of everywhere at once.

6. Real-world example

Incident 10, played out: a customer reports the Enterprise AI Assistant "got slow around 2 PM." LangSmith traces confirm the slowdown is isolated to the retrieval step. Checking for a concurrent event, the team finds a scheduled bulk re-embedding job (18.9's CronJob, 18.15) coincidentally started at 2 PM, saturating the shared vector database's query capacity — diagnosed correctly within minutes specifically because the investigation started from "is this isolated to retrieval" (ruling out half the playbook's other candidate causes immediately) rather than a broad, undirected search.

7. Architecture diagram

Symptom noticed

Which of the 10 incident SHAPES does this match?

That incident's own Investigation/Commandssection 4

Diagnosis → Immediate mitigation → Permanent fix → Prevention

Update THIS playbookif a genuinely new failure shape was found

8. Production considerations

Keep this playbook (or your own organization's equivalent) actually current — a runbook that hasn't been updated since the last architecture change is a false sense of preparedness, worse in some ways than no runbook at all if it actively misdirects an on-call engineer.

9. Common mistakes

• Treating incident 9 (agent tool-call errors) as an infrastructure problem and searching CloudWatch/kubectl output for something that will never appear there — the single most instructive cross-reference in this entire chapter (18.11's whole thesis, concretely realized).
• Investigating a cost spike (incident 8) by intuition instead of 18.18's actual breakdown-first order.
• Assuming any latency incident (1 or 10) is the LLM provider's fault before checking the trace breakdown (18.11/18.13).

10. Security considerations

Any incident involving a credential, secret, or unexpected access pattern should also trigger Part 9/18.19's security-incident process alongside this chapter's operational one — a security incident dressed as a performance incident (unusual traffic possibly indicating abuse, not organic growth) is a real, if less common, possibility worth explicitly ruling out for incidents 6-8 especially.

11. Performance considerations

This entire chapter is a performance/reliability reference — its value is specifically in reducing time-to-diagnosis during a real incident, which is itself the most important "performance" metric for an incident- response process (commonly tracked as MTTR — mean time to resolution).

12. Cost considerations

Incident 8 is this chapter's direct cost-incident entry; several others (7, 9) have real cost implications if left undiagnosed (rate-limit workarounds, or a bad agent looping on incorrect, possibly expensive tool calls).

13. When to use it

As a first-response reference during any real production incident matching one of these ten shapes, and as a study reference for interview preparation (18.22).

14. When NOT to over-apply it

Don't force a genuinely novel incident into one of these ten shapes if it doesn't actually fit — use the underlying diagnostic disciplines (18.1's process, 18.10's Kubernetes methodology, 18.13's bottleneck framework) to investigate a new failure shape from first principles, and add it to this playbook afterward.

15. Alternatives and trade-offs

A more exhaustive, formally-maintained runbook system (PagerDuty runbooks, an internal wiki with per-service playbooks) is the mature, larger-team version of what this chapter models at handbook scale — the underlying discipline (structured, evidence-based diagnosis) is identical regardless of where it's actually stored and maintained.

16. Practical example — a personal incident-response checklist

# First 5 Minutes of Any Incident
- [ ] What's the ACTUAL symptom, precisely? (not "it's broken" — WHAT,
      specifically, is failing or slow?)
- [ ] Which of this chapter's 10 shapes does it most resemble?
- [ ] Check that incident's specific Investigation/Commands FIRST
- [ ] Is this infrastructure (18.1-18.19) or AI-quality (Part 8)? —
      incident 9 is the reminder these are genuinely different
- [ ] Communicate status (Part 12.3's structure) BEFORE the full
      diagnosis is complete — "investigating, here's what we know so far"

17. Production-quality example — an incident postmortem template

# Postmortem: [Incident Title] — [Date]

## Summary
[One paragraph: what happened, user impact, duration]

## Timeline
[Detection → Investigation → Mitigation → Resolution, with timestamps]

## Root cause
[The actual, verified cause — not a guess; reference which of this
chapter's 10 shapes it matched, or note it as a genuinely new one]

## Immediate mitigation applied
## Permanent fix
## Prevention (what changes so this can't recur, or is caught faster)

## What went well / what didn't
[Honest, blameless — Part 13.2's honesty principle applied to
incident review specifically]

18. Short exercise

Pick any three of this chapter's ten incidents and, without looking at section 4, write out from memory their Possible Causes and Investigation steps — then check your answer against the actual text, noting specifically which cross-references (to 18.1/18.4/18.9/18.12/18.13/18.14 etc.) you missed.

19. Interview questions

• Walk through your investigation process for [any one of the 10 incidents] from first symptom to root cause.
• Why is incident 9 (agent tool-call errors) fundamentally different in diagnosis approach from the other nine?
• How would you distinguish incident 5 (max connections) from a connection LEAK versus genuine capacity need, from the evidence alone?
• What's the value of a structured incident-response playbook versus ad-hoc, from-scratch investigation each time?

20. FDE/customer scenario

A customer's on-call engineer pages you at 2 AM: "Something's wrong with the AI system, users are complaining." With no more information than that, walk through how you'd narrow down which of this chapter's ten incident shapes you're actually facing — the first two or three questions you'd ask, and the first one or two commands you'd run, before committing to a specific diagnosis path.

---

Key takeaways

• Most real AI-system production incidents fall into a small number of recurring shapes — recognizing which one quickly is most of the value of a structured incident-response process.
• Incident 9 (agent tool-call errors) is a genuinely different kind of incident than the other nine — infrastructure metrics will show nothing wrong, directly validating 18.11's infrastructure-vs-AI- observability distinction in a real, concrete scenario.
• A cost spike (incident 8) should always be investigated via an actual service-level breakdown (18.18), never intuition about which cost driver feels most likely.

Things you should be able to explain

• The distinguishing evidence for each of this chapter's ten incidents.
• Why an agent-reasoning incident requires AI-specific observability (Part 6/8), not infrastructure metrics, to diagnose.

Things you should be able to build

• A personal or team incident-response checklist and postmortem template.
• A rapid, evidence-based triage process for an unfamiliar new symptom.

Common mistakes

• Diagnosing an agent-reasoning incident with infrastructure tools alone.
• Investigating a cost spike by intuition instead of an actual breakdown.
• Assuming any latency issue is the LLM provider's fault without checking the actual trace.

Recommended next chapter

22-senior-system-design-cloud-ai.md