18.12 — Reliability Engineering for AI Systems

Relationship to Part 7.9: Part 7.9 already built an LLM gateway with retries, fallback models, and a circuit breaker (with a noted bug: every caller was allowed through in HALF_OPEN state). This chapter generalizes those patterns beyond the LLM-gateway-specific case, fixes that pattern correctly, and adds the infrastructure-wide reliability concepts (bulkheads, graceful degradation, disaster recovery) Part 7.9 didn't cover at the whole-system level.

1. What is it?

The set of patterns for building a system that keeps working — or fails in a controlled, limited way — when a dependency (an LLM provider, a database, a downstream service) is slow, erroring, or completely unavailable, applied specifically to the Enterprise AI Assistant's dependency graph.

2. Why does it exist?

An AI system has an unusually long chain of dependencies for a typical web application: the LLM provider, the vector database, the relational database, the cache, and often several external tool/API integrations (Part 3.3, Part 10.3) — and, distinctively, the LLM provider dependency is both essential to nearly every request and entirely outside your control (unlike your own database, you cannot fix Anthropic's or OpenAI's availability). This chapter exists because "what happens when the LLM provider is down or rate-limiting us" is not a hypothetical edge case for an AI system — it is one of the most common real production incidents an AI FDE will actually face.

3. What problem does it solve?

It solves "how do I stop one failing dependency from taking down my whole system" (via isolation and graceful degradation) and "how do I stop retries from making a bad situation worse" (via backoff and circuit breaking) — the concrete mechanisms behind an AI system continuing to provide some value during a partial outage rather than failing completely.

4. How does it work internally?

Timeouts — the first, most basic line of defense

A request with no timeout can hang indefinitely, holding a connection (and whatever resources the request-handling code holds — Part 1.1's async context) open forever if the dependency simply stops responding rather than erroring cleanly. Every outbound call — to the LLM provider, the database, an external tool API — needs an explicit timeout, sized to that specific dependency's realistic behavior (an LLM completion call reasonably takes longer than a database query, and should have a longer, but still bounded, timeout).

Retries and exponential backoff

A retry re-attempts a failed request. Exponential backoff increases the delay between successive retries (1s, 2s, 4s, 8s...) rather than retrying immediately and repeatedly — critical because immediate, repeated retries from many concurrent requests during a real outage create a retry storm: the retries themselves add enough additional load to a struggling or recovering dependency to prevent it from ever actually recovering, a self-inflicted amplification of the original problem. Jitter (adding a small random amount to each backoff delay) prevents many clients from retrying in synchronized lockstep, which would otherwise recreate exactly the same overload pattern at each backoff interval instead of spreading retries out. Only idempotent operations (Part 7.7's exact requirement, revisited here) should be retried safely by default — retrying a non-idempotent side-effecting call (charging a payment, sending a notification) can duplicate the effect.

import random
import asyncio

async def call_with_backoff(fn, max_attempts: int = 4, base_delay: float = 1.0):
    for attempt in range(max_attempts):
        try:
            return await fn()
        except RetryableError:
            if attempt == max_attempts - 1:
                raise
            delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)  # jitter
            await asyncio.sleep(delay)

Circuit breakers — done correctly this time

A circuit breaker tracks a dependency's recent failure rate and, once it crosses a threshold, "opens" — failing fast (without even attempting the call) for a cooldown period, instead of letting every request wait out a full timeout against a dependency that's very likely to fail anyway. After the cooldown, it enters half-open: allowing a limited number of trial requests through to test whether the dependency has recovered, not every caller simultaneously — Part 7.9's exact, flagged bug was allowing unlimited concurrent traffic through in half-open state, which both defeats the "test with limited traffic" purpose and can immediately re-trip the breaker if the dependency is still fragile. A correct half-open implementation limits concurrent trial requests explicitly:

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, cooldown_seconds: float = 30,
                 half_open_max_concurrent: int = 1):
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds
        self.half_open_max_concurrent = half_open_max_concurrent
        self._state = "CLOSED"
        self._failure_count = 0
        self._opened_at: float | None = None
        self._half_open_in_flight = 0

    def can_attempt(self) -> bool:
        if self._state == "CLOSED":
            return True
        if self._state == "OPEN":
            if time.monotonic() - self._opened_at >= self.cooldown_seconds:
                self._state = "HALF_OPEN"
                self._half_open_in_flight = 0
            else:
                return False
        if self._state == "HALF_OPEN":
            # THE FIX: only let a bounded number of trial requests through,
            # not every caller — this is what Part 7.9's version got wrong.
            if self._half_open_in_flight < self.half_open_max_concurrent:
                self._half_open_in_flight += 1
                return True
            return False
        return False

Rate limiting

Rate limiting protects a dependency (or your own service) from being overwhelmed by capping request rate — relevant in two directions for an AI system: outbound (respecting the LLM provider's own rate limits, Part 7.9's model-routing chapter) and inbound (protecting your own service from a traffic spike, or from one tenant consuming disproportionate shared capacity — Part 9.6's tenant-isolation concern, enforced here as a reliability mechanism, using Redis, 18.14's token-bucket pattern).

Bulkheads

A bulkhead isolates resources (a connection pool, a thread pool) per dependency or per tenant, so exhaustion in one doesn't starve the others — named for a ship's watertight compartments, which contain flooding to one section rather than sinking the whole vessel. Concretely for the Enterprise AI Assistant: a separate connection pool for the LLM provider's HTTP client versus the database's connection pool means a slow, backed-up LLM provider doesn't also exhaust the connections your application needs to serve requests that don't even call the LLM (a health check, a cached response, Part 7.8/18.14) — without a bulkhead, one dependency's slowness can starve unrelated request paths entirely.

Graceful degradation

Graceful degradation means providing reduced functionality instead of total failure when a dependency is unavailable — for the Enterprise AI Assistant specifically: if the reranker (Part 3.6) is unavailable, fall back to raw vector-similarity ranking rather than failing the whole request; if the primary LLM provider is down, fall back to a secondary provider or a smaller/cheaper model (Part 7.9's fallback-model pattern) rather than returning an error; if retrieval itself fails, consider whether answering from the model's parametric knowledge with an explicit caveat is an acceptable degraded mode for this specific use case, or whether an honest "I can't access our knowledge base right now" is the right call instead — a genuine, use-case-dependent judgment, not a universal rule.

Health checks, readiness, and liveness — the reliability angle

18.9 already covered these mechanically; the reliability-specific point worth adding is that a readiness check should reflect a pod's actual ability to serve requests usefully, which for an AI service arguably includes checking that its critical dependencies (the database, at minimum) are reachable — while a liveness check should stay minimal (is the process itself alive) specifically so a transient dependency blip doesn't cause Kubernetes to needlessly restart an otherwise-healthy process (18.9/18.10's exact liveness-vs-readiness distinction, now justified from the reliability-design angle rather than just the mechanical one).

Redundancy, failover, disaster recovery, and backups

Redundancy (multiple replicas, multiple AZs, 18.5) means no single instance's failure takes down the system. Failover is the mechanism that actually switches to a redundant resource when the primary fails (RDS Multi-AZ's automatic promotion, 18.4). Disaster recovery (DR) plans for a larger-scale event (an entire region becoming unavailable) — distinguished by two standard metrics: RTO (Recovery Time Objective — how long can the system be down) and RPO (Recovery Point Objective — how much data loss, measured in time, is acceptable). Backups (automated RDS snapshots, 18.4) are necessary but not sufficient for DR on their own — a backup that's never been tested for actual restoration is an unverified assumption, not a working recovery plan; the specific, common failure mode is only discovering a database backup is corrupted or incomplete during the actual disaster, when it's too late to fix.

What happens when the LLM provider itself becomes unavailable

Applying all of the above together, concretely, to the scenario the source spec asks about directly:

Application
   │
   ▼
Model Router (Part 7.9, with the CORRECT half-open circuit breaker above)
   ├── Provider A (primary) ── circuit breaker OPEN after threshold failures
   ├── Provider B (fallback) ── tried next, different provider entirely
   └── Cached/degraded response ── if BOTH providers are unavailable,
        return a cached answer for semantically similar past queries
        (18.14's semantic cache) if one exists, or an honest, clear
        "temporarily unavailable" response — never a silent failure or
        a fabricated answer presented as if retrieved normally

The trade-off worth stating explicitly, not glossing over: falling back to a different provider/model changes response characteristics (possibly quality, definitely cost, Part 7.9/7.11) — an acceptable trade during a genuine outage, but one that should be visible in logs/traces (18.11) so it's understood as a temporary degradation, not silently treated as equivalent to normal operation.

5. Simple mental model

Reliability engineering is building a system the way a well-designed ship is built: watertight compartments (bulkheads) so one breach doesn't sink the whole vessel, lifeboats (fallback providers, cached responses) for when the main vessel genuinely can't continue, and a tested (not just assumed) evacuation plan (disaster recovery) rather than discovering the lifeboats don't work during the actual emergency.

6. Real-world example

During a real LLM provider outage, the Enterprise AI Assistant's circuit breaker (correctly implemented, section 4) opens for the primary provider after five consecutive failures, and the model router shifts to a secondary provider for all new requests — visible in LangSmith traces (Part 6.1) as a distinct fallback_used: true tag, and in CloudWatch (18.11) as a metric spike on the fallback provider's usage. Users experience slightly different response characteristics but no outright failures. Once the primary provider recovers, the circuit breaker's half-open state (correctly limited to one trial request at a time, not flooding the recovering provider) verifies recovery before shifting traffic back — the entire incident resolved without a single page to a human being necessary, purely through correctly-implemented automated resilience.

7. Architecture diagram

Request

Timeout-bounded call → Circuit breakercorrect half-open limiting

Normal response

Fallback providerPart 7.9

Cached/degraded response18.14

Honest "temporarily unavailable" responsenever a silent or fabricated failure

8. Production considerations

• Every outbound call needs an explicit, dependency-appropriate timeout — no exceptions, including calls that "usually" respond quickly.
• Fix the half-open circuit-breaker bug explicitly if reusing Part 7.9's gateway code — limiting concurrent trial requests is not optional polish, it's what makes half-open state actually serve its purpose.
• Test disaster-recovery restoration procedures periodically, not just configure backups and assume they work.
• Use separate connection pools (bulkheads) per critical dependency so one slow dependency can't starve unrelated request paths.

9. Common mistakes

• Retrying without backoff or jitter, causing a retry storm that prevents a struggling dependency from ever recovering.
• A circuit breaker's half-open state allowing unlimited concurrent traffic through — Part 7.9's exact, real bug, restated here as a general anti-pattern to check for in any circuit-breaker implementation.
• Treating "we have RDS Multi-AZ" as equivalent to "we have a tested disaster recovery plan" — Multi-AZ handles a narrower class of failure than a full DR scenario (an entire region, or a corrupted backup).
• Silently falling back to a degraded mode with no visibility (logging/ tracing) that degradation occurred, making a real outage look identical to normal operation in dashboards.

10. Security considerations

A fallback provider/model (section 4) must meet the same data-handling requirements (Part 9.3/9.6, data residency, what data leaves the environment) as the primary — a resilience mechanism that silently routes sensitive data to a provider that wasn't vetted for that data classification is a real, easy-to-overlook security gap introduced by a reliability feature.

11. Performance considerations

Circuit breakers directly improve perceived performance during an incident by failing fast instead of making every caller wait out a full timeout against a dependency very likely to fail — the "fail fast" benefit is itself a performance property, not just a resilience one.

12. Cost considerations

A fallback provider is very often priced differently (Part 7.9/7.11) than the primary — a prolonged fallback period during an extended outage has a real, sometimes-significant cost implication worth monitoring, not assuming is identical to normal-operation cost.

13. When to use it

Every dependency an AI system relies on for a user-facing request benefits from at least a timeout and, for critical/flaky dependencies, a circuit breaker — LLM provider calls specifically warrant the full pattern set given both their criticality and their being outside your direct control.

14. When NOT to over-apply it

A low-stakes internal tool with a small, forgiving user base may not need a full circuit-breaker-plus-fallback-provider setup — matching resilience investment to the actual cost of an outage for that specific system, rather than building maximum resilience machinery everywhere by default.

15. Alternatives and trade-offs

Building resilience patterns yourself (as shown) versus adopting them from a dedicated library or an LLM gateway product (18.7/7.9's point about LiteLLM/Portkey/Kong) is a real trade-off between control/understanding and development time — a mature team often adopts an existing, tested implementation rather than maintaining a hand-rolled circuit breaker, once correctness at this level of subtlety (section 4's half-open bug) is weighed against the ongoing maintenance burden.

16. Practical example — a bulkheaded HTTP client setup

import httpx

# Separate clients (and therefore separate connection pools) per
# dependency — a bulkhead preventing one dependency's slowness from
# starving connections needed for another, unrelated dependency.
llm_client = httpx.AsyncClient(
    timeout=httpx.Timeout(30.0, connect=5.0),
    limits=httpx.Limits(max_connections=20, max_keepalive_connections=10),
)
vector_db_client = httpx.AsyncClient(
    timeout=httpx.Timeout(5.0, connect=2.0),
    limits=httpx.Limits(max_connections=20, max_keepalive_connections=10),
)

17. Production-quality example — a resilient LLM call combining every pattern

"""
Combines timeout, retry-with-backoff-and-jitter, a correctly-implemented
circuit breaker, and graceful degradation to a fallback provider — the
full reliability stack from this chapter applied to one real call path.
"""
async def resilient_llm_call(prompt: str, primary_breaker: CircuitBreaker,
                               fallback_breaker: CircuitBreaker) -> str:
    for breaker, provider in [(primary_breaker, primary_provider),
                                (fallback_breaker, fallback_provider)]:
        if not breaker.can_attempt():
            continue   # this provider's breaker is open — skip immediately,
                        # don't waste a timeout on a very-likely failure
        try:
            return await call_with_backoff(lambda: provider.complete(prompt))
        except RetryableError:
            breaker.record_failure()
            continue

    # BOTH providers exhausted — degrade honestly, never silently fabricate
    cached = await semantic_cache_lookup(prompt)   # 18.14
    if cached:
        logger.warning("llm_providers_unavailable_using_cache")
        return cached
    logger.error("llm_providers_unavailable_no_fallback")
    raise ServiceTemporarilyUnavailable(
        "Our AI assistant is temporarily unavailable. Please try again shortly."
    )

18. Short exercise

Trace through what happens, step by step using this chapter's mechanisms, if the vector database (not the LLM provider) becomes unavailable for the Enterprise AI Assistant — identify specifically what graceful-degradation options exist here (raw keyword search? parametric-knowledge answer with a caveat? an honest failure?) and argue for which is the right default, given this system's stated risk profile from Part 16.8's underwriting scenario specifically (a high-stakes, audit-required context).

19. Interview questions

• Why is a retry storm worse than the original failure it was retrying against?
• Walk through the correct half-open circuit-breaker behavior and why allowing unlimited concurrent trial requests defeats its purpose.
• What's the difference between a bulkhead and a circuit breaker, and when do you need both?
• What's the difference between RTO and RPO, and why does a backup alone not guarantee either?

20. FDE/customer scenario

A customer asks: "What happens to our AI assistant if OpenAI/Anthropic has an outage?" A strong answer walks through this chapter's concrete mechanism — a circuit breaker detecting the failure quickly, automatic fallback to a secondary provider or degraded cached response, with clear, visible logging of the degraded state rather than a silent failure — and is honest about the real trade-off involved (a fallback provider may have different quality/cost characteristics), rather than claiming the system is simply "always available" without qualification.

---

Key takeaways

• Retries without backoff and jitter can turn a recoverable outage into an unrecoverable one by overloading a dependency that's trying to recover.
• A circuit breaker's half-open state must limit concurrent trial requests — allowing every caller through defeats its entire purpose (Part 7.9's real, documented bug).
• Graceful degradation should be visible (logged, traced) and honest (never a silently fabricated answer) — a degraded mode is still a mode worth being transparent about.

Things you should be able to explain

• Why exponential backoff with jitter is necessary, not just "nice to have."
• The correct half-open circuit-breaker behavior, and why Part 7.9's original version was a real bug, not a stylistic choice.
• The difference between RTO/RPO and why backups alone don't satisfy DR.

Things you should be able to build

• A correctly-implemented circuit breaker with bounded half-open concurrency.
• A resilient LLM call path combining timeout, retry, circuit breaking, and honest graceful degradation.

Common mistakes

• Retries with no backoff/jitter, causing retry storms.
• Circuit breakers that flood a recovering dependency in half-open state.
• Silent, untraced fallback to a degraded mode.

Recommended next chapter

13-scalability-for-ai-workloads.md