5.7 — Durable Execution, Retries, and Failure Recovery

1. What is it?

This chapter covers what happens when something goes wrong mid-execution of a LangGraph graph — a node's LLM call fails transiently, a tool call throws an exception, or the entire process crashes — and how LangGraph's checkpointing (Part 5.3) combines with explicit retry policies and failure-handling code to make a multi-step workflow genuinely durable: able to recover from failure at exactly the point it occurred, rather than either losing all progress or (worse) silently continuing in a corrupted state.

2. Why does it exist?

Part 1's Testing and Linux chapters, and Part 7's upcoming resilience patterns, all establish that real production systems fail in all sorts of partial, transient ways — a network blip, a provider's momentary rate limit, a downstream service's brief unavailability. A single-call LLM interaction (Part 2.6) has one obvious failure-handling story: catch the exception, retry or fail the request. A multi-step graph has a genuinely harder version of this problem: if step 3 of 5 fails, do you lose steps 1-2's work? Retry just step 3? The entire run? Durable execution and LangGraph's retry mechanisms exist to answer this precisely, building directly on Part 5.3's checkpointing so that recovery can resume from the last successfully completed step rather than from scratch.

3. What problem does it solve?

It solves "how do I make a multi-step, potentially long-running, potentially expensive workflow robust to the transient failures that are a statistical certainty at any real production scale" — without either building fragile, ad hoc retry logic scattered through every node, or accepting that any transient failure anywhere in a long workflow means restarting the entire (potentially expensive, potentially partially-side-effecting) sequence from step one.

4. How does it work internally?

Node-level retry policies

LangGraph supports attaching a retry policy to individual nodes, specifying how many times to retry that specific node's execution on failure, with what backoff strategy, and for which exception types — directly implementing Part 1's tenacity-based retry pattern (Part 1.1's production example) as a first-class graph feature rather than code you'd hand-write inside every node:

from langgraph.pregel import RetryPolicy

graph.add_node(
    "credit_check",
    credit_check_node,
    retry=RetryPolicy(max_attempts=3, retry_on=TransientAPIError),
)

Verify the exact current RetryPolicy API surface (parameter names, default backoff behavior) against current LangGraph documentation. The key architectural point: a retry policy scoped to one specific node means only that node re-executes on transient failure — the graph doesn't need to re-run already-successfully-completed prior nodes (their results are already checkpointed, Part 5.3), and the retry logic doesn't need to be duplicated across every node's own implementation.

Why checkpointing is what makes recovery from a hard crash possible

A retry policy handles transient failures within a single execution attempt. A genuinely different failure mode — the entire process crashing (a server restart, an out-of-memory kill, Part 1.8's OOM-killer discussion) mid-graph-run — is handled by Part 5.3's checkpointing: since state is durably persisted after every completed superstep, a crashed process's in-flight graph run can be resumed (using the same thread_id, Part 5.3) by a different process entirely, continuing from the last successfully checkpointed superstep rather than losing all prior progress or requiring the exact same process to survive.

Superstep 1 (credit_check) completescheckpointed

Superstep 2 (income_check) completescheckpointed

Superstep 3 (fraud_check) — PROCESS CRASHESmid-execution

A new process instance resumessame thread_id

LangGraph loads the last checkpoint and re-executes ONLY superstep 3 onwardcredit_check and income_check are NOT re-run

The idempotency requirement this creates — a critical, often-missed detail

This recovery model has a direct, important implication that mirrors Part 1.2's idempotency discussion: a node that has a real-world side effect (sending an email, charging a payment, calling an external API with a non-idempotent effect) must be designed with the possibility that it could be retried or re-executed — if fraud_check_node in the example above had, say, already sent a notification email before the crash occurred partway through its own execution, resuming and re-running that node from scratch could send the notification twice. This is precisely Part 1.2's idempotency-key pattern, now required at the node level for any node with an external side effect, not just at the API-endpoint level.

Failure handling within a node — explicit, not implicit

Beyond the framework-level retry policy, a node's own implementation should still handle failure explicitly for cases that shouldn't simply retry (e.g., a permanent validation failure versus a transient network error) — exactly Part 3.3's tool-execution error-handling pattern, applied inside a graph node:

async def credit_check_node(state: LoanReviewState) -> dict:
    """Runs the credit check, distinguishing transient failures (let RetryPolicy
    handle them) from permanent failures (handle explicitly, don't retry uselessly)."""
    try:
        result = await run_credit_check(state["application_id"])
    except InvalidApplicationDataError as exc:
        # Permanent, not transient — retrying won't help; handle explicitly
        return {"credit_check_result": {"error": str(exc), "status": "failed_permanently"}}
    return {"credit_check_result": result}

Bounding a looping graph — recursion_limit and GraphRecursionError

Everything above handles a graph that fails outright. A genuinely different failure mode is a graph that never fails and never finishes — a conditional edge (Part 5.2) that keeps routing back into the same node(s) because some expected exit condition never actually becomes true (a supervisor, Part 5.6, that never decides the task is "done"; a self-correction loop that never converges). Left unbounded, this doesn't crash — it just keeps consuming LLM calls and wall-clock time indefinitely, which in production is a cost incident and an availability incident at once, not merely a bug.

LangGraph's structural defense against this is the recursion_limit config option, passed alongside thread_id at invocation time:

config = {
    "configurable": {"thread_id": "loan-application-4521"},
    "recursion_limit": 50,
}
result = await compiled.ainvoke(initial_state, config=config)

recursion_limit caps the total number of supersteps (Part 5.1) a single invocation is allowed to execute — not the number of nodes in the graph, but the number of steps actually taken, which is exactly what an unbounded loop keeps incrementing. When the limit is hit, LangGraph raises GraphRecursionError rather than continuing silently or being killed by some external timeout with no specific diagnostic signal — the exception itself is the concrete evidence that a loop ran away, and it identifies precisely which execution hit the ceiling (correlatable, via thread_id, to a specific trace in LangSmith, Part 6.1 section on diagnosing stuck graphs).

Verify the exact current default value and config key name (recursion_limit under configurable vs. top-level) against current LangGraph documentation — this is exactly the kind of config-surface detail that has shifted across versions.

Sizing it deliberately, not by copying a default: the right value is relative to the graph's actual topology, not a round number picked arbitrarily. Count the expected maximum number of supersteps a legitimate run of this specific graph should ever take — a fixed five-node sequential pipeline (Part 5.6) legitimately needs a limit only slightly above 5; a supervisor pattern that might legitimately route through eight sub-agent turns before completing needs a limit comfortably above 8, but not so high that a genuine runaway loop gets to burn hundreds of LLM calls before being stopped. Setting it too low causes GraphRecursionError on entirely legitimate long-but-finite runs (a false-positive failure); setting it too high (or leaving an oversized framework default unexamined) means a real runaway loop still costs significantly before being caught. This is a judgment call specific to each graph's topology, not a single value that's correct for every graph in a codebase.

5. Simple mental model

Think of Part 5.3's checkpointing plus node-level retry policies as a multi-stage assembly line with both quality-control checkpoints and a foreman who knows which specific stations are worth re-attempting versus reporting as broken. recursion_limit is the assembly line's own step counter with a hard stop — a safeguard against the line looping back on itself and running forever if a routing decision downstream never actually signals "finished," distinct from either the transient-failure retries or the crash-recovery mechanism above. If station 3 has a momentary glitch (a transient failure), the foreman (retry policy) has it retry a few times before giving up — no need to restart the entire line from station 1. If the entire factory loses power mid-shift (a process crash), when power returns, work resumes from the last completed, checkpointed station rather than remaking everything from scratch — but any station whose job was "seal and ship this specific box" (a non-idempotent side effect) needs its own explicit safeguard against accidentally shipping the same box twice if it gets re-run.

6. Real-world example

A document-processing pipeline's send_completion_notification_node sends an email to the customer when processing finishes. During testing of the recovery behavior, the team discovered that a process crash occurring right after this node sent its email, but before its checkpoint fully committed, would cause the resumed run to re-execute this node and send a duplicate email — precisely the idempotency gap described in section 4. The fix: adding an idempotency check at the start of the node itself (checking, via a durable record, whether a notification for this specific thread_id/application had already been sent, before sending another) — directly applying Part 1.2's idempotency-key pattern at the node level, closing the gap the recovery model's re-execution behavior had exposed.

7. Architecture diagram

Node-level RetryPolicyhandles TRANSIENT failures: credit_check_node fails (network blip) → retries up to 3x within the same attempt

Checkpointing (Part 5.3)handles PROCESS-LEVEL failures: process crashes → new process resumes from last checkpoint, re-executing only the incomplete superstep

Node-level idempotencyhandles SIDE-EFFECT SAFETY under retry/resume: send_notification_node checks "already sent?" before sending

recursion_limit / GraphRecursionErrorhandles RUNAWAY LOOPS: a conditional edge with no legitimate exit hits recursion_limit → error raised, stopping the loop deterministically

Four independent mechanisms, each handling a different failure class — not layers of one pipeline.

8. Production considerations

• Set an explicit, topology-sized recursion_limit on every graph with a conditional loop (any supervisor, Part 5.6, self-correction, or retry-style routing edge) — don't rely on an unexamined framework default; size it relative to the graph's actual legitimate maximum step count (section 4).
• Attach retry policies to every node making an external call (LLM API, database, third-party API) — exactly Part 1's discipline (timeouts, retries with backoff, Part 1.1's production example) now expressed declaratively at the graph level rather than duplicated by hand in every node's implementation.
• Audit every node with a real-world side effect for idempotency (section 4/6) — this is the single most commonly missed production-readiness gap in graph-based workflows, precisely because the failure mode (a crash at exactly the wrong instant) is rare enough to not show up in normal testing but statistically certain to eventually occur at real production scale and volume.
• Distinguish transient from permanent failures explicitly in node logic — retrying a permanent failure (a malformed input that will never succeed) wastes the retry budget and delays surfacing the real problem; only transient failures (network issues, rate limits) should be retried automatically.
• Log every retry attempt and every resume-from-checkpoint event — this observability (Part 4.6, Part 8.4) is what lets you actually verify the recovery model is working correctly in production, rather than assuming it silently is.

9. Common mistakes

• Nodes with real-world side effects (sending notifications, charging payments, calling non-idempotent external APIs) with no idempotency safeguard, creating exactly the duplicate-side-effect risk from section 6 the first time a crash happens to occur at the wrong instant.
• Applying a retry policy uniformly to every node regardless of whether its failures are typically transient or permanent, wasting retry budget on failures that will never succeed no matter how many times they're retried.
• Not testing the actual crash-and-resume behavior explicitly (Part 1.9) — assuming checkpointing "just works" without verifying it with a deliberate test that simulates a mid-run process failure and confirms correct resumption.
• Leaving recursion_limit at an unexamined default (too high to catch a runaway loop's cost quickly, or too low and triggering GraphRecursionError on legitimate long-but-finite runs) instead of sizing it deliberately against the specific graph's topology (section 4).

10. Security considerations

• A resumed execution picks up with whatever authorization context was captured in its checkpointed state (Part 5.3, section 10) — if that context can become stale (a user's permissions changed between the crash and the resume), a resumed node might execute with outdated authorization; consider whether long-paused or long-delayed resumes should re-verify authorization rather than blindly trusting checkpointed context that could be significantly out of date.

11. Performance considerations

• Retry backoff strategies (Part 1's tenacity discussion) should scale delay between attempts to avoid hammering an already-struggling downstream service — the same exponential-backoff discipline from Part 1.1 applies directly to RetryPolicy configuration.
• Checkpointing overhead (Part 5.3, section 11) is the same cost regardless of whether a run ultimately fails or succeeds — a workflow with very frequent transient failures and retries will checkpoint (and pay that cost) correspondingly more often.

12. Cost considerations

• Retries on an LLM-calling node have the same real, additive cost as any other retry (Part 3.2, section 12) — a node retried 3 times due to transient failures pays for up to 3x the LLM calls for that specific step, worth accounting for in cost estimates for workflows operating at real scale with a non-trivial transient-failure rate.

13. When to use it

Every production graph, without exception, should have deliberate retry policies on external-call nodes and explicit idempotency safeguards on any node with a real-world side effect — this isn't an optional hardening step, it's a core part of what makes a graph genuinely production-ready rather than merely "working in the demo."

14. When NOT to use it

A node with no external call and no side effect (pure computation on already-available state) has no meaningful failure mode to design retry/idempotency handling around — apply this discipline proportionally to where real failure risk actually exists, not uniformly to every node regardless of what it does.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Node-level RetryPolicy	Declarative, consistent, less duplicated code	Only handles retries within one execution attempt
Checkpoint-based resume (Part 5.3)	Recovers from full process crashes	Requires idempotency discipline for side-effecting nodes
No retry/recovery design (anti-pattern)	Simplicity for a demo	Guaranteed to fail unpredictably and non-recoverably at real production scale
recursion_limit / GraphRecursionError	Deterministically bounding a looping graph's cost/duration	A wrong-sized limit either false-fails legitimate long runs or catches runaway loops too late

16. Practical Python/code example

from langgraph.pregel import RetryPolicy

graph.add_node(
    "credit_check",
    credit_check_node,
    retry=RetryPolicy(max_attempts=3, retry_on=(TransientAPIError, TimeoutError)),
)
graph.add_node(
    "send_notification",
    send_notification_node,
    retry=RetryPolicy(max_attempts=3, retry_on=TransientAPIError),
)

17. Production-quality example

An idempotent notification node, directly implementing the section 6 fix using a durable "already sent" record:

import logging

logger = logging.getLogger("notification_node")


async def send_notification_node(state: "DocumentProcessingState") -> dict:
    """
    Sends a completion notification, checking a durable idempotency record first
    so a resumed/retried execution never sends a duplicate notification.

    Args:
        state: The graph's current state, including the thread/application identifier.

    Returns:
        dict: A state update recording that notification has been (or already was) sent.
    """
    idempotency_key = f"notification:{state['application_id']}"

    already_sent = await idempotency_store.check_and_set(idempotency_key)
    if already_sent:
        logger.info("notification for %s already sent, skipping duplicate", state["application_id"])
        return {"notification_sent": True}

    try:
        await send_email(
            to=state["applicant_email"],
            subject="Your application has been processed",
            body=render_notification_body(state),
        )
    except Exception:
        await idempotency_store.clear(idempotency_key)  # allow a genuine retry on real failure
        raise

    logger.info("notification sent for %s", state["application_id"])
    return {"notification_sent": True}

idempotency_store.check_and_set should be an atomic operation (e.g., backed by Redis's SET ... NX, Part 1.6, or a unique-constraint database insert) — the same atomicity requirement Part 1.2's idempotency-key discussion emphasized, now applied at the node level.

18. Short exercise

A graph node calls a third-party payment API to charge a customer. Using this chapter's idempotency pattern, describe (in plain language, no code required) exactly what could go wrong if this node has no idempotency safeguard and the process crashes immediately after the payment API call succeeds but before the node's checkpoint commits — and what specific safeguard would prevent it.

19. Interview questions

• Explain the difference between what a node-level RetryPolicy handles and what checkpoint-based resume handles — why are both necessary?
• Why does the checkpoint-and-resume recovery model create an idempotency requirement for nodes with real-world side effects?
• How would you distinguish, in a node's own error-handling code, a failure worth retrying automatically from one that should fail immediately without retry?
• What does recursion_limit bound, specifically, and how would you decide what value to set it to for a given graph?

20. FDE/customer scenario

Customer's operations team: "We had an incident where a customer got charged twice after our system had some kind of hiccup — can you explain how that happened?"

This maps directly onto section 4/6/9's core failure mode: a payment-charging node without an idempotency safeguard, re-executed after a process crash or transient-failure retry, can produce exactly this symptom. Diagnosing it correctly (checking whether the charging node has an idempotency check, rather than assuming a one-off unexplainable glitch) and proposing the concrete fix from section 17 is a credible, technically precise response that also demonstrates exactly the kind of systematic reliability thinking an enterprise customer needs to trust the system with real financial operations going forward.

---

Key takeaways

• Node-level RetryPolicy handles transient failures within one execution attempt; checkpoint-based resume (Part 5.3) handles full process crashes — both are necessary, and neither substitutes for the other.
• Any node with a real-world side effect must be designed to be idempotent, because the checkpoint-and-resume recovery model can re-execute a node whose prior attempt partially succeeded.
• Distinguish transient failures (worth retrying) from permanent failures (retrying wastes budget and delays surfacing the real problem) explicitly in node logic.
• recursion_limit bounds a graph's total superstep count and raises GraphRecursionError when exceeded — the structural defense against a routing loop that never legitimately terminates, sized deliberately against each graph's actual topology, not left at an unexamined default.

Things you should be able to explain

• Why checkpoint-based recovery creates an idempotency requirement for side-effecting nodes.
• The difference between what RetryPolicy and checkpointing each protect against.
• What recursion_limit bounds and why its correct value depends on the specific graph's topology.

Things you should be able to build

• A graph with node-level retry policies and an idempotent, side-effect-safe notification node.

Common mistakes

• No idempotency safeguard on side-effecting nodes.
• Uniform retry policies applied regardless of whether failures are typically transient or permanent.
• Never testing actual crash-and-resume behavior explicitly.
• Leaving recursion_limit at an unexamined default instead of sizing it to the graph's actual topology.

Recommended next chapter

08-production-architecture.md