7.6 — Async Processing Patterns: Beyond Request-Time Concurrency

1. What is it?

Part 1.1 covered asyncio as the concurrency model within a single request's handling — many requests interleaved on one event loop. This chapter covers a different, complementary concept: deferred/background processing, where a request's actual work happens outside the request/response cycle entirely, with the client notified of completion later (via polling or a webhook) rather than waiting synchronously for the full result.

2. Why does it exist?

Part 1.2 flagged that some AI operations (a long document analysis, a multi-step agent run, Part 3.7) can take much longer than a typical HTTP request should reasonably stay open — a client (a browser, a mobile app, another service) holding a connection open for 30+ seconds risks timeouts at multiple layers (the client itself, load balancers, Part 7.5's connection handling) and provides a poor user experience regardless. Deferred processing exists to decouple "the client's request completing" from "the actual work finishing," letting genuinely long-running AI operations complete reliably without requiring an equally long-lived, fragile HTTP connection.

3. What problem does it solve?

It solves "how do I handle an AI operation that genuinely takes minutes, not milliseconds, without forcing the client to hold a fragile long-lived connection open the entire time, and without blocking my own server's capacity to handle other requests in the meantime" — directly relevant for batch document processing, complex multi-agent workflows (Part 5.6), or any operation whose duration is inherently unpredictable and potentially long.

4. How does it work internally?

The job-creation-plus-polling pattern

Client                                  Server                    Background Worker
  │  POST /v1/reports/generate            │                              │
  ├──────────────────────────────────────►│                              │
  │                                        │  creates job record          │
  │                                        │  (status: "pending")          │
  │                                        ├─────────────────────────────►│
  │  202 Accepted                          │  enqueues job                │
  │  {"job_id": "abc123",                  │                              │
  │   "status": "pending"}                 │                              │
  ◄──────────────────────────────────────┤                              │
  │                                                                       │  processes job
  │  GET /v1/reports/abc123                                              │  (potentially
  ├──────────────────────────────────────►│                              │   minutes)
  │  {"status": "pending"}                 │                              │
  ◄──────────────────────────────────────┤                              │
  │  ... client polls again later ...                                    │
  │  GET /v1/reports/abc123                                              │  updates job
  ├──────────────────────────────────────►│◄─────────────────────────────┤  record: "complete"
  │  {"status": "complete", "result": ...} │                              │
  ◄──────────────────────────────────────┤                              │

This is precisely Part 1.2's 202 Accepted pattern, made concrete end-to-end: the initial request returns immediately with a job identifier, and the actual work happens asynchronously in a separate worker process (Part 7.7 covers the queue/worker infrastructure that makes this reliable), with the client checking back for the result rather than waiting on an open connection.

The webhook alternative — push instead of poll

Rather than the client repeatedly polling for status, the server can instead call back to a client-provided URL once the job completes — better for scenarios where polling overhead is undesirable or where the client isn't well-suited to repeated polling (e.g., another backend service integration). This trades polling overhead for the added complexity of managing webhook delivery reliability (Part 7's resilience patterns — retries, Part 1.2's idempotency — apply directly to webhook delivery, since a webhook call can itself fail transiently and needs the same retry discipline as any other external call).

Why this differs from Part 1.1's within-request async

It's worth being precise about the distinction this chapter draws, since both concepts are called "async" colloquially but solve different problems: Part 1.1's asyncio concurrency lets one process handle many concurrent requests efficiently while each individual request is still being actively worked on and will return its result directly. This chapter's deferred processing is about decoupling a single request's response from when its work actually completes — the client isn't waiting synchronously at all, and the work might not even happen on the same process or even the same machine that received the initial request (Part 7.7's worker processes).

5. Simple mental model

Within-request async (Part 1.1) is like a chef working on multiple dishes at once, moving between them while each cooks — every dish is being actively worked on and served directly to the customer who ordered it as soon as it's ready. Deferred processing is like a restaurant taking a large catering order, giving the customer a claim ticket and an estimated ready time, and calling them when it's actually ready — the customer doesn't stand at the counter waiting; they leave and come back (poll) or get a call (webhook) once the work — happening in a separate kitchen entirely, possibly hours later — is done.

6. Real-world example

A legal-document analysis platform processes uploaded contracts that can take 3-8 minutes to fully analyze (multiple RAG lookups, an agent-based clause-by-clause review, Part 3.5/3.7). Building this as a synchronous request would mean holding an HTTP connection open for up to 8 minutes — fragile against client timeouts, load-balancer idle-connection limits (Part 7.5), and any transient network blip along the way. Building it as a deferred job (client uploads the document, gets a job ID immediately, and either polls or receives a webhook when analysis completes) makes the actual multi-minute processing entirely decoupled from any single HTTP connection's fragility, and lets the platform queue and process many such jobs concurrently via a proper worker pool (Part 7.7) without any client-facing timeout risk at all.

7. Architecture diagram

Client

API serverPart 1.3 · POST returns immediately, 202 + job_id

Job queuePart 7.7

Worker processdoes the actual long-running AI work, updates job status/result

8. Production considerations

• Make job status/result storage durable (a database, Part 1.4/1.5, not just in-memory) — exactly Part 5.3's checkpointing argument, applied to job status specifically: a job's status must survive the API server or worker process restarting, since the whole point of this pattern is decoupling from any single process's lifetime.
• Design webhook delivery with retries and idempotency (Part 1.2, Part 5.7) — a webhook call is itself an external call subject to transient failure, and the receiving client should be able to safely handle a duplicate webhook delivery without adverse effect.
• Give clients a way to know approximately how long to expect to wait (an estimated completion time, or at least a reasonable polling-interval recommendation) — this is a real UX consideration Part 1.2's exercise touched on, and its absence leads to either wasteful rapid polling or an unnecessarily anxious, uninformed wait.
• Set a maximum job lifetime/timeout and a clear failure state — a job that's been "pending" for hours due to an unrecovered worker failure should eventually be marked failed rather than remaining silently, indefinitely pending.

9. Common mistakes

• Building a genuinely long-running AI operation (multi-minute agent run, batch processing) as a synchronous request, then discovering client/load-balancer timeout issues only under real production load and duration variance.
• Storing job status only in memory, losing all in-flight job state on any process restart.
• No retry/idempotency discipline on webhook delivery, causing missed notifications on transient failures or duplicate-processing issues on the receiving end.
• No maximum job lifetime, leaving genuinely stuck/failed jobs in a "pending" state indefinitely with no automated detection or client-facing failure signal.

10. Security considerations

• Job IDs should not be easily guessable/enumerable (use a UUID or equivalent, not a sequential integer) — a guessable job ID could let an unauthorized party poll for another user's job status/results if authorization isn't also separately enforced (Part 1.2's 403-vs-404 discussion applies directly: even with a hard-to-guess ID, explicit authorization checking on every status/result request remains necessary).
• Webhook URLs, if client-provided, are a real SSRF (server-side request forgery) risk surface if not validated — a naive implementation calling back to an arbitrary client-supplied URL could be abused to make your server issue requests to internal, otherwise-unreachable infrastructure; validate and restrict webhook destination URLs appropriately.

11. Performance considerations

• Polling interval design is a real trade-off: too frequent wastes server resources on largely-empty status checks; too infrequent produces a worse perceived-latency user experience — a client-side exponential backoff (poll frequently at first, less frequently as time passes) is a common, effective middle ground.

12. Cost considerations

• Deferred processing doesn't by itself reduce the underlying LLM/compute cost of the work itself (Part 2.6, Part 3.11) — it changes when and how that work is orchestrated, not what it costs; don't conflate "we made this asynchronous" with "we made this cheaper."
• That said, work already built as a deferred job (this chapter's pattern) is exactly the workload shape that qualifies for a provider's discounted batch inference API (Part 7.10) — since the client was never waiting on a synchronous response anyway, routing the actual LLM calls through the batch API on top of this chapter's job/polling infrastructure captures a real price-per-token reduction with no additional UX cost.

13. When to use it

Any AI operation whose duration is long enough (roughly, beyond a few seconds, and especially anything potentially reaching tens of seconds or minutes) or unpredictable enough that holding a synchronous connection open is fragile or provides a poor user experience — multi-step agent runs, batch document processing, complex multi-stage RAG pipelines with many retrieval/generation rounds.

14. When NOT to use it

A genuinely fast, predictable operation (a single classification call, a short RAG-grounded answer typically completing in a few seconds) doesn't need this added complexity — Part 1.1's within-request async concurrency is entirely sufficient, and adding a job-queue layer for something this fast is unnecessary overhead per Part 3.12's core discipline.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Synchronous request/response (Part 1.1)	Fast, predictable operations	Fragile/poor UX for genuinely long-running work
Job-creation + polling	Client controls check-in timing, simple to implement	Polling overhead, some latency between completion and client noticing
Job-creation + webhook	No polling overhead, immediate notification	Webhook delivery reliability/security considerations, more complex to implement correctly

16. Practical Python/code example

from fastapi import FastAPI
from pydantic import BaseModel
import uuid

app = FastAPI()


class JobStatus(BaseModel):
    """Status and, when complete, result of a deferred processing job."""
    job_id: str
    status: str  # "pending", "processing", "complete", "failed"
    result: dict | None = None


@app.post("/v1/reports/generate", status_code=202)
async def create_report_job(request: "ReportRequest", job_store, task_queue) -> JobStatus:
    """
    Creates a deferred report-generation job, returning immediately with a job ID
    rather than waiting for the potentially multi-minute analysis to complete.
    """
    job_id = str(uuid.uuid4())
    await job_store.create(job_id=job_id, status="pending")
    await task_queue.enqueue("generate_report", job_id=job_id, document_id=request.document_id)
    return JobStatus(job_id=job_id, status="pending")


@app.get("/v1/reports/{job_id}")
async def get_report_status(job_id: str, job_store, current_user) -> JobStatus:
    """Returns a job's current status, enforcing that only its owner can view it."""
    job = await job_store.get(job_id)
    if job is None or job.owner_id != current_user.id:
        raise HTTPException(status_code=404, detail="job not found")
    return JobStatus(job_id=job_id, status=job.status, result=job.result)

17. Production-quality example

Adding webhook delivery with retry and idempotency, per section 8/9's recommendations:

import logging
import httpx
from tenacity import retry, stop_after_attempt, wait_exponential

logger = logging.getLogger("webhook_delivery")

ALLOWED_WEBHOOK_HOST_SUFFIXES = (".customer-domain.com",)  # explicit allowlist, mitigating SSRF


@retry(stop=stop_after_attempt(5), wait=wait_exponential(multiplier=1, max=60))
async def deliver_webhook(webhook_url: str, job_id: str, result: dict) -> None:
    """
    Delivers a job-completion webhook with retry on transient failure, and an
    idempotency key so the receiving system can safely deduplicate retried deliveries.

    Args:
        webhook_url (str): The customer-provided callback URL, validated against
            an allowlist before use to mitigate SSRF risk.
        job_id (str): The completed job's identifier, used as the idempotency key.
        result (dict): The job's result payload.

    Raises:
        ValueError: If the webhook URL isn't on the allowed destination list.
    """
    from urllib.parse import urlparse
    host = urlparse(webhook_url).hostname or ""
    if not any(host.endswith(suffix) for suffix in ALLOWED_WEBHOOK_HOST_SUFFIXES):
        raise ValueError(f"webhook host not on allowlist: {host}")

    async with httpx.AsyncClient(timeout=10.0) as client:
        response = await client.post(
            webhook_url,
            json={"job_id": job_id, "result": result},
            headers={"Idempotency-Key": job_id},
        )
        response.raise_for_status()
    logger.info("webhook delivered for job_id=%s", job_id)

18. Short exercise

A customer's integration receives your webhook but their endpoint occasionally times out on their end, causing your retry logic to redeliver. Explain, using the Idempotency-Key header from section 17, what the customer's receiving endpoint should do to handle this safely, and what could go wrong if they don't implement deduplication on their end.

19. Interview questions

• Explain the difference between within-request async concurrency (Part 1.1) and deferred/background processing, and why both are sometimes called "async" despite solving different problems.
• Why is a durable, database-backed job status store necessary rather than storing job state in memory?
• What SSRF risk does accepting a client-provided webhook URL introduce, and how would you mitigate it?

20. FDE/customer scenario

Customer: "Our document analysis feature times out for large documents, but works fine for small ones."

This is a direct match for this chapter's core motivation: a synchronous request/response design that works for short operations breaks down as operation duration grows unpredictably with input size — recommending a shift to the job-creation-plus-polling (or webhook) pattern from section 4 directly and permanently resolves the timeout-fragility issue, regardless of how large a given document's processing time turns out to be, rather than the common but ultimately futile alternative of just increasing timeout thresholds further and further.

---

Key takeaways

• Deferred/background processing decouples a request's response from when its actual work completes — a genuinely different concept from within-request async concurrency (Part 1.1), despite the shared "async" label.
• The job-creation-plus-polling (or webhook) pattern makes genuinely long-running AI operations reliable, avoiding fragile long-held HTTP connections.
• Job status must be stored durably (a database), and webhook delivery needs the same retry/idempotency discipline as any other external call.

Things you should be able to explain

• The distinction between within-request async concurrency and deferred background processing.
• Why increasing timeout thresholds is not a durable fix for genuinely unpredictable-duration operations.

Things you should be able to build

• A job-creation-and-polling API with durable status storage, plus retry-safe, SSRF-mitigated webhook delivery.

Common mistakes

• Building long-running AI operations as synchronous requests.
• In-memory-only job status storage.
• Unvalidated client-provided webhook URLs (SSRF risk).

Recommended next chapter

07-queues-and-workers.md