18.15 — Message Queues and Async Architecture

Relationship to Part 7.6/7.7: Part 7.6 taught async processing patterns (background tasks, polling vs. webhooks) and Part 7.7 taught queue/worker architecture in depth, including the Celery acks_late gap. 18.4 already introduced SQS/SNS as AWS's managed implementation. This chapter completes the picture: synchronous vs. asynchronous processing as an architectural decision, message ordering, and a concrete, fully-worked AI document-ingestion pipeline tying 18.4, Part 7.7, and Part 3.4 together end to end.

1. What is it?

The architectural pattern of decoupling work from the request that triggers it — a producer places work on a queue and returns immediately; one or more independent consumers process it later, at their own pace — applied specifically to the Enterprise AI Assistant's document-ingestion pipeline (18.4's running example, completed here).

2. Why does it exist?

A synchronous request/response model requires the caller to wait for the entire operation to finish before getting a response — fine for a fast operation, actively harmful for a slow one (parsing and embedding a large document can take much longer than a reasonable HTTP request timeout, Part 1.1). Asynchronous, queue-based processing exists specifically to decouple "acknowledge the request was received" (fast) from "actually do the work" (potentially slow), without forcing the caller to hold an open connection the whole time.

3. What problem does it solve?

It solves "how do I accept a document-processing request instantly without making the user wait minutes for embedding to finish" and — just as important — "how do I smooth a bursty arrival pattern (100 documents uploaded at once) into a steady, sustainable processing rate" rather than needing to provision enough synchronous processing capacity for worst-case burst volume at all times.

4. How does it work internally?

Synchronous vs. asynchronous processing — the actual decision

SYNCHRONOUS is appropriate when:
  - The operation is fast (well within a reasonable request timeout)
  - The caller genuinely needs the result before it can proceed
  - Example: a chat request — the user is waiting for the answer

ASYNCHRONOUS (queue-based) is appropriate when:
  - The operation is slow, or its duration is unpredictable/unbounded
  - The caller doesn't need the result immediately (can poll, or be
    notified later via webhook, Part 10.1)
  - Volume can be bursty and benefits from being smoothed to a
    sustainable processing rate
  - Example: document ingestion — the uploader doesn't need to wait for
    embedding to complete before their upload is acknowledged

This is a genuine architectural decision, not a default — forcing everything through a queue adds real latency and complexity (a document's processing status now needs to be checked or pushed, Part 7.6's polling- vs-webhook trade-off) that a synchronous call wouldn't need; the decision should follow from the operation's actual latency/volume profile, not a blanket "async is more scalable so always use it" assumption.

Producers, consumers, and workers

A producer places a message on a queue (the API endpoint accepting a document upload, 18.4's S3-event-to-SNS-to-SQS pipeline). A consumer (often called a worker, when it runs as a standing, long-lived process polling the queue, Part 7.7) retrieves and processes messages. Multiple workers can consume from the same queue concurrently, each processing a different message — this is the actual mechanism behind smoothing burst volume: a spike of 500 uploaded documents queues up, and a fixed, smaller pool of workers processes them at a sustainable, predictable rate, rather than the system needing 500x its normal capacity instantaneously.

Message ordering

Most queues (SQS standard, Part 7.7's typical setup) provide no strict ordering guarantee across messages — two documents uploaded in sequence may be processed out of order. For document ingestion, this is usually fine (each document's processing is independent). It becomes a real concern specifically for updates to the same logical entity — e.g., a document re-uploaded twice in quick succession, where processing the older version after the newer one would leave stale data as the final state. The fix, when ordering genuinely matters for a specific message group, is a FIFO queue (SQS FIFO, 18.4) with a message group ID (all messages for the same document ID form one ordered group, while different documents' groups still process independently and concurrently) — a targeted fix for the specific entities needing order, not a blanket requirement across the whole pipeline.

Idempotency, dead-letter queues, and retries — restated at the architecture level

Part 7.7 and 18.4 already covered idempotent consumers and DLQs in depth at the implementation level; the architectural point worth adding here: every stage of a multi-stage pipeline needs this independently. The Enterprise AI Assistant's ingestion pipeline (18.4's diagram: parse → chunk → embed → write to vector DB) is not one atomic operation — a failure partway through (embedding succeeds, the vector DB write fails) needs its own retry/idempotency handling at that specific stage, not just an overall "redo the whole pipeline" retry, which would needlessly re-parse and re-embed (real LLM/embedding-API cost, Part 3.4/18.18) work that already succeeded.

Backpressure

Backpressure is what happens (and what should happen deliberately) when consumers can't keep up with producers — messages accumulate in the queue rather than being dropped or overwhelming a downstream system. This is precisely why introducing a queue between an ingestion API and its workers is itself a backpressure mechanism: without it, a burst of uploads would call the embedding API and vector database directly and synchronously, at whatever rate uploads arrive, with no smoothing at all — exactly the scenario 18.13's bottleneck-identification framework would flag as an avoidable, self-inflicted database/API overload.

5. Simple mental model

A queue is a restaurant's order ticket rail: the server (producer) hands in an order and immediately moves to the next table, rather than standing at the kitchen window waiting for the dish to be cooked; the kitchen (workers) cooks orders at its own sustainable pace, and a sudden rush of customers (a traffic burst) creates a longer ticket rail, not a kitchen meltdown — exactly the backpressure and burst-smoothing behavior section 4 describes.

6. Real-world example — the complete document-ingestion pipeline

Building fully on 18.4's diagram, now with every stage's own idempotency/retry handling made explicit:

1. Upload → S3 (18.4) → SNS → SQS "ingestion-queue"

2. Worker picks up message (document_id, s3_key, version)

3. STAGE: Parse
   - idempotency check: has this (document_id, version) already been
     parsed? (check a status table in RDS, 18.4)
   - on failure: message becomes visible again after the queue's
     visibility timeout (18.4) — retried independently of later stages

4. STAGE: Chunk + Embed (Part 3.4)
   - idempotency check: has embedding already completed for this
     (document_id, version)? If parsing succeeded but embedding failed
     on a PRIOR attempt, don't re-parse — resume from the embedding
     stage specifically, using the already-parsed, already-stored
     intermediate result
   - real cost implication (18.18): re-running a whole pipeline from
     scratch on every retry re-incurs embedding API cost unnecessarily

5. STAGE: Write to vector DB (Part 3.4)
   - idempotent write: upsert by a stable, content-derived vector ID
     (Part 7.7's idempotency-key pattern), so a retried write doesn't
     create duplicate vectors for the same chunk

6. Mark (document_id, version) as fully ingested in RDS — queryable
   status for the uploader (Part 7.6's polling/webhook completion signal)

7. Failure after N total attempts across the above → DLQ (18.4) →
   alerting (18.11) — a human-visible failure, not silent data loss

Stage-level idempotency (step 4's explicit point) is the detail that turns "our pipeline retries the whole thing on any failure, wasting embedding-API cost" into a genuinely efficient, resumable pipeline — a real, concrete improvement over a naive "just retry the whole message" approach.

7. Architecture diagram

Upload

S3

SNS

SQS: ingestion-queuevisibility timeout, DLQ after N attempts (18.4)

Worker pool (N replicas)ECS/EKS (18.3/18.9), scaled on QUEUE DEPTH via KEDA, not CPU

Stage: Parseidempotent, status in RDS

Stage: Embedidempotent, status in RDS

Stage: Write to vector DBidempotent upsert

RDS: ingestion_statusqueryable completion signal (Part 7.6)

8. Production considerations

• Scale the worker pool on queue depth (18.9's KEDA/Prometheus- Adapter point), not CPU — queue depth directly reflects backlog, which is the actual signal that more processing capacity is needed.
• Give each pipeline stage its own idempotency check, so a retry resumes from the failed stage rather than redoing already-successful, possibly-costly earlier stages.
• Configure a DLQ with alerting (18.11) so pipeline failures are visible and actionable, not silently retried forever or silently dropped.
• Use a FIFO queue with message-group IDs only for the specific entities where ordering genuinely matters (e.g., same-document re-uploads), not as a default for the whole pipeline, given FIFO's lower throughput ceiling relative to standard queues.

9. Common mistakes

• Retrying an entire multi-stage pipeline from scratch on any failure, re-incurring real embedding/LLM API cost for stages that already succeeded.
• Scaling worker replica count on CPU utilization instead of queue depth — CPU can look idle while a large backlog sits queued (18.1's I/O-bound CPU-signal point, restated here for a queue-consuming worker).
• Assuming message ordering across the whole queue when only specific same-entity updates actually need it — reaching for FIFO (with its throughput cost) unnecessarily broadly.
• No DLQ or no alerting on it, letting a specific document's ingestion fail silently and permanently with no one aware.

10. Security considerations

Message payloads may contain sensitive extracted content (Part 9.3's data-exposure concern, at the message-queue layer specifically) — encryption in transit and at rest for the queue (SQS supports both, 18.4) matters the same way it does for the underlying S3/RDS storage.

11. Performance considerations

Long-polling (18.4's cost point) also has a latency-smoothing benefit: workers waiting efficiently for new messages rather than tight-looping short polls reduces both unnecessary cost and unnecessary load on the queue service itself.

12. Cost considerations

Stage-level idempotency (section 6) is a direct, concrete cost lever specifically for AI pipelines — avoiding redundant embedding-API calls on retry is real, quantifiable savings (18.18) that a naive "just retry everything" pipeline design forfeits.

13. When to use it

Any AI workload whose processing time is long, unpredictable, or subject to bursty arrival volume — document ingestion, batch evaluation runs (Part 8.1, run as a CronJob per 18.9), and bulk re-embedding after a model upgrade are all natural fits.

14. When NOT to over-apply it

A fast, synchronous operation the caller genuinely needs an immediate result for (the chat endpoint itself) should stay synchronous — routing it through a queue would add latency and complexity for no benefit, directly contradicting section 4's actual decision criteria.

15. Alternatives and trade-offs

See 18.4's SQS/SNS-vs-Kafka-style-systems comparison — this chapter's document-ingestion pipeline is a textbook fit for SQS's simpler, sufficient model; a use case needing multiple independent consumers replaying the same event stream would be the concrete trigger to reconsider a Kafka-style system instead.

16. Practical example — stage-level idempotency in RDS

CREATE TABLE ingestion_status (
    document_id UUID NOT NULL,
    version     INT NOT NULL,
    stage       TEXT NOT NULL,   -- 'parsed', 'embedded', 'indexed'
    completed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (document_id, version, stage)
);

-- Before running a stage, check if it already completed:
SELECT 1 FROM ingestion_status
WHERE document_id = $1 AND version = $2 AND stage = 'embedded';
-- If found, SKIP straight to the next stage on retry.

17. Production-quality example — a resumable, stage-aware worker

"""
A worker that resumes from the correct stage on retry, rather than
redoing already-completed (and potentially costly) work — implementing
section 6's stage-level idempotency concretely.
"""
STAGES = ["parsed", "embedded", "indexed"]

async def process_document(document_id: str, version: int, s3_key: str) -> None:
    completed = await get_completed_stages(document_id, version)  # from RDS

    if "parsed" not in completed:
        parsed = await parse_document(s3_key)
        await mark_stage_complete(document_id, version, "parsed")
    else:
        parsed = await load_cached_parse_result(document_id, version)

    if "embedded" not in completed:
        chunks_with_embeddings = await embed_chunks(parsed)  # real API cost —
                                                                # skip if already done
        await mark_stage_complete(document_id, version, "embedded")
    else:
        chunks_with_embeddings = await load_cached_embeddings(document_id, version)

    if "indexed" not in completed:
        await upsert_to_vector_db(chunks_with_embeddings)  # idempotent upsert,
                                                              # stable vector IDs
        await mark_stage_complete(document_id, version, "indexed")

Checking completed stages before running each one is precisely what prevents an SQS redelivery (18.4's at-least-once guarantee) from re-incurring embedding-API cost for a document that already got past that stage on a prior attempt — a concrete, cost-relevant refinement beyond a purely message-level (not stage-level) idempotency check.

18. Short exercise

Design the message-group-ID scheme (section 4's FIFO discussion) for a scenario where the same document can be re-uploaded (a new version) while a previous version is still being processed — write out specifically what would go wrong with a standard (non-FIFO) queue in this exact scenario, and confirm your FIFO group-ID choice actually prevents it.

19. Interview questions

• When should an operation be synchronous versus queued, and what's the actual cost of routing something through a queue unnecessarily?
• Why does stage-level idempotency matter more for an AI pipeline specifically than for a typical CRUD background job?
• When do you actually need a FIFO queue, and what does it cost you relative to a standard queue?
• Why does scaling workers on CPU utilization often fail to reflect the actual backlog in a queue-based AI pipeline?

20. FDE/customer scenario

A customer says: "We need to process 100,000 documents, and we're worried about cost if something fails partway through and has to restart." A strong response directly addresses stage-level idempotency (section 6) as the concrete answer — a well-designed pipeline resumes from the last successfully-completed stage per document, rather than re-running (and re-paying for) already-successful embedding work on every retry, a specific, cost-relevant design decision rather than a vague assurance that "we handle failures."

---

Key takeaways

• Synchronous vs. asynchronous is a genuine architectural decision driven by operation latency/volume, not a default — forcing fast, immediately- needed operations through a queue adds unnecessary latency and complexity.
• A multi-stage AI pipeline needs idempotency at EACH stage, not just at the overall message level — otherwise a retry re-incurs real, avoidable embedding/LLM API cost for already-completed work.
• Worker pools should scale on queue depth, not CPU — CPU can look idle while a real backlog accumulates, for the same I/O-bound reasons 18.1 and 18.13 already established.

Things you should be able to explain

• The actual decision criteria for synchronous vs. asynchronous processing.
• Why stage-level idempotency matters specifically for cost in AI pipelines.
• When a FIFO queue is actually needed versus a default, throughput- costly overreach.

Things you should be able to build

• A resumable, stage-aware document-ingestion worker.
• A queue-depth-based (not CPU-based) autoscaling configuration for a worker pool.

Common mistakes

• Retrying entire pipelines from scratch instead of resuming from the failed stage.
• Scaling on CPU instead of queue depth for a queue-consuming worker pool.
• Reaching for FIFO ordering guarantees broadly when only specific same-entity updates actually need them.

Recommended next chapter

16-ai-infrastructure-deep-dive.md