18.4 — AWS Data and Messaging

Running example: from this chapter onward, the track builds one evolving system — the Enterprise AI Assistant: a FastAPI + LangGraph RAG/agent service, backed by PostgreSQL (application data + LangGraph checkpoints, Part 5.3), Redis (caching, Part 7.8/18.14), a vector database (Part 3.4), S3 (document storage), and SQS (async ingestion, Part 7.7/18.15) — fronted by Route 53 + an ALB, running on ECS/EKS (18.3), calling out to an external LLM provider, observed via CloudWatch and LangSmith (Part 6). Each chapter adds one more piece of this picture. This chapter adds the data and messaging layer.

1. What is it?

The AWS managed services that store and move an AI application's data and work: S3 (object storage), RDS (managed relational databases), ElastiCache (managed Redis/Memcached), SQS (queues), and SNS (pub/sub notifications) — taught as the AWS-native implementation of concepts Part 1.4–1.6, 7.7, and 7.8 already covered from first principles.

2. Why does it exist?

18.3 answered "where does my code run." This chapter answers "where does my data live, and how does work move between components without every piece needing to call every other piece directly" — the two questions that, together with networking (18.2/18.5), fully describe an AI system's runtime shape.

3. What problem does it solve?

Running your own PostgreSQL server, your own Redis instance, and your own message broker on raw EC2 is possible but means you now own patching, backups, failover, and scaling for each — operational work with no direct AI-engineering value. Managed services (RDS, ElastiCache, SQS/SNS) trade a service premium (18.18) for AWS handling that operational burden, letting an AI FDE spend effort on the application instead of database administration — the same "buy vs. build" trade-off Part 7's chapters raised for CI/CD and observability, applied here to data infrastructure.

4. How does it work internally?

S3 — object storage

S3 stores objects (files) in buckets, addressed by key, with effectively unlimited capacity and very high durability (AWS states multiple nines of durability for standard storage classes — verify the current published figure before quoting it to a customer). For the Enterprise AI Assistant, S3 is where source documents land before ingestion (Part 10.2's data pipeline, 18.15's queue-triggered processing) and where any large generated artifacts (a report, an export) are stored rather than returned inline in an API response. S3 is not a database — no query language, no transactions across objects — it's the right tool for whole-file storage, not for the structured, queryable data RDS handles.

S3 event notifications are the specific mechanism that turns "a file was uploaded" into "a Lambda function runs" or "a message lands on an SQS queue" — the trigger that starts the document-ingestion pipeline in section 6 below and in 18.15, without your application needing to poll S3 for new files.

RDS — managed relational databases

RDS runs PostgreSQL (or MySQL, etc.) as a managed service: AWS handles patching, automated backups, and (with Multi-AZ enabled) synchronous replication to a standby in a different availability zone for automatic failover. For the Enterprise AI Assistant, RDS Postgres holds both ordinary application data (users, conversation metadata, Part 1.4/1.5) and LangGraph's checkpoint state via PostgresSaver (Part 5.3) — meaning RDS's availability characteristics directly determine whether an in-progress, possibly-paused agent workflow (Part 5.4's human-in-the-loop interrupts) can resume correctly after an infrastructure event, not just whether ordinary CRUD queries succeed.

RDS vs. DynamoDB: RDS gives you a relational model, joins, and transactions, at the cost of a more constrained, harder-to-horizontally- scale write path (vertical scaling and read replicas, mainly). DynamoDB is a managed NoSQL key-value/document store that scales writes horizontally far more easily, at the cost of losing joins and relational query flexibility, and requiring your access patterns to be designed up front around its key structure. For the Enterprise AI Assistant's structured, relational application data and LangGraph's checkpoint model (which Part 5.3 already assumes a relational or key-value store, with PostgresSaver as the production-recommended default), RDS is the natural fit; DynamoDB becomes attractive for very high-write-throughput, simple-access-pattern data (e.g., raw event/audit logs at large scale) where its scaling characteristics matter more than relational flexibility.

ElastiCache — managed Redis

ElastiCache runs Redis (or Memcached) as a managed service — this is the AWS-hosted version of everything Part 7.8 and 18.14 (which goes deeper) teach about Redis's role in an AI system: LLM response caching, rate limiting, session state, and job coordination. The AWS-specific layer this chapter adds: ElastiCache handles patching and offers replication groups with automatic failover (a primary node plus replicas, promoted automatically if the primary fails) — the managed-service answer to "what happens if my cache goes down" that 18.12's reliability chapter revisits from the application-design side (never treating cache availability as a hard dependency for correctness, only for performance).

SQS — Simple Queue Service

SQS is a managed, fully-buffered message queue: a producer sends a message, it's durably stored, and a consumer polls for and processes it, deleting it only after successful processing. This is AWS's implementation of the queue/worker pattern Part 7.7 taught in depth — SQS specifically gives at-least-once delivery by default (a message can be delivered more than once, most commonly if a consumer takes longer than the message's visibility timeout to finish processing it, causing SQS to assume the consumer failed and redeliver it to another consumer) — which is exactly why Part 7.7's idempotency argument is not optional decoration, it is the concrete, required consequence of SQS's actual delivery guarantee. A dead-letter queue (DLQ) is a second queue SQS automatically routes a message to after it fails processing (is received but not deleted) more than a configured number of times — preventing one poison-pill message (a malformed document, say) from blocking or endlessly reprocessing in the main queue.

SNS — Simple Notification Service

SNS is publish/subscribe: a single message published to an SNS topic can fan out to multiple subscribers (one or more SQS queues, a Lambda function, an email address, an HTTP endpoint) — useful when one event (a document finished processing) needs to trigger several independent downstream actions (update a search index, notify a user, log an audit event) without the publisher needing to know about each consumer individually. The common combined pattern is SNS fan-out to SQS: SNS distributes one event to multiple durable SQS queues, each consumed independently at its own pace — giving both fan-out (SNS's strength) and durable, retryable, at-least-once processing (SQS's strength) together.

SQS/SNS vs. Kafka-style systems

SQS/SNS are managed, low-operational-overhead, and model messages as discrete units consumed and removed. A Kafka-style system (or AWS's own managed Kafka, MSK) is a durable, ordered, replayable log: consumers read from an offset and can re-read historical messages, multiple independent consumer groups can each process the full stream at their own pace without affecting each other, and message retention is time/size-based rather than "until consumed." For the Enterprise AI Assistant's document-ingestion pipeline (a straightforward "process this file once" workload), SQS is the simpler, sufficient, lower-overhead choice; Kafka-style streaming becomes worth its added complexity when you need multiple independent consumers replaying the same event stream (e.g., both a real-time indexing pipeline and a separate analytics pipeline consuming the same document- upload events independently) or need strict, replayable ordering at scale — a genuinely different requirement shape, not simply "Kafka is more powerful so use it."

5. Simple mental model

S3 is a warehouse for boxes (files) — you can store and retrieve a whole box, but you can't ask it "which boxes contain a red widget." RDS is a filing cabinet with a strict, well-organized structure you can query precisely. ElastiCache is a sticky-note board next to your desk — fast to check, but you'd never store your only copy of something important there (18.14 makes this explicit). SQS is a to-do list where a task doesn't disappear until someone explicitly checks it off — if the person doing it disappears mid-task, the task reappears for someone else. SNS is a loudspeaker announcement that any number of separately interested parties can hear and act on independently.

6. Real-world example — document ingestion, end to end

1. A user (or an integration, Part 10.1's webhook pattern) uploads a
   policy document → S3 (bucket: acme-docs-prod, prefix: /ingest/)

2. The S3 PUT event triggers an SNS topic ("document-uploaded")

3. SNS fans out to two SQS queues:
     - "ingestion-queue"   → triggers parsing/chunking/embedding
     - "audit-log-queue"   → triggers a compliance audit-log entry
   (each consumed independently, at its own pace, per section 4)

4. A worker (Part 7.7's worker pattern, running on ECS, 18.3) polls
   "ingestion-queue", parses the document, chunks it (Part 3.4),
   generates embeddings, and writes vectors to the vector database

5. If parsing fails repeatably (a corrupted PDF, say), SQS's
   redrive policy moves the message to a dead-letter queue after N
   attempts — a human/alerting workflow (18.11) picks it up from
   there rather than the ingestion pipeline retrying it forever

6. Application metadata (which document, which version, embedding
   status) is written to RDS Postgres — queryable, relational,
   transactional

Every step here is exactly the kind of "how does data actually move through this system" question a customer's technical stakeholder asks in a real architecture review (18.23).

7. Architecture diagram

Upload

S3

SNSfan-out

SQS: ingestDLQ after N failures

SQS: audit

Worker (ECS)

Audit worker

Vector DBembeddings

RDSmetadata

ElastiCachehot-path cache

8. Production considerations

• Enable RDS Multi-AZ for production — a single-AZ RDS instance is a single point of failure for both application data and LangGraph checkpoint state (Part 5.3), directly affecting whether paused agent workflows can resume after an availability-zone-level event.
• Configure SQS visibility timeout to comfortably exceed your worker's realistic processing time (with margin) — set too low, a slow-but- succeeding worker gets its message redelivered to a second worker mid- processing, causing duplicate work exactly the way Part 7.7 warns about.
• Always configure a DLQ for a production queue — without one, a poison-pill message either blocks the queue or gets silently retried forever, consuming worker capacity with no forward progress.
• Use S3 lifecycle policies to transition old, rarely-accessed documents to cheaper storage classes (18.18) rather than leaving everything on standard storage indefinitely.

9. Common mistakes

• Treating SQS's at-least-once delivery as "basically exactly-once in practice" and skipping idempotency in the consumer — this fails exactly when it matters most, under load or partial failure (Part 7.7's core argument, restated here at the AWS-service level).
• Storing large binary content (documents, images) directly in RDS rows instead of S3 with a reference stored in RDS — bloats the database, slows backups, and is generally the wrong tool for whole-file storage.
• Never configuring a DLQ, discovering the gap only when a malformed message causes unbounded reprocessing in production.
• Assuming ElastiCache's replication/failover means it's safe to treat as a durable store — a promoted replica does not guarantee zero data loss for in-flight writes, and Redis here should still be treated as a cache, not a system of record (18.14 elaborates).

10. Security considerations

• S3 buckets storing customer documents must not be public — bucket policies and IAM (18.3) should scope access to the specific roles that need it, and S3 Block Public Access should be enabled account-wide unless a specific, reviewed exception exists.
• RDS and ElastiCache should live in private subnets (18.2/18.5) with security groups scoped to the specific application tier's security group — never a broad CIDR range, and never publicly accessible.
• Encryption at rest (RDS, ElastiCache, S3 all support this) and in transit (TLS to each) should be enabled by default for any system handling customer data — a frequent, specific compliance requirement (Part 10.5, 18.19).
• SQS/SNS messages can carry sensitive data (a document's extracted text, say) — apply the same data-classification thinking from Part 9.6 to message payloads, not just to database rows.

11. Performance considerations

• RDS read replicas offload read-heavy query load (a common pattern for an AI application logging heavy read-side analytics or a dashboard) from the primary, which continues to handle writes.
• SQS has no built-in strict ordering by default (standard queues) — processing order across messages is not guaranteed; if your ingestion pipeline requires strict per-document ordering (rare, but possible for versioned-document updates), a FIFO queue trades some throughput for ordering guarantees — a genuine, workload-dependent trade-off.
• ElastiCache's in-memory design gives it latency far below RDS for cache-appropriate reads (Part 7.8's caching rationale) — but only for data that's actually safe to lose or recompute, not a substitute for RDS's durability guarantees.

12. Cost considerations

• RDS Multi-AZ roughly doubles compute cost relative to single-AZ (a standby instance is provisioned) — a concrete, real trade-off between availability and cost, not a free upgrade, revisited with numbers in 18.18.
• S3 storage is cheap per GB but has real costs for requests (especially many small objects) and for data transfer out to the internet — a document-heavy AI system processing large volumes should account for both, not just raw storage GB.
• SQS pricing is per-request — a very chatty polling pattern (short poll intervals, many empty receives) costs more than necessary; long polling (ReceiveMessage with a wait time) reduces empty-response requests and is the recommended default for cost and efficiency both.
• ElastiCache and RDS both bill for provisioned capacity whether or not it's fully utilized — right-sizing (18.1's utilization-first principle, applied to managed services) is a genuine, recurring cost lever.

13. When to use it

Any production AI system needs at least object storage (S3) and a relational store (RDS) for its structured data; most benefit from a cache (ElastiCache) and a queue (SQS) once there's any asynchronous work at all (document ingestion, background evaluation jobs, Part 7.6).

14. When NOT to over-apply it

A local prototype (Part 13.2) should use local Postgres/Redis (or even SQLite/in-memory structures) rather than provisioning RDS/ElastiCache before the design is validated — matching infrastructure investment to the stage of the engagement (Part 13.1).

15. Alternatives and trade-offs

See section 4's RDS-vs-DynamoDB and SQS/SNS-vs-Kafka comparisons — both are genuine, workload-shape-dependent trade-offs revisited from the system-design angle in Part 11 and 18.22.

16. Practical example — S3 event to SQS via SNS, in Terraform (previewed; full Terraform in 18.6)

resource "aws_sns_topic" "document_uploaded" {
  name = "document-uploaded"
}

resource "aws_sqs_queue" "ingestion_dlq" {
  name = "ingestion-dlq"
}

resource "aws_sqs_queue" "ingestion_queue" {
  name                       = "ingestion-queue"
  visibility_timeout_seconds = 300   # must exceed realistic worker processing time
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.ingestion_dlq.arn
    maxReceiveCount      = 5
  })
}

resource "aws_sns_topic_subscription" "ingestion" {
  topic_arn = aws_sns_topic.document_uploaded.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.ingestion_queue.arn
}

17. Production-quality example — an idempotent SQS worker

"""
Idempotent SQS consumer for the document-ingestion pipeline. Idempotency
matters because SQS guarantees at-least-once delivery (section 4) — this
worker must produce the same end state whether it processes a given
message once or three times.
"""
import boto3
import hashlib
from myapp.db import get_ingestion_status, mark_ingested

sqs = boto3.client("sqs")
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/ingestion-queue"


def process_message(body: dict) -> None:
    document_id = body["document_id"]
    # Idempotency key: derived from immutable message content, not a
    # random ID generated per-attempt, so a redelivered message maps to
    # the SAME key every time.
    idempotency_key = hashlib.sha256(
        f"{document_id}:{body['version']}".encode()
    ).hexdigest()

    if get_ingestion_status(idempotency_key) == "completed":
        return  # already processed — safe no-op on redelivery

    # ... parse, chunk, embed, write to vector DB ...
    mark_ingested(idempotency_key, document_id)


def poll_loop() -> None:
    while True:
        response = sqs.receive_message(
            QueueUrl=QUEUE_URL,
            MaxNumberOfMessages=5,
            WaitTimeSeconds=20,          # long polling — section 12's cost point
            VisibilityTimeout=300,        # must match/exceed queue's own setting
        )
        for message in response.get("Messages", []):
            try:
                process_message(json.loads(message["Body"]))
                sqs.delete_message(
                    QueueUrl=QUEUE_URL, ReceiptHandle=message["ReceiptHandle"]
                )
            except Exception:
                # Do NOT delete — let it become visible again for retry,
                # or route to the DLQ after maxReceiveCount (section 16).
                logger.exception("Failed processing message %s", message["MessageId"])

The idempotency key is derived from the message's own immutable content (document ID + version), not a per-attempt random value — this is the detail that actually makes redelivery safe; an idempotency key generated fresh on each attempt would defeat the entire mechanism.

18. Short exercise

For the Enterprise AI Assistant's document-ingestion pipeline (section 6), write out what would go wrong if the SQS visibility timeout were set to 10 seconds while the actual worker takes 45 seconds to embed a large document — trace the exact sequence of duplicate processing that results, using section 4's delivery-guarantee explanation.

19. Interview questions

• Why does SQS require idempotent consumers, and what's a concrete example of what breaks if a consumer isn't idempotent?
• When would you choose DynamoDB over RDS for an AI system's data, and when would that be the wrong choice?
• Walk through the SNS-fan-out-to-SQS pattern and why you'd use it instead of publishing directly to multiple queues from the producer.
• What's the difference between a queue's visibility timeout and a dead-letter queue's max-receive-count, and how do they interact?

20. FDE/customer scenario

A customer says: "We need to process 100,000 documents for our knowledge base, and we need to know if any fail." A strong response reasons through this chapter's pattern explicitly: S3 for storage, an SQS-based (or SNS-fan-out, if multiple downstream systems need the event) ingestion pipeline with a DLQ specifically so failures are visible and actionable rather than silently dropped or retried forever, with RDS holding ingestion status queryable for a progress dashboard — a concrete architecture answer grounded in this chapter's actual mechanisms, not a generic "we'll use a queue" reassurance.

---

Key takeaways

• S3/RDS/ElastiCache/SQS/SNS are the managed-service implementations of concepts Part 1 and Part 7 already taught from first principles — the AWS-specific value-add is offloaded operational burden, at a real cost.
• SQS's at-least-once delivery guarantee makes consumer idempotency mandatory, not optional — this is a direct, mechanical consequence of how the service actually works, not a general best practice suggestion.
• RDS Multi-AZ availability directly affects whether paused, checkpointed LangGraph workflows (Part 5.3/5.4) can resume correctly after an infrastructure event — data-layer availability decisions have AI- workflow-correctness consequences, not just uptime consequences.

Things you should be able to explain

• RDS vs. DynamoDB, and SQS/SNS vs. Kafka-style systems, as genuine workload-shape trade-offs.
• Why SQS visibility timeout misconfiguration causes duplicate processing.
• The role of a dead-letter queue and why production queues need one.

Things you should be able to build

• An idempotent SQS consumer using a content-derived idempotency key.
• An S3-event-to-SNS-to-SQS fan-out pipeline with a configured DLQ.

Common mistakes

• Non-idempotent SQS consumers, assuming at-least-once is "basically once."
• Storing large binary documents in RDS rows instead of S3.
• Skipping a DLQ and discovering the gap during a production incident.

Recommended next chapter

05-aws-networking-and-vpc-architecture.md