18.5 — AWS Networking and VPC Architecture

1. What is it?

The concrete AWS implementation of 18.2's networking concepts: how a VPC, subnets, route tables, an Internet Gateway, a NAT Gateway, and an Application Load Balancer combine into the actual network the Enterprise AI Assistant runs inside — plus the AWS services (Route 53, CloudFront, CloudWatch, Secrets Manager, KMS, CloudTrail) that complete the picture around it.

2. Why does it exist?

18.2 taught the concepts in the abstract; 18.3/18.4 placed compute and data inside that network without yet drawing the network itself. This chapter draws it — the actual VPC design the rest of Part 18 assumes from here forward.

3. What problem does it solve?

It solves "here is the complete, concrete network design for a production AI system on AWS" — the artifact you'd actually hand to (or build with) a customer's network/security team, and the one nearly every AI FDE system-design conversation eventually needs to produce.

4. How does it work internally?

Building the VPC, subnet by subnet

Following 18.2's CIDR design exactly, for the Enterprise AI Assistant:

VPC: 10.0.0.0/16

Public subnets (route to Internet Gateway):
  10.0.1.0/24   (AZ us-east-1a)  — ALB, NAT Gateway
  10.0.2.0/24   (AZ us-east-1b)  — ALB, NAT Gateway

Private subnets — application tier (route to NAT Gateway, outbound only):
  10.0.11.0/24  (AZ us-east-1a)  — ECS/EKS tasks
  10.0.12.0/24  (AZ us-east-1b)  — ECS/EKS tasks

Private subnets — data tier (NO route to NAT Gateway at all):
  10.0.21.0/24  (AZ us-east-1a)  — RDS, ElastiCache
  10.0.22.0/24  (AZ us-east-1b)  — RDS, ElastiCache

Two availability zones (AZs), not one, is the deliberate choice here — an AZ is an AWS data center (or small cluster of them) with independent power/cooling/networking from other AZs in the same region; spanning two means the loss of one AZ doesn't take down the whole system, the concrete AWS mechanism behind Part 7.4/18.9's "don't run a single replica" and 18.12's redundancy principle. This requires one NAT Gateway per AZ (not sharing a single one across AZs) for genuine AZ-independence — the resilience benefit and the doubled NAT cost (18.2's cost point, revisited concretely in 18.18) are directly linked; a single shared NAT Gateway would reintroduce a cross-AZ single point of failure for all outbound traffic, silently undermining the multi-AZ design's whole purpose.

VPC Endpoints — bypassing the NAT Gateway for AWS-service traffic

A VPC Endpoint lets a private-subnet resource reach a specific AWS service directly, over AWS's own internal network, without that traffic ever traversing the NAT Gateway (or the public internet at all). Two kinds matter in practice: a Gateway endpoint (S3 and DynamoDB only, no additional hourly charge) adds a route-table entry that sends traffic for that specific service directly to it; an Interface endpoint (most other AWS services — Secrets Manager, ECR, CloudWatch Logs, SQS, and dozens more, backed by a PrivateLink elastic network interface in your subnet, billed hourly plus per-GB) does the same for services a Gateway endpoint doesn't cover. For the Enterprise AI Assistant, the ECS tasks in the private application subnets routinely call S3 (document uploads/ downloads, 18.4), Secrets Manager (18.3's task role fetching the LLM API key), and ECR (pulling the container image) — every one of these calls, absent a VPC endpoint, exits through the NAT Gateway (incurring NAT's per-GB data-processing charge, section 12) and traverses AWS's public-facing network path even though both ends are AWS services. Adding an S3 Gateway endpoint and Secrets Manager/ECR Interface endpoints removes this traffic from the NAT Gateway's cost and path entirely — a genuine, often-overlooked cost lever precisely because NAT Gateway cost (section 12) is usually attributed to LLM-provider/external-API calls, when a meaningful share of it can actually be AWS-service traffic a VPC Endpoint would have avoided at lower cost and one fewer network hop. It's also a security improvement independent of cost: traffic to Secrets Manager for the LLM API key never needs to leave AWS's private network at all, closing off a class of exposure a NAT Gateway path (however unlikely to actually be intercepted) doesn't need to exist in the first place.

Security Groups vs. NACLs — stateful vs. stateless, and when the second one earns its complexity

18.2 introduced the mechanical distinction: a security group is stateful — allow inbound traffic on a port, and the matching outbound return traffic is automatically permitted, with no separate outbound rule needed — and attaches to individual resources (an ALB, an ECS task's ENI, an RDS instance). A network ACL (NACL) is stateless — an allowed inbound request's return traffic is NOT automatically permitted; a matching outbound rule must be configured explicitly, or return traffic is silently dropped — and attaches to a subnet as a whole, applying to every resource in it regardless of that resource's own security group. This is precisely why NACL misconfiguration is a distinctive debugging trap: an inbound rule that looks correct can still fail silently if the corresponding outbound rule for return traffic was never added, a mistake a security group's stateful behavior makes structurally impossible.

In practice, security groups do essentially all of the Enterprise AI Assistant's real access control (the app-tier-only, db-tier-only patterns throughout this chapter) — NACLs add genuine value in a narrower set of cases: a subnet-wide explicit deny against a specific, known-bad CIDR range (blocking a range identified in a threat feed or a prior incident, at the subnet boundary, before traffic ever reaches any individual resource's security group) is the clearest case where a NACL does something a security group cannot, since security groups support allow rules only, with no explicit-deny capability of their own. Reaching for NACLs as a general-purpose, per-resource access-control layer duplicating what security groups already do adds real operational complexity (two rule sets to keep synchronized, a stateless model that's easy to misconfigure) for little or no additional security benefit in a typical single-account, non-adversarial-insider architecture — a judgment call worth making deliberately (add a NACL rule for the one, specific known-bad-range case; don't rebuild the whole security-group ruleset a second time at the NACL layer for its own sake).

Route tables — what actually makes a subnet "public" or "private"

A subnet is public or private only because of what its associated route table says, not because of any label on the subnet itself. A public subnet's route table has a route for 0.0.0.0/0 (all internet-bound traffic) pointing at the Internet Gateway; a private subnet's route table instead points 0.0.0.0/0 at a NAT Gateway (application tier) or has no 0.0.0.0/0 route at all (data tier, per section 4 above). This is a common source of confusion worth being precise about: there is no aws_subnet "type" field — "public" and "private" are purely a consequence of route table association, which is exactly why a misconfigured route table can silently turn an intended-private subnet public (or vice versa) with no obvious warning anywhere in the console.

Application Load Balancer (ALB)

The ALB (18.2's concrete AWS implementation of a load balancer) sits in the public subnets, terminates TLS (a certificate from AWS Certificate Manager, renewed automatically), performs health checks against your application's /health endpoint (Part 7.1's health-check pattern), and routes traffic to targets (ECS tasks or EKS pods) in the private application subnets. An ALB operates at the HTTP layer (Layer 7) — it can route based on path or host header (useful for routing /api/* to one service and /admin/* to another within the same system), unlike a purely TCP-level load balancer.

ALB idle timeout and streaming responses. An ALB has a default idle-timeout setting for a connection with no data flowing (historically 60 seconds — verify the exact current default against AWS documentation before relying on it) — after which the ALB closes the connection, regardless of whether the backend is still actively working on it. This matters directly for the Enterprise AI Assistant's streaming chat responses (Part 7.5's SSE/token-by-token streaming): a slow-generating LLM response that goes quiet for longer than the idle timeout (a long tool call mid-agent-run, a large context prompt taking longer than usual to produce a first token) can have its connection killed by the ALB mid-stream, even though the backend would have eventually delivered a complete response — a failure mode that looks like a flaky backend but is actually a load-balancer configuration mismatch. The fix is to explicitly raise the ALB's idle-timeout attribute above your expected maximum streaming duration (with headroom), not to leave it at a default tuned for typical short request/response HTTP traffic — and, where your streaming protocol supports it, to periodically send some data (a keep-alive/heartbeat event over SSE) so a connection isn't relying on timeout margin alone during an unusually long silent gap.

Route 53 — DNS

Route 53 resolves api.acme-assistant.com to the ALB's address. Beyond basic resolution, Route 53 supports health-check-based failover routing (useful for a multi-region disaster-recovery design, 18.20) and latency- based routing (directing users to whichever region responds fastest) — capabilities that become relevant once the Enterprise AI Assistant grows beyond a single region, a scenario 18.20/18.22 return to.

CloudFront — CDN

CloudFront caches and serves content from edge locations close to users — relevant to the Enterprise AI Assistant primarily for its frontend static assets (a chat UI's JS/CSS bundle) rather than for the AI API responses themselves, which are per-user and typically not cacheable at the CDN layer (though see 18.14 for a distinct, different discussion of caching LLM responses specifically, which is an application-layer concern, not a CDN one).

CloudWatch — the AWS-native observability layer

CloudWatch collects metrics (CPU, memory, request count, custom application metrics), logs (via the awslogs driver, Part 7.1's logging configuration, or the CloudWatch agent), and can trigger alarms (feeding Auto Scaling, 18.3, or paging an on-call engineer, 18.11). This is AWS's own layer beneath the AI-specific observability tooling (LangSmith, Part 6) — CloudWatch tells you the ECS task is healthy and serving requests; LangSmith tells you whether the agent's reasoning inside those requests is actually correct (18.11 draws this distinction in full).

Secrets Manager and KMS

Secrets Manager stores secrets (database credentials, the LLM provider's API key) and can rotate database credentials automatically on a schedule — the concrete mechanism behind 18.3's ECS task definition example, which referenced a Secrets Manager ARN rather than a plaintext environment variable. KMS (Key Management Service) manages the encryption keys used to encrypt data at rest across nearly every AWS service (S3, RDS, ElastiCache, Secrets Manager itself) — you don't usually handle raw key material directly; you reference a KMS key, and AWS services use it to encrypt/decrypt transparently, with IAM (18.3) controlling who can use which key for what.

CloudTrail — audit logging

CloudTrail records every API call made against your AWS account — who (which IAM identity) did what (which API action) to which resource, when. This is the concrete AWS answer to a very common, very literal customer requirement (18.23's "we need audit logs for every AI action") — though it's worth being precise about scope: CloudTrail logs infrastructure API calls (someone changed a security group, an IAM policy was modified), not your application's business-logic actions (a specific user asked the AI assistant a specific question) — that second kind of audit log is an application-level concern (Part 9.4/10.5), and conflating the two is a common, consequential misunderstanding in a compliance conversation.

5. Simple mental model

If 18.2's mental model was "a VPC is a building," this chapter is the actual floor plan: which rooms have street doors (public subnets), which don't (private subnets), which mail-forwarding service each floor uses (NAT Gateway per AZ), and the building directory service (Route 53) that tells visitors which door to use.

6. Real-world example — the full request path, concretely

User → Route 53 (api.acme-assistant.com)
     → ALB (public subnets, TLS termination, health-checks targets)
     → ECS task (private app subnet, security group: ALB only)
     → RDS / ElastiCache (private data subnet, security group: app only)
     → NAT Gateway (private app subnet's outbound route)
     → External LLM Provider API
     ← response flows back through the same path

CloudWatch collects metrics/logs at every AWS-managed hop (ALB, ECS, RDS); LangSmith (Part 6) additionally traces what happened inside the application's own LangGraph execution — two complementary, non-overlapping observability layers, revisited fully in 18.11.

7. Architecture diagram

Route 53DNS

AZ-a · public 10.0.1.0/24

ALB

NAT Gateway

AZ-b · public 10.0.2.0/24

ALB

NAT Gateway

AZ-a · app (private) 10.0.11.0/24

ECS/EKS tasks

AZ-b · app (private) 10.0.12.0/24

ECS/EKS tasks

AZ-a · data (private, no NAT) 10.0.21.0/24

RDS primary

ElastiCache

AZ-b · data (private, no NAT) 10.0.22.0/24

RDS standbyMulti-AZ

ElastiCache replica

VPC 10.0.0.0/16 — two AZs, public/app/data tiers, one NAT Gateway per AZ.

CloudWatch (metrics/logs) · CloudTrail (API audit) · KMS (encryption) · Secrets Manager (credentials) — cross-cutting, apply account-wide rather than to any single tier above.

8. Production considerations

• Deploy across at least two AZs, with one NAT Gateway per AZ — the concrete implementation of Part 7's redundancy principle at the network layer, and the most common thing a first-pass architecture gets wrong by sharing a single NAT Gateway to save cost.
• Route table review should be part of any infrastructure change review — since "public" vs. "private" is route-table-derived (section 4), a seemingly small route table edit can silently expose a data-tier subnet.
• CloudTrail should be enabled account-wide, writing to a separate, access-restricted S3 bucket (ideally in a separate, dedicated logging account for a mature setup) — so audit logs survive even a compromise of the account being audited.

9. Common mistakes

• Sharing one NAT Gateway across multiple AZs "to save cost," quietly reintroducing the single point of failure the multi-AZ design was meant to eliminate.
• Confusing CloudTrail (infrastructure API audit) with an application- level audit log of user/AI actions — answering a customer's audit-log requirement with only one when they actually need both.
• Assuming a subnet is private because it has no aws_eip (Elastic IP) attached to instances in it, rather than actually checking its route table — the EIP is unrelated to whether the route exists.
• Placing the ALB's target health check on a path that itself depends on RDS/ElastiCache being healthy — a data-tier blip can then cascade into the load balancer marking every application instance unhealthy simultaneously, which is usually not the intended behavior (a /health endpoint checking only "is the process alive and can it serve requests" is usually the safer default, with a separate, deeper readiness check for orchestration-level decisions — Part 7.4/18.9 revisit this liveness-vs-readiness distinction precisely).

10. Security considerations

This chapter operationalizes 18.2's security posture concretely on AWS: private subnets for app and data tiers, security groups referenced by ID (18.2's pattern), Secrets Manager instead of plaintext credentials, KMS encryption at rest, and CloudTrail for infrastructure audit — together forming the baseline a customer's security review (18.19/18.23) will check for specifically, item by item.

11. Performance considerations

• ALB health-check interval/threshold tuning (18.2's point) matters more once you have multiple AZs — an overly aggressive health check can flap targets across AZ boundaries in a way that's harder to diagnose than a single-AZ setup.
• Cross-AZ data transfer (e.g., an ECS task in AZ-a calling RDS in AZ-b) has both a latency and a cost implication (section 12) — a well-designed system generally prefers same-AZ paths where the architecture allows it, while still maintaining genuine multi-AZ redundancy for failover.

12. Cost considerations

NAT Gateway (one per AZ, section 4) is often the single largest, most underestimated fixed network cost in a multi-AZ AI architecture — concretely quantified in 18.18. Cross-AZ data transfer is billed and adds up for chatty, high-volume internal traffic (frequent RDS/ElastiCache calls from an application tier in a different AZ) — a real, if usually secondary, cost lever.

13. When to use it

Every production AWS-hosted AI system needs this VPC design as its baseline network foundation — from here, Part 18's remaining chapters (Terraform in 18.6 onward) build compute and orchestration inside this already-established network shape.

14. When NOT to over-apply it

A quick prototype or an internal proof-of-concept (Part 13.2) can reasonably run in a single AZ, in a simpler network (even AWS's account default VPC) — multi-AZ, fully-private-tiered network design is a production-readiness investment, not a prototyping requirement.

15. Alternatives and trade-offs

A simpler single-public-subnet design (no NAT Gateway, application instances directly in a public subnet with restrictive security groups) is cheaper and simpler but leaves the application tier one security-group misconfiguration away from direct internet exposure — a real, if increasingly discouraged, trade-off some smaller or cost-constrained deployments still make deliberately, with eyes open about the risk.

16. Practical example — reasoning through a route table

Given this route table, is 10.0.11.0/24 public or private?

Destination         Target
10.0.0.0/16          local
0.0.0.0/0            nat-0abc123 (NAT Gateway)

Answer: PRIVATE. The 0.0.0.0/0 route points at a NAT Gateway, not an
Internet Gateway — inbound connections from the internet cannot reach
resources here directly (no route back from IGW to this subnet's
resources); only outbound traffic can leave, via NAT.

17. Production-quality example — the full VPC in AWS CLI (illustrative; Terraform is the real deployment tool, 18.6)

# Illustrative only — see 18.6 for the actual Terraform this maps to.
aws ec2 create-vpc --cidr-block 10.0.0.0/16 --tag-specifications \
  'ResourceType=vpc,Tags=[{Key=Name,Value=ai-assistant-vpc}]'

aws ec2 create-subnet --vpc-id vpc-0abc --cidr-block 10.0.1.0/24 \
  --availability-zone us-east-1a --tag-specifications \
  'ResourceType=subnet,Tags=[{Key=Name,Value=public-a},{Key=Tier,Value=public}]'

aws ec2 create-subnet --vpc-id vpc-0abc --cidr-block 10.0.11.0/24 \
  --availability-zone us-east-1a --tag-specifications \
  'ResourceType=subnet,Tags=[{Key=Name,Value=app-private-a},{Key=Tier,Value=app}]'

aws ec2 create-subnet --vpc-id vpc-0abc --cidr-block 10.0.21.0/24 \
  --availability-zone us-east-1a --tag-specifications \
  'ResourceType=subnet,Tags=[{Key=Name,Value=data-private-a},{Key=Tier,Value=data}]'

Tagging each subnet with its intended Tier (public/app/data) is a real, practical habit — it makes the route-table-derived public/private distinction (section 4) visible at a glance in the console, rather than requiring someone to open the route table to know what a subnet is for.

18. Short exercise

Draw the full VPC diagram (section 7) for a version of the Enterprise AI Assistant deployed across three AZs instead of two — identify exactly which resources need a third subnet in each tier, and recompute the NAT Gateway cost implication (18.18) of that change.

19. Interview questions

• What actually determines whether a subnet is public or private in AWS?
• Why does a multi-AZ design need one NAT Gateway per AZ, not one shared across all of them?
• What's the difference between CloudTrail and an application-level audit log, and why might a customer need both?
• Walk through the full request path for the Enterprise AI Assistant, AWS service by AWS service.

20. FDE/customer scenario

A customer's network architect reviews your VPC design and asks: "Why do you need a NAT Gateway per availability zone instead of just one?" A strong answer explains the specific failure mode a shared NAT Gateway reintroduces — the AZ containing that single NAT Gateway becomes a cross-AZ single point of failure for all outbound traffic, silently undermining the entire multi-AZ redundancy design — using the problem/ evidence/impact/recommendation structure this chapter's FDE communication approach models throughout Part 18.

---

Key takeaways

• "Public" and "private" subnets are entirely route-table-derived, not an inherent subnet property — a common, security-relevant misconception.
• Multi-AZ redundancy requires per-AZ NAT Gateways to actually deliver the redundancy it promises; sharing one NAT Gateway quietly defeats the design.
• CloudTrail (infrastructure audit) and application-level audit logging (Part 9.4/10.5) are two distinct, both-often-required layers.

Things you should be able to explain

• What actually makes a subnet public vs. private (route tables, not tags).
• The ALB → ECS/EKS → RDS/ElastiCache → NAT → LLM provider request path.
• Why CloudWatch and LangSmith are complementary, non-overlapping layers.

Things you should be able to build

• A complete, correctly-tiered multi-AZ VPC design for a given AI workload.
• A route-table-based public/private determination from raw route entries.

Common mistakes

• Sharing a single NAT Gateway across AZs to save cost.
• Conflating CloudTrail with application-level audit logging.
• Health checks that couple application liveness to data-tier health.

Recommended next chapter

06-infrastructure-as-code-terraform.md