Appearance
18.2 — Networking Fundamentals for Production Cloud Architectures
1. What is it?
The set of networking concepts — addressing, routing, DNS, TCP/HTTP/TLS, firewalls, load balancers, and private-vs-public network boundaries — needed to actually understand what a cloud architecture diagram is saying, rather than treating "VPC," "security group," and "load balancer" as unexplained boxes you copy from a template.
2. Why does it exist?
18.1 assumed you could reach a host once you knew where it was. This chapter explains how a request finds that host at all, and every boundary it crosses on the way — because every AI-system architecture from Part 7.3 onward (and every AWS service in 18.3) is fundamentally a networking diagram with compute, storage, and AI attached to it. Without this chapter, "put the database in a private subnet" is a rule to memorize; with it, it's an obvious consequence of understanding what a private subnet actually does and doesn't allow.
3. What problem does it solve?
It solves "I can build the AI application but I can't reason about why it's unreachable, why a security review flagged our network design, or how to answer a customer's question about data flow across a network boundary" — a gap that shows up constantly in FDE work, since customers with real security/compliance requirements (Part 9, Part 10.5) think and ask questions in exactly this vocabulary.
4. How does it work internally?
IP addresses, IPv4, and CIDR
An IP address identifies a device on a network. IPv4 addresses are four 8-bit numbers (10.0.1.25) giving ~4.3 billion possible addresses — a number exhausted for public internet use years ago, which is precisely why private address ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) exist for internal use behind NAT (below). CIDR notation (10.0.0.0/16) describes a block of addresses: the /16 says the first 16 bits are fixed (the network portion), leaving 16 bits free (65,536 addresses) for hosts within it. A VPC (18.3/18.5) is, at its core, exactly one CIDR block you choose — e.g., 10.0.0.0/16 — subsequently carved into smaller CIDR blocks (subnets) like 10.0.1.0/24 (256 addresses) for specific purposes.
10.0.0.0/16 → VPC: 65,536 addresses total
├─ 10.0.1.0/24 → public subnet AZ-a (256 addresses)
├─ 10.0.2.0/24 → public subnet AZ-b
├─ 10.0.11.0/24 → private subnet AZ-a (app servers, ECS/EKS)
├─ 10.0.12.0/24 → private subnet AZ-b (app servers, ECS/EKS)
├─ 10.0.21.0/24 → private subnet AZ-a (database, cache)
└─ 10.0.22.0/24 → private subnet AZ-b (database, cache)Private subnets in both AZs — not just AZ-a — is deliberate: this chapter's own NAT-Gateway-per-AZ resilience argument (and 18.5's full multi-AZ VPC design) only holds together if the application and data tiers actually have a subnet to run in in each AZ; a private tier confined to a single AZ would defeat the multi-AZ redundancy this design is for.
TCP vs. UDP, and ports
TCP is connection-oriented, reliable, ordered — the connection is established (the three-way handshake: SYN, SYN-ACK, ACK) before any data flows, and the protocol guarantees delivery and order, retransmitting lost packets. Nearly everything an AI service does — HTTP calls to an LLM provider, a Postgres connection, a Redis connection — runs over TCP, because losing or reordering part of an API response or a SQL query is unacceptable. UDP is connectionless and makes no delivery guarantee — used where low latency matters more than guaranteed delivery (DNS queries, some real-time media) — rarely something an AI engineer configures directly, but worth knowing exists so "why does DNS use UDP but my API calls use TCP" has an answer. A port (18.1) identifies which specific service on a host a TCP/UDP connection is for — port 443 for HTTPS, 5432 for Postgres, 6379 for Redis, by strong convention rather than hard rule.
DNS: turning names into addresses
DNS (Domain Name System) resolves a human-readable name (api.acme-assistant.com) to an IP address, via a hierarchical lookup (root servers → TLD servers → authoritative servers for the specific domain). In AWS, Route 53 (18.3/18.5) is the managed DNS service most architectures use for this. For an AI system, DNS matters in two specific, recurring ways: (1) your own service's public hostname needs a DNS record pointing at your load balancer; (2) DNS resolution failures for an outbound dependency (the LLM provider's API hostname, briefly unresolvable) is a real, if uncommon, transient-failure mode worth including in the retry/failure-handling thinking from Part 7.9/18.12.
HTTP/HTTPS and TLS
HTTP is the application-layer protocol nearly every AI system's request/response traffic runs over — both inbound (a user's browser or API client calling your FastAPI service) and much of the outbound side (your service calling the LLM provider's API, per Part 1.1/1.2). TLS (Transport Layer Security, the successor to SSL) is what turns HTTP into HTTPS: it encrypts the connection and (via a certificate, verified against a trusted certificate authority) authenticates that you're actually talking to the server you think you are, not an attacker intercepting the connection (a man-in-the-middle attack). TLS termination is the point in an architecture where encrypted traffic is decrypted — commonly at the load balancer (18.3/18.5's ALB), after which traffic to backend instances may travel unencrypted within a trusted private network, or re-encrypted end-to-end for stricter compliance requirements (a real, common customer ask covered again in 18.19/18.23).
NAT — Network Address Translation
Instances in a private subnet (no direct route to the internet) still often need outbound internet access — to call the LLM provider's API, to pull a package during a build, to reach an external SaaS integration (Part 10.3). A NAT Gateway provides exactly this: outbound-only internet access for private-subnet resources, without giving them a public IP or allowing inbound connections initiated from the internet. This is precisely how a database or application server can call out to Anthropic's API while remaining completely unreachable from the public internet directly — the architectural pattern underlying nearly every secure AI deployment in this handbook.
A VPC Endpoint is a related, narrower mechanism worth knowing exists alongside NAT: rather than routing ALL outbound traffic through a NAT Gateway, a VPC Endpoint lets private-subnet resources reach one specific AWS service (S3, Secrets Manager, and others) directly over AWS's own internal network, without a NAT Gateway or the public internet involved at all — 18.5 covers this concretely, including its cost implications for the NAT Gateway spend this chapter's cost section already flags.
Firewalls, security groups, and network ACLs
A firewall is a set of rules controlling which traffic is allowed to pass. In AWS specifically (18.3/18.5 go deeper), two firewall-like mechanisms exist at different layers: a security group is stateful (if you allow an inbound request, the matching outbound response is automatically allowed) and attaches to individual resources (an instance, a load balancer); a network ACL is stateless (inbound and outbound rules must both be explicitly configured) and applies at the subnet level as a coarser, secondary layer of defense. In practice, most day-to-day AI-system network configuration happens via security groups; network ACLs are a defense-in-depth layer, not the primary control.
Proxies, reverse proxies, and load balancers
A forward proxy sits in front of clients, making requests on their behalf (less common in typical AI-system architectures). A reverse proxy sits in front of servers, receiving requests on their behalf and forwarding them onward — Nginx used this way, or an AWS Application Load Balancer, doing exactly this for your FastAPI service. A load balancer is a reverse proxy specialized for distributing traffic across multiple backend instances, additionally performing health checks (18.1's section 6 touched this) and removing unhealthy targets from rotation automatically — the reason a single failing instance doesn't take down your whole service, and also, when misconfigured, the reason a perfectly healthy fleet can look completely down (a health-check path or timeout misconfiguration).
CDN — Content Delivery Network
A CDN (AWS's is CloudFront, 18.5) caches content at edge locations physically close to users, and serves a cached copy directly from the edge on a repeat request instead of going back to your origin server every time — an origin pull: the first request for a given piece of content still reaches your origin (the ALB, an S3 bucket), the CDN caches the response, and subsequent requests for that same content from nearby users are served from the edge cache, both faster (shorter physical/network distance) and with less load reaching your origin. This mechanism works well for genuinely static, shareable content — a chat UI's JS/CSS bundle, images, a public marketing page — where the same bytes are correctly served to many different users. It does not help with the Enterprise AI Assistant's actual chat API responses: those are per-user, dynamic, and frequently streamed token-by-token (Part 5.5) — there is no shared, cacheable response to serve from an edge location, since the whole point is that each user's answer is uniquely generated for their specific question. A CDN is a real, valuable piece of this architecture (18.5 covers CloudFront's role concretely) — just not a lever for LLM response latency, which is an application/caching-layer concern (Part 7.8, 18.14) entirely separate from what a CDN does.
Public vs. private networks, ingress, and egress
A public subnet has a route to an Internet Gateway — resources there can have a public IP and be directly reachable from the internet. A private subnet has no such route — its resources are unreachable from the internet directly, reaching the internet (if at all) only outbound via a NAT Gateway. Ingress describes traffic coming into a network or resource; egress describes traffic going out. Nearly every real AI system's baseline security posture (Part 9, 18.19) is built from exactly this distinction: load balancer in a public subnet (ingress from users), application and database in private subnets (no direct ingress from the internet at all, only from the load balancer), egress to the LLM provider allowed outbound through a NAT Gateway.
5. Simple mental model
A VPC is like a building you own: the building's street address is your CIDR block, floors are subnets, the lobby with a street-facing door is your public subnet, the internal offices with no direct street access are your private subnets, security guards at each office door are security groups, and a mail-forwarding service that lets internal offices send outgoing mail without anyone outside being able to walk in through that same channel is your NAT Gateway.
6. Real-world example — tracing a request end to end
Exactly as the source spec's trace requests, walked through with what happens at each hop:
User's browser
↓ DNS lookup: api.acme-assistant.com → 34.201.x.x (Route 53, 18.3/18.5)
Internet
↓ TLS handshake + HTTP request over port 443
Load Balancer (public subnet, health-checks its targets)
↓ TLS terminated here; forwards to a healthy target over the
private network (often re-encrypted internally for compliance)
Network (private subnet, security group allows only LB → app:8000)
↓
Application (FastAPI, Part 1.3) — receives the HTTP request
↓ invokes the compiled graph
LangGraph (Part 5) — routes through nodes, may call tools/retrievers
↓ outbound HTTPS call, via NAT Gateway (private subnet has no
direct internet route otherwise), DNS-resolving the provider's
API hostname
LLM API (Anthropic/OpenAI/etc.) — processes the request, returns a
response over the same TLS connection
↓
Database (private subnet, reachable only from the app's security
group, e.g. checkpoint state via PostgresSaver, Part 5.3)
↓
Response flows back: Application → Load Balancer → TLS re-encrypted
→ Internet → User's browserEvery hop in this trace is something a customer's network/security team will ask about in a real engagement (18.23) — being able to narrate this trace precisely, hop by hop, is a direct, practical FDE skill.
7. Architecture diagram
User's browser
Route 53DNS lookup
Internet Gateway
public subnet
Load BalancerTLS termination · health checks
NAT Gatewayoutbound only
private subnet
ApplicationFastAPI
LangGraphroutes through nodes/tools
private subnet
Databasecheckpoint state
LLM APIAnthropic/OpenAI/etc.
8. Production considerations
- Databases and caches should live in private subnets with no route to the internet at all — not even outbound — since they never need to initiate external connections; this is a concrete, checkable security posture.
- TLS should terminate at the load balancer at minimum, and be re-encrypted to backend targets for any workload with real compliance requirements (18.19/18.23) — "traffic is encrypted end-to-end" is a specific, frequently-asked customer security question with a specific, checkable answer in your architecture.
- Health-check configuration (path, interval, unhealthy threshold) on the load balancer should be tuned to your application's actual startup time and failure characteristics — a health check that's too aggressive during a slow cold start (a large model/tokenizer load, Part 7.1/18.16) can cause a load balancer to cycle instances that were about to become healthy.
9. Common mistakes
- Putting a database in a public subnet "because it was easier to connect to during development" and leaving it that way in production — a concrete, common, serious security lapse.
- Assuming a security group being "open" (0.0.0.0/0) on a non-web port is fine because "it's just for debugging" — this is exactly the kind of finding a customer's security review (Part 9, 18.23) will flag immediately, and rightly so.
- Confusing a stateless network ACL's need for explicit outbound rules (easy to forget) with a security group's automatic stateful behavior — a frequent source of "why can't this instance respond, even though the inbound rule looks right" confusion.
- Not accounting for NAT Gateway cost (18.18) when designing a multi-availability-zone architecture — one NAT Gateway per AZ (the resilient, recommended pattern) multiplies this specific cost line item.
10. Security considerations
This entire chapter is largely security architecture — the public/ private subnet split, security groups, and NAT Gateway pattern together form the baseline network security posture nearly every enterprise AI deployment is built on, and is very often the first thing a customer's security team asks to see (18.23's customer-scenario chapter engages this directly, e.g. "everything must run inside our private VPC").
11. Performance considerations
- TLS handshake overhead is real but usually amortized by connection reuse/keep-alive (18.1's socket-level connection reuse point) — matters most for very short-lived, high-frequency outbound calls.
- A load balancer adds a small, generally negligible latency hop; a misconfigured health check causing unnecessary instance churn is a far more common real performance problem than the load balancer hop itself.
- NAT Gateway throughput has real bandwidth limits per gateway — a very high-volume AI workload with heavy outbound traffic (e.g., large document uploads to an external processing API) should be aware this exists as a potential bottleneck, not an unlimited pass-through.
12. Cost considerations
NAT Gateways bill both hourly and per-GB of data processed — a frequently-underestimated line item for AI workloads that make many outbound calls (to LLM providers, external APIs) through them; this is revisited concretely with numbers in 18.18's FinOps chapter. Data transfer between availability zones also has a real, often-overlooked cost, relevant once an architecture spans multiple AZs for availability (18.5).
13. When to use it
Every cloud-hosted AI system requires this networking foundation; there is no meaningful AI-production-architecture chapter in this handbook (Part 7, Part 11, this Part 18) that doesn't implicitly rest on these concepts.
14. When NOT to over-apply it
A purely local prototype (Part 13.2's rapid-prototyping phase) doesn't need a VPC design — over-engineering network architecture before a design is even validated wastes the exact kind of premature effort Part 13 warns against; apply this chapter's depth once you're moving toward production.
15. Alternatives and trade-offs
Serverless-first architectures (heavy Lambda usage, 18.3) can reduce how much of this networking design you must own directly — at the cost of less control and some genuinely different constraints (VPC-attached Lambda cold starts, for instance) — a trade-off revisited in 18.3's EC2-vs-Lambda comparison.
16. Practical example — reasoning through a connectivity requirement
python
"""
A customer says: "Our compliance team requires that our database never
be reachable from the public internet, under any circumstance."
Translate this into concrete network design decisions:
"""
requirements_to_design = {
"database in a private subnet": (
"No route to an Internet Gateway at all — not just 'no public IP', "
"the subnet's route table itself has no path outward."
),
"security group locked to the app tier only": (
"Inbound rule: allow port 5432 ONLY from the application's own "
"security group (referenced by security-group ID, not a CIDR "
"range) — so even another private-subnet resource can't reach it "
"unless explicitly granted."
),
"no NAT Gateway route from the DB subnet": (
"The database never needs OUTBOUND internet access either — "
"omit any NAT route from its route table entirely, closing off "
"even a hypothetical exfiltration path via an outbound connection."
),
"verify, don't just assert": (
"Attempt a connection to the DB's endpoint from outside the VPC "
"and confirm it times out — a network security claim should be "
"tested, not just configured and assumed correct."
),
}17. Production-quality example — a minimal, correct security group set
hcl
# Terraform (introduced fully in 18.6) — illustrative security-group
# design implementing the public/private split from section 4/7.
resource "aws_security_group" "alb" {
name = "ai-app-alb-sg"
description = "Allow HTTPS from the internet to the load balancer only"
vpc_id = aws_vpc.main.id
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"] # public-facing HTTPS — intentional
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_security_group" "app" {
name = "ai-app-service-sg"
description = "Allow traffic ONLY from the load balancer"
vpc_id = aws_vpc.main.id
ingress {
from_port = 8000
to_port = 8000
protocol = "tcp"
security_groups = [aws_security_group.alb.id] # not a CIDR range
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"] # outbound to LLM provider, via NAT
}
}
resource "aws_security_group" "db" {
name = "ai-app-db-sg"
description = "Allow traffic ONLY from the application tier"
vpc_id = aws_vpc.main.id
ingress {
from_port = 5432
to_port = 5432
protocol = "tcp"
security_groups = [aws_security_group.app.id] # app tier only
}
# No egress rule at all — the database never initiates outbound
# connections, so no route is granted for it.
}Referencing security groups by ID rather than by CIDR range (as shown for the app and db security groups) is the specific, correct pattern — it remains valid even as instances are replaced or scaled, unlike a CIDR range that would need updating whenever the underlying IP addresses change.
18. Short exercise
Draw (on paper or a diagramming tool) the full network path for a request to a RAG-based AI application you've designed in an earlier part of this handbook (e.g., Part 11.2's enterprise RAG design) — labeling every subnet, security group, and the public/private boundary explicitly, then check it against section 8's production-considerations list.
19. Interview questions
- Explain the difference between a security group and a network ACL, and when each matters.
- Why does a private-subnet application server need a NAT Gateway but a private-subnet database typically shouldn't have one at all?
- Walk through what happens, hop by hop, when a user's request reaches your AI application and gets a response back.
- What's the security and compliance significance of TLS termination point, and why might a customer require re-encryption to the backend?
20. FDE/customer scenario
A customer's network security team, reviewing your proposed architecture, asks: "Can you confirm our vector database will never be exposed to the public internet, even indirectly?" A strong answer walks through exactly this chapter's public/private subnet split, security-group-by-ID pattern, and the absence of any NAT route from the database's subnet — concretely, specifically, and verifiably, rather than a general "yes, it's secure" reassurance; this is precisely the kind of question Part 12.3's layered communication (translating architecture into the specific vocabulary a network/security stakeholder is asking in) prepares you to answer well.
Key takeaways
- A cloud architecture diagram is fundamentally a networking diagram — CIDR blocks, subnets, security groups, and the public/private boundary are the substrate every AI-system architecture in this handbook sits on.
- The public-subnet-LB / private-subnet-app-and-data pattern, with a NAT Gateway for necessary outbound-only traffic, is the baseline secure architecture nearly every real enterprise AI deployment starts from.
- Being able to trace a request hop-by-hop (DNS → LB → app → LLM API → database → response) is a direct, frequently-used FDE communication skill.
Things you should be able to explain
- CIDR notation and what a VPC's address block actually represents.
- The difference between a security group and a network ACL.
- Why a NAT Gateway allows outbound-only access without inbound exposure.
Things you should be able to build
- A security-group design (section 17) correctly separating a public LB tier, a private app tier, and a private data tier with no unnecessary network access between them.
Common mistakes
- Databases placed in public subnets "temporarily" and left there.
- Overly permissive (0.0.0.0/0) security group rules on non-public ports.
- Ignoring NAT Gateway cost and per-AZ multiplication in architecture design.
Recommended next chapter
03-aws-compute-and-identity.md