18.22 — Senior System Design: Cloud AI

Relationship to Part 11: Part 11 already taught full AI-system design (the 14-point framework, and nine complete designs including enterprise RAG, a customer-support agent, and a multi-tenant SaaS platform) — this chapter does not re-derive the AI architecture decisions (RAG vs. agent, retrieval strategy, evaluation approach) those chapters already own. This chapter applies the infrastructure lens this track built (18.1–18.21) to six senior-level design prompts, using Part 11.1's Requirements→Trade-offs framework but focused on the cloud/DevOps decisions specifically: compute choice, network topology, scaling mechanism, and failure handling.

1. What is it?

Six worked, infrastructure-focused system-design problems, each following Requirements → Assumptions → Architecture → Components → Data Flow → Scaling → Reliability → Security → Observability → Cost → Trade-offs — the exact structure the source spec asks for, applied at the depth an actual senior AI FDE interview or design review operates at.

2. Why does it exist?

17.1 taught the interview process for AI system design. This chapter gives you material to actually practice that process against — six distinct prompts spanning this track's full range, each requiring you to choose (and defend) infrastructure decisions rather than simply recognize them from a single chapter in isolation.

3. What problem does it solve?

It solves "I understand each infrastructure chapter individually — can I actually design a coherent system under a specific prompt, live, the way 17.1's interview format demands." Six varied prompts, worked through the same disciplined structure, are what turns fragmented knowledge into a practiced, transferable design skill.

4. How does it work internally?

Design 1: A production AI platform for 100,000 requests/day

REQUIREMENTS: ~1.16 req/sec average, but AI traffic is rarely uniform —
  clarify peak-to-average ratio (Part 12.1's discovery discipline) before
  designing for average load alone.
ASSUMPTIONS (stated explicitly, per 17.1): peak ~10x average during
  business hours; each request involves one LLM call plus retrieval.
ARCHITECTURE: 18.17's full assembled diagram — ALB → ECS/EKS (18.3/18.9)
  → RDS/Redis/vector DB (18.4/18.14) → model router (Part 7.9/18.12).
COMPONENTS: this scale does NOT require Kubernetes-specific capability
  (18.9's decision framework) — ECS/Fargate (18.3) is the simpler,
  sufficient default absent another reason to choose EKS.
DATA FLOW: 18.17's traced request path, unchanged.
SCALING: HPA/ECS auto scaling on request concurrency (18.9/18.13), NOT
  CPU — apply 18.13's bottleneck framework to confirm the LLM
  provider's own rate limit, not compute, is the more likely ceiling at
  this volume.
RELIABILITY: circuit breaker + fallback provider (18.12); Multi-AZ RDS
  (18.4/18.5).
SECURITY: 18.19's full bridge review.
OBSERVABILITY: 18.11's two-layer model.
COST: 18.18 — at this volume, external LLM API almost certainly beats
  self-hosting (18.16) on cost; verify with the actual break-even
  calculation rather than assuming.
TRADE-OFFS: ECS over EKS (simplicity over Kubernetes-specific
  capability not needed at this scale); external API over self-hosting
  (16's break-even math favors it at this volume).

Design 2: A multi-tenant enterprise RAG platform

REQUIREMENTS: strict per-tenant data isolation (Part 9.6) is the
  defining constraint — clarify with the customer whether isolation
  must be logical (shared infrastructure, enforced boundaries) or
  physical (dedicated infrastructure per tenant) before designing,
  since these have very different cost/complexity profiles.
ARCHITECTURE: builds on Part 11.4's multi-tenant SaaS design (the AI-
  architecture substance lives there) — this chapter's contribution is
  the infrastructure implementation: separate vector-DB namespaces or
  collections per tenant (Part 3.4), RLS (row-level security) on shared
  RDS tables OR separate schemas/databases for stricter isolation tiers.
COMPONENTS: Kubernetes NAMESPACES per tenant (18.9) plus network
  policies — reiterating 18.9's explicit warning that a namespace ALONE
  is not a strong isolation boundary; add network policies or dedicated
  node pools for tenants with contractually stricter requirements.
DATA FLOW: request → tenant identification (Part 9.4) → tenant-scoped
  retrieval and tenant-scoped IAM role (18.19's tenant-tagged policy
  pattern) for any downstream resource access.
SCALING: per-tenant noisy-neighbor risk (18.12's bulkhead concept,
  applied at the tenant level) — a bulkhead per tenant tier (rate
  limits, Redis-based, 18.14) prevents one tenant's load from degrading
  others sharing infrastructure.
RELIABILITY: same patterns as Design 1, PLUS tenant-level SLO tracking
  (18.11) — an aggregate SLO can look healthy while one tenant is
  actually degraded.
SECURITY: 18.19's tenant-isolation bridge row is this design's central
  concern — verify the isolation boundary matches the STATED requirement,
  not a convenient, weaker default.
OBSERVABILITY: per-tenant dashboards/alerts (18.11), not only aggregate.
COST: shared infrastructure is cheaper (18.18) but weakens isolation;
  dedicated infrastructure per tenant is the reverse trade — a real,
  tier-dependent choice (e.g., a "dedicated" pricing tier for
  strict-compliance customers).
TRADE-OFFS: logical vs. physical isolation, and the cost/complexity gap
  between them, is this design's central, explicit trade-off to present.

Design 3: A highly available LangGraph agent platform

REQUIREMENTS: "highly available" needs a concrete target (17.1's
  discipline) — clarify acceptable downtime/RTO (18.12) before designing.
ARCHITECTURE: the AI-architecture substance (agent design, tool
  permissions) lives in Part 5/11.7 — this chapter's focus: LangGraph's
  checkpointed state (Part 5.3, `PostgresSaver`) requires RDS Multi-AZ
  (18.4/18.5) so a paused, human-in-the-loop-interrupted (Part 5.4)
  workflow survives an infrastructure failover, not just a stateless
  request.
COMPONENTS: ECS/EKS (18.3/18.9) with a PDB (18.9) — availability
  requires surviving BOTH crashes and planned disruption (node upgrades),
  a distinction 18.9 flagged explicitly.
DATA FLOW: an interrupted agent's state must be resumable from a
  DIFFERENT replica than the one that paused it (statelessness at the
  application layer, all state in RDS/checkpointer, Part 5.3) — a
  design detail easy to get wrong if session affinity is assumed.
SCALING: 18.13's framework — a long-running, potentially-paused agent
  workflow behaves differently under load than a stateless request;
  capacity planning must account for IN-PROGRESS, paused workflows
  consuming database rows (cheap) rather than active compute (they
  don't hold a process while paused, if correctly checkpointed).
RELIABILITY: 18.12's full stack for every tool call the agent makes,
  not just the LLM call itself — each external tool integration (Part
  3.3/10.3) needs its own timeout/circuit-breaker treatment.
SECURITY: 18.19's excessive-agency bridge row — least-privilege IAM for
  every tool (18.3), scoped per the agent's actual legitimate need.
OBSERVABILITY: LangSmith traces (Part 6.1) for agent reasoning,
  correlated (18.11) with infrastructure traces for the same execution.
COST: paused, checkpointed state is cheap (a database row); avoid
  designing an architecture that holds a live process/connection open
  for the ENTIRE duration of a possibly-long human-in-the-loop pause.
TRADE-OFFS: PostgresSaver (durable, RDS-backed) vs. a faster but less
  durable checkpointer — Part 5.3/18.14's durability-category reasoning
  determines this is not a close call for a system requiring genuine
  high availability.

Design 4: An AI document-processing pipeline

REQUIREMENTS: expected volume and burstiness (Part 12.1's discovery)
  determine whether this is even an async/queue design at all (18.15's
  actual decision criteria) — clarify before assuming.
ARCHITECTURE: 18.4/18.15's complete ingestion pipeline (S3 → SNS → SQS
  → worker pool → vector DB + RDS status), unchanged from this track's
  running example.
COMPONENTS: worker pool as ECS/EKS Jobs or a long-running Deployment
  polling SQS (18.9's Job-vs-Deployment distinction) depending on
  whether processing is naturally batch-shaped or continuous.
DATA FLOW: 18.15's stage-idempotent design (parse → embed → index),
  each stage independently retryable without re-incurring cost for
  already-completed stages.
SCALING: worker pool scaled on QUEUE DEPTH (18.9's KEDA point,
  18.13/18.15) — the single most important, most often-missed scaling
  decision for this design specifically.
RELIABILITY: DLQ with alerting (18.4/18.11); idempotent, stage-aware
  retries (18.15).
SECURITY: document content may be sensitive (Part 9.3) — encryption at
  rest/in transit (18.4/18.5) and scoped IAM per pipeline stage (18.3).
OBSERVABILITY: pipeline-specific SLI (18.11's exercise from that
  chapter) — "percentage ingested within N minutes," not a latency
  metric borrowed from the synchronous chat path.
COST: 18.18's stage-idempotency cost point is this design's most
  concrete, quantifiable lever — avoiding redundant embedding-API calls
  on retry.
TRADE-OFFS: standard vs. FIFO queue (18.4/18.15) — only needed if
  same-document re-uploads require strict ordering, not by default.

Design 5: An AI customer-support platform

REQUIREMENTS: the AI-architecture substance (when to escalate to a
  human, tone/quality bar) lives in Part 11.3 — this chapter's focus:
  what infrastructure requirement does "customer-facing, always-on
  support" impose that an internal tool wouldn't?
ARCHITECTURE: 18.17's full diagram, with an ADDED human-escalation path
  (Part 5.4's interrupt pattern) requiring the same durable,
  RDS-backed checkpointing as Design 3.
COMPONENTS: real-time channel support (chat widget, possibly a
  telephony integration, Part 10.3) may require WebSocket/SSE handling
  (Part 5.5) — revisit 18.5's ALB idle-timeout point (Part 7.5's flagged
  gap) explicitly for any long-lived streaming connection.
DATA FLOW: escalation to a human agent needs a real integration (a
  ticketing system, Part 10.3) — treat this exactly like any other
  external dependency requiring 18.12's resilience treatment, not an
  assumed-reliable internal call.
SCALING: customer support traffic is often predictably time-of-day
  bursty (business hours) — scheduled/predictive scaling (a variant of
  18.13's HPA discussion) may outperform purely reactive scaling for
  this specific traffic shape.
RELIABILITY: customer-facing means an outage is directly, immediately
  visible to paying customers — justifies a stricter SLO (18.11) and
  more investment in canary deployment (18.8) than an internal tool would.
SECURITY: customer PII flows through this system routinely (Part
  9.3/9.6) — this is not an optional or edge-case consideration here.
OBSERVABILITY: customer-facing SLAs (18.11) may be CONTRACTUAL — the
  SLO-below-SLA margin discipline (18.11) matters more here than in an
  internal-tool design.
COST: 24/7 availability requirement likely rules out aggressive
  scale-to-zero patterns that a batch/internal system (Design 4) could
  use freely.
TRADE-OFFS: real-time (WebSocket/SSE) vs. simpler polling for the
  human-escalation status — a genuine UX-vs-infrastructure-complexity
  trade-off worth stating explicitly.

Design 6: An AI system that must run inside a customer's AWS VPC

REQUIREMENTS: "inside the customer's VPC" is 18.23's exact recurring
  customer statement — clarify precisely what it means for THIS
  customer (their own AWS account entirely? A peered/shared VPC? A
  specific compliance driver behind the requirement, per Part 12.1?).
ARCHITECTURE: identical to 18.17's diagram in SHAPE, but deployed via
  Terraform (18.6) into the CUSTOMER's AWS account, not your own —
  this is precisely the reproducibility 18.6 argued Terraform provides
  for exactly this scenario (Part 14.1's "customer's cloud account"
  deployment model).
COMPONENTS: your CI/CD pipeline (18.8) needs a secure, scoped path to
  deploy into an external account — cross-account IAM roles (18.3)
  with tightly-scoped trust policies, not a shared, static credential.
DATA FLOW: unchanged in shape, but EVERY external call (the LLM
  provider, any SaaS integration, Part 10.3) now needs explicit
  confirmation it's acceptable under the customer's specific
  data-residency requirement (18.16's self-hosting decision may be
  forced here, not merely optional, if no compliant external-API path
  exists).
SCALING/RELIABILITY: identical patterns to Design 1, now operated
  within infrastructure you don't fully control access to for
  debugging — 18.10's `kubectl`-based diagnostics remain the same, but
  YOUR access to run them may be more limited/audited than in your own
  environment.
SECURITY: 18.19's full bridge review, PLUS the cross-account access
  pattern itself is a new, additional attack surface to scope carefully
  (least privilege on the deployment role specifically, 18.3).
OBSERVABILITY: the customer may require observability data (logs,
  traces) to ALSO stay within their account/region — verify this
  explicitly rather than assuming your own centralized observability
  stack (18.11) can simply aggregate across account boundaries.
COST: billed to the CUSTOMER's account directly — a genuinely different
  cost-conversation shape (18.18) than a vendor-hosted, usage-billed
  SaaS model.
TRADE-OFFS: this deployment model trades your own operational
  convenience (Part 14.1) for the customer's control/compliance
  requirement — state this explicitly as the deliberate reason for the
  added complexity, not an unfortunate necessity to apologize for.

5. Simple mental model

Each of these six prompts is the same building (18.17's architecture) with one load-bearing wall moved — practicing all six is what reveals which parts of the design are truly load-bearing (identity, network segmentation, checkpointed state durability) versus which parts are genuinely free to vary per prompt (compute choice, queue vs. no queue, which specific AWS account).

6. Real-world example

Design 6 is, concretely, Part 16.8's full FDE engagement narrative's Stage 5 (Deployment) — the same underwriting-automation customer's compliance-driven requirement to deploy within their own cloud tenancy, now viewed through this chapter's infrastructure-decision lens rather than that chapter's engagement-narrative lens.

7. Architecture diagram

(Each design's architecture is 18.17's base diagram with the specific modification described in section 4 — deliberately not reproduced six times here to keep the comparison across designs legible.)

8. Production considerations

Practice stating assumptions explicitly (17.1's discipline) for each design — none of these six prompts is fully specified as given, and a strong answer states what's being assumed before designing against it.

9. Common mistakes

• Re-deriving the AI-architecture decision (RAG vs. agent, Part 11's territory) instead of focusing on the infrastructure decisions this chapter and track actually teach.
• Presenting one design's architecture as a template applied unchanged to all six prompts, missing each prompt's actual, distinguishing requirement (isolation for Design 2, durability for Design 3, account boundary for Design 6).
• Skipping the explicit trade-offs section — a design with no stated alternative considered is a weaker answer regardless of correctness (17.1's exact point).

10. Security considerations

Design 2 (multi-tenant isolation) and Design 6 (cross-account access) are this chapter's two most security-critical prompts — both directly exercise 18.19's bridge reasoning under a concrete, specific scenario rather than in the abstract.

11. Performance considerations

Design 5 (customer support) is this chapter's clearest case for why non-functional requirements (Part 11.1) must be stated explicitly and early — "highly available, customer-facing" implies a meaningfully different SLO/investment level than an internal tool, and a design that doesn't distinguish them is under-specified.

12. Cost considerations

Design 6's cost conversation (billed to the customer's own account) is qualitatively different from the other five designs' vendor-hosted cost model — worth recognizing as a genuinely distinct conversation shape, not merely "the same cost analysis in a different account."

13. When to use it

As deliberate practice material for 17.1's interview process, and as a template for approaching a genuinely new, real customer design prompt using this track's full vocabulary.

14. When NOT to over-apply it

Don't force every real design prompt into exactly one of these six molds — use them as practiced patterns to draw from, not an exhaustive taxonomy; a real prompt will often combine elements of several (Design 2 plus Design 6's account-boundary constraint, for instance).

15. Alternatives and trade-offs

Each design's own trade-offs section (section 4) is this chapter's concrete content — there is no single alternative "way" to do system design generally beyond 17.1/Part 11.1's shared framework, consistently applied.

16. Practical example — a blank template for a new design prompt

# System Design: [Prompt]

## Requirements (clarified, not assumed)
## Assumptions (stated explicitly where not clarified)
## Architecture (reference 18.17's base diagram; note deviations)
## Components (compute/data/network choices, with WHY)
## Data flow
## Scaling (18.13's bottleneck-first framework)
## Reliability (18.12's patterns, applied to THIS design's dependencies)
## Security (18.19's bridge review, applied to THIS design's AI risks)
## Observability (18.11's two-layer model)
## Cost (18.18's investigation order)
## Trade-offs (at least one genuine alternative considered and rejected)

17. Production-quality example

Section 4's six worked designs ARE this chapter's production-quality examples — each is a complete, if intentionally concise, application of the section 16 template to a real, distinct prompt.

18. Short exercise

Pick a seventh prompt not covered here — "design an AI system for processing 10,000 support tickets per day with a 4-hour SLA" — and work through the full section-16 template yourself, in writing, before checking which of this chapter's six designs it most resembles and why.

19. Interview questions

Each of section 4's six prompts, verbatim, IS an interview question — practice delivering each one live, narrated, timed to 17.1's 45-minute structure, from memory rather than reading the worked answer.

20. FDE/customer scenario

A real customer conversation will rarely match one of these six prompts exactly — the actual skill this chapter builds is recognizing which combination of this track's infrastructure decisions (compute choice, isolation model, durability requirement, account boundary, scaling trigger) a genuinely novel prompt actually calls for, then reasoning through it live using 17.1's process, not recalling a memorized answer.

---

Key takeaways

• Six varied design prompts, worked through the same disciplined structure, reveal which architectural decisions are genuinely load-bearing (identity, checkpoint durability, isolation boundary) versus which are free to vary per prompt.
• The AI-architecture decision (RAG vs. agent, Part 11) and the infrastructure decision (this chapter) are distinct layers of the same design — a strong answer addresses both, but doesn't confuse one for the other.
• Stating assumptions and trade-offs explicitly (17.1's discipline) is what separates a strong design answer from a merely correct one.

Things you should be able to explain

• What infrastructure decision changes across each of the six design prompts, and why.
• Why Design 3's checkpoint durability requirement and Design 2's isolation requirement are each non-negotiable, not stylistic choices.

Things you should be able to build

• A complete, from-scratch system design for a novel seventh prompt, using section 16's template.

Common mistakes

• Re-deriving AI-architecture decisions instead of focusing on infrastructure trade-offs this track actually teaches.
• Applying one design as a template unchanged to a genuinely different prompt's distinguishing requirement.
• Omitting explicit trade-offs and assumptions.

Recommended next chapter

23-fde-cloud-customer-scenarios.md