7.3 — Cloud Fundamentals for AI Systems

1. What is it?

This chapter covers the cloud computing concepts an AI FDE needs regardless of specific provider (AWS, GCP, Azure) — compute, storage, networking, managed services, and the shared-responsibility model — with an emphasis on where these concepts intersect specifically with AI system requirements (GPU availability, data residency for LLM calls, VPC design for secure tool execution).

2. Why does it exist?

Almost no enterprise AI system runs on hardware you personally rack and maintain — cloud infrastructure exists to let you provision and pay for compute, storage, and networking on demand, without owning physical hardware, and to offload substantial operational burden (hardware failure, physical security, base infrastructure patching) to the cloud provider. For an AI FDE, cloud fluency is close to a prerequisite: virtually every enterprise deployment target (Part 14) is either the customer's existing cloud environment or a cloud environment you're setting up specifically for them.

3. What problem does it solve?

It solves "how do I get compute/storage/networking resources that scale with actual demand, are billed for actual usage rather than fixed capacity, and don't require me to personally manage physical hardware" — while introducing its own distinct considerations (the shared-responsibility model, data residency, cost management) that an AI FDE must understand precisely because customer requirements around data location, compliance, and cost are frequently central to real engagements (Part 9.6, Part 10.5, Part 15).

4. How does it work internally?

The shared-responsibility model

Cloud providers and customers split security/operational responsibility along a boundary that shifts depending on the service model:

IaaS (raw VMs)You manage: OS, runtime, app, data, access. Provider manages: hardware, network, virtualization.

PaaS (managed platform)You manage: application code, data, access config. Provider manages: OS, runtime, scaling, patching, hardware, network, virtualization.

SaaS (fully managed)You manage: your data, users, config. Provider manages everything else.

Understanding exactly where this boundary sits for a given service is critical for security review (Part 9) — a common, serious mistake is assuming the cloud provider handles a security responsibility that actually remains yours (e.g., a managed database service still requires you to configure access controls and network isolation correctly; the provider manages the underlying infrastructure, not your specific access policy).

Compute options and their AI-specific relevance

• Virtual machines (IaaS): full control, needed for GPU-accelerated workloads (self-hosted model inference/fine-tuning, Part 2.2) where you need specific hardware and driver configurations.
• Managed container services (e.g., a managed Kubernetes offering, or a simpler managed container-run service): the common choice for deploying a Dockerized AI backend (Part 7.1) without managing underlying VMs directly.
• Serverless functions: good for lightweight, infrequent, or highly bursty workloads, but often a poor fit for long-running LLM streaming responses or workloads needing persistent connections/state — verify a given serverless platform's request-duration and connection limits against your actual AI workload's characteristics before committing to this model.
• Managed AI/ML platforms (a cloud provider's own hosted model-serving or fine-tuning infrastructure): can reduce operational burden for self-hosted model workloads, at the cost of provider lock-in for that specific capability.

Networking — VPCs and why they matter for AI-specific security

A VPC (Virtual Private Cloud) is an isolated network segment within a cloud provider's infrastructure — the standard way to control exactly which resources can reach which other resources and the public internet. For AI systems specifically, VPC design matters directly for Part 9.6's secure tool execution: an agent's tool-execution environment (Part 3.3/9.6) should typically run in a tightly-scoped network segment with only the specific outbound access it genuinely needs (a specific internal API, not unrestricted internet egress), directly implementing the least-privilege principle at the network layer.

Data residency and cross-border data flow — a recurring, real constraint

Many enterprise customers (especially in regulated industries or specific jurisdictions, Part 10.5) have explicit requirements about where their data is processed and stored — which directly affects which cloud region you deploy in, and critically, which LLM provider region/endpoint you call, since an LLM API call sends data to wherever that provider's endpoint processes it. This is a genuinely common, concrete constraint in real FDE engagements: verify a given LLM provider's data-residency options and guarantees explicitly against a customer's specific regulatory requirements, rather than assuming any convenient default endpoint is compliant.

5. Simple mental model

Cloud infrastructure is like renting fully-serviced office space instead of buying and maintaining your own building — you get the space and services you need (compute, storage, networking) without owning or maintaining the underlying building (physical hardware, power, cooling), but you're still responsible for what happens inside your specific rented space (your application, your data, your access policies) — the landlord (cloud provider) maintains the building, not your office's specific lock configuration or who you hand keys to.

6. Real-world example

A European healthcare customer requires that patient data never leaves EU-based infrastructure, for GDPR compliance (Part 10.5). This requirement cascades through the entire architecture: the application's compute and database must run in an EU cloud region, and critically, the LLM provider called for any patient-data-touching workflow must have (and be configured to use) an EU-based processing endpoint with a documented data-residency guarantee — a detail easy to overlook if you default to a provider's global or US-based endpoint without explicitly verifying and configuring the region-specific option, and a genuine, common point of failure in real healthcare/EU engagements if not addressed deliberately from the start.

7. Architecture diagram

VPC · isolated network segment

App serverspublic-facing subnet

Databaseprivate subnet, no direct internet

Tool execution sandboxprivate subnet, scoped egress only

External LLM Provider APIverify region / data-residency

Cloud Provider (e.g., AWS/GCP/Azure) — everything inside sits within one VPC.

8. Production considerations

• Explicitly verify and configure LLM provider region/data-residency settings for any customer with data-locality requirements — don't assume a default endpoint is compliant.
• Design VPC network segmentation deliberately, especially around any component executing LLM-generated or agent-driven actions (Part 9.6) — least-privilege network access is a real, enforceable security control at the infrastructure layer, not just an application-layer concern.
• Understand the shared-responsibility boundary precisely for every managed service you use — a managed database or managed Kubernetes service handles infrastructure-level security, not your specific access-control configuration, which remains fully your responsibility.
• Budget and monitor cloud costs explicitly, separate from LLM API costs (Part 7.10 will cover the full cost picture) — compute, storage, and data-transfer costs are a distinct, real line item alongside LLM token costs.

9. Common mistakes

• Assuming a cloud provider's general security certifications automatically extend to cover a specific customer's specific compliance requirements without explicit verification.
• Deploying to a default, convenient cloud region without checking a customer's actual data-residency requirements, discovering the mismatch only during a compliance review late in the project.
• Overly permissive VPC/network configuration (broad outbound internet access for a tool-execution environment that only needs a few specific endpoints) — a real, common excessive-agency-adjacent risk at the infrastructure layer (Part 9.2/9.6).
• Confusing "the cloud provider is responsible for security" with "I don't need to think about security" — the shared-responsibility model always leaves real, substantial responsibility with you.

10. Security considerations

Sections 4/8 above cover this chapter's core security content directly — the shared-responsibility model, VPC-based least-privilege network design, and data-residency verification are all, fundamentally, security considerations specific to cloud deployment of AI systems (deepened further in Part 9's dedicated security parts).

11. Performance considerations

• Network latency between your compute and the LLM provider's API endpoint is a real, measurable factor in overall response latency (Part 2.6's tokens-per-second discussion is about generation speed; this is about the network hop before generation even starts) — choosing a compute region geographically/network-topologically close to your LLM provider's endpoint can meaningfully reduce this overhead.
• Managed services (a managed database, a managed container platform) often have different performance characteristics and tuning options than a self-managed equivalent — verify a managed service's specific performance profile against your workload's requirements rather than assuming parity with a self-hosted alternative.

12. Cost considerations

• Cloud compute/storage/networking costs are billed on genuinely different models than LLM API costs (often usage-based but with different units — compute-hours, storage-GB-months, data-transfer-GB) — model both cost categories explicitly and separately in a customer's total cost of ownership (Part 15), since conflating them obscures where the actual cost is coming from.
• Data-transfer costs (especially cross-region or cross-cloud-provider transfer) are a frequently underestimated cost line item, particularly relevant if your compute and your LLM provider's endpoint are in different regions/networks.

13. When to use it

Virtually every production AI deployment uses cloud infrastructure in some form — the practical question for an AI FDE is rarely "cloud or not" but "which specific cloud, which specific region, which specific service model," driven by the customer's actual existing infrastructure, compliance requirements, and cost constraints.

14. When NOT to use it

A customer's strict on-prem/air-gapped requirement (common in some government, defense, or highly regulated environments) may preclude public cloud infrastructure entirely — Part 14 covers this deployment scenario specifically, where the "cloud" concepts in this chapter still apply conceptually (VPC-equivalent network segmentation, the shared-responsibility model in a private-cloud/on-prem form) but without a public cloud provider in the loop.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Public cloud (AWS/GCP/Azure)	Scalability, managed services, no hardware ownership	Ongoing usage-based cost; potential data-residency/compliance friction for some customers
Private cloud / on-prem (Part 14)	Full data control, meets strict air-gapped/regulatory requirements	Higher operational burden, customer must manage more infrastructure themselves
Hybrid (some workloads cloud, some on-prem)	Balances flexibility and compliance for mixed requirements	More architectural complexity to manage two environments coherently

16. Practical Python/code example

A configuration pattern that makes LLM provider region an explicit, environment-driven setting rather than a hardcoded default — directly operationalizing section 8's data-residency verification:

import os
from anthropic import AsyncAnthropic

def build_region_appropriate_client() -> AsyncAnthropic:
    """
    Builds an LLM client configured for the region required by this deployment's
    data-residency requirements, rather than defaulting to a convenient endpoint.

    Returns:
        AsyncAnthropic: A client configured with the required base URL/region,
            explicitly sourced from environment configuration.
    """
    base_url = os.environ.get("LLM_PROVIDER_ENDPOINT")  # explicit, required, no silent default
    if base_url is None:
        raise RuntimeError(
            "LLM_PROVIDER_ENDPOINT must be explicitly configured — "
            "verify against the customer's data-residency requirements before deploying"
        )
    return AsyncAnthropic(base_url=base_url)

Making this an explicit, required environment variable (raising rather than silently defaulting) is a deliberate design choice: it forces a conscious decision at deployment time rather than allowing a compliance-relevant setting to be silently, accidentally left at a default.

17. Production-quality example

A least-privilege network policy definition (illustrative, cloud-provider-agnostic form) for a tool-execution environment, directly implementing section 8's VPC segmentation recommendation:

# Illustrative network policy — exact syntax varies by cloud provider/orchestrator;
# verify against your specific platform's current documentation.
tool_execution_network_policy:
  default_deny_all_egress: true
  allowed_egress:
    - destination: internal-crm-api.customer.internal
      port: 443
    - destination: internal-ticketing-api.customer.internal
      port: 443
  # Explicitly NOT allowed: unrestricted internet egress — an agent's tool
  # execution environment should only reach the specific systems its actual
  # tools need, per Part 9.6's sandboxing and least-privilege principles.

18. Short exercise

A customer's compliance team asks: "Where exactly does our data go when your AI assistant processes a request?" Using this chapter's concepts, list the specific components (compute region, database region, LLM provider endpoint region, any third-party integrations) you'd need to trace and document to give a complete, accurate answer.

19. Interview questions

• Explain the shared-responsibility model and give a concrete example of a security responsibility that remains yours even when using a fully managed cloud service.
• Why does an LLM provider's API endpoint region matter for a customer with data-residency requirements, beyond just where your own compute runs?
• Why would you design a tool-execution environment's network access with default-deny egress rather than default-allow?

20. FDE/customer scenario

Customer's compliance officer: "Can you guarantee our data never leaves [specific country/region]?"

This requires tracing the complete data-flow path this chapter describes: your application's compute region, your database's region, and critically, your LLM provider's actual processing endpoint region and its documented data-residency guarantees — a credible answer requires having verified all of these explicitly, not just the parts of the stack you directly provisioned, since the LLM provider's endpoint is very often the piece most likely to be overlooked in this kind of compliance conversation.

---

Key takeaways

• The shared-responsibility model always leaves real, substantial security/configuration responsibility with you, regardless of how much a cloud provider manages.
• LLM provider endpoint region is a frequently-overlooked but critical part of a data-residency compliance story — it's easy to focus only on your own compute/database region and miss it.
• VPC-based least-privilege network segmentation is a real, enforceable security control, especially relevant for agent tool-execution environments.

Things you should be able to explain

• The shared-responsibility model and where its boundary sits for different service models (IaaS/PaaS/SaaS).
• Why LLM provider endpoint region matters for data-residency compliance, not just your own infrastructure's region.

Things you should be able to build

• A deployment configuration that makes LLM provider region an explicit, required, verified setting, and a least-privilege network policy for a tool-execution environment.

Common mistakes

• Assuming cloud provider certifications automatically satisfy a specific customer's compliance requirements.
• Overlooking LLM provider endpoint region in a data-residency assessment.
• Overly permissive network egress for tool-execution environments.

Recommended next chapter

04-kubernetes.md