Appearance
18.16 — AI Infrastructure Deep Dive
This chapter is the reason this whole track exists: everything in 18.1–18.15 built the general cloud/DevOps foundation; this chapter is where it becomes distinctly AI infrastructure — GPUs, inference servers, quantization, batching, and the concrete external-API-vs- self-hosting decision an AI FDE is asked to make or defend constantly.
1. What is it?
The infrastructure specific to running AI models themselves — as opposed to the infrastructure around them (18.1–18.15's networking, compute, data, and operational layers) — covering GPU hardware concepts, inference serving, model-serving optimization techniques (batching, quantization), and the token/throughput economics that make self-hosting a model a genuinely different infrastructure problem than calling an external API.
2. Why does it exist?
Everything the Enterprise AI Assistant has done so far in this track treats the LLM as an external API call — a black box reached over HTTPS (18.2), fronted by a model router (Part 7.9/18.12). That's the right default for most AI FDE work, and this chapter doesn't argue otherwise — but it exists because "when, and why, would you self-host instead" is a question every AI FDE eventually faces from a customer, and answering it well requires actually understanding what's different about serving a model yourself: hardware, memory, and a completely different capacity- planning problem than a stateless web API.
3. What problem does it solve?
It solves "our customer wants (or is required, by data residency/ compliance) to run models on their own infrastructure instead of calling an external API — what does that actually involve, and is it even a good idea" — a real, recurring FDE decision point, not a hypothetical.
4. How does it work internally?
CPU vs. GPU, and why it matters for inference
A CPU has a small number of powerful, general-purpose cores optimized for sequential, branching logic. A GPU has thousands of simpler cores optimized for the same operation applied in parallel across huge amounts of data — exactly the shape of the matrix multiplications that dominate both training and inference for a transformer model (Part 2.4). This is the entire reason GPUs matter for LLMs at all: inference is overwhelmingly matrix multiplication, and a GPU performs it orders of magnitude faster than a CPU for models of any real size, at the cost of GPUs being specialized, expensive, and far less flexible for general- purpose application logic (your FastAPI service, LangGraph orchestration, Part 5, still runs on ordinary CPU — only the model's own forward pass benefits from GPU acceleration).
GPU memory — the actual constraint that shapes everything else
GPU memory (VRAM) is a distinctively hard, non-negotiable constraint: a model's weights (plus activation memory during inference, plus the KV cache — cached key/value attention states for previously-generated tokens, which grows with both context length and concurrent request count) must fit within the GPU's available VRAM, or inference for that request simply fails outright, unlike a CPU/RAM system where you can (up to a point) tolerate being slower under memory pressure (18.1's OOM discussion) rather than failing immediately. This is the single most important AI-infrastructure-specific capacity-planning fact: VRAM, not raw compute throughput, is very often the binding constraint on how large a model you can serve, and how many concurrent requests (each consuming additional KV-cache memory) a given GPU can actually handle — directly analogous to 18.13's bottleneck-identification discipline, now applied to a GPU-specific resource.
Quantization
Quantization reduces the numeric precision used to store a model's weights (e.g., from 16-bit floating point down to 8-bit or 4-bit integer representations) — directly reducing the VRAM required to hold the model, and often speeding up inference, at the cost of some model quality degradation (the size of that degradation varies by model, quantization method, and task, and should be evaluated empirically against your actual use case, Part 8's evaluation discipline, rather than assumed negligible or assumed severe). Quantization is the concrete lever that makes self-hosting a large model feasible on smaller/cheaper GPU hardware than its full-precision weights would require — a real, practical trade-off between infrastructure cost and model quality, not a free optimization.
Batching
Batching processes multiple inference requests together, amortizing some fixed per-request overhead and — more importantly for GPU utilization specifically — keeping the GPU's massively parallel compute actually busy, since a single request's forward pass often doesn't use a modern GPU's full parallel capacity on its own. Static batching groups a fixed batch upfront and waits for all requests in it to finish before starting the next batch — simple, but a fast request in the batch still waits for the slowest one. Continuous (in-flight) batching — the modern approach used by serving frameworks like vLLM and NVIDIA's Triton/TensorRT-LLM — allows new requests to join a batch and completed requests to leave it dynamically, token by token, dramatically improving GPU utilization and reducing average latency relative to static batching, at real implementation complexity that's exactly why dedicated inference- serving software (rather than a hand-rolled batching loop) is the practical, standard choice.
Inference servers and model serving
An inference server (vLLM, NVIDIA Triton, TensorRT-LLM, or a cloud provider's managed model-serving offering) is purpose-built software for serving a model efficiently: it implements continuous batching, manages the KV cache efficiently (vLLM's PagedAttention technique, for instance, specifically addresses KV-cache memory fragmentation), and exposes an API your application calls — this is the layer that sits between raw GPU hardware and your FastAPI/LangGraph application when you self-host, analogous in role (though very different in what it actually does internally) to how Uvicorn sits between raw sockets and your FastAPI application code.
External LLM API vs. self-hosting — the actual decision framework
EXTERNAL API (Anthropic, OpenAI, etc.) — the default for most AI FDE work:
+ No infrastructure to manage, provision, or capacity-plan for GPUs
+ Access to frontier-capability models without owning hardware
+ Pay-per-token, scales automatically with usage
- Data leaves your (or your customer's) environment, a real
consideration for strict data-residency/compliance requirements
(Part 9.6, 18.19, 18.23)
- No control over model version changes/deprecation timing (Part
2.6/18.8's version-pinning discussion)
- Subject to the provider's own rate limits (18.12's reliability
concern) and outages
SELF-HOSTING — a genuine, situation-dependent alternative, not a
strictly worse option:
+ Full control over data flow (nothing leaves your infrastructure) —
the single most common REAL reason a customer requires this
+ No per-token cost to an external provider (replaced by GPU
infrastructure cost — see 18.18's economics)
+ No dependency on an external provider's availability/rate limits
- Real, ongoing operational burden: GPU provisioning, inference-
server operation, model updates, capacity planning for VRAM/
throughput (this entire chapter's content becomes YOUR
responsibility, not a vendor's)
- Almost certainly a lower-capability model than the frontier
external-API options, unless the specific task doesn't need
frontier capability (a real, legitimate case — many production
use cases are well-served by a smaller, self-hosted, fine-tuned
or well-prompted model)
- Real upfront and ongoing GPU cost, which for LOW, bursty traffic
volumes is very often MORE expensive than pay-per-token API
pricing, not less (18.18 works through this trade-off with numbers)The decision genuinely depends on the specific requirement driving it — data residency/compliance is the most common legitimate driver for self-hosting in real FDE engagements; "we want to save money" is a far more situation-dependent claim that needs the actual volume/utilization math (18.18) before being accepted at face value, since GPU infrastructure sitting underutilized is very often more expensive than external API calls at moderate volume, not less.
Concurrency, rate limits, and embeddings infrastructure
For an external API, concurrency is governed by the provider's rate limits (Part 7.9/7.11, 18.12's resilience patterns) — your own infrastructure doesn't directly bound this. For self-hosted serving, concurrency is bounded by GPU VRAM (how many concurrent requests' KV caches fit) and compute throughput together — a genuinely different capacity-planning exercise requiring load testing against your specific model, hardware, and typical request shape (context length, output length) rather than a rate-limit number you can simply read from documentation. Embeddings infrastructure (Part 3.4's embedding models) follows the same external-API-vs-self-hosted decision, generally at a smaller GPU-footprint scale than a full generative LLM, since embedding models are typically much smaller than frontier generative models — often making self-hosted embeddings a more practical starting point for an organization exploring self-hosting at all, before committing to self-hosting a full generative model.
5. Simple mental model
Calling an external LLM API is like ordering from a restaurant — you get excellent food without owning a kitchen, paying per meal, subject to the restaurant's hours and menu. Self-hosting is building your own kitchen — real, ongoing cost and skill required (staffing the kitchen, i.e. operating the inference server; buying equipment, i.e. GPUs), but complete control over ingredients (data never leaving your building) and no per-meal markup — worthwhile specifically when that control is a genuine, non-negotiable requirement, not simply because owning a kitchen "sounds more capable."
6. Real-world example
A customer in a regulated industry says their compliance team will not approve any architecture where document content leaves their AWS account, ruling out an external LLM API entirely for the Enterprise AI Assistant's document-Q&A feature. The concrete infrastructure translation: an open-weights model, quantized (section 4) to fit a chosen GPU instance type's VRAM, served via vLLM on an EC2 GPU instance (18.3) inside the customer's own VPC (18.5) — with a real, upfront capacity-planning exercise (expected concurrent users × typical context length → required VRAM and instance count) replacing the "just call the API" simplicity the rest of this track has assumed, and a real, explicit conversation about the resulting model-capability trade-off relative to the frontier external-API alternative the customer would otherwise have used.
7. Architecture diagram
External API (default):
App (ECS/EKS)
LLM Provider's own infrastructureno GPU infrastructure of your own at all
Self-hosted (compliance/data-residency driven):
AppECS/EKS, private subnet (18.5)
Inference server (vLLM/Triton)on GPU instance(s), private subnet, SAME VPC — no external network path for the model call at all
8. Production considerations
- Capacity-plan self-hosted serving from VRAM first, not just raw GPU compute throughput — VRAM is very often the binding constraint on concurrent request capacity (section 4).
- Use a purpose-built inference server (vLLM, Triton) rather than a hand-rolled serving loop — continuous batching and KV-cache management are non-trivial to reimplement correctly and are exactly where these frameworks' value lies.
- Evaluate quantization's actual quality impact empirically (Part 8) for your specific use case before adopting it in production — the impact is real and use-case-dependent, not a fixed, universal number.
- Treat "self-host to save money" claims skeptically until the actual utilization math (18.18) is done — low/bursty volume very often favors external API pricing.
9. Common mistakes
- Assuming self-hosting is strictly more capable or more cost-effective than an external API, without doing the actual capacity/cost analysis.
- Under-provisioning VRAM by capacity-planning from a single request's memory footprint without accounting for concurrent requests' KV-cache growth.
- Hand-rolling a batching loop instead of adopting an existing, mature inference-serving framework, then being surprised by poor GPU utilization relative to what the framework would have achieved.
- Treating quantization's quality impact as either negligible or unacceptable without actually measuring it against your own evaluation suite (Part 8).
10. Security considerations
Self-hosted model infrastructure inherits the full security posture of this entire track (18.2/18.5's private-subnet placement, 18.3's IAM, 18.19's broader security chapter) — plus a self-hosting-specific consideration: the model weights themselves may be a protected artifact (licensing terms, or a fine-tuned model containing information derived from proprietary training data) requiring their own access controls, distinct from the data flowing through the model at inference time.
11. Performance considerations
Continuous batching (section 4) is the single largest lever on self-hosted serving throughput and GPU utilization — the difference between a naive serving setup and a properly-configured inference server here is often not a small percentage but a multiple.
12. Cost considerations
This chapter's entire self-hosting-vs-API decision is fundamentally a cost-and-control question, worked through with real numbers in 18.18 — the short version worth internalizing now: GPU infrastructure has real, continuous cost whether or not it's fully utilized (unlike pay-per-token API pricing, which scales exactly with usage), making utilization rate the single most important variable in whether self-hosting is actually cheaper for a given workload.
13. When to use it
Self-hosting is worth its real operational cost specifically when data residency/compliance genuinely requires it (the most common legitimate driver), when a smaller, well-suited model at high, steady volume makes the cost math favorable (18.18), or when no external provider offers a capability the use case specifically needs.
14. When NOT to over-apply it
Don't recommend self-hosting to a customer as a default or as a "more sophisticated" architecture choice — for the large majority of AI FDE engagements, an external API remains the right default given its dramatically lower operational burden, and self-hosting should be a deliberate response to a specific, real requirement, not a default posture.
15. Alternatives and trade-offs
A middle ground worth naming explicitly: some cloud providers offer managed, provisioned-throughput hosting of specific open-weights models (committing to a fixed capacity at a fixed price) — reducing some of self-hosting's operational burden (you're not managing the inference server or GPU provisioning yourself) while still keeping data within a chosen cloud environment, a real, intermediate trade-off between the two poles this chapter has framed, worth investigating against a specific customer's actual requirement rather than assuming it's strictly "external API" or "fully self-hosted" with nothing in between.
16. Practical example — a back-of-envelope VRAM capacity estimate
python
"""
A rough, illustrative VRAM budgeting calculation for self-hosting —
illustrative of the REASONING, not a precise formula; actual model
memory characteristics vary by architecture and should be verified
against the specific model/serving-framework's own documentation.
"""
def estimate_vram_gb(
model_params_billion: float,
bytes_per_param: float, # e.g. 2 for fp16, ~0.5-1 for 4-8 bit quantization
concurrent_requests: int,
kv_cache_gb_per_request: float, # grows with context length; measure empirically
) -> float:
weights_gb = model_params_billion * bytes_per_param
kv_cache_total_gb = concurrent_requests * kv_cache_gb_per_request
overhead_gb = 2.0 # activation memory, framework overhead — a rough buffer
return weights_gb + kv_cache_total_gb + overhead_gb
# A 7B-parameter model, 4-bit quantized, serving 20 concurrent requests:
estimate_vram_gb(
model_params_billion=7,
bytes_per_param=0.5, # 4-bit ≈ 0.5 bytes/param
concurrent_requests=20,
kv_cache_gb_per_request=0.5, # illustrative — measure for your real model
)
# ≈ 3.5 + 10 + 2 = 15.5 GB — informs which GPU instance type can actually
# serve this workload; ALWAYS validate against real load testing, not
# this estimate alone, before committing to a specific instance type.The function's structure — separating model-weight memory from per-request KV-cache memory explicitly — is the point: it makes visible exactly why concurrent request count, not just model size, drives VRAM requirements, directly operationalizing section 4's capacity-planning argument.
17. Production-quality example — reasoning through a self-hosting decision memo
markdown
# Self-Hosting Decision Memo — Enterprise AI Assistant, Document Q&A
## Requirement driving this analysis
Customer's compliance team requires document content never leave their
AWS account (confirmed: not solvable via a Bedrock/regional-endpoint
option that keeps data in-region but still leaves the account boundary
— verify current provider offerings before finalizing this conclusion).
## Options considered
1. External API via a regional/VPC-scoped offering (if the provider
offers one meeting the account-boundary requirement) — PREFERRED
if available, since it avoids this chapter's full operational burden.
2. Self-hosted open-weights model, quantized, on GPU instances inside
the customer's own VPC.
## Capacity estimate (self-hosting, if required)
[Section 16's calculation, validated against real load testing]
## Cost estimate
[18.18's framework — GPU instance cost at expected utilization,
compared against equivalent external-API token cost at expected volume]
## Recommendation
[Stated explicitly, with the specific requirement and trade-off that
drove it — never presented as a default architecture choice]This memo structure is itself the practical FDE deliverable this chapter prepares you to produce — grounded in section 4's actual technical trade-offs, not a generic architecture recommendation.
18. Short exercise
Using section 16's estimation function, compute the VRAM requirement for a larger model (13B parameters) at higher concurrency (50 concurrent requests) and identify which real GPU instance types (checking current AWS EC2 GPU instance specifications) would and wouldn't have sufficient VRAM — then identify what quantization level would be needed to fit a smaller, more cost-effective instance type instead.
19. Interview questions
- Why is GPU VRAM, not raw compute throughput, often the binding constraint on self-hosted model-serving capacity?
- What is continuous/in-flight batching, and why does it improve GPU utilization over static batching?
- Walk through the real trade-offs between an external LLM API and self-hosting, and what specific requirement would justify each.
- Why might "self-host to save money" be wrong for a low-volume workload?
20. FDE/customer scenario
A customer says: "We want to self-host our own LLM instead of using an API, to save money." A strong response doesn't accept this premise at face value — it asks for the customer's actual expected request volume and concurrency, walks through the GPU capacity and utilization math (section 16, 18.18) against equivalent external-API token cost at that same volume, and presents the real, calculated trade-off — which, for many realistic volumes, favors the external API on cost grounds alone, with self-hosting justified instead by a genuine data-residency or control requirement rather than the originally-stated cost motivation.
Key takeaways
- GPU VRAM — not raw compute throughput — is very often the binding constraint on self-hosted model-serving capacity, and must be capacity-planned from concurrent-request KV-cache growth, not just model size alone.
- Continuous batching (vLLM, Triton) is what makes self-hosted serving GPU-efficient; a hand-rolled serving loop leaves substantial utilization (and therefore cost-efficiency) on the table.
- The external-API-vs-self-hosting decision should be driven by a real requirement (data residency, a specific capability gap) — not accepted as a cost-saving move without doing the actual utilization math first.
Things you should be able to explain
- Why GPUs, not CPUs, are used for LLM inference, mechanically.
- What determines GPU VRAM requirements for serving a model at real concurrency.
- The genuine trade-offs between external API usage and self-hosting.
Things you should be able to build
- A back-of-envelope VRAM capacity estimate for a self-hosting scenario.
- A structured self-hosting decision memo grounded in an actual requirement and cost/capacity analysis.
Common mistakes
- Assuming self-hosting saves money without doing the utilization math.
- Capacity-planning VRAM from model size alone, ignoring concurrent KV-cache growth.
- Hand-rolling model serving instead of using a mature inference server.
Recommended next chapter
17-genai-production-architecture.md