AI FDE Handbook

Appearance

18.9 — Kubernetes for AI Engineers ​

Relationship to Part 7.4: Part 7.4 already gave you a working Deployment/Service/ConfigMap/Secret YAML set with liveness/readiness probes and resource limits for an AI service on EKS. This chapter assumes that and builds the surrounding pieces it didn't cover: the cluster/node/pod/container relationship explained properly, namespaces, StatefulSets, Jobs/CronJobs for batch AI workloads, PodDisruptionBudgets, GPU scheduling, RBAC, Helm, and HPA explained correctly (including what it actually needs to scale on anything beyond CPU/memory). 18.10 covers troubleshooting these once deployed.

1. What is it? ​

Kubernetes is a system for running and managing containers (18.7) across a cluster of machines — deciding which machine runs which container, restarting failed containers, routing traffic to healthy ones, and scaling the number of running instances based on demand. This chapter teaches it first from the perspective of an application engineer placing an AI service onto it, then from the perspective of the infrastructure underneath, per the source spec's explicit ordering.

2. Why does it exist? ​

18.3 already showed ECS as a simpler way to run containers reliably. Kubernetes exists because some organizations need capabilities ECS doesn't provide as directly — a large, complex multi-service system with intricate scheduling needs, a desire for cloud portability (running identically on AWS, Azure, GCP, or on-premises, 18.20), or — very commonly in FDE work — the customer already standardized on Kubernetes before you arrived (18.23's "we already use Kubernetes" scenario), making it the platform you must integrate with rather than choose independently.

3. What problem does it solve? ​

Beyond what 18.3's ECS discussion already covers (reliably running containers, restarting failures, load balancing), Kubernetes specifically solves: portability across cloud providers and on-prem, a rich scheduling model (placing GPU workloads on GPU nodes specifically, respecting affinity/anti-affinity rules), and a large ecosystem (Helm charts, operators, service meshes) that ECS doesn't have an equivalent of at the same scale.

4. How does it work internally? ​

The cluster → node → pod → container relationship ​

Cluster
Control planeAPI server, scheduler, etcd — usually managed by EKS (18.3)
Nodea worker machine — EC2 instance or Fargate
Podsmallest deployable unit — one or more containers sharing network namespace/volumes
Container18.7's container, running inside the pod

A pod — not a container directly — is what Kubernetes schedules onto a node. Most AI service pods run exactly one container, but a pod can run multiple containers sharing localhost networking and volumes — the sidecar pattern (a logging agent, or a service-mesh proxy, running alongside your FastAPI container in the same pod) is the main reason this matters in practice.

Deployments, ReplicaSets, and Services (brief, per Part 7.4) ​

A Deployment manages a ReplicaSet, which ensures a specified number of pod replicas are running, replacing any that fail — this is Part 7.4's territory in full detail. A Service gives a stable network identity (a DNS name, a virtual IP) to a set of pods selected by label, so other pods (or an Ingress) can reach "the API" without tracking individual, ephemeral pod IPs as pods are replaced. Ingress is the Kubernetes-native way to route external HTTP traffic into the cluster (roughly analogous to 18.5's ALB, often implemented using an ALB via the AWS Load Balancer Controller on EKS specifically).

Namespaces ​

A namespace partitions a cluster into logical groups — commonly one per environment (staging, production) or per team/tenant. Namespaces provide a scoping boundary for names (two api Deployments can coexist in different namespaces) and for RBAC policies (below) — but namespaces alone are not a strong security/tenant-isolation boundary by default; network policies (this chapter) and node-level isolation are needed for that, a distinction directly relevant to Part 9.6's tenant-isolation reasoning applied at the Kubernetes layer specifically.

StatefulSets ​

A StatefulSet is for workloads needing a stable, unique identity per replica (a predictable pod name and, typically, its own persistent volume) that survives rescheduling — unlike a Deployment's interchangeable pods. For the Enterprise AI Assistant, RDS/ElastiCache are managed services (18.4) run outside Kubernetes, so a StatefulSet is less commonly needed for this specific architecture — but it becomes relevant if you're self-hosting a database or a vector database directly on Kubernetes (a real, if less common, architecture choice, revisited in 18.16) rather than using a managed AWS service.

Jobs and CronJobs — the right primitive for batch AI workloads ​

A Job runs a pod to completion (unlike a Deployment, which keeps pods running indefinitely) — the correct primitive for a one-off batch task: a bulk document re-embedding run after upgrading an embedding model, or a one-time data-migration script. A CronJob runs a Job on a schedule — a nightly evaluation-suite run (Part 6.2/8.1) against production traffic samples, or a periodic cache-warming job, are both natural CronJob fits. Using a long-running Deployment for what's actually a finite, batch task (and then manually tracking whether it "finished") is a common Kubernetes anti-pattern this distinction directly avoids.

Probes, resource requests/limits — the connection to 18.1 ​

Part 7.4 already showed liveness/readiness probe YAML; the underlying mechanism is worth being precise about, connecting back to 18.1: a liveness probe failing causes Kubernetes to restart the container (useful for "the process is stuck, kill and restart it"); a readiness probe failing removes the pod from a Service's routable endpoints without restarting it (useful for "the process is fine but not ready to serve traffic yet," e.g., a startup phase loading a local model/tokenizer into memory, 18.1's memory-consumer point). Conflating these — using a liveness probe for a slow-starting AI service — causes Kubernetes to repeatedly kill and restart a pod that just needed more time to start, never actually reaching a healthy state (a real, common, self-inflicted crash loop, revisited in 18.10). Resource requests are what the scheduler uses to decide which node has room for a pod; resource limits are enforced by the node's cgroups (18.1/18.7) — a memory limit that's too low causes the exact cgroup-OOM behavior 18.1 explained, now at the Kubernetes-pod level specifically, surfaced as OOMKilled in kubectl describe pod (18.1's confirmation command, directly applicable here).

PodDisruptionBudgets — a gap worth closing explicitly ​

A PodDisruptionBudget (PDB) limits how many replicas of a workload can be voluntarily disrupted at once (during a node drain for a cluster upgrade, for instance) — without one, Kubernetes could legitimately evict all replicas of your AI API simultaneously during routine cluster maintenance, causing an entirely avoidable outage. This matters specifically because pod ephemerality (Part 7.4's core theme) is not just about crashes — planned, routine cluster operations also terminate and reschedule pods, and a PDB is the concrete mechanism ensuring enough replicas stay up throughout.

yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ai-assistant-pdb
spec:
  minAvailable: 2          # or maxUnavailable: 1 — either form works
  selector:
    matchLabels:
      app: ai-assistant

Graceful shutdown — terminationGracePeriodSeconds and preStop ​

When a pod is terminated (a deploy, a scale-down, a node drain), the default sequence is: the pod is removed from Service endpoints (so no new requests are routed to it), then SIGTERM is sent to the container, then Kubernetes waits up to terminationGracePeriodSeconds (default 30 seconds) before sending SIGKILL if the container hasn't exited. For a streaming AI response (Part 5.5's streaming, 18.11) that can legitimately take longer than 30 seconds to finish, the default grace period can truncate an in-flight response mid-stream — the fix is raising terminationGracePeriodSeconds explicitly and ensuring the application actually handles SIGTERM by finishing in-flight requests and refusing new ones, rather than exiting immediately or ignoring the signal entirely (18.1's PID-1 signal-handling point from 18.7 applies directly here too). A preStop hook adds a deliberate delay beforeSIGTERM is even sent — commonly used to give a load balancer's own health-check/deregistration delay time to catch up, avoiding a brief window where traffic is still routed to a pod that's already begun shutting down.

yaml
spec:
  terminationGracePeriodSeconds: 60   # enough for a long streaming response
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "10"]   # let LB deregistration catch up

Horizontal Pod Autoscaler (HPA) — what it actually needs ​

The HPA scales replica count based on a metric. Out of the box, Kubernetes' metrics-server provides CPU and memory metrics, so an HPA scaling on CPU utilization works with no additional components. Scaling on anything else — request rate, queue depth (18.13's better-fit signal for an I/O-bound AI service, per 18.1's CPU-isn't-the-right-signal point) — requires an additional metrics adapter: the Prometheus Adapter (exposing Prometheus-collected custom metrics to the HPA API) or KEDA (Kubernetes Event-Driven Autoscaling, which can scale directly on external signals like SQS queue depth, 18.4/18.15, without needing Prometheus in between for that specific case). This is a genuine, concrete gap worth stating precisely: "the HPA can scale on queue depth" is true only once one of these adapters is actually installed and configured — it is not a Kubernetes built-in capability by default.

yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-assistant-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai-assistant
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    # A request-concurrency or queue-depth metric here requires KEDA or
    # the Prometheus Adapter to be installed — NOT available by default.

GPU scheduling ​

A pod requesting a GPU declares it as a resource (nvidia.com/gpu: 1), and the NVIDIA device plugin (installed on GPU nodes) makes GPUs visible to the Kubernetes scheduler as a schedulable resource, the same way CPU/memory are — the scheduler then places the pod only on a node with an available, matching GPU. This matters directly for 18.16's self-hosted-model scenario: a GPU-serving pod's YAML looks like an ordinary Deployment except for this resource request, but requires GPU- enabled nodes and the device plugin to exist in the cluster at all — a CPU-only EKS node group will never satisfy this pod's scheduling requirement, and it will sit Pending indefinitely (18.10 covers diagnosing exactly this).

yaml
resources:
  limits:
    nvidia.com/gpu: 1

RBAC in Kubernetes ​

Kubernetes RBAC (distinct from, but conceptually parallel to, Part 9.4's application-level RBAC) controls which identities (users, or a pod's own ServiceAccount) can perform which actions (get, list, create, delete) on which Kubernetes resources (pods, secrets, deployments) — a Role (namespace-scoped) or ClusterRole (cluster-wide) grants permissions, bound to an identity via a RoleBinding/ClusterRoleBinding. The least-privilege principle (18.3's IAM-role reasoning, applied identically here) means an application pod's ServiceAccount should be scoped to exactly the Kubernetes API actions it needs (often none at all, for a typical stateless AI API that doesn't call the Kubernetes API itself) — not the cluster-admin-equivalent default some clusters grant too permissively out of convenience.

Helm ​

Helm packages a set of Kubernetes YAML manifests as a versioned, parameterized chart — the Kubernetes-ecosystem analogue of a Terraform module (18.6): the same chart deploys differently to staging vs. production via a values.yaml file, rather than maintaining duplicated, drifting copies of raw YAML per environment. Helm is not required to use Kubernetes at all (raw kubectl apply -f works fine for simple cases) but becomes valuable once you're managing the same application across multiple environments or need to install third-party software (many open-source tools, including KEDA and the Prometheus Adapter above, distribute as Helm charts) that expects to be installed this way.

5. Simple mental model ​

If ECS (18.3) is a hotel with a single, integrated front desk, Kubernetes is an entire small city's infrastructure — more powerful and more flexible (it can host far more varied kinds of buildings and services), but requiring you to understand zoning (namespaces), building codes (RBAC), utility hookups (Services/networking), and city planning (scheduling, including GPU-specific zoning) rather than simply checking in at a front desk.

6. Real-world example ​

A customer already runs Kubernetes (18.23's common scenario) and wants the Enterprise AI Assistant deployed onto their existing EKS cluster rather than a new ECS setup. The concrete translation: the FastAPI + LangGraph service becomes a Deployment (Part 7.4) with a PDB (this chapter) and a properly-tuned terminationGracePeriodSeconds for its streaming responses; the document-ingestion batch reprocessing becomes a CronJob rather than a long-running service; autoscaling on request concurrency (rather than misleading CPU%, per 18.1's point) requires confirming whether the customer's cluster already has KEDA or the Prometheus Adapter installed — a concrete, necessary discovery question (Part 12.1's discipline, applied to a technical constraint) before promising queue-depth-based autoscaling will simply work.

7. Architecture diagram ​

Users
Ingress / ALB18.5
EKS Control Planemanaged by AWS, 18.3
Worker Nodes
Node (CPU) → Pod: apiDeployment
Node (CPU) → Pod: apiDeployment
Node (GPU) → Pod: modelGPU request

8. Production considerations ​

  • Set terminationGracePeriodSeconds explicitly for any service with long-running requests (streaming AI responses) — the 30-second default silently truncates them.
  • Always define a PDB for a production workload — without one, routine cluster maintenance can evict every replica at once.
  • Confirm whether HPA scaling on anything beyond CPU/memory actually has the required metrics adapter installed before designing around it.
  • Use Jobs/CronJobs for finite batch AI work (re-embedding runs, scheduled evaluations) instead of a long-running Deployment you manually track completion for.

9. Common mistakes ​

  • Using a liveness probe (which triggers a restart) where a readiness probe (which only affects traffic routing) was actually needed — causing a slow-starting AI service to crash-loop unnecessarily.
  • Assuming HPA can scale on queue depth or request rate "because Kubernetes supports custom metrics," without checking whether the Prometheus Adapter or KEDA is actually installed in this specific cluster.
  • No PDB, discovering the gap only when a routine node upgrade causes an unexpected full outage.
  • A GPU-requesting pod stuck Pending because the cluster has no GPU node group or the NVIDIA device plugin isn't installed — a scheduling constraint mistaken for an application bug (18.10 covers diagnosing this).

10. Security considerations ​

  • Namespaces alone are not a tenant-isolation boundary (Part 9.6's principle, applied here) — real isolation needs network policies and, for strong requirements, dedicated node pools or separate clusters.
  • Scope pod ServiceAccounts to least privilege (section 4's RBAC point) — most application pods need zero Kubernetes API permissions at all.
  • Secrets stored as Kubernetes Secret objects are only base64-encoded, not encrypted, by default at the etcd storage layer unless encryption at rest is explicitly enabled for the cluster — a detail worth verifying rather than assuming, especially for a customer's own, possibly differently-configured cluster (18.23).

11. Performance considerations ​

Pod startup time (image pull, application init, e.g. loading a local tokenizer, 18.1/18.7) directly affects how fast HPA-driven scale-up can actually add usable capacity during a traffic spike — a slow-starting pod means autoscaling reacts more slowly than the metric alone suggests, revisited concretely in 18.13.

12. Cost considerations ​

Running your own EKS cluster (control plane fee, plus worker node capacity, versus ECS's simpler pricing model, 18.3) is a real, additional cost and operational-complexity layer — appropriate when Kubernetes's specific capabilities (portability, GPU scheduling flexibility, an existing customer standard) are actually needed, not by default.

13. When to use it ​

When you need Kubernetes-specific capabilities (GPU scheduling flexibility, multi-cloud portability, an existing organizational standard) — 18.3's decision framework still applies as the first filter.

14. When NOT to over-apply it ​

Don't adopt Kubernetes for a simple, single-service AI application "to be more scalable" when ECS (18.3) would meet the same requirements with meaningfully less operational overhead — a recurring, real over- engineering trap.

15. Alternatives and trade-offs ​

ECS (18.3) vs. EKS remains the primary alternative comparison — this chapter's added Kubernetes-specific capabilities (Jobs/CronJobs, GPU scheduling, Helm's ecosystem, RBAC granularity) are the concrete reasons to accept EKS's added complexity when they're actually needed.

16. Practical example — a CronJob for scheduled evaluation ​

yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-eval-suite
spec:
  schedule: "0 3 * * *"   # 3 AM daily
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: eval-runner
              image: <ecr-repo>/ai-assistant-eval:latest
              command: ["python", "scripts/run_eval_gate.py"]  # 18.8's script
          restartPolicy: Never
      backoffLimit: 2

17. Production-quality example — a Deployment with PDB, graceful shutdown, and HPA together ​

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-assistant
spec:
  replicas: 3
  selector: { matchLabels: { app: ai-assistant } }
  template:
    metadata: { labels: { app: ai-assistant } }
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: api
          image: <ecr-repo>/ai-assistant:1.4.2
          resources:
            requests: { cpu: "500m", memory: "512Mi" }
            limits:   { cpu: "1",    memory: "1Gi" }
          readinessProbe:
            httpGet: { path: /health/ready, port: 8000 }
            initialDelaySeconds: 5
          livenessProbe:
            httpGet: { path: /health/live, port: 8000 }
            initialDelaySeconds: 15
          lifecycle:
            preStop:
              exec: { command: ["sleep", "10"] }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: ai-assistant-pdb }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: ai-assistant } }
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: ai-assistant-hpa }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: ai-assistant }
  minReplicas: 3
  maxReplicas: 15
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }

Separate readinessProbe/livenessProbe paths (/health/ready vs. /health/live) reflect section 4's point precisely: they check genuinely different things and should not share one endpoint with one meaning.

18. Short exercise ​

Take the Deployment in section 17 and modify it to run on a GPU node (section 4's GPU scheduling), then trace through what would happen — using 18.10's methodology, previewed here — if the cluster has no GPU node group at all: what would kubectl get pods and kubectl describe pod actually show, and how would you distinguish this from an application crash?

19. Interview questions ​

  • What's the actual difference between a liveness and a readiness probe, and what goes wrong if you use the wrong one?
  • What does an HPA need beyond Kubernetes' defaults to scale on queue depth, and why?
  • Why does a PDB matter even if your application never crashes?
  • Walk through the cluster → node → pod → container relationship and where GPU scheduling fits into it.

20. FDE/customer scenario ​

A customer asks: "Can Kubernetes automatically scale our AI API based on how many requests are queued, not just CPU?" A strong answer states the real requirement precisely — yes, but only with KEDA or the Prometheus Adapter installed and configured against the actual queue metric (SQS depth, 18.4/18.15, or a custom Prometheus metric) — then asks whether either is already present in their cluster, rather than promising a capability Kubernetes doesn't provide by default and discovering the gap during implementation.

Key takeaways ​

  • A pod, not a container, is Kubernetes' scheduling unit — understanding this distinction clarifies sidecars, shared volumes, and shared networking.
  • HPA scaling beyond CPU/memory requires an additional component (KEDA or the Prometheus Adapter) that is not installed by default — a concrete, frequently-assumed-away gap.
  • PodDisruptionBudgets and correctly-tuned graceful shutdown (terminationGracePeriodSeconds/preStop) protect against planned disruption, not just crashes — an easy, consequential thing to omit.

Things you should be able to explain ​

  • The cluster → node → pod → container relationship.
  • Liveness vs. readiness probes, and the failure mode of confusing them.
  • What HPA actually needs to scale on a non-CPU/memory metric.

Things you should be able to build ​

  • A Deployment with correctly-tuned graceful shutdown, a PDB, and an HPA.
  • A CronJob for a scheduled batch AI workload (evaluation, re-embedding).
  • A GPU-scheduled pod spec, and a diagnosis of why it might stay Pending.

Common mistakes ​

  • Liveness probes used where readiness probes were needed.
  • Assuming HPA can scale on queue depth without the required adapter.
  • No PDB, discovered only during a routine cluster maintenance outage.

10-kubernetes-troubleshooting.md