7.4 — Kubernetes Concepts for AI Systems

1. What is it?

Kubernetes (K8s) is a container orchestration system — it automates deploying, scaling, healing, and networking containers (Part 7.1) across a cluster of machines. This chapter covers the concepts you need to reason about an AI backend running on Kubernetes, not a full operational Kubernetes administration course.

2. Why does it exist?

Part 7.1 established that Docker packages an application into a portable container. Running one container on one machine is simple; running many containers, across many machines, that need to scale up/down with demand, recover automatically from failures, and be discovered/networked correctly by each other, is a genuinely hard distributed-systems problem. Kubernetes exists to solve this orchestration problem once, as reusable infrastructure, rather than every team hand-building their own cluster-management tooling.

3. What problem does it solve?

For an AI backend specifically, it solves "how do I run enough instances of my service to handle real production load (Part 7.5's scaling discussion), automatically replace instances that crash or become unhealthy, and roll out new versions (Part 7.2's CI/CD output) without downtime" — at the scale and reliability enterprise customers expect from a production system.

4. How does it work internally?

Core objects

• Pod: the smallest deployable unit — one or more tightly-coupled containers sharing network/storage, usually just one container (your AI backend) per pod in practice.
• Deployment: manages a set of identical pod replicas, handling rolling updates (gradually replacing old-version pods with new-version pods, Part 7.2's deployment stage) and self-healing (automatically replacing a crashed pod).
• Service: a stable network endpoint routing traffic to a set of pods (which come and go as they scale/restart) — solving the problem that individual pods have ephemeral, changing IP addresses.
• Horizontal Pod Autoscaler (HPA): automatically adjusts the number of pod replicas based on observed metrics (CPU, memory, or custom metrics like request queue depth) — the direct mechanism for Part 7.5's horizontal scaling.
• ConfigMap/Secret: externalize configuration and sensitive values from the container image itself, directly implementing Part 7.1's "never bake secrets into an image" principle at the orchestration layer.

Kubernetes Cluster

HPA

Service"ai-backend-svc"

Kubernetes Cluster

Deployment"ai-backend"

Kubernetes Cluster

Podsreplicas

Readiness and liveness probes — how Kubernetes knows a pod is actually healthy

Kubernetes needs a way to know whether a pod is genuinely ready to receive traffic (readiness probe) and whether it's still functioning correctly and shouldn't be restarted (liveness probe) — typically implemented as the exact health-check endpoint Part 1.3/7.1 recommended building. A pod that's "running" (the process hasn't crashed) but not actually healthy (e.g., its database connection pool is exhausted) should fail its readiness probe so Kubernetes stops routing new traffic to it, even though it's not unhealthy enough to warrant a full restart (liveness failure) — this distinction matters for graceful degradation under partial failure rather than an all-or-nothing healthy/dead model.

PodDisruptionBudgets — protecting availability during voluntary disruptions

The rolling-update strategy (maxUnavailable/maxSurge, section 17) governs how Kubernetes replaces pods during a deployment you triggered. A PodDisruptionBudget (PDB) protects availability during a genuinely different category of event: voluntary disruptions initiated by cluster operations rather than your own deployment — a node drain for maintenance, a cluster version upgrade, or Kubernetes' own cluster-autoscaler removing an underutilized node. Without a PDB, a node drain is free to evict every pod scheduled on that node simultaneously, regardless of how many of your service's replicas that happens to be — a Deployment with 3 replicas could, in the worst case, have all 3 scheduled on the same node and lose all of them at once during a routine drain, even though nothing about your own rolling-update configuration was violated (rolling update simply doesn't apply here; nobody deployed anything).

A PDB declares a floor: the minimum number (or percentage) of pods that must remain available at all times, and Kubernetes' eviction API enforces it by refusing to evict a pod if doing so would violate the budget, forcing the node drain (or other voluntary disruption) to proceed more slowly, one pod at a time, rather than all at once.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ai-backend-pdb
spec:
  minAvailable: 2   # or: maxUnavailable: 1 — pick one, not both
  selector:
    matchLabels:
      app: ai-backend

With minAvailable: 2 on a 3-replica deployment, a node drain touching two of those pods' nodes is forced to evict them one at a time, always keeping at least 2 pods serving traffic — the exact same "don't take down more capacity than the service can absorb" principle the rolling-update strategy applies to your own deployments, now applied to disruptions you don't control the timing of at all.

Why this matters specifically for stateful, checkpointed AI workflows (Part 5.3)

A critical, easy-to-miss interaction: Kubernetes pods are inherently ephemeral and can be terminated/replaced at any time (during scaling, a rolling update, or node maintenance) — this is precisely why Part 5.3 argued so strongly for durable, shared checkpointing (PostgresSaver) rather than in-memory or single-instance state for any LangGraph-based workflow deployed on Kubernetes: a long-running agent's state must survive its specific pod being terminated and its work potentially resuming on an entirely different pod, which is exactly the guarantee PostgresSaver provides and MemorySaver/SqliteSaver do not.

5. Simple mental model

Kubernetes is like an automated building superintendent for a large apartment complex of identical units — it makes sure the right number of units (pods) are occupied and functioning based on demand (autoscaling), automatically has maintenance replace a unit that develops a problem (self-healing), and directs new tenants (requests) to a unit that's actually ready and functioning (via the Service and readiness probes) rather than one currently being renovated (unhealthy/not-ready) — all without a human superintendent manually checking every unit constantly.

6. Real-world example

An AI FDE deploys a customer support backend on the customer's existing Kubernetes cluster (a very common real deployment target, Part 14). During a rolling update deploying a new prompt version (Part 7.2's output), Kubernetes gradually replaces old-version pods with new-version pods one at a time, only routing traffic to new pods once they pass their readiness probe — meaning the deployment causes zero downtime and no dropped requests, and if the new version's readiness probe never passes (e.g., a misconfiguration preventing startup), Kubernetes can be configured to automatically halt the rollout and keep the old, working version serving traffic, rather than a bad deployment silently taking down the whole service.

7. Architecture diagram

See section 4's core-objects diagram — this chapter's architecture is that Deployment/Pod/Service/HPA relationship, directly underlying the horizontal scaling (Part 7.5) and CI/CD deployment (Part 7.2) this Part has built toward.

8. Production considerations

• Implement genuinely meaningful readiness/liveness probes (section 4) — a probe that only checks "is the process running" misses the more important question "is this pod actually able to serve requests correctly right now," and a probe that's too aggressive (failing on transient, self-recoverable conditions) can cause unnecessary pod churn.
• Use PostgresSaver or equivalent durable, shared state for any LangGraph workflow deployed on Kubernetes (section 4's critical interaction with Part 5.3) — pod ephemerality makes this non-negotiable, not optional hardening.
• Set explicit resource requests and limits per pod (Part 1.8's cgroups discussion, now at the Kubernetes scheduling layer) — this is what lets Kubernetes schedule pods appropriately across nodes and prevents one pod from starving others sharing the same physical node.
• Configure rolling update strategy deliberately (max unavailable, max surge) to balance deployment speed against risk — a very aggressive rollout speed increases the blast radius if a new version has an undiscovered problem.

9. Common mistakes

• Shallow health checks that don't actually verify the specific dependencies (database connection, LLM API reachability) a pod needs to function correctly, giving false confidence that a pod is healthy when it can't actually serve requests properly.
• Deploying a LangGraph-based workflow with MemorySaver onto Kubernetes, then being surprised when in-progress workflows are lost every time a pod is rescheduled (a routine, expected Kubernetes event, not a rare failure) — precisely Part 5.3's warning, made concrete by Kubernetes's normal operational behavior.
• Not setting resource requests/limits, leading to unpredictable scheduling behavior and one pod's resource usage affecting others on the same node.

10. Security considerations

• Kubernetes Secrets (for API keys, database credentials, Part 9.5) have their own access-control model (RBAC, Part 9.4) that must be configured correctly — a Kubernetes Secret is more secure than an environment variable baked into an image (Part 7.1) but is not automatically encrypted at rest in every cluster configuration by default; verify your specific cluster's secret-encryption configuration rather than assuming it.
• Network policies (Kubernetes's native mechanism, or a service mesh) implement the least-privilege network segmentation from Part 7.3's VPC discussion at the pod-to-pod level within the cluster, not just at the cluster's external network boundary.

11. Performance considerations

• Autoscaling reaction time (how quickly HPA responds to increased load by adding pods) has a real lag — for AI workloads with bursty traffic patterns, ensure your autoscaling configuration and, where relevant, pre-warmed minimum replica counts account for this lag rather than assuming instantaneous scale-up.
• Pod startup time (including your application's own initialization, e.g., establishing DB connection pools) adds to the effective time before a newly-scaled pod can serve traffic — a slow-starting application reduces the practical responsiveness of autoscaling.

12. Cost considerations

• Over-provisioned resource requests (requesting more CPU/memory than a pod actually needs) waste cluster capacity and cost, since Kubernetes schedules based on requested, not actual, resource usage — right-size these based on real observed usage (Part 1.8's monitoring discussion) rather than generous guesses.
• Autoscaling configuration directly controls the cost/responsiveness trade-off — scaling down aggressively during low traffic saves cost but risks slower response to a sudden traffic spike; tune deliberately against your actual traffic patterns and cost sensitivity.

13. When to use it

Production AI backends needing genuine horizontal scalability, self-healing, and zero-downtime deployment — the standard choice for enterprise-scale deployments, and a very common target given how many enterprise customers already operate Kubernetes clusters (Part 14).

14. When NOT to use it

A small-scale deployment, a single-tenant simple application, or an early-stage prototype may not need Kubernetes's operational complexity — a simpler managed container-run service (Part 7.3) can meet the same functional needs with substantially less configuration and operational overhead for a workload that doesn't yet need Kubernetes's full feature set.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Kubernetes	Full-featured orchestration, scaling, self-healing, wide enterprise adoption	Real operational complexity/learning curve
Simpler managed container platforms	Less operational overhead for straightforward deployments	Less flexibility/control for complex, custom orchestration needs
Single-VM/simple deployment (no orchestration)	Simplicity for very small-scale needs	No automated scaling, self-healing, or zero-downtime deployment

16. Practical Python/code example

A FastAPI health-check endpoint (Part 1.3, Part 7.1) designed to serve as a genuinely meaningful Kubernetes readiness probe, per section 8's recommendation:

from fastapi import FastAPI, Response, status

app = FastAPI()


@app.get("/ready")
async def readiness_check(response: Response) -> dict:
    """
    Checks the specific dependencies this pod needs to serve requests correctly,
    not just whether the process is running — used as the Kubernetes readiness probe.
    """
    checks = {
        "database": await check_database_connection(),
        "llm_provider_reachable": await check_llm_provider_health(),
    }
    if not all(checks.values()):
        response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE
    return {"ready": all(checks.values()), "checks": checks}

17. Production-quality example

A Kubernetes Deployment manifest incorporating this chapter's production recommendations (resource limits, meaningful probes, rolling-update strategy):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-backend
spec:
  replicas: 3
  strategy:
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    spec:
      containers:
        - name: ai-backend
          image: registry.internal/ai-backend:abc1234
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1000m"
              memory: "1Gi"
          readinessProbe:
            httpGet:
              path: /ready
              port: 8000
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 30
          envFrom:
            - secretRef:
                name: ai-backend-secrets
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ai-backend-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: ai-backend

The PodDisruptionBudget (section 4) is a distinct object from the Deployment's own rolling-update strategy above it — the rolling-update fields govern disruptions you trigger by deploying a new version; the PDB governs disruptions the cluster triggers (a node drain, a cluster upgrade) that your own deployment configuration has no say over at all. A production manifest missing the PDB is protected during your own deployments but still fully exposed to losing multiple replicas at once during routine cluster maintenance.

18. Short exercise

A team's Kubernetes deployment uses a liveness probe that checks the same /ready endpoint as its readiness probe (checking database and LLM provider connectivity). During a brief, transient LLM provider outage, all pods fail this shared probe and get restarted repeatedly, making the outage worse. Explain why using the same check for both readiness and liveness caused this, and describe the fix.

19. Interview questions

• Explain the difference between a Deployment, a Pod, and a Service in Kubernetes, and how they relate to each other.
• Why does pod ephemerality make durable, shared checkpointing (Part 5.3) non-negotiable for a LangGraph workflow deployed on Kubernetes?
• What's the difference between a readiness probe and a liveness probe, and why using the same overly-strict check for both can make a transient issue worse rather than better?

20. FDE/customer scenario

Customer's platform team: "We already run Kubernetes for everything — can your AI service just deploy into our existing cluster?"

This is a common, favorable scenario in real FDE engagements — the answer is generally yes, provided the service is properly containerized (Part 7.1), configured with meaningful health probes and appropriate resource limits, and — critically, per this chapter's core warning — any stateful, multi-step LangGraph workflow (Part 5) is configured with a durable, shared checkpointer rather than an in-memory one, since the customer's existing cluster will treat your pods exactly like any other workload, including routine, expected termination and rescheduling events that an improperly-configured stateful workflow would not survive.

---

Key takeaways

• Kubernetes automates scaling, self-healing, and zero-downtime deployment for containerized services — solving a genuinely hard distributed-systems problem as reusable infrastructure.
• Pod ephemerality is a normal, expected operational behavior, not a rare failure — this makes durable, shared state (Part 5.3's PostgresSaver) non-negotiable for any stateful AI workflow deployed on Kubernetes.
• Readiness and liveness probes serve different purposes and should check different things, at different strictness levels.

Things you should be able to explain

• The relationship between Deployment, Pod, Service, and HPA.
• Why pod ephemerality makes in-memory/single-instance checkpointing unsafe on Kubernetes.

Things you should be able to build

• A meaningful, dependency-checking readiness endpoint and a production Deployment manifest with appropriate resource limits and probe configuration.

Common mistakes

• Shallow health checks that don't verify actual dependencies.
• Deploying stateful LangGraph workflows with non-durable checkpointers onto Kubernetes.
• Using the same overly-strict check for both readiness and liveness probes.

Recommended next chapter

05-scaling-load-balancing.md