18.10 — Kubernetes Troubleshooting

1. What is it?

A systematic troubleshooting process for the Kubernetes failure modes an AI FDE actually encounters in practice — pods that won't start, pods that keep restarting, services traffic can't reach, and resource-pressure symptoms — each diagnosed with the same disciplined, symptom-to-fix structure, building directly on 18.9's concepts.

2. Why does it exist?

18.9 taught what Kubernetes objects mean when everything works. This chapter exists because "everything works" is not the steady state of a real production system — pods restart, images fail to pull, health checks fail — and an AI FDE needs a repeatable process for turning a vague symptom ("the deployment isn't working") into a specific, fixable cause, the same discipline 18.1 built for a raw Linux host, now for Kubernetes.

3. What problem does it solve?

It solves "something is wrong with my Kubernetes deployment and I don't know where to look" — replacing panic or random guessing with the same systematic, evidence-based process throughout: symptom → possible causes → what to check → command/tool → expected result → interpretation → fix → prevention.

4. How does it work internally?

The core diagnostic commands, and how to read their output

kubectl get pods                    # quick status: is it Running, Pending,
                                      # CrashLoopBackOff, ImagePullBackOff?
kubectl describe pod <pod>          # events, conditions, resource requests —
                                      # THE single most useful troubleshooting
                                      # command; read the Events section first
kubectl logs <pod>                  # the container's stdout/stderr
kubectl logs <pod> --previous       # logs from the PREVIOUS instance —
                                      # essential for a pod that already
                                      # restarted, since --logs alone shows
                                      # only the CURRENT (possibly log-less,
                                      # freshly-restarted) instance
kubectl exec -it <pod> -- /bin/sh   # a shell inside the container, for
                                      # direct inspection (18.1's tools apply
                                      # identically once you're inside)
kubectl top pod                      # live CPU/memory usage per pod
                                      # (requires metrics-server, 18.9)
kubectl get events --sort-by=.lastTimestamp   # cluster-wide recent events,
                                      # useful when you don't yet know
                                      # which pod/resource is involved
kubectl rollout status deployment/<name>   # is a rollout stuck?

kubectl describe pod's Events section is where the large majority of real diagnoses start — it records exactly what the scheduler and kubelet did and why (Failed to pull image, Insufficient cpu, Liveness probe failed, OOMKilled) in plain, specific language, and should always be the first thing checked, before reading application logs.

Incident: CrashLoopBackOff

SYMPTOM: kubectl get pods shows STATUS: CrashLoopBackOff, RESTARTS
         climbing.

POSSIBLE CAUSES:
  - Application crashes immediately on startup (a bug, a missing
    required environment variable, a bad config)
  - OOMKilled repeatedly (18.1/18.9's cgroup-OOM, at pod scale)
  - Liveness probe failing before the app finishes starting
    (18.9's liveness-vs-readiness confusion)
  - A dependency (database, another service) is unreachable at
    startup and the app doesn't retry/backoff, crashing instead

WHAT TO CHECK / COMMAND:
  kubectl describe pod <pod>            # check Events + "Last State"
  kubectl logs <pod> --previous         # the CRASHED instance's own output

EXPECTED RESULT / INTERPRETATION:
  - "Last State: Terminated, Reason: OOMKilled" → memory limit issue
    (18.1's confirmation pattern, at the pod level)
  - "Liveness probe failed" in Events, with the app's OWN logs showing
    it was still initializing → probe misconfiguration (18.9)
  - Application stack trace in --previous logs → an actual application
    bug or misconfiguration, not an infrastructure problem at all

FIX:
  - OOMKilled: raise the memory limit (after ruling out a genuine leak)
  - Liveness probe too aggressive: increase initialDelaySeconds, or
    switch to a startupProbe (delays liveness checks until an initial
    startup check first succeeds — the correct fix for a slow-starting
    AI service loading a local model/tokenizer, 18.1)
  - Dependency unreachable: fix the dependency, or add retry/backoff
    to the application's startup sequence

PREVENTION:
  - A startupProbe for any service with meaningful startup time
  - Memory limits set from REAL observed usage under load (18.1), not
    a guess

Incident: ImagePullBackOff

SYMPTOM: kubectl get pods shows STATUS: ImagePullBackOff or ErrImagePull.

POSSIBLE CAUSES:
  - Image tag doesn't exist in the registry (a typo, or CI pushed to
    the wrong tag)
  - The node's pull credentials/IAM role (18.3) lack ECR permissions
  - The image is in a private registry the cluster isn't authorized
    to pull from
  - Registry is unreachable from the node (a networking/NAT issue, 18.2)

WHAT TO CHECK / COMMAND:
  kubectl describe pod <pod>   # Events shows the EXACT pull error message

EXPECTED RESULT / INTERPRETATION:
  - "manifest unknown" → the tag genuinely doesn't exist — check what
    CI actually pushed (18.8)
  - "unauthorized" / "authentication required" → an IAM/ECR permission
    problem (18.3), not a missing image
  - Timeout reaching the registry → a networking problem (18.2), check
    NAT Gateway health if nodes are in a private subnet

FIX: correct the tag, fix the IAM policy scoping ECR pull permissions
     to the node/pod role, or fix the network path to the registry.

PREVENTION: CI should verify the pushed tag is pullable as part of the
            pipeline itself (18.8), not assume a successful push means
            a successful future pull.

Incident: application can't connect to the database

SYMPTOM: Application logs show connection timeouts/refused errors to
         RDS/ElastiCache (18.4).

POSSIBLE CAUSES:
  - Security group doesn't allow the pod's traffic (18.2/18.5's
    security-group-by-ID pattern misconfigured or missing)
  - Wrong endpoint/hostname in a ConfigMap/Secret (Part 7.4)
  - Database itself is down or at max connections (a distinct incident,
    covered in 18.21)
  - Pod is in a subnet with no route to the database's subnet at all
    (a VPC/route-table misconfiguration, 18.5)

WHAT TO CHECK / COMMAND:
  kubectl exec -it <pod> -- sh -c "nc -zv <db-host> 5432"
       # tests raw TCP reachability from INSIDE the pod's actual
       # network context — the most direct way to separate a
       # networking problem from an application/credentials problem
  kubectl exec -it <pod> -- env | grep DATABASE_URL
       # confirm the pod actually has the config you THINK it has

EXPECTED RESULT / INTERPRETATION:
  - nc connection times out → networking (security group, route table,
    or NACL) — move to 18.2/18.5's checklist
  - nc connects but the app still fails → likely credentials or the
    database itself (wrong password, database at max connections)
  - DATABASE_URL is wrong/missing entirely → a ConfigMap/Secret
    misconfiguration, unrelated to networking at all

FIX: correct the security group / ConfigMap / Secret as diagnosed above.

PREVENTION: a readiness probe that specifically checks DB connectivity
            (not just "is the process alive") catches this before the
            pod is marked ready for traffic at all.

Incident: service cannot reach pod

SYMPTOM: Requests to a Service's DNS name/IP fail or time out, even
         though `kubectl get pods` shows Running pods.

POSSIBLE CAUSES:
  - Service selector labels don't match the pod's actual labels
    (a very common, purely typo-driven bug)
  - Pods are Running but not Ready (readiness probe failing) — a
    Service only routes to READY pods, by design
  - Wrong targetPort in the Service spec (points at a port the
    container isn't actually listening on — 18.1's bind-address point,
    now at the Kubernetes-Service level)

WHAT TO CHECK / COMMAND:
  kubectl get endpoints <service-name>
       # THE key command: if this is EMPTY, the Service has no pods
       # it considers valid targets — even though pods exist and are
       # Running, they're either non-matching or not Ready

EXPECTED RESULT / INTERPRETATION:
  - Empty endpoints, pods exist and are Running → check
    `kubectl get pods --show-labels` against the Service's selector —
    mismatch is the most common cause
  - Empty endpoints, pods are Running but 0/1 READY → a readiness
    probe issue (see the CrashLoopBackOff incident's probe guidance)

FIX: correct the selector/label mismatch, or fix why readiness fails.

PREVENTION: use `kubectl apply` with manifests under version control
            (18.6's IaC discipline applied to Kubernetes YAML) so
            label mismatches are caught in review, not production.

Incident: high memory usage / CPU throttling / failing health checks

These map directly onto 18.1's OS-level explanations, now observed at the pod level:

kubectl top pod                              # current usage vs. what?
kubectl describe pod <pod> | grep -A3 Limits # what IS the configured limit?
kubectl describe pod <pod> | grep -i throttl # CPU throttling evidence,
                                               # if the container runtime
                                               # surfaces it in events

High memory usage approaching the limit → either a genuine leak (needs application-level investigation, Part 1.1's tooling) or an under- provisioned limit relative to real steady-state need (18.1's "know your real RSS before setting a limit" principle). CPU throttling with pods otherwise healthy → exactly 18.1's CFS-bandwidth explanation, now visible as intermittent latency spikes with no corresponding sustained high-CPU% reading — the fix is raising the CPU limit or reducing per-request CPU work, not chasing a phantom "sometimes slow" bug in application code. Failing health checks → apply the liveness-vs-readiness diagnostic from the CrashLoopBackOff incident above; the two failure modes (restart vs. traffic removal) look different in kubectl get pods output and point to different fixes.

Incident: deployment stuck / traffic not reaching new version

SYMPTOM: kubectl rollout status hangs, or a new Deployment's pods exist
         but users still see old behavior.

POSSIBLE CAUSES:
  - New pods failing readiness (a rolling update won't proceed past
    the configured maxUnavailable/maxSurge if new pods never become
    Ready — this is a SAFETY feature, not a bug in the rollout itself)
  - An old, cached image tag mismatch (18.7/18.8's environment-
    promotion point: did CI actually push a NEW tag, or reuse an old one)
  - A CDN/browser cache serving stale frontend assets — worth explicitly
    ruling out for a UI-facing symptom before assuming a backend issue

WHAT TO CHECK / COMMAND:
  kubectl rollout status deployment/<name>
  kubectl describe deployment <name>   # look at Conditions, and the
                                         # ReplicaSet events

FIX: fix why new pods fail readiness (apply the CrashLoopBackOff/
     readiness diagnostics above to the NEW ReplicaSet's pods
     specifically), or confirm and fix the image tag actually deployed.

PREVENTION: canary deployment (18.8) surfaces this kind of problem on
            5% of traffic instead of discovering it after a full rollout
            attempt stalls.

5. Simple mental model

kubectl describe pod's Events section is Kubernetes' own incident log, written by the system itself as things happen — troubleshooting Kubernetes well is mostly the discipline of reading that log before guessing, the same "check vitals before reading application code" order 18.1 established for a raw Linux host.

6. Real-world example

A customer reports the Enterprise AI Assistant "stopped responding" after a routine deploy. kubectl get pods shows the new ReplicaSet's pods stuck at 0/1 Ready, old pods still serving traffic (the rolling update correctly refused to proceed, section 4's "stuck deployment" incident). kubectl describe pod on a new pod's Events shows repeated Readiness probe failed: connection refused — kubectl logs on that same pod shows the application waiting on a database migration that hadn't been applied yet in this environment. The fix (apply the pending migration) and the prevention (a CI gate verifying migrations are applied before a new version's pods are expected to be ready, 18.8) both follow directly from having read the actual evidence rather than guessing "maybe it's a networking issue" first.

7. Architecture diagram

kubectl get podsstatus: Running / Pending / CrashLoopBackOff / ImagePullBackOff?

kubectl describe podEvents section — READ THIS FIRST

OOMKilled18.1's memory/cgroup-OOM diagnosis

Image pull errorregistry/IAM/network diagnosis

Probe failedliveness vs readiness diagnosis

Scheduling failedresource/GPU/node-capacity diagnosis

kubectl logs [--previous]application-level evidence

kubectl exec -it ...direct inspection (18.1's tools, inside the pod)

8. Production considerations

• Always check kubectl describe pod's Events before reading application logs — it frequently answers the question outright.
• Use --previous logs for any pod that has already restarted — current logs from a freshly-restarted pod are often nearly empty.
• Use startupProbe for any AI service with meaningful, variable startup time, rather than tuning liveness probe delays as a workaround.

9. Common mistakes

• Reading current (not --previous) logs on a crash-looping pod and concluding "there's nothing in the logs" when the crash evidence is in the previous instance's logs instead.
• Assuming a Service problem is a networking problem before checking kubectl get endpoints — an empty endpoints list from a label mismatch or failed readiness check is unrelated to actual network connectivity.
• Debugging CPU throttling by looking for a code-level performance bug, without first checking the CPU limit against actual usage (kubectl top plus kubectl describe).

10. Security considerations

kubectl exec into a production pod is itself a privileged, auditable action — RBAC (18.9) should restrict who can exec into which namespaces, and cluster audit logging (analogous to CloudTrail, 18.5, but for the Kubernetes API specifically) should record when it happens, directly relevant to a customer's audit-log requirement (18.23).

11. Performance considerations

kubectl top gives point-in-time usage; for understanding a trend (is memory usage climbing toward the limit over hours, indicating a leak, versus stable) you need the metrics pipeline 18.11 builds, not repeated manual kubectl top polling.

12. Cost considerations

Diagnosing a genuine resource under-provisioning (frequent OOMKilled events, sustained CPU throttling) versus over-provisioning (usage far below requests/limits, discovered the same way) both directly inform the right-sizing decisions 18.18's cost chapter quantifies.

13. When to use it

Any time a Kubernetes-hosted AI service isn't behaving as expected — this chapter's incidents cover the large majority of real, recurring cases an AI FDE encounters operating such a system.

14. When NOT to over-apply it

Don't reach for deep Kubernetes debugging when the actual problem is one layer up (the AI system's reasoning/quality, Part 8) or one layer down (18.1's raw host issues, still relevant when a node itself is unhealthy) — matching the right chapter's diagnostic process to the actual failure layer.

15. Alternatives and trade-offs

A managed observability platform with Kubernetes-aware dashboards (18.11) can surface many of these symptoms visually and faster than manual kubectl commands — a worthwhile investment for a mature system, though the manual commands in this chapter remain essential for the moments a dashboard itself is unavailable or insufficiently detailed (18.1's parallel point about OS fundamentals applies identically here).

16. Practical example — a troubleshooting decision script

#!/usr/bin/env bash
# k8s-triage.sh — first-response snapshot for a misbehaving Deployment.
set -euo pipefail
DEPLOYMENT="${1:?Usage: k8s-triage.sh <deployment-name> [namespace]}"
NAMESPACE="${2:-default}"

echo "== Pod status =="
kubectl get pods -n "$NAMESPACE" -l app="$DEPLOYMENT"

echo "== Recent events =="
kubectl get events -n "$NAMESPACE" --sort-by=.lastTimestamp | tail -20

echo "== Rollout status =="
kubectl rollout status deployment/"$DEPLOYMENT" -n "$NAMESPACE" --timeout=5s || true

echo "== Endpoints =="
kubectl get endpoints "$DEPLOYMENT" -n "$NAMESPACE" 2>/dev/null || echo "No matching service"

17. Production-quality example — describing and interpreting real output

$ kubectl describe pod ai-assistant-7d9f6-abcde
...
Events:
  Type     Reason     Age                From     Message
  ----     ------     ----               ----     -------
  Normal   Scheduled  2m                 default-scheduler  Successfully assigned...
  Normal   Pulled     2m                 kubelet  Container image already present
  Normal   Created    2m                 kubelet  Created container api
  Normal   Started    2m                 kubelet  Started container api
  Warning  Unhealthy  90s (x3 over 110s) kubelet  Readiness probe failed: Get
           "http://10.0.11.23:8000/health/ready": dial tcp: connect: connection refused
  Warning  BackOff    30s (x2 over 60s)  kubelet  Back-off restarting failed container

Interpretation, read in order: the image pulled and the container started successfully (ruling out 18.7/18.3's image/registry problems entirely) — the failure is specifically the readiness probe getting connection refused, meaning the application process itself never actually opened port 8000 successfully, not merely "isn't ready yet." Combined with kubectl logs --previous showing a stack trace on startup, this correctly points to an application-level startup crash (a missing environment variable, most commonly), not an infrastructure, networking, or Kubernetes-configuration problem at all — the value of reading Events in order is exactly this: it progressively rules out entire categories of possible cause before you've read a single line of application code.

18. Short exercise

Using the CrashLoopBackOff incident's structure (section 4), work through what kubectl describe pod and kubectl logs --previous would each show for three different underlying causes — an OOMKilled pod, a pod whose liveness probe is misconfigured, and a pod with a genuine application bug — and write down specifically how you'd tell them apart from the output alone, without guessing.

19. Interview questions

• Walk through your process for diagnosing a CrashLoopBackOff pod.
• Why would kubectl get endpoints show an empty result even when kubectl get pods shows healthy, Running pods?
• What's the difference in symptoms between a memory limit that's too low and a genuine memory leak, and how would you tell them apart?
• Why is kubectl logs --previous sometimes essential and kubectl logs alone insufficient?

20. FDE/customer scenario

A customer's on-call engineer reports: "Our AI service's pods keep restarting and we don't know why." Without yet having cluster dashboard access, you're given kubectl access to one namespace. Walk through this chapter's exact process live — kubectl get pods, kubectl describe pod's Events section first, kubectl logs --previous — narrating each finding to the customer's engineer as you go (Part 12.3's communication discipline, applied under incident pressure, exactly as 18.1 modeled for a raw host).

---

Key takeaways

• kubectl describe pod's Events section is the single most valuable troubleshooting artifact in Kubernetes — read it before application logs.
• kubectl logs --previous is essential for any pod that has already restarted; current logs from a freshly-restarted pod are often empty.
• An empty kubectl get endpoints result — not a networking problem — is the most common real cause of "service can't reach pod," usually from a label mismatch or a failed readiness check.

Things you should be able to explain

• The specific, distinguishing evidence for OOMKilled vs. a probe misconfiguration vs. a genuine application crash.
• Why a stuck rolling deployment is often a safety feature working as intended, not a bug.

Things you should be able to build

• A first-response Kubernetes triage script (section 16).
• A systematic diagnosis, from raw kubectl describe/logs output, of which of this chapter's incidents is actually occurring.

Common mistakes

• Reading current logs instead of --previous on a crash-looping pod.
• Assuming a connectivity symptom is networking before checking endpoints.
• Chasing a code-level performance bug that's actually CPU throttling.

Recommended next chapter

11-observability-for-ai-infrastructure.md