Appearance
18.1 — Linux for Production AI Systems
Track note: This opens a new, additional track — Cloud, DevOps, and AI Infrastructure — that sits underneath everything Part 7 (Production AI Engineering) already taught. Part 7 assumed you could already operate a Linux host, read a stack trace on a remote server, and reason about a container's resource limits. This chapter (and this track) builds that assumption from first principles, then reconnects to Part 7/8/9/11 at every point where they already went deep on a shared topic — this chapter will cite those chapters rather than re-teach what they already cover well. The goal of this whole track is stated once, here, and applies throughout: you are not becoming a general DevOps engineer. You are becoming an AI engineer / FDE who can take an application to production without being helpless the moment something below the Python layer breaks.
Prerequisites — what you should already know
This chapter assumes Part 1.1's treatment of Python's asyncio event loop, coroutines, and the blocking-call anti-pattern — this chapter uses that model directly (a process's threads, from the OS's point of view) rather than re-explaining it. It assumes basic command-line comfort (navigating directories, running a program, reading a file) but no prior Linux administration or kernel knowledge. Where a concept requires kernel-level detail to be accurate, this chapter gives you the minimum correct model, not the full mechanism — deeper kernel internals are explicitly out of scope (section 14).
1. What is it?
Linux, in this chapter's scope, is the operating system your AI application actually runs on in production — whether that's a raw EC2 instance, an ECS task, a Kubernetes pod, or a Lambda's execution environment. This chapter covers the operating-system concepts (processes, memory, filesystems, permissions, networking primitives) and the command-line tools you use to observe and control them, taught specifically through the lens of debugging and operating a production AI service — not as a general Linux course.
2. Why does it exist?
Every chapter in Parts 1–17 assumed a working, healthy runtime environment. In the real world, that environment fails in specific, Linux-shaped ways: a process runs out of memory and gets killed, a port is already bound, a disk fills up with log files, a file has the wrong permissions after a deploy. An AI engineer who can write a correct LangGraph agent but freezes the moment sshing into a host that's serving 500 errors is not yet production-capable — this chapter closes that specific, common gap.
3. What problem does it solve?
It solves "my AI application is misbehaving in production and I need to find out why, using only the operating system's own tools, often before I have access to a nicer dashboard." Every AI-specific observability tool in Part 6/8 (LangSmith, tracing, SLIs) sits on top of a host that is itself an ordinary Linux machine — when the AI-specific tooling itself won't load, or the problem is beneath it (out of memory, out of disk, network unreachable), Linux fundamentals are what's left.
4. How does it work internally?
Processes, threads, and what "running" actually means
A process is a running instance of a program: it owns its own memory address space, file descriptors, and at least one thread of execution. A thread is a unit of execution within a process, sharing that process's memory. Your FastAPI + Uvicorn AI service is one process (or several, if you run multiple workers); inside it, Python's asyncio event loop schedules many concurrent coroutines onto a small number of OS threads (see Part 1.1's asyncio internals) — this is why "my server has 100 concurrent LLM calls in flight" does not mean 100 OS threads, and why CPU-bound work (Part 1.1's warning about blocking the event loop) can stall all of them at once even though the process looks "alive" the whole time.
Kernel
Process (PID 4821)uvicorn worker — heap, stack, loaded libraries, open file descriptors
Thread 1asyncio event loop
Thread 2threadpool worker (blocking I/O, e.g. sync DB driver)
Thread 3threadpool worker
Every process has a PID (process ID), a PPID (parent PID — who started it), and a state, shown by ps/top as a single letter: R (running or ready to run), S (sleeping, waiting on something — the common, healthy state for an idle async worker), D (uninterruptible sleep, almost always waiting on disk or network I/O — a process stuck in D state cannot even be killed with SIGKILL until the I/O completes, which is why a process that "won't die" is often a symptom of a stuck disk or NFS mount, not a bug in the process itself), Z (zombie), and T (stopped). A zombie process is one that has finished executing but whose exit status hasn't been collected by its parent yet — harmless in small numbers, a sign of a bug in the parent's process-management code if they accumulate (a common cause: a supervisor process that spawns worker subprocesses and never calls wait() on them).
A note on threads and the GIL: the diagram above shows a threadpool worker handling blocking I/O, but it is important not to over-read this as "Python threads give you parallelism." CPython's Global Interpreter Lock (GIL) means only one thread executes Python bytecode at a time, regardless of how many OS threads or CPU cores exist. Threads still help here specifically because a blocking I/O call (reading a socket, waiting on a synchronous DB driver) releases the GIL while it waits — letting the event loop's thread keep running — not because the threadpool achieves CPU parallelism. A CPU-bound task (heavy local tokenization, a large JSON parse, in-process embedding computation) handed to a Python thread does not run any faster for having more threads or more CPU cores available to it; that requires a separate process (multiprocessing, or a subprocess call) to actually escape the GIL. This distinction matters directly for capacity planning: adding more threads to an AI service fixes I/O-bound stalls, not CPU-bound ones.
CPU: what "100% CPU" actually measures, and why it matters differently for AI workloads
CPU usage is measured per-core, over a time window — "container at 380% CPU" on a 4-core container means it is nearly saturating all four cores. For a typical AI service that spends most of its wall-clock time waiting on an LLM API response (I/O-bound, not CPU-bound — Part 1.1's async rationale exists precisely because of this), sustained high CPU usually means something other than "the LLM is slow": synchronous JSON parsing of huge payloads, an accidentally-synchronous embedding call blocking the event loop (Part 1.1's blocking-call anti-pattern), or a busy-loop bug. CPU throttling is a distinct, container-specific failure mode: a container with a CPU limit (Part 7.4's Kubernetes resource limits) that briefly needs more CPU than its limit allows is not killed — it is throttled (paused) by the kernel's CFS (Completely Fair Scheduler) bandwidth controller, which shows up as intermittent latency spikes with no corresponding CPU-usage-at-100% signal if you're only sampling every few seconds, a genuinely confusing failure mode covered again in 18.10's Kubernetes troubleshooting chapter.
Memory: RSS, virtual memory, and the OOM killer
A process's memory footprint is usually reported as RSS (Resident Set Size — physical RAM actually in use) versus VSZ (Virtual Size — total address space reserved, which can be far larger than physical RAM and is usually not the number you care about). RSS itself has a caveat worth knowing before you trust it: it includes memory the process has mapped, which can include shared libraries also mapped by other processes — summing the RSS of every Python worker process on a host will typically overcount real physical memory used, since they share the same loaded interpreter and libraries. For an AI service, the memory budget typically has three distinct consumers worth telling apart when debugging: the Python process's own heap (request/response objects, loaded prompt templates), any in-process model or tokenizer artifacts (a loaded local embedding model or tokenizer vocabulary can be hundreds of MB — see 18.16's AI-infrastructure chapter), and connection/buffer memory (open HTTP connections to the LLM provider, database connection pools).
Two different OOM mechanisms exist, and conflating them is a common, consequential mistake. The system-wide Linux OOM killer activates when the entire host runs critically low on physical memory: it scans every process on the machine and kills the one with the worst heuristic "badness" score (roughly, highest memory use, adjustable via /proc/<pid>/oom_score_adj) to save the system as a whole — it does not care which process caused the pressure, only which one scoring worst right now. Container/cgroup OOM is a different, more common trigger in practice: a container's memory is capped by a cgroup memory.max limit (surfaced as a Docker --memory flag or a Kubernetes pod's memory limit, Part 7.4/18.9); when a container's own cgroup exceeds its own limit, the kernel kills a process inside that cgroup immediately — even if the host as a whole has plenty of free RAM sitting idle. This is the mechanism that actually fires in nearly all container/Kubernetes OOM incidents, and it explains a detail the system-wide killer wouldn't: a single container can be repeatedly OOM-killed on a host that is otherwise completely healthy.
In both cases, the kill itself is a SIGKILL (signal 9) — abrupt, with no opportunity for the process to clean up or log anything — which is why the container simply exits with code 137 (128 + 9) and no application-level error explaining why; the process was killed before it could write one. Don't stop at "exit code 137" as a diagnosis — confirm it. The kernel logs every OOM kill; check for it directly rather than assuming:
bash
dmesg | grep -i "killed process" # host-level: see the OOM killer's own log line
journalctl -k | grep -i "out of memory" # same, via journalctl
docker inspect <container> --format '{{.State.OOMKilled}}' # true/false, cgroup OOM
kubectl describe pod <pod> # look for "Last State: Terminated,
# Reason: OOMKilled" under the container statusThis confirmation step is what turns "I think it might have been OOM- killed" into a verified diagnosis — and it's also how you tell the two mechanisms apart: a dmesg line naming your process means host-wide pressure (a noisy-neighbor problem, possibly other processes on the same host, or a limit set too high relative to host capacity); OOMKilled: true on one specific container with a healthy host otherwise means that container's own memory limit is the thing to raise (after first ruling out a genuine memory leak) — the single most common reason a "why did my pod restart with no error in the logs" investigation (18.9/18.10) turns out to be a memory limit, not an application bug.
Disk and filesystems
Linux organizes storage as a single hierarchical filesystem tree rooted at /, with different physical or logical devices mounted at various points (/, /var, /tmp, a mounted EBS volume, etc.). For an AI service, the disk-related failure that recurs most often in practice is log files filling the disk — an application logging every LLM request/response payload at high volume, with no rotation, on a host with a small root volume, until df reports 100% used and the application (or the whole host) starts failing writes in ways that look unrelated to disk at first (a database that can't write its WAL, a temp-file-based library that can't create a scratch file). The fix is log rotation (logrotate, or a container runtime's own log-driver size limits) plus routing structured logs to a centralized system (18.11's observability chapter) rather than accumulating them locally at all.
A specific, genuinely confusing variant worth knowing by name: df -h reports the disk as full, but du -sh /* doesn't add up to anywhere near that much — usually because some process still holds an open file handle to a file that was deleted (e.g., a log file removed by a naive cleanup script instead of being rotated). Linux does not actually free a deleted file's disk space until every process holding it open closes that handle — du walks the visible directory tree and won't find a file with no directory entry, but the space is still consumed until the holding process exits or closes the descriptor. Find these with:
bash
lsof +L1 2>/dev/null # lists open files with a link count of 0
# (i.e., deleted but still held open)Restarting the offending process (freeing the handle) recovers the space immediately — a fix that looks unrelated to the symptom until you know this specific mechanism.
Permissions, users, and groups
Every file has an owner (a user) and a group, plus three permission triads (read/write/execute for owner/group/other), commonly seen as -rw-r--r-- or the octal form 644. In a containerized AI service, the two permission issues that come up constantly: (1) a container running as root by default (a security anti-pattern — Part 9's least-privilege principle applies identically to the OS layer, covered again in 18.19) when it should run as a dedicated non-root user; (2) a mounted volume (a config file, a credentials file, a model-weights directory) owned by a UID that doesn't match the container's runtime user, causing a Permission denied error that has nothing to do with the application code itself. Permissions and ownership in Linux are enforced by numeric UID/GID, not by username — a username is just a label /etc/passwd maps to a number; a container process running as "UID 1000" is compared against a file's owning UID (also just a number) even if the container has no /etc/passwd entry for 1000 at all, which is why a file can appear owned by an unresolvable "UID 1000" or "nobody" inside a container while working fine on the host. Diagnosing a permission error, as distinct from fixing it, means checking both sides of that comparison explicitly rather than guessing:
bash
id # what UID/GID is THIS process/shell running as?
ls -l /path/to/file # what UID/GID and mode does the FILE have?
stat /path/to/file # same, with more detail (numeric owner, mode)For a running container specifically, docker exec <container> id and docker exec <container> ls -l <path> answer the same two questions from inside the container's own view — which is often different from the host's view of the same mounted path, and is usually the faster way to find the mismatch than reasoning about it from the host side alone.
Environment variables, ports, and sockets
Environment variables are how a process receives configuration without it being baked into the container image (Part 7.1's twelve-factor-style config pattern) — an AI service's API keys, database URLs, and feature flags (Part 3.1's prompt-governance flags) are almost always injected this way. A port is a 16-bit number identifying a specific communication endpoint on a host. A socket is the actual OS-level object representing one end of a connection — precisely, the combination of protocol, local IP, local port, and (for an established connection) remote IP and remote port. A port by itself identifies nothing until it's paired with an address; a process "listening on port 8000" has actually created a listening socket bound to a specific IP address and that port, and which address it binds matters enormously in practice.
Binding to 127.0.0.1 (loopback) vs. 0.0.0.0 (all interfaces) is one of the most common, most confusing sources of "it works on my machine but nobody else can reach it." A process bound to 127.0.0.1 only accepts connections that originate from the same host — reachable via curl localhost on that machine, and nowhere else, no matter how the firewall or port-publishing is configured. A process bound to 0.0.0.0 accepts connections on that port arriving via any of the host's network interfaces, subject to whatever firewall/security-group rules exist (18.2). Many frameworks default to 127.0.0.1 for local development safety — Uvicorn's own default is 127.0.0.1 unless you pass --host 0.0.0.0 explicitly. Inside a container this is a very common, specific failure: the container's port is correctly published (-p 8000:8000) and the security group/firewall correctly allows it, and it still isn't reachable from outside the container, because the process inside is listening only on 127.0.0.1 — which, from the container's own network namespace, does not include traffic arriving from outside that namespace at all. This is exactly the kind of thing to check explicitly in step 2 of section 6's investigation, not assume.
"The application won't start" very often means "another process already has this port bound" (Address already in use) — the single most common first thing to check with lsof or ss (section 6 below) before assuming an application bug.
Ephemeral ports and TIME_WAIT matter for the outbound side of an AI service specifically. Every outbound TCP connection your service opens (to an LLM provider, a database) uses a locally-assigned, temporary source port from the OS's ephemeral port range (roughly 32,768–60,999 by default on Linux — verify against /proc/sys/net/ipv4/ip_local_port_range on a given host, since it's configurable). After a connection closes, the kernel holds that port in TIME_WAIT state for a period (typically around 60 seconds) before it can be reused, specifically to correctly handle any delayed packets from the closed connection. A service that opens a new short-lived outbound connection for every LLM API call instead of reusing a persistent HTTP connection (Part 1.1/1.2's connection-reuse guidance) can, under high enough concurrent request volume, exhaust its available ephemeral ports faster than they free up from TIME_WAIT — new outbound connections then fail with Cannot assign requested address, a genuinely confusing symptom that looks like an LLM-provider-side problem but is actually a local port-exhaustion issue solved by connection pooling/reuse, not by anything on the provider's end. Check for this specifically with:
bash
ss -tan state time-wait | wc -l # count of sockets currently in TIME_WAITProcesses vs. services, and systemd
A raw process started from a terminal dies when that terminal session ends (unless explicitly detached). A service is a process managed by an init system — on nearly all modern Linux distributions, systemd — which handles starting it at boot, restarting it if it crashes, capturing its logs, and managing its dependencies on other services. Running your AI application as a systemd service (versus a bare python app.py in a terminal) is one of the most basic production-readiness upgrades on a raw VM — though in practice, most AI FDE work runs inside containers/Kubernetes (18.9) where the container runtime and Kubernetes itself take over this restart-supervision role instead, and you interact with systemd more when debugging the host underneath a container runtime (e.g., is the Docker daemon itself running: systemctl status docker).
5. Simple mental model
Think of a Linux host the way you'd think of a single, very well-instrumented patient in an ICU: top/htop is the vitals monitor (CPU, memory, at a glance), ps is asking "who's actually in this room right now and what are they doing," df/du is checking how much space is left in storage, ss/netstat is checking which doors (ports) are open and to whom, and journalctl/logs are the patient's chart — a record of what happened and when. Production debugging is almost always: check vitals first (is it CPU, memory, or disk), then narrow to the specific process, then read its chart.
6. Real-world example
Scenario, exactly as the source spec poses it: "Your FastAPI AI application is running but users cannot connect to it. How would you investigate the problem?" Walked through step by step:
1. Is the process even running?
$ ps aux | grep uvicorn
→ If nothing: it crashed or never started. Check systemd/container
logs for why (journalctl -u myapp, or `docker logs`, or
`kubectl logs` — 18.10).
→ If it IS running: continue.
2. Is it listening on the port you expect, and on the right address?
$ ss -tlnp | grep 8000
→ If nothing is listening on 8000: the app may have crashed AFTER
partially starting, or be listening on a different port than
configured (a common env-var misconfiguration).
→ If something IS listening, check WHICH address it's bound to in
the output itself: "127.0.0.1:8000" means loopback-only — reachable
locally on the host but NOT from outside it (section 4's bind-
address point) — while "0.0.0.0:8000" means it's listening on all
interfaces and the problem is further down this list, not here.
3. Can you reach it FROM the host itself?
$ curl -v http://localhost:8000/health
→ If this works but external requests don't: the problem is
between the host and the outside world, not the application —
move to networking (18.2): security group / firewall rule,
load balancer health check misconfiguration, or DNS.
4. Is a firewall or security group blocking the port?
$ sudo iptables -L -n # host-level firewall
(cloud) check the security group / network ACL attached to the
instance — 18.3/18.5's exact territory.
5. If this is a container, is the port actually published/mapped?
$ docker ps # check the PORTS column: 0.0.0.0:8000->8000/tcp ?
A container's internal port not being published to the host (or
published to the wrong host port) produces exactly this symptom
with a perfectly healthy application inside the container.
6. Check the load balancer's own health check status (if applicable)
— a load balancer that has marked every target unhealthy will
correctly refuse to route traffic to them, which looks
IDENTICAL to "the app is down" from a user's perspective even
though every instance is actually running fine (18.4's exact
failure mode, and a very common real incident).This exact ordering — process → port binding → local reachability → firewall/security-group → container port mapping → load balancer — is the systematic debugging process the source spec asks for, and it generalizes: almost every "users can't connect" AI-application incident is one of these six layers, checked from the inside out.
7. Architecture diagram
SSH/exec
Linux Host
systemd / container runtime
Linux Host
uvicorn (PID)thread 1 ← event loop, thread 2 ← threadpool, socket:8000 (LISTEN)
Linux Host
/var/logdisk — rotation!
/prockernel's live state
SSH/exec
Linux Host
systemd / container runtime
Linux Host
uvicorn (PID)thread 1 ← event loop, thread 2 ← threadpool, socket:8000 (LISTEN)
Linux Host
/var/logdisk — rotation!
/prockernel's live state
8. Production considerations
- Never run a production AI service as a bare foreground process — use systemd, a container runtime, or Kubernetes (18.9) so a crash is detected and recovered from automatically rather than silently ending the service.
- Set explicit log rotation for any process writing local log files — an AI service logging full request/response bodies (common for debugging prompt issues, per Part 6.1) can fill a disk surprisingly fast at real traffic volume.
- Run application processes as a non-root user inside containers — covered again, with the security rationale, in 18.19.
- Budget memory explicitly and know your process's real RSS under representative load before setting a container/pod memory limit (18.9) — a limit set below real steady-state usage causes recurring OOM kills that look like application instability.
9. Common mistakes
- Assuming "the process is running" means "the process is healthy" — a process can be alive, listening, and still deadlocked or serving errors.
- Debugging "users can't connect" by immediately reading application code, skipping the systematic layer-by-layer check in section 6 — wastes time when the actual cause is a security group or a
127.0.0.1bind. - Not knowing the difference between a process's exit code 137 (OOM-killed) and a normal application crash — these require completely different fixes (raise the memory limit or fix a leak, versus fix a bug) — and not confirming WHICH OOM mechanism fired (host-wide vs. this container's own cgroup limit, section 4) before deciding what to change.
- Treating high CPU% as automatically meaning "the LLM call is the bottleneck" for an I/O-bound AI service, when it more often means a blocking, synchronous call stalling the event loop (Part 1.1).
- Assuming more Python threads will speed up a CPU-bound task — the GIL means they won't; only a separate process actually escapes it.
10. Security considerations
- Least privilege at the OS layer: application processes should run as a dedicated non-root user with only the file permissions they actually need — a compromised process (Part 9.2's excessive-agency logic applies identically to OS permissions) with root access can do far more damage.
- SSH access to production hosts should be key-based, logged (
journalctl,/var/log/auth.log), and — where the architecture allows it (18.3's Systems Manager Session Manager, for example) — avoid opening an inbound SSH port at all, reducing the network attack surface directly. - Secrets should never be stored as plain files with broad read permissions or committed into an image — Part 9.5 and 18.19 cover the full secrets- management picture; at the OS layer, the immediate rule is
600permissions (owner read/write only) on any local secret file that must exist at all.
11. Performance considerations
nice/ionicecan deprioritize non-critical background processes (a local batch embedding job, say) relative to the latency-sensitive API process sharing the same host — rarely needed once you're on Kubernetes with proper resource requests/limits (18.9), but relevant on raw VMs.- Connection reuse (keep-alive) at the OS/socket level matters for an AI service making many outbound HTTPS calls to an LLM provider — exhausting ephemeral ports under high concurrency is a real, if uncommon, failure mode worth knowing exists (
ss -sshows socket-state counts, includingTIME_WAITaccumulation).
12. Cost considerations
Right-sizing a host or container's CPU/memory allocation depends on actually knowing its real utilization — top/htop sampled under representative load is the cheapest possible cost-optimization tool available before reaching for cloud-provider cost-analysis dashboards (18.18's FinOps chapter); over-provisioned instances discovered this way are frequently the single largest, easiest cost win in an early FDE engagement.
13. When to use it
Every production AI deployment touches these fundamentals somewhere, whether directly (a raw EC2 instance) or indirectly (Kubernetes and containers are themselves built out of Linux process/namespace/cgroup primitives — 18.9 makes this connection explicit).
14. When NOT to over-apply it
You do not need to become a kernel engineer. This chapter's depth is calibrated to "can debug and reason about a production host," not "can tune kernel scheduler parameters" — the latter is a specialized skill outside this handbook's FDE-focused scope.
15. Alternatives and trade-offs
The "alternative" to knowing these fundamentals is depending entirely on higher-level dashboards (a cloud console, an APM tool) — these are valuable and should be your first stop in a mature system (18.11), but they routinely fail to explain why exactly when you need the answer most (the dashboard itself is unreachable, or shows a symptom with no root cause) — which is precisely when OS-level fundamentals become the only way forward.
16. Practical example — a Linux command reference for AI-service debugging
bash
# --- Process inspection ---
ps aux | grep uvicorn # find your app's process(es)
ps -o pid,ppid,%cpu,%mem,cmd -p <PID> # detail on one process
top # live view: CPU/mem per process
htop # nicer live view (if installed)
# --- Memory ---
free -h # total/used/free RAM, human-readable
cat /proc/<PID>/status | grep VmRSS # exact RSS for one process
# --- Disk ---
df -h # disk space per mounted filesystem
du -sh /var/log/* # find what's consuming space where
lsof +L1 2>/dev/null # deleted-but-still-open files holding space
# --- Network / ports ---
ss -tlnp # listening TCP sockets + owning process
# (check the BOUND ADDRESS: 127.0.0.1 vs 0.0.0.0)
lsof -i :8000 # what process holds port 8000
curl -v http://localhost:8000/health # local reachability check
ss -tan state time-wait | wc -l # check for ephemeral-port TIME_WAIT buildup
# --- Logs ---
journalctl -u myapp.service -f # follow a systemd service's logs live
journalctl -u myapp.service --since "10 min ago"
tail -f /var/log/myapp/app.log
# --- OOM confirmation ---
dmesg | grep -i "killed process" # host-level OOM killer's own log line
docker inspect <container> --format '{{.State.OOMKilled}}'
kubectl describe pod <pod> # look for "Reason: OOMKilled"
# --- Permissions ---
id # UID/GID this shell (or `docker exec ... id`) runs as
ls -l /etc/myapp/secrets.env # owner/group/mode on the file itself
chmod 600 /etc/myapp/secrets.env
chown appuser:appuser /etc/myapp/secrets.env
# --- Process control ---
kill -TERM <PID> # graceful shutdown request (SIGTERM)
kill -9 <PID> # forceful kill (SIGKILL) — last resort
systemctl restart myapp.service
systemctl status myapp.service17. Production-quality example — a first-response triage script
bash
#!/usr/bin/env bash
# triage.sh — first-response snapshot for "app is unreachable" incidents.
# Run this FIRST, before deep-diving — it answers the six layers from
# section 6 in one pass and is safe to run on a live production host
# (read-only; makes no changes).
set -euo pipefail
APP_NAME="${1:?Usage: triage.sh <systemd-service-name> <port>}"
PORT="${2:?Usage: triage.sh <systemd-service-name> <port>}"
echo "== 1. Is the process running? =="
systemctl is-active "$APP_NAME" || echo "NOT ACTIVE"
ps aux | grep -v grep | grep "$APP_NAME" || echo "No matching process found"
echo "== 2. Is it listening on the expected port? =="
ss -tlnp | grep ":$PORT" || echo "Nothing listening on port $PORT"
echo "== 3. Can we reach it locally? =="
curl -s -o /dev/null -w "HTTP %{http_code} in %{time_total}s\n" \
"http://localhost:$PORT/health" || echo "Local curl failed"
echo "== 4. Recent memory/CPU snapshot =="
ps aux --sort=-%mem | head -5
echo "== 5. Disk space =="
df -h / /var
echo "== 6. Last 20 log lines =="
journalctl -u "$APP_NAME" -n 20 --no-pagerThis script is intentionally read-only and fast — the point of a triage script during an active incident is to get an answer in seconds, not to be a comprehensive diagnostic tool; deeper investigation follows once you know which of the six layers to focus on.
18. Short exercise
Start any simple FastAPI app locally, bind it to 127.0.0.1 instead of 0.0.0.0, and then try to reach it from another device (or a container) on the same network. Use ss -tlnp to see the bound address, and reproduce in your own hands exactly why this specific misconfiguration produces "users can't connect" while the app itself is completely healthy — the single most common real-world instance of this chapter's section 6 scenario.
19. Interview questions
- Walk through your investigation process for "the API is running but users can't reach it."
- What does an exit code of 137 tell you, and what would you check next?
- What's the difference between a process being CPU-throttled and a process legitimately using 100% CPU, and why does it matter for an I/O-bound AI service specifically?
- Why might
topshow low CPU usage on a host serving a slow AI API, and what does that tell you about where the bottleneck actually is?
20. FDE/customer scenario
A customer's on-call engineer pages you: "the AI assistant is down, users are getting connection errors." You don't yet have dashboard access to their environment — only SSH access to one affected host, granted for the incident. Apply this chapter's six-layer process live, narrating each step and its result to the customer's engineer as you go (Part 12.3's layered- communication discipline, applied under incident pressure) — this is precisely the kind of moment where OS fundamentals, not a fancier tool you don't yet have access to, are what actually resolves the incident.
Key takeaways
- Production AI debugging routinely happens beneath the AI-specific tooling (LangSmith, APM dashboards) — at the level of processes, memory, disk, and ports — and this layer has to be usable without those tools.
- "Users can't connect" has a systematic, layered diagnosis (process → port binding → local reachability → firewall/security group → container port mapping → load balancer health) that resolves the large majority of real connectivity incidents quickly.
- Exit code 137 (OOM-kill) and CPU throttling are container-specific failure modes that look like application bugs but require infrastructure fixes, not code fixes — and the host-wide OOM killer and a container's own cgroup OOM are different mechanisms with different fixes, confirmed with different commands (
dmesgvs.docker inspect/kubectl describe). - Binding to
127.0.0.1instead of0.0.0.0is one of the single most common reasons a healthy, running, correctly-networked application is still unreachable from outside its own host or container.
Things you should be able to explain
- The difference between a process and a thread, and why an AI service's "100 concurrent requests" doesn't mean 100 OS threads — and why more Python threads don't speed up CPU-bound work (the GIL).
- Why a container can be OOM-killed with no application-level error log, and how to confirm whether it was the host-wide OOM killer or this container's own cgroup limit.
- The six-layer diagnostic order for a "can't connect" incident, including the
127.0.0.1vs.0.0.0.0bind-address check specifically.
Things you should be able to build
- A first-response triage script (section 17) for your own AI service.
- A working mental model of
ps/top/ss/df/journalctlyou can use live, under pressure, without looking up syntax. - A permission-mismatch diagnosis (
id+ls -l/staton both sides) and an OOM-kill confirmation (dmesg/docker inspect/kubectl describe), each as a repeatable two-command habit rather than a guess.
Common mistakes
- Reading application code first instead of checking the six layers.
- Confusing high CPU% with "the LLM is slow" for an I/O-bound service.
- Not knowing exit code 137 means OOM-killed, not "crashed" — and not checking which OOM mechanism actually fired before changing a limit.
- Assuming a published container port and an allowed firewall rule are enough, without checking the process is actually bound to
0.0.0.0.
Recommended next chapter
02-networking-fundamentals.md