18.7 — Docker for AI Systems: A Deep Dive

Relationship to Part 7.1: Part 7.1 already taught you how to write a production Dockerfile for an AI service (multi-stage builds, health checks, resource limits) — this chapter does not repeat that. Instead it goes underneath it: what docker build/run/push actually do internally, how image layers and the build cache really work, Docker networking and volumes in depth, Docker Compose for a full local AI development stack, and named container-security tooling Part 7.1 left generic. Read Part 7.1 first if you haven't.

1. What is it?

The internal mechanics of Docker — images as layered, content-addressed filesystems; containers as Linux processes running in isolated namespaces and cgroups (18.1's primitives, now assembled); and the networking/volume model that lets a multi-container AI stack (FastAPI, LangGraph, Postgres, Redis, a vector DB) run and communicate locally exactly as it will in production.

2. Why does it exist?

Part 7.1 gave you a working, production-quality Dockerfile. This chapter exists because "it works" and "you understand why it works, and what to do when it doesn't" are different levels of competence — and the gap between them is almost always in exactly the internals this chapter covers: why a build is slow (layer caching), why a container can't reach another container (networking), why data disappears on restart (volumes), and why an image built successfully but is a security liability nobody noticed (scanning).

3. What problem does it solve?

It solves "my Docker build takes 8 minutes and I don't know why," "my FastAPI container can't reach my Postgres container," "I restarted my local stack and lost my test data," and "is this image actually safe to run" — all real, recurring problems that a purely Dockerfile-syntax-level understanding (Part 7.1) doesn't equip you to diagnose.

4. How does it work internally?

What docker build actually does

A Docker image is a stack of read-only layers, each corresponding to one instruction in the Dockerfile that changes the filesystem (RUN, COPY, ADD) — layers below the ones that don't (ENV, LABEL, which add metadata, not filesystem changes, though they still create a layer entry). Each layer is content-addressed: its identity is a hash of its content, and Docker's build cache reuses a layer from a previous build if an identical instruction runs against an identical preceding layer state. This is precisely why Dockerfile instruction order matters for build speed: COPY requirements.txt . followed by RUN pip installbefore COPY . . (the rest of your application code) means an application-code-only change reuses the cached pip install layer entirely, while COPY . . first would invalidate that (and every subsequent) layer's cache on every single code change — turning a multi-minute dependency install into something that reruns on every build for no reason. For a typical Python AI service with heavy dependencies (langchain, langgraph, embedding/tokenizer libraries), this ordering decision alone is often the single largest lever on local iteration speed.

Dockerfile layer order that CACHES WELL:
  FROM python:3.12-slim
  COPY requirements.txt .          ← layer invalidated only when THIS file changes
  RUN pip install -r requirements.txt   ← reused across code-only changes
  COPY . .                          ← invalidated on every code change (expected,
                                       cheap — just a file copy, not a rebuild)

Dockerfile layer order that CACHES POORLY:
  FROM python:3.12-slim
  COPY . .                          ← invalidated on EVERY change, including a
                                       one-line README edit
  RUN pip install -r requirements.txt   ← re-runs on every build as a result,
                                       even when requirements.txt didn't change

docker build walks the Dockerfile instruction by instruction, checking the cache at each step; the moment one instruction's cache misses, every subsequent instruction's cache misses too, even if those later steps' own inputs didn't change — cache invalidation cascades downward through the layer stack, which is the mechanical reason instruction order matters as much as it does.

Multi-stage builds, revisited from the internals angle

Part 7.1 taught multi-stage builds as a pattern (a builder stage with build tools, a slim final stage copying only the built artifacts). The internal reason this actually reduces image size: each stage produces its own independent layer stack, and only the layers from the stage you COPY --from=builder actually end up in the final image — the builder stage's layers (containing gcc, build headers, and other tools needed only to compile a package with C extensions, common for numerical/ML Python packages) are discarded entirely, not merely hidden.

What docker run actually does

docker run creates a new container — a set of Linux namespaces (PID, network, mount, UTS, IPC — each giving the container its own isolated view of processes, network interfaces, the filesystem, hostname, and IPC resources respectively) plus a cgroup (18.1's resource-limiting mechanism, now the actual enforcement point for a container's --memory and --cpus flags) — then starts the image's ENTRYPOINT/CMD as PID 1 inside that new PID namespace. This directly explains a detail from 18.1 revisited at the container level: PID 1 inside a container has special responsibilities (reaping zombie child processes, section 4 of 18.1) that most application processes don't handle by default — which is why a container running, say, a shell script that spawns subprocesses can accumulate zombies inside the container's own PID namespace unless an init process (docker run --init, or tini) is used as PID 1 instead of the application directly.

The container's filesystem is the image's layers (still read-only) plus one additional writable layer on top, using a union filesystem driver (overlay2, on modern Docker) — this is precisely why writes inside a running container disappear when the container is removed: they only ever existed in that ephemeral top writable layer, never in the underlying image, and section 4's volumes subsection is the actual, correct fix for anything that needs to survive a container's lifecycle.

What docker push actually does

docker push uploads an image's layers to a registry (ECR, 18.3) — but only the layers the registry doesn't already have, identified by their content hash (the same content-addressing from section 4's build-cache explanation). This is why pushing a new version of an image that changed only the last COPY . . layer is fast — every earlier, unchanged layer (the base image, the installed dependencies) is already present in the registry and is skipped, not re-uploaded.

Docker networking

By default, containers on the same user-defined bridge network (created via docker network create, or automatically by Docker Compose) can reach each other by container/service name — Docker runs an internal DNS resolver for the network, resolving postgres to whichever container is running with that name/alias on the same network. This is the mechanism behind a DATABASE_URL=postgresql://postgres:5432/app environment variable working correctly inside a Compose stack, and it's also the single most common source of "my app can't connect to the database" in local development: the application container and the database container must be on the same Docker network for name resolution to work at all — a container on Docker's legacy default bridge network, or a different user-defined network than the one you expect, will fail this exact way with no name resolution possible.

Docker volumes

A named volume (docker volume create pgdata, or declared in Compose) is storage managed by Docker itself, persisting independently of any container's lifecycle — the correct way to keep PostgreSQL's actual data files, or a locally-cached embedding model's weights, alive across docker compose down and up cycles. A bind mount instead maps a specific host directory into the container — useful for live-reloading application source code during local development (edit on the host, see the change immediately inside the running container) but not the right tool for a database's data directory, where you want Docker-managed, portable storage rather than a hardcoded host path.

Container security scanning, named

Part 7.1 mentioned image scanning conceptually without naming tools. The commonly used ones, concretely: Trivy (open-source, fast, checks both OS packages and language-level dependencies for known CVEs, easy to run in CI), Grype (a similar open-source vulnerability scanner), and Docker Scout (Docker's own, integrated into docker CLI/Docker Hub). Each works the same way conceptually: inspect every layer's installed packages against a vulnerability database and flag known CVEs by severity — run as a CI gate (18.8) so a critical vulnerability in a base image or a Python dependency blocks a deploy rather than being discovered later, and run periodically even against unchanged images, since new CVEs are disclosed continuously against packages already sitting in a registry.

.dockerignore

A .dockerignore file (parallel to .gitignore, Part 1.7) excludes files from the build context (everything sent to the Docker daemon before a build even starts) — without one, a COPY . . sends your entire project directory, including .git/ history, local virtual environments, __pycache__, and any local .env files, to the build context, both slowing the build meaningfully and creating a real risk of accidentally COPY-ing a secret-containing file into an image layer (where it remains recoverable from the image's layer history even if a later layer deletes it — image layers are immutable and additive, not truly overwritten).

# .dockerignore
.git
__pycache__/
*.pyc
.venv/
.env
.env.*
tests/
*.md

5. Simple mental model

An image is like a stack of transparency sheets on an overhead projector — each layer adds or changes something, and the final picture is all sheets viewed together; Docker's build cache is simply "if this sheet and every sheet below it are identical to last time, reuse them instead of drawing again." A running container adds one more, erasable sheet on top for anything it writes at runtime — remove the container, and that top sheet is gone, while the sheets underneath (the image) remain untouched and reusable for the next container.

6. Real-world example — a full local AI stack via Docker Compose

# docker-compose.yml — local development stack for the Enterprise AI
# Assistant: FastAPI + LangGraph, Postgres (app data + checkpoints,
# Part 5.3), Redis (caching, Part 7.8), a vector DB.
services:
  api:
    build: .
    ports: ["8000:8000"]
    environment:
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/app
      REDIS_URL: redis://redis:6379/0
    depends_on:
      postgres: { condition: service_healthy }
      redis: { condition: service_healthy }
    volumes:
      - ./app:/app/app   # bind mount: live code reload in local dev only

  postgres:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: postgres
    volumes:
      - pgdata:/var/lib/postgresql/data   # named volume: survives `down`/`up`
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      retries: 5

volumes:
  pgdata:

depends_on with a condition: service_healthy (not just plain depends_on, which only waits for the container to start, not to be ready) is the detail that prevents the classic "API container starts before Postgres is actually accepting connections yet" race condition — plain depends_on alone is a common, specific source of intermittent local-dev startup failures that look like flaky infrastructure but are actually a missing readiness gate.

7. Architecture diagram

Docker bridge network · "ai-assistant-net"

api containerbind mount: ./app

postgres containervolume: pgdata · DNS: "postgres"

redis containerephemeral · DNS: "redis"

host:8000developer's curl/browser

8. Production considerations

• Order Dockerfile instructions from least-to-most frequently changing (dependencies before application code, section 4) — the single highest- leverage change for CI build speed (18.8) on a typical AI service.
• Always ship a .dockerignore — both for build speed and to avoid accidentally baking secrets or .git history into an image layer.
• Scan every image in CI (Trivy/Grype/Docker Scout) before it's pushed to a registry an ECS/EKS deployment (18.3/18.9) will pull from, and re-scan periodically even for unchanged, already-deployed images.
• Use named volumes for anything requiring persistence (a database's data directory) and bind mounts only for local-development conveniences that have no place in a production image.

9. Common mistakes

• COPY . . before installing dependencies, silently making every build as slow as a from-scratch build regardless of what actually changed.
• Two containers that need to talk to each other placed on different Docker networks (or one left on the legacy default bridge), producing a DNS-resolution failure that looks like an application bug.
• Using a bind mount for a database's data directory in local development, then being surprised when host-path assumptions don't hold in CI or on a teammate's machine.
• Assuming depends_on alone (without a health-check-based condition) guarantees a dependency is actually ready to accept connections.

10. Security considerations

• Image layers are immutable and additive — a secret COPY-ed in one layer and "deleted" in a later layer is still recoverable from the image's layer history; secrets must never be baked into an image at all, only injected at runtime (environment variables from Secrets Manager, 18.3/18.5).
• Named vulnerability scanning tools (section 4) should be a required CI gate (18.8), not an occasional manual check — new CVEs are disclosed continuously against already-built, already-deployed images.
• docker run --init (or an explicit tini entrypoint) matters for security-relevant process hygiene too, not just zombie reaping — a container whose PID 1 doesn't correctly forward signals can prevent a graceful shutdown (Part 7.4/18.9's SIGTERM/preStop handling) from working as intended.

11. Performance considerations

Layer-cache-aware Dockerfile ordering (section 4) is the primary local build-performance lever; at the registry level, docker push's content-addressed, layer-diffing behavior means keeping your base image and dependency layers stable (changing rarely) minimizes both push and pull time across your whole team and CI pipeline, not just your own local builds.

12. Cost considerations

Faster builds (via good layer caching) reduce CI compute minutes (18.8, 18.18) directly — a concrete, measurable cost lever, not just a developer- experience nicety. Registry storage (ECR, 18.3) also has a real, if usually small, per-GB cost — pruning old, unused image tags periodically avoids unbounded registry growth over a project's lifetime.

13. When to use it

Every containerized AI service uses these mechanics whether or not you think about them explicitly — understanding them is what turns "the build is slow" or "containers can't talk to each other" from a mystery into a specific, fixable, named cause.

14. When NOT to over-apply it

You don't need to hand-optimize every layer of every Dockerfile in a prototype (Part 13.2) — apply this chapter's depth once build speed, image size, or container networking actually becomes a real, felt problem, not preemptively for every throwaway experiment.

15. Alternatives and trade-offs

BuildKit (Docker's modern build engine, enabled by default in current Docker versions) adds more advanced caching (cache mounts for package manager caches specifically, e.g. pip's own cache surviving across builds even when the pip install layer itself is invalidated) — a real, worthwhile upgrade over the classic builder for AI services with heavy, slow-to-reinstall dependencies; verify current Docker documentation for exact BuildKit syntax and default status in your version.

16. Practical example — diagnosing a networking failure

# "My API container can't reach my Postgres container"

docker network ls                       # what networks exist?
docker inspect api-container \
  --format '{{json .NetworkSettings.Networks}}'   # which network(s) is API on?
docker inspect postgres-container \
  --format '{{json .NetworkSettings.Networks}}'   # same, for Postgres

# If they don't share a network: that's the entire bug.
docker network connect ai-assistant-net postgres-container   # fix, or better,
                                                                # fix the compose file

# If they DO share a network, test DNS resolution directly:
docker exec api-container getent hosts postgres
# No result → DNS resolution itself is failing (wrong network, or
# Postgres container not actually running under that name/alias)

17. Production-quality example — a cache-optimized, scanned Dockerfile

# syntax=docker/dockerfile:1
FROM python:3.12-slim AS builder
RUN apt-get update && apt-get install -y --no-install-recommends gcc \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt   # dependency layer,
                                                               # cached across
                                                               # code-only changes

FROM python:3.12-slim
RUN useradd --uid 1000 --create-home appuser
WORKDIR /app
COPY --from=builder /root/.local /home/appuser/.local
COPY . .                                # application code — changes most often,
                                          # placed LAST deliberately (section 4)
RUN chown -R appuser:appuser /app
USER appuser
ENV PATH=/home/appuser/.local/bin:$PATH
ENTRYPOINT ["/usr/bin/tini", "--"]      # correct PID 1 signal/zombie handling
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

# CI step (18.8 shows the full pipeline this fits into):
docker build -t ai-assistant:$(git rev-parse --short HEAD) .
trivy image --severity HIGH,CRITICAL --exit-code 1 ai-assistant:$(git rev-parse --short HEAD)

The build's dependency-installation layer is separated from the application-code layer deliberately (section 4); tini as ENTRYPOINT addresses the PID-1 signal-handling point from section 4/10; trivy with --exit-code 1 fails the CI job outright on a high/critical CVE rather than merely reporting one.

18. Short exercise

Take a Dockerfile that does COPY . . before RUN pip install, time a rebuild after changing one line of application code, then reorder it per section 4 and time the same rebuild again — observe the cache-hit behavior directly in the build output (CACHED markers) rather than just trusting the explanation.

19. Interview questions

• Why does Dockerfile instruction order affect build speed, mechanically?
• What's the difference between a named volume and a bind mount, and when is each appropriate?
• Why can two containers on different Docker networks not resolve each other by name, and how would you diagnose that?
• Why are secrets baked into an image layer still recoverable even if a later layer "deletes" them?

20. FDE/customer scenario

A customer's engineering team says: "Our Docker builds in CI take 12 minutes and it's slowing down every deploy." A strong response starts by actually inspecting their Dockerfile's instruction order (section 4) before proposing anything else — a dependency-installation step placed after application code copying is a specific, common, and often single-largest cause of exactly this complaint, fixable with a Dockerfile reorder rather than a more expensive CI infrastructure upgrade.

---

Key takeaways

• Docker's build cache invalidates from the point of change downward through the entire layer stack — Dockerfile instruction order (least- to-most frequently changing) is the primary lever on build speed.
• Containers on different Docker networks cannot resolve each other by name — a very common, specific cause of local "can't connect" bugs.
• Named volumes persist independently of container lifecycle; bind mounts and a container's writable layer do not — using the wrong one for production data causes silent, avoidable data loss.

Things you should be able to explain

• Why Dockerfile instruction order affects build cache behavior.
• What docker build/run/push actually do at the OS/registry level.
• The difference between a named volume and a bind mount.

Things you should be able to build

• A cache-optimized, scanned, non-root Dockerfile for an AI service.
• A Docker Compose stack with correct health-check-gated depends_on and a named volume for persistent data.

Common mistakes

• Application code copied before dependency installation in a Dockerfile.
• Two related containers left on different or default Docker networks.
• Assuming plain depends_on guarantees dependency readiness.

Recommended next chapter

08-cicd-for-ai-systems-deep-dive.md