7.2 — CI/CD for AI Systems

1. What is it?

CI/CD (Continuous Integration/Continuous Deployment) is the automated pipeline that runs on every code change: building, testing, and (for CD) deploying the application without manual intervention. For AI systems specifically, this pipeline needs to include not just traditional code tests (Part 1.9) but the evaluation gates (Part 6.2/6.3) this book has argued for throughout — a distinctive, AI-specific extension of the standard CI/CD concept.

2. Why does it exist?

Part 1.9 established that automated testing exists to catch regressions immediately rather than after a customer reports them. CI/CD extends this principle to the entire path from code change to production: build the artifact (Part 7.1's Docker image), run every test suite automatically, and (for CD) deploy automatically once everything passes — removing manual, error-prone, inconsistent steps from the release process, and making "can we ship this" an objective, automated answer rather than a subjective judgment call made under time pressure.

3. What problem does it solve?

For AI systems, it solves a problem broader than traditional software CI/CD: not just "does the code work correctly" (Part 1.9's unit/integration tests) but "does the AI system's actual behavior meet quality bars" (Part 6.2's evaluation gates) — both need to be automated gates blocking a bad change from reaching production, and CI/CD is the infrastructure that enforces both consistently, on every single change, without relying on someone remembering to run the eval suite manually.

4. How does it work internally?

The pipeline stages, for an AI application specifically

Code pushed / PR opened

Lint/format checkfast, catches style/syntax issues immediately

Unit testsfast, deterministic logic — parsing, auth, retrieval ranking (Part 1.9)

Integration testsDB, mocked LLM calls (Part 1.9)

Evaluation gate (AI-SPECIFIC)regression dataset, minimum pass threshold — doesn't exist in traditional CI/CD (Part 6.2)

Build Docker imagetagged with commit SHA for traceability (Part 7.1)

Deploy to stagingautomated, for CD

Smoke tests on stagingverify the deployed artifact actually starts/responds, distinct from pre-deploy tests

Deploy to productionfull auto-deploy, a manual approval gate, or a canary/progressive rollout monitored against Part 8.4 SLOs

The evaluation gate is the distinctively AI-specific addition to an otherwise standard pipeline.

The evaluation gate stage is the distinctively AI-specific addition to an otherwise standard CI/CD pipeline — Part 1.7's CI-gate example and Part 6.2's run_regression_gate function are exactly this stage, made concrete.

Why ordering matters — fail fast, expensive checks last

The stages are ordered from fastest/cheapest to slowest/most-expensive deliberately: lint checks (seconds) run before unit tests (tens of seconds) before integration tests (potentially minutes) before the evaluation gate (Part 6.2, section 11 — can be the slowest stage, especially with LLM-as-judge evaluators making their own real LLM calls). This ordering means a change with an obvious problem fails fast and cheaply, without wasting the time and (for the evaluation gate specifically) real LLM-API cost of running the more expensive later stages on a change that was already going to fail an earlier, cheaper check.

Continuous Deployment vs. Continuous Delivery — a real distinction worth being precise about

Continuous Deployment means every change that passes all gates deploys to production automatically, with no manual approval step. Continuous Delivery means every change that passes all gates is ready to deploy, but an explicit human decision (a button click, a scheduled release window) triggers the actual production deployment. For AI systems, given the added risk profile of behavior changes (Part 3.1's prompt-change caution, Part 9's security stakes), many teams deliberately choose continuous delivery over full continuous deployment for changes with higher behavioral risk (a prompt or model change) while using full continuous deployment for lower-risk changes (a bug fix in unrelated infrastructure code) — this isn't inconsistency, it's risk-appropriate automation, directly paralleling Part 5.4's argument against blanket-applying human-in-the-loop gates uniformly regardless of actual stakes.

Canary / progressive rollout — the middle path between full auto-deploy and a manual gate

Section 4's "Deploy to production" stage has, so far, been presented as a binary choice: fully automatic, or gated behind manual approval. There's a genuinely distinct third option that a binary framing misses — canary (progressive) rollout: shift a small percentage of production traffic to the new prompt/model/code version, monitor it against the quality/cost/latency SLOs (Part 8.4) for a defined window, and only then progressively increase traffic toward 100%, halting or automatically rolling back (Part 8.4's SLO-triggered rollback) the instant the canary's metrics breach an SLO the old, still-serving-most-traffic version wasn't breaching.

Deploy v2 alongside v1

Route 5% of traffic to v2, 95% to v1

Monitor v2's SLOsdefined window · Part 8.4

SLOs hold

SLO breached

Increase to 25% → 50% → 100%

Automatic rollbackroute 100% back to v1, page on-call, halt rollout (Part 8.4)

This matters specifically for AI systems because a prompt or model change's failure mode is often a quality regression (a faithfulness or task-completion drop, Part 8.4's SLIs), not a crash — traditional software canarying watches error rates and latency, which a bad prompt change frequently won't move at all, so the canary's monitored metrics for an AI rollout must include Part 8.4's quality/cost SLIs, not just infrastructure health. Canary rollout is the natural middle path between the two extremes section 4 originally presented: full continuous deployment (fast, but a bad change reaches 100% of traffic immediately) and a manual gate (safe, but "safe" here just means "a human looked at it before launch," which catches nothing a human reviewer wouldn't have already caught in code review) — a canary catches exactly the failure mode neither extreme catches well: a change that looks fine in code review and passes the evaluation gate (section 4) but degrades quality in ways only real production traffic surfaces, while limiting the blast radius to a small percentage of traffic instead of everyone.

5. Simple mental model

CI/CD for an AI system is like a factory assembly line with both a mechanical quality-control station (traditional tests) and a taste-test station for a food product (the evaluation gate) — a food factory wouldn't ship based only on "the machinery ran without jamming" (traditional tests passing); it also needs an actual taste/quality check specific to what the product is supposed to deliver, run automatically on every batch, not just occasionally by hand.

6. Real-world example

A team's CI pipeline for their support-agent system runs, in order: lint (10 seconds), unit tests covering parsing/auth/retrieval logic (45 seconds), integration tests against a test database with mocked LLM calls (2 minutes), then the evaluation gate (Part 6.2) running the current prompt against a 200-example regression dataset with a mix of deterministic and LLM-as-judge evaluators (4 minutes, and the only stage incurring real LLM API cost) — a prompt change that accidentally breaks JSON parsing (Part 3.2) fails at the unit-test stage in under a minute, never reaching the evaluation gate's slower, costlier check; a prompt change that's syntactically fine but degrades response quality passes the fast stages and is caught specifically by the evaluation gate, exactly the failure mode traditional tests alone would have missed entirely.

7. Architecture diagram

See section 4's pipeline-stage diagram — this chapter's architecture is that ordered pipeline, with the evaluation gate as its AI-specific addition to an otherwise standard structure.

8. Production considerations

• Include the evaluation gate (Part 6.2) as a required, blocking CI stage for any change touching prompts, model configuration, or retrieval logic — not an optional, occasionally-run check, but a mandatory gate exactly like a failing unit test would be.
• Tag/version every deployed artifact with its exact commit SHA (Part 1.7) — this is what makes "which exact version is running in production, and what changed since the last version" a precise, answerable question rather than a guess, directly supporting Part 6.1's debugging value and Part 1.7's reproducibility argument.
• Deliberately choose continuous deployment vs. continuous delivery per change-risk category, rather than uniformly automating everything or uniformly gating everything behind manual approval.
• Use canary/progressive rollout for prompt and model changes specifically (section 4) — this is often a better fit than a binary manual-gate-or-not choice for exactly the changes whose failure mode is a quality regression rather than a crash, since a human approver reviewing a prompt diff can't reliably predict its production quality impact the way a monitored canary window can.
• Run smoke tests against the actually-deployed artifact in staging, not just pre-deployment tests against source code — this catches deployment-specific issues (a missing environment variable, a misconfigured connection string) that unit/integration tests running against source code wouldn't surface.

9. Common mistakes

• Treating evaluation (Part 6.2) as a manual, occasional practice rather than a required, automated CI gate — exactly the gap Part 3.1's original warning about ad hoc prompt iteration was written to close, now specifically about pipeline automation.
• Ordering pipeline stages inefficiently (e.g., running the slow, costly evaluation gate before fast unit tests), wasting time and real LLM-API cost on changes that were going to fail a cheaper check anyway.
• Full continuous deployment with no manual gate for genuinely high-risk changes (a major prompt rewrite, a new tool with real side effects, Part 9.2) — automating deployment doesn't mean every change deserves the same level of automated trust.
• Treating "manual approval gate" and "full automation" as the only two options for a risky change, missing canary/progressive rollout (section 4) as a third path that's often better-suited to catching AI-specific quality regressions than either extreme.
• Not tagging deployed artifacts with traceable version identifiers, making "what's actually running in production right now" a surprisingly hard question to answer during an incident.

10. Security considerations

• CI/CD pipelines themselves are a real, sensitive infrastructure component — they typically hold credentials for deploying to production, pulling from private registries, and calling real LLM APIs (for the evaluation gate) — secure pipeline configuration and credential management (Part 9.5) deserves the same scrutiny as the application it builds and deploys.
• A compromised CI/CD pipeline is effectively a compromise of the production deployment path itself — restrict who can modify pipeline configuration and require review for changes to it, exactly as you would for production code changes.

11. Performance considerations

• Parallelize independent test stages where possible (unit tests and lint checks often have no dependency on each other) rather than running everything strictly sequentially, reducing total pipeline time.
• Cache dependency installation (Part 7.1's Docker layer caching, and equivalent caching mechanisms most CI platforms provide for non-Docker dependency installation) to avoid redundant, slow reinstallation on every single run.

12. Cost considerations

• The evaluation gate's LLM-as-judge calls (Part 6.2, section 12) are a real, recurring CI cost, multiplied by how frequently the pipeline runs (every commit, every PR) — worth monitoring as part of overall AI system operating cost (Part 7.10), and a legitimate reason to run the full, expensive evaluation gate only on PRs targeting the main branch rather than on every single intermediate commit during active development.

13. When to use it

Any production AI system, without exception — the combination of traditional CI/CD discipline (Part 1.9) plus an AI-specific evaluation gate (Part 6.2) is close to a baseline requirement for a system this book would consider genuinely production-ready (echoing Part 5.8's checklist philosophy, applied to the deployment pipeline specifically).

14. When NOT to use it

A very early-stage, single-developer prototype might reasonably defer full CI/CD infrastructure temporarily — but per Part 1.9's general philosophy, this should be a deliberate, temporary trade-off during initial feasibility exploration, not a permanent habit that survives into any real production deployment.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Full CI/CD with AI evaluation gate	Consistent, automated, catches both code and behavior regressions	Real setup effort and ongoing evaluation-gate cost
Canary/progressive rollout (section 4)	Catches production-only quality regressions the evaluation gate missed, with limited blast radius	Needs per-canary-slice SLO monitoring (Part 8.4) and a longer time-to-full-rollout
CI only, manual deployment	Automated testing, controlled release timing	Manual deployment step is slower and more error-prone
No CI/CD (manual testing and deployment)	Minimal setup for a trivial prototype	High risk of regressions reaching production undetected

16. Practical Python/code example

A GitHub Actions workflow implementing section 4's staged pipeline, including the AI-specific evaluation gate:

name: ci-cd

on:
  pull_request:
    branches: [main]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt -r requirements-dev.txt
      - run: ruff check .
      - run: pytest tests/unit tests/integration

  evaluation-gate:
    needs: lint-and-test
    runs-on: ubuntu-latest
    if: contains(github.event.pull_request.changed_files, 'prompts/') || contains(github.event.pull_request.changed_files, 'src/agent/')
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python scripts/run_eval_gate.py --dataset support-agent-regression-v1 --min-pass-rate 0.90
        env:
          LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Note the evaluation gate is conditioned to run only when prompt or agent-related files changed (section 12's cost-consciousness applied directly), not on every single PR regardless of what it touches.

17. Production-quality example

Adding image build, staging deployment, and smoke tests to complete the pipeline through to production, per section 4's full sequence:

  build-and-push:
    needs: evaluation-gate
    if: always() && (needs.evaluation-gate.result == 'success' || needs.evaluation-gate.result == 'skipped')
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          docker build -t registry.internal/support-agent:${{ github.sha }} .
          docker push registry.internal/support-agent:${{ github.sha }}

  deploy-staging:
    needs: build-and-push
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy.sh staging ${{ github.sha }}
      - name: smoke test
        run: curl -f https://staging.internal/health || exit 1

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production  # requires manual approval in GitHub's environment protection rules
    steps:
      - run: ./deploy.sh production ${{ github.sha }}

environment: production with GitHub's environment protection rules is a concrete way to implement section 4's "continuous delivery, not full continuous deployment, for higher-risk changes" — requiring an explicit human approval click before the production deployment step runs, while everything before it remains fully automated.

A progressive-rollout controller implementing section 4's canary pattern, deciding whether to advance, hold, or automatically roll back a prompt/model version based on Part 8.4's SLOs:

from dataclasses import dataclass


@dataclass
class RolloutStage:
    """One step in a progressive traffic shift toward a new version."""
    traffic_pct: int
    min_observation_minutes: int


ROLLOUT_STAGES = [
    RolloutStage(traffic_pct=5, min_observation_minutes=30),
    RolloutStage(traffic_pct=25, min_observation_minutes=30),
    RolloutStage(traffic_pct=50, min_observation_minutes=30),
    RolloutStage(traffic_pct=100, min_observation_minutes=0),
]


def evaluate_canary_stage(
    canary_slo_breaches: list[str], observed_minutes: int, current_stage: RolloutStage
) -> str:
    """
    Decides whether a canary rollout should advance to the next traffic
    percentage, hold at the current one, or roll back, based on whether the
    canary's own SLOs (Part 8.4) have breached during the observation window.

    Args:
        canary_slo_breaches (list[str]): SLI names that breached their SLO for
            the canary's traffic slice specifically (not the whole fleet).
        observed_minutes (int): How long the canary has been running at this stage.
        current_stage (RolloutStage): The current traffic-percentage stage.

    Returns:
        str: "rollback" (immediately revert, Part 8.4's automatic-rollback
            trigger), "hold" (keep observing), or "advance" (move to next stage).
    """
    if canary_slo_breaches:
        return "rollback"
    if observed_minutes < current_stage.min_observation_minutes:
        return "hold"
    return "advance"

A canary rollout's monitored SLOs must include Part 8.4's quality/cost SLIs (faithfulness, task completion, cost per request) computed specifically for the canary's traffic slice, not the whole fleet's aggregate — otherwise a canary serving only 5% of traffic can breach badly while a fleet-wide average still looks healthy, defeating the entire point of canarying before full cutover.

18. Short exercise

A team's evaluation gate takes 8 minutes and runs on every single commit pushed to a feature branch (not just PRs to main), costing meaningful LLM API spend during active development with many small commits. Propose a specific change to when this gate runs that preserves its regression-catching value while reducing unnecessary cost during active development.

19. Interview questions

• What's the AI-specific addition to a standard CI/CD pipeline, and why can't traditional unit/integration tests substitute for it?
• Explain the distinction between continuous deployment and continuous delivery, and why an AI system might deliberately choose different automation levels for different types of changes.
• Why should pipeline stages be ordered from cheapest/fastest to most expensive/slowest?

20. FDE/customer scenario

Customer's engineering lead: "We're nervous about automating deployment for our AI features — what if a bad prompt change goes straight to production?"

This is a legitimate, well-founded concern that this chapter directly addresses: recommending continuous delivery (not full continuous deployment) specifically for prompt/model/agent-behavior changes — requiring an explicit approval step after the automated evaluation gate passes, per section 4's distinction — gives the team confidence that a bad change is still automatically screened (Part 6.2) while adding a deliberate human checkpoint for that specific category of higher-risk change, without giving up automation entirely or reverting to a fully manual, slower, less-consistent release process for lower-risk changes.

---

Key takeaways

• AI system CI/CD needs a distinctive addition beyond traditional testing: an automated evaluation gate (Part 6.2) checking behavior quality, not just code correctness.
• Pipeline stages should run from cheapest/fastest to most expensive/slowest, failing fast on obvious problems before incurring the evaluation gate's real cost.
• Continuous deployment vs. continuous delivery is a deliberate, risk-appropriate choice per change category, not an all-or-nothing decision.

Things you should be able to explain

• Why an evaluation gate is a necessary addition to standard CI/CD for AI systems.
• The distinction between continuous deployment and continuous delivery, and when each is appropriate.

Things you should be able to build

• A staged CI/CD pipeline including lint, tests, a conditional evaluation gate, image build, staging deployment with smoke tests, and a manually-gated production deployment.

Common mistakes

• Treating evaluation as manual/occasional instead of an automated, required gate.
• Inefficient stage ordering wasting time and cost on doomed changes.
• Full continuous deployment applied uniformly regardless of change risk.

Recommended next chapter

03-cloud.md