18.8 — CI/CD for AI Systems: A Deep Dive

Relationship to Part 7.2: Part 7.2 taught the CI/CD pipeline shape (build → test → lint → security scan → Docker build → push → deploy → smoke test) and the evaluation-gate concept (Part 6.2). This chapter assumes that and goes deeper on exactly the parts a real infrastructure deployment adds: deployment strategies (rolling/blue-green/canary) in mechanical detail, rollback, image signing/provenance, infrastructure (Terraform, 18.6) as a pipeline stage of its own, and handling a flaky LLM-judge evaluation gate (Part 8.3) in CI specifically.

1. What is it?

The complete, production-grade CI/CD pipeline for the Enterprise AI Assistant — from a git push through build, test, evaluation, security scanning, infrastructure change, deployment, and rollback — with GitHub Actions as the concrete implementation, per the source spec's requirement.

2. Why does it exist?

Part 7.2 established what stages a pipeline needs. This chapter exists because two of the hardest, most consequential parts of a real AI-system pipeline are under-specified by "have a pipeline": how do you deploy a new version without an outage or a bad rollback, and what do you do when your evaluation gate (Part 6.2/8.3) — which calls an LLM to judge quality — is occasionally flaky in a way a unit test never is?

3. What problem does it solve?

It solves "how do I safely ship a change to a production AI system," where "safely" specifically means: a bad deploy is caught before it serves most users (deployment strategy), can be undone quickly (rollback), the image running in production is provably the one that passed CI, not something substituted afterward (signing/provenance), and a genuinely regressive prompt or model change is blocked without an occasionally-noisy evaluator blocking every unrelated deploy (flaky-gate handling).

4. How does it work internally?

Deployment strategies, mechanically

Rolling deployment replaces old instances with new ones incrementally — e.g., an ECS service or Kubernetes Deployment (18.9) with a rolling update policy stops a few old tasks/pods, starts the same number of new ones, waits for them to pass health checks, and repeats until all instances run the new version. The risk: for a window of time, both old and new versions serve traffic simultaneously — fine for a stateless API, but worth checking explicitly for an AI service if a schema or behavior change between versions could break a client mid-rollout.

Blue-green deployment runs the new version ("green") fully, alongside the old version ("blue"), with zero production traffic on green until it's verified healthy — then traffic is switched all at once (a load balancer target-group swap, most commonly). The advantage over rolling: no window with mixed old/new versions serving simultaneously, and instant rollback (switch the target group back) if green shows problems immediately after the cutover. The cost: you run double the compute capacity for the duration of the deployment.

Canary deployment routes a small percentage of real traffic (5%, say) to the new version first, monitors it against the same SLIs (Part 8.1/8.4) the old version is held to, and only proceeds to 100% if the canary looks healthy — the deployment strategy best suited to catching an AI-specific regression (a subtly worse response-quality distribution, an increased hallucination rate, Part 8.2) that a purely infrastructure-level health check (is the process up, is it responding) would never catch, since infrastructure health and AI-quality health are genuinely different signals (18.11 develops this distinction fully).

Rolling:     [old][old][old] → [new][old][old] → [new][new][old] → [new][new][new]
                                  (brief mixed-version window)

Blue-Green:  Blue: [old][old][old]  (100% traffic)
             Green:[new][new][new]  (0% traffic, verified healthy)
             → instant switch →
             Blue: [old][old][old]  (0% traffic, kept briefly for rollback)
             Green:[new][new][new]  (100% traffic)

Canary:      [old][old][old][old] → [new][old][old][old] (5% traffic to new)
             → monitor SLIs/eval metrics specifically on the canary →
             [new][new][new][new] (100%, only if canary looked healthy)

Rollback

Rollback undoes a bad deployment — mechanically, kubectl rollout undo (Kubernetes, 18.9/18.10) reverts a Deployment to its previous ReplicaSet; on ECS, it means deploying the previous task definition revision. The critical, easy-to-miss requirement for rollback to actually work correctly: any database migration shipped alongside the code change must itself be backward-compatible with the previous application version during the rollback window — a migration that drops a column the old code still reads will break the rollback it was supposed to enable. The safe pattern (the same "expand/contract" migration discipline Part 1.5/1.6 implied) is deploying schema changes that are additive-only in the same release as the code that needs them, and removing the old column/field only in a later, separate release once the rollback window has safely passed.

Image signing and provenance

Image signing (commonly via cosign, part of the Sigstore project) cryptographically signs a built image, so a deployment step can verify "this exact image was built by our CI pipeline from this exact commit," not tampered with or substituted in the registry in between. SLSA (Supply-chain Levels for Software Artifacts) is a framework describing increasing levels of supply-chain integrity guarantees (build provenance, tamper-resistance) a pipeline can achieve — relevant to an AI FDE specifically because enterprise customers with mature security programs (18.19, 18.23) increasingly ask "can you prove this artifact wasn't tampered with between build and deploy," and "we sign our images" is the concrete, current answer.

CI-specific correctness: the changed-files pattern, done right

Part 7.2 flagged that github.event.pull_request.changed_files is an integer count, not a path list, and cannot power a "did prompts/ change" conditional the way it might appear to. The correct pattern uses a dedicated action designed for this:

- uses: dorny/paths-filter@v3
  id: filter
  with:
    filters: |
      prompts:
        - 'prompts/**'
- name: Run eval gate
  if: steps.filter.outputs.prompts == 'true'
  run: python scripts/run_eval_gate.py

paths-filter actually inspects the changed file paths in the diff and exposes a proper boolean output per filter — the mechanism the naive changed_files count could never provide.

Handling a flaky LLM-judge evaluation gate

Part 8.3 already covered LLM-as-judge's own reliability limitations (inter-run variance, the need for a validated correlation with human judgment). In a CI gate specifically, that variance becomes an operational problem: an evaluation gate that occasionally fails a perfectly good deploy (because the judge model itself gave a noisy score on one run) trains engineers to distrust or bypass the gate entirely — worse than not having a gate at all. Practical mitigations, applied together rather than any one alone: run the judge multiple times (e.g., 3 runs) and require a majority or averaged-threshold pass rather than a single run's verdict; set the pass threshold with real margin below the validated baseline (Part 8.3's correlation work) rather than at a razor's edge; and — critically — make a gate failure block merge with a clear, actionable message (which specific eval cases regressed, with a link to the actual traces, Part 6.1) rather than an opaque "eval failed," so a genuine regression is fast to diagnose and a flaky one is fast to recognize as flaky and re-run.

Environment promotion

Environment promotion means the same, already-built image is deployed to staging, tested, then promoted to production — never rebuilt separately for production. Rebuilding for each environment risks a subtle difference (a dependency resolving to a different version at build time, Part 7.1's reproducible-build concern) between what was tested in staging and what actually runs in production — defeating the entire purpose of staging as a pre-production check.

5. Simple mental model

A CI/CD pipeline is like an assembly line with quality-control stations: each station (test, lint, security scan, eval gate) can stop the line, and the further down the line a defect is caught, the more expensive it is to fix — catching a regressive prompt change in the eval-gate station (before production) is far cheaper than catching it via a customer complaint after full rollout, which is exactly why canary deployment exists as one more, later quality-control station specifically for things earlier stations can't catch.

6. Real-world example

A prompt-engineering change to the Enterprise AI Assistant's retrieval- answering prompt (Part 3.1) is pushed. The paths-filter step detects the change under prompts/, triggering the eval gate: three runs against Part 6.2's golden dataset, averaged, compared against the validated baseline threshold (Part 8.3). It passes. The image builds, is scanned (Trivy, 18.7), signed (cosign), and deployed as a canary to 5% of traffic. CloudWatch and LangSmith (18.11) are watched for 15 minutes; faithfulness and latency SLIs stay within bounds; the deployment proceeds to 100%. Two hours later, a genuine regression is spotted anyway (a rare edge case the eval dataset didn't cover) — kubectl rollout undo reverts within a minute, because the schema change shipped alongside it was additive-only and the previous application version still reads the old data shape correctly.

7. Architecture diagram

git push

Lint + unit testsPart 1.9

paths-filter: did prompts/ or app code change?

Eval gate3 runs, averaged (Part 8.3) — fail blocks merge, links to regressed trace (Part 6.1)

Docker build18.7's cache-optimized Dockerfile

Image scan (Trivy) → sign (cosign)

Push to ECR18.3

Terraform plan18.6 — infra changes, if any, reviewed

Deploy to stagingsame image — environment promotion

Smoke test staging

Canary (5% traffic)monitor SLIs for N minutes (18.11)

Full rollout (100%)or automatic rollback on SLI breach

8. Production considerations

• Never rebuild an image per environment — build once, promote the same artifact through staging to production.
• Ship schema migrations additive-only in the same release as the code that needs them; drop old columns/fields only in a later release, after the rollback window has safely passed.
• Automate canary-metric evaluation (an automatic rollback trigger on SLI breach, not only a human watching a dashboard) once you have enough deploys per week that manual canary-watching doesn't scale.

9. Common mistakes

• A changed_files-count-based conditional that silently never triggers (or always triggers) an eval gate — Part 7.2's exact, real bug.
• A flaky eval gate that fails intermittently for reasons unrelated to actual regression, training the team to --force merge past it, defeating its purpose entirely.
• A destructive database migration shipped in the same release as the code needing it, silently breaking kubectl rollout undo as an available recovery option.
• No image signing/provenance at all, leaving "is this the image CI actually built and tested" as an unverifiable assumption.

10. Security considerations

Image signing/provenance (section 4) is this chapter's primary security contribution beyond Part 7.2/18.7's scanning — it answers a distinct question (was this artifact tampered with after CI produced it) that vulnerability scanning alone doesn't address. CI pipeline credentials themselves (the role/token used to push to ECR, apply Terraform, deploy) should follow 18.3's least-privilege discipline — a compromised CI pipeline with overly broad permissions is a severe, realistic supply- chain risk.

11. Performance considerations

Canary and blue-green deployments both trade deployment speed for deployment safety — a canary's monitoring window (section 4) adds real wall-clock time to every deploy, a deliberate, worthwhile cost for a system whose failure mode (a subtly degraded AI response quality) isn't always immediately, obviously visible the way an infrastructure crash is.

12. Cost considerations

Blue-green deployment's double-capacity requirement (section 4) is a real, temporary cost during the deployment window — usually negligible against deployment frequency, but worth being explicit about for a customer asking why staging costs briefly spike during a release. CI compute minutes themselves (18.7's build-speed point) are a direct, ongoing cost line item, especially for a pipeline running an LLM-judge eval gate on every prompt change (real, per-run LLM API cost, distinct from compute cost).

13. When to use it

Every production AI system needs this full pipeline shape; canary deployment specifically becomes worth its added complexity once a bad deploy's blast radius (all users, immediately) is a real, unacceptable risk — which is true for essentially any customer-facing production AI system.

14. When NOT to over-apply it

A low-traffic internal tool or an early-stage prototype (Part 13.2) can reasonably use straightforward rolling deployment without the added canary/blue-green machinery — match deployment sophistication to actual blast-radius risk and deployment frequency, not universally maximum rigor.

15. Alternatives and trade-offs

Rolling vs. blue-green vs. canary (section 4) is a genuine, situation- dependent trade-off between deployment speed, infrastructure cost, and safety — not a strict hierarchy where canary is always "best"; a low-risk, easily-reversible internal service may reasonably prefer rolling's simplicity and lower cost.

16. Practical example — a GitHub Actions rollback-aware deploy job

name: deploy-production
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Deploy new task definition revision
        id: deploy
        run: |
          aws ecs update-service --cluster prod --service ai-assistant \
            --task-definition ai-assistant:${{ github.sha }} \
            --force-new-deployment

      - name: Wait for service stability
        id: wait
        run: aws ecs wait services-stable --cluster prod --services ai-assistant

      - name: Rollback on failure
        if: failure() && steps.wait.outcome == 'failure'
        run: |
          PREVIOUS_REVISION=$(aws ecs describe-services --cluster prod \
            --services ai-assistant --query 'services[0].deployments[1].taskDefinition' \
            --output text)
          aws ecs update-service --cluster prod --service ai-assistant \
            --task-definition "$PREVIOUS_REVISION" --force-new-deployment

The rollback step is conditioned specifically on the wait step failing (the new deployment never reached a stable, healthy state) — an automated safety net that doesn't require a human to notice and manually intervene during exactly the moment when a deploy has already started going wrong.

17. Production-quality example — an averaged, non-flaky eval gate

"""
Run the LLM-judge evaluation gate multiple times and require an averaged
pass — mitigating single-run judge variance (Part 8.3) from blocking a
good deploy, or passing a bad one, on one noisy sample.
"""
import statistics

RUNS = 3
BASELINE_THRESHOLD = 0.82  # set with real margin below the validated baseline

def run_eval_gate() -> bool:
    scores = [run_single_eval_pass() for _ in range(RUNS)]
    average = statistics.mean(scores)
    passed = average >= BASELINE_THRESHOLD

    print(f"Eval gate scores: {scores} (avg {average:.3f}, threshold {BASELINE_THRESHOLD})")
    if not passed:
        print("REGRESSED CASES:")
        for case in get_regressed_cases():
            print(f"  - {case.id}: {case.trace_url}")  # link to Part 6.1's trace
    return passed


if __name__ == "__main__":
    import sys
    sys.exit(0 if run_eval_gate() else 1)

Averaging across RUNS independent evaluation passes, plus printing the specific regressed cases with a direct trace link, together address section 4's flaky-gate problem: a single bad run doesn't block a good deploy, and when the gate genuinely fails, the failure message is immediately actionable rather than an opaque "eval score too low."

18. Short exercise

Design a rollback plan for a hypothetical release that both changes the retrieval-answering prompt (Part 3.1) and adds a new, required column to the conversations table — identify specifically what would break if the column were added as NOT NULL with no default, versus as nullable with a default, in terms of whether kubectl rollout undo/ECS rollback would actually work afterward.

19. Interview questions

• Explain rolling vs. blue-green vs. canary deployment and when you'd choose each.
• Why can a database migration break rollback, and how do you avoid it?
• How would you handle a flaky, LLM-judge-based evaluation gate in CI without either ignoring real regressions or blocking good deploys?
• What does image signing add beyond vulnerability scanning?

20. FDE/customer scenario

A customer's engineering lead says: "Our last AI feature deploy caused a quality regression that took two days to notice and roll back." A strong response walks through this chapter's canary-plus-SLI-monitoring pattern specifically — catching a quality regression within the canary's monitoring window (minutes, not days) by watching the right signals (Part 8.1/8.4's AI-quality SLIs, not just infrastructure health) on a small fraction of real traffic before full rollout, and pairing it with automated rollback so recovery doesn't depend on someone noticing manually.

---

Key takeaways

• Canary deployment, monitored against AI-quality SLIs (not just infrastructure health), is the deployment strategy best matched to catching AI-specific regressions before full rollout.
• Rollback safety depends on migration discipline (additive-only schema changes) as much as on the deployment mechanism itself — a destructive migration silently breaks the rollback it was meant to enable.
• A flaky LLM-judge eval gate needs averaging, real margin, and actionable failure messages — or it trains engineers to bypass it.

Things you should be able to explain

• Rolling vs. blue-green vs. canary deployment, mechanically.
• Why database migrations must be additive-only for safe rollback.
• What image signing/provenance adds beyond vulnerability scanning.

Things you should be able to build

• A paths-filter-based conditional eval gate.
• An automated rollback step conditioned on deployment-stability failure.
• An averaged, multi-run LLM-judge eval gate with actionable failure output.

Common mistakes

• changed_files-count-based path conditionals that don't work as intended.
• Destructive migrations shipped alongside the code that needs them.
• Single-run, unaveraged eval gates that are flaky enough to be bypassed.

Recommended next chapter

09-kubernetes-for-ai-engineers.md