16.8 — Project: A Complete FDE Engagement, First Meeting to Production

This capstone project narrates one continuous engagement — the mid-size insurance company underwriting-automation scenario referenced throughout Part 11.2, Part 12.2, and Part 13.1 — end-to-end, following Part 13.1's full eight-stage lifecycle explicitly. Every stage below cites the specific chapters whose techniques are actually being applied, so you can trace exactly how this handbook's individual skills compose into one real engagement.

Stage 1: First Contact and Discovery (Part 12.1, Part 12.2)

First call.

CUSTOMER (VP of Underwriting Operations): "We want to explore AI to help speed up our underwriting process. Our underwriters are drowning in paperwork."

Per Part 12.1's explicit warning against jumping to a technology recommendation, the FDE's response doesn't propose anything yet:

FDE: "That's helpful context — before we talk about any specific approach, can you walk me through what an underwriter actually does, start to finish, for a typical application? I want to understand where the time actually goes."

This opens Part 12.2's business-process-mapping exercise. Over two follow-up calls and a half-day on-site shadowing session, the mapped process (Part 12.2, section 4's exact structure) emerges:

1. Pull application + credit report (2 separate systems, ~5 min)
2. Check against a 12-point underwriting checklist (mostly routine, ~10 min)
3. IF any checklist item is borderline → escalate to senior underwriter
   (~15% of applications, adds 1-2 business days)
4. Document decision with justification (required for audit, ~5 min)

Applying Part 12.1's five discovery questions surfaces the critical additional facts:

• Data reality (Q2): application and credit-report data live in two different internal systems; underwriting guidelines live in a document library across three folder structures with meaningful duplication (directly foreshadowing Part 10.2's realistic data-engineering challenge).
• Risk tolerance (Q3): a wrong approval has real regulatory and financial consequences — this is not a low-stakes convenience feature.
• Access control (Q4): underwriters should only see cases assigned to their team; senior-underwriter escalations carry additional access restrictions.
• Success metric (Q5): "If average review time for routine applications dropped meaningfully without any increase in our error rate, that would be a clear win." — this exact sentence becomes the metric Stage 8 measures against, unaltered.

Feasibility assessment (Part 12.2): the FDE proposes — and the customer agrees to — a quick validation before any commitment: gathering 40 real, historical (anonymized, appropriately handled per Part 9.6) applications with known correct outcomes, to test whether structured extraction can reliably pull the required checklist fields.

Stage 2: Solution Architecture (Part 11.1, Part 11.2, Part 12.3)

Based on discovery, the FDE proposes — per Part 11.2's full system design — a RAG-based system for policy/guideline questions plus structured extraction for the routine checklist evaluation, explicitly recommending against full automation of the escalation-worthy 15% (Part 5.4/9.2's calibrated human-in-the-loop, chosen from actual process understanding, not generic caution).

Presenting this to a mixed audience (the VP, plus the company's IT director and compliance officer), the FDE applies Part 12.3's layered communication:

To the VP: "This will let your underwriters spend their time on the 15% of applications that genuinely need judgment, while the routine 85% get evaluated automatically against the same checklist they already use — every automated decision fully documented and explainable, and every borderline case still goes to a human, exactly like today."

To IT: "We're proposing pgvector on Postgres for the guideline retrieval [Part 1.5/3.4, appropriate at this scale, Part 11.2's exact reasoning], with structured extraction using schema-enforced generation plus independent validation [Part 3.2] for the checklist fields."

To Compliance: "Every extraction and decision is fully traced [Part 6.1] with the specific source documents cited, and access is scoped so an underwriter only sees their assigned cases [Part 9.4]."

Stage 3: Rapid Prototyping (Part 13.2)

The riskiest assumption, identified explicitly per Part 13.2's discipline: can structured extraction reliably pull the required fields from real underwriting documents at acceptable accuracy. The two-week prototype (Part 13.2, section 6's exact numbers) is a single script, no UI, testing against the 40 real documents gathered during discovery:

result = await run_extraction_prototype(client, real_sample_documents)
# {"overall_accuracy": 0.91, "breakdown_by_subtype": {"standard": 0.96, "non_standard_format": 0.61}}

The 91% overall accuracy clears the required bar — but the sub-type breakdown (Part 13.2's exact value: measuring by sub-type, not just in aggregate) reveals a specific, addressable gap: documents in a less-common formatting variant (8% of volume) perform notably worse. This directly and concretely shapes Stage 4's design: a fallback-to-human-review path specifically for that document sub-type, rather than either ignoring the gap or delaying the whole project to solve it before proceeding.

The demo to the customer is honest about both results (Part 13.2, section 4's trust-preserving practice) — presenting the 91% success and the specific 8% limitation together, with the concrete mitigation plan, rather than hiding the weaker number.

Stage 4: Production Engineering (Parts 1–9)

The full production system is built following the architecture from Stage 2, hardened with:

• Part 1's foundations: async FastAPI backend, PostgreSQL with the two source systems' data unified via a proper data pipeline (Part 10.2 — this took longer than expected, per that chapter's own real-world example, since the two source systems had inconsistent formats requiring genuine deduplication work).
• Part 3's AI engineering: RAG for guideline questions (Part 3.5), structured extraction with independent validation for the checklist (Part 3.2), the non-standard-format fallback identified in Stage 3 routed to human review (Part 5.4).
• Part 5's LangGraph: the review workflow built as a graph (Part 5.1/5.2) with PostgresSaver checkpointing (Part 5.3), since the escalation path can pause for a senior underwriter's review potentially across a business day.
• Part 7's production hardening: Dockerized (Part 7.1), CI/CD with an evaluation gate (Part 7.2/6.2) blocking any prompt change that regresses extraction accuracy below the validated threshold.
• Part 9's security: per-underwriter RBAC (Part 9.4), document-level permission preservation from the source document library (Part 10.4), tenant/case-level isolation (Part 9.6) even though this is a single-organization deployment (isolation between underwriting teams' assigned cases still matters).

Stage 5: Deployment (Part 14)

Discovery in Stage 1 didn't surface a deployment-environment constraint explicitly enough — a gap the FDE catches by asking directly, early in Stage 4 rather than assuming (Part 14.1's exact warning): the customer's compliance team requires the system run within their own cloud tenancy, not the vendor's, given regulatory data-residency requirements (Part 7.3/10.5). The Docker-based architecture (Part 7.1's portability benefit) makes this a configuration change, not a redesign — deployed into the customer's own cloud account (Part 14.1's "customer's cloud account" point on the deployment spectrum) with their IT team's network-policy approval process (Part 14.1, section 4) planned into the timeline from this point forward rather than discovered as a late blocker.

Stage 6: Production Monitoring and Support (Part 8)

Post-launch, faithfulness and extraction-accuracy SLIs (Part 8.1/8.2/8.4) are monitored continuously, with alerting on sustained trends (Part 8.4's exact discipline) rather than single noisy data points. A minor incident in month two — a slight accuracy dip traced, via full tracing (Part 6.1), to a source-document-library reorganization that shifted some guideline documents' locations without the ingestion pipeline's webhook (Part 10.1) catching the move correctly — is resolved within a day specifically because the observability infrastructure (Part 6.1/8.4) built in from Stage 4 made root-cause diagnosis fast rather than a multi-day investigation.

The full postmortem the FDE writes and shares with the customer's IT director and compliance officer (Stage 2's same audience), mirroring 18-cloud-devops-ai-infrastructure/21-production-incidents-playbook.md's postmortem structure:

# Postmortem: Guideline-Retrieval Accuracy Dip — 2026-03-11

## Summary
For approximately 19 hours, RAG-based guideline retrieval (Part 3.5)
returned stale or missing results for roughly 12% of policy-guideline
questions — the checklist-extraction path (unaffected, separate data
source) continued operating normally throughout. No incorrect
underwriting decision resulted: affected queries returned either a
"guideline not found, escalate to senior underwriter" fallback or a
slightly outdated (not fabricated) guideline excerpt, per Part 2.6's
never-hallucinate-a-fallback design. Detected via Part 8.4's faithfulness
SLI alert, not a customer report.

## Timeline (all times local)
- 09:14 — Faithfulness SLI (Part 8.2) drops from a 30-day baseline of
  0.94 to 0.81 on guideline-retrieval queries specifically; alert fires
  per Part 8.4's sustained-trend rule (three consecutive 10-minute
  windows below threshold, not a single noisy point).
- 09:22 — On-call FDE acknowledges; pulls the Part 6.1 trace sample for
  the affected window.
- 09:41 — Traces show retrieval returning zero or low-relevance chunks
  for guideline documents specifically, while checklist-extraction traces
  (a separate ingestion path) look entirely normal — narrows the
  investigation to the guideline ingestion pipeline (Part 10.1).
- 10:15 — DIAGNOSIS confirmed (below). Guideline document library was
  reorganized into new folder paths the prior evening; the ingestion
  webhook (Part 10.1) fired on the move events but the payload's old
  path no longer resolved, and the resulting fetch errors were logged
  but not alerted on — a monitoring gap, not just a pipeline gap.
- 10:30 — IMMEDIATE MITIGATION applied (below).
- 10:52 — Faithfulness SLI recovers to 0.93, within normal range.
- 04:10 (next day) — FIX (below) deployed and verified against a full
  re-ingestion of the reorganized library.

## Root cause
The guideline document library's folder reorganization (an action taken
by the customer's own document-management team, not the vendor) moved
files to new paths. The ingestion pipeline's webhook listener (Part
10.1) received the move events but its path-resolution logic assumed a
stable folder structure and silently failed to re-resolve the new
paths — chunks for the moved documents fell out of the vector index as
their content aged past the store's staleness window, with no ingestion
failure alert because the failure mode was "silently stale," not a hard
error (Part 8.4's exact "correlate an accuracy trend against a specific
change" diagnosis step, applied here against the document-library's own
change log rather than the vendor's own deploy log).

## Immediate mitigation
Manually triggered a full re-ingestion crawl (bypassing the webhook,
directly walking the current folder tree) against the guideline library,
restoring the index to current content within the hour.

## Permanent fix
Rewrote the webhook handler to resolve documents by a stable document ID
(already present in the source system's metadata) rather than by folder
path, so a future reorganization moves a document without breaking its
identity in the ingestion pipeline. Added a dedicated ingestion-staleness
alert (Part 8.4) — time-since-last-successful-ingest per source
document, independent of the downstream faithfulness SLI — so a future
occurrence is caught by the ingestion layer directly instead of only
being inferred from a retrieval-quality symptom an hour or more later.

## Prevention
- Ingestion-staleness alerting (above) now runs as a first-class SLI,
  not inferred indirectly through faithfulness.
- Added the document-library reorganization as an explicit item in the
  customer handoff runbook (below): the customer's document-management
  team now notifies the on-call channel before a planned reorganization,
  giving the ingestion pipeline a known change window to correlate
  against if anything looks off afterward.

## What went well / what didn't
**Well**: the faithfulness SLI caught a subtle, no-hard-error failure
mode before any customer-facing impact was reported, and full tracing
(Part 6.1) took the investigation from symptom to root cause in under an
hour. **Didn't**: the ingestion pipeline's silent-failure mode (a fetch
error logged but not alerted on) meant the underlying cause existed for
several hours before the downstream symptom crossed the alert threshold
— the new ingestion-staleness alert closes exactly this gap.

This is resolved within a day specifically because the observability infrastructure (Part 6.1/8.4) built in from Stage 4 made root-cause diagnosis fast rather than a multi-day investigation — and because the postmortem's Prevention step turns a one-time fast recovery into a permanently closed gap, not a recurring near-miss.

Customer handoff runbook

Alongside the postmortem process above, the customer's own IT operations team receives a short standing runbook at go-live — the same kind of artifact 01-enterprise-support-agent.md's customer handoff includes, adapted to this engagement's specific failure modes:

# Underwriting-Automation System — Ops Handoff Runbook

## Escalation contacts
- P1 (extraction or retrieval pipeline fully down): page vendor on-call
  (shared PagerDuty rotation, 30-min SLA per the support contract).
- P2 (accuracy-SLI alert fired, system still serving): vendor Slack
  channel, next-business-day response acceptable.
- P3 (single case looks wrong): file a ticket with the case ID; the
  senior-underwriter escalation path (Stage 4) already catches
  individual borderline cases regardless.

## Dashboards and logs
- Faithfulness/extraction-accuracy SLI dashboard (Part 8.4): Grafana →
  "Underwriting Automation Overview."
- Per-case trace lookup: LangSmith, filtered by case ID (Part 6.1).
- Ingestion-staleness alert (added per this postmortem's Prevention
  step): same Grafana dashboard, "Ingestion Health" tab.

## Common failure modes and first response
| Symptom | Likely cause | First action |
|---|---|---|
| Faithfulness SLI dips on guideline questions specifically | Document-library reorganization (this postmortem) or a genuine content change | Check "Ingestion Health" tab for staleness before assuming a model regression |
| Extraction accuracy dips on a specific document sub-type | A new, unseen formatting variant (Stage 3's known 8% gap) | Confirm it routes to human review (Stage 4's fallback) rather than a silent wrong answer |
| A case is stuck in the senior-underwriter escalation queue | Awaiting human review — expected behavior, not a system fault | Check the assigned senior underwriter's queue directly before treating as an incident |

## Change coordination
The customer's document-management team notifies #underwriting-ai-oncall
before any planned guideline-library reorganization, giving the ingestion
pipeline a known change window to correlate against (this postmortem's
direct Prevention outcome).

Stage 7: Iteration Based on Feedback

Six weeks of real underwriter usage surfaces a request discovery didn't anticipate: underwriters want the system to flag not just checklist pass/fail, but a brief natural-language explanation of why a borderline item was flagged, to speed up their escalation review. This is fed back into requirements (Part 12.1's discipline, applied mid-engagement per Part 13.1's lifecycle-loop point) rather than treated as a quick, undocumented patch — resulting in a deliberate, evaluated (Part 6.2) enhancement to the extraction prompt (Part 3.1) generating this explanation as a structured field (Part 3.2) alongside the pass/fail result.

Stage 8: Measuring Business Impact (Part 15)

At the nine-month mark, the impact report returns to the exact success metric from Stage 1 (Part 15, section 4's discipline: never a retroactively-chosen, more flattering metric): average review time for routine applications, measured via a controlled comparison (Part 15's attribution discipline) against a holdout subset still processed the prior, fully manual way during the same period.

report = generate_impact_report(
    treatment_times=automated_review_times,  # minutes, from the AI-assisted subset
    control_times=manual_review_times,        # minutes, from the holdout subset, same period
    hourly_cost_usd=underwriter_fully_loaded_hourly_cost,
    monthly_operating_cost_usd=system_total_monthly_cost,
)
# {"time_reduction_pct": 59.1, "net_benefit_usd": 47_200, "roi_ratio": 6.8}

A 59% controlled, attribution-aware time reduction, with the system's full operating cost (Part 7.10) netted out, produces a genuine, defensible 6.8x ROI ratio — the credible business case (Part 15, section 4's exact discipline) that directly justifies the customer's decision to expand the system to their claims-review department, closing Part 13.1's lifecycle loop back into a fresh Stage 1 discovery conversation for that new scope.

---

Key takeaways

• Every stage in this engagement applied a specific, previously-taught technique at the exact moment it was needed — discovery's five questions, feasibility's real-data probe, prototyping's single-riskiest-assumption focus, layered stakeholder communication, and attribution-aware impact measurement all compose into one coherent, real professional practice.
• The prototype's sub-type accuracy breakdown (Stage 3) is what turned a generic "extraction works pretty well" impression into a specific, actionable design decision (the non-standard-format fallback) — exactly the value Part 13.2 argues prototyping exists to provide.
• The engagement's success traces directly back to Stage 1's discovery-established metric being the exact metric measured in Stage 8, nine months and several iterations later — never substituted for something more convenient to report.

Recommended next chapter

Part 16 complete. Continue to handbook/17-interview-prep/01-system-design-interview.md.