11.5 — System Design: Document Intelligence Platform

Scenario: A logistics company needs a platform that ingests diverse incoming documents (bills of lading, customs forms, invoices — a mix of digital PDFs, scanned images, and occasional handwritten forms), extracts structured data from each, and validates it against expected business rules before routing to downstream systems.

1. Requirements

Automate what's currently manual data entry from diverse incoming logistics documents, extracting structured fields (shipment ID, weight, customs codes, dates) with high accuracy, flagging anything uncertain for human review rather than silently propagating errors into downstream systems.

2. Constraints

• Document quality varies enormously — clean digital PDFs, poor-quality scans, occasional handwriting (Part 3.10's multimodal discussion, and Part 3.10's precision-limitation warning is directly relevant given accuracy stakes here).
• Downstream systems (customs clearance, billing) require highly reliable structured data — an incorrect customs code has real regulatory/financial consequences.
• High volume: tens of thousands of documents per day, requiring genuine throughput, not just per-document accuracy.

3. Functional Requirements

• Multi-format document ingestion (PDF, scanned image, Part 3.10).
• Structured field extraction matching a defined schema per document type (Part 3.2).
• Business-rule validation (e.g., does the extracted weight fall within a plausible range; does the customs code match the described goods).
• Human-review queue for low-confidence or validation-failed extractions.

4. Non-Functional Requirements

• High extraction accuracy is paramount — given regulatory/financial stakes, a wrong-but-confident extraction is worse than a flagged-for-review one (directly connecting to Part 3.10's general-vs-specialized-model precision discussion).
• Throughput: must process the full daily volume within a defined business-hours window, requiring genuine batch/queue-based processing (Part 7.6/7.7), not real-time synchronous handling per document.
• Auditability: every extraction must be traceable to its source document and the specific model/version that produced it (Part 6.1/1.7).

5. Architecture

Incoming documentsbatch/streaming

Document queuePart 7.7

Format-specific pre-processingOCR for scans (Part 3.10)

Structured extractionschema per document type (Part 3.2)

Business-rule validationdeterministic checks, Part 3.2's semantic layer

Passes validation→ downstream systems

Flagged for human review queue

6. Components

• Queue/worker architecture (Part 7.6/7.7): given the throughput requirement and per-document processing time, this is a job-queue pattern, not synchronous request/response.
• Format-specific preprocessing: OCR for scanned documents (Part 3.10), with explicit confidence scoring from the OCR step itself feeding into overall extraction confidence.
• Structured extraction with independent semantic validation (Part 3.2's two-layer pattern) — schema-enforced generation plus business-rule checks, not schema conformance alone.
• Confidence-based routing: extractions below a confidence threshold, or failing a validation rule, route to human review rather than propagating automatically (Part 5.4/8.5's calibrated human-in-the-loop).
• Hybrid model strategy (Part 3.10): a general multimodal model handles the bulk of extraction, with the option to route particularly poor-quality or ambiguous documents to a specialized OCR/vision service for a second opinion where stakes warrant it.

7. Data Flow

1. Documents arrive (batch upload or a streaming intake) and enter the processing queue.
2. Each document is format-classified and preprocessed (OCR if scanned).
3. Structured extraction runs against the appropriate schema for that document type.
4. Extracted fields are validated against business rules (deterministic checks, Part 3.2).
5. Passing extractions route to downstream systems; failing or low-confidence ones route to a human review queue with the original document and extracted fields side-by-side for quick correction.
6. Human corrections are logged and, where a pattern emerges, feed back into evaluation datasets (Part 6.2/6.3's continuous-improvement loop) to identify systematic extraction weaknesses.

8. Failure Modes

• OCR fails on a very poor-quality scan: flagged immediately for human review rather than passing garbage text into extraction, which would produce a confidently wrong (not just uncertain) result.
• Queue backlog during a volume spike: worker pool scales based on queue depth (Part 7.7), with an explicit SLA-monitoring alert (Part 8.4) if the backlog threatens the business-hours processing window.
• Systematic extraction error on a specific document sub-type: caught via the human-correction feedback loop (Part 6.3) surfacing a pattern, rather than each individual error being treated as isolated.

9. Security

Documents may contain sensitive shipment/customer data (Part 9.6) — access to the review queue and extracted data scoped by RBAC (Part 9.4); documents themselves are untrusted input to any code-execution-adjacent processing step (Part 9.1's principle extended to document content, not just text prompts).

10. Scalability

Queue-based architecture (Part 7.6/7.7) scales naturally with worker-pool count, decoupled from any front-end request-serving capacity — directly matching this system's actual load pattern (high, somewhat predictable daily volume, not unpredictable user-driven traffic).

11. Observability

Extraction accuracy (validated against human-corrected ground truth, Part 8.1) as the primary quality SLI; queue depth and processing latency as throughput SLIs (Part 8.4), with explicit alerting if the backlog risks missing the business-hours processing window (constraint 3's throughput requirement).

12. Cost

Multimodal model calls (Part 3.10) are the primary per-document cost driver, multiplied by daily volume (Part 7.10) — worth evaluating whether a smaller/cheaper model suffices for cleaner digital-PDF documents (the majority case) while reserving a larger model or specialized OCR service for poor-quality scans specifically (Part 3.11's routing-by-complexity pattern, applied here by document quality rather than task type).

Worked numeric example: at "tens of thousands of documents per day" (constraint 3), take 50,000 documents/day as representative, with roughly 80% clean digital PDFs and 20% scans/handwriting needing full multimodal OCR (Part 3.10). Digital-PDF extraction on a small, cheap model (~800 input + 150 output tokens at $0.25/$1.25 per million): (800/1,000,000 × $0.25) + (150/1,000,000 × $1.25) ≈ $0.0002 + $0.0001875 ≈ $0.00039/doc. Scanned/handwritten extraction on a larger multimodal model (~1,500 token-equivalent input + 150 output at $3/$15 per million): (1,500/1,000,000 × $3) + (150/1,000,000 × $15) ≈ $0.0045 + $0.00225 ≈ $0.00673/doc. Weighted average: (0.8 × $0.00039) + (0.2 × $0.00673) ≈ $0.00031 + $0.00135 ≈ $0.00166/doc → daily cost ≈ 50,000 × $0.00166 ≈ $83/day, or roughly $2,490/month — directly validating section 12's quality-routing recommendation, since routing all 50,000 documents through the larger model instead would cost roughly 4x as much for the 80% majority that didn't need it.

For throughput: processing 50,000 documents within an 8-hour business-hours window (non-functional requirement 2) requires a sustained rate of 50,000 / (8 × 3,600) ≈ 1.74 documents/second from the worker pool (Part 7.6/7.7), with the queue absorbing any arrival-pattern burstiness above that average rate.

13. Trade-offs

Chose confidence-based human-review routing over either (a) fully automated processing (unacceptable given regulatory/financial stakes of a wrong extraction) or (b) full human review of every document (defeats the automation purpose) — a calibrated middle ground directly following Part 5.4/9.2's principle of reserving human intervention for genuinely uncertain or high-stakes cases.

14. Alternatives

A fully automated pipeline with no human review was considered and rejected given the real regulatory/financial consequences of a wrong customs code or shipment weight (constraint 2) — the cost of occasional human review is justified by the cost avoidance of downstream errors. A specialized, dedicated OCR/extraction model trained specifically on this company's document types was considered as a future optimization once sufficient human-corrected training data accumulates through the review-queue feedback loop (Part 2.2's fine-tuning discussion) — not pursued initially given the upfront data/training investment required.

15. Code Example

The confidence-based human-review routing check (section 6/9's central safety mechanism) — the component that decides whether an extraction is trustworthy enough to reach downstream systems unreviewed:

from dataclasses import dataclass


@dataclass
class ExtractionResult:
    document_id: str
    fields: dict
    field_confidences: dict[str, float]
    passed_business_rules: bool


CONFIDENCE_THRESHOLD = 0.90


def route_extraction(result: ExtractionResult) -> str:
    """
    Routes an extraction result to downstream systems or to human review,
    based on both per-field confidence and business-rule validation —
    never propagating a confidently wrong OR a rule-violating extraction.

    Args:
        result (ExtractionResult): The extraction output, including
            per-field confidence scores and business-rule validation status.

    Returns:
        str: "downstream" if every field clears the confidence threshold
            AND business rules pass; otherwise "human_review".
    """
    lowest_confidence = min(result.field_confidences.values())
    if lowest_confidence >= CONFIDENCE_THRESHOLD and result.passed_business_rules:
        return "downstream"
    return "human_review"

16. Interview questions

• Walk through estimating daily and monthly extraction cost given the document-quality mix (digital vs. scanned) and explain why routing by document quality, not a single model for everything, matters at this volume.
• Why does the routing check use the lowest per-field confidence rather than an average across fields?

17. FDE/customer scenario

CUSTOMER: "90% accuracy sounds great — can we just skip the human review queue entirely?"

The FDE-correct response explains the actual risk concretely (constraint 2's regulatory/financial stakes): 90% field-level accuracy across tens of thousands of documents/day still means thousands of wrong extractions daily if none are caught, and the review queue exists specifically to catch the ~10% the model itself signals uncertainty about, not to second-guess the 90% it's confident and correct on — a calibrated middle ground (section 13), not full manual review of everything.

---

Key takeaways

• Document intelligence at real volume is fundamentally a queue/worker architecture problem (Part 7.6/7.7), not a synchronous request/response one — matching the batch-oriented, throughput-driven nature of the actual workload.
• Confidence-based human-review routing is the calibrated middle ground between unacceptable full automation and unnecessary full manual review, directly applying Part 5.4/9.2's human-in-the-loop principles.
• Document quality variance (clean PDFs vs. poor scans) justifies a hybrid model-routing strategy by document quality, not just by task complexity.

Things you should be able to explain

• Why this system is a queue-based architecture rather than a synchronous API.
• Why confidence-based routing to human review is preferable to both full automation and full manual review.

Things you should be able to build

• A queue-driven extraction pipeline with schema-enforced extraction, business-rule validation, and confidence-based human-review routing.

Common mistakes

• Fully automated extraction with no human-review safety net for regulatory/financially-consequential data.
• Treating document processing as a real-time, synchronous problem rather than a batch/queue one.

Recommended next chapter

06-design-ai-workflow-automation.md