16.2 — Project: Document Intelligence Platform

This project builds the logistics document-extraction platform designed in Part 11.5 into a real, runnable system. Read Part 11.5 first.

1. Customer Scenario

A logistics company (Part 11.5's scenario) needs high-volume, high-accuracy structured extraction from diverse incoming documents (bills of lading, customs forms, scans, occasional handwriting), with confidence-based routing to human review rather than silent, unreliable full automation.

2. Requirements

Recap of Part 11.5, sections 3-4: multi-format ingestion including OCR for scans (Part 3.10), schema-based structured extraction (Part 3.2) per document type, deterministic business-rule validation, human-review queue for low-confidence/failed-validation cases, throughput sufficient for tens of thousands of documents/day within a business-hours window.

3. Architecture

Part 11.5's queue-driven pipeline: document queue → format-specific preprocessing → structured extraction → validation → route to downstream systems or human review.

4. Technology Selection

• Celery + Redis (Part 7.7): the job-queue backbone, given the batch/throughput-oriented nature of this workload (Part 7.6's "not a synchronous request/response problem" argument).
• A multimodal LLM (Part 3.10) for the bulk of extraction, with a specialized OCR fallback for particularly poor-quality scans.
• Pydantic schemas per document type (Part 3.2) as the structured-extraction contract.
• PostgreSQL (Part 1.5) for extraction results, validation state, and the human-review queue.

5. Folder Structure

document-intelligence/
├── src/
│   ├── ingestion/
│   │   ├── format_detector.py      # classifies incoming document format
│   │   └── ocr_preprocessor.py     # Part 3.10, confidence-scored OCR
│   ├── extraction/
│   │   ├── schemas.py              # per-document-type Pydantic schemas (Part 3.2)
│   │   ├── extractor.py            # schema-enforced extraction (Part 3.2)
│   │   └── validators.py           # deterministic business-rule checks
│   ├── workers/
│   │   └── tasks.py                # Celery tasks (Part 7.7), idempotent (Part 5.7)
│   ├── review_queue/
│   │   ├── models.py               # human-review queue schema
│   │   └── api.py                  # review UI backend endpoints
│   └── feedback/
│       └── correction_logger.py    # feeds human corrections into eval dataset (Part 6.3)
├── tests/
│   ├── unit/                       # schema validation, business rules
│   ├── integration/                # queue processing, idempotency (Part 5.7/7.7)
│   └── eval/
│       └── extraction_accuracy.py  # Part 8.1's outcome evaluation, per document sub-type
├── infra/
│   ├── Dockerfile
│   └── k8s/
│       ├── worker-deployment.yml   # HPA scaled on QUEUE DEPTH (Part 7.7), not request rate
│       └── api-deployment.yml
└── requirements.txt

6. Implementation (key excerpt)

The confidence-based routing logic, directly implementing Part 11.5/5.4's calibrated human-in-the-loop pattern:

async def process_document(document_id: str, extractor, validator, review_queue) -> dict:
    """Extracts, validates, and routes a document to downstream systems or
    human review based on confidence and validation outcome."""
    extraction_result = await extractor.extract(document_id)

    if extraction_result.confidence < CONFIDENCE_THRESHOLD:
        await review_queue.add(document_id, reason="low_confidence", extraction=extraction_result)
        return {"status": "pending_review", "reason": "low_confidence"}

    validation_result = validator.validate(extraction_result)
    if not validation_result.passed:
        await review_queue.add(document_id, reason="validation_failed", extraction=extraction_result)
        return {"status": "pending_review", "reason": "validation_failed"}

    await route_to_downstream_systems(extraction_result)
    return {"status": "processed"}

7. Testing

Idempotency tests verifying a redelivered queue job doesn't duplicate a downstream write (Part 5.7/7.7's exact concern). Schema/validation unit tests per document type. A dedicated test verifying documents below the confidence threshold always route to review, never silently pass through.

8. Evaluation

Extraction accuracy measured per document sub-type (Part 11.5's section 4 breakdown pattern, directly reused from Part 13.2's prototyping example) using human-corrected ground truth accumulated through the review-queue feedback loop (Part 6.3).

9. Security

Documents may carry sensitive shipment/customer data (Part 9.6) — RBAC-scoped review-queue access (Part 9.4); documents treated as untrusted input to any downstream processing (Part 9.1's principle extended to document content).

10. Observability

Queue depth and processing latency as throughput SLIs (Part 8.4), explicitly alerted against the business-hours processing-window requirement; extraction accuracy tracked per document sub-type as the primary quality SLI.

11. Deployment

Worker pool and API layer deployed and scaled independently (Part 7.4/7.7) — worker replica count driven by queue depth specifically, not request rate.

12. Scaling

Natural horizontal scaling via additional Celery workers (Part 7.7) — the queue-based architecture is precisely what makes this straightforward, decoupled from any front-end capacity planning.

13. Cost Considerations

Multimodal model calls are the primary per-document cost driver (Part 3.10/7.10) — route cleaner digital-PDF documents (the majority) to a smaller/cheaper model, reserving a larger model or specialized OCR service for poor-quality scans specifically (Part 3.11's routing-by-input-quality pattern).

14. Failure Scenarios

OCR failure on unreadable scans → immediate route to review, never passed to extraction as garbage text (Part 11.5, section 8). Queue backlog during a volume spike → worker autoscaling plus explicit SLA alerting (Part 7.7/8.4).

15. Improvements

Once sufficient human-corrected training data accumulates via the feedback loop, evaluate whether a fine-tuned, specialized extraction model (Part 2.2) for this company's specific document types would improve accuracy or reduce cost versus the general multimodal model — a genuine, data-driven future decision, not a day-one commitment.

16. Business Metrics

Reduction in manual data-entry hours (Part 15's time-savings ROI pattern), plus reduction in downstream error rate (customs/billing errors avoided, Part 15's error-reduction ROI pattern) — both netted against the system's full operating cost including the ongoing human-review queue's labor cost.

---

Key takeaways

• This project's central design decision — confidence-based routing to human review rather than full automation — is a direct, concrete application of Part 5.4/9.2's calibrated human-in-the-loop principle to a genuinely high-stakes extraction task.
• Queue-depth-based worker autoscaling (Part 7.7), not request-rate-based scaling, is the correct mechanism for this batch-oriented workload's actual traffic pattern.
• The human-review feedback loop is both a safety mechanism and a continuous-improvement data source (Part 6.3), feeding directly into both evaluation and potential future fine-tuning decisions.

Recommended next chapter

03-multi-tenant-rag-saas.md