11.2 — System Design: Enterprise RAG Platform

Scenario: A mid-size insurance company wants an internal system letting underwriters and claims adjusters ask natural-language questions against their policy documents, underwriting guidelines, and historical claims decisions (roughly 200,000 documents, growing by a few thousand per month), with answers citing specific source documents.

1. Requirements

Underwriters and adjusters need fast, accurate, cited answers to policy/guideline questions without manually searching through documents; the system must reduce time spent on document lookup measurably (Part 15's ROI framing) while never presenting an ungrounded or overconfident wrong answer as if it were verified fact.

2. Constraints

• Documents live across three internal systems (Part 10.2's realistic, messy starting point): an old document management system, a newer SharePoint deployment, and a claims database.
• Regulatory requirement: any answer influencing an actual underwriting/claims decision must be traceable to its source documents (an audit requirement, Part 10.5's governance discussion).
• Data residency: all data must remain within the company's home country's cloud regions (Part 7.3).
• Budget: a mid-size company, cost-sensitive at scale (Part 7.10 matters more here than at a large enterprise with more budget slack).

3. Functional Requirements

• Natural-language Q&A with source citations (Part 3.5).
• Handles both broad policy questions and precise lookups (e.g., "what's the exact deductible threshold in clause 4.2.1") — meaning hybrid search (Part 3.6) is likely necessary given the exact-clause-lookup pattern.
• Respects each document's existing access permissions from its source system (Part 10.4's document-level permission preservation).

4. Non-Functional Requirements

• p95 latency under 5 seconds for a typical question (acceptable given this is an internal productivity tool, not a real-time customer-facing chat).
• Must support roughly 500 concurrent internal users at peak (a specific, moderate scale — not hyperscale).
• Zero cross-permission-boundary leakage (Part 10.4) — the single hardest non-functional requirement given the regulatory stakes.
• 99.5% availability (an internal tool, not requiring the extreme availability bar of a customer-facing system).

5. Architecture

source systems

Old DMS

SharePoint

Claims DB

Data pipelineextract, dedupe, chunk, embed

pgvectorPostgres · Part 1.5/3.4

User queryFastAPI

Rewrite + hybrid search + rerankPart 3.6

Permission filterPart 10.4

Generation + citationPart 3.5/3.1

Enterprise RAG — nightly ingestion into pgvector on the left, the live query path filtered by permission before generation on the right.

6. Components

• Data pipeline: handles the three-source deduplication and permission-metadata preservation this scenario's messy real-world data requires (Part 10.2/10.4).
• pgvector on existing Postgres: chosen over a dedicated vector DB given moderate scale (Part 1.5's cross-reference: this is exactly the "customer already runs Postgres, scale doesn't yet demand a dedicated system" case).
• Hybrid search + reranking: necessary given the mixed query pattern (broad policy questions and precise clause lookups, Part 3.6).
• Document-level permission filter: the single most safety-critical component given the regulatory stakes (Part 10.4).
• Citation-generating LLM layer: grounded generation with explicit source attribution (Part 3.5), using a mid-tier model (Part 3.11) given this is a productivity tool, not requiring the largest available model for every query.

7. Data Flow

1. Documents are ingested nightly (batch, Part 10.2 — acceptable staleness given policy documents don't change hourly) from all three sources, deduplicated, chunked with permission metadata preserved.
2. A user submits a question via the internal web UI.
3. The query is rewritten if conversational (Part 3.6), embedded, and searched via hybrid search against pgvector.
4. Results are filtered to only documents the requesting user is authorized for (Part 10.4), then reranked.
5. The top results are passed to the LLM with an instruction to answer only from provided context and cite sources (Part 3.1/3.5).
6. The response, with citations, is returned and logged (Part 6.1) for audit purposes.

8. Failure Modes

• LLM provider outage: falls back to a secondary provider (Part 3.11/7.9) or, if both are unavailable, returns raw retrieved documents with a clear "AI summary unavailable, here are the source documents" message rather than failing entirely.
• Vector search returns nothing relevant: the system explicitly states it found no relevant information rather than the LLM guessing (Part 2.6/3.5's core mitigation).
• Permission filter bug: the highest-severity failure mode given regulatory stakes — mitigated by Part 10.4's defense-in-depth runtime assertion (a TenantVerifiedRetriever-style check) in addition to the primary filter.
• Stale document index: a document updated in SharePoint but not yet re-indexed (nightly batch) could surface outdated guidance for up to 24 hours — an accepted, explicitly communicated trade-off given this isn't a use case requiring real-time freshness (Part 8.2).

9. Security

• Document-level permission enforcement (Part 10.4) is the primary security control, tested explicitly and rigorously (Part 9.6's audit-every-layer principle) given the regulatory stakes.
• Prompt injection defense (Part 9.1) applied to retrieved document content, since an internal document corpus, while more trusted than public web content, is still not fully immune to accidental or malicious embedded content.
• All access logged with full audit trail (Part 6.1) for regulatory traceability (constraint 2).

10. Scalability

At 500 concurrent users and 200,000 documents growing modestly, this system doesn't need aggressive horizontal scaling infrastructure (Part 7.4/7.5) — a modest number of FastAPI instances behind a load balancer, with pgvector's HNSW index (Part 3.4) comfortably handling this document volume. Growth path: if the document corpus or user base grows an order of magnitude, revisit whether pgvector still meets latency requirements or a dedicated vector DB (Part 3.4) becomes justified.

11. Observability

Faithfulness and citation-accuracy as explicit online evaluation metrics (Part 6.3/8.2), given the regulatory stakes of an ungrounded answer; per-query cost and latency tracked (Part 7.10); permission-filter behavior specifically monitored/alerted (Part 8.4) given its safety-criticality.

12. Cost

Primary cost drivers: LLM generation calls (moderate volume, mid-tier model, Part 3.11's cost-appropriate routing), embedding costs for the nightly ingestion pipeline (proportional to document volume, not per-query), and existing Postgres infrastructure (no new dedicated vector DB cost, Part 1.5's cost advantage at this scale). Overall cost profile is modest relative to the underwriter/adjuster time savings it's designed to produce (Part 15's ROI framing).

Worked numeric example: with 500 concurrent users, assume each does roughly 20 queries/business day (a moderate, internal-productivity-tool usage rate) across ~22 business days/month → 500 × 20 × 22 = 220,000 queries/month. Each query retrieves ~5 chunks (~500 tokens each = 2,500 tokens) plus the question, generating a ~300-token cited answer, on a mid-tier model priced at roughly $3/million input tokens and $15/million output tokens (Part 3.11): cost per query ≈ (2,800/1,000,000 × $3) + (300/1,000,000 × $15) ≈ $0.0084 + $0.0045 ≈ $0.013/query. Monthly generation cost ≈ 220,000 × $0.013 ≈ $2,860/month — a small fraction of even a single underwriter's monthly fully-loaded cost, comfortably justifying the build against the time-savings ROI case (Part 15).

At 500 concurrent users with p95 latency under 5 seconds, peak throughput needed is modest: even a pessimistic assumption of all 500 users querying within the same 10-minute peak window is only 500/600 ≈ 0.83 QPS, well within a small FastAPI deployment's capacity (Part 7.4/7.5) without aggressive horizontal scaling.

13. Trade-offs

Chose pgvector over a dedicated vector database, trading some theoretical scale headroom for lower cost and operational simplicity (Part 1.5/3.4) — appropriate given this system's actual, moderate scale. Chose nightly batch ingestion over real-time webhook-driven updates (Part 10.1), trading some freshness for significantly simpler pipeline infrastructure — appropriate given policy documents' realistic update frequency.

14. Alternatives

A dedicated vector database (Part 3.4) was considered and rejected given current scale doesn't justify its added cost/complexity. A fully real-time, webhook-driven ingestion pipeline (Part 10.1) was considered and rejected as disproportionate engineering investment given this use case's actual staleness tolerance — revisit both decisions if scale or freshness requirements change materially.

15. Code Example

The document-level permission filter (section 6/9's single most safety-critical component), implemented as a defense-in-depth runtime assertion rather than trusted to the retrieval query alone:

class PermissionViolationError(Exception):
    """Raised when a retrieved document would leak across a permission boundary."""


def enforce_document_permissions(
    retrieved_docs: list[dict], requesting_user_id: str, get_authorized_doc_ids
) -> list[dict]:
    """
    Filters retrieved documents to only those the requesting user is
    authorized for, and raises loudly if the primary retrieval query's
    own filter appears to have failed — a second, independent check
    rather than trusting one filtering layer alone.

    Args:
        retrieved_docs (list[dict]): Documents returned by the retrieval
            query, each with a "doc_id" key.
        requesting_user_id (str): The user issuing the query.
        get_authorized_doc_ids: A callable returning the set of document
            IDs this user is authorized to see, sourced independently
            from the retrieval query's own permission filter.

    Returns:
        list[dict]: Only the documents the user is actually authorized for.

    Raises:
        PermissionViolationError: If the retrieval query returned any
            document outside the user's authorized set — signaling a
            bug in the primary filter that must never silently pass through.
    """
    authorized_ids = get_authorized_doc_ids(requesting_user_id)
    unauthorized = [d for d in retrieved_docs if d["doc_id"] not in authorized_ids]
    if unauthorized:
        raise PermissionViolationError(
            f"Primary retrieval filter leaked {len(unauthorized)} unauthorized "
            f"document(s) to user {requesting_user_id} — failing closed."
        )
    return retrieved_docs

16. Interview questions

• Given 500 concurrent users and moderate query volume, walk through estimating monthly LLM cost for this system, and explain why pgvector rather than a dedicated vector database is the right infrastructure choice at this scale.
• Why does the document-level permission filter need a second, independent runtime check rather than trusting the retrieval query's own filter alone?

17. FDE/customer scenario

CUSTOMER: "Just point it at all our SharePoint documents for now — we'll sort out permissions later once it's working."

The FDE-correct response declines to build this way even for a prototype (Part 13.2's exception for baseline data-handling discipline) — given this system's regulatory stakes (constraint 2) and the permission filter's status as the single highest-severity failure mode (section 8), permission enforcement is designed in from the first prototype, not retrofitted once "it's working," since retrofitting access control after users have already queried unfiltered content is a materially different, much worse problem than building it in from day one.

---

Key takeaways

• Enterprise RAG design must weigh real, messy multi-source data (Part 10.2) and document-level permission preservation (Part 10.4) as seriously as the retrieval/generation pipeline itself — often more so, given the regulatory stakes.
• Infrastructure choices (pgvector vs. dedicated vector DB, batch vs. real-time ingestion) should match actual, current scale and freshness requirements, not default to the most sophisticated available option.
• Failure modes for a RAG system must include what happens on retrieval failure, provider outage, and — most critically here — a permission-filter bug, each with an explicit, designed response.

Things you should be able to explain

• Why document-level permission enforcement is the single highest-stakes component in this specific design.
• Why pgvector, not a dedicated vector database, is the right choice at this system's actual scale.

Things you should be able to build

• A full enterprise RAG pipeline with hybrid search, reranking, document-level permission filtering, and citation-grounded generation.

Common mistakes

• Underinvesting in the data pipeline and permission-preservation work relative to the retrieval/generation pipeline.
• Choosing infrastructure sophistication disproportionate to actual scale requirements.

Recommended next chapter

03-design-customer-support-agent.md