11.8 — System Design: Enterprise Knowledge Assistant

Scenario: A large, multi-department enterprise (10,000+ employees) wants a single internal AI assistant that can answer questions spanning HR policy, IT support, finance procedures, and department-specific knowledge — each domain currently siloed in different systems, owned by different teams, with different sensitivity levels.

1. Requirements

One assistant employees across the whole company can use for any internal question, correctly routing to and respecting the right domain's specific knowledge, access controls, and update cadence — without becoming an unmaintainable monolith as more departments want to add their own content.

2. Constraints

• Different departments (HR, IT, Finance, individual business units) own and maintain their own content independently — a centralized, single-team content-curation model doesn't match how the organization actually operates.
• Sensitivity varies enormously by domain — general IT help-desk content is broadly accessible; HR content involving specific employee cases is highly restricted (Part 10.4's document-level permissions, now compounded across many distinct domains with different ownership).
• Different domains have very different update frequencies — IT knowledge-base articles might change weekly; company-wide policy documents change rarely.

3. Functional Requirements

• Cross-domain natural-language Q&A with correct routing to the relevant domain's knowledge (Part 3.7's routing pattern, applied at the knowledge-domain level).
• Federated content ownership — each department manages its own content ingestion without needing central-team involvement for routine updates.
• Consistent citation/sourcing regardless of which domain answered the question (Part 3.5).

4. Non-Functional Requirements

• An employee must never see content from a domain/sensitivity level they're not authorized for (Part 10.4's document-level permissions, now the central architectural challenge given the multi-domain, multi-owner structure).
• The system must scale to onboard new departments/domains without requiring a full system redesign each time (an architectural scalability requirement distinct from Part 7.5's traffic-scaling — this is "scales to more content domains," not just "scales to more requests").
• Consistent user experience across domains despite federated ownership and varying content quality/structure.

5. Architecture

Federated Domain Structure

HR domainown ingestion, own permission model

IT domainown ingestion, own permission model

Finance domainown ingestion, own permission model

Business unit domainsown ingestion, own permission model

Domain routerclassifies query into the relevant domain(s) (Part 3.7)

Domain-specific RAGretrieval + permission filter, per-domain (Part 3.5/10.4)

Unified response generationconsistent citation format

6. Components

• Federated domain structure: each department owns its own ingestion pipeline and permission model (Part 10.2/10.4), plugging into a shared platform rather than a single, centrally-maintained corpus — directly addressing constraint 1's organizational reality.
• Domain router: classifies a query into the relevant domain(s) (Part 3.7's routing pattern, with the possibility of a query genuinely spanning multiple domains, requiring parallel retrieval across them, Part 3.7's parallelization pattern).
• Per-domain permission enforcement: each domain's own access-control rules (which can differ significantly — HR's rules aren't IT's rules) are enforced independently at retrieval time (Part 10.4), composed together for a query that spans domains.
• Unified generation layer: presents a consistent citation/response format to the end user regardless of which domain(s) actually answered, hiding the federated complexity from the employee's experience.

7. Data Flow

1. An employee asks a question via the shared assistant interface.
2. The domain router classifies which domain(s) the question relates to.
3. For each relevant domain, retrieval runs against that domain's own index, applying that domain's own specific permission rules to the requesting employee.
4. Results from all relevant domains are combined (if multi-domain) and passed to generation with unified citation formatting.
5. The response is returned, citing sources with clear domain attribution so the employee understands where each piece of information came from.

8. Failure Modes

• A query spans domains, but one domain's retrieval fails or times out: the system should degrade gracefully, answering from the domains that succeeded with an explicit note that one area of information wasn't available, rather than failing the entire response.
• A new department's onboarding introduces a permission-model bug: isolated to that specific department's domain given the federated architecture, not able to affect other domains' isolation (a direct architectural benefit of federation over a single monolithic system).
• Domain misclassification: a query classified into the wrong domain retrieves irrelevant results — mitigated by allowing multi-domain retrieval when router confidence is ambiguous (better to retrieve from a slightly broader set than to miss the correct domain entirely).

9. Security

Federated permission enforcement (section 6) is the central security architecture — critically, a bug in one domain's onboarding doesn't compromise other domains' isolation, a genuine security benefit of avoiding a single, shared permission-enforcement codebase across genuinely different business functions with genuinely different sensitivity models. Cross-domain query results must never leak a domain's restricted content into a response the requesting employee wasn't authorized for in that specific domain, even if they're authorized in others.

10. Scalability

Onboarding a new department means adding a new domain with its own ingestion/permission plugin, not modifying a shared, growing monolith (constraint 3's architectural-scalability requirement) — this federated pattern is specifically what makes constraint 3 achievable, versus a single unified corpus that would need redesign as more, increasingly heterogeneous content sources are added.

11. Observability

Per-domain quality and usage metrics (Part 8.1/8.4) let each department's content owners see how well their specific domain is performing (are their documents actually answering employees' questions well), supporting the federated-ownership model with federated accountability for content quality, not just federated technical ownership.

12. Cost

Cost attribution per domain (Part 7.9/7.10) supports a genuinely federated organizational model where departments might reasonably be charged back for their own domain's usage, aligning incentives for content-quality investment with the teams who actually benefit from good answers in their area.

Worked numeric example: at 10,000+ employees, assume 10% are active daily users (1,000) each asking ~2 questions/day → 2,000 queries/day. Assume roughly 30% of queries span more than one domain (constraint/section 6's multi-domain case), so the average query triggers 1.3 domain-retrieval passes. With ~2,000 context tokens per domain retrieved on average, an average query costs 1.3 × 2,000 = 2,600 input tokens plus a ~250-token unified answer, on a mid-tier model at $3/$15 per million: (2,600/1,000,000 × $3) + (250/1,000,000 × $15) ≈ $0.0078 + $0.00375 ≈ $0.01155/query → daily generation cost ≈ 2,000 × $0.01155 ≈ $23.10/day, or roughly $693/month company-wide — small enough that per-domain chargeback (section 12) is more about incentive alignment than materially significant cost recovery at this scale, worth stating explicitly when a department objects to being charged for a small number.

Peak-hour throughput: concentrating the 2,000 daily queries into an 8-hour work day gives an average of 2,000 / (8 × 3,600) ≈ 0.07 QPS, though a company-wide event (a benefits open-enrollment announcement, say) could plausibly spike this 5-10x briefly — worth explicit capacity headroom given how bursty internal-tool usage often is around specific announcements.

13. Trade-offs

Chose a federated domain architecture over a single unified corpus/permission model, trading some cross-domain consistency and a bit more architectural complexity (multiple ingestion pipelines, a routing layer) for genuinely better fit with the organization's actual, federated content-ownership reality and stronger isolation between domains' very different sensitivity/permission requirements.

14. Alternatives

A single, centrally-curated corpus with one unified permission model was considered and rejected — it would require a central team to become the bottleneck for every department's content updates (unrealistic at this organizational scale and structure, constraint 1) and would mix genuinely different sensitivity models into one permission system, increasing the risk of a single permission-modeling mistake affecting the entire system rather than being contained to one domain.

15. Code Example

The cross-domain permission composer (section 6/9's central safety mechanism for a query spanning domains) — each domain's own permission rule is evaluated independently, and a query result includes only what the user is authorized for in that specific domain:

def compose_multi_domain_results(
    domain_results: dict[str, list[dict]], requesting_user_id: str, domain_permission_checkers: dict
) -> dict[str, list[dict]]:
    """
    Applies each domain's own, independent permission check to that
    domain's retrieved results, never allowing a user's access in one
    domain to imply access in another.

    Args:
        domain_results (dict[str, list[dict]]): Raw retrieved results
            keyed by domain name (e.g., "hr", "it", "finance").
        requesting_user_id (str): The querying employee.
        domain_permission_checkers (dict): Maps domain name to that
            domain's own, independently-implemented permission-check
            callable — deliberately not a single shared function, so a
            bug in one domain's checker can't affect another's.

    Returns:
        dict[str, list[dict]]: Filtered results per domain, containing
            only content this user is authorized for IN THAT domain.
    """
    filtered: dict[str, list[dict]] = {}
    for domain, results in domain_results.items():
        check_domain_permission = domain_permission_checkers[domain]
        filtered[domain] = [
            r for r in results if check_domain_permission(requesting_user_id, r)
        ]
    return filtered

16. Interview questions

• Walk through estimating daily query cost given employee count, active-user rate, and the fraction of queries that span multiple domains — and explain why multi-domain queries cost more per query than single-domain ones.
• Why does each domain get its own independent permission-checker function rather than one shared permission-checking function used across all domains?

17. FDE/customer scenario

CUSTOMER (IT department lead): "Since we're onboarding fastest, can our domain just reuse HR's existing permission-checking code to save time?"

The FDE-correct response declines the shortcut — section 6/9's federated design deliberately keeps each domain's permission logic independent specifically so a bug in one domain's checker (or a genuine difference in how HR's and IT's sensitivity models actually work) can't silently affect the other domain's isolation guarantees; a small amount of duplicated implementation effort here is a worthwhile trade against the containment benefit section 9 describes.

---

Key takeaways

• Federated domain architecture — each department owning its own ingestion and permission model — matches how large, multi-department enterprises actually operate, and contains a permission-modeling mistake to one domain rather than risking the whole system.
• A domain-routing layer (Part 3.7) lets one unified assistant experience sit on top of genuinely federated, heterogeneous backend domains.
• Onboarding new departments should mean adding a new domain plugin, not modifying a growing shared monolith — this is what makes long-term organizational scalability achievable.

Things you should be able to explain

• Why federated domain ownership better matches large-enterprise organizational reality than a single centralized corpus.
• Why containing a permission bug to one domain is a genuine architectural security benefit of federation.

Things you should be able to build

• A domain-routing layer composing independently-permissioned, independently-owned retrieval domains into one unified assistant experience.

Common mistakes

• A single centralized corpus/permission model that becomes a bottleneck and mixes incompatible sensitivity models.
• No graceful degradation when one domain's retrieval fails for a multi-domain query.

Recommended next chapter

09-design-ai-api-platform.md