11.4 — System Design: Multi-Tenant AI SaaS Platform

Scenario: A startup is building a SaaS product offering AI-powered meeting-notes summarization and action-item extraction to other businesses — ranging from small teams (a handful of users) to a few large enterprise customers with strict security/compliance requirements, all on one shared product.

1. Requirements

Serve hundreds of separate customer organizations from one product, with strong data isolation between them, while remaining cost-efficient enough to serve small customers profitably and secure enough to satisfy large enterprise customers' compliance requirements (Part 10.5).

2. Constraints

• Highly variable customer size and sophistication — from a 5-person startup to a 5,000-employee enterprise, on the same underlying product.
• A small number of large enterprise prospects have explicitly required database-per-tenant isolation (Part 10.5) as a contractual condition of purchase.
• Engineering team is small — the multi-tenant architecture must not require disproportionate per-tenant operational overhead for the (numerous) small customers.

3. Functional Requirements

• Meeting transcript upload/ingestion, summarization, and action-item extraction (Part 3.1/3.2's structured extraction).
• Per-tenant customization of summary format/focus (a real, common SaaS requirement — different customers want different things emphasized).
• Admin dashboard for each tenant's own usage/billing visibility.

4. Non-Functional Requirements

• Zero cross-tenant data leakage under any circumstance (Part 9.6/10.5) — the single most important non-functional requirement for this product category.
• Cost-to-serve must remain low enough for small-customer pricing tiers to be profitable (Part 7.10/15).
• Large enterprise tenants require the option of dedicated, isolated infrastructure (Part 10.5's database-per-tenant pattern) as a paid tier.

5. Architecture

Shared Application Layerstateless API servers (Part 7.4/7.5), serving ALL tenants regardless of isolation tier

Small/mid tenantsshared database, tenant_id scoping (Part 9.6)

Larger customersschema-per-tenant

Contractually-required dedicated isolationdatabase-per-tenant (Part 10.5)

6. Components

• Stateless application layer: the same codebase serves every tenant tier; tenant-tier routing (Part 10.5's get_database_connection_for_tenant pattern) determines which database connection a given request uses, keeping engineering overhead low despite supporting multiple isolation strengths.
• Per-tenant customization store: a structured configuration table (not baked into code) holding each tenant's summary-format preferences, read at request time.
• Billing/usage tracking: per-tenant cost attribution (Part 7.9/7.10's gateway-level visibility, applied per-tenant rather than per-internal-service) feeding both customer-facing usage dashboards and internal margin analysis per tenant tier.

7. Data Flow

1. A meeting transcript is uploaded by a user, tagged with their tenant ID and (if applicable) resolved via SSO group mapping (Part 10.1) to internal roles.
2. The request routes to the correct database connection based on the tenant's isolation tier (Part 10.5).
3. Summarization/extraction runs using the tenant's customized format preferences.
4. Results are stored back in the tenant-appropriate database, and usage is logged for billing attribution.

8. Failure Modes

• Tenant-isolation bug: the highest-severity failure mode for this entire product category — mitigated by Part 9.6's full-layer audit discipline (database, cache, any shared infrastructure) plus defense-in-depth runtime assertions.
• A large, dedicated-tier tenant's database instance failure: isolated to that specific tenant, not cascading to others — a direct benefit of the stronger isolation tier justifying its added operational cost for exactly this failure-containment property.
• Shared-tier database contention from one heavy-usage tenant: a "noisy neighbor" risk specific to the shared-database pattern — mitigated with per-tenant rate limiting/quota (Part 7.9's gateway pattern, applied per-tenant) to prevent one tenant's usage spike from degrading others' experience.

9. Security

Full-layer tenant isolation audit (Part 9.6) is the central security requirement — database, cache (Part 7.8), any shared observability/tracing access (Part 6.1), and billing data must all independently enforce tenant scoping. RBAC (Part 9.4) within each tenant for their own internal user roles, layered on top of the cross-tenant isolation.

10. Scalability

The shared-tier database must scale to serve the (numerous) small/mid customers efficiently (Part 7.4/7.5's standard horizontal scaling), while the dedicated tier scales per-tenant independently — a genuine architectural benefit of the tiered approach, since a large dedicated-tier tenant's own scale needs don't affect the shared tier's capacity planning at all.

11. Observability

Per-tenant cost, usage, and quality metrics (Part 7.9/8.4) are necessary both for customer-facing dashboards (a functional requirement) and for internal margin analysis — critically, tenant-isolation-specific monitoring (Part 9.6/10.4) as its own dedicated observability concern, given its outsized security importance relative to other metrics.

12. Cost

The tiered architecture directly optimizes cost-to-serve: the shared-database tier keeps small-customer costs low (Part 1.5/10.5's cost comparison), while dedicated-tier pricing should reflect and recover the genuinely higher infrastructure cost of database-per-tenant isolation (Part 10.5, section 12) — a direct connection between architecture and pricing strategy (Part 15).

Worked numeric example: assume 300 shared-tier tenants averaging 20 users each, with each user summarizing roughly 5 meetings/day across ~22 business days/month → 300 × 20 × 5 × 22 = 660,000 summarizations/month. Each summarization processes a ~3,000-token transcript and produces a ~300-token summary on a mid-tier model at $3/$15 per million tokens: cost ≈ (3,000/1,000,000 × $3) + (300/1,000,000 × $15) ≈ $0.009 + $0.0045 ≈ $0.0135/summarization → shared-tier generation cost ≈ 660,000 × $0.0135 ≈ $8,910/month, spread across 300 tenants ≈ $29.70/tenant/month — a number that must sit comfortably under the shared tier's price point for the product to be profitable at that tier (Part 15). Separately, assume 10 dedicated-tier enterprise tenants each on their own ~$200/month Postgres instance → $2,000/month additional infrastructure cost, which the dedicated tier's pricing premium must explicitly recover (section 12), not silently absorb.

13. Trade-offs

Chose a tiered multi-tenant architecture (not a single uniform pattern) specifically to serve both cost-sensitive small customers and isolation-requiring large enterprise customers from one product — trading some engineering complexity (supporting multiple isolation patterns) for meaningfully better fit across a genuinely heterogeneous customer base, versus either uniformly cheap-but-insufficiently-isolated or uniformly isolated-but-too-expensive-for-small-customers.

14. Alternatives

A uniform shared-database-only architecture was considered and rejected because it couldn't satisfy large enterprise customers' contractual isolation requirements (constraint 2), risking losing exactly the highest-value customer segment. A uniform database-per-tenant architecture for all customers was considered and rejected as disproportionately expensive for the numerous small customers whose pricing tier couldn't absorb that cost (Part 10.5, section 14).

15. Code Example

The tenant-tier-aware connection resolver (section 6's central component), with a defense-in-depth assertion ensuring a request can never silently resolve to the wrong tenant's database:

class TenantIsolationError(Exception):
    """Raised when a resolved database connection doesn't match the requesting tenant."""


def get_database_connection_for_tenant(tenant_id: str, tenant_registry: dict) -> "Connection":
    """
    Resolves the correct, tier-appropriate database connection for a
    tenant, and asserts the returned connection is actually scoped to
    that tenant before returning it — never trusting the registry
    lookup alone for a mistake this costly to get wrong.

    Args:
        tenant_id (str): The requesting tenant's ID.
        tenant_registry (dict): Maps tenant_id to its isolation tier and
            connection details (shared, schema-per-tenant, or dedicated).

    Returns:
        Connection: A connection scoped correctly to this tenant's tier.

    Raises:
        TenantIsolationError: If the resolved connection's own recorded
            tenant scope doesn't match the requested tenant_id.
    """
    tier_config = tenant_registry[tenant_id]
    connection = build_connection(tier_config)
    if connection.scoped_tenant_id != tenant_id:
        raise TenantIsolationError(
            f"Resolved connection scoped to {connection.scoped_tenant_id}, "
            f"expected {tenant_id} — failing closed rather than serving "
            f"a request against the wrong tenant's data."
        )
    return connection

16. Interview questions

• Walk through estimating monthly LLM cost for the shared tier given a stated number of tenants and users, and explain how that per-tenant cost should inform the shared tier's pricing.
• Why does the connection resolver assert the resolved connection's tenant scope rather than trusting the registry lookup alone?

17. FDE/customer scenario

CUSTOMER (a small-tier prospect): "Can we just get the same dedicated database your big enterprise customers get? We don't want to be on 'the cheap tier.'"

The FDE-correct response explains the actual cost driver honestly (section 12's per-tenant infrastructure cost) rather than either refusing outright or quietly absorbing the cost — offering the dedicated tier at its true, cost-reflective price, and letting the customer make an informed choice between the price difference and the isolation guarantee it buys, precisely the honest, itemized cost communication Part 12.3 argues for.

---

Key takeaways

• A tiered multi-tenant architecture (Part 10.5's three patterns, applied per customer segment) lets one product serve both cost-sensitive small customers and isolation-requiring enterprise customers without forcing a single, ill-fitting pattern on the whole customer base.
• Full-layer tenant-isolation auditing (Part 9.6) is the single highest-stakes design requirement for any multi-tenant AI SaaS product.
• Architecture and pricing strategy are directly connected — the dedicated isolation tier's real infrastructure cost should inform its pricing, not be absorbed silently.

Things you should be able to explain

• Why a tiered isolation architecture serves a heterogeneous customer base better than any single uniform pattern.
• Why "noisy neighbor" risk is specific to the shared-database tier and how to mitigate it.

Things you should be able to build

• A tenant-tier-aware routing layer supporting multiple isolation patterns from one shared application codebase.

Common mistakes

• Uniform architecture that either can't satisfy enterprise isolation requirements or is too expensive for small customers.
• Not pricing the dedicated isolation tier to reflect its genuinely higher infrastructure cost.

Recommended next chapter

05-design-document-intelligence-platform.md