11.9 — System Design: AI API Platform

Scenario: A company wants to expose several of its internal AI capabilities (document summarization, entity extraction, sentiment classification) as a public, developer-facing API product that external companies can integrate into their own applications and pay for by usage — directly analogous to Part 7.9's internal LLM gateway, but now facing external, paying, less-trusted developers instead of internal teams.

1. Requirements

Offer reliable, well-documented, metered API access to internal AI capabilities for external developers, with billing accuracy, strong abuse resistance (since external traffic is inherently less trusted than internal traffic), and a developer experience good enough to drive adoption and integration.

2. Constraints

• External developers are a fundamentally less-trusted traffic source than Part 7.9's internal services — every security consideration from Part 9.5's API-security discussion applies with maximum severity here, not just as good practice.
• Billing must be accurate to the individual API call, since customers pay based on usage — an under-counted or over-counted usage metric is a direct financial/trust problem with paying customers.
• Developers expect industry-standard API ergonomics (clear docs, predictable rate limits, versioning, Part 1.2) since they're evaluating this against other API products they could integrate instead.

3. Functional Requirements

• Multiple distinct AI capabilities exposed as separate, well-documented endpoints (Part 1.2/1.3).
• API key-based authentication and per-key usage metering (Part 9.4/9.5).
• Tiered rate limiting and pricing plans (free tier, paid tiers with different rate limits/quotas).

4. Non-Functional Requirements

• High availability — external developers building production applications on top of this API expect strong uptime guarantees, likely with a published SLA.
• Strict, enforced rate limiting per API key — both for fair resource allocation across customers and, critically, as a security/cost-abuse control given untrusted external traffic (Part 9.5's exact cost-attack concern, now facing genuinely external, potentially adversarial callers rather than internal misconfigurations).
• Accurate, real-time (or near-real-time) usage metering for billing.

5. Architecture

External developer application

API Gatewayfacing EXTERNAL traffic — API key auth (Part 9.4), per-key rate limiting/quota (Part 9.5), request validation (Part 1.9)

Internal AI capability servicessummarization, extraction, classification (Part 3.1-3.12)

Usage metering + billing pipelineaccurate, per-call (Part 7.9 usage-logging pattern, applied to billing)

6. Components

• API gateway with strict external-facing security (Part 9.5): every request from an external caller is untrusted input requiring full validation (Part 1.9), API-key authentication, and enforced rate limiting — a meaningfully stricter posture than Part 7.9's internal gateway, which could reasonably trust its internal callers more.
• Internal AI capability services: the actual summarization/extraction/classification logic, built using this book's full Part 3 toolkit, exposed only through the gateway, never directly to external traffic.
• Usage metering and billing pipeline: must be accurate and auditable (constraint 2) — a distinct, dedicated system from operational cost-tracking (Part 7.10), since billing errors have direct financial/trust consequences with paying external customers in a way internal cost-tracking inaccuracies don't.

7. Data Flow

1. An external developer's application calls a documented API endpoint with their API key.
2. The gateway authenticates the key, checks rate limit/quota, and validates the request payload.
3. The request routes to the relevant internal AI capability service.
4. The response returns to the developer, and the call is logged for both operational observability (Part 8.4) and billing metering, ideally as one atomic logging step to avoid any discrepancy between what's served and what's billed.

8. Failure Modes

• A malicious or buggy external caller sends malformed/adversarial input: caught by strict input validation at the gateway (Part 1.9/9.1), never reaching internal AI services unvalidated.
• A billing/metering discrepancy (a call served but not correctly logged for billing, or vice versa): the highest-severity failure mode for this specific business model — mitigated by atomic, transactional logging (serving and metering as one unit, not two separately-failable steps) and reconciliation monitoring (Part 8.4) comparing served-request counts against billed-usage counts.
• A rate-limit bypass or abuse pattern: Part 9.5's cost-attack concern realized concretely — mitigated by the gateway's enforced rate limiting plus anomaly monitoring (Part 8.4) for unusual usage patterns suggesting key compromise or abuse.

9. Security

This is the highest-external-exposure design in this Part, warranting the strictest application of Part 9.5's API-security principles: never trust external input under any circumstance, enforce rate limiting as a hard security control (not just a fairness mechanism), and treat API key compromise as a real, expected event requiring rotation/revocation capability (Part 9.4's OAuth-adjacent token-management discipline, applied to API keys specifically).

10. Scalability

The gateway and internal AI services should scale independently (Part 7.4/7.5) — a traffic spike from one large external customer's legitimate high-volume usage shouldn't be conflated with an abuse pattern, requiring the rate-limiting/quota system to distinguish "high but legitimate, paid-for usage" from "abuse," typically via the customer's actual subscribed tier and quota rather than a one-size-fits-all limit.

11. Observability

Per-API-key usage, latency, and error-rate metrics (Part 8.4) serve both operational health monitoring and developer-facing usage dashboards (a real product feature developers expect) — and billing-reconciliation monitoring (section 8) as a dedicated, high-priority observability concern given its direct financial-trust implications.

12. Cost

The underlying LLM/compute cost per API call (Part 7.10) must be well understood and priced with a healthy margin into the external-facing pricing tiers — a common, serious business-model mistake is pricing an AI API product without accurately modeling the actual per-call cost (Part 7.10's cost-equation discipline), risking a pricing structure that's unprofitable at real usage volume.

Worked numeric example: assume 500 API customers averaging 10,000 calls/month each → 5,000,000 calls/month. Each call performs a lightweight classification/extraction task (~300 input + 100 output tokens) on a small, cheap model at $0.25/$1.25 per million tokens: (300/1,000,000 × $0.25) + (100/1,000,000 × $1.25) ≈ $0.000075 + $0.000125 ≈ $0.0002/call → underlying compute cost ≈ 5,000,000 × $0.0002 ≈ $1,000/month. If the platform prices calls at, say, $0.002/call (a 10x margin over raw compute cost — a healthy, defensible margin per section 12's warning), monthly revenue ≈ $10,000/month against ~$1,000/month underlying cost, leaving margin to cover the gateway infrastructure, billing pipeline, and support cost this platform also requires (section 6) — pricing at a thinner margin than this risks the exact unprofitable-at-volume mistake section 12 warns against.

At 5,000,000 calls/month with a plausible 5x peak-to-average traffic ratio (external developer traffic is typically far burstier than internal traffic), average throughput is 5,000,000 / (30 × 86,400) ≈ 1.93 calls/sec, implying a peak design target of roughly 10 calls/sec — the number that should actually drive the gateway's rate-limiting and autoscaling configuration (Part 7.5), not the monthly average.

13. Trade-offs

Chose strict, security-maximalist input validation and rate limiting for this specific platform (versus Part 7.9's more trusting internal-gateway posture) explicitly because the traffic source (external, less-trusted developers) genuinely warrants a different security calculus — the same underlying gateway pattern from Part 7.9, deliberately configured more strictly given a fundamentally different trust boundary.

14. Alternatives

A less strict, more internally-trusting API posture (treating external developers similarly to internal services) was considered and rejected as inappropriate given the fundamentally different trust and abuse-risk profile of external, paying, potentially adversarial traffic versus internal services (Part 9.5's core distinction) — the security investment here is not optional hardening but a direct requirement of the business model itself.

15. Code Example

Atomic serve-and-meter logging (section 8's mitigation for the platform's highest-severity failure mode — a billing/serving discrepancy) — recording that a call was served and that it was billed as one indivisible operation:

class BillingLogFailedError(Exception):
    """Raised when usage cannot be durably recorded — the response must not ship regardless."""


async def serve_and_meter(api_key: str, request_payload: dict, call_ai_capability, usage_store) -> dict:
    """
    Serves an API call and records its billing-relevant usage as one
    atomic operation, so a served-but-unbilled or billed-but-unserved
    discrepancy (section 8's highest-severity failure mode) cannot occur.

    Args:
        api_key (str): The calling developer's API key.
        request_payload (dict): The validated request body.
        call_ai_capability: The internal AI capability service to invoke.
        usage_store: A transactional store recording per-call usage
            for billing, supporting an atomic commit alongside serving.

    Returns:
        dict: The AI capability's response, returned ONLY after usage
            was durably recorded.

    Raises:
        BillingLogFailedError: If usage cannot be durably recorded —
            the response is withheld rather than served unbilled.
    """
    response = await call_ai_capability(request_payload)
    try:
        await usage_store.record_call(api_key=api_key, tokens_used=response["usage"]["total_tokens"])
    except Exception as exc:
        raise BillingLogFailedError(
            f"Usage recording failed for key {api_key} — withholding response "
            f"rather than serving a call that won't be billed."
        ) from exc
    return response

16. Interview questions

• Walk through estimating monthly underlying compute cost given customer count and calls/customer, and explain how that number should inform a defensible per-call price with margin.
• Why does the serve-and-meter operation need to be atomic, and what does the code choose to do when the billing side of that operation fails?

17. FDE/customer scenario

CUSTOMER (a developer integrating the API): "We got billed for calls that returned errors on our end — shouldn't failed calls be free?"

The FDE-correct response applies the reasoning skeleton (Part 12.3): the problem (perceived unfair billing) may have real evidence behind it (verify whether "failed" means the platform's error or the developer's own malformed request) — a request that reached and consumed the underlying AI capability's compute (Part 9.5) is a real cost regardless of what the developer's client did with the response, while a request rejected at the gateway before reaching that capability (invalid API key, malformed payload) should never be billed in the first place — the honest next step is confirming which case actually occurred from the logs, not assuming the customer's framing is correct or incorrect without evidence.

---

Key takeaways

• An externally-facing AI API platform warrants a meaningfully stricter security posture than an internal LLM gateway (Part 7.9), given the fundamentally different trust boundary of external, paying, potentially adversarial developer traffic.
• Billing accuracy is a distinct, high-stakes requirement from operational cost-tracking — atomic serve-and-meter logging and reconciliation monitoring are necessary to avoid direct financial/trust consequences with paying customers.
• Rate limiting functions as both a fairness mechanism and a hard security control against cost-abuse, requiring the ability to distinguish legitimate high-volume paid usage from actual abuse.

Things you should be able to explain

• Why an external-facing AI API platform needs a stricter security posture than an internal gateway serving the same underlying capabilities.
• Why billing accuracy requires dedicated, atomic logging distinct from general operational metering.

Things you should be able to build

• A strictly-validated, rate-limited, per-key-metered external API gateway with reconciliation monitoring between served and billed usage.

Common mistakes

• Applying an internally-trusting security posture to externally-facing traffic.
• Pricing an AI API product without accurately modeling the actual per-call underlying cost.
• Non-atomic serve-and-bill logging risking billing discrepancies.

Recommended next chapter

Part 11 complete. Continue to handbook/12-customer-engineering/01-customer-discovery-and-requirements.md.