Appearance
7.9 — LLM Gateways
1. What is it?
An LLM gateway is a centralized proxy service sitting between all of your application's code and every LLM provider you call — every LLM request flows through this one layer, which handles cross-cutting concerns (rate limiting, circuit breaking, unified logging/cost tracking, model routing, Part 3.11) in one place, rather than each application/service independently re-implementing these concerns against each provider's raw SDK.
2. Why does it exist?
Part 3.11 established model routing conceptually — deciding which model handles which request. Part 7.5/7.7 established resilience patterns (rate limiting, retries, circuit breakers) generally. As an organization's AI usage grows beyond one application to many services, teams, and use cases, each independently calling LLM providers directly, several real problems emerge: inconsistent rate-limit handling across services (one service's burst can exhaust a shared provider quota for everyone), no centralized visibility into total organization-wide cost/usage, and duplicated integration code for every provider across every service. An LLM gateway exists to solve these once, centrally, rather than per-service.
3. What problem does it solve?
It solves "how do I get consistent rate limiting, cost visibility, provider fallback, and access control across every LLM call my organization makes, without every individual service/team re-implementing this independently and inconsistently" — a genuinely organizational-scale problem that individual application-level patterns (Part 3.11's in-application routing, Part 1.1's per-service concurrency bounds) don't fully address once you have more than one service calling LLMs.
4. How does it work internally?
The gateway as a unified proxy
Multiple internal services/teams
Service A
Service B
Service C
LLM Gatewayunified auth/access control · rate limiting/circuit breaking · request/response logging + cost tracking · model routing · provider fallback
Provider AAnthropic
Provider BOpenAI
Provider Cself-hosted model
Every internal service calls the gateway using one consistent internal API/SDK, rather than each service directly holding provider credentials and calling provider APIs independently — the gateway holds the actual provider credentials (Part 9.5) and handles the provider-specific translation (conceptually similar to Part 4.2's BaseChatModel abstraction, but implemented as a centralized network service rather than a client-side library).
Centralized rate limiting and quota management
Because every request from every internal service passes through one gateway, the gateway can enforce rate limits and quotas across services against a shared provider limit — preventing one service's traffic burst from silently exhausting a rate limit that other services also depend on, a coordination problem that's difficult or impossible to solve correctly if each service independently manages its own rate limiting against a shared, external provider quota it can't see other services' usage of.
Circuit breaking at the gateway layer
A circuit breaker (a resilience pattern this Part builds toward) tracks a provider's recent failure rate and, once it exceeds a threshold, "opens" — temporarily stopping new requests to that failing provider entirely (failing fast instead of piling up slow, doomed requests) and periodically testing whether the provider has recovered before "closing" again and resuming normal traffic. Implementing this centrally, in the gateway, means every service benefits from the same circuit-breaker protection and the same automatic provider-fallback behavior (Part 3.11's fallback pattern) without each service needing to implement this resilience logic independently.
Circuit breaker states:
CLOSED (normal) ──[failure rate exceeds threshold]──► OPEN (fail fast, no calls to provider)
│
[after cooldown period]
▼
HALF-OPEN (test with limited traffic)
│ │
[succeeds] [fails again]
▼ ▼
CLOSED OPENExisting gateway products — build vs. buy
Every mechanism in this chapter (circuit breaking, unified logging, model routing, fallback) is available in existing, mature gateway products, and a genuine build-vs-buy evaluation should happen before committing to a custom-built gateway. LiteLLM (open-source, self-hosted proxy with a unified OpenAI-compatible API across 100+ providers, built-in rate limiting and cost tracking) and Portkey (a managed gateway with observability, caching, and guardrail integrations layered on top of similar routing/fallback functionality) are the two most commonly evaluated options specifically for LLM-shaped traffic; Kong AI Gateway (an AI-specific plugin on top of Kong's established general-purpose API gateway, a natural fit for an organization already running Kong for non-AI APIs) and Cloudflare AI Gateway (a managed edge-layer gateway with built-in caching, rate limiting, and analytics, a natural fit for an organization already on Cloudflare's network) round out the commonly-evaluated set.
The buy-vs-build trade-off: an existing gateway product gets you section 4's circuit breaking, cost visibility, and provider fallback in days rather than the weeks-to-months of engineering effort building and hardening this chapter's LLMGateway/CircuitBreaker classes into genuinely production-grade shared infrastructure would take (section 8's "design the gateway itself for high availability" is nontrivial work to get right) — at the cost of adopting the product's specific API shape, its release cadence for new provider support, and (for a managed/hosted option specifically) a new vendor dependency sitting on the critical path of every LLM call in the organization (section 10's "the gateway becomes a high-value target" applies to a third-party-hosted gateway with extra force, since it now also means a third party sees every request). A common, pragmatic pattern: adopt an existing gateway product for the standard cross-cutting mechanics (routing, rate limiting, cost dashboards) this chapter describes, and reserve custom engineering effort for genuinely organization-specific policy logic (a bespoke routing rule tied to internal business logic, a non-standard access-control integration) that an off-the-shelf product doesn't already cover well.
Unified cost and usage visibility
Because the gateway sees every LLM call across the entire organization, it's the natural place to aggregate Part 4.2's response_metadata usage data into organization-wide cost dashboards, broken down by service, team, or feature — directly enabling Part 7.10's cost-optimization work at an organizational scale, rather than each service having only its own siloed view of its own usage.
5. Simple mental model
An LLM gateway is like a corporate travel-booking desk that every employee books through, instead of each employee independently negotiating with airlines — the desk gets a unified view of total company travel spend (cost tracking), can negotiate better rates by aggregating volume (potentially better provider terms at scale), enforces consistent policies (rate limits, allowed providers) across every department, and if one airline has a major disruption, can rebook affected travelers through a different airline automatically (provider fallback/circuit breaking) — all without every individual employee needing to understand airline-specific booking systems or independently discover and react to a disruption themselves.
6. Real-world example
A large enterprise with a dozen internal teams building various AI features initially had each team calling LLM providers directly, with no visibility into total organizational spend until a monthly bill arrived, and no coordination when one team's traffic spike exhausted a shared provider rate-limit tier, causing unrelated teams' unrelated features to start failing with rate-limit errors they had no way to diagnose (since they didn't know another team's traffic was the actual cause). Introducing a centralized LLM gateway gave immediate, real-time cost visibility per team/feature, centralized rate-limit coordination (preventing one team's burst from silently starving others), and one place to implement provider fallback and circuit breaking that every team automatically benefited from — turning a previously invisible, uncoordinated organizational problem into a directly managed, visible one.
7. Architecture diagram
See section 4's unified-proxy diagram — this chapter's architecture is that centralized gateway pattern, sitting between every internal consumer and every external LLM provider.
8. Production considerations
- Design the gateway itself for high availability — since every LLM-dependent feature across the organization now depends on it, the gateway becoming a new single point of failure would be a serious regression from the (admittedly less coordinated) status quo of direct provider calls; apply Part 7.4/7.5's own resilience patterns to the gateway service itself.
- Implement circuit breaking and fallback routing centrally (section 4) — this is one of the gateway's highest-value capabilities, since it benefits every consuming service automatically once implemented once, correctly, in one place.
- Expose cost/usage data broken down by consuming service/team, not just an aggregate organization-wide number — the actionable cost-optimization work from Part 7.10 requires this granularity to know where to actually focus effort.
- Keep the gateway's own latency overhead minimal — since it's now on the critical path of every single LLM call across the organization, even a small added latency per call compounds significantly at organizational scale.
9. Common mistakes
- Introducing a gateway without designing it for its own high availability, creating a new organization-wide single point of failure.
- Not exposing per-service cost breakdown, leaving the gateway's cost-visibility benefit largely theoretical (an aggregate number without actionable granularity).
- Implementing model routing (Part 3.11) inconsistently across services because each service still makes its own routing decisions, missing the opportunity to centralize this logic in the gateway where it can be tuned and improved once for everyone.
- Underestimating the gateway's own operational complexity and treating it as a lightweight afterthought rather than a genuinely critical piece of shared infrastructure deserving proper engineering investment.
10. Security considerations
- Centralizing provider credentials in the gateway (rather than distributing them across every service) is actually a security improvement in most cases — fewer places holding sensitive API keys (Part 9.5), with the gateway enforcing consistent access control (Part 9.4) for which internal services/teams can make which kinds of requests.
- The gateway itself becomes a high-value target precisely because it holds every provider credential and sees every request across the organization — secure it with correspondingly serious rigor (network isolation, Part 7.3's VPC design, strict access control on the gateway's own admin/configuration surface).
11. Performance considerations
- Every added network hop (service → gateway → provider, instead of service → provider directly) adds some latency — this should be measured and kept minimal (section 8), since it's now a permanent, universal addition to every LLM call's critical path.
- A well-implemented gateway's circuit breaking can actually improve aggregate perceived performance during a provider incident, by failing fast and routing to a healthy fallback rather than every service independently timing out slowly against a struggling provider.
12. Cost considerations
- Centralized cost visibility (section 4/8) is itself the primary cost-optimization enabler this chapter provides — you cannot systematically optimize organization-wide AI spend (Part 7.10, Part 15) without first being able to see where it's actually going, broken down meaningfully.
- A gateway can also enable organization-wide volume-based provider pricing negotiations that individual, siloed service-level usage couldn't access — a real, concrete financial benefit of aggregating usage centrally.
13. When to use it
Organizations with more than a small handful of services/teams independently calling LLM providers — the coordination, visibility, and resilience benefits scale with the number of independent consumers; a single small team with one application has much less need for this additional infrastructure layer.
14. When NOT to use it
A single, small application or a small team with one clear LLM integration point doesn't need a separate gateway service — Part 3.11's in-application routing and Part 4.2's provider-abstraction patterns provide most of the relevant benefit at a much smaller scale without the added infrastructure and operational overhead of a standalone gateway service.
15. Alternatives and trade-offs
| Approach | Good for | Weak point |
|---|---|---|
| Centralized LLM gateway (custom-built) | Full control over routing/policy logic; no vendor dependency | Real engineering effort to build and hardening effort to make highly available |
| Existing gateway product (LiteLLM, Portkey, Kong AI Gateway, Cloudflare AI Gateway) | Section 4's mechanics available in days, not months; actively maintained provider support | Adopts the product's API shape/release cadence; a hosted option adds a new vendor on the critical path |
| Per-service direct provider integration (Part 3.11/4.2) | Simpler for a single application/small team | No cross-service coordination, visibility, or shared resilience benefit |
16. Practical Python/code example
A minimal circuit-breaker implementation illustrating section 4's core mechanism (in production, this logic typically lives in the gateway service itself, and mature libraries exist for this — shown here for conceptual clarity):
python
import time
import logging
from enum import Enum
logger = logging.getLogger("circuit_breaker")
class CircuitState(str, Enum):
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
class CircuitBreaker:
"""A simple circuit breaker protecting calls to a specific LLM provider."""
def __init__(self, failure_threshold: int = 5, cooldown_seconds: float = 30.0):
self._failure_threshold = failure_threshold
self._cooldown_seconds = cooldown_seconds
self._failure_count = 0
self._state = CircuitState.CLOSED
self._opened_at: float | None = None
def _maybe_transition_to_half_open(self) -> None:
if self._state == CircuitState.OPEN and self._opened_at is not None:
if time.monotonic() - self._opened_at >= self._cooldown_seconds:
self._state = CircuitState.HALF_OPEN
logger.info("circuit transitioning to half-open, testing provider")
def can_attempt(self) -> bool:
"""Returns whether a call should be attempted, given the current circuit state."""
self._maybe_transition_to_half_open()
return self._state != CircuitState.OPEN
def record_success(self) -> None:
"""Resets the circuit to closed on a successful call."""
self._failure_count = 0
self._state = CircuitState.CLOSED
def record_failure(self) -> None:
"""Tracks a failure, opening the circuit if the threshold is exceeded."""
self._failure_count += 1
if self._failure_count >= self._failure_threshold:
self._state = CircuitState.OPEN
self._opened_at = time.monotonic()
logger.warning("circuit opened after %d consecutive failures", self._failure_count)17. Production-quality example
A gateway request handler using the circuit breaker plus fallback routing, combining sections 4 and Part 3.11's fallback pattern:
python
import logging
logger = logging.getLogger("llm_gateway")
class LLMGateway:
"""Central gateway routing requests to primary/fallback providers with
circuit breaking and unified usage logging."""
def __init__(self, primary_client, fallback_client):
self._primary_client = primary_client
self._fallback_client = fallback_client
self._primary_breaker = CircuitBreaker()
async def generate(self, service_name: str, **kwargs) -> dict:
"""
Routes a generation request, using the fallback provider if the primary
provider's circuit is open, and logs usage attributed to the calling service.
Args:
service_name (str): Identifies which internal service made this request,
for cost/usage attribution (Part 7.10).
**kwargs: Model call parameters.
Returns:
dict: The response plus metadata on which provider actually served it.
"""
if self._primary_breaker.can_attempt():
try:
response = await self._primary_client.messages.create(**kwargs)
self._primary_breaker.record_success()
self._log_usage(service_name, "primary", response)
return {"response": response, "provider": "primary"}
except Exception as exc:
self._primary_breaker.record_failure()
logger.warning("primary provider failed for service=%s: %s", service_name, exc)
response = await self._fallback_client.messages.create(**kwargs)
self._log_usage(service_name, "fallback", response)
return {"response": response, "provider": "fallback"}
def _log_usage(self, service_name: str, provider: str, response) -> None:
"""Logs usage attributed to the calling service, for cost dashboards (Part 7.10)."""
logger.info(
"service=%s provider=%s input_tokens=%s output_tokens=%s",
service_name, provider,
response.usage.input_tokens, response.usage.output_tokens,
)18. Short exercise
An organization's five internal AI-powered features all call the same LLM provider directly and, during a company-wide product launch event, one feature's traffic spike exhausts the shared provider rate limit, causing the other four unrelated features to fail with confusing rate-limit errors their own teams can't explain. Using this chapter's gateway pattern, describe specifically how a centralized gateway would have prevented this cross-team incident.
19. Interview questions
- What problem does a centralized LLM gateway solve that Part 3.11's per-application model routing doesn't address on its own?
- Explain circuit breaking and why implementing it centrally in a gateway benefits every consuming service simultaneously.
- Why does centralizing provider credentials in a gateway represent, in most cases, a security improvement rather than a new risk, despite concentrating access in one place?
20. FDE/customer scenario
Customer's platform team: "We now have six different teams building AI features, and we have no idea what our total LLM spend actually is until the bill arrives — is there a better way?"
This is close to a direct match for section 6's real-world example: recommending a centralized LLM gateway gives the customer immediate, per-team cost visibility (closing the "no idea until the bill arrives" gap directly), consistent rate-limit coordination across teams, and shared resilience benefits — a concrete, high-value piece of infrastructure investment that pays for itself quickly once an organization has passed the single-application scale where Part 3.11's simpler in-application patterns sufficed.
Key takeaways
- An LLM gateway centralizes rate limiting, circuit breaking, cost tracking, and model routing across an entire organization's LLM usage, rather than each service implementing these independently and inconsistently.
- Centralizing provider credentials in a gateway is generally a security improvement, not a new risk, given consistent access control benefits.
- The gateway itself must be engineered for high availability, since it becomes a new critical dependency for every LLM-dependent feature across the organization.
Things you should be able to explain
- What a gateway provides beyond per-application model routing (Part 3.11).
- How circuit breaking protects against a struggling provider, and why implementing it centrally benefits every consumer at once.
Things you should be able to build
- A gateway request handler combining circuit breaking, provider fallback, and per-service usage logging.
Common mistakes
- Introducing a gateway without ensuring its own high availability.
- No per-service cost/usage breakdown, losing the actionable value of centralized visibility.
Recommended next chapter
10-cost-and-token-optimization.md