7.5 — Scaling and Load Balancing for AI Systems

1. What is it?

Scaling is increasing a system's capacity to handle more load; load balancing is distributing incoming requests across multiple instances so no single instance is overwhelmed while others sit idle. This chapter covers both concepts specifically as they apply to AI backends, which have load characteristics (long-held connections for streaming, Part 4.6/5.5; expensive, slow per-request work, Part 2.6) that differ meaningfully from a typical stateless CRUD API.

2. Why does it exist?

A single server instance has finite capacity — finite CPU, memory, and (for an async service, Part 1.1) a finite number of connections it can meaningfully juggle before latency degrades. Scaling and load balancing exist to let a system handle more total traffic than any single instance could, by adding more instances (horizontal scaling) or more powerful instances (vertical scaling), and distributing requests intelligently among them.

3. What problem does it solve?

For an AI backend, it solves "how do I serve a growing number of concurrent users making slow, expensive LLM-backed requests, without response times degrading as load increases, and without paying for far more capacity than actual demand requires" — a genuinely distinct problem profile from scaling a typical fast, cheap CRUD endpoint, because the bottleneck resource (LLM provider rate limits, not just your own compute) is partially outside your control.

4. How does it work internally?

Horizontal vs. vertical scaling

• Vertical scaling: making a single instance bigger (more CPU/memory) — simple, but has a hard ceiling (the largest available instance size) and doesn't provide redundancy (one instance is still a single point of failure).
• Horizontal scaling: adding more instances — Part 7.4's Kubernetes HPA is the automated mechanism for this; provides both more total capacity and redundancy (one instance failing doesn't take down the whole service), and is the standard approach for genuinely scalable production systems.

For an AI backend specifically, since most of the actual work happens in an external LLM provider's infrastructure (not your own compute, Part 1.1's discussion of I/O-bound workloads), horizontal scaling of your own service is often less about raw compute capacity and more about handling more concurrent, in-flight requests — meaning your scaling bottleneck may be connection/memory limits per instance (Part 1.1's semaphore-bounded concurrency) well before it's CPU-bound.

Load balancing algorithms and why the choice matters for AI workloads

• Round-robin: distributes requests evenly, in order, across instances — simple, works well when requests have roughly uniform cost/duration.
• Least-connections: routes to whichever instance currently has the fewest active connections — better suited to AI workloads specifically, because request duration varies enormously (a simple classification call vs. a long multi-step agent run, Part 3.7/3.11) — round-robin can send a new request to an instance already juggling several long-running agent executions, while an instance mostly finished with shorter requests sits comparatively idle; least-connections accounts for this actual load imbalance rather than assuming uniform request cost.
• Latency-based/weighted: routes based on observed response times or instance capacity weighting — most sophisticated, most appropriate when instances have genuinely different capacity or when request latency varies significantly and predictably enough to route around it.

The distinctive challenge of streaming connections

Part 5.5/4.6 established that AI backends frequently stream responses (token-by-token) over long-held connections. Load balancers need to be configured correctly for this: some load-balancer configurations (particularly ones designed around short-lived request/response cycles) can time out or misbehave with long-held streaming connections if not explicitly configured to support them — verify your specific load balancer/ingress configuration supports the connection duration and streaming behavior your AI backend actually produces, rather than assuming default settings designed for typical short HTTP requests will work correctly.

Concretely, on AWS an Application Load Balancer has a default idle timeout (historically 60 seconds — verify the current default against AWS documentation) after which it closes a connection with no data flowing. A slow-generating streaming LLM response that goes quiet longer than that timeout (a long tool call mid-agent-run, an unusually slow first token) gets its connection killed mid-stream unless the timeout is explicitly raised above your expected maximum streaming duration — Part 18.5 covers this specific ALB setting, and the failure mode it causes, in full.

Bottlenecks beyond your own infrastructure

A distinctive AI-system scaling consideration: your own service scaling out doesn't help if the actual bottleneck is the LLM provider's rate limit — Part 3.11's model-routing and Part 7.9's LLM-gateway patterns become directly relevant here, since adding more of your own server instances just means more instances competing for the same shared, provider-imposed rate limit, not more actual throughput. Recognizing this distinction (is the bottleneck my infrastructure, or the upstream provider) is a critical, AI-specific scaling diagnostic that doesn't have a direct parallel in typical web-service scaling.

5. Simple mental model

Horizontal scaling with load balancing is like adding more checkout lanes at a grocery store and having a smart "next available" system directing customers to whichever lane will actually get them through fastest — round-robin is like a naive system sending customers to lanes in strict rotation regardless of how many items are already in each lane's queue; least-connections is the smarter system that actually looks at each lane's current queue before directing the next customer. But no number of additional checkout lanes helps if the actual bottleneck is a single, shared resource behind all the lanes (like one overworked price-checker everyone needs) — exactly the LLM-provider-rate-limit scenario from section 4.

6. Real-world example

A customer support AI platform scales its backend from 3 to 10 instances during a load test, expecting proportional throughput improvement, but observes only marginal gains — investigation reveals the actual bottleneck is the LLM provider's requests-per-minute rate limit for their API tier, which all 10 instances share and collectively saturate identically to how 3 instances did, just with more instances competing for the same fixed ceiling. The real fix wasn't more instances — it was negotiating a higher rate-limit tier with the provider and implementing Part 3.11's model-routing to shift a portion of traffic to a less rate-limited, smaller model for simpler requests, distributing load across a resource that wasn't already saturated rather than adding more consumers of the one that was.

7. Architecture diagram

Client requests

Load Balancerleast-connections, streaming-aware

Horizontally scaled · Part 7.4 HPA

Instance 1

Instance 2

Instance 3

Shared LLM provider rate limitexternal, outside your control — the ACTUAL bottleneck may live here, not your instance count

8. Production considerations

• Diagnose whether your actual bottleneck is your own infrastructure or the upstream LLM provider before scaling out (section 4/6) — scaling your own instances is wasted effort and cost if the real ceiling is a shared, external rate limit.
• Choose a load-balancing algorithm suited to your request-duration variance (section 4) — least-connections or latency-based routing generally outperforms plain round-robin for AI workloads with highly variable per-request cost.
• Verify your load balancer/ingress configuration explicitly supports your streaming connection patterns (section 4) — test this directly rather than assuming default configuration handles it correctly.
• Combine horizontal scaling with the bounded-concurrency discipline from Part 1.1 — more instances each running unbounded concurrent requests doesn't actually solve a rate-limit or connection-pool-exhaustion problem; each instance still needs its own sensible concurrency bounds.

9. Common mistakes

• Scaling out (adding instances) as the default response to any performance problem, without first diagnosing whether the actual bottleneck is your infrastructure or a shared external resource (the LLM provider's rate limit) that more instances don't help with at all.
• Using round-robin load balancing for a workload with highly variable request duration (simple queries mixed with long multi-step agent runs, Part 3.7), causing real, avoidable load imbalance across instances.
• Not verifying load balancer support for streaming/long-held connections, discovering timeout or connection-drop issues only under real production streaming traffic.

10. Security considerations

• A load balancer is often also the natural point for TLS termination, and sometimes for rate-limiting and basic request filtering (Part 9.5's API security discussion) — verify these are configured correctly, since a misconfigured load balancer can either weaken security (TLS misconfiguration) or become a bottleneck/single point of failure itself if it's the only layer providing certain protections.

11. Performance considerations

This entire chapter is fundamentally about performance at scale — the section 4 discussion of load-balancing algorithm choice and the section 8 bottleneck-diagnosis discipline are the core, actionable performance considerations.

12. Cost considerations

• Scaling out instances that don't actually address the real bottleneck (an external rate limit) is pure wasted infrastructure cost — the bottleneck-diagnosis discipline from section 8 is as much a cost-control practice as a performance one.
• Autoscaling configuration (Part 7.4) directly trades cost against responsiveness — scale-down aggressiveness during low-traffic periods should be tuned deliberately against your actual cost sensitivity and traffic predictability.

13. When to use it

Any production AI system expecting meaningful, growing, or variable concurrent traffic — which describes essentially any real customer-facing enterprise AI deployment beyond an initial prototype.

14. When NOT to use it

A genuinely low-traffic, single-tenant internal tool may not need sophisticated load balancing across multiple instances — a single, adequately-provisioned instance (with the resilience patterns from Part 5.7/7 still applied) may be entirely sufficient, and adding scaling infrastructure prematurely is unnecessary complexity per Part 3.12's core discipline.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Horizontal scaling + load balancing	Genuine capacity growth, redundancy	Doesn't help if bottleneck is external (LLM provider rate limit)
Vertical scaling only	Simplicity	Hard ceiling, no redundancy, single point of failure
Model routing to relieve provider rate limits (Part 3.11)	Addresses external bottlenecks scaling instances can't fix	Requires evaluation to ensure quality isn't compromised by routing to smaller models

16. Practical Python/code example

A bottleneck-diagnostic helper — logging which resource (local concurrency bound vs. provider rate limit) is actually being hit, directly implementing section 8's diagnostic discipline:

import logging
from anthropic import RateLimitError

logger = logging.getLogger("bottleneck_diagnostics")


async def call_llm_with_bottleneck_tracking(client, semaphore, **kwargs):
    """
    Calls the LLM, distinguishing local concurrency-bound waits from provider
    rate-limit errors, to correctly diagnose where the actual scaling bottleneck is.
    """
    if semaphore.locked():
        logger.info("local concurrency bound reached — consider scaling instances")

    async with semaphore:
        try:
            return await client.messages.create(**kwargs)
        except RateLimitError:
            logger.warning(
                "PROVIDER rate limit hit — scaling instances will NOT help; "
                "consider model routing (Part 3.11) or a higher provider tier"
            )
            raise

17. Production-quality example

A least-connections-aware routing simulation for illustrating the algorithm choice from section 4 (in a real deployment, this logic typically lives in your load balancer/ingress configuration, not application code — shown here for conceptual clarity):

class LeastConnectionsRouter:
    """Illustrative least-connections routing logic, showing why it outperforms
    round-robin for AI workloads with variable request duration."""

    def __init__(self, instance_ids: list[str]):
        self._active_connections = {instance_id: 0 for instance_id in instance_ids}

    def route(self) -> str:
        """Routes to whichever instance currently has the fewest active requests."""
        return min(self._active_connections, key=self._active_connections.get)

    def on_request_start(self, instance_id: str) -> None:
        self._active_connections[instance_id] += 1

    def on_request_end(self, instance_id: str) -> None:
        self._active_connections[instance_id] -= 1

18. Short exercise

A team scales their AI backend from 5 to 20 instances in response to slow response times, but latency doesn't improve. List the specific diagnostic steps (referencing section 8/16) you'd take to determine whether the bottleneck is their own infrastructure or an external LLM provider limit, before recommending further scaling.

19. Interview questions

• Why might scaling out your own service's instances fail to improve throughput for an AI backend, in a way that wouldn't happen for a typical CPU-bound web service?
• Explain why least-connections load balancing generally outperforms round-robin for AI workloads specifically.
• What configuration risk does streaming introduce for a load balancer designed around typical short HTTP request/response cycles?

20. FDE/customer scenario

Customer: "We doubled our server capacity but our AI assistant is still slow under load — what are we missing?"

This is almost exactly the section 6 real-world example: the first diagnostic question should be whether the customer has verified their actual bottleneck is local infrastructure versus their LLM provider's rate limit — a very common, non-obvious root cause that "add more servers" (the intuitive first response) doesn't address at all, and correctly diagnosing this (rather than recommending yet more scaling) is a clear demonstration of the kind of AI-specific systems thinking that distinguishes an AI FDE from a generic infrastructure engineer.

---

Key takeaways

• AI backends have a distinctive scaling profile: the actual bottleneck is frequently an external LLM provider rate limit, not your own infrastructure capacity — diagnose this before scaling out.
• Least-connections or latency-based load balancing generally outperforms round-robin for AI workloads with highly variable per-request duration.
• Load balancer configuration must be explicitly verified to support streaming, long-held connections, not assumed from defaults designed for short request/response cycles.

Things you should be able to explain

• Why more server instances don't always improve throughput for an AI backend.
• Why request-duration variance makes load-balancing algorithm choice matter more for AI workloads than typical web services.

Things you should be able to build

• A bottleneck-diagnostic wrapper distinguishing local concurrency limits from provider rate limits.

Common mistakes

• Scaling out as a default response without diagnosing the actual bottleneck.
• Round-robin load balancing for highly variable-duration AI requests.
• Unverified load balancer streaming/connection-duration support.

Recommended next chapter

06-async-processing.md