3.11 — Model Routing, Selection, Latency, and Token Economics

1. What is it?

Model selection is the discipline of choosing which model (and which provider) to use for a given task, out of a landscape of options that differ in capability, latency, and cost. Model routing is the closely related production pattern of dynamically sending different requests to different models based on the request's characteristics — not committing to one model for an entire application. Token economics is the underlying cost math (Part 2.6 introduced the foundation; this chapter builds the full framework) that makes these decisions concrete and quantifiable rather than a matter of taste.

2. Why does it exist?

Part 2.6 established that bigger, more capable models cost more per token and are generally slower — this isn't a minor implementation detail, it's a structural trade-off (Part 2.2's discussion of scaling laws and inference cost). No single model is simultaneously the cheapest, fastest, and most capable option for every task a real application needs to perform. Model routing and deliberate model selection exist because a production AI system typically has a mix of tasks with very different requirements — a simple classification step and a complex multi-step reasoning task within the same application have genuinely different optimal model choices, and treating them identically (always using the most capable, most expensive model for everything) leaves substantial cost and latency improvement on the table for no corresponding quality benefit on the simpler tasks.

3. What problem does it solve?

It solves "how do I get the best overall cost/latency/quality trade-off across an entire application's worth of different LLM calls," rather than the naive approach of picking one model for the whole system. For an AI FDE, this is a direct, concrete, and often surprisingly large lever for controlling a customer's ongoing AI operating cost (Part 15's ROI framing depends heavily on getting this right) — and it's one of the clearest, most quantifiable ways an FDE demonstrates engineering value beyond "the AI feature works."

4. How does it work internally?

The three-way trade-off, made concrete

Every model choice sits somewhere on a genuine three-way trade-off: capability (how well it performs on hard, nuanced, or multi-step tasks), latency (time to first token and tokens per second, Part 2.6), and cost (dollars per million input/output tokens). Historically and currently, more capable models tend to cost more per token and can be slower — but this relationship is not perfectly linear or fixed across time, since model efficiency genuinely improves across generations (a newer smaller model can sometimes match an older larger model's capability at a fraction of the cost/latency) — meaning model selection is not a "set it once" decision but requires periodic reassessment as the available model landscape evolves. Always verify current model capability, pricing, and latency characteristics against the provider's current documentation and your own benchmarks (Part 8) rather than relying on stale assumptions about which model is "the good one" or "the cheap one."

Task-complexity-based routing

The core routing insight: not every request in your application needs your most capable (and most expensive) model. A practical routing architecture classifies incoming requests by their actual complexity/stakes and sends each to an appropriately-sized model:

Incoming request
       │
       ▼
┌─────────────────┐
│  Complexity/stakes  │   (a fast, cheap classification step — Part 3.7's
│  classifier          │    "routing" workflow pattern applied specifically
└────────┬─────────┘    to model selection)
         │
    ┌────┴────┬─────────────┐
    ▼         ▼              ▼
 Simple    Moderate      Complex/high-stakes
 (small,   (mid-tier      (largest, most
  cheap     model)         capable model)
  model)

A concrete example: classifying a support ticket's category (Part 3.7's routing workflow) is a simple, well-defined, low-stakes task that a smaller, cheaper, faster model handles perfectly well; synthesizing a complex multi-document legal analysis is a task where the largest, most capable model's quality advantage is worth its higher cost and latency. Routing every request — including the simple classification step — to the largest model wastes money and adds latency for zero measurable quality benefit on that specific step.

Fallback and redundancy routing

A distinct but related routing pattern: routing to a fallback model (potentially from a different provider entirely) when the primary model/provider is unavailable, rate-limited, or returns an error — a direct application of the resilience patterns from Part 7 (retries, circuit breakers) applied specifically to model availability. This requires designing your application against a reasonably provider-agnostic interface where feasible, so a fallback doesn't require completely different integration code (Part 7.9's LLM gateway pattern is the production-grade version of this).

Provider abstraction and its real limits

It's tempting to build a fully abstracted "any model, any provider" interface for maximum flexibility — but be aware this has real limits: different providers' models have genuinely different strengths, different structured-output mechanisms (Part 3.2), different tool-calling conventions (Part 3.3), and different failure modes, and a prompt carefully tuned for one model's specific behavior often needs re-tuning (not just re-pointing) when switched to a different model or provider (Part 3.1's warning that prompting techniques don't transfer identically across models applies directly here). Full provider-agnosticism is a spectrum, not a binary — decide deliberately how much abstraction genuinely serves your application versus adding complexity for a flexibility you won't actually exercise often.

Prompt caching — a cost lever independent of which model you route to

Routing (sending a simpler task to a cheaper model) and caching (Part 3.3, section 12) are two entirely separate levers on the same cost equation from section 12 below — routing changes the per-token price you pay; caching changes how many tokens are billed at full price at all. A large, stable prompt prefix that's resent on every call (a long system prompt, a fixed set of tool definitions, a big retrieved-context block reused across a session, Part 3.5) can be cached: the provider marks a breakpoint on that stable prefix, and a subsequent request sharing the same bytes is billed at a fraction of the normal input-token price for the cached portion. Critically, the two levers compose — you get the full benefit of both by caching whatever prefix is genuinely stable and routing the actual generation to the cheapest model that meets your quality bar for that specific task. They can also work against each other if applied carelessly: caches are model-scoped, so a routing scheme that varies which model handles logically-similar requests (rather than pinning one model per route) forfeits the caching benefit on that route entirely — a real, easy-to-miss interaction worth checking explicitly, not an assumption that the two savings simply add up.

5. Simple mental model

Model routing is like a hospital's triage system, not a single doctor seeing every patient regardless of severity. A minor issue gets seen quickly by a nurse or general practitioner (a small, fast, cheap model) — appropriately, since a specialist's deeper expertise wouldn't change the outcome for a simple case, and using one anyway would waste the specialist's limited time (or, in the model case, waste money and add latency) with no benefit. A genuinely complex, high-stakes case gets routed to the specialist (the largest, most capable model) — appropriately, since that's exactly where the specialist's greater capability changes the actual outcome.

6. Real-world example

A customer support platform processes three genuinely different task types within one application: (1) classifying incoming tickets into categories, (2) drafting a first-pass response to routine, well-understood ticket types, and (3) handling escalated, ambiguous, or emotionally sensitive tickets requiring nuanced judgment. Routing all three to the same large, expensive model was the initial (simple, but costly) implementation. Measuring actual task-specific quality (Part 8) revealed that a much smaller, cheaper, faster model performed statistically indistinguishably from the larger model on tasks (1) and (2) — meaning the extra cost and latency for those two task types was providing zero measured benefit — while task (3) showed a real, measurable quality gap favoring the larger model, justifying its cost specifically there. Routing accordingly cut the platform's overall LLM cost substantially while maintaining (in fact, slightly improving, due to lower latency on the majority-volume simple tasks) the customer-facing experience.

7. Architecture diagram

Incoming request

Complexity classifierfast, cheap · Part 3.7 routing

Small modelsimple

Mid-tier modelmoderate

Largest modelcomplex, high-stakes

Fallback modelPart 7 resilience patterns

Any tier can fail over to the fallback model — not just the tier shown.

8. Production considerations

• Build the complexity/routing classifier itself as a small, cheap, fast model or even a non-LLM heuristic (Part 2.3's classical NLP discussion applies directly) — the routing decision itself shouldn't become a significant cost/latency line item.
• Pin exact model versions explicitly (Part 2.6, section 8) rather than routing to a generic "latest" alias where determinism/reproducibility matters, since provider-side model updates can shift behavior even at a stable-seeming name.
• Design for provider fallback deliberately, even if you never expect to need it — provider outages happen, and a production system with zero fallback path has a hard, unmitigated single point of failure at the LLM layer (Part 7 covers the broader resilience architecture this fits into).
• Re-benchmark routing decisions periodically, not just once at launch — the cost/capability landscape across models shifts over time (new model releases, pricing changes), and a routing decision that was optimal a year ago may no longer be.

9. Common mistakes

• Defaulting to "always use the most capable model available" for an entire application, leaving substantial, measurable cost/latency improvement on the table for tasks where it provides no quality benefit.
• The opposite mistake: routing aggressively to the cheapest model everywhere without measuring quality impact, degrading user experience on the subset of tasks that genuinely needed more capability.
• Building an elaborate, fully abstracted multi-provider architecture before validating that the flexibility will actually be exercised, adding complexity for a hypothetical future need.
• Not re-evaluating model choice over time, missing meaningful cost/capability improvements as the model landscape evolves.

10. Security considerations

• Different providers have different data-handling, retention, and compliance postures (Part 9.6, Part 10.5) — routing customer data to a fallback provider during an outage must respect the same data-residency/compliance constraints as the primary provider, not be treated as a purely technical failover decision.
• A routing/classification step that itself processes potentially sensitive request content is subject to the same data-handling scrutiny as any other model call touching that data — "it's just a cheap classifier" doesn't exempt it from compliance requirements if it sees sensitive input.

11. Performance considerations

• Smaller models are generally faster (both time-to-first-token and tokens-per-second, Part 2.6) — routing simple, high-volume tasks to smaller models improves aggregate application latency, not just cost, which is a real, often underemphasized benefit of routing beyond pure cost savings.
• A routing/classification step adds a small amount of latency itself — ensure this overhead is genuinely smaller than the latency/cost savings it enables, especially for very simple, low-stakes tasks where the classification overhead could become a larger fraction of total latency than the savings justify.

12. Cost considerations

• This entire chapter is fundamentally a cost-optimization discipline — the core, quantifiable insight is: cost scales with (number of calls) × (input + output tokens per call) × (per-token price of the model used) × (frequency), and routing optimizes the last factor specifically, without needing to reduce call volume or token count (Part 7.10 covers the other factors in this equation).
• Track cost per task-type in production (not just aggregate cost) to know where routing improvements would actually matter — you can't optimize what you haven't measured at the right granularity.
• Caching and routing are complementary, not substitutes (section 4) — routing reduces cost per token by picking a cheaper model for a given task; caching reduces the number of tokens billed at full price at all for a stable, repeated prefix. A cost review that only asks "are we using the right model for each task" and never asks "is our stable prompt content actually being cached" is leaving one of the two biggest levers unexamined.

13. When to use it

Any production application with more than one distinct task type or complexity tier — which describes essentially every non-trivial AI application. Even a modest routing scheme (two tiers: simple/complex) typically captures most of the available benefit without the complexity of many finely-graded tiers.

14. When NOT to use it

• A genuinely single-task, single-complexity-tier application (a narrow, uniform use case) gains little from routing complexity — one well-chosen model may be entirely sufficient, and adding a routing layer would be complexity without corresponding benefit.
• Very early-stage prototypes, where establishing whether the product works at all matters more than cost/latency optimization — premature optimization here can slow down validating the actual product hypothesis.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Single model for everything	Simplicity, fastest to build initially	Leaves cost/latency optimization on the table; single point of failure
Complexity-based routing	Captures most of the cost/latency benefit with modest added complexity	Requires building and maintaining a classification step
Full multi-provider abstraction	Maximum flexibility, resilience	Real complexity cost; prompts often need per-model tuning anyway

16. Practical Python/code example

from enum import Enum

class TaskComplexity(str, Enum):
    """Complexity tiers used to route requests to an appropriately capable model."""
    SIMPLE = "simple"
    MODERATE = "moderate"
    COMPLEX = "complex"


MODEL_FOR_COMPLEXITY = {
    TaskComplexity.SIMPLE: "claude-haiku-4-5",
    TaskComplexity.MODERATE: "claude-sonnet-4-5",
    TaskComplexity.COMPLEX: "claude-opus-4-5",
}
# Verify current model identifiers and their relative capability/cost/latency
# positioning against current provider documentation before using in production —
# this mapping should be revisited periodically, not treated as permanent.


async def route_and_generate(client, task_complexity: TaskComplexity, prompt: str) -> str:
    """
    Routes a request to the model tier appropriate for its assessed complexity.

    Args:
        client: An async LLM client.
        task_complexity (TaskComplexity): The assessed complexity tier for this request.
        prompt (str): The prompt to send.

    Returns:
        str: The model's response.
    """
    model = MODEL_FOR_COMPLEXITY[task_complexity]
    response = await client.messages.create(model=model, max_tokens=500, messages=[{"role": "user", "content": prompt}])
    return response.content[0].text

17. Production-quality example

Adding a fallback path across models/providers and per-tier cost tracking, tying together sections 8, 9, and 12:

import logging
from dataclasses import dataclass

logger = logging.getLogger("model_router")


@dataclass
class RoutedCallResult:
    """The outcome of a routed model call, including which model actually served it
    (which may differ from the intended model if a fallback was used)."""
    text: str
    model_used: str
    input_tokens: int
    output_tokens: int
    used_fallback: bool


async def route_generate_with_fallback(
    primary_client, fallback_client, task_complexity: TaskComplexity, prompt: str
) -> RoutedCallResult:
    """
    Routes a request by complexity tier, falling back to a secondary provider/model
    if the primary call fails, and records which model actually served the request
    for cost and reliability tracking.

    Args:
        primary_client: The primary async LLM client.
        fallback_client: A secondary async LLM client, potentially a different provider.
        task_complexity (TaskComplexity): The assessed complexity tier.
        prompt (str): The prompt to send.

    Returns:
        RoutedCallResult: The response text plus metadata on which model served it.
    """
    model = MODEL_FOR_COMPLEXITY[task_complexity]
    try:
        response = await primary_client.messages.create(
            model=model, max_tokens=500, messages=[{"role": "user", "content": prompt}]
        )
        return RoutedCallResult(
            text=response.content[0].text,
            model_used=model,
            input_tokens=response.usage.input_tokens,
            output_tokens=response.usage.output_tokens,
            used_fallback=False,
        )
    except Exception as exc:
        logger.warning("primary model %s failed (%s); falling back", model, exc)
        fallback_model = "fallback-provider-model-id"  # a real, pre-vetted fallback identifier
        response = await fallback_client.messages.create(
            model=fallback_model, max_tokens=500, messages=[{"role": "user", "content": prompt}]
        )
        return RoutedCallResult(
            text=response.content[0].text,
            model_used=fallback_model,
            input_tokens=response.usage.input_tokens,
            output_tokens=response.usage.output_tokens,
            used_fallback=True,
        )

Logging model_used and used_fallback on every call gives you exactly the per-tier, per-model cost and reliability visibility referenced in section 12 — without this, you can't actually verify a routing scheme is delivering the savings it was designed for.

18. Short exercise

A customer's application currently routes 100% of requests to the largest, most expensive model, at a volume of 1 million requests/month. You measure that 70% of those requests are simple classification/extraction tasks where a smaller model performs statistically indistinguishably in evaluation (Part 8). Using rough, illustrative per-token pricing you look up for real current models, estimate the monthly cost difference this routing change would produce, and state what evaluation evidence you'd want in hand before recommending the change to the customer.

19. Interview questions

• Explain the three-way trade-off (capability, latency, cost) in model selection, and why it isn't perfectly linear or permanent across model generations.
• Design a routing architecture for an application with three genuinely different task types, and justify your tiering.
• Why is full multi-provider abstraction not automatically the right architecture, even though it sounds more flexible?

20. FDE/customer scenario

Customer: "Our AI costs are much higher than we expected, and we're not sure why."

A first, high-leverage diagnostic question, directly informed by this chapter: is every request in the system — including simple, low-stakes ones — being routed to the same, most capable (and most expensive) model? This is one of the most common, most fixable root causes of unexpectedly high AI operating costs in real enterprise deployments, and proposing a measured, evaluation-backed routing scheme is often the single highest-leverage recommendation an FDE can make in a cost-focused engagement (Part 15 builds directly on this kind of concrete, quantified improvement for ROI reporting).

---

Key takeaways

• Model capability, latency, and cost form a genuine three-way trade-off that shifts over time as the model landscape evolves — reassess periodically, don't decide once and forget.
• Routing by task complexity is often the single highest-leverage, most underused cost/latency optimization in production AI systems.
• Full multi-provider abstraction has a real complexity cost — decide deliberately how much flexibility you actually need.

Things you should be able to explain

• The three-way capability/latency/cost trade-off and why it isn't fixed permanently.
• Why routing simple tasks to smaller models often loses zero measured quality while cutting cost and latency.

Things you should be able to build

• A complexity-based model router with logged per-call model attribution and a fallback path for provider failures.

Common mistakes

• Using the most capable model for every task regardless of complexity.
• Never re-evaluating model/routing choices as the model landscape evolves.
• Building full provider abstraction before validating the need for it.

Recommended next chapter

12-ai-application-architecture.md