7.10 — Cost and Token Optimization (Synthesis)

1. What is it?

This chapter synthesizes every cost lever this book has introduced so far — model routing (Part 3.11), caching (Part 1.6/7.8), prompt engineering efficiency (Part 3.1), context management (Part 3.9), architectural minimalism (Part 3.12), and gateway-level visibility (Part 7.9) — into one coherent cost model and optimization framework, so cost reduction becomes a systematic, measured practice rather than an ad hoc reaction to a surprising bill.

2. Why does it exist?

Nearly every chapter in Parts 2-7 has flagged a cost consideration specific to its own topic. This chapter exists because cost optimization is most effective when approached holistically, understanding how these individual levers interact and compound, rather than optimizing one in isolation while missing larger opportunities elsewhere — exactly the kind of end-to-end thinking Part 3.12's "business problem → business impact" teaching rule argues for, applied specifically to the cost dimension.

3. What problem does it solve?

It solves "how do I systematically understand, control, and reduce an AI system's real operating cost" — a genuinely core AI FDE skill, since cost is one of the most concrete, quantifiable ways to demonstrate engineering value to a customer (Part 15's ROI framing depends directly on being able to do this work credibly).

4. How does it work internally?

The cost equation

Every LLM-related cost, fundamentally, decomposes into:

Total Cost = Σ (calls) × (input_tokens + output_tokens per call) × (price per token for the model used)

Every optimization technique in this book works by reducing one of these three factors, and organizing them this way clarifies which lever addresses which part of the equation:

Reduce NUMBER OF CALLS:
  - Caching (exact-match, semantic, provider-level prefix caching — Part 7.8)
  - Avoiding unnecessary agent loop iterations (Part 3.7's bounded-loop discipline)
  - Batching independent calls where the task allows it (Part 4.1's .abatch())

Reduce TOKENS PER CALL:
  - Context management / summarization (Part 3.9) — bounding conversation history growth
  - Chunking and retrieval precision (Part 3.5/3.6) — retrieving fewer, more relevant
    chunks rather than over-retrieving
  - Concise prompt engineering (Part 3.1) — trimming verbose, rarely-useful instructions
  - Reranking (Part 3.6) — letting you retrieve a large candidate set cheaply via ANN
    search, but pass only the best few (not all) to the expensive generation step

Reduce PRICE PER TOKEN:
  - Model routing (Part 3.11) — using smaller, cheaper models for simpler tasks
  - Provider-level prompt caching (Part 7.8) — often priced at a reduced rate for
    cache-hit tokens specifically
  - Batch processing discount (Part 7.6) — providers price asynchronous batch
    inference at a meaningfully reduced rate versus real-time calls, since batch
    requests can be scheduled against provider capacity with slack instead of
    needing an immediate response
  - Architectural minimalism (Part 3.12) — not using a component (an unnecessary
    agent loop, unneeded reranking pass) at all when it isn't measurably needed

Batch processing discount — the natural cost lever for already-deferred work

Provider batch inference APIs accept a large set of requests submitted together for asynchronous processing (typically with a completion window measured in hours, not the immediate response a real-time call guarantees) and price them at a meaningfully reduced rate — a commonly cited figure is roughly 50% off standard per-token pricing, though this varies by provider and changes over time, so verify the exact current discount against your provider's live pricing page before quoting it to a customer, rather than treating any specific percentage as a fixed fact.

This lever is the direct, natural pairing for Part 7.6's deferred/background-processing pattern: any workload already built as a job-creation-plus-polling or webhook flow (Part 7.6 — a client isn't waiting synchronously on the response) has, by construction, already given up the one thing a batch API asks you to give up — an immediate, real-time response — so routing that already-async work through the batch API captures a real price-per-token reduction essentially for free, with no user-experience cost, since no user was waiting on a live connection for it in the first place. Concretely: a nightly document-summarization job, an overnight bulk re-scoring of a support-ticket backlog, or a periodic re-embedding of an updated document corpus (Part 3.5) are all naturally already deferred and are the first places to look for batch-API savings before considering any other lever in this section.

The one thing worth being deliberate about: routing a workload to the batch API turns a request that used to complete in seconds into one that completes within the batch window (often hours) — appropriate for genuinely non-real-time work, but not a substitute for the caching/model-routing/context-management levers below when a user is actually waiting on the response. Don't retrofit a batch API onto a workload with a real-time UX requirement just to chase the discount; that trades a cost win for a latency regression a user will actually notice.

Why holistic analysis beats optimizing one lever in isolation

A team that aggressively optimizes model routing (Part 3.11) but ignores unbounded conversation-history growth (Part 3.9) may see far less overall savings than expected, because the token-count factor is growing even as the price-per-token factor shrinks — the two effects can partially or fully offset each other. Conversely, a team that fixes context-window bloat but never questions whether an agent architecture (Part 3.7) was ever necessary for a task with genuinely fixed steps may still be paying for many more LLM calls than a simpler workflow would require. The cost equation's three factors are genuinely multiplicative, not additive — a 50% reduction in call count and a 30% reduction in tokens-per-call compound to roughly a 65% total reduction, not an 80% one, and understanding this multiplicative relationship helps set realistic expectations when stacking multiple optimizations.

Measuring before optimizing — avoiding the wrong-target trap

Part 7.9's gateway-level, per-service cost visibility is what makes this holistic analysis possible at all — without knowing where cost is actually concentrated (which service, which feature, which specific call pattern), optimization effort risks being spent on a lever that sounds appealing but addresses only a small fraction of actual spend, while the real largest cost driver goes unaddressed. This is directly analogous to Part 7.5's bottleneck-diagnosis discipline applied to cost rather than latency: measure first, then optimize the actual largest lever, not the most interesting-sounding one.

5. Simple mental model

Think of AI system cost like a household budget with three independently-adjustable dials: how often you go out to eat (call count), how much you order each time (tokens per call), and which restaurant's price tier you choose (price per model) — a serious budgeting effort looks at where the actual spending is concentrated first (which of the three dials, and which specific "restaurant visits," account for most of the total) rather than assuming any one dial is automatically the most important one to turn down, and recognizes that turning down two dials at once compounds rather than simply adding up.

6. Real-world example

A customer's support-agent system's monthly LLM bill was dominated (per gateway-level breakdown, Part 7.9) by a single feature: a document-summarization tool called far more frequently than expected, each call including the full, un-chunked source document (often tens of thousands of tokens) plus a lengthy, rarely-revisited system prompt, using the organization's largest, most expensive model regardless of document complexity. Applying this chapter's framework in order of measured impact: first, retrieving only the relevant sections (Part 3.5/3.6's chunking and reranking, addressing the tokens-per-call factor, since most documents didn't need their entire content summarized) produced the largest single reduction; second, routing simpler, shorter documents to a smaller model (Part 3.11, addressing price-per-token) produced a further reduction; third, caching repeated summarization requests for documents that hadn't changed (Part 7.8, addressing call count) captured the remainder. Attacking these in order of measured impact (largest lever first, per the gateway data) rather than in an arbitrary or "most interesting" order delivered the bulk of total savings from the very first change.

7. Architecture diagram

Gateway-level cost visibilityPart 7.9 · per service/feature breakdown reveals WHERE cost is concentrated

Decompose that feature's costinto the three-factor equation

Call count too high?caching (7.8), bounded loops (3.7)

Tokens/call too high?context mgmt (3.9), chunking/reranking (3.5/3.6)

Wrong price tier?model routing (3.11), provider prompt caching (7.8)

8. Production considerations

• Instrument cost tracking at the finest useful granularity (per feature, per service, ideally per request type, Part 7.9) — coarse, aggregate cost visibility can't drive the targeted optimization this chapter describes.
• Re-run this analysis periodically, not just once — usage patterns, model pricing, and available provider capabilities (Part 3.11's "the model landscape shifts over time" warning) all change, meaning a cost model that was optimal six months ago may no longer be.
• Set cost budgets/alerts per feature or service (building on Part 7.9's gateway visibility) — proactive alerting on cost anomalies catches problems (a runaway agent loop, Part 3.7; an accidentally-disabled cache, Part 7.8) faster than waiting for a monthly bill to reveal them.
• Weigh cost optimization against quality explicitly and with evidence (Part 6.2's evaluation gates) — every cost lever in section 4 has a potential quality trade-off (a smaller model may perform slightly worse on some tasks, aggressive context summarization may lose information, Part 3.9); never optimize cost without verifying quality holds via evaluation, not assumption.
• Audit which workloads are already deferred (Part 7.6) and route them through the provider's batch API (section 4) before assuming a more complex optimization is needed — this is frequently the single highest-leverage, lowest-effort change available, since the workload's UX already tolerates the batch API's non-real-time turnaround.

9. Common mistakes

• Optimizing the most interesting-sounding or most recently-learned-about cost lever rather than the one gateway data actually shows is the largest contributor to real spend.
• Treating cost optimization as a one-time project rather than an ongoing practice, missing drift as usage patterns and the model landscape evolve.
• Cutting cost via a lever (a cheaper model, aggressive caching, shorter context) without evaluating (Part 6.2) whether quality has actually held — trading a real, immediate cost win for an unmeasured, potentially serious quality regression.
• Not accounting for the multiplicative (not additive) nature of stacked optimizations, leading to unrealistic savings projections when presenting a cost-reduction plan to a customer.

10. Security considerations

Cost-optimization work itself is generally not a direct security concern, but note the interaction with Part 9.2: aggressively bounding agent loops and tool access for cost reasons (fewer iterations, narrower tool scope) often also reduces excessive-agency security risk — cost discipline and security discipline frequently point in the same direction for agent-based systems, worth mentioning explicitly when justifying a cost-driven architectural change to a security-conscious stakeholder.

11. Performance considerations

Most of this chapter's cost levers also improve latency (fewer calls, fewer tokens per call, and often faster smaller models) — cost and performance optimization are frequently aligned rather than in tension for AI systems specifically, a genuinely favorable property worth highlighting when proposing these changes (a rare case where you're not asking a customer to trade one desirable property for another).

12. Cost considerations

This entire chapter is the cost-considerations synthesis — the section 4 equation and section 8's measurement-first discipline are the core, actionable content.

13. When to use it

Any production AI system, on an ongoing basis — cost optimization should be a standing practice (informed by Part 7.9's continuous visibility), not a one-time exercise performed only when a bill surprises someone.

14. When NOT to use it

Very early-stage prototypes still validating basic product feasibility shouldn't over-invest in cost optimization before validating the product concept itself — premature cost optimization here can slow down the more urgent question of whether the product works at all (echoing Part 3.11, section 14's similar caution).

15. Alternatives and trade-offs

This chapter's core "alternative" framing is between measured, prioritized optimization (this chapter's approach: find the actual largest lever via gateway data, address it first) versus ad hoc, intuition-driven optimization (attacking whichever lever seems most interesting or most recently read about) — the former reliably produces better results for the same effort investment, precisely because it's grounded in where cost actually concentrates rather than assumption.

16. Practical Python/code example

A cost-breakdown report generator, operationalizing section 8's "measure before optimizing" discipline using gateway usage logs (Part 7.9):

from dataclasses import dataclass
from collections import defaultdict


@dataclass
class CostBreakdown:
    """A per-service/feature cost summary, sorted to reveal the largest levers first."""
    service_name: str
    total_calls: int
    total_input_tokens: int
    total_output_tokens: int
    estimated_cost_usd: float


def build_cost_breakdown(usage_logs: list[dict], pricing: dict[str, dict[str, float]]) -> list[CostBreakdown]:
    """
    Aggregates gateway usage logs into a per-service cost breakdown, sorted by
    total cost descending, to identify where optimization effort should focus first.

    Args:
        usage_logs (list[dict]): Records with service_name, model, input_tokens, output_tokens.
        pricing (dict[str, dict[str, float]]): Per-model pricing, e.g.
            {"claude-sonnet-4-5": {"input_per_million": 3.0, "output_per_million": 15.0}}.
            Verify current pricing against your provider's current documentation.

    Returns:
        list[CostBreakdown]: One entry per service, sorted by cost descending.
    """
    aggregated = defaultdict(lambda: {"calls": 0, "input": 0, "output": 0, "cost": 0.0})

    for log in usage_logs:
        service = log["service_name"]
        model_pricing = pricing[log["model"]]
        cost = (
            log["input_tokens"] / 1_000_000 * model_pricing["input_per_million"]
            + log["output_tokens"] / 1_000_000 * model_pricing["output_per_million"]
        )
        aggregated[service]["calls"] += 1
        aggregated[service]["input"] += log["input_tokens"]
        aggregated[service]["output"] += log["output_tokens"]
        aggregated[service]["cost"] += cost

    breakdowns = [
        CostBreakdown(
            service_name=service, total_calls=data["calls"],
            total_input_tokens=data["input"], total_output_tokens=data["output"],
            estimated_cost_usd=data["cost"],
        )
        for service, data in aggregated.items()
    ]
    return sorted(breakdowns, key=lambda b: b.estimated_cost_usd, reverse=True)

17. Production-quality example

A cost-optimization recommendation engine applying section 4's three-factor decomposition to flag which lever is likely most relevant for a given service, based on its usage pattern:

def recommend_optimization_focus(breakdown: CostBreakdown, avg_calls_per_user: float) -> str:
    """
    Suggests which cost-optimization category (call count, tokens/call, or model
    tier) is likely most impactful for a given service, based on its usage shape.

    Args:
        breakdown (CostBreakdown): This service's aggregated usage/cost data.
        avg_calls_per_user (float): Average calls per user — a high value suggests
            repeated/redundant requests worth investigating for caching (Part 7.8).

    Returns:
        str: A specific, actionable recommendation, not a generic suggestion.
    """
    avg_tokens_per_call = (
        (breakdown.total_input_tokens + breakdown.total_output_tokens) / breakdown.total_calls
    )

    if avg_calls_per_user > 20:
        return (
            f"{breakdown.service_name}: high calls-per-user ({avg_calls_per_user:.1f}) — "
            "investigate caching (Part 7.8) or unbounded agent loop iterations (Part 3.7) first."
        )
    elif avg_tokens_per_call > 8000:
        return (
            f"{breakdown.service_name}: high tokens-per-call ({avg_tokens_per_call:.0f}) — "
            "investigate context management (Part 3.9) or retrieval precision (Part 3.5/3.6) first."
        )
    else:
        return (
            f"{breakdown.service_name}: moderate call count and token size — "
            "investigate model routing (Part 3.11) for tasks not needing the largest model."
        )

This kind of triage logic directly implements section 6's "attack the actual largest, most relevant lever first" discipline as a repeatable, automatable practice rather than a one-off manual analysis.

18. Short exercise

Using the build_cost_breakdown function's output format, a service shows: 50,000 calls/month, average 12,000 tokens/call, using the organization's largest model for every call regardless of task complexity. Applying section 4's three-factor framework, identify which factor(s) you'd investigate first, and write a one-paragraph justification referencing the specific chapters (Part 3.9, 3.11, 3.5/3.6, or 7.8) whose techniques would address each factor you flagged.

19. Interview questions

• Explain the three-factor cost equation and why stacking multiple optimizations produces multiplicative, not additive, savings.
• Why is gateway-level, per-service cost visibility (Part 7.9) a prerequisite for effective cost optimization, rather than a nice-to-have?
• Give an example of a cost optimization that could also introduce a quality regression, and explain how you'd verify it hasn't, before shipping it.

20. FDE/customer scenario

Customer's CFO: "Our AI costs have tripled in six months and we need to understand why and fix it, without breaking what's working."

This is precisely the engagement this chapter's framework was built for: start with gateway-level, per-service cost data (Part 7.9) to find where the actual growth is concentrated, decompose that specific driver into the three-factor equation (section 4), propose the highest-leverage fix first with an evaluation plan (Part 6.2) to verify quality holds, and present the realistic, multiplicative (not naively additive) expected savings — a rigorous, evidence-based process that builds far more credibility with a CFO than a vague promise to "make the AI more efficient."

---

Key takeaways

• AI cost decomposes into three multiplicative factors — call count, tokens per call, price per token — and every optimization technique in this book addresses one of them.
• Optimizing without measuring first (Part 7.9's gateway visibility) risks spending effort on a lever that isn't actually where cost is concentrated.
• Every cost optimization has a potential quality trade-off and must be verified against evaluation (Part 6.2), never assumed safe.

Things you should be able to explain

• The three-factor cost equation and why stacked optimizations compound multiplicatively.
• Why measurement must precede optimization for it to be effective.

Things you should be able to build

• A per-service cost-breakdown report and an automated optimization-focus recommendation engine based on usage shape.

Common mistakes

• Optimizing the most interesting lever instead of the measured largest one.
• Treating cost optimization as a one-time project.
• Cutting cost without verifying quality holds via evaluation.

Recommended next chapter

Part 7 complete. Continue to handbook/08-reliability-genaiops/01-llm-rag-agent-evaluation.md.