7.8 — Caching Strategies for AI Systems

1. What is it?

Part 1.6 covered Redis and the general cache-aside pattern. This chapter goes deeper into caching strategies specific to AI pipelines: exact-match response caching (Part 1.6's original example, revisited), semantic caching (matching on meaning-similarity rather than exact text match, using embeddings, Part 2.5), and provider-level prompt caching (a distinct, increasingly important mechanism where the LLM provider itself caches and reuses portions of a prompt's processing across calls) — three genuinely different techniques addressing different cost/latency opportunities in an AI pipeline.

2. Why does it exist?

Part 1.6 established that repeated, identical LLM calls are pure waste — caching solves that directly. But AI-specific traffic patterns present two additional opportunities plain exact-match caching misses entirely: (1) many user queries are semantically similar without being textually identical ("what's your return policy" vs. "how do returns work" vs. "can I return this item") — exact-match caching (keyed on the literal prompt string) would treat these as entirely different, cache-missing requests despite them warranting essentially the same answer; and (2) many prompts share large, static portions (a long system prompt, a big retrieved-context block, Part 3.5) across many different calls, where only a small part (the actual user question) genuinely varies — provider-level prompt caching exists to let you avoid re-processing that large, static portion's tokens on every single call.

3. What problem does it solve?

Exact-match caching solves "don't pay for the same exact request twice" (Part 1.6). Semantic caching solves "don't pay for the same underlying request twice, even when phrased differently." Provider-level prompt caching solves "don't pay full price to reprocess the same large, unchanging context on every call that shares it" — three distinct, complementary cost/latency levers, each addressing a different real pattern in production AI traffic.

4. How does it work internally?

Semantic caching — the mechanism

1. New query arrives: "how do I return an item"
2. Embed the query (Part 2.5)
3. Search a cache index (a vector store, Part 3.4, but used as a cache rather than a
   knowledge base) for previously-cached queries with high similarity
4. If a sufficiently similar cached query exists (above a similarity threshold) AND
   its cached answer is still considered fresh (a TTL, Part 1.6) → return the cached
   answer directly, no LLM call
5. Otherwise → cache miss → call the LLM, cache the new query's embedding + answer
   for future semantic matches

The key new design parameter beyond Part 1.6's plain caching is the similarity threshold — set it too loose (accepting weakly-similar queries as cache hits) and you risk returning a subtly or entirely wrong cached answer for a genuinely different question; set it too strict and you get few cache hits, losing most of the benefit. This threshold should be tuned and evaluated (Part 6.2/8) against real query pairs, not guessed, since getting it wrong in the loose direction produces a genuinely serious quality problem (a confidently wrong cached answer) that's harder to detect than a simple cache miss.

Provider-level prompt caching — a fundamentally different mechanism

Several major LLM providers offer a "prompt caching" feature (verify the exact current mechanism, activation syntax, and pricing against your specific provider's current documentation — this is an actively evolving area of provider APIs) that lets you mark a portion of your prompt (typically a prefix — a long system prompt, a large retrieved-context block that repeats across many calls) as cacheable. On a cache hit (a subsequent call sharing that exact marked prefix), the provider skips re-processing that portion's tokens through the model from scratch, at typically substantially reduced cost and latency for the cached portion specifically — this is happening on the provider's infrastructure, not your own cache (Part 1.6/this chapter's semantic cache), and is a fundamentally different mechanism from either.

Call 1: [large system prompt + retrieved context]  (marked cacheable) + "question A"
              → provider caches the marked prefix's processed representation
Call 2: [SAME large system prompt + retrieved context] (cache hit!) + "question B"
              → provider reuses its cached processing of the prefix,
                only processes "question B" fresh — cheaper, faster

This is directly relevant to Part 3.5's RAG pattern specifically: if the same large retrieved-context block is reused across several follow-up questions in one conversation (or across similar queries retrieving the same top documents), marking it as a cacheable prefix can produce substantial, concrete cost/latency savings — a genuinely different lever from either exact-match or semantic caching, and one worth actively designing your prompt structure around (placing the static, shareable content as a stable prefix, and the variable, per-call content afterward) to take advantage of.

Minimum prefix length for a cache hit. Provider-level prompt caching typically requires the cacheable prefix to meet a minimum token count before the provider will actually cache it at all — a short system prompt or a small retrieved-context block may simply fall under this threshold and never be cached, regardless of how you mark it. Verify the exact current minimum against your specific provider's current documentation before relying on it (this is an actively evolving, provider- and model-specific number — historically on the order of roughly one to two thousand tokens for some providers' smaller models, and different again for larger models, so treat any specific figure, including approximate ones like this, as something to confirm rather than assume). A prompt structure that reuses a large, stable prefix (Part 3.5's retrieved-context block, a long system prompt) clears this bar easily; a short, mostly-boilerplate system prompt with little else shared across calls may not — worth checking explicitly (via the provider's cache-hit reporting in the response, where available) rather than assuming prompt caching is helping just because you marked a prefix as cacheable.

Layering all three together

A mature production RAG pipeline can legitimately use all three simultaneously, at different points: provider-level prompt caching for the (large, often-repeated) system prompt and retrieved-context prefix on every call; semantic caching for entire common-question-and-answer pairs that recur across different users (Part 1.6's FAQ example, upgraded to catch paraphrases); and exact-match caching (Part 1.6) as the simplest, cheapest first check before either of the more sophisticated layers, since an exact match is definitionally also a semantic match, and checking for it first avoids the embedding-and-search overhead of the semantic cache layer for the common case of genuinely repeated, identical queries.

5. Simple mental model

Exact-match caching (Part 1.6) is a filing cabinet drawer labeled with the exact question, only useful if someone asks that literal question again word-for-word. Semantic caching is a librarian who recognizes when a new question means essentially the same thing as one they've already answered, even phrased completely differently, and hands back the same answer. Provider-level prompt caching is like handing someone a thick reference binder once and having them remember its contents for your next several questions, so you only need to state the new specific question each time, rather than re-reading and re-explaining the entire binder's contents on every single question.

6. Real-world example

A customer support platform's most common queries ("how do I reset my password," "how do returns work," "where's my order") are phrased in dozens of different ways by real users but map to a small number of genuinely distinct underlying answers. Exact-match caching alone caught almost none of this variation (near-zero hit rate on paraphrased queries). Adding semantic caching (with a carefully tuned, evaluated similarity threshold, Part 8) raised the cache hit rate for these common FAQ-shaped queries substantially, directly cutting both LLM cost and response latency for a large fraction of real traffic — while a separate, simultaneous adoption of provider-level prompt caching for the platform's shared system prompt and product-policy context block (included in every single call regardless of cache hit/miss on the question itself) reduced the cost of every remaining cache-miss call too.

7. Architecture diagram

Query arrives

Exact-match cachePart 1.6

Semantic cacheembedding similarity search, Part 3.4 as cache

Return cached answer

Call LLMshared system prompt/context marked for PROVIDER-level prompt caching — reduces cost/latency even on a full cache miss

Cache the new answerat both application-cache layers, for future hits

8. Production considerations

• Tune and evaluate the semantic cache's similarity threshold explicitly (Part 6.2/8) — this is not a "set once" configuration value; it should be validated against real query pairs and re-evaluated as query patterns evolve.
• Design prompt structure to maximize provider-level cache hits — put the largest, most stable content (system prompt, shared context) as an early, unchanging prefix, and variable content (the specific user question) afterward, since most provider caching mechanisms cache based on a shared prefix match.
• Set explicit TTLs on cached answers (Part 1.6) at every cache layer, especially the semantic cache, since a stale but plausible-sounding cached answer returned via a semantic near-match is a particularly hard-to-notice failure mode (Part 1.6's original TTL argument applies with extra force here).
• Monitor cache hit rates per layer explicitly (Part 6.1/8.4) — you can't tell whether your semantic-caching investment is paying off without measuring its actual hit rate and the cost/latency delta it's producing.

9. Common mistakes

• Setting a semantic cache's similarity threshold too loosely, causing genuinely different questions to receive a wrong, cached answer — a subtle, quality-degrading failure mode that's much harder to detect than a simple cache miss, since the returned answer looks plausible and confident.
• Structuring prompts with variable content first and shared/static content afterward, missing the prefix-matching requirement most provider-level prompt-caching mechanisms depend on.
• Caching answers with no TTL for content that can legitimately change over time (Part 1.6's staleness warning, doubly relevant since a stale semantic-cache hit is harder to notice than a stale exact-match hit).
• Not measuring cache hit rates, making it impossible to know whether a caching investment (particularly the more complex semantic-caching layer) is actually paying for its added complexity.

10. Security considerations

• A semantic cache, like Part 3.4's vector store, must enforce tenant/user scoping on lookups — returning tenant A's cached answer for a semantically-similar query from tenant B is exactly the cross-tenant leakage risk from Part 3.4/9.6, now applying to a cache layer specifically.
• Cached answers containing personalized or sensitive information must be scoped narrowly (per-user or per-tenant, not shared globally) — a global semantic cache is only safe for genuinely non-personalized, shareable answers (general policy questions), not anything containing or derived from one specific user's private data.

11. Performance considerations

• Semantic cache lookups have their own latency cost (an embedding call plus a vector similarity search, Part 2.5/3.4) — for this to be a net win, the semantic-cache-hit path's total latency must be meaningfully lower than an actual LLM call, which is usually true but worth verifying rather than assuming for your specific setup.
• Provider-level prompt caching's latency benefit is realized specifically on the cached-prefix portion — the variable, non-cached portion of a call still incurs its normal processing time.

12. Cost considerations

• This entire chapter is fundamentally a cost-optimization discipline, directly feeding Part 7.10's broader cost framework — semantic caching and provider-level prompt caching are two of the highest-leverage, most AI-specific cost levers available, often underused relative to their potential impact because they're less obvious than simple exact-match caching.
• Provider-level prompt caching pricing (often a reduced rate for cache-hit tokens versus fresh-processed tokens) varies by provider and is worth modeling explicitly against your specific prompt structure and call patterns — verify current pricing against your provider's current documentation rather than assuming a fixed percentage discount.

13. When to use it

Exact-match caching: any deterministic, frequently-repeated exact query (Part 1.6). Semantic caching: high-volume, FAQ-shaped traffic with genuine paraphrase variation, where a tuned similarity threshold can be validated to avoid wrong-answer risk. Provider-level prompt caching: any pipeline with a large, stable, frequently-reused prompt prefix (RAG systems with repeated retrieved-context reuse, agents with long, stable system prompts, Part 3.1/3.5) — close to a default optimization worth applying wherever your provider supports it and your prompt structure allows it.

14. When NOT to use it

Semantic caching is inappropriate for queries requiring precise, individually-correct answers where even a small risk of a subtly-wrong near-match is unacceptable (Part 3.4's exact-match-data warning applies here too) — don't semantically cache answers to questions like "what is my current account balance," where only an exact, current, individually-computed answer is ever correct. Provider-level prompt caching provides little benefit for pipelines with genuinely unique, non-repeated prompts on every call.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Exact-match caching (Part 1.6)	Simple, safe, zero wrong-answer risk	Misses paraphrase variation entirely
Semantic caching	Catches paraphrase variation, high hit-rate potential for FAQ-shaped traffic	Wrong-answer risk if threshold is mistuned; added latency/complexity
Provider-level prompt caching	Reduces cost/latency for large, shared prompt prefixes, no added application complexity	Only helps the specific cached-prefix portion; provider-specific mechanism/pricing

16. Practical Python/code example

A semantic cache lookup, extending Part 1.6's exact-match cache with an embedding-similarity layer:

async def semantic_cache_lookup(
    vector_store, embed_fn, query: str, tenant_id: str, similarity_threshold: float = 0.92
) -> str | None:
    """
    Checks for a semantically similar, previously-cached answer, scoped to tenant.

    Args:
        vector_store: A vector store used as a cache index (Part 3.4).
        embed_fn: An async embedding function (Part 2.5).
        query (str): The new query to check against cached entries.
        tenant_id (str): Tenant scope, enforced as a pre-filter.
        similarity_threshold (float): Minimum similarity to accept a cache hit —
            must be tuned and evaluated against real query pairs (Part 6.2/8),
            not left at an unvalidated default in production.

    Returns:
        str | None: The cached answer if a sufficiently similar entry exists, else None.
    """
    query_embedding = (await embed_fn([query]))[0]
    results = await vector_store.query(
        vector=query_embedding, filter={"tenant_id": {"$eq": tenant_id}}, top_k=1
    )
    if results and results[0]["score"] >= similarity_threshold:
        return results[0]["metadata"]["cached_answer"]
    return None

17. Production-quality example

Layering exact-match, semantic, and provider-level caching together, per section 4's combined-strategy discussion:

import logging

logger = logging.getLogger("layered_cache")


async def answer_with_layered_cache(
    redis, vector_store, embed_fn, llm_client, query: str, tenant_id: str,
    shared_system_prompt: str,
) -> str:
    """
    Answers a query using exact-match, then semantic, then a full LLM call with
    provider-level prompt caching applied to the shared, stable prompt prefix.
    """
    exact_cached = await get_cached_or_call_llm.__wrapped__(redis, "exact", query) if False else None
    exact_key = f"exact_cache:{tenant_id}:{hash(query)}"
    exact_cached = await redis.get(exact_key)
    if exact_cached:
        logger.info("exact-match cache hit")
        return exact_cached.decode()

    semantic_hit = await semantic_cache_lookup(vector_store, embed_fn, query, tenant_id)
    if semantic_hit:
        logger.info("semantic cache hit")
        return semantic_hit

    logger.info("cache miss at both layers, calling LLM with cacheable prefix")
    response = await llm_client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=500,
        system=[
            {
                "type": "text",
                "text": shared_system_prompt,
                "cache_control": {"type": "ephemeral"},  # verify exact current syntax against docs
            }
        ],
        messages=[{"role": "user", "content": query}],
    )
    answer = response.content[0].text

    await redis.set(exact_key, answer, ex=3600)
    await cache_semantic_entry(vector_store, embed_fn, query, answer, tenant_id)
    return answer

Spot the bug. Look again at the function's first real line:

exact_cached = await get_cached_or_call_llm.__wrapped__(redis, "exact", query) if False else None
exact_key = f"exact_cache:{tenant_id}:{hash(query)}"
exact_cached = await redis.get(exact_key)

The first line assigns to exact_cached via a conditional expression gated on the literal False — that branch (which also calls get_cached_or_call_llm.__wrapped__, a function that doesn't even exist anywhere in this module) can never execute, and the value it would assign is immediately thrown away by the very next line's real cache lookup, which reassigns exact_cached unconditionally. This is dead code: harmless here only because it's clobbered before it's ever read, but it's exactly the kind of leftover-from-a-refactor line — probably an earlier version of this function that called a since-renamed or since-removed helper — that should be deleted in code review, not shipped. A linter with dead-code/unreachable-branch detection would flag the if False immediately; absent that tooling, this is the kind of line that survives silently in a real codebase for a surprisingly long time precisely because it doesn't cause a visible bug, it just wastes a reader's attention figuring out what it does. The fix is simply to delete that first line entirely — the function is correct without it.

18. Short exercise

A team's semantic cache, tuned with a similarity threshold of 0.85, occasionally returns the cached answer for "how do I cancel my subscription" when a user asks "how do I cancel my order" — two genuinely different questions with superficially similar phrasing. Explain what this reveals about the threshold, and describe the evaluation process (referencing Part 6.2) you'd use to find a better value.

19. Interview questions

• Explain the difference between semantic caching and provider-level prompt caching — what does each actually cache, and at what layer?
• Why is a mistuned semantic-cache similarity threshold a more dangerous failure mode than a simple cache miss?
• How would you structure a RAG pipeline's prompt to take advantage of provider-level prompt caching for its retrieved-context block?

20. FDE/customer scenario

Customer: "Our support bot answers the same handful of common questions constantly, phrased differently every time — is there a way to make that cheaper and faster without hurting quality?"

This is precisely the semantic-caching use case (section 6), and the FDE-correct response includes the caveat this chapter emphasizes: semantic caching can deliver real, substantial savings for exactly this FAQ-shaped pattern, but only with a carefully tuned and evaluated similarity threshold — proposing it without also proposing the evaluation process to validate the threshold would risk introducing a subtle, hard-to-detect wrong-answer problem in exchange for the cost savings, a trade-off no credible technical recommendation should make silently.

---

Key takeaways

• Exact-match, semantic, and provider-level prompt caching are three distinct mechanisms addressing different AI-specific traffic patterns — a mature system often layers all three.
• Semantic caching's similarity threshold is a genuine, evaluatable parameter, not a default to leave untuned — mistuning it risks a subtle, hard-to-detect wrong-answer failure mode.
• Provider-level prompt caching rewards structuring prompts with stable, shared content as an early prefix and variable content afterward.

Things you should be able to explain

• The distinct mechanism and use case for each of the three caching strategies.
• Why a mistuned semantic-cache threshold is more dangerous than a simple cache miss.

Things you should be able to build

• A layered caching pipeline combining exact-match, semantic, and provider-level prompt caching with tenant-scoped, TTL-bounded entries.

Common mistakes

• Mistuned semantic-cache similarity thresholds.
• Prompt structures that don't take advantage of provider-level prefix caching.
• No TTL on cached answers for content that can change.

Recommended next chapter

09-llm-gateways-model-routing.md