1.6 — Redis

1. What is it?

Redis is an in-memory data structure store, used primarily as a cache, a message broker, and a fast key-value store. In AI systems it shows up everywhere: caching LLM responses, rate limiting, session/conversation state, distributed locks, and as the backing store for job queues (Part 7).

2. Why does it exist?

Databases like Postgres are optimized for durability and correctness at the cost of speed (disk-backed, transactional overhead). Many problems don't need that: "has this exact prompt been answered in the last hour," "how many requests has this user made in the last minute," "what's the current step of this in-flight agent run" — these need to be fast far more than they need to survive a server crash perfectly. Redis exists to serve that class of problem: everything lives in memory, operations are typically O(1) or O(log n), and the whole system is built around sub-millisecond latency.

3. What problem does it solve?

For AI applications specifically, Redis solves the "LLM calls are slow and expensive, don't repeat them unnecessarily" problem (caching), the "don't let one user or one bug bankrupt us" problem (rate limiting/token bucket), and the "an agent's conversation needs shared, fast, cross-request state" problem (session/memory storage) — see 09-memory.md in Part 3 for the deeper agent-memory architecture that often sits on top of Redis.

4. How does it work internally?

Single-threaded core, why it's still fast

Redis's core command processing is single-threaded (newer versions offload some I/O to background threads, but command execution itself is single-threaded) — this sounds like a limitation but is actually a large part of why Redis is simple and fast: no locks are needed for most operations, because only one command executes at a time. The trade-off: a single slow command (e.g., KEYS * on a huge keyspace, or a large SORT) blocks every other client during that command — this is a real, common production incident (see "common mistakes").

Cache-aside (the dominant pattern for LLM response caching)

1. Check cacheRedis — hit returns immediately, miss continues

2. Computecall the LLM, only on a miss

3. Write resultto Redis, with a TTL

4. Return result to caller

Your application code — not Redis itself — is responsible for checking the cache, falling back to the real computation on a miss, and writing the result back. This is "cache-aside" (a.k.a. lazy loading), the standard pattern for LLM response caching: hash the (model, prompt, parameters) tuple into a cache key, check Redis first, only call the LLM on a miss.

TTL and invalidation

Every cached entry should have a TTL (time-to-live) — Redis will expire it automatically. This matters enormously for AI caching because LLM outputs can go stale (a cached answer about "today's promotions" is wrong tomorrow) in ways a generic web cache wouldn't need to worry about as sharply. There is no general-purpose automatic invalidation for semantic staleness — the famous line "there are only two hard things in computer science: cache invalidation and naming things" applies directly: you must design when a cached LLM answer should be considered stale (time-based TTL is the simplest; event-based invalidation — e.g., invalidate on underlying document update — is more correct but more work).

Distributed locks

Redis's SET key value NX PX ttl (set-if-not-exists with expiry) is the basis for a simple distributed lock — used when multiple AI worker processes must not process the same job/agent-run concurrently. Real distributed locking at scale needs more care than a single SETNX (see Redlock discussion and its critiques) but the basic primitive is this atomic set-if-absent operation.

5. Simple mental model

Redis is a whiteboard next to the office, not the filing cabinet down the hall. The filing cabinet (Postgres) is where the permanent, trustworthy record lives — slower to get to, but it survives a fire. The whiteboard is right there, instantly readable and writable, but if the power goes out (without persistence configured) or someone erases it, whatever wasn't also written to the filing cabinet is just gone. You put things on the whiteboard that are either disposable (a cache) or that you're actively working with and will persist properly soon (a lock, a rate-limit counter).

6. Real-world example

An e-commerce customer support AI answers "what's your return policy" hundreds of times a day, verbatim or near-verbatim across different customers. Without caching, that's hundreds of redundant LLM calls a day for effectively identical output. With cache-aside and a well-chosen cache key (e.g., a hash of the normalized question + a policy-version identifier so the cache invalidates when the policy actually changes), the vast majority of those calls are served from Redis in under a millisecond, cutting both latency and LLM cost substantially for FAQ-shaped traffic.

7. Architecture diagram

AI Backend

Redis1. check (miss) → 4. write + TTL

LLM API2-3. call on miss, slow, costs money

Cache-aside: the application, not Redis, owns the miss/compute/write-back logic.

8. Production considerations

• Persistence: decide explicitly whether you need RDB snapshots, AOF (append-only file), both, or neither. Pure-cache use cases often need neither (losing the cache on restart just means more cache misses temporarily); rate-limit counters or distributed locks may need more care.
• Eviction policy (maxmemory-policy): when Redis hits its memory limit, what does it evict — least-recently-used, random, or does it refuse new writes? allkeys-lru is the common default for pure cache workloads; get this wrong and Redis either OOMs or evicts the wrong data.
• Redis Cluster / Sentinel for high availability — a single Redis instance is a single point of failure; production AI systems typically run replicated Redis with automatic failover.
• Connection pooling applies here too — the redis-py async client should be a shared, pooled instance, not recreated per request.

9. What happens if Redis goes down

This is the question every FDE should force themselves to answer explicitly for every Redis use case in a system, because the failure modes are very different depending on what Redis was doing:

Use case	If Redis goes down	Correct design response
LLM response cache	Every request becomes a cache miss — slower and more expensive, but not incorrect	Design so cache failures degrade gracefully (catch connection errors, fall through to the real LLM call) — never let a cache outage become an application outage
Rate limiting	If you fail open (allow all requests when Redis is unreachable), you lose protection right when you might need it most; if you fail closed, a Redis outage becomes a full outage	Decide deliberately; many systems fail closed for cost-sensitive LLM rate limits (accept an outage over an unbounded bill)
Session/conversation state	Users lose in-progress conversation context	Consider whether critical conversation state should also be durably persisted (Postgres) with Redis as an accelerating cache in front of it, not the sole source of truth
Distributed lock for job processing	Depends on failure mode — if the lock can't be acquired, jobs may stall (safe) or, with a badly implemented lock, may double-process (unsafe)	Prefer well-tested locking libraries over hand-rolled SETNX logic for anything where double-processing is costly

The single biggest architectural mistake FDEs see: treating Redis as a source of truth when it was only ever meant to be an accelerator. If losing Redis's data would be a genuine data-loss incident, that data belongs in a durable store, with Redis as a cache in front of it, not instead of it.

10. Security considerations

• Redis has no meaningful default authentication in many deployment configs — an exposed Redis instance (default port, no requirepass/ACLs, no network isolation) is a common, serious real-world breach vector; never expose Redis to the public internet.
• Use Redis ACLs (Redis 6+) to scope which commands/keys a given service credential can touch — an LLM-response-cache service shouldn't have the same Redis permissions as an admin tool.
• Be careful caching responses that include user-specific or tenant-specific data under a key that doesn't include the tenant/user identity — this is a direct cross-tenant data leakage vector, exactly the kind of subtle bug that shows up in multi-tenant AI SaaS (Part 11.4).

11. Performance considerations

• Avoid KEYS * in production (O(n), blocks the single-threaded server) — use SCAN for iteration instead.
• Pipeline multiple commands when you don't need each result before issuing the next, to avoid round-trip latency stacking.
• Choose data structures deliberately: a HASH for structured per-key data is often better than JSON-serializing into a STRING, since it allows partial field reads/writes.

12. Cost considerations

• Caching directly reduces LLM API cost for repeated/similar queries — often one of the highest-leverage, lowest-effort cost optimizations available (Part 7.10 covers this as part of a broader cost strategy).
• Redis itself has a cost (memory is the expensive resource for an in-memory store) — caching everything indiscriminately, including large or rarely-reused LLM responses, can turn into meaningful infrastructure spend for little hit-rate benefit. Cache what's actually likely to repeat.

13. When to use it

Caching expensive/repeated LLM calls, rate limiting, ephemeral session state, distributed locks, and as a message broker for lightweight job queues.

14. When NOT to use it

• As the sole, durable source of truth for data that must survive a crash without loss (use Postgres or another durable store).
• For complex relational queries or full-text/semantic search — Redis's data structures are simple by design; that's the trade for its speed.
• For very large values (large documents, big embedding batches) — Redis is optimized for many small, fast operations, not large blob storage.

15. Alternatives and trade-offs

Tool	Good for	Weak point
Redis	Fast cache, rate limiting, locks, lightweight queues	Not durable by default; single-threaded core for commands
Memcached	Pure caching, simpler, multi-threaded	Fewer data structures/features (no pub/sub, streams, etc.)
Postgres (as cache substitute)	Durable, transactional	Much slower for high-frequency cache-shaped access patterns
In-process cache (e.g., functools.lru_cache)	Zero infra, simplest	Not shared across processes/instances — useless in a horizontally-scaled deployment

16. Practical Python/code example

import hashlib
import json
from redis.asyncio import Redis


def _cache_key(model: str, prompt: str) -> str:
    """Builds a stable cache key from the model name and prompt text."""
    digest = hashlib.sha256(f"{model}:{prompt}".encode()).hexdigest()
    return f"llm_cache:{digest}"


async def get_cached_or_call_llm(
    redis: Redis, model: str, prompt: str, call_llm, ttl_seconds: int = 3600
) -> str:
    """
    Returns a cached LLM response if present, otherwise calls the LLM and caches the result.

    Args:
        redis (Redis): Shared async Redis client.
        model (str): Model identifier, included in the cache key so different models
            don't share cached answers.
        prompt (str): The exact prompt sent to the model.
        call_llm: An async callable that performs the real LLM call.
        ttl_seconds (int): How long the cached response remains valid.

    Returns:
        str: The LLM's response text, from cache or freshly computed.
    """
    key = _cache_key(model, prompt)
    cached = await redis.get(key)
    if cached is not None:
        return json.loads(cached)["response"]

    response = await call_llm(model, prompt)
    await redis.set(key, json.dumps({"response": response}), ex=ttl_seconds)
    return response

17. Production-quality example

import hashlib
import json
import logging
from redis.asyncio import Redis
from redis.exceptions import RedisError

logger = logging.getLogger("llm_cache")


class LLMResponseCache:
    """Cache-aside layer for LLM responses that degrades gracefully if Redis is unavailable."""

    def __init__(self, redis: Redis, default_ttl_seconds: int = 3600):
        """
        Args:
            redis (Redis): Shared, pooled async Redis client.
            default_ttl_seconds (int): Default cache entry lifetime.
        """
        self._redis = redis
        self._default_ttl = default_ttl_seconds

    def _key(self, model: str, prompt: str, tenant_id: str) -> str:
        """Builds a tenant-scoped cache key so responses never leak across tenants."""
        digest = hashlib.sha256(f"{model}:{prompt}".encode()).hexdigest()
        return f"llm_cache:{tenant_id}:{digest}"

    async def get(self, model: str, prompt: str, tenant_id: str) -> str | None:
        """
        Attempts to fetch a cached response, treating any Redis failure as a cache miss.

        Args:
            model (str): Model identifier.
            prompt (str): Exact prompt text.
            tenant_id (str): Tenant scope for the cache key.

        Returns:
            str | None: The cached response, or None on miss or Redis failure.
        """
        try:
            cached = await self._redis.get(self._key(model, prompt, tenant_id))
        except RedisError:
            logger.warning("cache read failed, falling back to live call", exc_info=True)
            return None
        return json.loads(cached)["response"] if cached else None

    async def set(self, model: str, prompt: str, tenant_id: str, response: str) -> None:
        """
        Writes a response to the cache, swallowing (but logging) any Redis failure.

        Args:
            model (str): Model identifier.
            prompt (str): Exact prompt text.
            tenant_id (str): Tenant scope for the cache key.
            response (str): The LLM response to cache.
        """
        try:
            await self._redis.set(
                self._key(model, prompt, tenant_id),
                json.dumps({"response": response}),
                ex=self._default_ttl,
            )
        except RedisError:
            logger.warning("cache write failed, continuing without caching", exc_info=True)

Note the deliberate design: a cache failure never becomes a request failure. This is the core lesson of section 9 encoded directly into the code.

18. Short exercise

Design a rate limiter using Redis (INCR + EXPIRE, or a sorted-set sliding window) that allows a tenant a maximum of 100 LLM calls per minute. Decide and justify: does it fail open or closed if Redis is unreachable, given this is protecting against runaway LLM cost?

19. Interview questions

• Why is Redis single-threaded for command execution, and what production incident can that cause if you're not careful?
• Walk through the cache-aside pattern and explain where cache invalidation logic actually needs to live.
• For an LLM response cache, a rate limiter, and agent conversation state — which of these, if any, should have Redis as their only backing store, and why?

20. FDE/customer scenario

Customer: "Our AI costs spiked 4x last week for no reason we can find."

One of the first things worth checking: was there a Redis outage or a cache-key bug (e.g., a deploy that accidentally included a timestamp or request ID in the cache key, making every request a guaranteed cache miss) around that time? Cache-related cost spikes are common enough in production AI systems that this is a standard early diagnostic question, not a long-shot guess.

---

Key takeaways

• Redis trades durability for speed — treat it as an accelerator in front of a durable store, never as the sole source of truth for data you can't afford to lose.
• Every Redis-dependent code path needs an explicit, deliberate answer to "what happens if Redis is unreachable."
• Cache-aside is the standard LLM response caching pattern; TTL is the simplest invalidation strategy, and often good enough.

Things you should be able to explain

• Why Redis being single-threaded doesn't make it slow, but does create a specific footgun.
• The fail-open vs fail-closed decision for rate limiting under a Redis outage.

Things you should be able to build

• A tenant-scoped LLM response cache that degrades gracefully on Redis failure.

Common mistakes

• Treating Redis as durable storage.
• Cache keys that don't account for tenant scope (leakage) or that vary per-request unintentionally (guaranteed misses).
• Using KEYS * in production.

Recommended next chapter

07-git.md