3.9 — Memory

1. What is it?

Memory, in an AI system, is the mechanism by which information persists across interactions — within a single conversation (short-term/working memory) and across separate conversations or sessions (long-term memory). Because an LLM itself is stateless (Part 2.6 — each API call is independent; the model retains nothing from a previous call unless you explicitly include it in the new call's context), every form of "memory" in an AI application is something you build and manage outside the model, not a capability the model has natively.

2. Why does it exist?

Without any memory mechanism, every single message to an LLM would need to include the entire relevant history from scratch, and the system would have no way to recall anything from a previous session at all — every conversation would start from zero, every time, with a user who might reasonably expect the system to remember "we talked about this yesterday." Memory exists to solve this by explicitly managing what information gets carried forward, in what form, and for how long — a genuine engineering discipline, not a switch you flip on.

3. What problem does it solve?

It solves two related but distinct problems: within a conversation, keeping the model's context relevant and complete enough to maintain coherence over a long interaction, without the context growing unboundedly (which would eventually exceed context window limits and cost, Part 2.4/2.6); and across conversations, letting a system recall relevant facts, preferences, or history from a user's past interactions without re-explaining everything every single time.

4. How does it work internally?

Short-term (working) memory — the conversation history problem

The simplest form of memory is just replaying the full conversation history in every new API call — this is literally what "having a conversation" with an LLM means mechanically: each new call includes every previous user and assistant message as context (Part 2.4's role-tagged message structure). This works, but doesn't scale indefinitely: context grows with every turn, eventually hitting context-window limits and, well before that, becoming expensive (Part 2.6 — every token in history is paid for, on every single call, for the entire conversation) and potentially degrading quality (Part 2.4's "lost in the middle" effect applies to a long, undifferentiated conversation history just as it does to a long document).

Production systems manage this growth with several strategies:

• Sliding window: keep only the most recent N messages, dropping older ones entirely. Simple, but loses potentially important early context permanently.
• Summarization: periodically (or continuously) compress older parts of the conversation into a shorter summary, replacing the verbatim messages with that summary while keeping recent messages verbatim. This is a genuine trade-off: summarization is itself an LLM call (with its own cost and its own risk of losing or subtly distorting details, Part 2.6's hallucination discussion applies to lossy summarization too), but it lets the system preserve the gist of a long conversation within a bounded token budget.
• Selective retention: explicitly extracting and retaining only specific, structured facts from the conversation (e.g., "user's account tier: enterprise," "user's stated goal: migrate off legacy system") rather than either the full verbatim history or a lossy general summary — often the most reliable approach for information that will genuinely matter later, at the cost of needing to define ahead of time what's worth extracting.

Long-term memory — three genuinely different architectural approaches

• Semantic memory (vector-store-backed): store facts/conversation excerpts as embeddings (Part 2.5) in a vector database (Part 3.4), and retrieve relevant ones at the start of a new session using similarity search against the current query — structurally, this is RAG (Part 3.5) applied to a user's own history instead of a document corpus. This is powerful for "recall something relevant from a large history of past interactions" but inherits every RAG failure mode (Part 3.5): retrieval can miss relevant memories, and there's no guarantee the retrieved memory is actually still accurate/current.
• Structured memory (database-backed): store specific, explicit facts in a traditional structured store (Part 1.4/1.5) — e.g., a user_preferences table with defined fields. More reliable and precise than semantic retrieval for facts you know ahead of time you'll need (a user's stated communication preference, their subscription tier), but requires deciding in advance what structured facts are worth capturing — it can't spontaneously capture something you didn't anticipate wanting to remember.
• Episodic/summary memory: periodically generate and store a natural-language summary of a completed conversation/session, to be included (in full or via retrieval) in future sessions — a middle ground between full verbatim recall (too much) and rigid structured fields (too little flexibility), at the cost of the summarization-quality risk noted above.

Real production systems typically combine more than one of these — structured memory for the small set of facts that are reliably worth precise tracking, semantic memory for open-ended "has this user asked about something like this before," and short-term conversation management (sliding window or summarization) for the current session's own coherence.

5. Simple mental model

Short-term memory is like keeping the current conversation's relevant notes on your desk while you're actively talking to someone — you don't re-read every word ever said in the meeting, but you do refer back to what was just discussed, and you might jot down a condensed summary of an earlier part of a long meeting so you don't need to hold every word in your head. Long-term memory is like a filing system you consult before a new meeting with someone you've met before — you don't remember every word of every past conversation with them from memory, but you can pull the relevant file (structured facts: their title, their account tier) and, if useful, search your notes for anything relevant to today's specific topic (semantic memory).

6. Real-world example

An enterprise IT helpdesk assistant needs to remember, across sessions, that a specific employee's laptop was already replaced last month (relevant context for a new "my laptop is slow" ticket, since a hardware-replacement suggestion would be redundant and confusing) — this is a good fit for structured memory (a specific, known-ahead-of-time fact worth tracking explicitly: last_hardware_replacement_date). The same assistant might also want to recall, more loosely, "has this employee previously described a similar intermittent issue in a past conversation, even if phrased completely differently" — this is a better fit for semantic memory (embedding past ticket summaries and retrieving similar ones), since you can't enumerate every possible phrasing of "intermittent issue" as a structured field ahead of time.

7. Architecture diagram

Within one conversation

Turn 1kept verbatim if recent

Turn 2kept verbatim if recent

Turn Ncurrent, verbatim

Summary of older turnscompressed, if history grows long

Across sessions (long-term)

Structured memoryDB: known facts, e.g. account tier

Semantic memoryvector store: embedded session summaries

Short-term (within a conversation) and long-term (across sessions) memory are different mechanisms.

8. Production considerations

• Decide explicitly, per fact, which memory strategy fits — don't default to "embed everything" (inherits RAG's retrieval-miss risk for facts that should be reliably remembered) or "structure everything" (can't capture open-ended, unanticipated relevant context).
• Summarization for context management needs its own evaluation (Part 8) — a summarization step that silently drops a critical detail (a stated constraint, a promised commitment) can cause real downstream errors that are hard to trace back to "the summary lost this," since the original detail is simply gone from what the model sees afterward.
• Memory needs explicit staleness/expiry handling — a structured fact like "user's subscription tier" can become outdated if not refreshed from the actual source of truth; treat cached/remembered facts about mutable real-world state with the same staleness skepticism as any cache (Part 1.6).
• Memory storage is itself sensitive data (Part 9.6) — conversation history and extracted facts about a specific user often qualify as personal data, subject to the same access-control, retention, and deletion-request (e.g., GDPR "right to be forgotten") requirements as any other PII store, and this is frequently a real, contractually-binding customer requirement, not a hypothetical concern.

9. Common mistakes

• Naively growing conversation history unboundedly until hitting a context-window error in production, with no proactive summarization/windowing strategy in place beforehand.
• Using semantic (vector-store) memory for facts that should be reliably, deterministically remembered (e.g., a legal agreement's specific terms) — semantic retrieval's inherent approximation (Part 3.4) is the wrong reliability profile for facts that must never be missed or misremembered.
• Treating a remembered fact as permanently current without any mechanism to detect or handle staleness.
• Not considering data-retention/deletion requirements for stored conversation memory until a compliance review forces the question late in a project.

10. Security considerations

• Long-term memory stores are a concentrated, high-value target — a semantic memory store containing embeddings of past conversations (Part 2.5's embedding-inversion risk) or a structured-facts database about users is exactly the kind of system that needs the tenant-isolation and access-control discipline from Part 1.5/3.4/9.6, and a breach here is potentially more damaging than a single conversation leak, since it aggregates history across many interactions.
• Memory that persists across sessions creates a longer-lived injection risk surface: if an earlier session's stored memory was influenced by injected content (Part 9.1), that influence can resurface and affect a completely different, later session — a subtler and longer-lived version of the single-conversation injection risk.

11. Performance considerations

• Retrieving long-term memory (a semantic search, Part 3.4, or a database query, Part 1.4/1.5) adds a retrieval step's latency to the start of every session that uses it — profile this the same way you would any other RAG-adjacent retrieval step (Part 3.5, section 11).
• Summarization-based short-term memory management trades a periodic extra LLM call (the summarization itself) for keeping ongoing per-turn context smaller and cheaper — a real, measurable trade-off worth tuning (how often to summarize, how aggressively).

12. Cost considerations

• Unbounded conversation history growth directly and compoundingly increases per-turn cost over a long conversation (Part 2.6) — this is one of the more common, underestimated cost drivers in production conversational AI systems, since it's invisible turn-by-turn but very real in aggregate.
• Long-term memory storage (vector store entries, database rows) accumulates over time across a user base — worth an explicit retention policy (how long is memory kept, and is old, likely-stale memory ever pruned) rather than indefinite, unbounded accumulation.

13. When to use it

Any conversational AI system needing coherence within a session beyond a couple of turns (short-term memory, essentially always relevant), and any system where recalling user-specific context across separate sessions provides real value (customer support history, personalization, ongoing project context) — long-term memory specifically.

14. When NOT to use it

• Stateless, single-turn interactions (a one-off classification or extraction call, Part 3.2) have no need for memory machinery at all — don't add the complexity where it provides no value.
• Long-term memory that would primarily surface stale or potentially incorrect "remembered" facts about frequently-changing real-world state is often better replaced by a direct, current lookup (a tool call to the actual source of truth, Part 3.3) rather than a remembered, potentially-outdated fact.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Full verbatim history (short-term)	Simplicity, perfect fidelity within budget	Doesn't scale — cost and context-window limits eventually force a strategy
Summarization	Bounded cost/context for long conversations	Lossy; summarization itself can introduce errors
Structured (DB) long-term memory	Reliable, precise for known-ahead-of-time facts	Can't capture unanticipated, open-ended context
Semantic (vector) long-term memory	Flexible, open-ended recall across large history	Inherits RAG's retrieval-miss and staleness risks

16. Practical Python/code example

A sliding-window-with-summarization conversation manager, directly implementing the short-term memory strategy from section 4:

async def manage_conversation_context(
    client, messages: list[dict], max_verbatim_messages: int = 10
) -> list[dict]:
    """
    Keeps the most recent messages verbatim and summarizes older ones once the
    conversation exceeds a threshold, bounding context growth over a long session.

    Args:
        client: An async LLM client, used for the summarization call itself.
        messages (list[dict]): The full conversation so far, role-tagged.
        max_verbatim_messages (int): How many of the most recent messages to keep
            in full detail; older messages beyond this are condensed.

    Returns:
        list[dict]: A context-managed message list: an optional summary message
            followed by the most recent verbatim messages.
    """
    if len(messages) <= max_verbatim_messages:
        return messages

    older_messages = messages[:-max_verbatim_messages]
    recent_messages = messages[-max_verbatim_messages:]

    older_text = "\n".join(f"{m['role']}: {m['content']}" for m in older_messages)
    summary_response = await client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=300,
        system="Summarize this conversation excerpt concisely, preserving any specific facts, commitments, or constraints mentioned.",
        messages=[{"role": "user", "content": older_text}],
    )
    summary_message = {
        "role": "user",
        "content": f"[Earlier conversation summary]: {summary_response.content[0].text}",
    }
    return [summary_message, *recent_messages]

17. Production-quality example

A structured long-term memory store with explicit staleness handling, addressing the section 8 concern about treating remembered facts as permanently current:

import logging
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

logger = logging.getLogger("user_memory")


@dataclass
class RememberedFact:
    """A single structured fact about a user, with staleness tracking."""
    key: str
    value: str
    recorded_at: datetime
    max_age: timedelta


class StructuredUserMemory:
    """Stores and retrieves structured facts about a user, treating each as
    potentially stale rather than permanently true."""

    def __init__(self, db_session_factory):
        """
        Args:
            db_session_factory: Async session factory for the facts store.
        """
        self._session_factory = db_session_factory

    async def remember(self, user_id: str, key: str, value: str, max_age: timedelta) -> None:
        """
        Stores or updates a structured fact about a user, with an explicit staleness window.

        Args:
            user_id (str): The user this fact is about.
            key (str): The fact's identifier (e.g., "subscription_tier").
            value (str): The fact's value.
            max_age (timedelta): How long this fact should be trusted before being
                treated as stale and requiring re-verification from the source of truth.
        """
        async with self._session_factory() as session:
            await session.execute(
                upsert_fact_query(user_id=user_id, key=key, value=value, recorded_at=datetime.now(timezone.utc)),
            )
            await session.commit()

    async def recall(self, user_id: str, key: str) -> str | None:
        """
        Retrieves a fact if present and not yet stale; returns None if stale or missing,
        signaling the caller to re-fetch from the authoritative source instead of trusting
        a potentially outdated remembered value.

        Args:
            user_id (str): The user to recall a fact about.
            key (str): The fact's identifier.

        Returns:
            str | None: The fact's value if fresh, otherwise None.
        """
        async with self._session_factory() as session:
            row = await session.execute(get_fact_query(user_id=user_id, key=key))
            fact = row.first()

        if fact is None:
            return None
        if datetime.now(timezone.utc) - fact.recorded_at > fact.max_age:
            logger.info("fact %s for user=%s is stale, treating as unknown", key, user_id)
            return None
        return fact.value

18. Short exercise

A customer support assistant remembers, via structured memory, that a user's subscription tier is "enterprise" — recorded three months ago. The customer downgraded their subscription last week through a self-service portal your AI system doesn't currently receive events from. Explain, using the staleness mechanism from section 17, what would happen with a max_age of 30 days versus a max_age of 6 months, and which is the safer default given this scenario.

19. Interview questions

• Why is an LLM itself stateless, and what does that imply about where "memory" actually lives in an AI system?
• Compare summarization-based and sliding-window short-term memory strategies — what does each preserve and what does each risk losing?
• When would you choose structured (database) memory over semantic (vector-store) memory for a specific fact, and why?

20. FDE/customer scenario

Customer: "We want the assistant to remember everything about every past interaction with a customer, forever."

This deserves the same reasoning-before-recommending discipline as any other FDE request: "remember everything, forever" has real cost implications (unbounded storage and retrieval growth), real compliance implications (data retention policy, right-to-deletion requirements depending on jurisdiction/industry, Part 10.5), and real reliability implications (old, potentially stale "memories" being trusted as current). The FDE-correct response distinguishes what's actually valuable to remember long-term (structured, durable facts; a bounded amount of semantic history) from what should have a defined retention window or be re-verified from source rather than trusted indefinitely from memory.

---

Key takeaways

• LLMs are stateless; every form of memory is explicitly engineered by the application, not a native model capability.
• Short-term (within-conversation) memory management (windowing, summarization) and long-term (cross-session) memory (structured vs. semantic) are distinct problems needing distinct strategies.
• Remembered facts about mutable real-world state need explicit staleness handling — memory is not automatically current.

Things you should be able to explain

• Why unbounded conversation history growth is a real production problem, and the trade-offs of the mitigation strategies.
• When structured memory is the right choice versus semantic memory.

Things you should be able to build

• A sliding-window-with-summarization context manager and a structured memory store with explicit staleness expiry.

Common mistakes

• Unbounded conversation history growth with no windowing/summarization strategy.
• Using semantic memory for facts that need reliable, exact recall.
• Treating remembered facts as permanently current.

Recommended next chapter

10-multimodal.md