2.5 — Embeddings

1. What is it?

An embedding is a dense vector of numbers (typically hundreds to thousands of dimensions) that represents the meaning of a piece of content (text, image, audio) such that semantically similar content produces vectors that are close together in that vector space. Embeddings are the foundational technology underneath RAG, semantic search, recommendation, clustering, and deduplication in AI systems.

2. Why does it exist?

Computers can't directly compare "meaning" — they compare numbers. Embeddings exist to translate the fuzzy, human notion of semantic similarity ("these two sentences are about the same thing, even though they share almost no words") into something mathematically comparable: distance or angle between two vectors. This solves the core limitation of classical keyword search — matching exact terms — by matching meaning instead.

3. What problem does it solve?

It solves "find things that mean the same thing, not just things that use the same words." A customer searching "how do I get my money back" should find a document titled "Refund Policy" even though it shares zero exact keywords — keyword search fails here; embedding-based semantic search succeeds because both phrases map to nearby points in the embedding space.

4. How does it work internally?

From transformer representations to a single vector

An embedding model (often itself a transformer, frequently encoder-only or a specially-trained variant) processes input text through its layers, producing a representation for each token (Part 2.4). To get one vector representing the whole input, the model pools these token representations (commonly using a special aggregate token, or averaging) into a single fixed-size vector. Embedding models are typically trained with a contrastive objective: pull embeddings of semantically similar pairs closer together, push embeddings of dissimilar pairs farther apart — this training objective is specifically what makes the resulting space useful for similarity search, not an incidental property of "any transformer output."

Measuring similarity

cosine_similarity(A, B) = (A · B) / (‖A‖ · ‖B‖)

Cosine similarity measures the angle between two vectors, ignoring magnitude — the standard metric for text embedding similarity, because it captures "pointing in the same semantic direction" regardless of vector length. Euclidean (L2) distance and dot product are the other common choices; which one is "correct" depends on how the specific embedding model was trained (some models are trained/normalized specifically for cosine similarity; check the model's documentation rather than assuming).

"refund policy"        ●
                          ╲  small angle → high cosine similarity
"how do I get my money back" ●

"quarterly sales report" ────────────● large angle → low cosine similarity

Dimensionality and what it trades off

Higher-dimensional embeddings (e.g., 3072 vs. 384 dimensions) can capture more nuance but cost more to store and compare, and don't automatically mean "better" for a given task — this is a real trade-off to evaluate (Part 3.4), not a spec to maximize blindly.

5. Simple mental model

Embeddings turn meaning into geography. Imagine every sentence, document, or image as a physical location on a map, where "meaning-similar" content is placed near each other and "meaning-different" content is placed far apart. Semantic search becomes "find the nearest neighbors to this point on the map" — a well-defined, computable operation, instead of the fuzzy human task of "read everything and decide what's relevant."

6. Real-world example

An insurance company's claims documents include phrases like "vehicle collision," "auto accident," and "car crash" used inconsistently across decades of records. A keyword search for "car accident" misses documents that only say "vehicle collision" — but an embedding-based search finds both, because the embedding model has learned (from its training data) that these phrases are used in near-identical contexts and therefore mean nearly the same thing.

7. Architecture diagram

Raw textquery or document

Embedding modeltransformer, contrastively trained

Dense vectore.g. 1536-dim

Vector indexPart 2.5 → Part 3.4

Both queries and documents pass through the same embedding model into the same vector space.

8. Production considerations

• Never mix embeddings from different models in the same index — different embedding models produce vector spaces that are not comparable to each other; distances between a vector from model A and a vector from model B are meaningless, even if the dimensionality happens to match.
• Re-embedding is required if you change embedding models — swapping providers/model versions means the entire corpus must be re-embedded, not just new documents; plan for this operationally (it's a real migration, not a config change).
• Chunking strategy (how you split documents before embedding) materially affects retrieval quality — this is covered in depth in Part 3.5 (RAG), because embeddings are rarely used standalone in production; they're almost always one stage of a retrieval pipeline.
• Batch embedding calls where possible — most embedding APIs support batched input, which is far more efficient than one call per document.

9. Common mistakes

• Assuming cosine similarity is universally correct without checking whether a given embedding model was trained/normalized for it.
• Mixing embedding model versions in one index after a silent provider upgrade, causing degraded and confusing retrieval quality that's hard to diagnose because "it used to work."
• Embedding entire large documents as a single vector, losing granularity — a single vector can't represent a 50-page document's many distinct topics well (chunking exists to address this, Part 3.5).
• Treating embedding similarity as a proxy for truth or correctness — high similarity means "topically related," not "factually consistent" or "safe to trust," a distinction that matters for hallucination/faithfulness evaluation (Part 8.2).

10. Security considerations

• Embeddings of sensitive text are not automatically anonymized or safe to store/share freely — research has shown embeddings can, under some conditions, be partially inverted to recover information about the original text (an active, evolving research area). Treat an embedding of PII as itself sensitive data requiring the same access controls as the original text (Part 9.6).
• Cross-tenant embedding index contamination is a real, serious multi-tenant risk — the same tenant-isolation discipline from Part 1.5 (pgvector RLS) applies to any vector store, regardless of which one you use.

11. Performance considerations

• Exact nearest-neighbor search is O(n) per query — fine for small corpora, prohibitively slow at scale. Approximate Nearest Neighbor (ANN) indexes (HNSW, IVF) trade a small amount of recall accuracy for large speed gains, and are necessary once a corpus grows past roughly tens of thousands of vectors (exact threshold depends on latency requirements and hardware).
• Embedding dimensionality directly affects both storage size and comparison speed — larger isn't free.

12. Cost considerations

• Embedding API calls have a per-token cost, generally far lower than generation calls, but non-trivial at the scale of embedding an entire enterprise document corpus (millions of chunks) — worth estimating explicitly before committing to a re-embedding migration.
• Storage cost for high-dimensional vectors at scale (millions of documents × thousands of dimensions × 4 bytes/float) is a real infrastructure line item, not negligible at enterprise document-corpus scale.

13. When to use it

Semantic/similarity-based search, RAG retrieval, deduplication, clustering, recommendation — any task where "meaning-similarity" (not exact match) is the relevant notion of relevance.

14. When NOT to use it

• Exact-match lookups (an order ID, an account number) — a SQL query is faster, cheaper, and more precise than embedding-based similarity search for exact structured lookups (Part 1.4).
• Tasks needing precise, auditable, rule-based matching (e.g., exact regulatory keyword compliance checks) where "approximately similar" isn't good enough and a false negative/positive has real consequences.

15. Alternatives and trade-offs

Approach	Good for	Weak point
Embedding-based semantic search	Meaning-based matching, synonym/paraphrase robustness	Can miss exact-term precision; not inherently explainable
Keyword/full-text search (BM25, etc.)	Exact term matching, explainable ranking, cheap	Misses semantic paraphrases entirely
Hybrid search (both, combined)	Captures both exact-term and semantic matches	More complex pipeline, needs a fusion/reranking step (Part 3.6)

16. Practical Python/code example

import numpy as np


def cosine_similarity(a: list[float], b: list[float]) -> float:
    """
    Computes cosine similarity between two embedding vectors.

    Args:
        a (list[float]): First embedding vector.
        b (list[float]): Second embedding vector.

    Returns:
        float: Cosine similarity in [-1, 1], where 1 means identical direction.
    """
    a_arr, b_arr = np.array(a), np.array(b)
    return float(np.dot(a_arr, b_arr) / (np.linalg.norm(a_arr) * np.linalg.norm(b_arr)))


async def embed_texts(client, texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
    """
    Embeds a batch of texts using the provider's embedding API.

    Args:
        client: An initialized async embedding API client.
        texts (list[str]): Texts to embed.
        model (str): Embedding model identifier — must match the model used for
            anything this will be compared against.

    Returns:
        list[list[float]]: One embedding vector per input text, in order.
    """
    response = await client.embeddings.create(model=model, input=texts)
    return [item.embedding for item in response.data]

17. Production-quality example

A guarded embedding-index writer that refuses to mix embedding models in the same collection — encoding the "never mix embedding spaces" rule from section 9 as an enforced check, not just a documented warning:

import logging

logger = logging.getLogger("embedding_index")


class EmbeddingModelMismatchError(Exception):
    """Raised when an attempt is made to write embeddings from a different model into an index."""


class GuardedEmbeddingIndex:
    """Wraps a vector store write path, enforcing a single consistent embedding model per index."""

    def __init__(self, store, expected_model: str, expected_dim: int):
        """
        Args:
            store: The underlying vector store client.
            expected_model (str): The only embedding model identifier this index accepts.
            expected_dim (int): The expected embedding dimensionality for that model.
        """
        self._store = store
        self._expected_model = expected_model
        self._expected_dim = expected_dim

    async def upsert(self, chunk_id: str, embedding: list[float], model_used: str, metadata: dict) -> None:
        """
        Writes an embedding to the index, refusing writes from a mismatched model.

        Args:
            chunk_id (str): Unique identifier for the chunk.
            embedding (list[float]): The embedding vector to store.
            model_used (str): Which embedding model produced this vector.
            metadata (dict): Additional metadata to store alongside the vector.

        Raises:
            EmbeddingModelMismatchError: If model_used or dimensionality doesn't match
                the index's expected configuration.
        """
        if model_used != self._expected_model:
            raise EmbeddingModelMismatchError(
                f"index expects '{self._expected_model}', got '{model_used}'"
            )
        if len(embedding) != self._expected_dim:
            raise EmbeddingModelMismatchError(
                f"expected dim {self._expected_dim}, got {len(embedding)}"
            )
        await self._store.upsert(chunk_id, embedding, metadata)

18. Short exercise

Your team silently upgraded from text-embedding-3-small to a newer embedding model, and retrieval quality got noticeably worse afterward, but only for documents indexed before the upgrade. Diagnose the likely cause and describe the fix.

19. Interview questions

• Explain why embeddings from two different models generally can't be compared to each other, even at the same dimensionality.
• What is a contrastive training objective, and why does it matter for producing a useful similarity space?
• Why is exact-match SQL usually better than embedding similarity search for looking up an account by ID?

20. FDE/customer scenario

Customer: "Our search is finding almost nothing relevant even though the right documents are clearly in the system."

A first diagnostic question: was the corpus re-embedded consistently, or does the index contain a mix of embeddings from different model versions (e.g., from an incremental ingestion pipeline that upgraded models mid-stream without a full re-index)? This single, easy-to-miss operational mistake is one of the most common real causes of "semantic search suddenly got bad" incidents in production RAG systems.

---

Key takeaways

• Embeddings map meaning to geometry — similar meaning, nearby vectors — via contrastively trained models.
• Never mix embedding models/versions in one index; re-embedding the whole corpus is required on any model change.
• Embedding similarity means topical relatedness, not factual correctness.

Things you should be able to explain

• Why cosine similarity works for measuring semantic similarity.
• Why embeddings from different models aren't comparable.

Things you should be able to build

• An embedding-index writer that enforces model/dimension consistency.

Common mistakes

• Mixing embedding model versions in one index.
• Treating similarity as truth/correctness.
• Estimating embedding storage/compute cost as negligible at enterprise scale.

Recommended next chapter

06-llms.md