Appearance
3.4 — Vector Databases
1. What is it?
A vector database is a data store purpose-built to index and search over embedding vectors (Part 2.5) at scale — finding the k nearest vectors to a query vector, fast, even across millions or billions of entries, typically while also supporting metadata filtering (find the nearest vectors among only the documents belonging to tenant X). Examples include Pinecone, Weaviate, Qdrant, Milvus, Chroma, and Postgres with the pgvector extension (Part 1.5).
2. Why does it exist?
Part 2.5 established that semantic search means "find the nearest vectors in embedding space." The naive way to do this — compute the distance from your query vector to every vector in your collection, then sort — is an exact answer, but it's O(n) per query: linear in the number of stored vectors. At a few thousand documents this is fine (milliseconds). At tens of millions of document chunks (a realistic enterprise RAG corpus), a linear scan becomes seconds or minutes per query — completely unusable for an interactive application. Vector databases exist specifically to make "find the nearest neighbors" fast at scale, using approximate algorithms that trade a small amount of accuracy for orders-of-magnitude speed gains.
3. What problem does it solve?
It solves the retrieval half of RAG (Part 3.5): given a query embedding, return the most semantically relevant stored chunks, fast enough for an interactive user-facing request (typically tens of milliseconds), filtered correctly by whatever metadata constraints matter (tenant, document type, date range, access permissions), at whatever scale the customer's actual document corpus requires (thousands to billions of chunks).
4. How does it work internally?
Why exact nearest-neighbor search doesn't scale, precisely
For a query vector, exact k-NN search computes the distance (Part 2.5 — cosine, L2, dot product) between the query and every single stored vector, then finds the k smallest. This is exactly correct, but its cost grows linearly with the number of stored vectors and linearly with the vector's dimensionality (each distance computation touches every dimension). At 10 million vectors of 1536 dimensions each, that's roughly 15 billion multiply-add operations per single query — technically parallelizable, but still a real, scaling bottleneck that gets worse as the corpus grows, with no way around it while staying exact.
Approximate Nearest Neighbor (ANN) — the core trick
ANN algorithms accept a small, tunable amount of inaccuracy (occasionally missing the true single nearest neighbor, returning the 2nd or 3rd closest instead) in exchange for searching a much smaller fraction of the data per query. The two dominant algorithm families you'll encounter:
HNSW (Hierarchical Navigable Small World) — builds a multi-layered graph structure at index time, where each vector is a node connected to a handful of its nearest neighbors, with sparser "highway" layers on top for fast long-distance jumps and denser layers at the bottom for fine-grained local search.
Layer 2 (sparse, "highways"): A ─────────────── F
│ │
Layer 1 (medium density): A ─── B ─── D ─── E ─── F
│ │
Layer 0 (dense, all vectors): A─B─C─D─E─F─G─H─I─J─K─LA search starts at the top, sparse layer, quickly narrows to the right neighborhood by hopping toward the query vector, then descends layer by layer, doing progressively finer-grained search only in the relevant region — never touching the vast majority of the dataset. This is conceptually similar to how you'd navigate a map: start with highways to get to the right city, then use local streets to find the exact address, rather than checking every street in the country.
IVF (Inverted File Index) — partitions the vector space into clusters ("Voronoi cells") at index time using a clustering algorithm (like k-means, Part 2.1's unsupervised learning); at query time, it identifies the few clusters closest to the query vector and only searches within those clusters, skipping the rest entirely.
Full vector space, partitioned into clusters at index time:
┌─────────┬─────────┬─────────┐
│ Cluster1│ Cluster2│ Cluster3│ ← query vector falls near Cluster2's center
├─────────┼─────────┼─────────┤ → only Cluster2 (and maybe its neighbors)
│ Cluster4│ Cluster5│ Cluster6│ get searched; the rest are skipped entirely
└─────────┴─────────┴─────────┘HNSW generally gives better recall (accuracy) at a given speed than IVF, at the cost of slower index-building and higher memory usage (the graph structure itself takes space); IVF tends to be more memory-efficient and faster to build, at somewhat lower recall for the same speed budget. Both expose tunable parameters that trade recall against speed/memory (HNSW's ef_construction/ef_search, m; IVF's number of clusters/nprobe) — the right values depend on your actual corpus size, latency budget, and acceptable recall, and should be benchmarked against your real data, not guessed.
Quantization and truncatable embeddings — a third lever, orthogonal to ANN algorithm choice
HNSW vs. IVF above is a choice about how the nearest-neighbor search itself is structured. Independent of that choice, there's a third lever for controlling memory and cost at scale: reducing the size of each stored vector itself, rather than changing how the index searches over them.
- Scalar quantization reduces each vector's per-dimension precision (e.g., from 32-bit floats down to 8-bit integers), shrinking memory footprint roughly 4x with a small, generally acceptable recall cost — the most common, lowest-risk quantization option.
- Binary quantization goes further, reducing each dimension to a single bit (roughly a 32x memory reduction versus float32), which meaningfully speeds up distance computation (comparing bits is far cheaper than comparing floats) at the cost of a larger, more workload-dependent recall hit — usually paired with a reranking pass (Part 3.6) using the original full-precision vectors on a small candidate set, to recover most of the lost precision cheaply.
- Product quantization splits each vector into sub-vectors and quantizes each sub-vector independently against a learned codebook, achieving high compression ratios with a more tunable, and often better, recall/memory trade-off than naive scalar quantization at the same compression level, at the cost of more complex index-building.
Separately, some newer embedding models (trained with a technique often called Matryoshka representation learning) produce embeddings that remain meaningfully useful when truncated — the first N dimensions of, say, a 1536-dimension embedding are themselves a valid, if slightly less accurate, lower-dimensional embedding, unlike an arbitrarily-truncated embedding from a model not trained this way (which typically becomes close to useless). This gives you a cheap, tunable memory/recall knob at storage and retrieval time: store and search over a truncated, smaller vector for most queries, with the option to re-rank a small top-k using the full-dimensional embedding when precision matters more. Verify whether your specific embedding model was actually trained to support this against its current documentation before relying on truncation — it's a training-time property of that specific model, not a universal property of embeddings in general.
Quantization and truncation are both fully compatible with, and independent of, the HNSW-vs-IVF choice above — they compress what's stored and compared, not how the search traverses the index — and both should be evaluated the way section 8 argues for ANN parameters generally: benchmark recall against your actual data and query patterns before committing to an aggressive compression level in production.
Metadata filtering — the part that's easy to get subtly wrong
Real queries are almost never pure similarity search — they're "find the most similar documents among only the ones this tenant/user is allowed to see." There are two fundamentally different ways this filtering can be implemented, and which one a given database/index uses has real, sometimes surprising performance and correctness implications:
- Pre-filtering: apply the metadata filter first (narrow down to the allowed documents), then do similarity search only within that filtered set. Correct, but can be slow if the filter doesn't align well with how the ANN index is structured, sometimes forcing something closer to a full scan of the filtered subset.
- Post-filtering: run the ANN similarity search first (get the top-k globally), then discard results that don't match the metadata filter. Fast, but has a serious correctness pitfall: if the top-k globally-similar results happen to all belong to other tenants, post-filtering can return fewer than k results (or zero) even though highly relevant results exist for the correct tenant, just outside the initial top-k window.
This distinction is not academic — it's the direct, common root cause of "our RAG system sometimes returns nothing relevant even though we know the right document is in there" bugs in real multi-tenant systems. Know which strategy your chosen vector database uses (and whether it's configurable) before you're debugging this in production.
5. Simple mental model
A vector database is a library that's been organized for someone standing at the door with only a rough description of the book they want, not a librarian who reads every book's title one by one. HNSW is like a library organized into: a top floor with just directional signs to wings ("fiction," "history"), each wing with signs to specific shelves, each shelf with the actual books in order — you narrow down floor by floor instead of walking every aisle. IVF is like the library being pre-sorted into themed rooms; you figure out which one or two rooms are relevant to your rough description, and only search those rooms.
6. Real-world example
A multi-tenant legal-document AI platform stores document chunks from thousands of law firms in one shared vector index, distinguished by a firm_id metadata field. Early in development, they used post-filtering for simplicity, and it worked fine in testing with a handful of firms. In production, once a few large firms had heavily populated the index, smaller firms started reporting that searches returned few or no results even for documents they knew existed — because the top-k globally similar chunks were dominated by the large firms' much larger document volume, silently squeezing out the correct results for smaller tenants before the post-filter even got a chance to see them. Switching to pre-filtering (or a hybrid approach with a much larger candidate pool before filtering) fixed it — a direct, concrete illustration of why the filtering-strategy detail in section 4 is not a minor implementation nuance.
7. Architecture diagram
User query
Embedding modelPart 2.5
Vector Database
ANN index + metadata filterHNSW/IVF · tenant_id, doc_type, ...
RAG generationPart 3.5
8. Production considerations
- Choose pre-filtering (or a database that handles filtered ANN search correctly) for any multi-tenant or access-controlled system. Post-filtering is a real, common production bug waiting to happen at scale (section 6).
- Re-embedding and re-indexing are operationally real events, not config changes. Changing embedding models (Part 2.5) means rebuilding the entire index from scratch — plan for this (a blue-green index swap is common: build the new index fully, then cut over) rather than trying to migrate in place.
- Benchmark recall against your actual data, not vendor marketing numbers — ANN parameter tuning (HNSW's
ef_search, IVF'snprobe) that gives 99% recall on one dataset's distribution can behave differently on yours. - Decide your consistency requirements explicitly. Some vector databases have eventual consistency for newly-inserted vectors (a document just added might not be immediately searchable) — if your application needs "upload a document, immediately search for it," verify this behavior for your chosen database rather than assuming synchronous consistency.
9. Common mistakes
- Using post-filtering in a multi-tenant system without realizing it can silently starve smaller tenants of correct results (section 6).
- Not planning for the operational reality of re-embedding an entire corpus when swapping embedding models.
- Over-indexing on a single "best" vector database without evaluating whether your actual scale (a few thousand documents) even needs a specialized system rather than
pgvectorin an existing Postgres instance (Part 1.5's cross-reference on this exact trade-off). - Ignoring index-build time as a real operational cost — HNSW index construction on a very large corpus can take substantial time and resources, relevant for planning re-indexing windows.
10. Security considerations
- Tenant isolation enforcement at the vector-search layer (not just application-layer awareness of tenants) is critical — a filtering bug here is a direct data-leakage incident, not a minor quality issue (Part 9.6, Part 11.4).
- Vector databases are a real attack surface like any other data store — authentication, network isolation, and encryption at rest apply exactly as they would to any database holding potentially sensitive derived data (recall from Part 2.5 that embeddings themselves can leak information about their source text).
11. Performance considerations
- ANN parameters are a genuine speed/recall/memory three-way trade-off — there's no universally "correct" setting; it must be tuned against your real latency budget and acceptable miss rate.
- Query latency typically grows (slowly, sub-linearly with good ANN tuning) as the index grows — monitor this over time rather than assuming performance measured at launch scale holds indefinitely as the corpus grows.
12. Cost considerations
- Specialized vector databases (managed, hosted) typically price by vector count and/or query volume — at very large scale this is a real, ongoing infrastructure cost to model explicitly against
pgvector's "use infrastructure you already pay for" alternative (Part 1.5). - Memory requirements for HNSW indexes (holding the graph structure, often largely in RAM for speed) scale with corpus size and can become a meaningful infrastructure cost driver at large scale.
- Quantization (scalar/binary/product) and Matryoshka-style truncated embeddings (section 4) are a direct, often underused lever for reducing the memory cost driver above — a scalar-quantized index can shrink memory footprint roughly 4x, and binary quantization further still, both independent of which ANN algorithm (HNSW/IVF) you're using. The trade-off is a measurable, benchmarkable recall cost, not a free lunch — validate against your own data before adopting an aggressive compression level purely for cost reasons.
13. When to use it
Semantic search over any corpus large enough that exact linear scan is too slow for your latency requirements (a rough, benchmarkable rule of thumb: tens of thousands of vectors and up, though the exact threshold depends on your latency budget and hardware) — most production RAG systems, at almost any real enterprise document scale, need this.
14. When NOT to use it
- Small corpora (a few thousand documents or fewer) where exact linear scan is fast enough and the operational simplicity of not running a specialized index outweighs the marginal speed gain.
- Already covered by Part 1.5: when the customer's constraints (existing infrastructure, compliance approval process) favor
pgvectorin an existing Postgres instance over introducing a new specialized system, and scale doesn't yet demand otherwise.
15. Alternatives and trade-offs
| Option | Good for | Weak point |
|---|---|---|
| Dedicated vector DB (Pinecone, Weaviate, Qdrant, Milvus) | Purpose-built, often best-in-class recall/speed at very large scale, rich filtering features | New system to operate/secure; another source of truth |
| pgvector (Part 1.5) | Unified with existing transactional data, inherits existing security posture (RLS) | Scales vector search less far than dedicated systems at extreme volume |
| In-memory/exact search (small scale) | Simplicity, no approximation error | Doesn't scale past a modest corpus size |
16. Practical Python/code example
Using a vector database client abstractly (the pattern generalizes across most providers' SDKs) to make the pre-filtering discipline from section 4 concrete:
python
async def search_with_pre_filtering(
vector_store, query_embedding: list[float], tenant_id: str, k: int = 5
) -> list[dict]:
"""
Performs a similarity search scoped to a tenant via pre-filtering, ensuring the
ANN search only ever considers vectors this tenant is allowed to see.
Args:
vector_store: A vector database client supporting filtered search.
query_embedding (list[float]): The query's embedding vector.
tenant_id (str): Tenant identifier, applied as a hard filter BEFORE the ANN
search runs, not applied by discarding results afterward.
k (int): Number of results to return.
Returns:
list[dict]: The top-k results, guaranteed to belong only to this tenant.
"""
return await vector_store.query(
vector=query_embedding,
filter={"tenant_id": {"$eq": tenant_id}}, # applied as a pre-filter, not post-hoc
top_k=k,
)17. Production-quality example
A blue-green re-indexing helper, addressing the "re-embedding is a real operational event" production consideration from section 8:
python
import logging
logger = logging.getLogger("vector_reindex")
async def blue_green_reindex(
vector_store_client,
old_index_name: str,
new_index_name: str,
documents,
embed_fn,
batch_size: int = 100,
) -> None:
"""
Builds a new vector index fully before cutting traffic over, avoiding any window
where searches run against a partially-migrated, inconsistent index.
Args:
vector_store_client: Client for the vector database, supporting named indexes.
old_index_name (str): The currently live index, left untouched until cutover.
new_index_name (str): The new index to build fully before switching traffic.
documents: An iterable of document chunks to (re-)embed and index.
embed_fn: An async function embedding a batch of texts (Part 2.5).
batch_size (int): Number of documents to embed/index per batch.
"""
await vector_store_client.create_index(new_index_name)
batch = []
async for doc in documents:
batch.append(doc)
if len(batch) >= batch_size:
embeddings = await embed_fn([d.text for d in batch])
await vector_store_client.upsert(
index_name=new_index_name,
items=[
{"id": d.id, "embedding": e, "metadata": d.metadata}
for d, e in zip(batch, embeddings)
],
)
batch = []
if batch:
embeddings = await embed_fn([d.text for d in batch])
await vector_store_client.upsert(
index_name=new_index_name,
items=[
{"id": d.id, "embedding": e, "metadata": d.metadata}
for d, e in zip(batch, embeddings)
],
)
logger.info("new index '%s' fully built; ready for traffic cutover", new_index_name)
# Traffic cutover (e.g., updating a config value your app reads for the active
# index name) happens as a separate, deliberate, instantaneous step — never
# gradually, to avoid a window of inconsistent results.18. Short exercise
A customer's multi-tenant RAG system uses post-filtering and has one very large tenant (10 million chunks) and many small tenants (a few hundred chunks each). Explain, step by step, why a small tenant's search is more likely to return empty or irrelevant results than the large tenant's search, using the mechanism from section 4 — and describe the specific fix.
19. Interview questions
- Explain HNSW's layered graph structure and why it makes search sub-linear in the number of stored vectors.
- What's the difference between pre-filtering and post-filtering in a metadata-filtered vector search, and why does the choice matter for multi-tenant correctness?
- Why can't you simply swap an embedding model in a live vector index without a full re-index?
20. FDE/customer scenario
Customer: "Our smaller customers are complaining that AI search finds nothing, but our biggest customer says it works great."
This is close to a diagnostic script at this point given section 6: ask specifically whether metadata filtering is applied before or after the similarity search, and check whether the underlying vector database's filtering strategy is even configurable. This exact symptom pattern (works for high-volume tenants, silently fails for low-volume ones) is close to a fingerprint for the post-filtering bug, and being able to name the likely cause before even opening the codebase is a credibility-building moment in a customer engagement.
Key takeaways
- ANN algorithms (HNSW, IVF) trade a small amount of exactness for the speed needed at real enterprise scale — understand the trade-off, don't treat it as a black box.
- Pre-filtering vs. post-filtering metadata is a correctness-critical decision in any multi-tenant or access-controlled vector search system, not an implementation detail.
- Changing embedding models requires a full re-index — plan for this as a real migration.
Things you should be able to explain
- Why exact k-NN search doesn't scale, and how HNSW/IVF address it.
- The pre-filtering vs. post-filtering correctness trap in multi-tenant systems.
Things you should be able to build
- A tenant-scoped, pre-filtered vector search function and a blue-green re-indexing pipeline.
Common mistakes
- Post-filtering in a multi-tenant system.
- Treating a vector database swap or embedding model change as a config change rather than a full migration.
Recommended next chapter
05-rag.md