3.6 — Hybrid Search, Reranking, and Query Rewriting

1. What is it?

These are the three most impactful production-grade upgrades to the minimal RAG pipeline from Part 3.5, each fixing a specific, well-understood failure mode of pure semantic (embedding) search:

• Hybrid search: combining semantic (vector) search with classical keyword search (BM25), so exact-term matches aren't missed.
• Reranking: a second, more accurate scoring pass over a larger initial candidate set, to improve precision of the final top-k.
• Query rewriting: transforming the user's raw question into a better retrieval query before searching, since what a user types is often not what should be searched for.

2. Why does it exist?

Part 3.5 already flagged that "just embed and retrieve" leaves real gaps. This chapter goes deep on exactly why each gap exists mechanistically and how each technique closes it — because understanding the mechanism of the failure is what tells you which fix actually applies to a given customer's retrieval problem, rather than throwing all three techniques at every problem indiscriminately (a real, common overengineering mistake).

3. What problem does it solve?

Each technique solves a distinct, separable problem:

• Hybrid search solves: pure semantic search sometimes misses exact terms that matter (a product SKU, a legal clause number, a person's name) because embeddings represent meaning, not exact tokens — two strings can be semantically similar while differing in the one specific token that actually matters for the query.
• Reranking solves: fast approximate retrieval (Part 3.4's ANN search) optimizes for speed over a huge candidate space, which means its top-k by embedding similarity alone isn't always the true top-k by actual relevance to the specific query's intent.
• Query rewriting solves: the literal text a user types is frequently a poor search query — pronouns referring to earlier conversation, vague phrasing, or a phrasing mismatch between how users ask questions and how the source documents are written.

4. How does it work internally?

Hybrid search: why embeddings alone miss exact terms

Recall from Part 2.5 that embedding models are trained to place semantically similar content near each other in vector space. This is exactly the wrong property for exact-term matching: an embedding model has no special mechanism to give this token's exact match extra weight — "invoice #INV-2024-0451" and "invoice #INV-2024-0452" are semantically nearly identical (both are clearly about an invoice reference), so their embeddings will be very close together, even though for a user searching for one specific invoice number, only an exact match is actually correct. A classical keyword algorithm like BM25 (a refinement of TF-IDF — scoring based on term frequency in a document, weighted down for terms that are common across all documents and weighted up for rare, distinctive terms, with length normalization) handles this precisely: it will strongly favor the document containing the exact string "INV-2024-0451" over one merely discussing invoices generally.

Hybrid search runs both search methods (embedding similarity and BM25) and fuses their results — typically using Reciprocal Rank Fusion (RRF), a simple, robust method that combines two ranked lists without needing the two systems' raw scores to be on comparable scales (embedding cosine similarity and BM25 scores are not directly comparable numbers):

RRF_score(document) = Σ  1 / (rank_in_list_i + k)     (k is a small constant, often 60)

Example: document D ranks #2 in semantic search, #15 in keyword search
RRF_score(D) = 1/(2+60) + 1/(15+60) = 0.0161 + 0.0133 = 0.0294

Documents that rank well in either list get a meaningful boost, and documents that rank well in both get the strongest boost — exactly the property you want: catch what semantic search finds, catch what keyword search finds, and prioritize agreement between them.

Reranking: trading a little latency for real precision

The insight behind reranking is that the fast embedding-similarity search (used because it must scale to potentially millions of candidates via ANN, Part 3.4) trades some accuracy for that speed. Reranking exploits this by deliberately over-fetching a larger, cheap-to-retrieve candidate set (e.g., retrieve the top 50 by embedding similarity instead of the top 5 you actually need) and then applying a slower, more accurate model — typically a cross-encoder — to just those 50 candidates, before selecting the final top 5 to actually use.

The key architectural difference between the embedding model used for initial retrieval and a cross-encoder reranker: the embedding model (a "bi-encoder") encodes the query and each document completely independently, so their similarity is just a distance between two independently-computed vectors — this is what makes it fast enough to compare against millions of pre-computed document vectors (the documents' vectors were computed once, offline; only the query needs encoding at search time). A cross-encoder, by contrast, takes the query and a specific candidate document together as a single input and directly outputs a relevance score for that specific pair — this lets it model interactions between the query and document text directly (does this specific document actually answer this specific question), which is more accurate but means it must be run once per query-document pair at query time — computationally infeasible to run against millions of documents, but entirely feasible against the 50 candidates a bi-encoder already narrowed things down to.

Bi-encoder (used for initial retrieval, Part 3.4):

encode(query)

encode(doc)precomputed, offline

compare via cosine similarityfast

Cross-encoder (used for reranking, only on the narrowed candidate set):

encode(query + doc) together

Single relevance scoremust run once per candidate — too slow for millions, fine for dozens

Query rewriting: three specific, common transformations

• Conversational query resolution: rewriting a follow-up question that depends on prior conversation turns ("what about for enterprise plans?") into a self-contained query ("what is the cancellation policy for enterprise plans?") before it's used for retrieval — because embedding "what about for enterprise plans?" alone, with no conversation context, produces a nearly meaningless retrieval query.
• HyDE (Hypothetical Document Embeddings): instead of embedding the user's question directly, ask an LLM to first generate a hypothetical, plausible-sounding answer to the question, and embed that for retrieval instead. This works because a hypothetical answer is often structurally and lexically closer to how the real answer appears in the source documents than the question phrasing is (a question "how do I reset my password" and a document that begins "To reset your password, navigate to..." are somewhat different in phrasing; a generated hypothetical answer like "To reset your password, you should..." is much closer in style to the real document).
• Query decomposition: breaking a complex, multi-part question ("compare our refund policy for physical goods versus digital goods") into multiple simpler sub-queries, retrieving for each separately, then combining the results — because a single embedding of the compound question may retrieve documents that partially match one part of the question at the expense of the other.

5. Simple mental model

Think of a librarian analogy extended from Part 3.4/3.5: hybrid search is using both "find books about this general topic" (semantic) and "find books containing this exact word" (keyword) simultaneously, and trusting a book more if both methods point to it. Reranking is first quickly grabbing 50 plausible books off the shelves (fast, approximate), then actually sitting down and reading the first page of each of those 50 carefully to pick the best 5 (slow, but only on the 50 you already narrowed down, not the entire library). Query rewriting is translating a vague question into the specific, well-formed question a librarian would actually need to hear to help you ("the thing from before" becomes "your enterprise plan's cancellation policy").

6. Real-world example

A software company's technical documentation search needs to handle both "how do I configure SSO" (semantic — many valid phrasings of the same underlying need) and "error code E4021" (exact-match — the error code string itself is the entire query, and semantic similarity to "error code E4020" is actively unhelpful). Pure semantic search handles the first well and the second poorly (retrieving documents about various error codes, not specifically E4021); pure keyword search handles the second well and the first poorly (missing documents that describe SSO configuration without using the literal words "configure SSO"). Hybrid search handles both classes of query well within the same pipeline, without needing to detect which "mode" a given query is in ahead of time.

7. Architecture diagram

User query+ conversation history

Query rewriting

Semantic searchembeddings, top 50 · Part 3.4

Keyword searchBM25, top 50

Fusion (RRF)merged, deduplicated

Rerankingcross-encoder scores each candidate

Final top-kto generation · Part 3.5

Two independent retrieval paths, fused by RRF, then reranked directly against the query.

```

8. Production considerations

• Not every query needs every stage. For a small, homogeneous corpus with consistent query patterns, hybrid search and reranking may add latency/complexity without measurable quality gain — validate each stage's contribution with evaluation (Part 8.1's precision@k/recall@k/MRR/NDCG metrics) before assuming all three are necessary. Concretely: run the same labeled query set through the pipeline with and without a given stage (reranking, query rewriting) and compare NDCG@k or recall@k before adding the stage's latency/cost permanently — "it feels more relevant" is not evidence a stage earned its keep.
• HyDE has a real failure mode worth knowing: if the LLM's hypothetical answer is confidently wrong or off-topic (itself a form of hallucination, Part 2.6), retrieval based on that hypothetical answer inherits the error — HyDE trades one class of retrieval-quality problem for a different, subtler one, and should be evaluated (Part 8.1's metrics), not assumed to be a strict improvement.
• Reranking latency is real and additive — a cross-encoder pass over 50 candidates, while much cheaper than running it against millions, still adds measurable time to the request path; benchmark this against your actual latency budget rather than assuming it's negligible.
• Cache query rewrites and, where safe, retrieval results for repeated/similar queries (Part 1.6) — query rewriting typically requires its own LLM call, adding to both latency and cost on every single request unless mitigated.

9. Common mistakes

• Adding hybrid search, reranking, and query rewriting all at once as a default "best practice" stack without measuring whether each one actually improves quality for the specific corpus and query patterns at hand — this adds real latency and cost for potentially little to no benefit if, say, the corpus rarely has exact-match-critical content.
• Using RRF or similar fusion incorrectly by trying to directly combine raw semantic-similarity scores and BM25 scores (which live on entirely different, incomparable scales) instead of fusing based on rank position.
• Assuming a reranker's cross-encoder score is itself an absolute measure of relevance rather than a relative ranking signal among the specific candidate set it was given.
• Skipping query rewriting for conversational interfaces, then being confused why follow-up questions retrieve poorly compared to standalone questions.

10. Security considerations

• Query rewriting that incorporates conversation history means earlier turns in a conversation (potentially including injected or adversarial content from an earlier retrieved document, Part 9.1) can influence what gets retrieved in a later turn — the injection surface isn't limited to the current message alone in a multi-turn RAG conversation.
• A reranking model, like any model in the pipeline, should be evaluated for whether it's trained/hosted in a way consistent with the data sensitivity requirements of the content it's scoring (e.g., a third-party hosted reranking API seeing your retrieved document snippets has the same data-handling implications as any other third-party API call with sensitive content, Part 9.6).

11. Performance considerations

• The added latency budget, roughly in order of typical cost: query rewriting (an extra LLM call) > reranking (a cross-encoder pass over dozens of candidates) > hybrid search fusion (cheap, just combining two already-fast lookups) — prioritize accordingly if latency is tight.
• Running semantic and keyword search concurrently (Part 1.1's asyncio.gather pattern) rather than sequentially avoids paying their latencies additively.

12. Cost considerations

• Query rewriting's extra LLM call is a real, recurring per-request cost — worth measuring its actual retrieval-quality lift against this cost, especially at high request volume (Part 7.10).
• Reranking model hosting/API costs scale with the number of candidates reranked per query — tune the initial candidate-set size (how many results you over-fetch before reranking) as a deliberate cost/quality trade-off, not an arbitrary large number.

13. When to use it

• Hybrid search: whenever the corpus contains content where exact terms matter (identifiers, codes, names, specific terminology) alongside content better served by semantic understanding — which describes most real enterprise document corpora.
• Reranking: whenever initial retrieval precision is measurably a problem (Part 8.1) and the added latency is acceptable for the use case.
• Query rewriting: whenever the interface is conversational (follow-up questions depend on context) or user queries are measurably vague/underspecified relative to how source documents are written.

14. When NOT to use it

• Small, homogeneous corpora with simple, consistent query patterns where the minimal RAG pipeline (Part 3.5) already performs well by evaluation — added complexity here is pure cost with no measured benefit.
• Extremely latency-sensitive use cases where even reranking's modest added latency isn't acceptable, and where the initial retrieval's precision, while imperfect, is good enough for the task.

15. Alternatives and trade-offs

Technique	Fixes	Cost
Hybrid search	Missed exact-term matches from pure semantic search	Modest — running two searches and fusing is cheap
Reranking	Imprecise top-k from fast approximate initial retrieval	Moderate — added latency for a cross-encoder pass
Query rewriting	Poor retrieval queries from vague/conversational user input	Moderate to high — typically an extra LLM call per request

16. Practical Python/code example

Reciprocal Rank Fusion, implemented directly to make the mechanism from section 4 concrete:

def reciprocal_rank_fusion(
    ranked_lists: list[list[str]], k: int = 60
) -> list[tuple[str, float]]:
    """
    Fuses multiple ranked lists of document IDs into a single ranking using RRF,
    which combines rank positions rather than raw, incomparable scores.

    Args:
        ranked_lists (list[list[str]]): Each inner list is one search method's
            results, ordered from most to least relevant.
        k (int): Fusion constant; higher values reduce the influence of high ranks
            from any single list, smoothing the combined ranking.

    Returns:
        list[tuple[str, float]]: Document IDs and their fused scores, sorted
            descending by score.
    """
    scores: dict[str, float] = {}
    for ranked_list in ranked_lists:
        for rank, doc_id in enumerate(ranked_list, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (rank + k)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

17. Production-quality example

A full hybrid-retrieve-then-rerank pipeline, combining sections 4's mechanisms end-to-end:

import asyncio


async def hybrid_retrieve_and_rerank(
    vector_store, bm25_index, cross_encoder, query: str, tenant_id: str,
    candidate_k: int = 50, final_k: int = 5,
) -> list[dict]:
    """
    Retrieves candidates via both semantic and keyword search concurrently, fuses
    them with RRF, then reranks the fused candidate set with a cross-encoder to
    produce the final, precise top-k for generation.

    Args:
        vector_store: Vector database client (Part 3.4), supporting tenant-filtered search.
        bm25_index: A keyword search index over the same tenant-scoped corpus.
        cross_encoder: A reranking model scoring (query, document) pairs directly.
        query (str): The (already rewritten, if applicable) retrieval query.
        tenant_id (str): Tenant scope enforced as a pre-filter on both search methods.
        candidate_k (int): Number of candidates to over-fetch before reranking.
        final_k (int): Number of results to return after reranking.

    Returns:
        list[dict]: The final, reranked top-k document chunks.
    """
    query_embedding = (await embed_texts(llm_client, [query]))[0]

    semantic_results, keyword_results = await asyncio.gather(
        vector_store.query(
            vector=query_embedding,
            filter={"tenant_id": {"$eq": tenant_id}},
            top_k=candidate_k,
        ),
        bm25_index.search(query=query, tenant_id=tenant_id, top_k=candidate_k),
    )

    fused = reciprocal_rank_fusion(
        [
            [r["id"] for r in semantic_results],
            [r["id"] for r in keyword_results],
        ]
    )
    candidate_ids = [doc_id for doc_id, _ in fused[:candidate_k]]
    candidates = await fetch_chunks_by_id(candidate_ids)

    rerank_scores = await cross_encoder.score_pairs(
        [(query, c["content"]) for c in candidates]
    )
    reranked = sorted(zip(candidates, rerank_scores), key=lambda x: x[1], reverse=True)
    return [chunk for chunk, _ in reranked[:final_k]]

18. Short exercise

A customer's RAG system handles general policy questions well but consistently fails to find the right document when users search by exact contract clause numbers (e.g., "Section 4.2.1(b)"). Using the mechanisms from section 4, explain precisely why pure semantic search struggles with this specific query pattern, and which single technique from this chapter you'd add first to address it most directly.

19. Interview questions

• Explain the architectural difference between a bi-encoder (used for initial retrieval) and a cross-encoder (used for reranking), and why that difference determines where each is used in the pipeline.
• Why can't you directly combine raw BM25 scores and embedding cosine-similarity scores, and how does Reciprocal Rank Fusion avoid this problem?
• What's the failure mode HyDE introduces, and why is it not a strict improvement over embedding the raw query?

20. FDE/customer scenario

Customer: "Search works great for general questions but completely misses when someone searches for a specific part number or error code."

This is close to a direct diagnostic match for the exact-term-matching gap in pure semantic search (section 4/6). The concrete, credible recommendation: add hybrid search combining the existing embedding-based retrieval with a BM25 keyword index over the same corpus, fused via RRF — not "switch to a better embedding model" (the wrong fix, since this isn't a semantic-quality problem, it's a structural limitation of semantic search for exact-token matching).

---

Key takeaways

• Hybrid search fixes exact-term-matching gaps that are structural to how embeddings represent meaning, not fixable by "better" embeddings alone.
• Reranking exploits the speed/accuracy trade-off between bi-encoders (fast, independent encoding) and cross-encoders (slow, joint encoding, more accurate) by only applying the expensive method to an already-narrowed candidate set.
• Query rewriting addresses the gap between how users phrase questions (especially conversationally) and how retrieval actually needs to be queried.
• None of these three techniques should be added by default — each should be justified by a measured, specific retrieval failure mode.

Things you should be able to explain

• Why Reciprocal Rank Fusion combines ranks, not raw scores.
• The bi-encoder vs. cross-encoder distinction and why it determines retrieval vs. reranking roles.

Things you should be able to build

• A concurrent hybrid-search-then-rerank retrieval pipeline.

Common mistakes

• Adding all three techniques by default without measuring their individual contribution.
• Directly combining incomparable raw scores from different search methods.

Recommended next chapter

07-agents-and-workflows.md