Dense vs Hybrid Retrieval: Measuring What Each Layer Adds

Agentic AI Stress Suite, Post 2 of 9. Reproducible benchmarks on where production RAG-to-agent stacks break: one harness, one corpus, negative results published alongside the positives.

Correction, 8 October 2026. The first version of this post reported that hybrid search lowered MRR from 0.739 to 0.690. That number came from a bug in the benchmark, not from BM25. The hybrid pipeline chunked the corpus twice, once per index, and the two copies of each chunk received different IDs. Reciprocal Rank Fusion merges by ID, so no chunk was ever credited by both retrievers. A second problem affected latency: the reranker's result cache stayed on across the three benchmark runs, so runs two and three measured cache hits, which the post explained as model warmup. The same ID bug sat upstream of the reranker, so the FlashRank numbers were wrong too, and the earlier conclusion that FlashRank trades recall for a better MRR is withdrawn. Both problems are fixed, all three configurations were rerun from clean indexes, and every number below comes from the corrected run. The dense-only Recall@5 and MRR reproduced exactly. The original artifact stays in the repository beside the corrected one.

Clarified, 11 October 2026. Hybrid's ranking gain is described as promising rather than settled, two explanations of dense retrieval are corrected, the artifact's uncommitted-changes flag is explained, and code links point to a fixed repository commit.

The short answer, measured on a 30-query tech-docs set: hybrid search (BM25 + dense + RRF) ranked the first relevant source higher than dense-only retrieval, lifting MRR from 0.739 to 0.794 and top-1 hits from 21 to 23 of 30, while Recall@5 slipped from 0.761 to 0.744. Adding FlashRank reranking on top did not help on this set: MRR fell back to 0.756, Recall@5 to 0.733, and every query paid about 0.7 seconds of cross-encoder inference. On 30 queries these gaps come from two to six queries changing rank, so read them as direction, not proof.

Article 1 measured query-side transformations on a dense-only baseline (Chroma + BGE-base-en-v1.5): the corrected results put dense retrieval at 0.838 to 0.847 Recall@5, HyDE at 0.854, and HyDE + decomposition at 0.877 on the 142-question tech-docs set, with none of the recall gains established. Article 2 moves to index-side and re-scoring techniques. Three failure modes are now in scope. First, keyword-critical queries: exact API terms like @Async annotation or HTTPException status_code=422 can bypass semantic search because dense embeddings compress token identity into geometric proximity. Second, cosine similarity is a weak ranking signal; a vector that lands close to the query centroid is not necessarily the passage most useful for generation. Third, document structure is invisible to dense-only pipelines: metadata fields, section boundaries, and PDF tables need ingestion-time handling before retrieval can use them.

This article introduces four techniques that target those failure modes: hybrid search (BM25 + dense via Reciprocal Rank Fusion), neural reranking (FlashRank cross-encoder, with Cohere as a hosted alternative), metadata-filtered retrieval (pre-filter vs. post-filter trade-offs), and sentence-aware / context-prefixed chunking (boundary-aware splits with section context). Three configurations are benchmarked in the verified run: dense-only, hybrid (BM25 + dense + RRF), and hybrid + FlashRank. The source artifact is results/data/article_02_benchmarks_corrected_2026-10-04.json, which also holds per-query rows for each configuration.

The methodology detail matters: benchmarks/run_article_02.py uses the shared BenchmarkRunner, and BenchmarkRunner.run_single_query() calls pipeline.query(...). Both Article 2 pipelines implement query() as retrieve plus answer generation. The quality metrics come from retrieved context_nodes, but the latency column is full query latency, not retrieval-only latency. The artifact's cost_per_1k_queries field is 0.0 for every measured configuration, but the pipelines do not expose token or dollar measurements to the runner, so this article does not report real cost.

Failure Mode 1: Keyword-Critical Queries

Consider this query against FastAPI documentation:

HTTPException status_code=422 raise detail

Dense search can return chunks about generic error handling patterns even when the query contains exact API terms. The failure mode is structural: 422, HTTPException, and status_code are token-level evidence, while dense retrieval ranks by vector proximity. A chunk about general HTTP error handling can be semantically close without containing the exact status code or parameter name.

This is the general problem: dense embeddings trade token identity for geometric proximity. For queries containing exact identifiers - error codes, API method signatures, library-specific parameter names, version strings - the trade destroys signal rather than capturing it.

Many teams skip hybrid search because "dense is good enough" - until they see their first production query with error codes or API signatures. At that point, the failure is hard to explain to a user who searched for an exact error message and got back a generic best-practices article.

BM25 does not have this problem.

BM25: Probabilistic Keyword Search

BM25 (Best Matching 25) treats retrieval as a probabilistic ranking problem, not a geometric one. Where dense search asks "which document is closest to the query in vector space?", BM25 asks "given this query, how likely is each document to be relevant?" - a fundamentally different framing.

The scoring function for a query Q and document D:

score(Q, D) = sum over query terms qi of:
    IDF(qi) * (f(qi, D) * (k1 + 1)) / (f(qi, D) + k1 * (1 - b + b * |D| / avgdl))

Three mechanisms: IDF weights rare terms higher (HTTPException is stronger evidence than return); k1 saturates term frequency so a term appearing 5× is only marginally better than 3× (no keyword stuffing); b normalizes for document length so longer documents don't dominate by accident. Dense search has none of these - rare tokens get averaged away during embedding, and there's no analog to document length normalization.

Here is how HybridSearchPipeline builds and queries the BM25 index:

def build_index(self, documents: list[Document]) -> tuple[VectorStoreIndex, BM25Okapi]:
    """
    Build both BM25 and dense indices.

    Teaching note: Dual indexing
    -----------------------------
    We build TWO indices from the same chunked documents:

    1. Dense index (Chroma):
       - Chunks → embeddings → vector store
       - Same process as NaiveRAG

    2. BM25 index (rank_bm25):
       - Chunks → tokenized text → inverted index
       - BM25Okapi calculates IDF statistics during initialization

    Both indices operate on the SAME chunks, ensuring:
    - Results can be merged (same chunk IDs)
    - Fair comparison between methods
    - Consistent chunking strategy
    """
    # Chunk once. RRF merges candidates by node_id, and the node parser
    # assigns a fresh UUID on every parse, so both indices must be built
    # from these exact node objects.
    self._chunks = self.node_parser.get_nodes_from_documents(documents)

    # Build the dense index from a clean collection, using the same chunks
    chroma_client = self._get_or_create_chroma_client()
    existing = {getattr(c, "name", c) for c in chroma_client.list_collections()}
    if self.collection_name in existing:
        chroma_client.delete_collection(name=self.collection_name)
    chroma_collection = chroma_client.get_or_create_collection(name=self.collection_name)
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
    storage_context = StorageContext.from_defaults(vector_store=vector_store)
    self._dense_index = VectorStoreIndex(
        nodes=self._chunks, storage_context=storage_context, show_progress=True
    )

    # Build BM25 index
    # Teaching note: BM25 requires tokenized text
    # We use simple whitespace + lowercase tokenization here
    # Production should use:
    # - Stemming (Porter stemmer)
    # - Stopword removal (language-specific)
    # - Subword tokenization for code (CamelCase splitting)
    tokenized_chunks = [self._tokenize(chunk.get_content()) for chunk in self._chunks]
    self._bm25 = BM25Okapi(tokenized_chunks, k1=self.k1, b=self.b)

    return self._dense_index, self._bm25

View on GitHub

The "chunk once" step is the part that is easy to get wrong, and the first version of this pipeline did. It built the dense index with VectorStoreIndex.from_documents(...) and then parsed the documents a second time for BM25. The text of the chunks was identical. Their IDs were not, because LlamaIndex gives every parsed node a new UUID. The fusion step then saw two disjoint candidate lists and could only interleave them. The unit test for fusion passed throughout, because it handed the same node objects to both lists. tests/unit/rag/test_hybrid_search_index_identity.py now builds both indexes for real and checks that a shared chunk keeps one identity and receives both rank contributions.

At query time, BM25 retrieval scores all chunks and returns the top-K by rank:

@traced_retrieval
def retrieve_bm25(self, query: str, top_k: int | None = None) -> list[tuple[TextNode, float]]:
    """
    Teaching note: BM25 scoring
    ---------------------------
    Key properties:
    - Rare terms score higher (IDF)
    - Repeated terms saturate (k1 controls saturation)
    - Longer docs penalized (b controls normalization)

    BM25 excels at:
    - Exact term matching
    - Rare technical terms
    - Acronyms and abbreviations
    - Code identifiers (function names, error codes)
    """
    k = top_k or self.top_k
    query_tokens = self._tokenize(query)
    scores = self._bm25.get_scores(query_tokens)
    top_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:k]
    return [(self._chunks[i], float(scores[i])) for i in top_indices]

View on GitHub

BM25 has no embedding cost - it operates on token statistics computed at index time. The useful distinction is qualitative but important: BM25 preserves exact token evidence, while dense retrieval preserves semantic neighborhood evidence. Hybrid search is attractive when production queries actually contain exact identifiers, and on the Article 2 smoke set the second signal improved ranking even though few queries contain such identifiers.

Reciprocal Rank Fusion

The challenge of merging BM25 and dense results is that their scores are incompatible. BM25 scores are roughly [0, 100] depending on corpus statistics; cosine similarity scores are [0, 1]. Normalizing each to [0, 1] is unreliable because the distributions differ per query: a BM25 score of 12 could be the maximum in one query and rank 8th in another.

Reciprocal Rank Fusion (RRF) sidesteps this entirely by discarding raw scores and fusing on rank position alone:

RRF_score(d) = sum over each retrieval method i of:
    weight_i / (k + rank_i(d))

Where k=60 is a bias constant that prevents early ranks from dominating. A document at rank 1 contributes weight / 61; a document at rank 2 contributes weight / 62 - a smooth reciprocal decay. Documents absent from one method's results contribute zero from that method.

RRF works for three reasons: scores from BM25 and cosine never interact (no normalization needed); the reciprocal decay is gentle enough that rank 1 doesn't dominate ranks 2-3; and a document appearing in both result sets accumulates contributions from both - consensus is rewarded without penalizing documents that only appear in one.

Here is the implementation:

def _reciprocal_rank_fusion(
    self,
    bm25_results: list[tuple[TextNode, float]],
    dense_results: list[NodeWithScore],
    top_k: int,
) -> list[NodeWithScore]:
    """
    Teaching note: Reciprocal Rank Fusion (RRF)
    --------------------------------------------
    RRF is a simple but effective rank aggregation method.

    Example:
    Document appears at:
    - BM25 rank 2 → score = 0.5 / (60 + 2) = 0.00806
    - Dense rank 5 → score = 0.5 / (60 + 5) = 0.00769
    - Total RRF = 0.01575

    Why RRF works:
    1. Rank-based (not score-based):
       - Robust to score magnitude differences
       - Robust to incompatible BM25 and dense score scales

    For most cases, RRF is the sweet spot: simple, robust, effective.
    """
    k = _RRF_K

    # Build rank maps from each retrieval method
    bm25_ranks: dict[str, int] = {}
    for rank, (node, _score) in enumerate(bm25_results, start=1):
        node_id = node.node_id if hasattr(node, "node_id") else node.id_
        bm25_ranks[node_id] = rank

    dense_ranks: dict[str, int] = {}
    for rank, node_with_score in enumerate(dense_results, start=1):
        dense_ranks[node_with_score.node.node_id] = rank

    # Calculate RRF scores across the union of both result sets
    rrf_scores: dict[str, float] = {}
    for node_id in set(bm25_ranks.keys()) | set(dense_ranks.keys()):
        score = 0.0
        if node_id in bm25_ranks:
            score += self.bm25_weight / (k + bm25_ranks[node_id])
        if node_id in dense_ranks:
            score += self.dense_weight / (k + dense_ranks[node_id])
        rrf_scores[node_id] = score

    sorted_node_ids = sorted(
        rrf_scores.keys(), key=lambda nid: rrf_scores[nid], reverse=True
    )

    # Rebuild NodeWithScore objects with RRF scores for downstream use
    node_map: dict[str, TextNode | Any] = {}
    for node, _score in bm25_results:
        node_id = node.node_id if hasattr(node, "node_id") else node.id_
        node_map[node_id] = node
    for node_with_score in dense_results:
        node_map[node_with_score.node.node_id] = node_with_score.node

    return [
        NodeWithScore(node=node_map[nid], score=rrf_scores[nid])
        for nid in sorted_node_ids[:top_k]
    ]

View on GitHub

The bm25_weight and dense_weight parameters (defaulting to 0.5 each) allow tuning influence without changing the formula's structure. Setting (0.7, 0.3) favors keyword precision for jargon-heavy technical documentation; (0.3, 0.7) favors semantic coverage for conversational queries.

LlamaIndex vs Haystack: Same Feature, Different Philosophy

Both frameworks support hybrid search, but their API philosophy differs sharply. LlamaIndex is data-centric: you build indices and compose retrieval operations programmatically. Haystack is pipeline-centric: you declare explicit components, connect them, and run the graph.

LlamaIndex (from HybridSearchPipeline.retrieve):

# Retrieve from both indices, then merge with RRF
bm25_results = self.retrieve_bm25(query, top_k=retrieval_k * 2)
dense_results = self.retrieve_dense(query, top_k=retrieval_k * 2)
merged_results = self._reciprocal_rank_fusion(bm25_results, dense_results, top_k=retrieval_k)

View on GitHub

Haystack (from HaystackHybridSearchPipeline.build_index):

# Declare components, connect them explicitly, then run
pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder(...))
pipeline.add_component("bm25_retriever", InMemoryBM25Retriever(document_store=self._document_store, top_k=self.top_k * 2))
pipeline.add_component("embedding_retriever", InMemoryEmbeddingRetriever(document_store=self._document_store, top_k=self.top_k * 2))
pipeline.add_component("joiner", DocumentJoiner(join_mode="reciprocal_rank_fusion", weights=[self.bm25_weight, self.dense_weight]))
pipeline.connect("text_embedder.embedding", "embedding_retriever.query_embedding")
pipeline.connect("bm25_retriever.documents", "joiner.documents")
pipeline.connect("embedding_retriever.documents", "joiner.documents")

# At query time:
result = pipeline.run({"text_embedder": {"text": query}, "bm25_retriever": {"query": query}})

View on GitHub

LlamaIndex keeps the retrieval logic close to the Python method. Haystack makes the dataflow explicit through named pipeline components. The repo unit tests exercise the Haystack wrapper's construction and retrieval shape, but Article 2 does not benchmark Haystack head-to-head with LlamaIndex, so this article treats the comparison as API design, not performance evidence.

Benchmark Results

Measured on 30 queries spanning the tech-docs corpus (Spring, React, FastAPI, Pydantic), 3 runs each, top_k=5. Numbers are read directly from results/data/article_02_benchmarks_corrected_2026-10-04.json:

Recall@5 here is the fraction of a query's expected source documents that appear among the top five retrieved chunks, averaged over queries. Hit@1 counts queries whose first retrieved chunk comes from an expected source. Retrieval is deterministic, so all three runs returned identical quality numbers. The +/- on latency is run-to-run spread on the same 30 queries. It says nothing about how the result would vary across a different set of queries.

Three findings matter.

Hybrid improves ranking. MRR moves from 0.739 to 0.794 and top-1 hits from 21 to 23. Per query, the first relevant source moved up for three queries and down for one. The likely mechanism is agreement: when a chunk is found by both retrievers, RRF adds both rank contributions and it rises above chunks that only one retriever liked. The artifact records final ranks, not which retriever contributed each chunk, so this is the expected behaviour of the fusion formula, not something measured per query.

Hybrid does not improve recall on this corpus. Recall@5 drops from 0.761 to 0.744. One query gained an expected source in its top five and two queries lost one. All three have two expected sources, so each change is worth half a query, and the net loss is half a query out of 30. BM25 surfaced few documents that dense retrieval had missed.

The latency cost is negligible. Full query latency moves from 2,278 ms to 2,310 ms, inside the run-to-run spread. The benchmark includes answer generation, which dominates. Absolute latency is about twice the first published run. The generator settings did not change, and the artifact does not record which provider in the fallback chain served each answer, so the likely cause is provider latency on the day. Compare latency within this table, not against other runs.

The caveat is workload shape. N=30 is a smoke set, not a broad evaluation set. The dataset has 19 filtered queries, 4 simple queries, 3 procedural queries, 3 comparison queries, and 1 cross_framework query. The filtered queries ran without their metadata filter in every configuration, because the filter path is not benchmarked. A two-query swing in top-1 hits and a one-query net swing in recall are not statistically established. One confound was measured after the fact. LlamaIndex embeds each chunk's metadata with its text, and here that includes the Chroma collection name, so the dense half of the hybrid pipeline is not byte-identical to dense-only. Rebuilding dense-only under the hybrid run's collection name moved Recall@5 from 0.761 to 0.750 and MRR from 0.739 to 0.750, and left top-1 hits at 21. So about one point of hybrid's MRR gain may come from the name, and none of its top-1 gain. The narrow summary is that hybrid ranked the first relevant source higher more often than it hurt, at a small recall cost and no real latency cost.

When to Use Hybrid Search

Hybrid is not a free upgrade, but on this small set it showed a promising ranking improvement. The measured result says: higher MRR, slightly lower recall, and about the same latency. The decision still depends on the query distribution.

Use hybrid when:

• Documentation contains precise technical terminology that the embedding model fails on: API names, error codes, version strings, parameter names, library-specific identifiers
• Users are developers querying by function signature or exact error message
• Code search is part of the retrieval scope (variable names, class names) - dense embeddings on code are notoriously weak
• The dense baseline has visible recall holes that look lexical (you can see the gold passage if you grep for the query terms verbatim, but the embedding model ranks it below the top-K)

Skip hybrid (or measure first) when:

• The consumer needs every relevant source in the top-K more than it needs the first one ranked high - on this benchmark hybrid cost 1.7 pp of Recall@5
• Corpus is conversational or informal (user reviews, forum posts, FAQs) - vocabulary is already informal, BM25 IDF weights add little signal
• Documents are very short, making IDF statistics weak
• The corpus is large enough that in-memory BM25 scoring becomes a latency problem; at that point an inverted-index backend is the right shape, not the in-memory rank_bm25 approach shown here

The right way to settle the argument is the same way this section settled it: run both configurations against your own query set, look at Recall@K and MRR side by side, and only ship the layer if the data agrees.

Failure Mode 2: Vector Match != Best Answer

Dense retrieval returns chunks that are semantically proximate to the query, not chunks that are most useful for answering the query. These are different things. A query "How do FastAPI background tasks work?" may retrieve chunks that mention background tasks frequently - but the most useful chunk is the one that explains the lifecycle, startup, and error handling, not the one that just uses the term most.

The root cause: embedding models optimize for semantic similarity, not answer quality. Cosine similarity measures geometric closeness, not relevance. The top-K list after dense retrieval is approximately right, but its ranking within K is weak.

Reranking addresses this with a second-pass neural model that scores query-document pairs jointly.

Cross-Encoders: Scoring Query-Document Pairs Together

Two architectures, fundamentally different trade-offs:

• Bi-encoder (the embedding model used in retrieval): encodes query and document independently, then compares their vectors via cosine similarity. Fast at query time because document vectors are pre-computed, so only the query is encoded, and an approximate nearest-neighbour index keeps search well below a full scan in practice. But there is no query-document interaction during encoding. The model never sees the query and the document at the same time; it can only compare the shapes their independent representations leave in vector space.

• Cross-encoder: takes the query and document together as a single input sequence. The transformer's attention mechanism runs across both, letting query tokens attend to document tokens and vice versa. This produces a direct relevance score rather than inferring relevance from geometric proximity. Slower - O(N) per query, no pre-indexing possible - and usually more accurate, though the measurement below is a case where it was not.

The key insight: cross-encoders are impractical for full corpus retrieval (scoring a million chunks per query is prohibitive), but they are exactly the right tool for re-scoring a small candidate set. The two-stage pipeline - bi-encoder retrieves top-20, cross-encoder reranks to top-5 - aims for the precision of cross-encoders at the latency cost of scoring only 20 documents.

FlashRank: Local Cross-Encoder Reranking

def rerank(
    self,
    query: str,
    documents: list[NodeWithScore] | list[dict[str, Any]],
    top_k: int = 5,
) -> tuple[list[NodeWithScore] | list[dict[str, Any]], float]:
    """Rerank documents using local FlashRank cross-encoder.

    Args:
        query: User query
        documents: Retrieved documents (LlamaIndex NodeWithScore or Haystack dicts)
        top_k: Number of top results to return after reranking

    Returns:
        Tuple of (reranked_documents, reranking_latency_ms)
    """
    if not documents:
        return documents, 0.0

    start = time.perf_counter()

    ranker = self._get_ranker()

    is_llamaindex = isinstance(documents[0], NodeWithScore)

    # Convert to FlashRank passage format
    passages = []
    for i, doc in enumerate(documents):
        if is_llamaindex:
            text = doc.node.get_content()
        else:
            text = doc.get("content", "")
        passages.append({"id": str(i), "text": text})

    from flashrank import RerankRequest

    rerank_request = RerankRequest(query=query, passages=passages)
    reranked = ranker.rerank(rerank_request)[:top_k]

    # Map back to original document format, replacing retrieval score with reranker score
    reranked_docs: list[NodeWithScore] | list[dict[str, Any]] = []
    for passage in reranked:
        idx = int(passage["id"])
        original_doc = documents[idx]

        if is_llamaindex:
            reranked_docs.append(
                NodeWithScore(
                    node=original_doc.node,
                    score=passage["score"],
                )
            )
        else:
            updated_doc = original_doc.copy()
            updated_doc["score"] = passage["score"]
            reranked_docs.append(updated_doc)

    end = time.perf_counter()
    latency_ms = (end - start) * 1000

    return reranked_docs, latency_ms

View on GitHub

The configured model is ms-marco-MiniLM-L-12-v2, wrapped by FlashRank in an ONNX runtime. The ranker is lazy-loaded: FlashRankReranker._get_ranker() creates the Ranker on first use and reuses it on subsequent calls.

The measured two-stage pipeline uses settings.reranking_top_k=20. For a final top_k=5, HybridSearchPipeline.retrieve() fetches 40 BM25 candidates and 40 dense candidates, fuses them down to 20 with RRF, reranks those 20 with FlashRank, and returns 5. The candidate multiplier is a code setting, not a universal rule.

Cohere Rerank: Same Interface, No Local Measurement

src/rag/reranking.py also implements CohereReranker behind the same create_reranker() factory used for FlashRank. The implementation extracts text from each candidate, calls client.rerank(...), and maps Cohere's returned indices back onto the original NodeWithScore objects.

The Article 2 verification run does not include Cohere. benchmarks/run_article_02.py only appends the hybrid_cohere configuration when --cohere is passed and COHERE_API_KEY is configured. This run did not pass --cohere, so there is no Cohere latency, quality, or cost number in the artifact. Cohere is an implementation option in the source tree, not a measured row in this article.

Benchmark Results

Measured on the same 30 queries, 3 runs each, top_k=5. The full numbers are in results/data/article_02_benchmarks_corrected_2026-10-04.json:

The result is more interesting than the standard "reranking always helps" story. Three findings:

FlashRank did not improve on hybrid here. Against plain hybrid, MRR drops from 0.794 to 0.756, top-1 hits from 23 to 21, and Recall@5 from 0.744 to 0.733. Per query, the first relevant source moved up for two queries and down for four. Against dense-only the picture is mixed: MRR is 1.7 points higher, Recall@5 is 2.8 points lower, and top-1 hits are the same 21. The cross-encoder reorders the 20 fused candidates on its own judgment, and on this set that judgment was not better than the fused rank it replaced.

MRR is not top-1. MRR rises when a relevant source moves from rank 5 to rank 2, which does nothing for a consumer that reads only the first result. That is why the table reports Hit@1 directly. If the downstream consumer shows a single citation or a single snippet, Hit@1 is the number to read, and on it FlashRank ties dense-only and trails hybrid.

The reranker costs about 0.7 seconds on every query. The runner now times the rerank call in isolation, on the same 20 fused candidates per query, on an Apple M4 Pro CPU:

The first version of this post saw run latencies of 1,913 / 1,024 / 1,086 ms and concluded that the first run paid a warmup cost. That reading was wrong. HybridSearchPipeline wraps its reranker in a result cache keyed on the query and candidate IDs, and the benchmark repeats the same 30 queries each run, so runs two and three were cache hits. Model load accounts for roughly 240 ms of one call. Warm inference still costs about 0.72 seconds on CPU for 20 chunks of roughly 500 tokens. The cache hit across runs was possible because each configuration builds its index once, so candidate IDs stay the same from run to run. The benchmark now disables the cache, and the latency row above is uncached. The full-query difference between hybrid and hybrid + FlashRank is 842 ms. The rerank call explains 724 ms of it, and the reranking path also retrieves 40 candidates per retriever instead of 10. Caching still helps a real workload that repeats queries. It does not make the reranker cheap.

Caveat (worth reading before generalising). N=30 again, and the hybrid -> FlashRank pipeline is the full chain - dense retrieval -> BM25 retrieval -> RRF merge -> cross-encoder rerank. A two-query difference in top-1 hits is direction, not proof. A benchmark configuration that runs dense -> FlashRank directly would isolate the reranker's own contribution; it is not measured here. Neither is a larger cross-encoder, which may rank better than the MiniLM model at a higher latency. FlashRank's 512-token input limit covers the query and the passage together, so the tail of the longest chunks is truncated before scoring, which may cost some ranking quality.

When to Use Reranking

The decision is not "always on". It is a function of which metric the downstream consumer actually optimises for.

Use FlashRank when:

• You have measured a Hit@1 or MRR gain over your un-reranked pipeline on your own queries - this benchmark did not find one over plain hybrid
• The candidate set from the upstream retriever already has high recall and the remaining problem is ordering
• A fully local reranker is a hard requirement for privacy, air-gapped deployments, or cost control
• Latency budget can absorb about 0.7 seconds of CPU inference per uncached query for 20 candidates, the cost measured on this run

Skip FlashRank (or test first) when:

• Downstream consumer reads the full top-K and fuses multiple chunks; on this benchmark adding FlashRank to hybrid lowered Recall@5 by a further 1.1 pp
• Very low retrieval latency is a hard requirement
• Upstream retrieval is already weak on recall - reranking cannot recover documents that retrieval never surfaced; fix retrieval first

Evaluate Cohere separately when a hosted reranker is acceptable and retrieval quality is valuable enough to justify sending query-document pairs to an external service. Article 2 does not provide the evidence for that decision; it only shows that the source tree has the backend and that the benchmark runner can include it behind --cohere.

Failure Mode 3: Invisible Structure

Dense retrieval treats every document as a flat bag of text. Metadata fields (framework, version, date, doc_type), section boundaries, and structured content like tables are stripped at ingest and never retrieved. This creates two distinct problems:

• Precision failure from missing constraints: "What are the Spring APIs released after version 5.3?" should filter by framework=spring and version>=5.3 before searching. Without metadata, dense search retrieves Spring 4.x and Spring 6.x results equally - the version constraint is simply invisible.

• Quality failure from poor chunking: A 500-token fixed chunk that splits mid-sentence, or a table extracted as a flat string, loses the structural context that makes the content interpretable.

Metadata Filtering: Adding Structured Constraints

During indexing, MetadataExtractor attaches structured fields to every chunk from two sources: the file path convention (datasets/tech_docs/{framework}/{filename}.md yields framework) and content headers (first # heading yields section, filename keywords yield doc_type). This avoids a separate metadata database while enabling rich filtering - the trade-off is that quality depends on consistent naming conventions.

The fields attached per chunk:

metadata: dict[str, Any] = {
    "source": str(doc_path),      # Full path for debugging
    "framework": "fastapi",        # Extracted from path component
    "section": "Exception Handling",  # From first H1/H2 heading
    "doc_type": "reference",       # Classified from filename + content keywords
}

View on GitHub

Filters are built using composable FilterCondition objects and combined with AND/OR/NOT logic:

# "FastAPI or Pydantic docs, after 2024, excluding tutorials"
filter_ = MetadataFilter.and_(
    MetadataFilter.or_(
        MetadataFilter([FilterCondition("framework", FilterOperator.EQ, "fastapi")]),
        MetadataFilter([FilterCondition("framework", FilterOperator.EQ, "pydantic")]),
    ),
    MetadataFilter.not_(
        MetadataFilter([FilterCondition("doc_type", FilterOperator.EQ, "tutorial")])
    ),
)

View on GitHub

The composable static methods (and_, or_, not_) nest freely because each returns a MetadataFilter - they operate on the same interface they receive. The internal _sub_filters list and _sub_logic field separate composed filters from leaf conditions, so the matches method handles both cases uniformly.

The two application points are pre_filter and post_filter, which target different stages of the pipeline:

Pre-filter - restrict the candidate pool before vector search:

def pre_filter(
    nodes: list[TextNode],
    filter_: MetadataFilter,
) -> list[TextNode]:
    # Teaching note: Pre-filtering trade-offs
    # ----------------------------------------
    # Pre-filtering modifies the candidate pool:
    # - Faster retrieval: Fewer documents to search through
    # - Risk: May eliminate relevant documents tagged with unexpected metadata
    # - Use when: Metadata is reliable and query intent is clear
    #
    # Example scenario:
    # Query: "How does FastAPI handle authentication?"
    # Pre-filter: framework="fastapi"
    # Result: Only FastAPI docs searched -> faster, but misses Spring comparison docs
    return [node for node in nodes if filter_.matches(node.metadata)]

View on GitHub

Post-filter - apply constraints after retrieval, preserving ranking:

def post_filter(
    results: list[NodeWithScore],
    filter_: MetadataFilter,
) -> list[NodeWithScore]:
    # Teaching note: Post-filtering trade-offs
    # -----------------------------------------
    # Post-filtering narrows already-ranked results:
    # - Preserves ranking quality from hybrid search + reranking
    # - Risk: May return fewer than top-K results
    # - Use when: Ranking quality matters more than speed
    #
    # Example scenario:
    # Query: "What are dependency injection patterns?"
    # Retrieve: top-20 from all frameworks
    # Post-filter: framework="spring"
    # Result: Only Spring docs from ranked results -> quality ranking preserved
    return [r for r in results if filter_.matches(r.node.metadata)]

View on GitHub

The trade-off is concrete:

• Pre-filter: Reduces the search space before BM25 and dense retrieval run. Faster. Lower recall risk when the filter is precise and the filtered corpus is still large enough to return K results. Use when the query intent is unambiguous (user explicitly selects "FastAPI" in a UI dropdown, for example).
• Post-filter: Retrieves unrestricted top-K, then filters. Preserves the ranking quality from hybrid search and reranking - the relative order of surviving results reflects the full-corpus ranking, not a pre-filtered subset. The risk is getting fewer than K results when the filter is strict and the retrieved set has few matching documents.

Pre-filter is a strong candidate when metadata is reliable (path-derived, version tags, framework labels assigned at ingest). Post-filter is safer during early experimentation when metadata quality is still uncertain - you retrieve more and rank it before applying the constraint.

Metadata Filtering: Benchmark Status

Pre-filter and post-filter are implemented in src/rag/metadata_filter.py and exercised in unit tests, but they are not benchmarked as configurations in results/data/article_02_benchmarks_corrected_2026-10-04.json. The real-run path in benchmarks/run_article_02.py skips hybrid_metadata_prefilter and hybrid_metadata_postfilter with the message that they need per-query filter plumbing.

Why pre-filter is still a plausible default for framework-tagged corpora, on first principles:

• The metadata is path-derived, not inferred (no classification model, no LLM extraction step), so false positives from the filter itself are essentially zero
• Each per-framework subcorpus in datasets/tech_docs/ still contains hundreds of chunks - large enough that BM25 and dense retrieval both have a meaningful candidate pool to rank inside the filtered set
• Pre-filter should shrink the work both retrievers do per query, but the size and quality effect still need a benchmark row before publication as a measured claim

The measurement status is: this repo does not yet have a metadata-filter row in the Article 2 verification JSON. Until the runner has per-query filter plumbing, treat the prose above as a design argument, not a measured result.

Chunking Strategies: When Fixed Chunks Fail

Fixed chunking is simple and predictable, but it splits content at arbitrary boundaries. Article 1's SentenceSplitter uses 500 tokens with 50 overlap; the FixedChunker shown here uses 500 characters with 50 overlap and attempts to break at sentence boundaries within the last 20% of each chunk:

# Try to break at sentence boundary within last 20% of chunk
if end < len(text):
    boundary_zone_start = max(start, end - self.chunk_size // 5)
    best_break = end

    for sep in [". ", ".\n", "\n\n", "\n", " "]:
        pos = text.rfind(sep, boundary_zone_start, end)
        if pos != -1:
            best_break = pos + len(sep)
            break

    end = best_break

View on GitHub

This boundary heuristic helps, but it cannot prevent a function signature at character 480 from being split from its docstring at character 510 when no sentence separator falls in the boundary zone. The three strategies address this at different levels.

Fixed chunking (baseline): 500 characters, 50 character overlap. Splits at natural sentence boundaries when possible, arbitrary character positions otherwise. Simple and fast, suitable for uniform prose. Fails on code files and reference docs with dense, heterogeneous structure.

Sentence-aware chunking: Groups consecutive sentences up to a size limit, never splitting mid-sentence. The class is named SemanticChunker, but it does not detect topic changes with embeddings, so "sentence-aware" is the accurate label. From SemanticChunker.chunk:

for sentence in sentences:
    sentence_len = len(sentence)

    # If adding this sentence exceeds max, flush current chunk
    if current_length + sentence_len > self.max_chunk_size and current_sentences:
        chunk_text = " ".join(current_sentences)
        chunks.append(Chunk(text=chunk_text, ...))
        current_sentences = []
        current_length = 0

    current_sentences.append(sentence)
    current_length += sentence_len + 1  # +1 for space

# Merge tiny trailing chunk with previous if possible
if len(chunk_text) < self.min_chunk_size and chunks:
    merged_text = last_chunk.text + " " + chunk_text
    chunks[-1] = Chunk(text=merged_text, ...)

View on GitHub

The implementation uses regex sentence splitting ((?<=[.!?])\s+) rather than a full embedding-based similarity comparison. As the module docstring notes, full semantic chunking with per-sentence embeddings and a cosine-similarity threshold would be more accurate at detecting topic boundaries - but it introduces a sentence-level embedding dependency at index time and requires threshold tuning per domain (0.5-0.8 typical). The simplified version respects sentence boundaries, avoids mid-sentence splits, and merges tiny trailing chunks to prevent sub-100-character noise chunks.

Context-prefixed chunking: Prepends document context to every chunk before embedding. The class is named LateChunker, but this is not late chunking in the sense Jina describes, where a long-context model embeds the whole document and token embeddings are pooled per chunk afterwards. Here the context is a text prefix added before a normal embedding call. From LateChunker.chunk:

def chunk(self, text: str, metadata: dict[str, Any]) -> list[Chunk]:
    context = self._extract_context(text)  # First heading or first paragraph
    base_chunks = self._fixed_chunker.chunk(text, metadata)

    prefix = f"[{context}] "
    contextualized = []
    for chunk in base_chunks:
        contextualized.append(
            Chunk(
                text=prefix + chunk.text,
                metadata={**chunk.metadata, "context_prefix": context},
                ...
            )
        )
    return contextualized

View on GitHub

_extract_context scans the first five lines of the document for a heading (#) or the first non-empty prose line (capped at 200 characters), producing a prefix like [FastAPI - Exception Handling] . Every chunk's embedding then carries the document-level context - the chunk about "It also supports..." knows it is in the FastAPI exception handling section, not just a generic exception paragraph. The module docstring notes this is the simple version inspired by Anthropic's "Contextual Retrieval" paper (2024); the advanced version generates per-document summaries via LLM, which is more accurate for documents without clear headings but adds cost at index time.

PDF Tables: The Hidden Retrieval Problem

Standard PDF text extraction reads left-to-right, top-to-bottom, and can destroy table structure. Column names and row values can become plain adjacent tokens. Dense search may still retrieve the chunk on semantic similarity, but BM25 loses the structural relationship between a header, a row label, and a cell value.

PDFProcessor addresses this using PyMuPDF's built-in find_tables():

for page_num, page in enumerate(doc):
    page_text = page.get_text("text")
    if page_text.strip():
        text_parts.append(page_text)

    try:
        page_tables = page.find_tables()
        for table in page_tables:
            table_data = table.extract()
            if table_data:
                tables.append({"page": page_num + 1, "rows": table_data})
    except Exception as exc:
        logger.debug("Table extraction skipped on page %d: %s", page_num + 1, exc)

View on GitHub

Extracted tables are returned as row-major lists of lists in the current implementation. Article 2 does not benchmark PDF table retrieval, so the safe claim is limited to code behavior: PDFProcessor.extract() attempts table extraction with PyMuPDF and preserves any extracted rows for downstream processing.

When to Use Each Chunking Strategy

Full Pipeline: Measured Stack

The runner measures three configurations end-to-end, and the reranking results table above holds all three rows. Read together:

No row wins on every metric. Dense-only has the best Recall@5, by half a query. Hybrid has the best MRR and the most top-1 hits, for almost no latency. Hybrid + FlashRank is behind hybrid on all three quality metrics and adds 842 ms to the full query, 724 ms of it in the rerank call.

Each layer has a clear job, but the job has to match the workload. Hybrid improved ranking on the measured smoke set and cost a little recall. FlashRank did not improve on hybrid and was the only layer with a real latency cost. Pre-filter is still implementation-only in this benchmark. None of these layers should be enabled without measuring the actual query distribution.

Decision Framework

Picking a stack is a measurement decision, not a recipe. The framework is short:

# Retrieval stack selection - measured on YOUR query distribution

baseline = dense_only(BGE_base_en_v1_5)
# Run baseline first. Record Recall@K and MRR. This is the bar.

IF corpus has exact technical identifiers AND baseline has visible lexical recall holes:
    EVALUATE hybrid_bm25_dense_rrf  # measured here: -1.7 pp recall, +5.5 pp MRR, +2 top-1 hits, 1.01x latency
    # Ship only if it beats baseline on the metric your downstream consumer reads

IF downstream consumer reads top-1 (single citation, button-press answer):
    MEASURE Hit@1, not only MRR
    EVALUATE hybrid_plus_flashrank  # measured here vs hybrid: -1.1 pp recall, -3.8 pp MRR, -2 top-1 hits, +0.84 s full query
    # Ship only if Hit@1 improves on your queries; it did not here

IF downstream consumer reads full top-K (long-form generation fusing chunks):
    PREFER higher-recall configuration even if MRR is slightly lower
    # No measured winner here: dense-only leads hybrid by half a query out of 30

IF documents have reliable, path-derived structured metadata:
    EVALUATE pre-filter (not benchmarked here yet - see Metadata section)

IF a hosted reranker is acceptable:
    EVALUATE Cohere rerank in place of FlashRank (not benchmarked here)

Observability

Understanding where latency comes from requires tracing each pipeline stage independently. The retrieval methods in src/rag/hybrid_search.py use @traced_retrieval: retrieve_bm25(), retrieve_dense(), and the top-level retrieve() path are all traced. Answer generation uses @traced_generation. FlashRank latency is returned by the reranker and used inside HybridSearchPipeline.retrieve(), but the benchmark JSON only reports full query latency; it does not break out BM25, dense, RRF, reranking, and generation as separate metric columns.

Access the full trace waterfall at http://localhost:6006 after starting the Phoenix container via docker-compose -f infra/docker-compose.yml up -d.

What to take away

The conventional story says hybrid search wins, FlashRank wins on top, and quality rises as layers are added. The measured story on the 30-query smoke set agrees with the first part and not the rest:

• Dense-only is not a baseline you should assume you can beat on every metric. BGE-base-en-v1.5 with Chroma got Recall@5 = 0.761 and MRR = 0.739 on this corpus. Hybrid (BM25 + RRF) raised MRR to 0.794 and lowered Recall@5 to 0.744. Hybrid + FlashRank was behind plain hybrid on both. Neither dominated the baseline on both metrics.

• Hybrid is a query-distribution decision, not a default. It pays off when the corpus contains exact identifiers (function signatures, error codes, version strings) that the embedding model collapses, and when users query with those identifiers verbatim. On this natural-language tech-docs query set it still improved ranking, most likely because a chunk that both retrievers agree on rises above one that only dense retrieval liked. The first version of this post concluded that BM25 added rank noise to a correct dense ranking. The corrected run does not support that.

• Reranking has to earn its latency. FlashRank cost about 0.7 seconds per query and did not beat plain hybrid on MRR, Hit@1, or Recall@5 on this set. MRR alone would have hidden that for a top-1 consumer, so measure Hit@1 directly when only the first result is read. Time the reranker without its result cache, or repeated benchmark queries will make it look free.

• Cost is not measured here. BenchmarkRunner calls pipeline.query(...), so the benchmark is not retrieval-only. The artifact stores cost_per_1k_queries: 0.0, but the pipelines do not expose token or dollar metrics to the runner.

• The concepts hold whether or not a given measurement agrees. BM25's IDF + length normalisation, RRF's score-free rank fusion, cross-encoder joint attention, pre-filter vs post-filter trade-offs, the document context prefix on each chunk - these are the right concepts to reach for. What the runner shows is that whether a given technique helps on your corpus is an empirical question, not a foregone conclusion. Run both configurations, look at Recall@K and MRR side by side, ship only the layer the data agrees with.

What is not in this article's measured set: Cohere reranker, metadata pre/post-filter, context-prefixed chunking, sentence-aware chunking. These are implemented and covered by focused unit tests, but the real Article 2 runner only measured dense_only, hybrid_bm25_dense, and hybrid_flashrank.

Article 3 covers evaluation infrastructure - RAGAS, DeepEval, LLM-as-Judge, A/B testing - which is the right way to settle "does X help on our corpus" with a methodology that scales past a 30-query smoke set.

Reproducibility. The measured numbers in this article are the JSON in results/data/article_02_benchmarks_corrected_2026-10-04.json. It records the repository commit, library versions, model identifiers, per-query results, and the isolated reranker timings. Its git_dirty: true flag is accurate: the run used commit 60e6228 plus the uncommitted Part 2 fixes, which were committed unchanged as 081e743. The runner drops and rebuilds each Chroma collection and disables the reranker cache. The original, pre-correction results are preserved as results/data/article_02_benchmarks_verify.json. To re-run the benchmark:

./scripts/start_dev_stack.sh
uv run python benchmarks/run_article_02.py --runs 3 --top-k 5 --output results/data/article_02_benchmarks_rerun.json

If your numbers diverge from the ones quoted above, the JSON your runner produces is the source of truth for your hardware and your corpus, not the prose here.

References

Published papers, separate from the benchmark numbers above:

• Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval. BM25.
• Cormack, G. V., Clarke, C. L., & Buettcher, S. (2009). Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR 2009. Reciprocal Rank Fusion.
• Nogueira, R., & Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085. Cross-encoder reranking.
• Anthropic (2024). Introducing Contextual Retrieval. The inspiration for the context prefix.
• Gao, Y., Xiong, Y., et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997.

Technical documentation:

• FlashRank: repository.
• Cohere Rerank: API reference.
• rank_bm25: Python library.
• LlamaIndex: BM25 retriever and hybrid search.
• Haystack: retrievers API.
• Arize Phoenix: documentation.