Agentic AI Stress Suite, Post 1 of 9. Reproducible benchmarks on where production RAG-to-agent stacks break: one harness, one corpus, negative results published alongside the positives.
Correction, 8 October 2026. The first version of this post reported a naive baseline of 0.487 Recall@5 on 300 queries, and a HyDE lift of 2.2 points. Both came from a broken question set. The file held 300 rows but only 152 distinct questions, two of which had no matching document in the corpus and were dropped. 97 of its source labels pointed at documents that had been renamed or had never existed, and in 60 rows every label was broken, so those rows scored zero for every pipeline. The set was deduplicated and relabelled, then labelled a second time independently. The two passes agreed exactly on 96 of 150 questions and shared at least one document on 137, and disagreements were settled from document content without looking at retrieval results. Eight questions about Pydantic validation pitfalls were removed because no document has that topic as its main subject, leaving 142. The comparison also gained a like-for-like baseline, because the original naive baseline used a differently built index from the transformed configurations. The corrected result: HyDE's recall lift shrinks to 1.6 points, which is not established, and every transform lowers ranking. The original artifact stays in the repository beside the corrected one.
Retrieval-Augmented Generation (RAG) has become the default pattern for grounding LLMs in external knowledge. The basic shape is simple enough: embed documents, store vectors, retrieve top-K matches, generate an answer. The usual argument for going further is that naive top-K struggles on multi-hop questions, on queries that share no vocabulary with the source documents, and on anything where semantic similarity alone is the wrong signal.
The practical question is not whether advanced retrieval sounds smarter than top-K. The practical question is whether a technique improves retrieval quality enough to justify its latency, operational complexity, and additional LLM calls on the workload in front of you.
The benchmark behind this article compares retrieval variants on the same 142-question dataset: two dense baselines that differ in how the corpus is indexed, HyDE, query decomposition, and the HyDE + decomposition composite. The results are less dramatic than a synthetic demo. Against its like-for-like baseline, HyDE adds 1.6 points of Recall@5 at 1.46x the latency and lowers ranking slightly. Decomposition leaves recall unchanged and lowers top-1 hits from 120 to 100. The composite adds the most recall, 3.9 points, at 3.4x the latency, and also lowers ranking. None of the recall gains is established on 142 questions, and the clearest result is a negative one: decomposition hurts ranking.
Graph RAG belongs in the discussion, but not in the same table. The repository implementation returns entity paths, while the benchmark metrics compare retrieved document paths. The honest treatment is architectural analysis plus a clear statement that Graph RAG needs a path-oriented metric before it can be compared numerically.
Benchmark methodology
The benchmark compares the baseline and advanced RAG pipelines against the same 142-question technical-docs workload. The important implementation boundary is small: benchmarks/run_article_01.py orchestrates the run, src/core/benchmarking.py calculates Recall@5/MRR/latency, src/rag/naive_rag.py implements the baseline, and src/rag/advanced_rag.py implements HyDE and decomposition.
The benchmark runs three repetitions at top_k=5, which matters for latency; the quality numbers below are single-run, for reasons given with the results table, and the relevant unit suite passed with 79 tests. One methodology detail matters for interpretation: BenchmarkRunner.run_single_query() times pipeline.query(...), and both Article 1 RAG pipelines implement query() as retrieve plus answer generation. The quality metrics come from retrieved context_nodes, but latency is full query latency, not retrieval-only latency.
The metric definitions are intentionally simple. Recall@5 is the fraction of a query's expected source documents that appear among the top five retrieved chunks, averaged over queries. For a query with two expected sources, retrieving one scores 0.5, so this is not the same as a hit rate. MRR asks how early the first expected source appears. Latency measures the full query path used by the benchmark runner. Cost is not reported because the benchmark artifact does not contain real token or dollar measurements.
The corpus and the questions are both synthetic, and that limits what the numbers mean. The corpus is 200 technical documents on FastAPI, Pydantic, React, and Spring. A small early set was adapted from the official documentation. Most were written by an LLM in that style from a topic list, and were checked for length and format, not for technical accuracy. The 142 questions and their reference answers were also LLM-generated. The source labels come from two independent passes that read document titles, section headings and content, with the second pass blind to the first and to all retrieval results. A document is labelled as a source when its main subject is what the question asks about, with at most three per question: 75 questions have one source, 44 have two, and 23 have three. Some topics, such as React hooks in general, have no single document, so those labels are judgment calls. The question categories used for slices below, such as the 34 comparison questions, are the ones the generator assigned.
There are two dense baselines. NaiveRAGPipeline indexes every file in the corpus into a Chroma collection. AdvancedRAGPipeline, which implements HyDE and decomposition, indexes the markdown files in memory. The embedding model and chunk settings are the same. A transform has to be compared with the baseline that shares its index, so the comparison table reports both.
One reproducibility caveat applies to every number in this post. LlamaIndex embeds a document's metadata together with each chunk's text, and here that metadata includes the absolute path of the file on disk. The naive pipeline also embeds the Chroma collection name. Retrieval therefore depends on where the repository is checked out and what the collection is called. Rebuilding the same indexes from a different path moved Recall@5 by 1.3 points for the in-memory baseline and 2.5 points for the naive one. That is as large as most of the effects reported below, so a rerun from another directory will not reproduce these numbers exactly. Rebuilding from the original path reproduced the stored naive retrievals for all 142 questions.
The Naive RAG Baseline
The baseline is the floor: chunk, embed, store, retrieve, generate. The naive pipeline is the same RAG shape most teams build first, and that makes it the right comparison point for every more complex variant.
The important source path is src/rag/naive_rag.py. NaiveRAGPipeline.__init__() wires the settings-backed LLM client, BGE embeddings, Chroma, and the SentenceSplitter. retrieve() is deliberately plain LlamaIndex vector retrieval:
k = top_k or self.top_k
retriever = self._index.as_retriever(similarity_top_k=k)
nodes = retriever.retrieve(query)
return nodes
The benchmark measures query(), not only retrieve(), because the source pipeline retrieves chunks and then calls generation before returning context_nodes:
context_nodes = self.retrieve(query_str, top_k=top_k)
answer = self.generate(query_str, context_nodes)
return {
"answer": answer,
"context_nodes": context_nodes,
"metadata": {
"query": query_str,
"num_retrieved": len(context_nodes),
"top_k": top_k or self.top_k,
"collection": self.collection_name,
},
}
Chunking strategy: fixed vs semantic
Chunk size is the hyperparameter that quietly decides everything else. Too small and you fragment context until retrieval is just noise. Too large and the relevant sentence drowns in the irrelevant 400 tokens around it (and your prompt budget evaporates). For technical documentation, 500 tokens (~375 words, one or two paragraphs) is a defensible default.
The benchmark uses SentenceSplitter with 50-token overlap so a sentence straddling a chunk boundary is not lost on either side. The splitter respects sentence structure while targeting 500 tokens per chunk. The result is better than naive character splitting, not as strong as proper semantic chunking, and appropriate for a baseline.
A note on parent-document and late chunking. The baseline only ever returns the chunk itself, which can leave the LLM staring at a fragment without the surrounding paragraph. Parent-document retrieval returns the full document a chunk came from. Late chunking runs a long span of text through the embedding model first and pools the token embeddings into per-chunk vectors afterwards, so each chunk vector carries context from the text around it. Both produce better generations, both cost you more tokens and latency. I do not use either here because the goal is to measure the floor, not the ceiling.
Benchmark results
142 questions across the tech-docs corpus (Spring, React, FastAPI, Pydantic), from results/data/article_01_benchmarks_corrected_2026-10-05.json:
The reading is simple. On average, 85% of each question's expected sources are in the top five. At least one expected source is in the top five for 137 of 142 questions, and the first retrieved chunk comes from an expected source for 117. Retrieval for this baseline is deterministic, so three runs give identical quality numbers. The ± on latency is run-to-run spread on the same questions, because the benchmark times the full query() path including answer generation.
That is a strong baseline, and it should be read with the corpus in mind. Most documents cover one topic and the questions were generated from the same topic vocabulary, so a dense retriever has an easy job. The first version of this post explained a 0.487 baseline as the product of a hard, mixed workload. It was the product of broken labels. The weakest slice is the 34 comparison questions, at 0.779 Recall@5, because each needs a source for both sides.
When naive RAG fails
The failures these techniques target come in three shapes, and each one motivates a technique later in this article. They are illustrations of the risk, not counts from this dataset:
- Vocabulary mismatch. "How do I make Spring async?" and a doc paragraph titled "asynchronous processing in Spring Framework" mean the same thing and embed in different neighborhoods.
- Multi-hop reasoning. "Compare FastAPI and Spring async patterns" needs at least two retrievals, one per framework. A single dense query has to land both. On the 34 comparison questions in this set the dense baselines found 73% to 78% of the expected sources, so it often does.
- Ambiguity. "React hooks" matches JavaScript React, Java reactive programming, and a few unrelated things involving the word "hook".
HyDE: hypothetical document embeddings
HyDE attacks the vocabulary-mismatch problem head-on. Instead of embedding the user's query, you ask the LLM to write a hypothetical document snippet in the style of the corpus, and embed that.
How HyDE is meant to work
The query is short and casual: fastapi dependency injection example.
One benchmark question asks for an example of FastAPI's Depends() function. The relevant corpus section is written like framework documentation: it introduces dependency injection, then shows a get_db() dependency and a route parameter declared as db: Session = Depends(get_db). HyDE is useful when the user query and the documentation answer the same question in different registers.
The concept is the same, but the neighborhoods in vector space differ. HyDE closes the gap by manufacturing text that looks like the documents you want to retrieve. The implementation is in src/rag/advanced_rag.py:
@traced_generation
def _generate_hypothetical_document(
self,
query: str,
correlation_id: str | None = None,
) -> str:
prompt = f"""You are generating a hypothetical technical document snippet.
Query: {query}
Write 2-3 sentences of technical documentation that would answer this query.
Use formal technical language and specific terminology.
Do not say "this document explains" - just write the content directly.
Hypothetical document:"""
response = self.llm_client.generate(
prompt=prompt,
temperature=0.7, # Some creativity for vocabulary variation
max_tokens=150, # Short snippet
)
hypothetical_doc = response.content.strip()
return hypothetical_doc
The hypothetical answer does not have to be correct to be useful. We are not showing it to the user; we are using it as an embedding probe that lives in the same neighborhood as the real docs. It does have to be about the right thing: a hypothetical answer that confidently describes the wrong concept pulls retrieval toward the wrong neighborhood, and this benchmark does not measure how often that happens.
HyDE vs query expansion
These two get conflated and they should not be. Query expansion adds synonyms: async becomes async, asynchronous, concurrent. HyDE writes a paragraph: async becomes Asynchronous processing allows non-blocking operations through callback patterns or async/await syntax.... Expansion gives you alternative tokens. HyDE gives you the style and structure of the target text, which is what is missing when the query and the documents are written in different registers.
Cost-benefit
The cost shape is one extra LLM call before retrieval. The benchmark artifact does not contain token counts or real dollar cost; benchmarks/run_article_01.py sets cost_per_1k_queries to 0.0 for every configuration, so this article reports latency only.
The measured HyDE result is a small recall gain that is not established, and a small ranking loss. Against the dense baseline on the same in-memory index, Recall@5 moves from 0.838 to 0.854, a 1.6 point lift. HyDE raised recall on 15 questions and lowered it on 9, which a sign test puts at p of about 0.3. Before relabelling, three HyDE runs on the same questions ranged over 2 points of recall, so run-to-run variation alone is about the size of the lift. MRR moves from 0.898 to 0.870, and top-1 hits from 120 to 116, so the first result is not better. Full query latency is 3,568 ms against 2,439 ms for a dense query measured in the same execution, a 1.46x multiplier, for the extra LLM call.
The rule of thumb: HyDE earns a trial when the gap between how users ask and how docs answer is real, the latency budget can spend about a second on the extra query transformation, and a missed source is more expensive than the added LLM call. This corpus does not have that gap, because its questions and documents share a vocabulary, and this run gives no reason to add HyDE to it.
Query decomposition: multi-hop reasoning
Some queries cannot be answered by a single retrieval, no matter how good the embedding is. "Compare FastAPI and Spring async patterns" needs information about FastAPI and information about Spring, and a single dense vector has to sit somewhere between the two topics, which can leave both sides under-represented in the top five. That is the risk, not an observed failure in this dataset. Decomposition splits the query into sub-queries, retrieves them in parallel, and merges the results.
Implementation
@traced_generation
def _decompose_query(
self,
query: str,
correlation_id: str | None = None,
) -> list[str]:
prompt = f"""Break down this complex query into 2-4 simpler sub-queries.
Query: {query}
Rules:
- Each sub-query should be self-contained and answerable independently
- Cover different aspects of the original query
- Keep sub-queries concise (1 sentence each)
- Return as numbered list
Sub-queries:"""
response = self.llm_client.generate(
prompt=prompt,
temperature=0.3, # Low creativity for consistent decomposition
max_tokens=200,
)
sub_queries = []
for line in response.content.strip().split("\n"):
line = line.strip()
if line and (line[0].isdigit() or line.startswith("-")):
query_text = line.lstrip("0123456789.-) ").strip()
if query_text:
sub_queries.append(query_text)
if not sub_queries:
sub_queries = [query]
return sub_queries
Parallel retrieval
Sub-queries are independent and I/O-bound, so a ThreadPoolExecutor is enough; multiprocessing would buy us nothing here except the cost of forking.
@traced_retrieval
def retrieve(
self,
query: str,
top_k: int = 5,
correlation_id: str | None = None,
) -> list[dict[str, Any]]:
if self.use_decomposition:
sub_queries = self._decompose_query(
query=query,
correlation_id=correlation_id,
)
return self._retrieve_with_decomposition(
sub_queries=sub_queries,
top_k=top_k,
correlation_id=correlation_id,
)
processed_query = query
if self.use_hyde:
processed_query = self._generate_hypothetical_document(
query=query,
correlation_id=correlation_id,
)
return self._retrieve_single(processed_query, top_k)
When decomposition hurts
Decomposition is not free, and it makes some queries worse. "What is FastAPI?" gains nothing from being split, and pays the LLM-call tax anyway. Narrow topics produce sub-queries that all retrieve the same chunks, so deduplication wipes out the parallelism. And over-decomposition (5+ sub-queries) drowns the strong signal from the right chunk in mediocre matches from peripheral ones.
The usual advice is to put a cheap classifier in front of decomposition, so that simple queries stay on the plain pipeline. This benchmark does not test a classifier, and it gives that advice no support: on the 34 comparison questions, the slice a classifier would route to decomposition, Recall@5 was 0.686 against 0.725 for the like-for-like baseline. That slice result is 3 questions up and 7 down, so it is direction, not proof.
Graph RAG: entity-relationship traversal
Vector search is good at "documents that talk about the same thing" and bad at "documents that share a specific structured relationship". Graph RAG goes after that second case by building a knowledge graph from the corpus and walking it at query time.
When graph RAG is the right tool
Graph RAG pays off when the answer is a path, not a paragraph. "What Java framework is similar to React hooks?" wants the chain React -> hooks -> reactive programming -> Spring WebFlux, not the documents nearest to "React hooks" in vector space. "How are FastAPI and Pydantic related?" is a single edge lookup. A corpus with structured dependencies - specs, APIs, framework relationships - is the sweet spot.
Graph RAG falls down on fuzzy semantic queries, paraphrases of the same concept, and corpora large enough that the graph becomes unwieldy. "Explain async/await" lives in vector space, not in a graph.
Production systems usually need both, dispatched per query: dense retrieval for semantic questions, graph traversal for relationship questions, and sometimes a fusion of the two.
Implementation with NetworkX
NetworkX is here for teaching: pure Python, easy to inspect, easy to visualize. The repo also includes an optional Neo4j-backed class for persistence and concurrent access. The Article 1 benchmark does not compare NetworkX with Neo4j, so this section treats Neo4j as an implementation path, not a speed claim.
The source implementation is src/rag/graph_rag.py. GraphRAGPipeline keeps the graph in memory and injects the LLM client instead of hiding it behind global state:
def __init__(self, llm_client: UnifiedLLMClient | None = None) -> None:
self.llm_client = llm_client or UnifiedLLMClient()
self.graph: nx.DiGraph = nx.DiGraph()
Entity extraction and relation extraction are both traced generation calls. The prompts request JSON, the parser handles markdown code fences, and parse failures return an empty list. That design is simple and testable, but it is not free: graph construction spends LLM calls before the first query can benefit from the graph.
Graph Construction
Graph construction processes each document by extracting entities, extracting relations, and adding the resulting nodes and edges to the NetworkX graph:
entities = self.extract_entities(text)
for entity in entities:
name = entity["name"]
entity_type = entity.get("type", "unknown")
self.graph.add_node(name, type=entity_type, label=name)
relations = self.extract_relations(text, entities)
for relation in relations:
subject = relation.get("subject", "")
predicate = relation.get("predicate", "related_to")
obj = relation.get("object", "")
if subject and obj:
self.graph.add_edge(subject, obj, relation=predicate, label=predicate)
Graph Traversal
Query execution finds relevant nodes, traverses up to max_hops, and returns structured graph paths instead of document chunks:
path = nx.shortest_path(self.graph, node, target)
if len(path) <= max_hops + 1:
subgraph_nodes.update(path)
paths.append(path)
return {
"answer": answer,
"nodes": list(subgraph_nodes),
"paths": paths,
"metadata": {
"max_hops": max_hops,
"num_nodes": len(subgraph_nodes),
"num_paths": len(paths),
},
}
Cost analysis
The repository does not include an Article 1 Graph RAG timing or token-cost benchmark, so this article does not report Graph RAG dollars or latency.
The code shape is still useful: build_graph() calls entity extraction and relation extraction before adding nodes and edges to a NetworkX graph, and query() does keyword matching plus shortest-path traversal before returning structured paths and metadata. The implementation explains why Graph RAG needs a different benchmark, but it is not evidence for a cost table.
Neo4j as the production alternative
NetworkX is in-memory. The repo's production-oriented alternative is Neo4jGraphRAGPipeline, which adds a persistent driver, entity indexes, relationship writes, and Cypher traversal. That is an operational migration path, not a measured speedup in this article.
MATCH (n:Entity)
WHERE ANY(term IN $terms WHERE toLower(n.id) CONTAINS term
OR toLower(n.label) CONTAINS term)
RETURN n.id AS id
LIMIT 10
UNWIND $relevant_nodes AS start_id
MATCH (start:Entity {id: start_id})
MATCH path = (start)-[:RELATES_TO*1..$max_hops]->(end:Entity)
WITH path, start, end
RETURN [node IN nodes(path) | node.id] AS path_nodes
LIMIT $limit
The full NetworkX and Neo4j implementations are in src/rag/graph_rag.py. The Article 1 benchmark does not run Neo4j.
Putting it together: contextual retrieval
This section describes a design, not a measured result. Production retrieval systems rarely keep these techniques isolated. A pipeline for "Compare FastAPI and Spring async patterns" can look roughly like this:
- Query decomposition into
["What is FastAPI async?", "What is Spring async?"]. - HyDE transformations on each sub-query to generate corpus-style hypothetical answers.
- Vector retrieval of the top 5 chunks for each sub-query, in parallel.
- Knowledge-graph lookup for direct relationships between FastAPI and Spring.
- Result fusion with deduplication and relevance reranking.
- Final answer generation with citations into both chunks and graph paths.
Each stage is optional, and each has to earn its place. The one combination measured here, HyDE plus decomposition, cost 3.38x the latency of a dense query, added 2.3 pp of recall over HyDE alone and lowered MRR, so treat the full pipeline as something to benchmark stage by stage, not as a target architecture.
Empirical comparison
Retrieval quality
The retrieval-quality table uses the 142 questions, scored from results/data/article_01_benchmarks_corrected_2026-10-05.json. Hit@1 is the number of questions whose first retrieved chunk comes from an expected source. Deltas for the transformed rows are against the in-memory dense baseline, which shares their index:
How these numbers were produced matters. The retrieval for every row was recorded on 4 October: the naive and transformed rows in one benchmark execution, and the in-memory baseline in a separate run later that day on the same index construction as the transformed rows. The rescore uses run 1 of 3 and drops the 8 removed questions, and scripts/rescore_article_01.py reproduces every number in the table from the stored artifact. After the second labelling pass, the stored retrievals were rescored against the new labels instead of rerunning, so the quality columns are single-run. A rerun was not possible for the three transformed rows: their LLM calls went through a fallback chain that started with a Groq Llama 3.1 8B model, which Groq retired on 5 October, and any rerun would measure a different model. Before relabelling, three runs of HyDE ranged over 2 points of Recall@5, and three runs of the composite over 1.3 points, which is the repeatability to keep in mind. The latency column comes from the same three-run execution, over the 150-question set used before relabelling. The in-memory baseline's latency was measured in a separate execution and is not comparable, so latency multipliers below use the 2,439 ms dense query.
Cost is not reported because the benchmark artifact does not contain real token or dollar measurements. The latency column is full pipeline.query() latency, not retrieval-only latency.
Four findings.
Decomposition hurts ranking. Recall@5 is unchanged, MRR drops by 0.091, and top-1 hits fall from 120 to 100, at 1.31x the latency. The first relevant source moved down for 37 questions and up for 13, which a sign test puts at p of about 0.001. This is the clearest result in the table. A plausible cause is that the merged list interleaves chunks from several sub-queries, so the best chunk for the original question is less often first. That is a hypothesis, not something this benchmark isolates. On the 34 comparison questions it is built for, it was not better: 0.686 Recall@5 against 0.725, with 3 questions up and 7 down, which is not significant.
HyDE's recall gain is not established. It adds 1.6 pp Recall@5, with 15 questions up and 9 down, and costs 1.46x the latency. MRR and top-1 hits are lower. The first version of this post reported a 2.2 point lift on broken labels against a baseline that was not like-for-like.
The composite (HyDE + decomposition) adds the most recall and costs the most. Recall@5 is 3.9 pp above the baseline, with 21 questions up and 11 down, which a sign test puts at p of about 0.1. MRR is 0.052 lower, top-1 hits are 108 against 120, and full query latency is 8,244 ms, 3.38x a dense query. On the comparison slice it reached 0.779 against 0.725, the same as the Chroma dense baseline without any LLM call. If the consumer reads the whole top five and can wait 8 seconds, this is the one row worth a second look on a larger set.
The two dense baselines are within measurement noise of each other. The Chroma baseline is 0.9 pp ahead on recall and 0.015 behind on MRR, which is smaller than the effect of changing the checkout path, described above.
Graph RAG has no row. Graph RAG returns paths through an entity graph, not document chunks - it isn't directly comparable on Recall@K / MRR (which measure chunk retrieval against ground-truth source documents). The architectural section above covers when to reach for it; quantitative head-to-head needs a different metric (entity-path overlap) and is out of scope for this benchmark.
Decision framework
The routing policy should follow query shape:
MEASURE dense baseline first # 0.838 to 0.847 Recall@5 here
IF recall_matters_more_than_first_result AND latency_budget_allows:
BENCHMARK HyDE # +1.6 pp Recall@5 (not established), -0.028 MRR, 1.46x latency here
BENCHMARK HyDE + decomposition # +3.9 pp Recall@5 (p ~ 0.1), -0.052 MRR, 3.38x latency here
IF consumer_reads_first_result:
AVOID decomposition # -0.091 MRR, 20 fewer top-1 hits here (p ~ 0.001)
ELIF query_type == "relationship_traversal":
EVALUATE Graph_RAG # different metric framework; benchmark separately
ELSE:
USE Naive_RAG
No query transform delivered an established gain on this set, and decomposition did not help the comparison questions it targets. That is a result about this corpus, where a dense retriever already finds a source for more than nine questions in ten. A corpus with a real vocabulary gap or real multi-hop questions may behave differently, and the way to find out is to split the dataset by query shape and benchmark that slice before routing production traffic there.
Observability and tracing
The RAG modules use the tracing decorators in src/core/observability.py. The source excerpts above show the important applications: @traced_retrieval wraps retrieve(), and @traced_generation wraps HyDE, decomposition, and answer generation.
Observability decorators create Phoenix spans with latency and input/output attributes when OBSERVABILITY_ENABLED is true, and traced_generation attaches token and cost attributes if the LLM response exposes them. The Article 1 benchmark JSON does not aggregate those token or cost fields, so the empirical section above does not report them. The Phoenix UI is at http://localhost:6006 when the local stack is running.
What to take away
The practical takeaway is narrower than the architecture diagrams usually imply.
Dense retrieval sets a bar of 0.838 to 0.847 Recall@5 on this 142-question, four-framework set. The baseline should come first, the measurement should use the real query distribution, and extra machinery should wait until the failure mode is visible.
HyDE's recall gain is not established here: +1.6 pp Recall@5, -0.028 MRR, 1.46x latency against a like-for-like baseline.
Decomposition does not earn a global default on this set: no recall change, -0.091 MRR, 20 fewer top-1 hits, and 1.31x latency. It did not help the comparison slice either.
The HyDE + decomposition composite added the most recall, 3.9 pp, and is the most expensive configuration, at 3.38x the latency, with lower ranking. It is a candidate for a larger benchmark, not a default.
Check what your pipeline embeds. Here every chunk carried the absolute file path into its embedding, which made the results depend on the checkout directory by up to 2.5 pp, more than HyDE's effect.
Graph RAG is the right tool when the questions are about relationships and the corpus encodes structured dependencies. We do not benchmark it head-to-head here because its output (entity paths) is not directly comparable to top-K document retrieval on Recall@K / MRR. The architectural section above is the honest version: when graph traversal is the right shape for the question, evaluate it with relationship-specific metrics, not the table above.
Three details matter for interpreting the numbers: chunking uses 500 tokens with 50-token overlap, the benchmark measures the whole query() path and not only retrieval, and the labels that every number depends on had to be rebuilt and then checked by a second independent pass. A benchmark is only as good as its ground truth and its baseline. The validator in the repository now fails on a label that points at a missing file or on a duplicated question, and the runner now includes the like-for-like baseline.
Routing rule from this run: measure a dense baseline first, keep decomposition away from ranking-sensitive consumers, benchmark HyDE or the composite only where recall matters more than the first result and latency allows, and use Graph RAG only with relationship-specific metrics.
The raw benchmark artifact is results/data/article_01_benchmarks_corrected_2026-10-05.json, which includes per-question scores for each configuration and records how they were produced. The question set is datasets/synthetic_queries/article_01.json, version 2.1, and its metadata records the label policy. Both labelling passes and every adjudication decision are in datasets/synthetic_queries/review/. The original, pre-correction results are preserved as results/data/article_01_benchmarks_verify.json. To rerun the benchmark, which will not reproduce these numbers exactly for the reasons above, use:
uv run python benchmarks/run_article_01.py --runs 3 --top-k 5 --output results/data/article_01_benchmarks_rerun.json
The focused validation command is:
uv run pytest tests/unit/rag/test_advanced_rag.py tests/unit/rag/test_naive_rag.py tests/unit/rag/test_graph_rag.py tests/unit/core/test_benchmarking.py
The goal is not a universal winner. The goal is a benchmark-backed argument, so the next "graph RAG is the future" / "HyDE is overkill" / "naive RAG is fine" debate has real code and real measurements behind it.
References
Published papers, separate from the benchmark numbers above:
- Gao, L., Ma, X., Lin, J., & Callan, J. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels. ACL 2023. The HyDE paper.
- Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022). Measuring and Narrowing the Compositionality Gap in Language Models. EMNLP 2023. Query decomposition.
- Edge, D., Trinh, H., Cheng, N., et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130.
- Gao, Y., Xiong, Y., Gao, X., et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997.
Technical documentation:
- LlamaIndex: documentation and query transformations.
- ChromaDB: documentation and hybrid search.
- BGE embeddings (BAAI): model card and the MTEB leaderboard.
- NetworkX: documentation and algorithms.
- Neo4j: documentation and the Cypher manual.
- Arize Phoenix: documentation and the LlamaIndex integration.
Series navigation
Written by
Jagdish Salgotra
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Keep reading · rest of the series
Runnable Java experiments on tail latency: why averages lie, queueing theory, hedged requests, coordinated omission, backpressure, and SLOs.