Code repositoryagentic-ai-stress-suite
J
Written by
Jagdish Salgotra
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Was this article helpful? or email →anonymous · no account needed
Naive RAG: 0.487 Recall@5 on 300 tech-docs queries. HyDE reaches 0.509 at 1.14x latency; decomposition drops to 0.477.
Written by
Distributed systems, cloud-native architecture, and the JVM. mostly shipping, occasionally reading.
Agentic AI Stress Suite, Post 1 of 9. Reproducible benchmarks on where production RAG-to-agent stacks break: one harness, one corpus, negative results published alongside the positives.
Retrieval-Augmented Generation (RAG) has become the default pattern for grounding LLMs in external knowledge. The basic shape is simple enough: embed documents, store vectors, retrieve top-K matches, generate an answer. The problem shows up the moment a real query lands. Naive top-K falls apart on multi-hop questions, on queries that share no vocabulary with the source documents, and on anything where semantic similarity alone is the wrong signal.
The practical question is not whether advanced retrieval sounds smarter than top-K. The practical question is whether a technique improves retrieval quality enough to justify its latency, operational complexity, and additional LLM calls on the workload in front of you.
The benchmark behind this article compares four retrieval variants on the same 300-query Article 1 dataset: Naive RAG, HyDE, query decomposition, and the HyDE + decomposition composite. The results are deliberately less dramatic than a synthetic demo. HyDE improves the baseline, but only modestly. Decomposition hurts the mixed workload when applied globally. The composite adds only a tiny Recall@5 gain over HyDE while losing MRR and adding substantial latency.
Graph RAG belongs in the discussion, but not in the same table. The repository implementation returns entity paths, while the benchmark metrics compare retrieved document paths. The honest treatment is architectural analysis plus a clear statement that Graph RAG needs a path-oriented metric before it can be compared numerically.
The benchmark compares the baseline and advanced RAG pipelines against the same 300-query technical-docs workload. The important implementation boundary is small: benchmarks/run_article_01.py orchestrates the run, src/core/benchmarking.py calculates Recall@5/MRR/latency, src/rag/naive_rag.py implements the baseline, and src/rag/advanced_rag.py implements HyDE and decomposition.
The run uses three repetitions at top_k=5, and the relevant unit suite passed with 79 tests. One methodology detail matters for interpretation: BenchmarkRunner.run_single_query() times pipeline.query(...), and both Article 1 RAG pipelines implement query() as retrieve plus answer generation. The quality metrics come from retrieved context_nodes, but latency is full query latency, not retrieval-only latency.
The metric definitions are intentionally simple. Recall@5 asks whether the expected source document appears anywhere in the top five retrieved chunks. MRR asks how early the first expected source appears. Latency measures the full query path used by the benchmark runner. Cost is not reported because the benchmark artifact does not contain real token or dollar measurements.
The baseline is the floor: chunk, embed, store, retrieve, generate. The naive pipeline is the same RAG shape most teams build first, and that makes it the right comparison point for every more complex variant.
The important source path is src/rag/naive_rag.py. NaiveRAGPipeline.__init__() wires the settings-backed LLM client, BGE embeddings, Chroma, and the SentenceSplitter. retrieve() is deliberately plain LlamaIndex vector retrieval:
k = top_k or self.top_k
retriever = self._index.as_retriever(similarity_top_k=k)
nodes = retriever.retrieve(query)
return nodes
The benchmark measures query(), not only retrieve(), because the source pipeline retrieves chunks and then calls generation before returning context_nodes:
context_nodes = self.retrieve(query_str, top_k=top_k)
answer = self.generate(query_str, context_nodes)
return {
"answer": answer,
"context_nodes": context_nodes,
"metadata": {
"query": query_str,
"num_retrieved": len(context_nodes),
"top_k": top_k or self.top_k,
"collection": self.collection_name,
},
}
Chunk size is the hyperparameter that quietly decides everything else. Too small and you fragment context until retrieval is just noise. Too large and the relevant sentence drowns in the irrelevant 400 tokens around it (and your prompt budget evaporates). For technical documentation, 500 tokens (~375 words, one or two paragraphs) is a defensible default.
The benchmark uses SentenceSplitter with 50-token overlap so a sentence straddling a chunk boundary is not lost on either side. The splitter respects sentence structure while targeting 500 tokens per chunk. The result is better than naive character splitting, not as strong as proper semantic chunking, and appropriate for a baseline.
A note on parent-document and late chunking. The baseline only ever returns the chunk itself, which can leave the LLM staring at a fragment without the surrounding paragraph. Parent-document retrieval returns the full document a chunk came from. Late chunking embeds larger sections and only splits at retrieval time. Both produce better generations, both cost you more tokens and latency. I do not use either here because the goal is to measure the floor, not the ceiling.
300 queries across the tech-docs corpus (Spring, React, FastAPI, Pydantic), 3 runs, mean ± std from results/data/article_01_benchmarks_verify.json:
The honest reading is simple: the expected source document is in the top 5 about 49% of the time, and the first expected hit averages to an MRR of 0.450. The quality metrics are identical across the three runs; latency varies because the benchmark times the full query() path, including answer generation.
That 0.487 is lower than the kind of baselines you see in single-framework demos. It is supposed to be. The query set spans four frameworks and includes paraphrases, multi-hop questions, and queries that share little vocabulary with the source paragraph. The whole point of the next three sections is to see which added machinery actually earns its place against this measured floor.
The baseline is the floor. Anything more complicated has to earn its place against these numbers, not against an intuition that "graph RAG sounds smarter".
The interesting failures cluster into three shapes, and each one motivates one of the techniques later in this article:
HyDE attacks the vocabulary-mismatch problem head-on. Instead of embedding the user's query, you ask the LLM to write a hypothetical document snippet in the style of the corpus, and embed that.
The query is short and casual: fastapi dependency injection example.
A representative benchmark query asks for an example of FastAPI's Depends() function. The relevant corpus section is written like framework documentation: it introduces dependency injection, then shows a get_db() dependency and a route parameter declared as db: Session = Depends(get_db). HyDE is useful when the user query and the documentation answer the same question in different registers.
The concept is the same, but the neighborhoods in vector space differ. HyDE closes the gap by manufacturing text that looks like the documents you want to retrieve. The implementation is in src/rag/advanced_rag.py:
@traced_generation
def _generate_hypothetical_document(
self,
query: str,
correlation_id: str | None = None,
) -> str:
prompt = f"""You are generating a hypothetical technical document snippet.
Query: {query}
Write 2-3 sentences of technical documentation that would answer this query.
Use formal technical language and specific terminology.
Do not say "this document explains" - just write the content directly.
Hypothetical document:"""
response = self.llm_client.generate(
prompt=prompt,
temperature=0.7, # Some creativity for vocabulary variation
max_tokens=150, # Short snippet
)
hypothetical_doc = response.content.strip()
return hypothetical_doc
The accuracy of the hypothetical answer is irrelevant. We are not showing it to the user; we are using it as an embedding probe that lives in the same neighborhood as the real docs.
These two get conflated and they should not be. Query expansion adds synonyms: async becomes async, asynchronous, concurrent. HyDE writes a paragraph: async becomes Asynchronous processing allows non-blocking operations through callback patterns or async/await syntax.... Expansion gives you alternative tokens. HyDE gives you the style and structure of the target text, which is usually what was actually missing.
The cost shape is one extra LLM call before retrieval. The benchmark artifact does not contain token counts or real dollar cost; benchmarks/run_article_01.py sets cost_per_1k_queries to 0.0 for every configuration, so this article reports latency only.
The measured HyDE result moves Recall@5 from 0.487 to 0.509, a +2.2 percentage point lift. MRR moves from 0.450 to 0.483, and full query latency moves from 1,624 ms to 1,855 ms, a 1.14x multiplier. The result is the cleanest win in the benchmark: modest recall lift, better ranking, and a measured +231 ms latency cost.
The rule of thumb: HyDE earns a trial when the gap between how users ask and how docs answer is real, the latency budget can spend the extra query transformation, and a missed retrieval is more expensive than the added LLM call. Otherwise, the baseline should remain in place.
Some queries cannot be answered by a single retrieval, no matter how good the embedding is. "Compare FastAPI and Spring async patterns" needs information about FastAPI and information about Spring, and a single dense vector cannot point at both at once. Decomposition splits the query into sub-queries, retrieves them in parallel, and merges the results.
@traced_generation
def _decompose_query(
self,
query: str,
correlation_id: str | None = None,
) -> list[str]:
prompt = f"""Break down this complex query into 2-4 simpler sub-queries.
Query: {query}
Rules:
- Each sub-query should be self-contained and answerable independently
- Cover different aspects of the original query
- Keep sub-queries concise (1 sentence each)
- Return as numbered list
Sub-queries:"""
response = self.llm_client.generate(
prompt=prompt,
temperature=0.3, # Low creativity for consistent decomposition
max_tokens=200,
)
sub_queries = []
for line in response.content.strip().split("\n"):
line = line.strip()
if line and (line[0].isdigit() or line.startswith("-")):
query_text = line.lstrip("0123456789.-) ").strip()
if query_text:
sub_queries.append(query_text)
if not sub_queries:
sub_queries = [query]
return sub_queries
Sub-queries are independent and I/O-bound, so a ThreadPoolExecutor is enough; multiprocessing would buy us nothing here except the cost of forking.
@traced_retrieval
def retrieve(
self,
query: str,
top_k: int = 5,
correlation_id: str | None = None,
) -> list[dict[str, Any]]:
if self.use_decomposition:
sub_queries = self._decompose_query(
query=query,
correlation_id=correlation_id,
)
return self._retrieve_with_decomposition(
sub_queries=sub_queries,
top_k=top_k,
correlation_id=correlation_id,
)
processed_query = query
if self.use_hyde:
processed_query = self._generate_hypothetical_document(
query=query,
correlation_id=correlation_id,
)
return self._retrieve_single(processed_query, top_k)
Decomposition is not free, and it makes some queries worse. "What is FastAPI?" gains nothing from being split, and pays the LLM-call tax anyway. Narrow topics produce sub-queries that all retrieve the same chunks, so deduplication wipes out the parallelism. And over-decomposition (5+ sub-queries) drowns the strong signal from the right chunk in mediocre matches from peripheral ones.
A cheap classifier belongs in front of decomposition: simple queries stay on the plain pipeline, and the decomposition LLM call is reserved for queries whose complexity justifies it.
Vector search is good at "documents that talk about the same thing" and bad at "documents that share a specific structured relationship". Graph RAG goes after that second case by building a knowledge graph from the corpus and walking it at query time.
Graph RAG pays off when the answer is a path, not a paragraph. "What Java framework is similar to React hooks?" wants the chain React -> hooks -> reactive programming -> Spring WebFlux, not the documents nearest to "React hooks" in vector space. "How are FastAPI and Pydantic related?" is a single edge lookup. A corpus with structured dependencies - specs, APIs, framework relationships - is the sweet spot.
Graph RAG falls down on fuzzy semantic queries, paraphrases of the same concept, and corpora large enough that the graph becomes unwieldy. "Explain async/await" lives in vector space, not in a graph.
Production systems usually need both, dispatched per query: dense retrieval for semantic questions, graph traversal for relationship questions, and sometimes a fusion of the two.
NetworkX is here for teaching: pure Python, easy to inspect, easy to visualize. The repo also includes an optional Neo4j-backed class for persistence and concurrent access. The Article 1 benchmark does not compare NetworkX with Neo4j, so this section treats Neo4j as an implementation path, not a speed claim.
The source implementation is src/rag/graph_rag.py. GraphRAGPipeline keeps the graph in memory and injects the LLM client instead of hiding it behind global state:
def __init__(self, llm_client: UnifiedLLMClient | None = None) -> None:
self.llm_client = llm_client or UnifiedLLMClient()
self.graph: nx.DiGraph = nx.DiGraph()
Entity extraction and relation extraction are both traced generation calls. The prompts request JSON, the parser handles markdown code fences, and parse failures return an empty list. That design is simple and testable, but it is not free: graph construction spends LLM calls before the first query can benefit from the graph.
Graph construction processes each document by extracting entities, extracting relations, and adding the resulting nodes and edges to the NetworkX graph:
entities = self.extract_entities(text)
for entity in entities:
name = entity["name"]
entity_type = entity.get("type", "unknown")
self.graph.add_node(name, type=entity_type, label=name)
relations = self.extract_relations(text, entities)
for relation in relations:
subject = relation.get("subject", "")
predicate = relation.get("predicate", "related_to")
obj = relation.get("object", "")
if subject and obj:
self.graph.add_edge(subject, obj, relation=predicate, label=predicate)
Query execution finds relevant nodes, traverses up to max_hops, and returns structured graph paths instead of document chunks:
path = nx.shortest_path(self.graph, node, target)
if len(path) <= max_hops + 1:
subgraph_nodes.update(path)
paths.append(path)
return {
"answer": answer,
"nodes": list(subgraph_nodes),
"paths": paths,
"metadata": {
"max_hops": max_hops,
"num_nodes": len(subgraph_nodes),
"num_paths": len(paths),
},
}
The repository does not include an Article 1 Graph RAG timing or token-cost benchmark, so this article does not report Graph RAG dollars or latency.
The code shape is still useful: build_graph() calls entity extraction and relation extraction before adding nodes and edges to a NetworkX graph, and query() does keyword matching plus shortest-path traversal before returning structured paths and metadata. The implementation explains why Graph RAG needs a different benchmark, but it is not evidence for a cost table.
NetworkX is in-memory. The repo's production-oriented alternative is Neo4jGraphRAGPipeline, which adds a persistent driver, entity indexes, relationship writes, and Cypher traversal. That is an operational migration path, not a measured speedup in this article.
MATCH (n:Entity)
WHERE ANY(term IN $terms WHERE toLower(n.id) CONTAINS term
OR toLower(n.label) CONTAINS term)
RETURN n.id AS id
LIMIT 10
UNWIND $relevant_nodes AS start_id
MATCH (start:Entity {id: start_id})
MATCH path = (start)-[:RELATES_TO*1..$max_hops]->(end:Entity)
WITH path, start, end
RETURN [node IN nodes(path) | node.id] AS path_nodes
LIMIT $limit
The full NetworkX and Neo4j implementations are in src/rag/graph_rag.py. The Article 1 benchmark does not run Neo4j.
Production retrieval systems rarely keep these techniques isolated. A pipeline for "Compare FastAPI and Spring async patterns" can look roughly like this:
["What is FastAPI async?", "What is Spring async?"].Each stage is optional. The pipeline only earns its complexity if the query justifies it, which is why a query classifier in front of all of this is more important than any single retrieval trick.
The retrieval-quality table uses 300 queries across the tech-docs corpus, 3 runs each, mean ± std from results/data/article_01_benchmarks_verify.json:
Cost is not reported because the benchmark artifact does not contain real token or dollar measurements. The runner sets cost_per_1k_queries to 0.0 for every configuration. The latency column is full pipeline.query() latency, not retrieval-only latency.
The pattern is straightforward.
HyDE is the best single-technique upgrade in the measured run. It adds +2.2 pp Recall@5 and +0.033 MRR for +231 ms. The lift is modest, but it is still the cleanest accuracy/ranking improvement measured here.
Decomposition underperforms naive on this mixed query set. Recall@5 drops by 0.9 pp, MRR drops by 0.023, and full query latency rises by 552 ms. That does not prove decomposition is bad; it proves this unfiltered 300-query mix is not enough to justify turning it on globally.
The composite (HyDE + decomposition) is not justified by this run. It adds only +0.3 pp Recall@5 over HyDE, but MRR is worse (0.434 versus 0.483) and latency is much higher (4,208 ms versus 1,855 ms). On this mixed set, HyDE alone is the more defensible choice.
Graph RAG has no row. Graph RAG returns paths through an entity graph, not document chunks - it isn't directly comparable on Recall@K / MRR (which measure chunk retrieval against ground-truth source documents). The architectural section above covers when to reach for it; quantitative head-to-head needs a different metric (entity-path overlap) and is out of scope for this benchmark.
The routing policy should follow query shape:
IF vocabulary_mismatch_likely:
TRY HyDE # +2.2 pp Recall@5, +0.033 MRR, 1.14x latency here
ELIF query_is_multi_hop:
BENCHMARK decomposition # -0.9 pp Recall@5 on the unfiltered mixed set
ELIF query_type == "relationship_traversal":
EVALUATE Graph_RAG # different metric framework; benchmark separately
ELSE:
USE Naive_RAG # 0.487 Recall@5, 1.62s full query latency
Pure decomposition did not win on the measured mixed set. If you believe the workload is dominated by multi-hop questions, split the dataset by query shape and benchmark that slice before routing production traffic there.
The RAG modules use the tracing decorators in src/core/observability.py. The source excerpts above show the important applications: @traced_retrieval wraps retrieve(), and @traced_generation wraps HyDE, decomposition, and answer generation.
Observability decorators create Phoenix spans with latency and input/output attributes when OBSERVABILITY_ENABLED is true, and traced_generation attaches token and cost attributes if the LLM response exposes them. The Article 1 benchmark JSON does not aggregate those token or cost fields, so the empirical section above does not report them. The Phoenix UI is at http://localhost:6006 when the local stack is running.
The practical takeaway is narrower than the architecture diagrams usually imply.
Naive RAG sets a 0.487 Recall@5 floor on this 300-query, four-framework set. The baseline should come first, the measurement should use the real query distribution, and extra machinery should wait until the failure mode is visible.
HyDE is the best single upgrade in the measured run: +2.2 pp Recall@5, +0.033 MRR, 1.14x latency. The lift is modest, but it improves both recall and ranking.
Decomposition does not earn a global default on this mixed set: -0.9 pp Recall@5, -0.023 MRR, and 1.34x latency versus naive. It may still help a multi-hop-only slice, but that needs its own benchmark.
The HyDE + decomposition composite is not a clear win here. It adds only +0.3 pp Recall@5 over HyDE, but loses MRR and costs 2.27x HyDE's latency.
Graph RAG is the right tool when the questions are about relationships and the corpus encodes structured dependencies. We do not benchmark it head-to-head here because its output (entity paths) is not directly comparable to top-K document retrieval on Recall@K / MRR. The architectural section above is the honest version: when graph traversal is the right shape for the question, evaluate it with relationship-specific metrics, not the table above.
Two implementation details matter for interpreting the numbers: chunking uses 500 tokens with 50-token overlap, and the benchmark measures the whole query() path, not only retrieval.
Routing rule from this run: naive for the floor, HyDE when vocabulary mismatch is likely, decomposition only after benchmarking the multi-hop slice, and Graph RAG only with relationship-specific metrics.
The raw benchmark artifact is results/data/article_01_benchmarks_verify.json. The exact reproduction command is:
uv run python benchmarks/run_article_01.py --runs 3 --top-k 5 --output results/data/article_01_benchmarks_verify.json
The focused validation command is:
uv run pytest tests/unit/rag/test_advanced_rag.py tests/unit/rag/test_naive_rag.py tests/unit/rag/test_graph_rag.py tests/unit/core/test_benchmarking.py
The goal is not a universal winner. The goal is a benchmark-backed argument, so the next "graph RAG is the future" / "HyDE is overkill" / "naive RAG is fine" debate has real code and real measurements behind it.
Published papers, separate from the benchmark numbers above:
Technical documentation: