Two RAG Evaluators, Two Very Different Scores

Agentic AI Stress Suite, Post 3 of 9. Reproducible benchmarks on where production RAG-to-agent stacks break: one harness, one corpus, negative results published alongside the positives.

Updated, 8 October 2026. The first version of this post reported RAGAS answer_correctness of 0.425 against a judge correctness of 0.960, and told readers to inspect the samples where the two disagree without showing any. The benchmark was rerun with per-sample records so that the disagreement can be inspected here. The rerun generated fresh answers, on a question file that has since been deduplicated, so its numbers differ from the first run and replace them throughout. Three inspected samples are added below. The drift-detection section corrects a statistical error, and the summaries of Articles 1 and 2 reflect their corrected results.

Clarified, 11 October 2026. The sample readings were made by AI models, not human reviewers, and the post now says so. The zero-factual-overlap explanation is marked as unverified, and code links point to a fixed repository commit.

The short answer, measured on 30 naive RAG answers: RAGAS scored them 0.370 answer_correctness, while a gpt-4o-mini judge scored the same answers 0.887 correctness. Both evaluators are given the same reference answer, so the gap is not explained by one of them never seeing it. On the samples inspected below, RAGAS gave the identical score of 0.22 to a correct answer, a refusal, and a hedge, while the judge gave them 1.0, 0.0, and 0.6. Neither score is calibrated against human ratings.

Retrieval metrics made the first two articles measurable. Article 1 compared dense retrieval, HyDE, decomposition, and the HyDE + decomposition composite on the 142-question Article 1 set. The corrected results put dense retrieval at 0.838 to 0.847 Recall@5 and HyDE at 0.854, a gain that is not established. Article 2 moved to a 30-query retrieval workload and found a mixed result: hybrid search ranked the first relevant source higher than dense-only but lost a little Recall@5, and FlashRank reranking added latency without beating plain hybrid.

Those numbers answer a retrieval question: did the right source land in the retrieved context? They do not answer the production question: did the model produce an answer that was correct, complete, grounded, and relevant?

The gap matters. A RAG pipeline can retrieve the right chunk and still produce a weak answer. The prompt can bury the key evidence. The generator can paraphrase correctly but fail a reference-aware metric. A judge can give high scores because the answer sounds plausible. Retrieval quality is necessary, but it is not the same as answer quality.

What Recall@5 Cannot See

Recall@5 is the fraction of a query's expected source documents that appear among the top five retrieved chunks, averaged over queries. That is a useful retrieval signal. It stops before the answer is written.

Three answer-level failures sit outside that metric:

Hallucination. The right chunk appears in context, but the generated answer adds a claim unsupported by the chunk. A FastAPI answer can correctly explain Depends() and then add an unsupported comparison to Spring @Autowired. Recall@5 stays perfect because retrieval succeeded. Faithfulness drops because generation added an unsupported claim.

Tangential correctness. The context is relevant, and the answer is grounded, but the answer dodges the actual user question. A query about a custom HTTPException detail body can receive a generic explanation of HTTP errors. Retrieval succeeded. Usefulness did not.

Incomplete synthesis. The right evidence appears in retrieved chunks four and five, but the final prompt or generator gives most attention to earlier chunks. Recall@5 is still one. Completeness is not.

These three are illustrations of what the metric cannot see, not failures counted in this dataset. The evaluation framework exists because an answer can fail after retrieval has succeeded, and a retrieval metric cannot tell you how often.

The Evaluation Contract

The shared contract is small. EvalSample captures the sample being judged:

@dataclass
class EvalSample:
    sample_id: str
    query: str
    answer: str
    expected_answer: str
    contexts: list[str] = field(default_factory=list)
    source_docs: list[str] = field(default_factory=list)
    metadata: dict[str, Any] = field(default_factory=dict)

View on GitHub

The fields are deliberately plain:

EvalResult carries scores produced by any evaluator:

@dataclass
class EvalResult:
    sample_id: str
    evaluator: str
    scores: dict[str, float]
    metadata: dict[str, Any] = field(default_factory=dict)

View on GitHub

The important design choice is not the dataclass syntax. The important choice is that every evaluator converts from the same shape. RAGAS wants a HuggingFace Dataset. DeepEval wants an LLMTestCase. The LLM judge wants a prompt. The rest of the code should not care.

That contract keeps the comparison fair. The Article 3 runner generates answers once with the naive RAG pipeline, then sends the same 30 EvalSample objects to both evaluators. The evaluators disagree on the same answers, not on two different generations.

The Benchmark Setup

The benchmark for this article, rerun on 4 October 2026, exercises two evaluators: RAGAS and LLM-as-judge. DeepEval, A/B testing, and drift detection are implementation-backed and test-backed, but they are not part of the measured Article 3 benchmark output. That boundary is intentional. Evaluation articles become misleading when methodology, tests, and benchmark results are blended into one "it works" claim.

The 4 October run used this command:

uv run python benchmarks/run_article_03.py \
  --queries 30 \
  --runs 3 \
  --top-k 5 \
  --ragas-llm gpt-4o-mini \
  --judge-model gpt-4o-mini \
  --output results/data/article_03_benchmarks_corrected_2026-10-04.json

View on GitHub

The source artifact is results/data/article_03_benchmarks_corrected_2026-10-04.json. The runner also writes results/data/article_03_benchmarks_corrected_2026-10-04_samples.json, which holds each question, reference answer, generated answer, retrieved contexts, and every score from both evaluators.

The runner uses:

The generator is not pinned: NaiveRAGPipeline answers through the project's provider fallback chain, and the artifact does not record which model produced each answer. Whether the gpt-4o-mini judge favours answers from its own model family is therefore not measured here.

The collection hygiene check matters. NaiveRAGPipeline.build_index() uses Chroma get_or_create_collection, so rerunning into an existing collection can duplicate chunks. The runner now drops the a03 collection before indexing, and it contained 1,369 chunks afterward.

The benchmark generates the 30 RAG answers once. RAGAS and the LLM judge then score those same 30 generated answers three times each. That means evaluator variance is measured without changing the underlying pipeline answers.

RAGAS: Strict on Overlap with the Reference

RAGASEvaluator converts the shared samples into the dataset shape RAGAS expects:

data = {
    "question": [s.query for s in samples],
    "answer": [s.answer for s in samples],
    "contexts": [s.contexts for s in samples],
    "ground_truth": [s.expected_answer for s in samples],
}

View on GitHub

The four metrics used in this run are:

answer_correctness compares the generated answer with the expected answer. In RAGAS 0.4.3, the version used here, it breaks both into statements, has an LLM classify which statements match in each direction, and blends that factual-overlap score with an embedding similarity between the two texts. The default weights are 0.75 for the factual part and 0.25 for similarity. That makes it strict in a specific way: an answer loses score for statements the reference does not contain, and for reference statements the classifier does not find in the answer, as well as for statements it gets wrong.

The RAGAS results from that run:

The Std column is the spread of the three run means on the same 30 answers. It measures evaluator repeatability, not uncertainty across questions. Two of the 360 RAGAS metric calls timed out and are excluded from their means.

The headline is not a single score. The headline is the shape:

Faithfulness is high. A score of 0.975 means the answers stay close to the retrieved contexts.

Answer relevance is split. The mean of 0.743 hides two groups. Twenty-three answers score 0.89 or higher. Seven score 0.32 or lower, six of them exactly zero, including the three that open with a statement that the context does not contain enough information. RAGAS treats a noncommittal answer as irrelevant even when it goes on to answer the question.

Reference correctness is much lower. answer_correctness lands at 0.370. That is not a retrieval failure by itself. It means the generated answer and the expected answer do not overlap strongly under RAGAS's correctness metric. The inspected samples below show what that does and does not tell you.

LLM-as-Judge: Lenient and Useful

LLMJudge scores each sample with a fixed rubric:

Scores are requested on a 0-5 scale and normalized to 0-1. The implementation sends the rubric as the system prompt and the sample as the user prompt:

response = self._client.generate(
    prompt=prompt,
    system_prompt=JUDGE_SYSTEM_PROMPT,
    temperature=0.0,
    max_tokens=512,
    preferred_provider=self._preferred_provider,
    preferred_model=self._preferred_model,
)

View on GitHub

The judge requests temperature 0, but the suite's LLM client treats 0 as unset and substitutes its default of 0.7, so the judge sampled at 0.7. Its three runs agree within 0.02 on every dimension.

The judge results from that run:

The judge is much more generous than RAGAS. The same generated answers that receive 0.370 RAGAS answer_correctness receive 0.887 judge correctness. The judge gave 21 of the 30 answers a perfect correctness score. RAGAS scored those same 21 answers anywhere from 0.21 to 0.97. Across all 30 samples, using each sample's mean over the three runs, the two correctness scores have a Pearson correlation of 0.33. With 30 samples and 21 judge scores at the ceiling, that is weak evidence of any relationship at all.

That disagreement is the most important result in the article. It is not a difference between a reference-aware evaluator and one that ignores the reference. Both receive the expected answer, and the judge rubric explicitly asks whether the answer is factually correct against it. The difference is in how the reference is used. RAGAS measures statement overlap in both directions, so an answer scores low whenever its statements are not matched to the reference's: because it is wrong, because it says more, or because the classifier did not recognise a paraphrase. The judge is looser. In the first sample below it gave full marks to an answer that took a different route from the one the reference names, which suggests it leans on the retrieved context as much as on the reference.

A production team should not average these scores. The gap is diagnostic:

Three Samples Where the Evaluators Disagree

The aggregate gap says the evaluators disagree. It does not say which one to believe. These three samples come from the per-sample file. They were picked by hand to show different causes: one from the 21 answers the judge scored perfectly, and two from the nine it did not. They are not a random draw. Every score shown is the mean of the three evaluator runs, and none of the three samples was affected by the two RAGAS timeouts. The assessment under each one combines two independent readings of the same data, both made by AI models (Claude), with no human reviewer. They agreed on the main verdict for each sample, and the second reading corrected two details, noted below. These are model judgments about evaluator behaviour, not calibrated human ratings, and they share whatever blind spots the models have.

A good answer scored against a thin reference (q007)

Question. How do you validate JSON data using Pydantic models?

Reference answer. "You can validate JSON data using the validate() method on a Pydantic model."

Generated answer. A worked example that defines a Book model, parses a JSON string with json.loads(), constructs the model from the parsed dictionary, and points out that the string "2008" is coerced to an integer. The retrieved contexts contain that pattern.

Assessment. The answer is correct and grounded, with working code. It misses the most direct API, model_validate_json() in Pydantic v2 or parse_raw() in v1. The reference is thin and also partly wrong. In v1, validate() takes a dictionary, not a JSON string. In v2 it is a deprecated alias of model_validate. Neither version validates JSON text with it, and none of the retrieved contexts mentions it. A score of 0.22 is what the RAGAS formula produces when the factual part is zero and only the similarity part contributes, 0.25 times a similarity of about 0.88. That is one reading of the score, not read from RAGAS's intermediate output, which was not retained; a small factual score with a lower similarity would give the same total. The judge's 1.00 is slightly generous, because the answer skips the direct JSON method, but it is much closer than RAGAS. The low RAGAS score is mostly a finding about the reference.

Consequence. A low answer_correctness cannot be read as a wrong answer until the reference has been checked. These references were generated by an LLM and never reviewed, which is common for synthetic evaluation sets, and this one is factually off.

A refusal that both evaluators mark down (q027)

Question. What is the key difference between a Spring bean and a Pydantic model?

Reference answer. "A Spring bean is a class that is managed by the Spring IOC container, while a Pydantic model is a class that is used to validate data."

Generated answer. A refusal. It says the context does not contain enough information, and explains that the retrieved text covers Bean Validation and Pydantic but never defines a Spring bean.

Assessment. The answer is unhelpful, and the refusal is honest. Retrieval returned chunks about Bean Validation instead of the IoC container document, so the generator had nothing to work with. This is a retrieval failure that surfaces as an answer failure. The judge is right that the question went unanswered. Its groundedness score of 0.00 is wrong, because the refusal describes the retrieved context accurately, and RAGAS faithfulness of 0.90 reflects that.

Consequence. RAGAS gave this refusal the same 0.22 it gave the correct answer above. On these two samples answer_correctness does not separate a good answer from no answer. The judge separates them but folds "did not answer" into every dimension, so its groundedness column cannot be read on its own.

A hedge that contains the right answer (q008)

Question. What is the difference between the @autowired and @bean annotations in Spring?

Reference answer. "@autowired is used to inject a bean into a method, while @bean is used to define a bean."

Generated answer. It opens by saying the context does not contain enough information to fully answer. It then states that @Autowired tells the container to resolve and inject dependencies and that @Bean declares a bean in a configuration class, which is the distinction the reference draws.

Assessment. This one is ambiguous. The substance matches the reference. The framing tells the user the question cannot be answered. Whether that is a good answer depends on a product decision about how the assistant should behave when the context covers both halves of a comparison but never compares them. RAGAS scores relevance at zero because the answer is noncommittal. The judge scores relevance near the top because the content addresses the question. The RAGAS correctness of 0.22 is again consistent with little or no factual overlap, even though the answer states the same distinction as the reference. That points at the statement matching, not at the answer. Two more details cut both ways. The sentence that states the distinction comes from the model's own knowledge, not from the retrieved contexts, so it is correct but not grounded. And the judge's written justification says the answer never states the difference, which is false, so its 0.60 is somewhat harsh.

Consequence. The evaluators disagree here because they encode different policies about hedging, not because one is broken. A team has to decide that policy first and then pick or tune the evaluator to match it.

What the three samples show

All three answers received 0.22 from RAGAS answer_correctness, a score consistent with little or no factual-overlap contribution. The intermediate RAGAS outputs were not retained, and 0.22 does not uniquely determine the factual part of a weighted score, so that explanation remains unverified. The judge scored them 1.00, 0.00, and 0.60. The causes differ. For q007 the reference is too thin and partly wrong. For q027 retrieval failed. For q008 the substance matched and RAGAS still found no overlap. Seven of the 30 answers sit at 0.25 or below on this metric. The setup makes that likely: the 30 references have a median length of 16 words, 25 of them are a single sentence, and the answers have a median length of 236 words. For the q007 kind of case, improving the references is the first fix, before changing the pipeline or the evaluator. The judge tracks answer quality more closely on these three samples, but it is generous, with 21 perfect scores out of 30, and it lets one failure colour unrelated dimensions.

Three samples explain the disagreement. They do not calibrate either evaluator. That needs a set of human-rated answers and a correlation check against it.

Parse Failures Are Reported Separately

In the first run of this benchmark, one of 90 judge replies failed JSON parsing. LLMJudge._parse_judge_response() returned an empty object, _extract_scores() turned it into four zeros, and those zeros were averaged into the quality scores. That mixed two different things. A reply that cannot be parsed is a missing assessment. It is not an answer that was assessed and scored zero.

The judge now flags a failed parse, and the runner reports three things separately: quality scores over assessed samples only, the identifiers of failed evaluations, and the share of evaluations that produced a valid score. It also keeps a failure-penalised score, with zeros included, labelled as such for anyone who wants an operational number. In the rerun all 90 judge replies parsed, so valid-score coverage is 1.0 and the two scores are identical. One failure in 180 evaluations across both executions of the benchmark is too few to estimate a rate.

Pipeline Latency and Evaluator Wall Time

The pipeline latency in the artifact is full pipeline.query() latency for the 30 generated samples:

The benchmark does not expose stage-level latency for retrieval versus generation, so this article does not claim retrieval-only latency.

Evaluator wall time:

The serving pass generated 30 answers once. Using the measured mean query latency, those 30 queries account for about 67 seconds of pipeline query time. The two evaluators then spent about 2,424 seconds scoring the same generated samples. On this run, evaluation wall time was roughly 36x the measured pipeline query time. RAGAS wall time was about five times its first-run figure of 439 seconds for a comparable workload, with two metric calls timing out, so it depends heavily on provider throughput on the day.

That ratio is more useful than a dollar estimate because the benchmark artifact does not log RAGAS token usage or RAGAS cost. The judge persistence table stores provider cost for judge calls, but RAGAS does not write a comparable field into the benchmark artifact. Cost is therefore not reported as a measured result.

DeepEval: Implemented, Not Benchmarked Here

DeepEvalEvaluator is source-backed and test-backed. It converts an EvalSample into DeepEval's LLMTestCase:

return LLMTestCase(
    input=sample.query,
    actual_output=sample.answer,
    expected_output=sample.expected_answer,
    retrieval_context=sample.contexts,
)

View on GitHub

Its four configured metrics are faithfulness, contextual precision, contextual recall, and answer relevancy. The useful idea is the diagnostic quadrant:

DeepEval is not in the benchmark artifact. No DeepEval score appears in this article. The implementation is covered by tests/unit/rag/test_evaluation.py, where the adapter, metric mapping, batch behavior, and metric-failure fallback are tested with mocks.

A/B Testing: The Decision Layer

ABTestRouter assigns queries deterministically:

query_hash = hashlib.md5(query.encode()).hexdigest()
hash_int = int(query_hash[:8], 16)
return "a" if hash_int % 2 == 0 else "b"

View on GitHub

ABTestAnalyzer.calculate_statistics() then reports descriptive statistics, Welch's t-test, and Cohen's d. That code is for deciding between two pipelines. Article 3 benchmarks only one pipeline, so the A/B framework is not exercised by the 4 October run.

The focused test run did exercise it. tests/integration/rag/test_ab_testing.py covers deterministic routing, approximate 50/50 split over 1,000 queries, SQLite logging, report generation, metadata persistence, multi-metric analysis, significant differences, and no-significant-difference cases.

The implementation handles two mundane but important evaluation-infrastructure details:

• SQLite indexes must be created with separate CREATE INDEX statements, not inside CREATE TABLE.
• Zero-variance samples need an explicit statistical path so Cohen's d and t-test output do not depend on undefined SciPy behavior.

Those fixes are small, but they matter. Evaluation infrastructure that cannot store or compare its own results is not infrastructure.

Drift Detection: The Production Layer

DriftDetector monitors query embedding distributions. It establishes a baseline, keeps a rolling current window, and compares the two with KL divergence and a multi-dimensional KS test.

The detector defaults:

The implementation uses per-dimension histograms for KL divergence and a KS test on every dimension. It takes the smallest per-dimension p-value and Bonferroni-corrects it by the number of dimensions:

ks_stat, p_val = stats.ks_2samp(baseline_dim, current_dim)
corrected_p = min(1.0, float(np.min(p_values)) * len(p_values))
return float(np.max(ks_statistics)), corrected_p

View on GitHub

The first version of this post called the uncorrected minimum "the most conservative" p-value. It is the opposite. With 384 dimensions, at least one falls below 0.05 by chance on almost any window. In a simulation of 200 windows drawn from the same distribution as the baseline, the uncorrected minimum raised an alert on all 200, and the corrected value on 7. The detector as first published would have alerted constantly on a stream that was not drifting. One unit test missed this because it compared two sets of unit-normalised vectors that differed only in scale, which normalisation removes, and passed on the false alarm. Bonferroni assumes nothing about dependence between dimensions, so it over-corrects for correlated embedding dimensions and will miss small real shifts.

The thresholds are project defaults, not universal constants. A production deployment should calibrate them against a stable historical period and then check whether known topic shifts cross the threshold.

Drift detection is not in the Article 3 benchmark JSON. A 30-query static sample cannot model an evolving query stream. The relevant evidence here is unit coverage: tests/unit/rag/test_drift_detection.py covers baseline establishment, no-drift behavior, mean-shift detection, variance-change detection, KL calculation, KS execution, alert severity, metadata logging, window limits, and invalid embedding shape handling.

Test Evidence

The focused Article 3 test command was:

uv run pytest \
  tests/unit/rag/test_evaluation.py \
  tests/unit/rag/test_drift_detection.py \
  tests/integration/rag/test_ab_testing.py

The result was:

49 passed, 1 warning

The remaining warning is from testcontainers Redis deprecation code, not from the Article 3 evaluation modules.

Test coverage by claim:

Operating Rules

The LLM judge is the fast directional check during development, but its JSON output must be monitored, with parse failures counted separately from scores. RAGAS and judge results should be read together before deployment, and a wide gap between them is a reason to audit the reference answers before trusting either number.

A/B analysis belongs only after two pipeline variants produce comparable result records. Welch's t-test answers whether the difference is statistically visible, and Cohen's d answers whether it is large enough to matter. Drift detection belongs on live query embeddings after a stable baseline period exists, not on one synthetic static batch. LLMJudge.calculate_correlation() against a human-rated golden set is a first check, not a guarantee: correlation alone does not establish calibration or how the judge fails on hard cases. This article did not build a human-rated golden set, so the absolute judge scores should not be treated as calibrated truth.

Reproducibility

The benchmark command is in The Benchmark Setup and the test command is in Test Evidence above.

Artifacts and implementation map:

References

• Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217.
• Zheng, L., Chiang, W. L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.
• Confident AI. DeepEval: The Open-Source LLM Evaluation Framework.
• SciPy documentation for ks_2samp, ttest_ind, and pearsonr: scipy.stats reference.