Multi-Agent Patterns: What the Benchmark Actually Measured

Agentic AI Stress Suite, Post 5 of 9. Reproducible benchmarks on where production RAG-to-agent stacks break: one harness, one corpus, negative results published alongside the positives.

All 36 multi-agent trials in this run reported completion. An independent model judge approved two answers. Reading the traces overturned both approvals: the judge said a writer had revised an answer after critique, but no such writer pass had happened. Both answers also ended mid-sentence.

Part 4 showed why an agent's success flag is a weak measure of work done. Adding a researcher, writer, critic, or supervisor does not repair that measurement. This run adds another complication: a judge can misread the same workflow too.

There are useful operational results here, but they need their boundaries. The parallel route was fast because it overlapped three model calls without executing retrieval or synthesis. The critic loop used every allowed rewrite while hitting output limits. The old cost counter missed generation inside the RAG tool. And a separate SQL follow-up finally tests the valid-empty-result cases readers asked about after Part 4.

The code and complete evidence are available, including raw judge verdicts and a separate trace audit.

Experiment setup

The experiments ran on 25 October 2026. All generating calls used claude-sonnet-5-5 at effort medium, with no temperature parameter and no provider fallback. The separate judge was gpt-4o-mini at temperature 0.7.

The main experiment used the existing 12-task multi-agent dataset, three runs per task. It retained the existing prompts, routing, and generation limits. A separate six-case SQL fixture ran through ReAct and Plan-and-Execute three times each, adding 36 trials. Five later trials examined larger output budgets on selected multi-agent tasks.

The follow-up records every generating call, including calls inside RAG, along with intermediate states, final answers, tool results, retrieved source text, and judge verdicts. Acceptance criteria were frozen before generation. The criteria and trace reviews were produced by assistants; this is not a human-calibrated benchmark.

The four measured routes

The sequential and feedback pipelines use LangGraph. Fan-out uses Python's ThreadPoolExecutor. The artifact's framework label should not imply that every path is implemented as a LangGraph graph. CrewAI and AutoGen were not benchmark participants.

These rows ran different tasks. They are not a paired architecture comparison, and neither latency nor approval counts establish a winner. Latency excludes initial index construction and subsequent judging; generating cost excludes the judge. It includes the inner RAG generator when called.

The completion flag is still the original runner's check:

success = bool(output) and not error

A non-empty concatenation, a truncated draft, or a response that never used a required database can pass it. It is not a task-acceptance check.

The two judge approvals were sequential trials: q001/run 2 and q008/run 1. For q008, the judge wrote that the writer incorporated missing pieces after receiving critic feedback. The recorded node sequence is researcher, writer, critic, then termination. There is no later writer node. The guide ends halfway through a word. For q001, the output ends during its critique, without the claimed subsequent refinement.

Neither approval survived trace inspection. That does not establish that every factual statement in all 36 answers was wrong. It establishes that these approvals did not verify the requested work. The raw verdicts remain unchanged in the artifact; the audit corrections are separate.

The cost hidden inside a tool

The earlier runner injected an accumulating LLM client into the outer agents. Its shared RAGTool wrapped a NaiveRAGPipeline that created its own LLM client. A research tool call therefore retrieved context and generated an answer, outside the outer accumulator.

Task latency included that work, while recorded calls, tokens, and cost omitted the inner generation. The follow-up gives both layers the same accounting client.

In this run, inner RAG generation contributed 21 calls and $0.3276, or 26.8% of the $1.2229 multi-agent generation cost. In the sequential route alone, its $0.1954 was almost half the generating cost. A trace showing three outer roles was actually paying for four model calls.

This also changes the proposed cost dashboard. Dividing the sequential route's $0.3958 by its two judge approvals would produce $0.198 per approval. Calling that cost per supported answer would hide the two false positives. With no trace-confirmed approvals, that ratio is undefined. It is not zero, and it is not a useful architecture price comparison.

What each pattern actually does

Sequential review is not automatic refinement. The baseline uses ResearcherWriterCriticPipeline(max_refinements=0), so all three roles run but no rewrite follows the critic. It is not the simpler two-agent researcher-writer pipeline also present in the repository. This matters for tasks such as q008 that explicitly ask the writer to add missing pieces after review.

The handoff state makes the routing inspectable: the researcher writes research_findings, the writer writes draft, and the critic writes critic_feedback and critic_score. Nodes return new dictionaries with updated fields. That is a useful contract for following execution; the field types do not establish whether the content is correct or the task complete.

The critic loop is bounded, but its budget matters. It routes a score below 4 on the implemented 1–5 scale back to the writer, with at most two rewrites:

research -> write -> critique -> [refine -> critique] -> [refine -> critique]

All nine feedback trials used both rewrites: seven outer calls plus one inner RAG call, eight generating calls in total. The critic sees the task and draft, without the raw retrieved evidence. Its score is a review signal, not an independent groundedness verdict. Some dataset wording also requests a 1–10 policy; the implemented 1–5 policy is what actually ran.

Parallel prompting is narrower than parallel research. SpecialistAgent.analyze() describes tools in its prompt, but never executes them. The runner also assigns generic labels such as Specialist_1 and concatenates responses without a synthesis call. That falls short of q009's requested database counts and q012's requested synthesized comparison table. Faster completion does not establish faster completion of the same work.

Arbitration cost changes with the workload. q006 uses three candidates and three serial scoring calls. q011 uses two candidates and one supervisor call. The resulting six-call versus three-call difference changes both candidate generation and arbitration. It cannot establish which resolver is more efficient at the same task. Generic candidate labels also proved weak role definitions: in the larger-budget check, some candidates treated their names as unresolved placeholders instead of producing proposals.

An output-budget sensitivity check

Every multi-agent trial had at least one generating call end at max_tokens: 168 of 174 calls did so. All outer research, writing, critique, specialist, and arbitration calls reached their limits. A capped critic is not necessarily a truncated final answer, but several final drafts visibly ended mid-sentence or inside code.

That is too large a confound to attribute the poor task-acceptance results to orchestration alone. After the baseline, I added a post-hoc, single-run sensitivity check, reusing the same model, index, tasks, and criteria. Each generating call was allowed at least 2,000 output tokens; the judge's limit stayed unchanged.

None of those 22 generating calls hit its limit. Outputs were longer, and q003 used one rewrite instead of two. Raising limits changes research, drafts, feedback, and routing; this does not isolate truncation as the sole cause of a failure.

The judge again approved two outputs. Trace inspection again found workflow gaps. q001 labelled its initial summary “Refined,” but did not show a revision after its critique. q002 was exactly the concatenation of three specialist findings, with no retrieval or synthesis node, even though the judge claimed a completed synthesis supported by evidence. Its text still included a handoff to a synthesizer that never ran.

The judge also made an error in the other direction. It rejected q003 for using deprecated @validator examples. That syntax appeared in an intermediate RAG response; the final answer actually used @field_validator and @model_validator after a real refinement. That mistaken reason is recorded without promoting the whole answer to verified correctness.

The larger budgets removed observed token-limit stops in this small check. They did not add the missing tools, fix role definitions, or make the judge reliable. Five single-run checks cannot support an architecture ranking.

The promised empty-result follow-up

After Part 4, readers asked whether ReAct would keep trying when zero rows was the correct answer, and whether a retry might quietly relax the original filters. This needed a separate experiment from the multi-agent routes.

The frozen SQL fixture contains eight rows and six cases:

• Three valid empties: an absent framework, an impossible fastapi AND react conjunction on a single-valued column, and an absent exact title.
• Three repairs: a framework case mismatch, a difficulty case mismatch, and a misspelled column. Expected matching IDs are [3, 7], [2], and [5] respectively.

Both agents received the starting SQL, were told that zero rows could be correct, and were explicitly required to preserve the requested filters. Each case ran three times per agent. An assistant inspected every final answer and every executed SQL statement against the fixture references; a second assistant independently checked them.

“Supported” here means the requested IDs and count are backed by executed SQL with the logical filters retained. It does not establish that every explanatory sentence was correct. The cost includes generation across all 18 trials per agent; it excludes judging and review labor. These are new SQL-only tasks, not a continuation of Part 4's multi-tool cost comparison.

ReAct did query again in two valid-empty trials. Both checked whether Django existed under a different capitalization or difficulty, then still returned the correct zero answer. Plan-and-Execute performed more checks, but they were written into its upfront plan. They are not evidence of reactive replanning. Its nine valid-empty trials made 14 additional SQL calls; ReAct's made two.

Broader diagnostic queries are not automatically harmful constraint drift. In the impossible-conjunction case, Plan-and-Execute checked the two frameworks separately, while retaining zero as the answer to the original conjunction. Some answers explicitly said that an OR would be a different question. No unrelated rows were substituted as the requested answer in these trials.

The judge wrongly rejected two valid repairs because they used LOWER() to normalize case, despite the task explicitly allowing case-insensitive difficulty matching. Those two false negatives are corrected in the assistant audit, alongside the original verdicts.

This answers the narrow reader question under these conditions: extra checking occurred, but it did not turn a correct empty answer into a wrong non-empty answer. The fixture is small, the starting query was supplied, and the prompts explicitly warned about preserving filters. It does not establish how often harmful drift occurs in production or with less explicit instructions.

What I would carry into a real workflow

Match the graph to the task's acceptance conditions. If the request requires database counts, a RAG description in a prompt cannot substitute for a query. If it requires a synthesis, concatenation is a different operation. If it requires revision after review, there must be a recorded post-review writer pass.

Make the cost boundary include tools that generate. Here, the omitted inner RAG client accounted for more than a quarter of multi-agent generation cost. A cost-per-supported-answer metric needs both that numerator and a denominator that survives inspection.

Treat critic scores and external judge verdicts as evidence to inspect. The critic lacked raw research evidence, and the external judge invented handoffs, missed an explicit case-normalization instruction, and confused an intermediate example with final output. Adding a second model did not eliminate the need to read traces.

Use output budgets as an experimental variable. Keeping small inherited limits made this run a poor test of the broader potential of multi-agent systems. Raising them made outputs more complete and changed loop counts, but the remaining workflow gaps still mattered. A paired task set, appropriate per-role budgets, and human-reviewed criteria would be needed for a stronger quality comparison.

Evidence, cost, and reproduction

The evidence README links the compressed trial and call records, raw summaries, installed environment, frozen criteria, and separate audit. The runner preserves model selection and records a durable budget reservation before each request. Resumed trials retain their execution commit and runner hash.

Reported usage cost was $1.87 for the baseline, sensitivity checks, and initial judge overhead: about $1.81 generating and $0.0617 judging. Two generator requests rejected before an API credit top-up returned no usage; conservatively reserving their maximum costs brought the local cap ledger to $1.9066. These are calculated token costs and budget reservations, not a provider invoice. Downstream error costs are not priced.

The 32 targeted offline tests check instrumentation, accounting, and fixtures; they do not certify answer quality. Both the criteria and audit remain assistant-produced. Three repetitions per task characterize this small workload, not a population of production requests. Retrieval also includes file-path and collection metadata, so a different checkout or fresh index can change results.

To start a new 72-trial experiment after the repository's normal dependency and local Chroma setup:

uv run python benchmarks/run_article_05_followup.py \
  --credentials-from /path/to/existing/checkout \
  --output results/data/a05_new_run \
  --run-label new_run --cap 4.8

Use a fresh output path for a new experiment and keep its ledger intact when resuming. The README gives the separate command options for the sensitivity check.