arxiv: 2603.28488 · v2 · submitted 2026-03-30 · 💻 cs.CL · cs.AI· cs.MA

Recognition: no theorem link

Courtroom-Style Multi-Agent Debate with Progressive RAG and Role-Switching for Controversial Claim Verification

Masnun Nuha Chowdhury , Nusrat Jahan Beg , Umme Hunny Khan , Syed Rifat Raiyan , Md Kamrul Hasan , Hasan Mahmud

Authors on Pith no claims yet

Pith reviewed 2026-05-14 22:09 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.MA

keywords multi-agent debateprogressive RAGclaim verificationhallucination mitigationadversarial deliberationCOVID-19 claimszero-shot evaluation

0 comments

The pith

Courtroom-style debate with progressive retrieval raises claim verification accuracy to 81.7 percent on Check-COVID.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces PROClaim, a framework that recasts controversial claim verification as an adversarial courtroom proceeding with plaintiff, defense, judge, and jury roles. It combines these roles with Progressive RAG that expands and refines the evidence pool turn by turn during the debate, plus evidence negotiation, self-reflection, and heterogeneous multi-judge voting. In zero-shot tests on the Check-COVID benchmark this yields 81.7 percent accuracy, a 10-point gain over ordinary multi-agent debate, with most of the lift coming from the progressive retrieval component. The central goal is to curb hallucinations and systematic biases that plague single-pass LLM verification.

Core claim

PROClaim reformulates verification as structured adversarial deliberation among specialized roles integrated with Progressive RAG that dynamically expands the evidence pool, together with evidence negotiation, self-reflection, and heterogeneous multi-judge aggregation; this delivers 81.7 percent zero-shot accuracy on Check-COVID and a 10-point margin over standard multi-agent debate, driven primarily by the progressive retrieval mechanism.

What carries the argument

The PROClaim framework: courtroom roles (Plaintiff, Defense, Judge) plus Progressive RAG that iteratively grows the evidence set during debate, combined with negotiation and multi-judge aggregation.

If this is right

Progressive evidence gathering during debate supplies the main accuracy lift of 7.5 points.
Role specialization and evidence negotiation improve calibration and reduce bias compared with unstructured debate.
Heterogeneous multi-judge aggregation increases output diversity and stability.
The method works in zero-shot settings without task-specific training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same role-plus-progressive-retrieval pattern could transfer to other high-stakes verification domains such as legal or medical claims.
If gains persist across base models, the architecture itself rather than model scale becomes the dominant factor.
Adding live web search inside the progressive retrieval loop would be a direct next test of the mechanism.

Load-bearing premise

The courtroom structure and ongoing evidence expansion supply genuine robustness against hallucinations rather than merely fitting the Check-COVID benchmark or the particular models used.

What would settle it

Running PROClaim on a second claim-verification dataset unrelated to COVID, such as political or scientific claims, and finding no accuracy gain over plain multi-agent debate would falsify the claimed robustness.

Figures

Figures reproduced from arXiv: 2603.28488 by Hasan Mahmud, Masnun Nuha Chowdhury, Md Kamrul Hasan, Nusrat Jahan Beg, Syed Rifat Raiyan, Umme Hunny Khan.

**Figure 1.** Figure 1: Overview of the pipeline Before retrieval, the raw claim is decomposed into atomic, independently testable premises (Hu et al., 2025a; Lawrence & Reed, 2017). This serves two purposes: first, decomposing complex claims allows the retrieval system to cast a wider and more targeted net; second, the resulting premises act as an explicit checklist for scoring argument completeness during self-reflection a… view at source ↗

**Figure 2.** Figure 2: Termination distribution and convergence speed across 360 debate instances. Dataset. To evaluate the framework’s capacity for adversarial resolution, we focus on the subset of the Check-COVID (Wang et al., 2023a) test set possessing definitive binary ground-truths (SUPPORT or REFUTE). This task formulation, which we term Adversarial Resolution of Hard-Binary Claims, ensures that the system is tested on … view at source ↗

**Figure 3.** Figure 3: P-RAG evidence novelty across debate rounds Representative reflection score trajectories (n 10 per panelall runs) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: Reflection score trajectories across plateau, judicial, and critic resolution patterns [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

**Figure 5.** Figure 5: Cost–accuracy Pareto front across system configurations. [PITH_FULL_IMAGE:figures/full_fig_p033_5.png] view at source ↗

read the original abstract

Large language models (LLMs) remain unreliable for high-stakes claim verification due to hallucinations and shallow reasoning. While retrieval-augmented generation (RAG) and multi-agent debate (MAD) address this, they are limited by one-pass retrieval and unstructured debate dynamics. We propose a courtroom-style multi-agent framework, PROClaim, that reformulates verification as a structured, adversarial deliberation. Our approach integrates specialized roles (e.g., Plaintiff, Defense, Judge) with Progressive RAG (P-RAG) to dynamically expand and refine the evidence pool during the debate. Furthermore, we employ evidence negotiation, self-reflection, and heterogeneous multi-judge aggregation to enforce calibration, robustness, and diversity. In zero-shot evaluations on the Check-COVID benchmark, PROClaim achieves 81.7% accuracy, outperforming standard multi-agent debate by 10.0 percentage points, with P-RAG driving the primary performance gains (+7.5 pp). We ultimately demonstrate that structural deliberation and model heterogeneity effectively mitigate systematic biases, providing a robust foundation for reliable claim verification. Our code and data are publicly available at https://github.com/mnc13/PROClaim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

PROClaim's courtroom roles plus progressive RAG lift accuracy 10 points over plain multi-agent debate on Check-COVID, but the single-benchmark results leave the robustness claim under-supported.

read the letter

The main point is that this setup reaches 81.7% zero-shot accuracy on Check-COVID and beats standard multi-agent debate by 10 points, with most of the lift coming from the progressive retrieval step. The courtroom framing (plaintiff, defense, judge roles), evidence negotiation during the debate, and final aggregation across heterogeneous judges is the actual addition here. It moves past one-shot retrieval and unstructured back-and-forth by making the process more iterative and role-constrained, which is a reasonable extension of existing MAD work. The public code and data are a plus for anyone who wants to reproduce or extend it. The reported numbers are clear enough on the surface and the partial ablation on P-RAG helps isolate what matters. That said, everything rests on one dataset whose claim distribution and evidence properties are not compared to others. No variance across seeds or model families is shown, and there is no control that gives the baseline the same retrieval budget to test whether the structure itself drives the gain or whether extra tokens do. If the improvement shrinks on a different claim set or under tighter retrieval controls, the causal story weakens. This is the kind of paper that fits a reading group focused on multi-agent verification systems. It is concrete enough to deserve peer review so the authors can add the missing controls and cross-dataset checks, even if the current scope limits how far the robustness claim can travel right now.

Referee Report

3 major / 1 minor

Summary. The manuscript introduces PROClaim, a courtroom-style multi-agent debate framework for controversial claim verification that incorporates specialized roles (Plaintiff, Defense, Judge), Progressive RAG (P-RAG) for dynamic evidence expansion, evidence negotiation, self-reflection, and heterogeneous multi-judge aggregation. In zero-shot evaluation on the Check-COVID benchmark, it reports 81.7% accuracy, a 10 percentage point improvement over standard multi-agent debate, with P-RAG credited for +7.5 pp gains, and claims that the structured adversarial deliberation and model heterogeneity mitigate hallucinations and biases.

Significance. If the reported gains prove robust, the work offers a structured approach to improving LLM reliability in high-stakes verification tasks by combining adversarial roles with progressive retrieval. The public release of code and data is a clear strength that aids reproducibility.

major comments (3)

[Experimental Evaluation] Experimental results section: the headline accuracy of 81.7% and the attributed gains (+10 pp over MAD, +7.5 pp from P-RAG) are reported without variance across random seeds, statistical significance tests, or full ablation tables isolating each component (e.g., P-RAG vs. fixed retrieval budget, role-switching vs. fixed roles).
[§4] §4 (Results and Discussion): the robustness claim against hallucinations and biases rests on a single benchmark (Check-COVID) with no cross-dataset evaluation or analysis of claim distribution/evidence density, leaving open whether gains are benchmark-specific rather than structural.
[Methodology] Methodology section: no control experiment equates total retrieval volume between P-RAG and the standard MAD baseline, so it is impossible to determine whether performance lifts derive from the progressive mechanism or simply from retrieving more tokens.

minor comments (1)

[Abstract and §3] The title mentions 'Role-Switching' but the abstract and method description emphasize fixed specialized roles; clarify whether dynamic switching occurs and in which section this is detailed.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their insightful comments, which have helped us improve the manuscript. We address each major comment below and indicate the revisions made.

read point-by-point responses

Referee: [Experimental Evaluation] Experimental results section: the headline accuracy of 81.7% and the attributed gains (+10 pp over MAD, +7.5 pp from P-RAG) are reported without variance across random seeds, statistical significance tests, or full ablation tables isolating each component (e.g., P-RAG vs. fixed retrieval budget, role-switching vs. fixed roles).

Authors: We agree that reporting variance and statistical tests would strengthen the results. In the revised version, we will run experiments with multiple random seeds (reporting mean and std), include statistical significance tests for the gains, and provide a more comprehensive ablation table that isolates the contribution of P-RAG, role-switching, and other elements. This addresses the concern directly. revision: yes
Referee: [§4] §4 (Results and Discussion): the robustness claim against hallucinations and biases rests on a single benchmark (Check-COVID) with no cross-dataset evaluation or analysis of claim distribution/evidence density, leaving open whether gains are benchmark-specific rather than structural.

Authors: This is a fair criticism. While we focused on Check-COVID as a representative high-stakes benchmark, we will add an analysis of claim distribution and evidence density in the revised manuscript to better characterize the dataset. However, performing full cross-dataset evaluations would require substantial additional experiments, which we cannot complete within the revision timeline. We will explicitly discuss the generalizability limitations and suggest it as future work. revision: partial
Referee: [Methodology] Methodology section: no control experiment equates total retrieval volume between P-RAG and the standard MAD baseline, so it is impossible to determine whether performance lifts derive from the progressive mechanism or simply from retrieving more tokens.

Authors: We appreciate this observation. To clarify, we will include a new control experiment in the revised manuscript where the baseline MAD is allowed to retrieve an equivalent total number of tokens (matched to P-RAG's average retrieval volume). This will help determine if the gains come from the progressive, dynamic nature of P-RAG or merely from increased retrieval volume. We expect the results to support the progressive mechanism's value, but the control will provide the necessary evidence. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmark results are externally validated

full rationale

The paper reports zero-shot accuracy on the public Check-COVID benchmark (81.7%, +10 pp over standard MAD, +7.5 pp attributed to P-RAG) using direct comparison to external baselines. No equations, fitted parameters, or self-referential definitions appear in the provided text; the central claims rest on measured performance against independent test data rather than any derivation that reduces to its own inputs by construction. Self-citations are absent from the abstract and results description, and the evaluation protocol does not invoke uniqueness theorems or ansatzes from prior author work.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The approach builds on existing LLM capabilities and RAG techniques without introducing new free parameters or invented entities; relies on domain assumptions about prompt engineering and retrieval dynamics.

axioms (2)

domain assumption Specialized prompting can make LLMs adopt distinct adversarial roles effectively
Fundamental to the multi-agent courtroom setup
domain assumption Iterative retrieval during debate improves evidence quality and reduces hallucinations
Basis for Progressive RAG contribution

pith-pipeline@v0.9.0 · 5536 in / 1270 out tokens · 61166 ms · 2026-05-14T22:09:54.875625+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · 2 internal anchors

[1]

The Faiss library

URLhttps://arxiv.org/abs/2401.08281. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. InForty-first international conference on machine learning, 2024. Wei Fan, JinYi Yoon, and Bo Ji. imad: Intelligent multi-agent debate for efficient and accurate...

work page internal anchor Pith review Pith/arXiv arXiv 2024
[2]

A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions , volume=

URLhttps://arxiv.org/abs/2505.17762. Shuzhi Gong, Richard O Sinnott, Jianzhong Qi, Cecile Paris, Preslav Nakov, and Zhuohan Xie. Multi-sourced, multi-agent evidence retrieval for fact-checking.arXiv preprint arXiv:2603.00267, 2026. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, S...

work page doi:10.1145/3703155 2026
[3]

doi: 10.18653/v1/2021.findings-emnlp

Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-emnlp

work page doi:10.18653/v1/2021.findings-emnlp 2021
[4]

Reflexion: Language Agents with Verbal Reinforcement Learning

URLhttps://aclanthology.org/2021.findings-emnlp.297/. Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.),Findings of the Association for Computational Linguistics: EMNLP 2023, pp....

work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2023.findings-emnlp.620 2021
[5]

innocent until proven guilty

URLhttps://arxiv.org/abs/2212.10509. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating llm generations with a panel of diverse models, 2024. URL https: //arxiv.org/abs/2404.18796. Gengyu Wang, Kate Harwood, Lawrence Chi...

work page arXiv 2024
[6]

Relevance: How directly does this evidence address the premises of the claim? (0.0 - 1.0)

work page
[7]

thinking

Credibility: Does the evidence come from a reliable scientific context or contain high-quality data? (0.0 - 1.0) H.3 Plaintiff Counsel Prompt Agent:GPT-5-mini System Prompt: You are the Plaintiff Counsel in a legal proceeding. Your role is to present arguments supporting the claim, interpret evidence favorably, challenge opposing arguments, and conduct ex...

work page
[8]

Logical Coherence: Argument flow and structure

work page
[9]

Evidence Coverage: How well they used admitted exhibits

work page
[10]

unresolved

Rebuttal Coverage: Did they address the opponent’s strongest points? Identify any premises that remain "unresolved" or under-supported. Provide actionable recommendations for both sides to improve their discovery and arguments. Respond ONLY in valid JSON format: { "plaintiff": { "logic": 0.0, "evidence": 0.0, "rebuttal": 0.0, "reasoning": "..." }, "defens...

work page
[11]

Logical Coherence: Evaluate the flow and structural integrity of your arguments

work page
[12]

Evidence Novelty: Have you introduced truly new information or just repeated old points?

work page
[13]

scores": {

Rebuttal Coverage: How effectively did you address the{opp side} counsel’s latest points? Identify: - Critical gaps in your current evidence base. - Premises you haven’t sufficiently supported. Respond ONLY in valid JSON format: { "scores": { "logic": 0.0-1.0, "novelty": 0.0-1.0, "rebuttal": 0.0-1.0 }, "flaws_identified": ["...", "..."], "discovery_need":...

work page
[14]

Hospitalized COVID-19 patients have detectable levels of cardiac biomarkers in- dicative of heart muscle cell damage

work page
[15]

The prevalence of elevated cardiac biomarkers in hospitalized COVID-19 patients is comparable to a control group without COVID-19

work page
[16]

Incidence rates of heart muscle cell damage in hospitalized COVID-19 patients are not higher than in patients with other viral respiratory infections

work page
[17]

Clinical studies on hospitalized COVID-19 patients do not report significant occur- rences of heart muscle cell damage

work page
[18]

There is no statistical association between COVID-19 infection severity and markers of heart muscle cell damage in hospitalized patients

work page
[19]

Autopsy findings of deceased hospitalized COVID-19 patients do not show evidence of heart muscle cell damage

work page
[20]

not an associated condition

Hospitalized COVID-19 patients with pre-existing cardiac conditions do not have higher rates of heart muscle cell damage compared to those without pre-existing conditions. 24 Preprint. Under review. Evidence Negotiation & Admission Initial RAG retrieved 5 candidate documents; negotiation and arbitration admitted21 ex- hibits(weights ranging 0.54–0.81), in...

work page 2022
[21]

Role-Play Consistency (0–10) During the role-switching consistency test (Section 2.7), an independent consistency ana- lyzer evaluates whether an agent successfully argues the opposing position using identical evidence without logically contradicting its prior arguments. The score reflects adherence to the persona constraints on a 10-point scale; lower sc...

work page
[22]

I concede,

Concession Rate We programmatically track explicit linguistic markers of concession and conversational yielding (e.g.,“I concede,” “you make a good point,” “I partially agree”) within the counsel transcripts. To normalize for varying debate lengths, the metric is reported as the frequency of such triggers per 1,000 generated words. A near-zero rate indica...

work page
[23]

The early-stopping criterion conservatively halts the debate if ∆S< 0.05 (stagnation)

Reflection Plateau (∆S) It is computed as the average absolute change in the cumulative self-reflection score (Stotal) between consecutive debate rounds: ∆S=|S (t) total −S (t−1) total | For a given round, the maximum possible change is ∼ 1.0 (depending on reflection ad- justments). The early-stopping criterion conservatively halts the debate if ∆S< 0.05 ...

work page
[24]

rubber- stamping

Judicial Conformity (Fleiss’κ) To measure whether the three structurally heterogeneous LLM judges exhibit “rubber- stamping” or independent evaluation, we calculate Fleiss’ Kappa (κ) over their final verdicts (SUPPORTED, NOT SUPPORTED, INCONCLUSIVE). A κ≈ 0.4513 indicates moderate, au- thentic agreement. While confirming they reach consensus on clear-cut ...

work page