REVIEW 5 major objections 5 minor 88 references
Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate
T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read DEAR regulates who each debater consults and how it responds, using group-level evidence to mitigate blind conformity in multi-agent LLM debate.
desk verdict DEAR is a plausible efficiency-plus-accuracy contribution to multi-agent debate, but its central claim about mitigating blind conformity is asserted rather than measured, and the experimental reporting lacks error bars and code. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the group evidence matrix built from pairwise scaled cosine similarities between embedded conclusions and embedded reasoning processes of debating LLMs. Subjective Logic decouples this evidence into consultation tendencies (belief masses) and uncertainty, yielding the observation for the Selection RL-Agent, which outputs independent Bernoulli decisions about which peer references to keep. A Dempster-Shafer fusion rule then combines the selected peers' evidence, and the Behavior RL-Agent maps the fused evidence to a generation behavior—temperature and top-p—for each debater. Heterogeneous-Agent Proximal Policy Optimization (HAPPO) trains the two agents end-to-end by chaining their surrogate advantages, so the behavior policy's update is explicitly conditioned on the selection policy's update.
What would settle it
Take a constructed debate in which two debaters share high embedding similarity but give different final answers, while a third diverges in embedding but is correct; DEAR's selection policy, trained on similarity-based evidence, would be expected to sever the useful reference and keep the misleading one. Observing such a trajectory—or a near-zero correlation across a large sample between similarity scores and answer agreement—would falsify the central claim.
Extended reading notes
Core claim
DEAR's central claim is that regulating debate relationships—which peers each LLM may consult and with what generation behavior—mitigates blind conformity more effectively than individual-level evaluation. The paper shows that a fully connected debate amplifies erroneous majority reasoning, while confidence- or perplexity-based filters can propagate unreliable individual signals. DEAR instead builds a group evidence matrix from pairwise scaled cosine similarities of conclusions and reasoning, decouples it into belief and uncertainty via Subjective Logic, and trains Selection and Behavior RL-Agents that decide, each round, which references to keep and how to sample the next answer. The joint HAPPO training treats this as a sequential decision process. On GPT-4o-mini, DEAR obtains 65.1% average math accuracy and 75.0% average QA accuracy, against 57.9% and 69.6% for the closest baselines, while using roughly half the tokens.
Load-bearing premise
The paper's load-bearing premise is that pairwise cosine similarity between embedded conclusions and reasoning is a reliable, task-agnostic measure of consensus and divergence; if that measure does not track who is actually right, both RL agents receive corrupted inputs and the claimed accuracy gain loses its foundation.
Editorial extensions
If this is right
- Multi-agent systems can be made more reliable without fine-tuning the LLMs themselves; only the two external policies are trained.
- Token costs drop because the selection policy severs needless peer references; on GPT-4o-mini average tokens fall from about 22K for standard MAD to about 11K for DEAR on math tasks.
- The framework transfers across domains: policies trained on math still improve QA accuracy, and vice versa, with some decay from distribution shift.
- The gains persist when scaling the number of debaters or debate rounds, suggesting that learned regulation rather than a fixed topology is what helps.
Reading between the lines
- A natural test is to replace the learned selection policy with random peer masking at the same token budget; DEAR's margin over that baseline would show whether learned selection, rather than mere sparsity, drives the gain.
- If cosine similarity of embeddings is a reliable group signal, the same evidence pipeline could extend to multimodal debate, using embeddings of generated text and images.
- The What-Who-How split suggests a general design pattern: any multi-agent reasoning system can be decomposed into perception of group state, routing of information, and control of generation behavior.
- The paper does not show that its similarity scores correlate with answer correctness; if they do not, policies trained on them may latch onto superficial phrasing.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DEAR, a framework for multi-agent debate (MAD) that regulates which peers each LLM references and how each LLM generates (temperature/top_p) via two RL agents trained jointly with HAPPO. Group evidence is computed as pairwise cosine similarity between LLM conclusions and reasoning embeddings (Eqs. 2-3), decoupled by Subjective Logic into belief masses and uncertainty (Section 4.2), used to select reference peers (Section 4.3), with DST-based fusion guiding behavior adjustment (Section 4.4). Experiments on four math and four QA benchmarks against several MAD baselines report higher accuracy and lower token consumption (Tables 1-3, Figures 2-4), including closed-source models (Table 2). The authors claim this demonstrates that regulating debate relationships mitigates blind conformity.
Significance. If the central claim held, the paper would make a useful contribution: shifting MAD regulation from individual confidence/perplexity to group-level interaction evidence is an interesting direction, and the reported token-efficiency gains are practically relevant. The framework is nontrivial: it couples two heterogeneous RL agents with HAPPO and connects Subjective Logic and DST in a coherent pipeline. The paper reports consistent accuracy improvements over a wide range of baselines and models, and the cross-domain transfer experiment (Table 8) is a genuinely informative check on task-agnosticism. However, the headline claim that DEAR mitigates blind conformity is not directly measured, and the lack of error bars or significance tests weakens the empirical case, as noted below.
major comments (5)
- [Section 1; Section 5.2; Appendix G] The paper's title and Section 1 claim that DEAR mitigates blind conformity, but no experiment measures blind conformity. All reported metrics are final-answer accuracy and token consumption, and the RL reward in Eq. (13) is assigned solely on final correctness. The only conformity evidence is a single anecdotal case study (Appendix G). The authors should either add a direct measurement—e.g., the rate at which initially-correct agents switch to an incorrect majority answer, or a comparison of answer-switch dynamics versus baselines—or reframe the central claim as improved accuracy/efficiency, not conformity mitigation.
- [Tables 1-3] All tables report single accuracy numbers without error bars, confidence intervals, or significance tests. AIME24 has only 30 problems and GPQA Diamond 150, so headline gains such as AIME24 16.7 vs 10.0 (Table 1) or GPQA 48.0 vs 42.0 could be within sampling noise. I ask for standard errors or hypothesis tests on at least the small benchmarks, and for a statement of how many seeds/runs the reported numbers average over.
- [Sections 4.1-4.4] The weakest assumption is that scaled cosine similarity between embeddings (Eqs. 2-3) is a valid, task-agnostic measure of consensus/divergence that can guide regulation. The paper does not show that similarity correlates with answer correctness, nor that the learned policies depend on similarity rather than spurious patterns (e.g., persona-specific wording or length). A concrete validation test would be to compare the distribution of similarity scores for correct-incorrect, correct-correct, and incorrect-incorrect answer pairs, and to check whether the Selection RL-Agent's masking decisions change when embeddings are replaced by a different encoder.
- [Section 4.5 and Eq. (13)] The RL formulation assigns a single terminal reward R_debate for the final answer, so the two RL agents are optimized only for accuracy, not for any conformity-related objective. This means the observed accuracy gains could arise from filtering noisy peers, increasing output diversity via temperature/top_p, or other mechanisms unrelated to reducing blind conformity. To support the mechanism claim, the reward or evaluation should include a conformity metric, or the paper should report ablation on initial-round correctness versus final-round correctness.
- [Appendix E.2, Table 7] The comparison with RUMAD (6 rounds, 6 LLMs) uses configurations different from the main experiments (3 rounds, 3 LLMs in Table 4), and the paper does not state whether DEAR was retrained for the 6-agent/6-round setting or evaluated with the same policy. Without this detail, the claimed superiority over RUMAD is hard to interpret, since RL policies trained for one debate configuration may not transfer to another.
minor comments (5)
- [Section 5.1, Implementation Details] The text says DEAR is trained on GSM8K for math and ARC-C for QA, but does not state the size of the training subset or how many episodes/seeds were used for RL training; Appendix D gives total training time but not the number of trajectories.
- [Equation (7)] The notation p_{i<-j} is used both as the Bernoulli parameter and as the preference matrix entry P; consider renaming one for clarity.
- [Section 4.4, Definition 4.4.1] The DST combination rule is written for two belief sets, but Eq. (8) applies it to V sets; the paper should clarify whether the combination is associative in this formulation or specify the order of pairwise fusion.
- [Appendix G] The case study is illustrative but contains an apparent inconsistency: Agent 2's Round 1 reasoning says 'Agent 1 misinterprets' while Agent 3's Round 1 reasoning says 'Agent 1 correctly identifies'; this could confuse readers about the actual trajectories.
- [Throughout] There are occasional typos and formatting issues, e.g., 'the results confirm' at the start of Section 5.4, and unrendered math symbols in the extracted text; a careful proofread is recommended.
Circularity Check
No significant circularity: DEAR's accuracy and token-efficiency claims are trained and evaluated against external benchmark labels, and the self-citations are not load-bearing.
full rationale
The paper's central empirical claims are not circular. The two RL agents are trained with a reward R_debate that is assigned solely from correctness of the final answer (Eq. 13), using supervised signal from standard benchmarks (GSM8K and ARC-C training splits), and then evaluated on held-out test sets (Tables 1-3). The reported accuracy and token consumption are therefore external benchmark outcomes, not quantities forced by the method's construction. The group evidence (Eqs. 2-3) is a cosine-similarity feature used as input to the policies; the policies are not fitted to reproduce that similarity, and success is judged by downstream accuracy. There is no fitted parameter that is later renamed as a prediction. The self-citations [72-74] appear only in Appendix C.2 to support the commonplace statement that reward sharing is standard in cooperative MARL; they are not used to justify the central derivation, to import a uniqueness theorem, or to smuggle in an ansatz. The paper's assertion that regulating debate relationships 'is the key to mitigating blind conformity' is not directly tested, since no experiment measures the rate at which initially-correct agents abandon their answer for an incorrect majority; that is a construct-validity and evidence gap, not a circular reduction, because the measured accuracy gains do not by definition entail the conformity mechanism. Overall, the derivation is self-contained against external benchmarks and no circular step was found.
Assumptions & free parameters
free parameters (5)
- SL base weight =
V (3 in main experiments)
- Number of debate agents V =
3
- Maximum debate rounds T =
3
- Temperature and Top_p action bounds =
T in [0,1], Top_p in [0,1]
- RL hyperparameters =
lr 5e-4, gamma 1.0, lambda 0.95, clip 0.2, buffer 128
assumptions (5)
- domain assumption Cosine similarity in a pretrained embedding space captures task-agnostic consensus and divergence between LLM responses.
- domain assumption LLM behavior can be meaningfully steered by temperature and top-p alone.
- domain assumption Dempster's combination rule is valid for fusing LLM beliefs.
- ad hoc to paper Subjective Logic's zero-sum constraint with base weight V models consultation tendency and uncertainty.
- domain assumption Final-answer accuracy is an adequate proxy for reduced blind conformity.
invented entities (1)
-
Debate Relationships
Cite this review
Pith. "Pith review of Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate." pith.science (2026). https://pith.science/paper/YHZUEWK6
@misc{pith2026260803648,
author = {Pith},
title = {Pith review of: Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate},
year = {2026},
howpublished = {\url{https://pith.science/paper/YHZUEWK6}},
note = {Machine review of arXiv:2608.03648}
}
read the original abstract
Multi-Agent Debate (MAD) improves the reasoning performance of Large Language Models (LLMs) through multi-round interaction. However, LLMs in MAD are highly susceptible to blind conformity. Existing individual evaluation methods, typically based on confidence or perplexity, fail to reflect the correctness of reasoning and may even exacerbate blind conformity. To address this, we shift the perspective from individual evaluation to group interaction. We define mutual referencing among LLMs as \textbf{Debate Relationships} and recognize that regulating these relationships is the key to mitigating blind conformity. In this paper, we propose a novel framework for \textbf{D}ynamically r\textbf{E}gulating deb\textbf{A}te \textbf{R}elationships (DEAR) from the group perspective. At first, DEAR quantifies consensus and divergence as \textit{group evidence} to capture the debate state. Then, DEAR operates through three stages: 1) What: perceiving group consultation tendency and uncertainty; 2) Who: introducing a Selection RL-Agent to dynamically select reference peers; and 3) How: adopting a Behavior RL-Agent to adaptively adjust generation behaviors. Notably, we formulate the execution of the two RL-Agents as a sequential decision-making process, jointly optimizing via multi-agent reinforcement learning. Extensive experiments demonstrate that DEAR achieves superior performance while significantly reducing token consumption.
Figures
Reference graph
Works this paper leans on
-
[1]
Improving factuality and reasoning in language models through multiagent debate
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. InForty-first international conference on machine learning, 2024
2024
-
[2]
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate.arXiv preprint arXiv:2308.07201, 2023
arXiv 2023
-
[3]
Debate-to-write: A persona-driven multi-agent framework for diverse argument generation
Zhe Hu, Hou Pong Chan, Jing Li, and Yu Yin. Debate-to-write: A persona-driven multi-agent framework for diverse argument generation. InProceedings of the 31st International Conference on Computational Linguistics, pages 4689–4703, 2025
2025
-
[4]
Encouraging divergent thinking in large language models through multi-agent debate
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 17889–17904, 2024
2024
-
[5]
Debate helps supervise unreliable experts.arXiv preprint arXiv:2311.08702, 2023
Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, and Samuel R Bowman. Debate helps supervise unreliable experts.arXiv preprint arXiv:2311.08702, 2023
arXiv 2023
-
[6]
Towards detecting llms hallucination via markov chain-based multi-agent debate framework
Xiaoxi Sun, Jinpeng Li, Yan Zhong, Dongyan Zhao, and Rui Yan. Towards detecting llms hallucination via markov chain-based multi-agent debate framework. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025
2025
-
[7]
An empirical study of group conformity in multi-agent systems
Min Choi, Keonwoo Kim, Sungwon Chae, and Sangyeop Baek. An empirical study of group conformity in multi-agent systems. InFindings of the Association for Computational Linguistics: ACL 2025, pages 5123–5139, 2025
2025
-
[8]
Mengdie Flora Wang, Haochen Xie, Guanghui Wang, Aijing Gao, Guang Yang, Ziyuan Li, Qucy Wei Qiu, Fangwei Han, Hengzhi Qiu, Yajing Huang, et al. From debate to decision: Conformal social choice for safe multi-agent deliberation.arXiv preprint arXiv:2604.07667, 2026
arXiv 2026
Show all 88 references
-
[9]
Epistemic context learning: Building trust the right way in llm-based multi-agent systems.arXiv preprint arXiv:2601.21742, 2026
Ruiwen Zhou, Maojia Song, Xiaobao Wu, Sitao Cheng, Xunjian Yin, Yuxi Xie, Zhuoqun Hao, Wenyue Hua, Liangming Pan, Soujanya Poria, et al. Epistemic context learning: Building trust the right way in llm-based multi-agent systems.arXiv preprint arXiv:2601.21742, 2026
2026
-
[10]
Cortexdebate: Debating sparsely and equally for multi-agent debate
Yiliu Sun, Zicheng Zhao, Sheng Wan, and Chen Gong. Cortexdebate: Debating sparsely and equally for multi-agent debate. InFindings of the Association for Computational Linguistics: ACL 2025, pages 9503–9523, 2025
2025
-
[11]
Confidencecal: Enhancing llms reliability through confidence calibration in multi-agent debate
Yilin Bai. Confidencecal: Enhancing llms reliability through confidence calibration in multi-agent debate. In2024 10th International Conference on Big Data and Information Analytics (BigDIA), pages 221–226. IEEE, 2024
2024
-
[12]
Demystifying multi-agent debate: The role of confidence and diversity.arXiv preprint arXiv:2601.19921, 2026
Xiaochen Zhu, Caiqi Zhang, Yizhou Chi, Tom Stafford, Nigel Collier, and Andreas Vlachos. Demystifying multi-agent debate: The role of confidence and diversity.arXiv preprint arXiv:2601.19921, 2026
2026 arXiv
-
[13]
Mad-logic: Multi-agent debate enhances symbolic translation and reasoning
Haocheng Yang, Fengxiang Cheng, Tianjun Yao, Mengyue Yang, Jiajun Chai, Xiaohan Wang, Guojun Yin, Wei Lin, Soummya Kar, Fenrong Liu, et al. Mad-logic: Multi-agent debate enhances symbolic translation and reasoning. InThe Fourteenth International Conference on Learning Represen...
2026
-
[14]
Multi-agent debate with memory masking.arXiv preprint arXiv:2603.20215, 2026
Hongduan Tian, Xiao Feng, Ziyuan Zhao, Xiangyu Zhu, Rolan Yan, and Bo Han. Multi-agent debate with memory masking.arXiv preprint arXiv:2603.20215, 2026
2026
-
[15]
Confidence calibration and rationalization for llms via multi-agent deliberation.arXiv preprint arXiv:2404.09127, 2024
Ruixin Yang, Dheeraj Rajagopal, Shirley Anugrah Hayati, Bin Hu, and Dongyeop Kang. Confidence calibration and rationalization for llms via multi-agent deliberation.arXiv preprint arXiv:2404.09127, 2024. 10
2024 arXiv
-
[16]
Perplexity cannot always tell right from wrong.arXiv preprint arXiv:2601.22950, 2026
Petar Veliˇckovi´c, Federico Barbero, Christos Perivolaropoulos, Simon Osindero, and Razvan Pascanu. Perplexity cannot always tell right from wrong.arXiv preprint arXiv:2601.22950, 2026
2026
-
[17]
Can perplexity reflect large language model’s ability in long text understanding?arXiv preprint arXiv:2405.06105, 2024
Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model’s ability in long text understanding?arXiv preprint arXiv:2405.06105, 2024
2024 arXiv
-
[18]
When two llms debate, both think they’ll win.arXiv preprint arXiv:2505.19184, 2025
Pradyumna Shyama Prasad and Minh Nhat Nguyen. When two llms debate, both think they’ll win.arXiv preprint arXiv:2505.19184, 2025
2025 arXiv
-
[19]
Springer Publishing Company, Incorporated, 2018
Audun Jsang.Subjective Logic: A formalism for reasoning under uncertainty. Springer Publishing Company, Incorporated, 2018
2018
-
[20]
Trust region policy optimisation in multi-agent reinforcement learning.arXiv preprint arXiv:2109.11251, 2021
Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning.arXiv preprint arXiv:2109.11251, 2021
2021 arXiv
-
[21]
Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022
2022
-
[22]
Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022
2022 arXiv
-
[23]
Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023
2023
-
[24]
Graph of thoughts: Solving elaborate problems with large language models
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. InProceedings of the AAAI confer...
2024
-
[25]
Tab-cot: Zero-shot tabular chain of thought
Jin Ziqi and Wei Lu. Tab-cot: Zero-shot tabular chain of thought. InFindings of the Association for Computational Linguistics: ACL 2023, pages 10259–10277, 2023
2023
-
[26]
Automatic prompt augmentation and selection with chain- of-thought from labeled data
KaShun Shum, Shizhe Diao, and Tong Zhang. Automatic prompt augmentation and selection with chain- of-thought from labeled data. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 12113–12139, 2023
2023
-
[27]
Advancing collaborative debates with role differentiation through multi-agent reinforcement learning
Haoran Li, Ziyi Su, Yun Xue, Zhiliang Tian, Yiping Song, and Minlie Huang. Advancing collaborative debates with role differentiation through multi-agent reinforcement learning. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1:...
2025
-
[28]
From debate to equilibrium: Belief-driven multi-agent llm reasoning via bayesian nash equilibrium.arXiv preprint arXiv:2506.08292, 2025
Xie Yi, Zhanke Zhou, Chentao Cao, Qiyu Niu, Tongliang Liu, and Bo Han. From debate to equilibrium: Belief-driven multi-agent llm reasoning via bayesian nash equilibrium.arXiv preprint arXiv:2506.08292, 2025
2025 arXiv
-
[29]
Free-mad: Consensus-free multi-agent debate.arXiv preprint arXiv:2509.11035, 2025
Yu Cui, Hang Fu, Haibin Zhang, Licheng Wang, and Cong Zuo. Free-mad: Consensus-free multi-agent debate.arXiv preprint arXiv:2509.11035, 2025
2025 arXiv
-
[30]
Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021
-
[31]
Pal: Program-aided language models.arXiv preprint arXiv:2211.10435, 2022
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models.arXiv preprint arXiv:2211.10435, 2022
2022 arXiv
-
[32]
Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023
2023 arXiv
-
[33]
Think you have solved question answering? try arc, the ai2 reasoning challenge
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018
2018 arXiv
-
[34]
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290, 2024
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95...
2024
-
[35]
Truthfulqa: Measuring how models mimic human falsehoods
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. InProceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3214–3252, 2022
2022
-
[36]
Gpqa: A graduate-level google-proof q&a benchmark
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst conference on language modeling, 2024
2024
-
[37]
Breaking mental set to improve reasoning through diverse multi-agent debate
Yexiang Liu, Jie Cao, Zekun Li, Ran He, and Tieniu Tan. Breaking mental set to improve reasoning through diverse multi-agent debate. InThe Thirteenth International Conference on Learning Representations, 2025
2025
-
[38]
Understanding agent scaling in llm-based multi-agent systems via diversity.arXiv preprint arXiv:2602.03794, 2026
Yingxuan Yang, Chengrui Qu, Muning Wen, Laixi Shi, Ying Wen, Weinan Zhang, Adam Wierman, and Shangding Gu. Understanding agent scaling in llm-based multi-agent systems via diversity.arXiv preprint arXiv:2602.03794, 2026
2026
-
[39]
Hear both sides: Efficient multi-agent debate via diversity-aware message retention.arXiv preprint arXiv:2603.20640, 2026
Manh Nguyen, Anh Nguyen, Dung Nguyen, Svetha Venkatesh, and Hung Le. Hear both sides: Efficient multi-agent debate via diversity-aware message retention.arXiv preprint arXiv:2603.20640, 2026
2026 arXiv
-
[40]
Dy- nadebate: Breaking homogeneity in multi-agent debate with dynamic path generation.arXiv preprint arXiv:2601.05746, 2026
Zhenghao Li, Zhi Zheng, Wei Chen, Jielun Zhao, Yong Chen, Tong Xu, and Enhong Chen. Dy- nadebate: Breaking homogeneity in multi-agent debate with dynamic path generation.arXiv preprint arXiv:2601.05746, 2026
2026
-
[41]
Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024
2024
-
[42]
Qwen3 technical report, 2025
Qwen Team. Qwen3 technical report, 2025. URLhttps://arxiv.org/abs/2505.09388
2025 arXiv
-
[43]
Thang Luong, Dawsen Hwang, Hoang H. Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu H. Trinh, Quoc V . Le, and J...
2025
-
[44]
Medxpertqa: Benchmarking expert-level medical reasoning and understanding
Yuxin Zuo, Shang Qu, Yifei Li, Zhang-Ren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. In International Conference on Machine Learning, pages 80961–80990. PMLR, 2025
2025
-
[45]
Epistemic gain, aleatoric cost: Uncertainty decomposition in multi-agent debate for math reasoning.arXiv preprint arXiv:2603.01221, 2026
Dan Qiao, Binbin Chen, Fengyu Cai, Jianlong Chen, Wenhao Li, Fuxin Jiang, Zuzhi Chen, Hongyuan Zha, Tieying Zhang, and Baoxiang Wang. Epistemic gain, aleatoric cost: Uncertainty decomposition in multi-agent debate for math reasoning.arXiv preprint arXiv:2603.01221, 2026
2026 arXiv
-
[46]
Agentscourt: Building judicial decision-making agents with court debate simulation and legal knowledge augmentation
Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Kang Liu, and Jun Zhao. Agentscourt: Building judicial decision-making agents with court debate simulation and legal knowledge augmentation. InFindings of the Association for Computational Lin...
2024
-
[47]
Debatrix: Multi-dimensional debate judge with iterative chronological analysis based on llm
Jingcong Liang, Rong Ye, Meng Han, Ruofei Lai, Xinyu Zhang, Xuan-Jing Huang, and Zhongyu Wei. Debatrix: Multi-dimensional debate judge with iterative chronological analysis based on llm. InFindings of the Association for Computational Linguistics: ACL 2024, pages 14575–14595, 2024
2024
-
[48]
Improving multi-agent debate with sparse communication topology
Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene Ie. Improving multi-agent debate with sparse communication topology. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 7281–7294, 2024
2024
-
[49]
S2-mad: Breaking the token barrier to enhance multi-agent debate efficiency
Yuting Zeng, Weizhe Huang, Lei Jiang, Tongxuan Liu, Xitai Jin, Chen Tianying Tiana, Jing Li, and Xiaohua Xu. S2-mad: Breaking the token barrier to enhance multi-agent debate efficiency. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associat...
2025
-
[50]
Rumad: Reinforcement-unifying multi-agent debate.arXiv preprint arXiv:2602.23864, 2026
Chao Wang, Han Lin, Huaze Tang, Huijing Lin, and Wenbo Ding. Rumad: Reinforcement-unifying multi-agent debate.arXiv preprint arXiv:2602.23864, 2026
2026
-
[51]
Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion.arXiv preprint arXiv:2409.14051, 2024
Tongxuan Liu, Xingyu Wang, Weizhe Huang, Wenjiang Xu, Yuting Zeng, Lei Jiang, Hailong Yang, and Jing Li. Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion.arXiv preprint arXiv:2409.14051, 2024. 12
2024
-
[52]
Debate only when necessary: Adaptive multiagent collaboration for efficient llm reasoning.arXiv preprint arXiv:2504.05047, 2025
Sugyeong Eo, Hyeonseok Moon, Evelyn Hayoon Zi, Chanjun Park, and Heuiseok Lim. Debate only when necessary: Adaptive multiagent collaboration for efficient llm reasoning.arXiv preprint arXiv:2504.05047, 2025
2025 arXiv
-
[53]
imad: Intelligent multi-agent debate for efficient and accurate llm inference
Wei Fan, JinYi Yoon, and Bo Ji. imad: Intelligent multi-agent debate for efficient and accurate llm inference. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29403–29411, 2026
2026
-
[54]
Peacemaker or troublemaker: How sycophancy shapes multi-agent debate.arXiv preprint arXiv:2509.23055, 2025
Binwei Yao, Chao Shang, Wanyu Du, Jianfeng He, Ruixue Lian, Yi Zhang, Hang Su, Sandesh Swamy, and Yanjun Qi. Peacemaker or troublemaker: How sycophancy shapes multi-agent debate.arXiv preprint arXiv:2509.23055, 2025
2025
-
[55]
Upper and lower probabilities induced by a multivalued mapping
Arthur P Dempster. Upper and lower probabilities induced by a multivalued mapping. InClassic works of the Dempster-Shafer theory of belief functions, pages 57–72. Springer, 2008
2008
-
[56]
Dempster-shafer theory.Encyclopedia of artificial intelligence, 1(330-331):3, 1992
Glenn Shafer. Dempster-shafer theory.Encyclopedia of artificial intelligence, 1(330-331):3, 1992
1992
-
[57]
Interpretation and fusion of hyper opinions in subjective logic
Audun Jøsang and Robin Hankin. Interpretation and fusion of hyper opinions in subjective logic. In2012 15th International Conference on Information Fusion, pages 1225–1232. IEEE, 2012
2012
-
[58]
The modified dempster-shafer approach to classification.IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 27(1):96–104, 2002
Dale Fixsen and Ronald PS Mahler. The modified dempster-shafer approach to classification.IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 27(1):96–104, 2002
2002
-
[59]
A k-nearest neighbor classification rule based on dempster-shafer theory.IEEE transactions on systems, man, and cybernetics, 25(5):804–813, 1995
Thierry Denoeux. A k-nearest neighbor classification rule based on dempster-shafer theory.IEEE transactions on systems, man, and cybernetics, 25(5):804–813, 1995
1995
-
[60]
Combining the classification results of independent classifiers based on the dempster/shafer theory of evidence
Eberhard Mandler and Jürgen Schümann. Combining the classification results of independent classifiers based on the dempster/shafer theory of evidence. InMachine Intelligence and Pattern Recognition, volume 7, pages 381–393. Elsevier, 1988
1988
-
[61]
A neural network classifier based on dempster-shafer theory.IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 30(2):131–150, 2000
Thierry Denoeux. A neural network classifier based on dempster-shafer theory.IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 30(2):131–150, 2000
2000
-
[62]
A new correlation belief function in dempster-shafer evidence theory and its application in classification.Scientific Reports, 13(1):7609, 2023
Yongchuan Tang, Xu Zhang, Ying Zhou, Yubo Huang, and Deyun Zhou. A new correlation belief function in dempster-shafer evidence theory and its application in classification.Scientific Reports, 13(1):7609, 2023
2023
-
[63]
Trusted multi-view classification with dynamic evidential fusion.IEEE transactions on pattern analysis and machine intelligence, 45(2): 2551–2566, 2022
Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification with dynamic evidential fusion.IEEE transactions on pattern analysis and machine intelligence, 45(2): 2551–2566, 2022
2022
-
[64]
Detecting misbehaviors of large vision-language models by evidential uncertainty quantification.arXiv preprint arXiv:2602.05535, 2026
Tao Huang, Rui Wang, Xiaofei Liu, Yi Qin, Li Duan, and Liping Jing. Detecting misbehaviors of large vision-language models by evidential uncertainty quantification.arXiv preprint arXiv:2602.05535, 2026
2026
-
[65]
From observation to understanding: Front-door adjustments with uncertainty calibration for enhancing egocentric reasoning in lvlms
Shenshen Li, Wenxin Meng, Lei Wang, Hao Yang, Chong Peng, Peng Yan, Fumin Shen, Jingkuan Song, Heng Tao Shen, and Xing Xu. From observation to understanding: Front-door adjustments with uncertainty calibration for enhancing egocentric reasoning in lvlms. InFindings of the Asso...
2025
-
[66]
Not all inconsistency is equal: Decomposing lvlm uncertainty into belief divergence and belief conflict
Jie Shi, Xiaodong Yue, Wei Liu, Yufei Chen, and Feifan Dong. Not all inconsistency is equal: Decomposing lvlm uncertainty into belief divergence and belief conflict. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 25339–25347, 2026
2026
-
[67]
Visual hallucination detection in large vision-language models via evidential conflict.International Journal of Approximate Reasoning, 186: 109507, 2025
Tao Huang, Zhekun Liu, Rui Wang, Yang Zhang, and Liping Jing. Visual hallucination detection in large vision-language models via evidential conflict.International Journal of Approximate Reasoning, 186: 109507, 2025
2025
-
[68]
Llm-based corroborating and refuting evidence retrieval for scientific claim verification.arXiv preprint arXiv:2503.07937, 2025
Siyuan Wang, James R Foulds, Md Osman Gani, and Shimei Pan. Llm-based corroborating and refuting evidence retrieval for scientific claim verification.arXiv preprint arXiv:2503.07937, 2025
2025 arXiv
-
[69]
Jailjudge: A comprehensive jailbreak judge benchmark with multi-agent enhanced explanation evaluation framework.arXiv preprint arXiv:2410.12855, 2024
Fan Liu, Yue Feng, Zhao Xu, Lixin Su, Xinyu Ma, Dawei Yin, and Hao Liu. Jailjudge: A comprehensive jailbreak judge benchmark with multi-agent enhanced explanation evaluation framework.arXiv preprint arXiv:2410.12855, 2024
2024 arXiv
-
[70]
Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017
2017 arXiv
-
[71]
The surpris- ing effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surpris- ing effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022. 13
2022
-
[72]
Think how your teammates think: Active inference can benefit decentralized execution
Hao Wu, Shoucheng Song, Chang Yao, Sheng Han, Huaiyu Wan, Youfang Lin, and Kai Lv. Think how your teammates think: Active inference can benefit decentralized execution. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29749–29757, 2026
2026
-
[73]
Code: Communication delay-tolerant multi-agent collaboration via dual alignment of intent and timeliness
Shoucheng Song, Youfang Lin, Sheng Han, Chang Yao, Hao Wu, Shuo Wang, and Kai Lv. Code: Communication delay-tolerant multi-agent collaboration via dual alignment of intent and timeliness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23304–...
2025
-
[74]
From general relation patterns to task-specific decision-making in continual multi-agent coordination.arXiv preprint arXiv:2507.06004, 2025
Chang Yao, Youfang Lin, Shoucheng Song, Hao Wu, Yuqing Ma, Shang Han, and Kai Lv. From general relation patterns to task-specific decision-making in continual multi-agent coordination.arXiv preprint arXiv:2507.06004, 2025
2025 arXiv
-
[75]
The llama 3 herd of models
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024
2024 arXiv
-
[76]
Chatglm: A family of large language models from glm-130b to glm-4 all tools
Team Glm, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024
2024 arXiv
-
[77]
Deepseek-v3
Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025
2025 arXiv
-
[78]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025
2025 arXiv
-
[79]
Let’s think step by step
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. A Detailed Related Works A.1 Single-Agent Reasoning. Large Language Models (LLMs) ha...
2009 arXiv
-
[80]
MLC [27] employs reinforcement learning for task-specific role differentiation
introduce judge panels and multi-dimensional evaluation criteria for collaborative assessment, respectively. MLC [27] employs reinforcement learning for task-specific role differentiation. ECON
-
[81]
CortexDebate [ 10] and ConfidenceCal [11] attempt to select opinions using explicit confidence as soft weights
reformulates the collaborative process as an incomplete information game, seeking Bayesian Nash Equilibrium (BNE) through hierarchical reinforcement learning. CortexDebate [ 10] and ConfidenceCal [11] attempt to select opinions using explicit confidence as soft weights. MAD-M²
-
[82]
YES,” “NO,
adopts a dual-component policy for memory filtering. The subjective component involves agents labeling historical records manually. The objective component utilizes model perplexity as the 14 primary screening criterion. Sparse MAD (S-MAD) [48] and Selective Sparse MAD (S2-MAD...
2024
-
[84]
Output the final result inside\boxed{}. Example:\boxed{42}
Summarize your reasoning into a conclusion that includes the final answer. {Output Instruction} Where the output instructions are: •Math: “Output the final result inside\boxed{}. Example:\boxed{42}.” • QA: “Output the final choice selection using the format: \boxed{X} where X ...
-
[85]
BEFORE reading peers, rederive the key steps independently
-
[86]
Only consider an objection if it identifies a SPECIFIC step where a particular number, formula, or factual claim is wrong
THEN scan peers. Only consider an objection if it identifies a SPECIFIC step where a particular number, formula, or factual claim is wrong. Instruction:
-
[88]
Summarize your reasoning into a conclusion that includes the final answer. {Output Instruction} Case 2: Without Peer Responses Note that when no peer responses are available, the peer context block is replaced with: Debate Round Prompt (No Peer Responses Available) {Persona Pr...
-
[89]
Provide your reasoning process
-
[90]
{Output Instruction} 25
Summarize your reasoning into a conclusion that includes the final answer. {Output Instruction} 25
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.