Pith. sign in

REVIEW 5 major objections 5 minor 88 references

Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read DEAR regulates who each debater consults and how it responds, using group-level evidence to mitigate blind conformity in multi-agent LLM debate.

desk verdict DEAR is a plausible efficiency-plus-accuracy contribution to multi-agent debate, but its central claim about mitigating blind conformity is asserted rather than measured, and the experimental reporting lacks error bars and code. read the letter →

arxiv 2608.03648 v1 pith:YHZUEWK6 submitted 2026-08-04 cs.MA

classification cs.MA
keywords multi-agentdebateblindconformityrelationshipsgroupevidencereinforcementlearningsubjectivelogicHAPPOLLMreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's thesis is that blind conformity in multi-agent LLM debate is not a failure of any single debater's confidence but a group-level phenomenon, and that the right lever is the relationship between debaters: who is allowed to influence whom and how strongly. DEAR operationalizes this by extracting pairwise similarity between each debater's conclusion and reasoning, converting these similarities into consensus and uncertainty signals with Subjective Logic, and then training two policies—a selection policy that masks which peers are referenced and a behavior policy that sets temperature and top-p—jointly with HAPPO. Reported results show higher average accuracy and substantially lower token consumption than standard debate and individual-evaluation baselines across math and QA tasks. The contribution, if correct, is a way to get more reliable multi-model reasoning without trusting confidence scores or perplexity, which the paper argues are both misleading.

What carries the argument

The central object is the group evidence matrix built from pairwise scaled cosine similarities between embedded conclusions and embedded reasoning processes of debating LLMs. Subjective Logic decouples this evidence into consultation tendencies (belief masses) and uncertainty, yielding the observation for the Selection RL-Agent, which outputs independent Bernoulli decisions about which peer references to keep. A Dempster-Shafer fusion rule then combines the selected peers' evidence, and the Behavior RL-Agent maps the fused evidence to a generation behavior—temperature and top-p—for each debater. Heterogeneous-Agent Proximal Policy Optimization (HAPPO) trains the two agents end-to-end by chaining their surrogate advantages, so the behavior policy's update is explicitly conditioned on the selection policy's update.

What would settle it

Take a constructed debate in which two debaters share high embedding similarity but give different final answers, while a third diverges in embedding but is correct; DEAR's selection policy, trained on similarity-based evidence, would be expected to sever the useful reference and keep the misleading one. Observing such a trajectory—or a near-zero correlation across a large sample between similarity scores and answer agreement—would falsify the central claim.

Watch

Extended reading notes

Core claim

DEAR's central claim is that regulating debate relationships—which peers each LLM may consult and with what generation behavior—mitigates blind conformity more effectively than individual-level evaluation. The paper shows that a fully connected debate amplifies erroneous majority reasoning, while confidence- or perplexity-based filters can propagate unreliable individual signals. DEAR instead builds a group evidence matrix from pairwise scaled cosine similarities of conclusions and reasoning, decouples it into belief and uncertainty via Subjective Logic, and trains Selection and Behavior RL-Agents that decide, each round, which references to keep and how to sample the next answer. The joint HAPPO training treats this as a sequential decision process. On GPT-4o-mini, DEAR obtains 65.1% average math accuracy and 75.0% average QA accuracy, against 57.9% and 69.6% for the closest baselines, while using roughly half the tokens.

Load-bearing premise

The paper's load-bearing premise is that pairwise cosine similarity between embedded conclusions and reasoning is a reliable, task-agnostic measure of consensus and divergence; if that measure does not track who is actually right, both RL agents receive corrupted inputs and the claimed accuracy gain loses its foundation.

Editorial extensions

If this is right

  • Multi-agent systems can be made more reliable without fine-tuning the LLMs themselves; only the two external policies are trained.
  • Token costs drop because the selection policy severs needless peer references; on GPT-4o-mini average tokens fall from about 22K for standard MAD to about 11K for DEAR on math tasks.
  • The framework transfers across domains: policies trained on math still improve QA accuracy, and vice versa, with some decay from distribution shift.
  • The gains persist when scaling the number of debaters or debate rounds, suggesting that learned regulation rather than a fixed topology is what helps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test is to replace the learned selection policy with random peer masking at the same token budget; DEAR's margin over that baseline would show whether learned selection, rather than mere sparsity, drives the gain.
  • If cosine similarity of embeddings is a reliable group signal, the same evidence pipeline could extend to multimodal debate, using embeddings of generated text and images.
  • The What-Who-How split suggests a general design pattern: any multi-agent reasoning system can be decomposed into perception of group state, routing of information, and control of generation behavior.
  • The paper does not show that its similarity scores correlate with answer correctness; if they do not, policies trained on them may latch onto superficial phrasing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DEAR, a framework for multi-agent debate (MAD) that regulates which peers each LLM references and how each LLM generates (temperature/top_p) via two RL agents trained jointly with HAPPO. Group evidence is computed as pairwise cosine similarity between LLM conclusions and reasoning embeddings (Eqs. 2-3), decoupled by Subjective Logic into belief masses and uncertainty (Section 4.2), used to select reference peers (Section 4.3), with DST-based fusion guiding behavior adjustment (Section 4.4). Experiments on four math and four QA benchmarks against several MAD baselines report higher accuracy and lower token consumption (Tables 1-3, Figures 2-4), including closed-source models (Table 2). The authors claim this demonstrates that regulating debate relationships mitigates blind conformity.

Significance. If the central claim held, the paper would make a useful contribution: shifting MAD regulation from individual confidence/perplexity to group-level interaction evidence is an interesting direction, and the reported token-efficiency gains are practically relevant. The framework is nontrivial: it couples two heterogeneous RL agents with HAPPO and connects Subjective Logic and DST in a coherent pipeline. The paper reports consistent accuracy improvements over a wide range of baselines and models, and the cross-domain transfer experiment (Table 8) is a genuinely informative check on task-agnosticism. However, the headline claim that DEAR mitigates blind conformity is not directly measured, and the lack of error bars or significance tests weakens the empirical case, as noted below.

major comments (5)
  1. [Section 1; Section 5.2; Appendix G] The paper's title and Section 1 claim that DEAR mitigates blind conformity, but no experiment measures blind conformity. All reported metrics are final-answer accuracy and token consumption, and the RL reward in Eq. (13) is assigned solely on final correctness. The only conformity evidence is a single anecdotal case study (Appendix G). The authors should either add a direct measurement—e.g., the rate at which initially-correct agents switch to an incorrect majority answer, or a comparison of answer-switch dynamics versus baselines—or reframe the central claim as improved accuracy/efficiency, not conformity mitigation.
  2. [Tables 1-3] All tables report single accuracy numbers without error bars, confidence intervals, or significance tests. AIME24 has only 30 problems and GPQA Diamond 150, so headline gains such as AIME24 16.7 vs 10.0 (Table 1) or GPQA 48.0 vs 42.0 could be within sampling noise. I ask for standard errors or hypothesis tests on at least the small benchmarks, and for a statement of how many seeds/runs the reported numbers average over.
  3. [Sections 4.1-4.4] The weakest assumption is that scaled cosine similarity between embeddings (Eqs. 2-3) is a valid, task-agnostic measure of consensus/divergence that can guide regulation. The paper does not show that similarity correlates with answer correctness, nor that the learned policies depend on similarity rather than spurious patterns (e.g., persona-specific wording or length). A concrete validation test would be to compare the distribution of similarity scores for correct-incorrect, correct-correct, and incorrect-incorrect answer pairs, and to check whether the Selection RL-Agent's masking decisions change when embeddings are replaced by a different encoder.
  4. [Section 4.5 and Eq. (13)] The RL formulation assigns a single terminal reward R_debate for the final answer, so the two RL agents are optimized only for accuracy, not for any conformity-related objective. This means the observed accuracy gains could arise from filtering noisy peers, increasing output diversity via temperature/top_p, or other mechanisms unrelated to reducing blind conformity. To support the mechanism claim, the reward or evaluation should include a conformity metric, or the paper should report ablation on initial-round correctness versus final-round correctness.
  5. [Appendix E.2, Table 7] The comparison with RUMAD (6 rounds, 6 LLMs) uses configurations different from the main experiments (3 rounds, 3 LLMs in Table 4), and the paper does not state whether DEAR was retrained for the 6-agent/6-round setting or evaluated with the same policy. Without this detail, the claimed superiority over RUMAD is hard to interpret, since RL policies trained for one debate configuration may not transfer to another.
minor comments (5)
  1. [Section 5.1, Implementation Details] The text says DEAR is trained on GSM8K for math and ARC-C for QA, but does not state the size of the training subset or how many episodes/seeds were used for RL training; Appendix D gives total training time but not the number of trajectories.
  2. [Equation (7)] The notation p_{i<-j} is used both as the Bernoulli parameter and as the preference matrix entry P; consider renaming one for clarity.
  3. [Section 4.4, Definition 4.4.1] The DST combination rule is written for two belief sets, but Eq. (8) applies it to V sets; the paper should clarify whether the combination is associative in this formulation or specify the order of pairwise fusion.
  4. [Appendix G] The case study is illustrative but contains an apparent inconsistency: Agent 2's Round 1 reasoning says 'Agent 1 misinterprets' while Agent 3's Round 1 reasoning says 'Agent 1 correctly identifies'; this could confuse readers about the actual trajectories.
  5. [Throughout] There are occasional typos and formatting issues, e.g., 'the results confirm' at the start of Section 5.4, and unrendered math symbols in the extracted text; a careful proofread is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DEAR's accuracy and token-efficiency claims are trained and evaluated against external benchmark labels, and the self-citations are not load-bearing.

full rationale

The paper's central empirical claims are not circular. The two RL agents are trained with a reward R_debate that is assigned solely from correctness of the final answer (Eq. 13), using supervised signal from standard benchmarks (GSM8K and ARC-C training splits), and then evaluated on held-out test sets (Tables 1-3). The reported accuracy and token consumption are therefore external benchmark outcomes, not quantities forced by the method's construction. The group evidence (Eqs. 2-3) is a cosine-similarity feature used as input to the policies; the policies are not fitted to reproduce that similarity, and success is judged by downstream accuracy. There is no fitted parameter that is later renamed as a prediction. The self-citations [72-74] appear only in Appendix C.2 to support the commonplace statement that reward sharing is standard in cooperative MARL; they are not used to justify the central derivation, to import a uniqueness theorem, or to smuggle in an ansatz. The paper's assertion that regulating debate relationships 'is the key to mitigating blind conformity' is not directly tested, since no experiment measures the rate at which initially-correct agents abandon their answer for an incorrect majority; that is a construct-validity and evidence gap, not a circular reduction, because the measured accuracy gains do not by definition entail the conformity mechanism. Overall, the derivation is self-contained against external benchmarks and no circular step was found.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim rests on an unvalidated similarity-based evidence measure, on learned RL policies whose architectures are not shipped, and on an accuracy-only reward. The subjective logic and Dempster-Shafer machinery introduces choices, such as base weight and independence of evidence, that are not empirically grounded.

free parameters (5)
  • SL base weight = V (3 in main experiments)
    Used in Eq. (5) to convert evidence scores into belief masses and uncertainty; the choice changes uncertainty magnitudes and is not empirically justified.
  • Number of debate agents V = 3
    Main results fix three LLMs; scaling experiments vary this, but central tables use 3.
  • Maximum debate rounds T = 3
    Terminal round for reward; the behavior policy only acts within this horizon.
  • Temperature and Top_p action bounds = T in [0,1], Top_p in [0,1]
    Eqs. (10) and (11); chosen ranges constrain the behavior policy output.
  • RL hyperparameters = lr 5e-4, gamma 1.0, lambda 0.95, clip 0.2, buffer 128
    Table 4; standard PPO-family settings, chosen by the authors.
assumptions (5)
  • domain assumption Cosine similarity in a pretrained embedding space captures task-agnostic consensus and divergence between LLM responses.
    Section 4.1, Eqs. (2) and (3). No validation against correctness; the entire regulation pipeline depends on this.
  • domain assumption LLM behavior can be meaningfully steered by temperature and top-p alone.
    Section 4.4; the Behavior RL-Agent's only outputs are T and Top_p.
  • domain assumption Dempster's combination rule is valid for fusing LLM beliefs.
    Definition 4.4.1 and Eq. (8) require independent belief sources; LLMs share training data and prompts, so independence is questionable.
  • ad hoc to paper Subjective Logic's zero-sum constraint with base weight V models consultation tendency and uncertainty.
    Section 4.2, Eqs. (4) and (5); chosen to split evidence, not derived from data.
  • domain assumption Final-answer accuracy is an adequate proxy for reduced blind conformity.
    Reward in Eq. (13) and all experiments measure accuracy, not whether agents abandon independent reasoning.
invented entities (1)
  • Debate Relationships
    purpose: Reifies the pattern of mutual referencing among LLMs as the object that DEAR regulates.
    Defined in Section 1 as a conceptual construct; not directly measured, only manipulated through selection masks and indirectly evaluated through accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate." pith.science (2026). https://pith.science/paper/YHZUEWK6

@misc{pith2026260803648,
  author       = {Pith},
  title        = {Pith review of: Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YHZUEWK6}},
  note         = {Machine review of arXiv:2608.03648}
}
read the original abstract

Multi-Agent Debate (MAD) improves the reasoning performance of Large Language Models (LLMs) through multi-round interaction. However, LLMs in MAD are highly susceptible to blind conformity. Existing individual evaluation methods, typically based on confidence or perplexity, fail to reflect the correctness of reasoning and may even exacerbate blind conformity. To address this, we shift the perspective from individual evaluation to group interaction. We define mutual referencing among LLMs as \textbf{Debate Relationships} and recognize that regulating these relationships is the key to mitigating blind conformity. In this paper, we propose a novel framework for \textbf{D}ynamically r\textbf{E}gulating deb\textbf{A}te \textbf{R}elationships (DEAR) from the group perspective. At first, DEAR quantifies consensus and divergence as \textit{group evidence} to capture the debate state. Then, DEAR operates through three stages: 1) What: perceiving group consultation tendency and uncertainty; 2) Who: introducing a Selection RL-Agent to dynamically select reference peers; and 3) How: adopting a Behavior RL-Agent to adaptively adjust generation behaviors. Notably, we formulate the execution of the two RL-Agents as a sequential decision-making process, jointly optimizing via multi-agent reinforcement learning. Extensive experiments demonstrate that DEAR achieves superior performance while significantly reducing token consumption.

Figures

Figures reproduced from arXiv: 2608.03648 by the authors.

Figure 1
Figure 1. The overall framework of DEAR. We employ a two-round debate as an example to detail [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Effect of scaling the number of agents in the case of [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Effect of scaling the number of debate rounds in the case of [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablation study of the proposed DEAR on four datasets. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 33 canonical work pages

  1. [1]

    Improving factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. InForty-first international conference on machine learning, 2024

  2. [2]

    Chateval: Towards better llm-based evaluators through multi-agent debate.arXiv preprint arXiv:2308.07201, 2023

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate.arXiv preprint arXiv:2308.07201, 2023

  3. [3]

    Debate-to-write: A persona-driven multi-agent framework for diverse argument generation

    Zhe Hu, Hou Pong Chan, Jing Li, and Yu Yin. Debate-to-write: A persona-driven multi-agent framework for diverse argument generation. InProceedings of the 31st International Conference on Computational Linguistics, pages 4689–4703, 2025

  4. [4]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 17889–17904, 2024

  5. [5]

    Debate helps supervise unreliable experts.arXiv preprint arXiv:2311.08702, 2023

    Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, and Samuel R Bowman. Debate helps supervise unreliable experts.arXiv preprint arXiv:2311.08702, 2023

  6. [6]

    Towards detecting llms hallucination via markov chain-based multi-agent debate framework

    Xiaoxi Sun, Jinpeng Li, Yan Zhong, Dongyan Zhao, and Rui Yan. Towards detecting llms hallucination via markov chain-based multi-agent debate framework. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025

  7. [7]

    An empirical study of group conformity in multi-agent systems

    Min Choi, Keonwoo Kim, Sungwon Chae, and Sangyeop Baek. An empirical study of group conformity in multi-agent systems. InFindings of the Association for Computational Linguistics: ACL 2025, pages 5123–5139, 2025

  8. [8]

    From debate to decision: Conformal social choice for safe multi-agent deliberation.arXiv preprint arXiv:2604.07667, 2026

    Mengdie Flora Wang, Haochen Xie, Guanghui Wang, Aijing Gao, Guang Yang, Ziyuan Li, Qucy Wei Qiu, Fangwei Han, Hengzhi Qiu, Yajing Huang, et al. From debate to decision: Conformal social choice for safe multi-agent deliberation.arXiv preprint arXiv:2604.07667, 2026

Show all 88 references
  1. [9]

    Epistemic context learning: Building trust the right way in llm-based multi-agent systems.arXiv preprint arXiv:2601.21742, 2026

    Ruiwen Zhou, Maojia Song, Xiaobao Wu, Sitao Cheng, Xunjian Yin, Yuxi Xie, Zhuoqun Hao, Wenyue Hua, Liangming Pan, Soujanya Poria, et al. Epistemic context learning: Building trust the right way in llm-based multi-agent systems.arXiv preprint arXiv:2601.21742, 2026

  2. [10]

    Cortexdebate: Debating sparsely and equally for multi-agent debate

    Yiliu Sun, Zicheng Zhao, Sheng Wan, and Chen Gong. Cortexdebate: Debating sparsely and equally for multi-agent debate. InFindings of the Association for Computational Linguistics: ACL 2025, pages 9503–9523, 2025

  3. [11]

    Confidencecal: Enhancing llms reliability through confidence calibration in multi-agent debate

    Yilin Bai. Confidencecal: Enhancing llms reliability through confidence calibration in multi-agent debate. In2024 10th International Conference on Big Data and Information Analytics (BigDIA), pages 221–226. IEEE, 2024

  4. [12]

    Demystifying multi-agent debate: The role of confidence and diversity.arXiv preprint arXiv:2601.19921, 2026

    Xiaochen Zhu, Caiqi Zhang, Yizhou Chi, Tom Stafford, Nigel Collier, and Andreas Vlachos. Demystifying multi-agent debate: The role of confidence and diversity.arXiv preprint arXiv:2601.19921, 2026

  5. [13]

    Mad-logic: Multi-agent debate enhances symbolic translation and reasoning

    Haocheng Yang, Fengxiang Cheng, Tianjun Yao, Mengyue Yang, Jiajun Chai, Xiaohan Wang, Guojun Yin, Wei Lin, Soummya Kar, Fenrong Liu, et al. Mad-logic: Multi-agent debate enhances symbolic translation and reasoning. InThe Fourteenth International Conference on Learning Represen...

  6. [14]

    Multi-agent debate with memory masking.arXiv preprint arXiv:2603.20215, 2026

    Hongduan Tian, Xiao Feng, Ziyuan Zhao, Xiangyu Zhu, Rolan Yan, and Bo Han. Multi-agent debate with memory masking.arXiv preprint arXiv:2603.20215, 2026

  7. [15]

    Confidence calibration and rationalization for llms via multi-agent deliberation.arXiv preprint arXiv:2404.09127, 2024

    Ruixin Yang, Dheeraj Rajagopal, Shirley Anugrah Hayati, Bin Hu, and Dongyeop Kang. Confidence calibration and rationalization for llms via multi-agent deliberation.arXiv preprint arXiv:2404.09127, 2024. 10

  8. [16]

    Perplexity cannot always tell right from wrong.arXiv preprint arXiv:2601.22950, 2026

    Petar Veliˇckovi´c, Federico Barbero, Christos Perivolaropoulos, Simon Osindero, and Razvan Pascanu. Perplexity cannot always tell right from wrong.arXiv preprint arXiv:2601.22950, 2026

  9. [17]

    Can perplexity reflect large language model’s ability in long text understanding?arXiv preprint arXiv:2405.06105, 2024

    Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model’s ability in long text understanding?arXiv preprint arXiv:2405.06105, 2024

  10. [18]

    When two llms debate, both think they’ll win.arXiv preprint arXiv:2505.19184, 2025

    Pradyumna Shyama Prasad and Minh Nhat Nguyen. When two llms debate, both think they’ll win.arXiv preprint arXiv:2505.19184, 2025

  11. [19]

    Springer Publishing Company, Incorporated, 2018

    Audun Jsang.Subjective Logic: A formalism for reasoning under uncertainty. Springer Publishing Company, Incorporated, 2018

  12. [20]

    Trust region policy optimisation in multi-agent reinforcement learning.arXiv preprint arXiv:2109.11251, 2021

    Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning.arXiv preprint arXiv:2109.11251, 2021

  13. [21]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

  14. [22]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

  15. [23]

    Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023

  16. [24]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. InProceedings of the AAAI confer...

  17. [25]

    Tab-cot: Zero-shot tabular chain of thought

    Jin Ziqi and Wei Lu. Tab-cot: Zero-shot tabular chain of thought. InFindings of the Association for Computational Linguistics: ACL 2023, pages 10259–10277, 2023

  18. [26]

    Automatic prompt augmentation and selection with chain- of-thought from labeled data

    KaShun Shum, Shizhe Diao, and Tong Zhang. Automatic prompt augmentation and selection with chain- of-thought from labeled data. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 12113–12139, 2023

  19. [27]

    Advancing collaborative debates with role differentiation through multi-agent reinforcement learning

    Haoran Li, Ziyi Su, Yun Xue, Zhiliang Tian, Yiping Song, and Minlie Huang. Advancing collaborative debates with role differentiation through multi-agent reinforcement learning. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1:...

  20. [28]

    From debate to equilibrium: Belief-driven multi-agent llm reasoning via bayesian nash equilibrium.arXiv preprint arXiv:2506.08292, 2025

    Xie Yi, Zhanke Zhou, Chentao Cao, Qiyu Niu, Tongliang Liu, and Bo Han. From debate to equilibrium: Belief-driven multi-agent llm reasoning via bayesian nash equilibrium.arXiv preprint arXiv:2506.08292, 2025

  21. [29]

    Free-mad: Consensus-free multi-agent debate.arXiv preprint arXiv:2509.11035, 2025

    Yu Cui, Hang Fu, Haibin Zhang, Licheng Wang, and Cong Zuo. Free-mad: Consensus-free multi-agent debate.arXiv preprint arXiv:2509.11035, 2025

  22. [30]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

  23. [31]

    Pal: Program-aided language models.arXiv preprint arXiv:2211.10435, 2022

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models.arXiv preprint arXiv:2211.10435, 2022

  24. [32]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

  25. [33]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

  26. [34]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95...

  27. [35]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. InProceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3214–3252, 2022

  28. [36]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst conference on language modeling, 2024

  29. [37]

    Breaking mental set to improve reasoning through diverse multi-agent debate

    Yexiang Liu, Jie Cao, Zekun Li, Ran He, and Tieniu Tan. Breaking mental set to improve reasoning through diverse multi-agent debate. InThe Thirteenth International Conference on Learning Representations, 2025

  30. [38]

    Understanding agent scaling in llm-based multi-agent systems via diversity.arXiv preprint arXiv:2602.03794, 2026

    Yingxuan Yang, Chengrui Qu, Muning Wen, Laixi Shi, Ying Wen, Weinan Zhang, Adam Wierman, and Shangding Gu. Understanding agent scaling in llm-based multi-agent systems via diversity.arXiv preprint arXiv:2602.03794, 2026

  31. [39]

    Hear both sides: Efficient multi-agent debate via diversity-aware message retention.arXiv preprint arXiv:2603.20640, 2026

    Manh Nguyen, Anh Nguyen, Dung Nguyen, Svetha Venkatesh, and Hung Le. Hear both sides: Efficient multi-agent debate via diversity-aware message retention.arXiv preprint arXiv:2603.20640, 2026

  32. [40]

    Dy- nadebate: Breaking homogeneity in multi-agent debate with dynamic path generation.arXiv preprint arXiv:2601.05746, 2026

    Zhenghao Li, Zhi Zheng, Wei Chen, Jielun Zhao, Yong Chen, Tong Xu, and Enhong Chen. Dy- nadebate: Breaking homogeneity in multi-agent debate with dynamic path generation.arXiv preprint arXiv:2601.05746, 2026

  33. [41]

    Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

  34. [42]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025. URLhttps://arxiv.org/abs/2505.09388

  35. [43]

    Thang Luong, Dawsen Hwang, Hoang H. Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu H. Trinh, Quoc V . Le, and J...

  36. [44]

    Medxpertqa: Benchmarking expert-level medical reasoning and understanding

    Yuxin Zuo, Shang Qu, Yifei Li, Zhang-Ren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. In International Conference on Machine Learning, pages 80961–80990. PMLR, 2025

  37. [45]

    Epistemic gain, aleatoric cost: Uncertainty decomposition in multi-agent debate for math reasoning.arXiv preprint arXiv:2603.01221, 2026

    Dan Qiao, Binbin Chen, Fengyu Cai, Jianlong Chen, Wenhao Li, Fuxin Jiang, Zuzhi Chen, Hongyuan Zha, Tieying Zhang, and Baoxiang Wang. Epistemic gain, aleatoric cost: Uncertainty decomposition in multi-agent debate for math reasoning.arXiv preprint arXiv:2603.01221, 2026

  38. [46]

    Agentscourt: Building judicial decision-making agents with court debate simulation and legal knowledge augmentation

    Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Kang Liu, and Jun Zhao. Agentscourt: Building judicial decision-making agents with court debate simulation and legal knowledge augmentation. InFindings of the Association for Computational Lin...

  39. [47]

    Debatrix: Multi-dimensional debate judge with iterative chronological analysis based on llm

    Jingcong Liang, Rong Ye, Meng Han, Ruofei Lai, Xinyu Zhang, Xuan-Jing Huang, and Zhongyu Wei. Debatrix: Multi-dimensional debate judge with iterative chronological analysis based on llm. InFindings of the Association for Computational Linguistics: ACL 2024, pages 14575–14595, 2024

  40. [48]

    Improving multi-agent debate with sparse communication topology

    Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene Ie. Improving multi-agent debate with sparse communication topology. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 7281–7294, 2024

  41. [49]

    S2-mad: Breaking the token barrier to enhance multi-agent debate efficiency

    Yuting Zeng, Weizhe Huang, Lei Jiang, Tongxuan Liu, Xitai Jin, Chen Tianying Tiana, Jing Li, and Xiaohua Xu. S2-mad: Breaking the token barrier to enhance multi-agent debate efficiency. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associat...

  42. [50]

    Rumad: Reinforcement-unifying multi-agent debate.arXiv preprint arXiv:2602.23864, 2026

    Chao Wang, Han Lin, Huaze Tang, Huijing Lin, and Wenbo Ding. Rumad: Reinforcement-unifying multi-agent debate.arXiv preprint arXiv:2602.23864, 2026

  43. [51]

    Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion.arXiv preprint arXiv:2409.14051, 2024

    Tongxuan Liu, Xingyu Wang, Weizhe Huang, Wenjiang Xu, Yuting Zeng, Lei Jiang, Hailong Yang, and Jing Li. Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion.arXiv preprint arXiv:2409.14051, 2024. 12

  44. [52]

    Debate only when necessary: Adaptive multiagent collaboration for efficient llm reasoning.arXiv preprint arXiv:2504.05047, 2025

    Sugyeong Eo, Hyeonseok Moon, Evelyn Hayoon Zi, Chanjun Park, and Heuiseok Lim. Debate only when necessary: Adaptive multiagent collaboration for efficient llm reasoning.arXiv preprint arXiv:2504.05047, 2025

  45. [53]

    imad: Intelligent multi-agent debate for efficient and accurate llm inference

    Wei Fan, JinYi Yoon, and Bo Ji. imad: Intelligent multi-agent debate for efficient and accurate llm inference. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29403–29411, 2026

  46. [54]

    Peacemaker or troublemaker: How sycophancy shapes multi-agent debate.arXiv preprint arXiv:2509.23055, 2025

    Binwei Yao, Chao Shang, Wanyu Du, Jianfeng He, Ruixue Lian, Yi Zhang, Hang Su, Sandesh Swamy, and Yanjun Qi. Peacemaker or troublemaker: How sycophancy shapes multi-agent debate.arXiv preprint arXiv:2509.23055, 2025

  47. [55]

    Upper and lower probabilities induced by a multivalued mapping

    Arthur P Dempster. Upper and lower probabilities induced by a multivalued mapping. InClassic works of the Dempster-Shafer theory of belief functions, pages 57–72. Springer, 2008

  48. [56]

    Dempster-shafer theory.Encyclopedia of artificial intelligence, 1(330-331):3, 1992

    Glenn Shafer. Dempster-shafer theory.Encyclopedia of artificial intelligence, 1(330-331):3, 1992

  49. [57]

    Interpretation and fusion of hyper opinions in subjective logic

    Audun Jøsang and Robin Hankin. Interpretation and fusion of hyper opinions in subjective logic. In2012 15th International Conference on Information Fusion, pages 1225–1232. IEEE, 2012

  50. [58]

    The modified dempster-shafer approach to classification.IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 27(1):96–104, 2002

    Dale Fixsen and Ronald PS Mahler. The modified dempster-shafer approach to classification.IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 27(1):96–104, 2002

  51. [59]

    A k-nearest neighbor classification rule based on dempster-shafer theory.IEEE transactions on systems, man, and cybernetics, 25(5):804–813, 1995

    Thierry Denoeux. A k-nearest neighbor classification rule based on dempster-shafer theory.IEEE transactions on systems, man, and cybernetics, 25(5):804–813, 1995

  52. [60]

    Combining the classification results of independent classifiers based on the dempster/shafer theory of evidence

    Eberhard Mandler and Jürgen Schümann. Combining the classification results of independent classifiers based on the dempster/shafer theory of evidence. InMachine Intelligence and Pattern Recognition, volume 7, pages 381–393. Elsevier, 1988

  53. [61]

    A neural network classifier based on dempster-shafer theory.IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 30(2):131–150, 2000

    Thierry Denoeux. A neural network classifier based on dempster-shafer theory.IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 30(2):131–150, 2000

  54. [62]

    A new correlation belief function in dempster-shafer evidence theory and its application in classification.Scientific Reports, 13(1):7609, 2023

    Yongchuan Tang, Xu Zhang, Ying Zhou, Yubo Huang, and Deyun Zhou. A new correlation belief function in dempster-shafer evidence theory and its application in classification.Scientific Reports, 13(1):7609, 2023

  55. [63]

    Trusted multi-view classification with dynamic evidential fusion.IEEE transactions on pattern analysis and machine intelligence, 45(2): 2551–2566, 2022

    Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification with dynamic evidential fusion.IEEE transactions on pattern analysis and machine intelligence, 45(2): 2551–2566, 2022

  56. [64]

    Detecting misbehaviors of large vision-language models by evidential uncertainty quantification.arXiv preprint arXiv:2602.05535, 2026

    Tao Huang, Rui Wang, Xiaofei Liu, Yi Qin, Li Duan, and Liping Jing. Detecting misbehaviors of large vision-language models by evidential uncertainty quantification.arXiv preprint arXiv:2602.05535, 2026

  57. [65]

    From observation to understanding: Front-door adjustments with uncertainty calibration for enhancing egocentric reasoning in lvlms

    Shenshen Li, Wenxin Meng, Lei Wang, Hao Yang, Chong Peng, Peng Yan, Fumin Shen, Jingkuan Song, Heng Tao Shen, and Xing Xu. From observation to understanding: Front-door adjustments with uncertainty calibration for enhancing egocentric reasoning in lvlms. InFindings of the Asso...

  58. [66]

    Not all inconsistency is equal: Decomposing lvlm uncertainty into belief divergence and belief conflict

    Jie Shi, Xiaodong Yue, Wei Liu, Yufei Chen, and Feifan Dong. Not all inconsistency is equal: Decomposing lvlm uncertainty into belief divergence and belief conflict. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 25339–25347, 2026

  59. [67]

    Visual hallucination detection in large vision-language models via evidential conflict.International Journal of Approximate Reasoning, 186: 109507, 2025

    Tao Huang, Zhekun Liu, Rui Wang, Yang Zhang, and Liping Jing. Visual hallucination detection in large vision-language models via evidential conflict.International Journal of Approximate Reasoning, 186: 109507, 2025

  60. [68]

    Llm-based corroborating and refuting evidence retrieval for scientific claim verification.arXiv preprint arXiv:2503.07937, 2025

    Siyuan Wang, James R Foulds, Md Osman Gani, and Shimei Pan. Llm-based corroborating and refuting evidence retrieval for scientific claim verification.arXiv preprint arXiv:2503.07937, 2025

  61. [69]

    Jailjudge: A comprehensive jailbreak judge benchmark with multi-agent enhanced explanation evaluation framework.arXiv preprint arXiv:2410.12855, 2024

    Fan Liu, Yue Feng, Zhao Xu, Lixin Su, Xinyu Ma, Dawei Yin, and Hao Liu. Jailjudge: A comprehensive jailbreak judge benchmark with multi-agent enhanced explanation evaluation framework.arXiv preprint arXiv:2410.12855, 2024

  62. [70]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

  63. [71]

    The surpris- ing effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surpris- ing effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022. 13

  64. [72]

    Think how your teammates think: Active inference can benefit decentralized execution

    Hao Wu, Shoucheng Song, Chang Yao, Sheng Han, Huaiyu Wan, Youfang Lin, and Kai Lv. Think how your teammates think: Active inference can benefit decentralized execution. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29749–29757, 2026

  65. [73]

    Code: Communication delay-tolerant multi-agent collaboration via dual alignment of intent and timeliness

    Shoucheng Song, Youfang Lin, Sheng Han, Chang Yao, Hao Wu, Shuo Wang, and Kai Lv. Code: Communication delay-tolerant multi-agent collaboration via dual alignment of intent and timeliness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23304–...

  66. [74]

    From general relation patterns to task-specific decision-making in continual multi-agent coordination.arXiv preprint arXiv:2507.06004, 2025

    Chang Yao, Youfang Lin, Shoucheng Song, Hao Wu, Yuqing Ma, Shang Han, and Kai Lv. From general relation patterns to task-specific decision-making in continual multi-agent coordination.arXiv preprint arXiv:2507.06004, 2025

  67. [75]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

  68. [76]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    Team Glm, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024

  69. [77]

    Deepseek-v3

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

  70. [78]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

  71. [79]

    Let’s think step by step

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. A Detailed Related Works A.1 Single-Agent Reasoning. Large Language Models (LLMs) ha...

  72. [80]

    MLC [27] employs reinforcement learning for task-specific role differentiation

    introduce judge panels and multi-dimensional evaluation criteria for collaborative assessment, respectively. MLC [27] employs reinforcement learning for task-specific role differentiation. ECON

  73. [81]

    CortexDebate [ 10] and ConfidenceCal [11] attempt to select opinions using explicit confidence as soft weights

    reformulates the collaborative process as an incomplete information game, seeking Bayesian Nash Equilibrium (BNE) through hierarchical reinforcement learning. CortexDebate [ 10] and ConfidenceCal [11] attempt to select opinions using explicit confidence as soft weights. MAD-M²

  74. [82]

    YES,” “NO,

    adopts a dual-component policy for memory filtering. The subjective component involves agents labeling historical records manually. The objective component utilizes model perplexity as the 14 primary screening criterion. Sparse MAD (S-MAD) [48] and Selective Sparse MAD (S2-MAD...

  75. [84]

    Output the final result inside\boxed{}. Example:\boxed{42}

    Summarize your reasoning into a conclusion that includes the final answer. {Output Instruction} Where the output instructions are: •Math: “Output the final result inside\boxed{}. Example:\boxed{42}.” • QA: “Output the final choice selection using the format: \boxed{X} where X ...

  76. [85]

    BEFORE reading peers, rederive the key steps independently

  77. [86]

    Only consider an objection if it identifies a SPECIFIC step where a particular number, formula, or factual claim is wrong

    THEN scan peers. Only consider an objection if it identifies a SPECIFIC step where a particular number, formula, or factual claim is wrong. Instruction:

  78. [88]

    Summarize your reasoning into a conclusion that includes the final answer. {Output Instruction} Case 2: Without Peer Responses Note that when no peer responses are available, the peer context block is replaced with: Debate Round Prompt (No Peer Responses Available) {Persona Pr...

  79. [89]

    Provide your reasoning process

  80. [90]

    {Output Instruction} 25

    Summarize your reasoning into a conclusion that includes the final answer. {Output Instruction} 25

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.