Pith. sign in

REVIEW 3 major objections 5 minor 8 cited by

Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Across 36 model-benchmark combinations, five multi-agent debate methods rarely beat plain chain-of-thought prompting, and the paper proposes model heterogeneity as the fix.

desk verdict The negative result on MAD is credible and worth publishing; the Heter-MAD 'universal antidote' claim is not supported by the baselines used. read the letter →

arxiv 2502.08788 v3 pith:E37RSHB3 submitted 2025-02-12 cs.CL cs.LG

Hangfan Zhang , Zhiyao Cui , Jianhao Chen , Xinrun Wang , Qiaosheng Zhang , Zhen Wang , Dinghao Wu , Shuyue Hu This is my paper · ORCID
classification cs.CLcs.LG
keywords multi-agentdebatelargelanguagemodelschain-of-thoughtpromptingself-consistencymodelheterogeneityinference-timecomputationbenchmarkevaluationLLMreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that multi-agent debate (MAD) research has been overvalued because its evaluations are narrow and its baselines weak. Across 36 configurations (four LLMs times nine benchmarks), five representative MAD methods rarely outperform the simple single-agent baseline chain-of-thought (CoT), and they lag self-consistency when both use a comparable inference budget. MAD does beat direct answer-only prompting, but not the stronger single-agent methods. The paper then proposes Heter-MAD, a twist that assigns each debating agent a model sampled from a pool of different LLMs, and reports that this consistently improves every MAD framework tested. If these results hold, the field should adopt strong single-agent baselines and efficiency metrics, and treat model heterogeneity as a core design principle rather than an optional extra.

What carries the argument

The central object is Heter-MAD, a minimal modification that samples a foundation model per agent response from a candidate pool (with probabilities summing to one) while keeping each MAD method's prompts, roles, and aggregation rules unchanged. Its role is to turn same-model debate into cross-model debate, so that questions one model gets right and the other misses can be corrected by the partner model. The evaluation scaffold—nine benchmarks spanning general knowledge, math, and programming, four models, matched LLM-call budgets, and CoT and SC baselines—supports the negative claim that homogeneous MAD is not a reliable inference-time booster.

What would settle it

Run Heter-MAD against the stronger single model rather than the averaged baseline: for example, compare Heter-SoM's MATH score to GPT-4o-mini CoT alone (72.87) and to a majority vote among the same two models run independently. If Heter-MAD does not beat both on most benchmarks, the 'universal antidote' claim loses its support.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a negative result with a proposed remedy: when evaluated across nine benchmarks and four foundation models, five representative multi-agent debate (MAD) methods do not reliably outperform the single-agent chain-of-thought baseline, and they underperform self-consistency when both consume comparable inference budget. The failure is not a tuning artifact—increasing the number of agents or debate rounds rarely reverses it. The remedy is Heter-MAD, which makes each agent draw its foundation model at random from a pool of different LLMs at inference time. Heter-MAD improves every MAD framework tested, and the paper argues this shows that epistemic diversity, not the debate protocol itself, is what carries collaborative gains.

Load-bearing premise

The 'universal antidote' claim assumes that the right baseline is the average of the two models' chain-of-thought scores, so beating CoT-Average means heterogeneity helps; if the correct baseline is the stronger single model or a mixed-model single-agent ensemble, the claim weakens.

Editorial extensions

If this is right

  • MAD evaluations should always include chain-of-thought and self-consistency as baselines, with token and call budgets reported; claims of improvement are otherwise uninterpretable.
  • Published MAD results on one or two overlapping benchmarks should not be taken at face value; evaluation needs to span general knowledge, math, and programming.
  • Tuning the number of agents or debate rounds will not in general convert a losing MAD method into a winning one.
  • Sampling agents from a heterogeneous model pool should be a default design choice, since it improves all five tested frameworks.
  • The useful niche of MAD may be tasks requiring genuinely diverse knowledge, rather than the single-knowledge-point questions that dominate current benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stricter baseline than CoT-Average would test whether the 'universal antidote' claim survives: in MATH, Heter-SoM scores 71.1 while GPT-4o-mini CoT alone scores 72.87, so averaging in a weaker Llama model lowers the bar.
  • Much of the benefit attributed to debate could come from heterogeneous ensembling per se; a testable prediction is that a heterogeneous single-agent majority vote without debate recovers most of the gain.
  • Because the paper fixes temperature at T=1, the negative result may not transfer to lower-temperature or greedy decoding; re-running the comparison under different decoding budgets is a cheap way to test the claim's scope.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper reports a systematic empirical evaluation of five multi-agent debate (MAD) methods (SoM, MP, EoT, ChatEval, AgentVerse) across nine benchmarks and four LLMs, comparing them against single-agent baselines (CoT and Self-Consistency). The authors find that MAD methods generally fail to outperform CoT and perform worse than SC on most benchmarks, and they argue that current evaluation practices in MAD research (limited benchmarks, weak baselines, inconsistent setups) overstate MAD's benefits. They then propose Heter-MAD, a simple modification that randomly selects a foundation model from a pool for each agent generation, and claim that this 'consistently improves' all examined MAD frameworks, motivating a call to embrace model heterogeneity as a core design principle. The paper also provides a comparison of previous MAD evaluations and a discussion of future research questions.

Significance. If the negative result is correct, it is an important corrective to the MAD literature, and the paper's breadth (36 model-dataset configurations, 5 methods, 3 baseline types, public code) is a clear strength. The heterogeneity proposal is simple and testable, and the paper is honest about its own limitations (e.g., Section 4.1 footnote, Section 5, Appendix B). However, the empirical support for the positive claim is currently compromised by the choice of baselines in Section 4.2, and the statistical analysis in Section 3.2 is underpowered. The paper is therefore more persuasive as a critique of current evaluation practice than as a demonstration that heterogeneity is a 'universal antidote.'

major comments (3)
  1. [4.2 (Table 4)] The central claim that Heter-MAD 'consistently improves' MAD is based on comparing Heter-MAD to CoT-Average and MAD-Average, defined as the arithmetic mean of the GPT-4o-mini and Llama3.1-70b configurations. Because Llama3.1-70b is far weaker than GPT-4o-mini on several benchmarks (e.g., MATH CoT: 37.13 vs 72.87), these averages dilute the strong model and inflate the apparent gains. For example, Heter-SoM on MATH scores 71.1, which is +29.3% relative to CoT-Average 55.0, but is below GPT-4o-mini CoT (72.87) and below homogeneous SoM-GPT (75.4); on MBPP, all four Heter-MAD variants (54.7, 49.8, 54.3, 53.0) are below GPT-4o-mini CoT (62.26). The paper never reports comparisons against the best single model in the pool or against a same-budget heterogeneous single-agent baseline (e.g., Self-Consistency sampling from both models). Therefore the evidence does not establish that heterogeneity, rather than access to the stronger model in the pool, drives the improvements, and the word 'consistently' is contradicted by the best-member comparisons. Moreover, the statement in Section 4.2 that 'all the considered MAD methods outperform CoT-Average' is false for several entries (e.g., Heter-EoT on MMLU: -2.5%; Heter-ChatEval on MBPP: -10.8%; Heter-AGV on GSM8K: -2.3%).
  2. [3.2 (Figure 2)] The aggregate Win/Tie/Lose analysis uses an ANOVA with three repeats per condition and treats p > 0.05 as 'no significant differences—a tie.' With n = 3, the test is severely underpowered, so many comparisons that differ will be non-significant, conflating absence of evidence with evidence of equivalence. This is load-bearing because the paper's headline aggregate rates (e.g., MAD win rates around 15%) depend on this classification. The authors should either use an equivalence test with a pre-specified margin (e.g., TOST), report effect sizes and confidence intervals, or present the raw differences with appropriate statistical inference. At minimum, the tie category should be labeled 'not statistically distinguishable' rather than 'no difference.'
  3. [4.1] The Heter-MAD sampling protocol is underspecified: it is not clear whether each agent's model assignment is fixed for the entire debate or re-sampled at every generation step. This distinction changes the mechanism claimed to improve performance (fixed assignment gives some agents the stronger model; re-sampling mixes models within an agent's trajectory). The paper also sets p_i = 0.5 without any sensitivity analysis, so we cannot tell whether the results are robust to the mixing probability. Please specify the protocol precisely and test at least one alternative p_i (e.g., 0.25/0.75) to support the heterogeneity mechanism.
minor comments (5)
  1. [Throughout] The benchmark name 'CommensenseQA' is misspelled; it should be 'CommonsenseQA' (Tables 1 and 3, Figure 2, and Appendix C.1).
  2. [3.2] The sentence 'We acknowledge that ChatEval and SoM were not compared to CoT though, EoT, AgentVerse, and MP were compared to CoT in their papers' is grammatically incomplete and unclear; please rewrite.
  3. [Figure 4] The caption says 'the Fred bar represents' but should be 'the red bar.'
  4. [Table 4] The color coding described in the caption ('light green,' 'green,' 'red') is not visible in the typeset table; please use explicit markers or ensure the colors are rendered.
  5. [4.2] The first reason for excluding MP ('previous studies indicate that the agent roles in MP are unbalanced') lacks a citation; please add a reference or specify the study.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical evaluation against external benchmarks, and the Heter-MAD comparison baselines, while debatable, are not defined in terms of the results they are used to support.

full rationale

This is an empirical benchmark study rather than a derivation. The central claims—that MAD methods often fail to outperform CoT/SC, and that mixing two foundation models in Heter-MAD yields gains over the average of the two models run separately—are supported by measured accuracies on nine external benchmarks (MMLU, GSM8K, MATH, HumanEval, MBPP, etc.) with no fitted parameters and no equation that defines a prediction in terms of an input. The Heter-MAD protocol (randomly sampling GPT-4o-mini or Llama3.1-70b with probability 0.5) is a defined intervention, not a fit; its reported gains are comparisons to averages of separately measured configurations, which is a defensible if contestable baseline choice. The 'universal antidote' claim may be weakened by the absence of best-member comparisons (e.g., Heter-SoM falls below GPT-4o-mini CoT on MATH and MBPP), but that is a question of baseline validity and generalization, not circular reasoning: the conclusion is not equivalent to its inputs by construction. There are no load-bearing self-citations or imported uniqueness theorems; the few references to prior MAD work are contextual, and the empirical results stand on the paper's own benchmark runs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper is empirical, not a derivation, so the ledger is short. The central claims rest on benchmark selection, faithful reproduction of prior methods, and the choice of average baseline for Heter-MAD. The main load-bearing choices are the CoT-Average baseline and the ANOVA 'tie' rule.

free parameters (2)
  • model selection probability p_i = 0.5 for each of two models
    Chosen by hand in Section 4.2, not tuned; the paper does not show sensitivity to this probability. It is an experimental hyperparameter, not a fitted constant, but it is load-bearing for the Heter-MAD result.
  • inference budget (number of LLM calls) = 6 by default
    Chosen in Section 3.1 following prior conventions; MAD methods' debate rounds are adjusted to match. The negative result is claimed to be robust to varying this, but the baseline comparisons depend on the choice.
assumptions (4)
  • domain assumption The authors' open-source implementations of SoM, MP, EoT, ChatEval, and AgentVerse are faithful to the original methods after adjusting debate rounds.
    Section 3.1: 'we follow the authors' open-source implementations' and adjust debate rounds to align LLM calls; if the adjustments change behavior, conclusions about the original methods could be wrong.
  • domain assumption CoT-Average and MAD-Average are meaningful baselines for evaluating Heter-MAD.
    Section 4.2, Table 4: gains are reported relative to the average of the two models separately, not the best model or a mixed single-agent ensemble; this assumption is load-bearing for the 'universal antidote' claim.
  • domain assumption p > 0.05 from an ANOVA on three repeats constitutes a tie.
    Section 3.2: the paper explicitly states 'p-value greater than 0.05 as an indicator that no significant differences, a tie'. This is a statistical assumption that is not justified and affects Figure 2.
  • domain assumption The nine benchmarks and zero-shot settings are representative enough to support general statements about MAD.
    Section 3.1: benchmarks cover general knowledge, math, programming; the paper generalizes from these to 'MAD often fails', but does not include open-ended dialogue, agentic tasks, or translation, where MAD may behave differently.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity." pith.science (2026). https://pith.science/paper/E37RSHB3

@misc{pith2026250208788,
  author       = {Pith},
  title        = {Pith review of: Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E37RSHB3}},
  note         = {Machine review of arXiv:2502.08788}
}
read the original abstract

Multi-agent debate (MAD) has gained significant attention as a promising line of research to improve the factual accuracy and reasoning capabilities of large language models (LLMs). Despite its conceptual appeal, current MAD research suffers from critical limitations in evaluation practices, including limited benchmark coverage, weak baseline comparisons, and inconsistent setups. This paper presents a systematic evaluation of 5 representative MAD methods across 9 benchmarks using 4 foundational models. Surprisingly, our findings reveal that MAD often fail to outperform simple single-agent baselines such as Chain-of-Thought and Self-Consistency, even when consuming significantly more inference-time computation. To advance MAD research, we further explore the role of model heterogeneity and find it as a universal antidote to consistently improve current MAD frameworks. Based on our findings, we argue that the field must stop overvaluing MAD in its current form; for true advancement, we must critically rethink evaluation paradigms and actively embrace model heterogeneity as a core design principle.

Figures

Figures reproduced from arXiv: 2502.08788 by the authors.

Figure 1
Figure 1. The promotion and recognition of MAD research require a systematic and comprehensive [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison of MAD across 4 LLMs and 9 benchmarks, covering 3 top-level [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We explore the impact of hyperparameters on the performance of MAD frameworks by [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparing the behavior of inference strategies to direct prompting a single-agent. The [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Heter-MAD performance analysis. We split questions in a benchmark into four parts each [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Comparing scaling efficiency of MAD methods. We present performance regarding number [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: Heter-MAD performance analysis. We split questions in a benchmark into four parts each [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Does Multi-Agent Debate Improve AI Feedback on Research Papers?

    econ.GN 2026-07 accept novelty 7.0 of 10

    Authors of economics meta-analyses found a single-pass AI report more useful than two multi-agent debate tools that cost up to thirty times more to run.

  2. Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges

    cs.SE 2026-07 accept novelty 6.0 of 10

    A systematic review of 141 papers derives a three-axis taxonomy of multi-agent debate design (participants, interaction, agreement) and shows the field has converged on a narrow default pattern.

  3. Beyond a Joke: Multi-Angle Reasoning for Detecting and Explaining Harmful Humor in Memes

    cs.AI 2026-07 conditional novelty 6.0 of 10

    MAR-12 improves humor and hate detection in memes by prompting a VLM through twelve reasoning perspectives, attention-weighting them, and generating explanations from the weighted evidence.

  4. The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Reasoning messages between heterogeneous VLMs can be routed through the image-token span: a distilled universal codec plus affine alignment transmits latent traces across model families, cutting wall-clock time in sma...

  5. Exploring Advanced LLM Multi-Agent Systems Based on Blackboard Architecture

    cs.MA 2025-07 conditional novelty 6.0 of 10

    A blackboard-based LLM multi-agent system with controller-selected agents achieves competitive benchmark accuracy at lower token cost than several static and dynamically optimized baselines.

  6. Simulating Ethics: Using LLM Debate Panels to Model Deliberation on Medical Dilemmas

    cs.CY 2025-05 conditional novelty 6.0 of 10

    Two AI ethics debates with differently composed panels reached the same policy recommendation but through different arguments and coalitions, showing that panel membership can shift reasoning even when the facts are fixed.

  7. The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants

    cs.CL 2025-05 reject novelty 5.0 of 10

    Clustering-based routing plus self-consistency voting among ten 7B open models reportedly outranks GPT-4.1 and GPT-4.5 on average over 15 diverse benchmarks.

  8. Swarm Intelligence Enhanced Reasoning: A Density-Driven Framework for LLM-Based Multi-Agent Optimization

    cs.MA 2025-05 conditional novelty 5.0 of 10

    SIER uses kernel density estimation and Pareto-style selection to preserve diversity in PRM-guided multi-agent reasoning, improving pass@8 and prm@8 on math benchmarks at higher token cost.

Reference graph

Works this paper leans on

51 extracted references · 33 canonical work pages · cited by 8 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := ...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    , Odena, A

    Austin, J. , Odena, A. , Nye, M. , Bosma, M. , Michalewski, H. , Dohan, D. , Jiang, E. , Cai, C. , Terry, M. , Le, Q. et al. (2021). Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  4. [4]

    , Chen, W

    Chan, C.-M. , Chen, W. , Su, Y. , Yu, J. , Xue, W. , Zhang, S. , Fu, J. and Liu, Z. (2024). Chateval: Towards better LLM -based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations

    work page 2024

  5. [5]

    , Saha, S

    Chen, J. , Saha, S. and Bansal, M. (2024 a ). R e C oncile: Round-table conference improves reasoning via consensus among diverse LLM s. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (L.-W. Ku, A. Martins and V. Srikumar, eds.). Association for Computational Linguistics, Bangkok, Thailand

    work page 2024

  6. [6]

    , Tworek, J

    Chen, M. , Tworek, J. , Jun, H. , Yuan, Q. , de Oliveira Pinto, H. P. , Kaplan, J. , Edwards, H. , Burda, Y. , Joseph, N. , Brockman, G. , Ray, A. , Puri, R. , Krueger, G. , Petrov, M. , Khlaaf, H. , Sastry, G. , Mishkin, P. , Chan, B. , Gray, S. , Ryder, N. , Pavlov, M. , Power, A. , Kaiser, L. , Bavarian, M. , Winter, C. , Tillet, P. , Such, F. P. , Cum...

    work page 2021

  7. [7]

    , Zhang, S

    Chen, P. , Zhang, S. and Han, B. (2024 b ). Comm: Collaborative multi-agent, multi-reasoning-path prompting for complex problem solving. In Findings of the Association for Computational Linguistics: NAACL 2024

    work page 2024

  8. [8]

    Chen, W. , Su, Y. , Zuo, J. , Yang, C. , Yuan, C. , Chan, C.-M. , Yu, H. , Lu, Y. , Hung, Y.-H. , Qian, C. , Qin, Y. , Cong, X. , Xie, R. , Liu, Z. , Sun, M. and Zhou, J. (2024 c ). Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations

    work page 2024

Show all 51 references
  1. [9]

    , You, Z

    Chen, W. , You, Z. , Li, R. , Guan, Y. , Qian, C. , Zhao, C. , Yang, C. , Xie, R. , Liu, Z. and Sun, M. (2024 d ). Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence. CoRR abs/2407.07061

    2024 arXiv

  2. [10]

    and Lee, H.-y

    Chiang, C.-H. and Lee, H.-y. (2023). Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937

    2023 arXiv

  3. [11]

    , Cowhey, I

    Clark, P. , Cowhey, I. , Etzioni, O. , Khot, T. , Sabharwal, A. , Schoenick, C. and Tafjord, O. (2018). Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1

    2018 arXiv

  4. [12]

    , Kosaraju, V

    Cobbe, K. , Kosaraju, V. , Bavarian, M. , Chen, M. , Jun, H. , Kaiser, L. , Plappert, M. , Tworek, J. , Hilton, J. , Nakano, R. , Hesse, C. and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  5. [13]

    Du, Y. , Li, S. , Torralba, A. , Tenenbaum, J. B. and Mordatch, I. (2023). Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning

    2023

  6. [14]

    , ..., A

    Dubey, A. , ..., A. J. and Zhao, Z. (2024). The llama 3 herd of models

    2024

  7. [15]

    , Hedayatnia, B

    Gopalakrishnan, K. , Hedayatnia, B. , Chen, Q. , Gottardi, A. , Kwatra, S. , Venkatesh, A. , Gabriel, R. and Hakkani-Tur, D. (2023). Topical-chat: Towards knowledge-grounded open-domain conversations. arXiv preprint arXiv:2308.11995

    2023 arXiv

  8. [16]

    , Yang, D

    Guo, D. , Yang, D. , Zhang, H. , Song, J. , Zhang, R. , Xu, R. , Zhu, Q. , Ma, S. , Wang, P. , Bi, X. et al. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  9. [17]

    , Wang, T

    He, J. , Wang, T. , Xiong, D. and Liu, Q. (2020). The box is in the pen: Evaluating commonsense reasoning in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020 (T. Cohn, Y. He and Y. Liu, eds.). Association for Computational Li...

    2020

  10. [18]

    , Burns, C

    Hendrycks, D. , Burns, C. , Basart, S. , Critch, A. , Li, J. , Song, D. and Steinhardt, J. (2021 a ). Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  11. [19]

    , Burns, C

    Hendrycks, D. , Burns, C. , Basart, S. , Zou, A. , Mazeika, M. , Song, D. and Steinhardt, J. (2021 b ). Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  12. [20]

    , Burns, C

    Hendrycks, D. , Burns, C. , Kadavath, S. , Arora, A. , Basart, S. , Tang, E. , Song, D. and Steinhardt, J. (2021 c ). Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  13. [21]

    u gamer, D. , H \

    Herrmann, M. , Lange, F. J. D. , Eggensperger, K. , Casalicchio, G. , Wever, M. , Feurer, M. , R \"u gamer, D. , H \"u llermeier, E. , Boulesteix, A.-L. and Bischl, B. (2024). Position: Why we must rethink empirical research in machine learning. PMLR

    2024

  14. [22]

    Hosseini, M. J. , Hajishirzi, H. , Etzioni, O. and Kushman, N. (2014). Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)

    2014

  15. [23]

    , Hughes, J

    Khan, A. , Hughes, J. , Valentine, D. , Ruis, L. , Sachan, K. , Radhakrishnan, A. , Grefenstette, E. , Bowman, S. R. , Rocktäschel, T. and Perez, E. (2024). Debating with more persuasive llms leads to more truthful answers. In ICML

    2024

  16. [24]

    , Hajishirzi, H

    Koncel-Kedziorski, R. , Hajishirzi, H. , Sabharwal, A. , Etzioni, O. and Ang, S. D. (2015). Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics 3 585--597

    2015

  17. [25]

    Li, Y. , Du, Y. , Zhang, J. , Hou, L. , Grabowski, P. , Li, Y. and Ie, E. (2024). Improving multi-agent debate with sparse communication topology. In Findings of the Association for Computational Linguistics: EMNLP 2024 (Y. Al-Onaizan, M. Bansal and Y.-N. Chen, eds.). Associat...

    2024

  18. [26]

    Liang, T. , He, Z. , Jiao, W. , Wang, X. , Wang, Y. , Wang, R. , Yang, Y. , Shi, S. and Tu, Z. (2024). Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing ...

    2024

  19. [27]

    , Yogatama, D

    Ling, W. , Yogatama, D. , Dyer, C. and Blunsom, P. (2017). Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146

    2017 arXiv

  20. [28]

    , Feng, B

    Liu, A. , Feng, B. , Xue, B. , Wang, B. , Wu, B. , Lu, C. , Zhao, C. , Deng, C. , Zhang, C. , Ruan, C. et al. (2024). Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  21. [29]

    , Tandon, N

    Madaan, A. , Tandon, N. , Gupta, P. , Hallinan, S. , Gao, L. , Wiegreffe, S. , Alon, U. , Dziri, N. , Prabhumoye, S. , Yang, Y. , Gupta, S. , Majumder, B. P. , Hermann, K. , Welleck, S. , Yazdanbakhsh, A. and Clark, P. (2023). Self-refine: Iterative refinement with self-feedback

    2023

  22. [30]

    and Eskenazi, M

    Mehri, S. and Eskenazi, M. (2020). Unsupervised evaluation of interactive dialog with dialogpt

    2020

  23. [31]

    Gpt-4o-mini

    OpenAI (2024 a ). Gpt-4o-mini

    2024

  24. [32]

    Introducing openai o1

    OpenAI (2024 b ). Introducing openai o1

    2024

  25. [33]

    , Bhattamishra, S

    Patel, A. , Bhattamishra, S. and Goyal, N. (2021). Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191

    2021 arXiv

  26. [34]

    , Liu, B

    Pham, C. , Liu, B. , Yang, Y. , Chen, Z. , Liu, T. , Yuan, J. , Plummer, B. A. , Wang, Z. and Yang, H. (2023). Let models speak ciphers: Multiagent debate through embeddings. arXiv preprint arXiv:2310.06272

    2023 arXiv

  27. [35]

    , Xie, Z

    Qian, C. , Xie, Z. , Wang, Y. , Liu, W. , Dang, Y. , Du, Z. , Chen, W. , Yang, C. , Liu, Z. and Sun, M. (2024). Scaling large-language-model-based multi-agent collaboration. arXiv preprint arXiv:2406.07155

    2024 arXiv

  28. [36]

    and Roth, D

    Roy, S. and Roth, D. (2016). Solving general arithmetic word problems

    2016

  29. [37]

    , Suzgun, M

    Shi, F. , Suzgun, M. , Freitag, M. , Wang, X. , Srivats, S. , Vosoughi, S. , Chung, H. W. , Tay, Y. , Ruder, S. , Zhou, D. , Das, D. and Wei, J. (2022). Language models are multilingual chain-of-thought reasoners

    2022

  30. [38]

    , Cassano, F

    Shinn, N. , Cassano, F. , Gopinath, A. , Narasimhan, K. and Yao, S. (2024). Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems 36

    2024

  31. [39]

    Smit, A. P. , Grinsztajn, N. , Duckworth, P. , Barrett, T. D. and Pretorius, A. (2024). Should we be going mad? a look at multi-agent debate strategies for llms. In Forty-first International Conference on Machine Learning

    2024

  32. [40]

    , Rastogi, A

    Srivastava, A. , Rastogi, A. , Rao, A. , Shoeb, A. A. M. , Abid, A. , Fisch, A. , Brown, A. R. , Santoro, A. , Gupta, A. , Garriga-Alonso, A. , Kluska, A. , Lewkowycz, A. , Agarwal, A. , Power, A. , Ray, A. , Warstadt, A. , Kocurek, A. W. , Safaya, A. , Tazarv, A. , Xiang, A. ...

    2023

  33. [41]

    , Rastogi, A

    Srivastava, A. , Rastogi, A. , Rao, A. , Shoeb, A. A. M. , Abid, A. , Fisch, A. , Brown, A. R. , Santoro, A. , Gupta, A. , Garriga-Alonso, A. et al. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615

    2022 arXiv

  34. [42]

    , Herzig, J

    Talmor, A. , Herzig, J. , Lourie, N. and Berant, J. (2019). C ommonsense QA : A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language T...

    2019

  35. [43]

    , Wang, Z

    Wang, Q. , Wang, Z. , Su, Y. , Tong, H. and Song, Y. (2024 a ). Rethinking the bounds of llm reasoning: Are multi-agent discussions the key? arXiv preprint arXiv:2402.18272

    2024 arXiv

  36. [44]

    , Wei, J

    Wang, X. , Wei, J. , Schuurmans, D. , Le, Q. V. , Chi, E. H. , Narang, S. , Chowdhery, A. and Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations

    2023

  37. [45]

    Wang, Y. , Ma, X. , Zhang, G. , Ni, Y. , Chandra, A. , Guo, S. , Ren, W. , Arulraj, A. , He, X. , Jiang, Z. , Li, T. , Ku, M. , Wang, K. , Zhuang, A. , Fan, R. , Yue, X. and Chen, W. (2024 b ). MMLU -pro: A more robust and challenging multi-task language understanding benchmar...

    2024

  38. [46]

    , Wang, X

    Wei, J. , Wang, X. , Schuurmans, D. , Bosma, M. , Xia, F. , Chi, E. , Le, Q. V. , Zhou, D. et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 24824--24837

    2022

  39. [47]

    , Ding, X

    Xiong, K. , Ding, X. , Cao, Y. , Liu, T. and Qin, B. (2023). Examining inter-consistency of large language models collaboration: An in-depth analysis via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023 (H. Bouamor, J. Pino and K. Bali, eds.). A...

    2023

  40. [48]

    , Zhao, J

    Yao, S. , Zhao, J. , Yu, D. , Du, N. , Shafran, I. , Narasimhan, K. and Cao, Y. (2022). React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629

    2022 arXiv

  41. [49]

    , Sun, Q

    Yin, Z. , Sun, Q. , Chang, C. , Guo, Q. , Dai, J. , Huang, X. and Qiu, X. (2023). Exchange-of-thought: Enhancing large language model capabilities through cross-model communication. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  42. [50]

    Zhang, J. , Xu, X. , Zhang, N. , Liu, R. , Hooi, B. and Deng, S. (2024). Exploring collaboration mechanisms for LLM agents: A social psychology view. In ICLR 2024 Workshop on Large Language Model (LLM) Agents

    2024

  43. [51]

    , Cui, R

    Zhong, W. , Cui, R. , Guo, Y. , Liang, Y. , Lu, S. , Wang, Y. , Saied, A. , Chen, W. and Duan, N. (2024). AGIE val: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024 (K. Duh, H. Gomez and S. Bet...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.