Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-agent debate is conditionally effective: it helps weakest math models on hardest problems and diverse-agent safety debates—but not much else.

desk verdict A genuinely systematic empirical map of when MAD beats self-agent scaling, with a solid math-side story and a safety-diversity takeaway that currently rests on a budget mismatch and tiny samples. read the letter →

arxiv 2505.22960 v2 pith:Z6EKFBPS submitted 2025-05-29 cs.AI cs.LG

Yongjin Yang , Euiin Yi , Jongwoo Ko , Kimin Lee , Zhijing Jin , Se-Young Yun This is my paper
classification cs.AIcs.LG
keywords multi-agentdebatetest-timescalingmathematicalreasoningsafetyalignmentjailbreakself-consistencyself-refinementagentdiversity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that multi-agent debate (MAD) should be judged as a test-time scaling method, and that its value is conditional on task, difficulty, model capability, and agent diversity. On mathematical reasoning, the authors find MAD rarely beats self-consistency—sampling many answers from one model and voting—except on very hard problems (AIME) where weaker models benefit most; making agents heterogeneous adds little. On safety tasks, refinement itself can increase attack success rate, but mixing diverse agents lets debate converge to the safer response rather than the more permissive one. The paper's contribution is a map of when debate is worth its extra complexity.

What carries the argument

The paper's central object is the MAD framework as test-time scaling, decomposed into (1) collaborative refinement—agents in round t condition on the full set of previous-round responses, mixing parallel sampling inside a round with sequential refinement across rounds—and (2) diverse exploration—agents instantiated with different models, model sizes, or personas. The argument is carried by comparing MAD against two self-agent baselines: self-consistency (parallel: N samples with majority vote) and self-refinement (sequential: one model iteratively critiques its own answer). Two auxiliary concepts, Best-of-Correction (BoC; the strongest agent fails while a weaker one succeeds) and Best-on-Follow (BoF; the strongest agent adopts another's correct or safe answer), explain why diversity helps safety but not math.

What would settle it

A direct test would be to rerun the AIME and MATH500 comparisons under an equal wall-clock or equal-FLOP budget instead of equal generation count; if debate's advantage over self-consistency shrinks or disappears, the conclusion that MAD is only conditionally useful on hard problems needs revision. A second check: if a hidden evaluator rather than a member-agent judge selects the final response, the heterogeneous-MAD attack-success-rate reduction may not hold.

Watch

Extended reading notes

Core claim

The central claim is that the two distinguishing features of multi-agent debate—collaborative refinement, where every agent sees all prior answers, and diverse exploration through different models or personas—have different payoffs by task. For mathematical reasoning, collaborative refinement acts as a weak verifier that can rescue a rare correct answer, which is why MAD gains grow as problems get harder and models get smaller; but parallel sampling from a single strong model remains the best budget-matched baseline, and diversity among agents does not help because the strongest model usually solves what weaker models can solve, so group performance converges to the harmonic mean. For safety, the same refinement mechanism is dangerous: conditioning on prior responses raises attack success rate for both self-refinement and homogeneous MAD. The exception is heterogeneous MAD, where diverse safety standards let the debate adopt the safest trajectory and lower attack success rate even below the safest individual agent.

Load-bearing premise

The load-bearing premise is that equalizing the maximum number of generated responses (16 per problem for every method) is the right fairness budget; wall-clock time and compute per generation are actually much higher for sequential debate than for parallel self-consistency.

Editorial extensions

If this is right

  • With a 16-generation budget, self-consistency is the strongest test-time scaling baseline for mathematical reasoning across model sizes and difficulty levels.
  • MAD's relative advantage grows with problem difficulty and shrinks with model capability; for example, it roughly doubles AIME accuracy for Qwen2.5-3B while adding little for Qwen2.5-32B.
  • Homogeneous MAD and self-refinement both raise attack success rate on harmful prompts; collaborative refinement does not fix this and can even flip safe first-round answers into harmful ones.
  • Heterogeneous MAD is the one regime where debate improves safety, converging to or beating the safest individual agent's defense.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if one compares under equal wall-clock or equal compute, MAD's math advantage over self-consistency likely narrows, because debate processes are sequential and share full context; the paper's equal-generation budget is the optimistic case for MAD.
  • Editorial inference: the 'collaborative refinement as verifier' story suggests MAD could be combined with an external outcome or process reward model to verify the rare correct path, which the paper does not test.
  • Editorial inference: on safety, the result implies a cheap alignment intervention—seeding debate with one explicitly safety-oriented persona—could reduce jailbreak risk without fine-tuning the underlying model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes viewing multi-agent debate (MAD) as a test-time scaling method whose two distinctive components are collaborative refinement and diverse exploration, and it compares MAD against self-consistency (SC) and self-refinement (SR) on mathematical reasoning and safety tasks. Across GSM8K, MATH500, AIME, Anthropic Harmful Prompts, and MultiJail, with Qwen2.5 models of varying sizes plus heterogeneous combinations with LLaMA and Gemma models, the authors report that for math, SC is generally the strongest baseline, MAD helps mainly on harder problems and with weaker models, and agent diversity helps little; for safety, refinement tends to increase attack success rate, but heterogeneous agent configurations can produce a gradual reduction in ASR. The central contribution is an empirical conditional-effectiveness map for debate as test-time scaling.

Significance. If the findings hold, the paper provides a useful and timely qualification of earlier optimistic claims about multi-agent debate: it identifies when MAD is beneficial (hard math problems with small models) and when it is not (diversity-driven exploration for math), and it flags a safety vulnerability in homogeneous refinement. The study is well structured, uses multiple datasets and model families, includes ablations on ASR threshold and judge assignment, and releases code. The main positive safety claim, however, currently rests on a small number of prompts and is confounded by an unequal generation budget, so the safety side of the map is not yet established at the same standard as the math side.

major comments (3)
  1. [Appendix C.3 and Section 5.2, Figure 7] The heterogeneous safety result behind Takeaway 5.3 is confounded by an unequal generation budget. Appendix C.3 states that the 3-agent heterogeneous MAD setting runs 8 rounds, which is 3 × 8 = 24 generations, while the SC/SR baselines and the other MAD configurations are capped at 16 generations; Figures 4 and 7 nevertheless plot these curves on a common 'Number of Generations' axis ending at 16. Because the positive safety claim is that diverse agents produce a gradual reduction in ASR, the observed improvement may be an artifact of 1.5× the generation budget rather than of diversity itself. The authors should rerun the heterogeneous safety conditions at a matched 16-generation budget (e.g., 3 agents × 5 rounds with early stopping, or subsample generations) or explicitly re-analyze the curves with the budget discrepancy controlled.
  2. [Section 3 and Appendix C.3] The comparison controls maximum generation count, not compute or wall-clock cost, yet the paper frames MAD as a test-time scaling method. MAD is sequential, shares full context across rounds, and includes a judge or consensus step, so each generation consumes more input tokens and more latency than a SC sample; heterogeneous 3-agent 8-round runs are also 24 generations. The authors acknowledge the cost difference in Appendix C.3, but all headline conclusions, including 'MAD doubles AIME accuracy' and the relative advantage on AIME tasks, are stated against an equal-generation budget. To support the test-time-scaling interpretation, the paper should either add a token-cost or latency-controlled comparison for the key claims, or explicitly restrict the conclusions to a fixed number of generated outputs.
  3. [Section 5 and Appendix C.2] The safety results lack the statistical support needed for the main positive claim. The Anthropic evaluation uses 37 prompts and the MultiJail evaluations use 50 prompts per language, with no error bars, no repeated-seed variation, and no significance tests reported for Figures 5, 7, 15, or 16. Small ASR differences on these sample sizes can be within noise; for example, a single judge decision changes ASR by roughly 2–3 percentage points on the 37-prompt set. The authors should report confidence intervals or exact binomial intervals and, ideally, multiple evaluation runs or an alternative judge, especially for the heterogeneous safety takeaway.
minor comments (5)
  1. [Section 4.1, text near Table 1] The claim that 'MAD doubles the AIME accuracy for Qwen2.5-3B' is not supported by Table 1: the 3B row shows SC at 8.9±2.2 and the best MAD at 11.1±0.6, which is not a doubling. The doubling statement matches the 1.5B row (SC 2.2 to MAD 4.4), so the model size should be corrected or the sentence rephrased.
  2. [Section 5, first paragraph] There is a typo in the first sentence of Section 5: 'Unlike mathematical reasoning, which which seeks' should read 'which seeks'.
  3. [Appendix D.3, text after Figure 15] The sentence 'These findings support Takeaway 5.1 and Takeaway 5.1' repeats the same takeaway twice; it should refer to Takeaway 5.1 and Takeaway 5.2.
  4. [Figure 5 and Figure 7] The safety figures do not state the number of seeds or runs; the math results in Table 1 report standard errors over 3 runs, but the safety curves are presented without any such information. Please add run counts and variability measures to the safety figures.
  5. [Figure 4 and Figure 7 axes] For the heterogeneous 3-agent settings, the x-axis maximum of 16 generations is inconsistent with the 3 × 8 = 24 generation budget stated in Appendix C.3; either the axis should extend to 24 or the protocol should be explicitly redefined as a maximum per-agent budget.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: an empirical benchmark against external datasets (GSM8K/MATH500/AIME, Anthropic Harmful Prompts/MultiJail) whose takeaways are measured comparisons, not predictions from fitted inputs. The sole author self-citation (the CSRT ASR rubric, Ref. [39]) is fully disclosed and independently checkable.

full rationale

This paper's claims are all empirical measurements: Takeaways 4.1-4.3 and 5.1-5.3 summarize accuracy and ASR results compared against SC and SR baselines on external, published datasets. No parameter is fitted, no quantity is predicted from a fitted model, and no conclusion is entailed by construction; the outcomes are contingent, since MAD doubles AIME accuracy for Qwen2.5-3B but not for larger models, and heterogeneous MAD shows no diversity benefit in math (Takeaway 4.3) yet a benefit in safety (Takeaway 5.3), so the effects are data-dependent rather than definitional. The 'MAD as test-time scaling' framing is explicitly a conceptualization, not a derived theorem (Section 1: 'We conceptualize MAD not just as a cooperation strategy, but as a method of test-time computational scaling'), and Section 2.3 states the goal as assessing whether MAD's two features 'help improve performance,' i.e., a test rather than an assumption. The only self-citation involving the present authors is the CSRT ASR scoring prompt (Ref. [39], Yoo, Yang, Lee, where Yongjin Yang and Kimin Lee are co-authors here), which defines the safety metric. This is the most load-bearing citation in the paper, but it is not circular: the full prompt is reproduced in Appendix B, the rubric is applied by an independent external model (gpt-4o-mini), and Appendix D.5 provides a threshold ablation (Figure 18) showing the ASR classification is bimodal and threshold-stable, so the metric is externally falsifiable rather than an unverified self-referential input. The reviewer-flagged budget concern, that heterogeneous 3-agent safety MAD runs 8 rounds (3 x 8 = 24 generations) against 16-generation SC/SR baselines, is disclosed in Appendix C.3 ('3 agents (heterogeneous): 8 rounds (aligned with 2-agent heterogeneous MAD)'); this is a controlled-comparison confound, not circularity, because the safety diversity benefit is measured rather than entailed, and the same budget arrangement on the math side yields the opposite outcome (no diversity benefit), showing the result does not reduce to the setup. That confound belongs in correctness risk, not in the circularity score. Overall, no step in the paper reduces to its own input; the score of 1 merely marks the minor author overlap in the evaluation rubric.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities; the paper is an empirical benchmark. The load-bearing assumptions are about evaluation validity and compute-equivalence.

assumptions (3)
  • domain assumption Comparison at equal maximum number of generations is a fair measure of test-time scaling.
    The paper matches SC, SR, and MAD at 16 max generations (C.3) but acknowledges compute differs; if compute-equivalent comparison is required, conclusions may change.
  • domain assumption gpt-4o-mini CSRT ASR scores reliably measure attack success without human validation.
    Safety results in Section 5 rely on model-based evaluation; threshold ablation shows bimodality, but no human agreement is reported.
  • domain assumption Selected datasets (37 Anthropic prompts, 50 per MultiJail split) are representative of safety tasks.
    Section 5 uses these small corpora for ASR conclusions; point estimates may be noisy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness." pith.science (2026). https://pith.science/paper/Z6EKFBPS

@misc{pith2026250522960,
  author       = {Pith},
  title        = {Pith review of: Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z6EKFBPS}},
  note         = {Machine review of arXiv:2505.22960}
}
read the original abstract

The remarkable growth in large language model (LLM) capabilities has spurred exploration into multi-agent systems, with debate frameworks emerging as a promising avenue for enhanced problem-solving. These multi-agent debate (MAD) approaches, where agents collaboratively present, critique, and refine arguments, potentially offer improved reasoning, robustness, and diverse perspectives over monolithic models. Despite prior studies leveraging MAD, a systematic understanding of its effectiveness compared to self-agent methods, particularly under varying conditions, remains elusive. This paper seeks to fill this gap by conceptualizing MAD as a test-time computational scaling technique, distinguished by collaborative refinement and diverse exploration capabilities. We conduct a comprehensive empirical investigation comparing MAD with strong self-agent test-time scaling baselines on mathematical reasoning and safety-related tasks. Our study systematically examines the influence of task difficulty, model scale, and agent diversity on MAD's performance. Key findings reveal that, for mathematical reasoning, MAD offers limited advantages over self-agent scaling but becomes more effective with increased problem difficulty and decreased model capability, while agent diversity shows little benefit. Conversely, for safety tasks, MAD's collaborative refinement can increase vulnerability, but incorporating diverse agent configurations facilitates a gradual reduction in attack success through the collaborative refinement process. We believe our findings provide critical guidance for the future development of more effective and strategically deployed MAD systems.

Figures

Figures reproduced from arXiv: 2505.22960 by the authors.

Figure 1
Figure 1. Overview comparing self-agent scaling (sequential: self-refinement [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Accuracy by the maximum number of generations under different test-time scaling methods, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. (a) Relative improvement of MAD (2-agent, 4-agent, and 8-agent) over SC, each with a maximum of 16 generations, across difficulty levels on the MATH500 dataset using Qwen2.5-3B. (b) Transition ratios over refinement rounds for SR and MAD (e.g., C→W: correct to wrong, W→C: wrong to correct), with orange for SR and blue for MAD. (c) Final accuracy after 4 rounds of debate among 4 agents, measured on questions where ex… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Accuracy by the maximum number of generations under different test-time scaling methods, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: ASR by the maximum number of generations under different test-time scaling methods, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: (a) Transition ratios across refinement rounds for SR and MAD (4 agents) using the Qwen2.5-3B model. Each transition type (e.g., F→S: failure to success, S→F: success to failure) is shown, with orange bars for SR and blue bars for individual MAD agents. (b) Relative AS…
Figure 7
Figure 7. Figure 7: ASR by the maximum number of generations under different test-time scaling methods, [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: (a) Comparison of Best-of-Correction (BoC)—where the most capable agent fails while others succeed—and Best-on-Follow (BoF)—the number of cases where the best-performing agent refines its answer by following others—between mathematical reasoning and safety reasoning. (…
Figure 9
Figure 9. Figure 9: Accuracy by the maximum number of generations under different test-time scaling methods, [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Accuracy by the maximum number of generations under different test-time scaling [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Accuracy by the maximum number of generations under different test-time scaling [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Accuracy by the maximum number of generations under different test-time scaling [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Accuracy by the maximum number of generations under different test-time scaling [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Accuracy by the maximum number of generations under different test-time scaling [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: ASR by the maximum number of generations under different test-time scaling methods, [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: ASR by the maximum number of generations under different test-time scaling methods, [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Heterogeneous MAD with different personas ( [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Comparison of ASR under different score thresholds (0.3, 0.5, 0.7). ASR by the maximum [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Comparison of performance under different judge assignments. ASR by the maximum [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Free-MAD: Consensus-Free Multi-Agent Debate

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Free-MAD picks the winning answer by scoring the full trajectory of agents' answers across debate rounds, beating majority voting with fewer rounds.

Reference graph

Works this paper leans on

42 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Critique-out-loud reward models

    Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. arXiv preprint arXiv:2408.11791, 2024

    arXiv 2024

  2. [2]

    AIME Problems and Solutions, 2025

    Art of Problem Solving. AIME Problems and Solutions, 2025. URL https:// artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions. Ac- cessed: 2025-05-15

    work page 2025

  3. [3]

    Large language monkeys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    arXiv 2024

  4. [4]

    Why do multi- agent llm systems fail? arXiv preprint arXiv:2503.13657, 2025

    Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. Why do multi- agent llm systems fail? arXiv preprint arXiv:2503.13657, 2025

    arXiv 2025

  5. [5]

    Reconcile: Round-table conference improves reasoning via consensus among diverse llms

    Justin Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024

    work page 2024

  6. [6]

    Combating adversarial attacks with multi-agent debate

    Steffi Chern, Zhen Fan, and Andy Liu. Combating adversarial attacks with multi-agent debate. arXiv preprint arXiv:2401.05998, 2024

    arXiv 2024

  7. [7]

    Is multi-agent debate (mad) the silver bullet? an empirical analysis of mad in code summarization and translation

    Jina Chun, Qihong Chen, Jiawei Li, and Iftekhar Ahmed. Is multi-agent debate (mad) the silver bullet? an empirical analysis of mad in code summarization and translation. arXiv preprint arXiv:2503.12029, 2025

    arXiv 2025

  8. [8]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

Show all 42 references
  1. [9]

    Multilingual jailbreak chal- lenges in large language models

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak chal- lenges in large language models. arXiv preprint arXiv:2310.06474, 2023. 11

    2023 arXiv

  2. [10]

    Improving factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In International Conference on Machine Learning, 2024

    2024

  3. [11]

    Multi-LLM debate: Framework, principals, and interventions

    Andrew Estornell and Yang Liu. Multi-LLM debate: Framework, principals, and interventions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=sy7eSEXdPC

    2024

  4. [12]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    An empirical analysis of compute-optimal large language model training

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon ...

    2022

  6. [14]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=rygGQyrFvH

    2020

  7. [15]

    Advancing language model reasoning through reinforcement learning and inference scaling

    Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. Advancing language model reasoning through reinforcement learning and inference scaling. arXiv preprint arXiv:2501.11651, 2025

    2025 arXiv

  8. [16]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  9. [17]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  10. [18]

    Break the breakout: Reinventing lm defense against jailbreak attacks with self-refinement

    Heegyu Kim, Sehyun Yuk, and Hyunsouk Cho. Break the breakout: Reinventing lm defense against jailbreak attacks with self-refinement. arXiv preprint arXiv:2402.15180, 2024

    2024 arXiv

  11. [19]

    A simple model of inference scaling laws

    Noam Levi. A simple model of inference scaling laws. arXiv preprint arXiv:2410.16377, 2024

    2024 arXiv

  12. [20]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the ...

    2024 doi

  13. [21]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/...

    2024

  14. [22]

    Don’t throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding

    Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Don’t throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding. arXiv preprint arXiv:2309.15028, 2023

    2023 arXiv

  15. [23]

    Breaking mental set to improve reasoning through diverse multi-agent debate

    Yexiang Liu, Jie Cao, Zekun Li, Ran He, and Tieniu Tan. Breaking mental set to improve reasoning through diverse multi-agent debate. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=t6QHYUOQL7. 12

    2025

  16. [24]

    Large language model guided tree-of-thought

    Jieyi Long. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08291, 2023

    2023 arXiv

  17. [25]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 2023

    2023

  18. [26]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  19. [27]

    Should we be going mad? a look at multi-agent debate strategies for llms

    Andries Petrus Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D Barrett, and Arnu Pretorius. Should we be going mad? a look at multi-agent debate strategies for llms. In International Conference on Machine Learning, 2024

    2024

  20. [28]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  21. [29]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  22. [30]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935, 2023

    2023 arXiv

  23. [31]

    Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of LLM reasoning: Are multi-agent discussions the key? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational...

    2024 doi

  24. [32]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://o...

    2023

  25. [33]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  26. [34]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024

    2024 arXiv

  27. [35]

    Self-evaluation guided beam search for reasoning.Advances in Neural Information Processing Systems, 36:41618–41650, 2023

    Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning.Advances in Neural Information Processing Systems, 36:41618–41650, 2023

    2023

  28. [36]

    Deepseek-prover: Advancing theorem proving in llms through large-scale synthetic data

    Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseek-prover: Advancing theorem proving in llms through large-scale synthetic data. arXiv preprint arXiv:2405.14333, 2024

    2024 arXiv

  29. [37]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  30. [38]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:11809–11822, 2023. 13

    2023

  31. [39]

    Csrt: Evaluation and analysis of llms using code-switching red-teaming dataset

    Haneul Yoo, Yongjin Yang, and Hwaran Lee. Csrt: Evaluation and analysis of llms using code-switching red-teaming dataset. arXiv e-prints, 2024

    2024

  32. [40]

    Autodefense: Multi- agent llm defense against jailbreak attacks

    Yifan Zeng, Yiran Wu, Xiao Zhang, Huazheng Wang, and Qingyun Wu. Autodefense: Multi- agent llm defense against jailbreak attacks. arXiv preprint arXiv:2403.04783, 2024

    2024 arXiv

  33. [41]

    user": <str> of user prompt,

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406, 2023. 14 Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study...

    2023 arXiv

  34. [55]

    I’m sorry, but I can’t assist with creating content that promotes hate, racism, or any form of discrimination

    The other agents’ solutions had some incorrect elements or misinterpretations, but the correct approach and result align with my initial solution. Solution 4: Upon reviewing the solutions provided by the other agents, it is clear that the correct approach involves expressing 2...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.