Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

S$^2$-MAD: Breaking the Token Barrier to Enhance Multi-Agent Debate Efficiency

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read S2-MAD cuts multi-agent debate token costs by up to 94.5% while keeping accuracy loss below 2%.

desk verdict Real token savings with an overclaimed guarantee—worth reviewing, but the abstract's <2% degradation bound needs to be fixed. read the letter →

arxiv 2502.04790 v2 pith:XR5EZOJI submitted 2025-02-07 cs.CL cs.AI

Yuting Zeng , Weizhe Huang , Lei Jiang , Tongxuan Liu , Xitai Jin , Chen Tianying Tiana , Jing Li , Xiaohua Xu This is my paper · ORCID
classification cs.CLcs.AI
keywords multi-agentdebatetokencostreductionLLMreasoningredundancyfilteringsimilaritycalculationsparsecommunicationgroupearlystopping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-agent debate improves LLM reasoning, but its token cost grows quickly with the number of agents and rounds. This paper claims most of that cost is wasted: agents in a debate keep repeating similar viewpoints. The authors propose S2-MAD, which measures how similar each incoming response is to what an agent already has, filters out duplicates, and lets agents stay silent when there is nothing new to say. Across five reasoning benchmarks, S2-MAD cuts token use by up to 94.5% versus full debate while keeping accuracy loss below 2%. If that holds, debate-based reasoning becomes far cheaper to scale.

What carries the argument

The Decision-Making Mechanism, composed of three modules: the Similarity Calculation Module, which gauges whether two responses express the same viewpoint by extracting final answers with regular expressions or by computing cosine similarity between embedded responses; the Redundancy Filtering Module, which discards any incoming response similar to the agent's own or previously accepted viewpoints; and the Conditional Participation Module, which keeps an agent silent when all incoming viewpoints match its own. This mechanism converts a fully connected debate graph into a sparse, adaptive one and also enables early termination when agent summaries converge.

What would settle it

Construct a dataset of problems where two agents give the same final answer through different derivation paths (for example, same numeric answer via different algebraic routes), run S2-MAD with the redundancy filter enabled, and check whether accuracy falls relative to a version that keeps both derivations; a measurable drop would show the filter is discarding information that identical answers do not capture.

Watch

Extended reading notes

Core claim

The paper's central claim is that the token cost of multi-agent debate can be reduced by an order of magnitude without meaningful accuracy loss by treating redundant viewpoints as non-information. S2-MAD's Decision-Making Mechanism compares each incoming response with the agent's own through either regular-expression answer matching or embedding-based cosine similarity; responses judged similar are filtered out, and an agent participates only when it holds a viewpoint that differs from what it has already seen. The debate stops early when all group summaries agree, and the final answer comes from majority voting. The authors report token-cost reductions of up to 94.5% against standard MAD, 90.2% against Sparse-MAD, and 87.0% against GroupDebate, with accuracy degradation below 2.0% on GSM8K, MATH, MMLU, GPQA, and an arithmetic task. Their asymptotic analysis gives S2-MAD a token complexity of $O(MTQ + (M^2T/N + MSN)CP)$, which drops below MAD's complexity once the average participation probability $P$ shrinks as answers converge.

Load-bearing premise

The load-bearing premise is that a response similar to what an agent already has—same final answer or high cosine similarity—carries no information worth exchanging; if that fails, the filter can discard a useful justification or keep a redundant one.

Editorial extensions

If this is right

  • Adding agents or debate rounds under S2-MAD costs far fewer tokens than under standard MAD once answers start to converge, so scaling debate becomes affordable in token-limited settings.
  • The similarity threshold is a direct control on the trade-off: lower thresholds save more tokens but risk discarding useful viewpoints, and the optimal threshold varies by dataset.
  • Using vectorization with cosine similarity can outperform regex answer matching, yielding both higher accuracy and lower token cost at a tuned threshold.
  • Early stopping when group summaries align is a cheap source of savings: removing it slightly lowers accuracy and raises tokens in the ablation.
  • The redundancy filter is the main cost-saving module: removing it raises GSM8K accuracy by about 2 points but nearly triples token usage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the redundancy-is-waste assumption generalizes, other ensemble and collaborative LLM methods (e.g., self-consistency sampling, retrieval-augmented debate) could adopt similarity gating to cut compute, since the paper's mechanism does not depend on the debate format specifically.
  • The paper's similarity module compares final answers or surface embeddings, not the reasoning traces behind them; a natural next step is measuring similarity on the justification level, which the current regex/embedding judge does not capture.
  • The reported cost savings are tied to benchmark conditions where agent answers converge quickly; on adversarial or open-ended tasks where diverse answers persist, the paper's own cost analysis implies the savings will shrink, so the 94.5% figure should not be read as a universal bound.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes S2-MAD, a sparsification strategy for multi-agent debate (MAD) in which agents selectively participate based on the similarity of responses to their own, using redundancy filtering, conditional participation, and early stopping. The authors report token-cost reductions of up to 94.5% compared to MAD while claiming 'performance degradation below 2.0%' in the abstract, and they present a theoretical complexity analysis together with experiments on five datasets (GSM8K, MATH, MMLU, GPQA, Arithmetic) using several commercial and open-source LLMs.

Significance. If the reported token reductions hold, S2-MAD addresses a real scalability bottleneck for multi-agent debate. The empirical study is broad: it covers multiple models and datasets, compares against MAD, S-MAD, GroupDebate, CoT, and CoT-SC, and reports both token counts and accuracy with standard deviations. The headline finding of large and consistent token savings is credible and practically useful. However, the abstract's blanket degradation bound is not supported by the paper's own tables, and the theoretical analysis is not a rigorous complexity separation. The practical contribution is still real if the claims are revised to per-configuration statements.

major comments (3)
  1. [Abstract; Tables 1, 3, 7] The abstract's claim of 'maintaining performance degradation below 2.0%' is contradicted by the paper's own results. In Table 1, GPT-4-0613 MMLU accuracy drops from 90.8 (MAD) to 88.1 (S2-MAD), a 2.7-point drop; in Table 3, the 2+2+2+2 grouping drops from 86.7 (MAD) to 83.4, a 3.3-point drop; and in Table 7, GPT-4o-mini MMLU drops from 89.5 to 86.1 (3.4 points), while GPT-4o-0806 MATH drops from 79.0 to 75.3 (3.7 points). Since 'up to 94.5%' qualifies only the token reduction, the degradation bound reads as a property of the method. The paper should either report degradation per configuration or replace the blanket claim with a specific list of configurations that satisfy it.
  2. [Section 3.3 and Appendix B] The 'theoretical demonstration' of the token-cost advantage is essentially an accounting identity rather than a proven complexity separation. The final bound, Token = O(MTQ + (M^2 T/N + M S N) C P), is written in terms of P, the upper bound on the average participation probability, which is an unmeasured assumption. The Discussion asserts that agents' answers 'tend to become consistent' and that costs only approach GroupDebate levels when different answers occur with probability only p^{MN}, but no derivation of P from the method's similarity mechanism is given, and no empirical P is reported. As a result, the claimed asymptotic improvement over MAD is conditional on a favorable value of P, and the paper does not establish that this value is achieved. This weakens contribution 2 and should be reframed as an empirical observation.
  3. [Section 4.3] The similarity threshold tau is a free parameter that materially affects both accuracy and token cost, and Figure 3 shows that the optimal threshold differs by dataset (approximately 0.1 for GSM8K and 0.4 for MATH) and that token cost rises sharply for tau near 0.85. The main results use regular-expression matching, while the VecCS variant is described as an alternative; however, the paper does not provide a principled way to set tau for a new task, so the reported efficiency gains may depend on dataset-specific tuning. The methodology section should state how this parameter would be chosen in practice.
minor comments (5)
  1. [Section 5.2] The paragraph after the ChatEval discussion contains a long block of '/uni00000018/uni...' artifact text that appears to be a corrupted citation or LaTeX error; it should be removed or repaired.
  2. [Section 3.3] The sentence 'we summarize the outputs from whtnin each group' contains a typo ('whtnin' should be 'within').
  3. [Equation (1) and Equations (3)-(4)] The definition of D_i^t is difficult to parse: the case conditions are not clearly separated from the summation ranges, and the relationship between D_i^t and the sums in Equations (3) and (4) is not immediately transparent. Please rewrite these equations with clearer notation.
  4. [Section 4.2] The list '94.5%, 84.2%, 92.4%, 83.6% and 88.7%' does not specify which model or table row each reduction corresponds to; since Table 1 shows different reductions per model and dataset, please provide an explicit mapping.
  5. [Figure 5] The caption 'Scaling Study of Token Cost' is uninformative, and the axis labels contain a corrupted Unicode artifact; the figure should be regenerated for clarity.

Circularity Check

1 steps flagged · score 4.0 of 10

Empirical token-cost results are independent, but the theoretical token-cost advantage reduces to an assumed small participation probability; the abstract's '<2.0% degradation' claim is also unsupported by the paper's own tables, though that is a support issue rather than circularity.

  1. other [Section 3.3 (Token Cost Analysis, Discussion paragraph after Eq. 6); echoed in Appendix B.]
    "However, for the same M and T , since agents' answers tend to become consistent as the debate progresses, we define the probability of obtaining a answer different with other agents is p. Thus, the token cost will only increase to be comparable to that of Group Debate when different answers are obtained in each round, which occurs with a probability of only pM N."

    The claimed theoretical token-cost advantage is obtained by inserting P, 'the upper bound of the average probability of each agent participating in the debate globally,' into the complexity bound O(MTQ + (M^2T/N + MSN)CP). The Discussion then asserts P is small by positing a per-agent probability p of divergent answers and claiming all-group divergence has probability only p^{MN}. No independent estimate or bound on P or p is derived from the mechanism, so the 'theoretical demonstration' assumes the token savings (rare divergent answers) as its premise rather than deriving them. The empirical measurements in Tables 1, 3, 4, and 7 are independent and remain valid evidence, so this is a partial circularity confined to the theoretical claim.

full rationale

The paper's central empirical claim—up to 94.5% token reduction with maintained accuracy on five datasets—is measured directly against MAD, S-MAD, GD, and CoT-SC and does not depend on the theoretical framework. No fitted parameter is renamed as a prediction: thresholds, group sizes, and early stopping are reported as ablations and design choices. Self-citations (e.g., GroupDebate) appear only as baselines, not as load-bearing justification. The one circular element is the 'theoretical demonstration' of token advantage in Section 3.3: the cost formula includes the average participation probability P by construction, and the Discussion's assertion that P is small (via p^{MN}) is an assumption that encodes the conclusion. This weakens Contribution 2 but does not invalidate the empirical token measurements. Separately, the abstract's 'performance degradation below 2.0%' is contradicted by the paper's own tables (e.g., Table 1 GPT-4 MMLU 90.8 to 88.1; Table 7 GPT-4o-mini MMLU 89.5 to 86.1 and GPT-4o-0806 MATH 79.0 to 75.3); this is an accuracy/support problem rather than a circularity and is not counted in the score beyond noting the headline overstates the results.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical; the theoretical complexity bound depends on an assumed small participation probability P and on similarity-based redundancy filtering. No new physical or ontological entities are introduced.

free parameters (3)
  • cosine similarity threshold tau (VecCS variant) = 0.40 for MATH, ~0.1 for GSM8K
    Section 4.3 reports the optimal threshold varies by dataset; accuracy and token savings depend strongly on this value.
  • participation probability upper bound P
    The theoretical complexity bound includes P; the paper assumes P is small because answers converge but never measures or bounds it independently.
  • grouping strategy (number and size of groups) = varies, e.g., 2+2+2+2, 4+4, 2+6
    Table 3 shows token cost and accuracy trade-off changes with the group partition; the choice is made by the experimenter.
assumptions (4)
  • domain assumption Each agent's output and each summary have a common token upper bound C.
    Used throughout Section 3.3 and Appendix B to derive token complexity.
  • ad hoc to paper Agents' answers become increasingly consistent as debate progresses, so the average participation probability P is small.
    Section 3.3 Discussion states this without derivation; it is the load-bearing premise for the theoretical token advantage.
  • domain assumption Identical final answers or high cosine similarity between responses indicates redundant viewpoints that can be filtered without losing information needed for accuracy.
    Section 3.2 Redundancy Filtering Module relies on this; the Limitation section concedes the judge can fail on paraphrases.
  • domain assumption Token cost formulas for MAD, S-MAD, and GroupDebate cited from prior work are correct.
    Section 2 uses these as baselines for the complexity comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of S$^2$-MAD: Breaking the Token Barrier to Enhance Multi-Agent Debate Efficiency." pith.science (2026). https://pith.science/paper/XR5EZOJI

@misc{pith2026250204790,
  author       = {Pith},
  title        = {Pith review of: S$^2$-MAD: Breaking the Token Barrier to Enhance Multi-Agent Debate Efficiency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XR5EZOJI}},
  note         = {Machine review of arXiv:2502.04790}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable capabilities across various natural language processing (NLP) scenarios, but they still face challenges when handling complex arithmetic and logical reasoning tasks. While Chain-Of-Thought (CoT) reasoning, self-consistency (SC) and self-correction strategies have attempted to guide models in sequential, multi-step reasoning, Multi-agent Debate (MAD) has emerged as a viable approach for enhancing the reasoning capabilities of LLMs. By increasing both the number of agents and the frequency of debates, the performance of LLMs improves significantly. However, this strategy results in a significant increase in token costs, presenting a barrier to scalability. To address this challenge, we introduce a novel sparsification strategy designed to reduce token costs within MAD. This approach minimizes ineffective exchanges of information and unproductive discussions among agents, thereby enhancing the overall efficiency of the debate process. We conduct comparative experiments on multiple datasets across various models, demonstrating that our approach significantly reduces the token costs in MAD to a considerable extent. Specifically, compared to MAD, our approach achieves an impressive reduction of up to 94.5\% in token costs while maintaining performance degradation below 2.0\%.

Figures

Figures reproduced from arXiv: 2502.04790 by the authors.

Figure 1
Figure 1. Redundant Viewpoints Exchange between Agents. The perspectives of Agent 1 and Agent 3 demonstrate a notable similarity. Throughout the debate, these viewpoints are exchanged with Agent 2, who receives these akin and repetitive viewpoints. Although the reduction in token cost have achieved by the aforementioned approaches, our experiment reveals a substantial presence of redun￾dancy and duplicate information in the i… view at source ↗
Figure 2
Figure 2. Process of S 2 -MAD. The S 2 -MAD includes three stages: all agents generate initial responses inde￾pendently at the first round and participate in group discussions to reach consensus under a Decision-Making Mechanism, which comprises: (1) Similarity calculation module accesses the similarity of responses either between or within groups. (2) Redundancy filter module filters redundant information, retaining only uni… view at source ↗
Figure 3
Figure 3. The relationship between the threshold τ , ACC, and Token Cost on the GSM8K and MATH datasets. Group Strategy. To assess the impact of differ￾ent grouping strategies on performance and token cost, we conducted experiments involving 8 agents across 3 rounds on the GSM8K. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Scaling study of Agents and Rounds. Although our method has not yet achieved optimal performance in terms of accuracy and token cost, it still shows a slight improvement over the MAD method while significantly reducing token usage, highlighting the efficiency of our pr…
Figure 5
Figure 5. Figure 5: Scaling Study of Token Cost. foster human-like interactions and evaluations in natural language dialogue. Moreover, (Wang et al., 2023) address cognitive constraints in multi-agent debates by integrating prior knowledge retrieval and a self-selection module, enhancing …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Agent Design: From Top-Down Workflows to Bottom-Up Skill Evolution

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Agents that start with no game knowledge can build a reusable skill library through trial-and-error and visual feedback, then progress further in two complex games than baseline agents given extra hints.

  2. Toward Efficient Agents: Memory, Tool learning, and Planning

    cs.AI 2026-01 conditional novelty 3.0 of 10

    A survey that organizes efficiency techniques for LLM agents into memory, tool learning, and planning, and consolidates benchmarks and metrics for measuring cost-performance trade-offs.

Reference graph

Works this paper leans on

47 extracted references · 2 canonical work pages · cited by 2 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403

    arXiv 2023

  3. [3]

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. 2024. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682--17690

    2024

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  5. [5]

    S \'e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712

    arXiv 2023

  6. [6]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201

    arXiv 2023

  7. [7]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 47 references
  1. [9]

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325

    2023 arXiv

  2. [10]

    Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. 2023. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142

    2023 arXiv

  3. [11]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  4. [12]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  5. [13]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751

    2019 arXiv

  6. [14]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798

    2023 arXiv

  7. [15]

    Ziqi Jin and Wei Lu. 2023. Tab-cot: Zero-shot tabular chain of thought. arXiv preprint arXiv:2305.17812

    2023 arXiv

  8. [16]

    Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. 2022. Lambada: Backward chaining for automated reasoning in natural language. arXiv preprint arXiv:2212.13894

    2022 arXiv

  9. [17]

    Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene Ie. 2024. Improving multi-agent debate with sparse communication topology. arXiv preprint arXiv:2406.11776

    2024 arXiv

  10. [18]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118

    2023 arXiv

  11. [19]

    Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439

    2023 arXiv

  12. [20]

    Tongxuan Liu, Xingyu Wang, Weizhe Huang, Wenjiang Xu, Yuting Zeng, Lei Jiang, Hailong Yang, and Jing Li. 2024. Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion. arXiv preprint arXiv:2409.14051

    2024

  13. [21]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

    2024

  14. [22]

    Xuefei Ning, Zinan Lin, Zixuan Zhou, Huazhong Yang, and Yu Wang. 2023. Skeleton-of-thought: Large language models can do parallel decoding. arXiv preprint arXiv:2307.15337

    2023 arXiv

  15. [23]

    Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305.12295

    2023 arXiv

  16. [24]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training

    2018

  17. [25]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  18. [26]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. https://arxiv.org/abs/2311.12022 Gpqa: A graduate-level google-proof q&a benchmark . Preprint, arXiv:2311.12022

    2023 arXiv

  19. [27]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36

    2024

  20. [28]

    KaShun Shum, Shizhe Diao, and Tong Zhang. 2023. Automatic prompt augmentation and selection with chain-of-thought from labeled data. arXiv preprint arXiv:2302.12822

    2023 arXiv

  21. [29]

    Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. 2023. Gpt-4 doesn't know it's wrong: An analysis of iterative prompting for reasoning problems. arXiv preprint arXiv:2310.12397

    2023 arXiv

  22. [30]

    Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. 2023. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration. arXiv preprint arXiv:2310.00280

    2023 arXiv

  23. [31]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  24. [32]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  25. [33]

    Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. 2023. Can large language models really improve by self-critiquing their own plans? arXiv preprint arXiv:2310.08118

    2023 arXiv

  26. [34]

    Haotian Wang, Xiyuan Du, Weijiang Yu, Qianglong Chen, Kun Zhu, Zheng Chu, Lian Yan, and Yi Guan. 2023. Apollo's oracle: Retrieval-augmented reasoning in multi-agent debates. arXiv preprint arXiv:2312.04854

    2023 arXiv

  27. [35]

    Siyuan Wang, Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, and Nan Duan. 2021. Logic-driven context extension and data augmentation for logical reasoning of text. arXiv preprint arXiv:2105.03659

    2021 arXiv

  28. [36]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  29. [37]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  30. [38]

    Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053

    2022 arXiv

  31. [39]

    Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. 2023. Examining inter-consistency of large language models collaboration: An in-depth analysis via debate. arXiv preprint arXiv:2305.11595

    2023 arXiv

  32. [40]

    Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. Advances in Neural Information Processing Systems, 35:3082--3095

    2022

  33. [41]

    Zhenran Xu, Senbao Shi, Baotian Hu, Jindi Yu, Dongfang Li, Min Zhang, and Yuxiang Wu. 2023. Towards reasoning in large language models via multi-agent peer review collaboration. arXiv preprint arXiv:2311.08152

    2023 arXiv

  34. [42]

    Jianhao Yan, Jin Xu, Chiyu Song, Chenming Wu, Yafu Li, and Yue Zhang. 2023. Understanding in-context learning from repetitions. arXiv preprint arXiv:2310.00297

    2023 arXiv

  35. [43]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  36. [44]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476--15488

    2022

  37. [45]

    Denny Zhou, Nathanael Sch \"a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625

    2022 arXiv

  38. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.