Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training a single leader LLM on the outputs of a fixed team of off-the-shelf agents—MLPO—outperforms single-agent and multi-agent baselines on BBH, MATH, and MMLU, and the trained leader stays ahead even when deployed alone.

desk verdict Worthwhile paper with a plausible single-leader training method and strong multi-agent results, but the zero-shot claim needs a matched-compute control before it can be believed. read the letter →

arxiv 2507.08960 v1 pith:GRN4XRHB submitted 2025-07-11 cs.MA cs.AIcs.LG

Andrew Estornell , Jean-Francois Ton , Muhammad Faaiz Taufiq , Hang Li This is my paper
classification cs.MAcs.AIcs.LG
keywords multi-agentLLMcollaborationhierarchicalreasoningGRPOleaderpolicyoptimizationreinforcementlearningfine-tuningzero-shottransferbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the benefits of multi-agent collaboration can be captured by training a single 'leader' model while the rest of the team stays fixed and off-the-shelf. Its method, MLPO, is a variant of Group Relative Policy Optimization in which the leader's training prompt includes candidate solutions from three untrained peer agents, forcing the leader to evaluate, weigh, and synthesize them rather than solve from scratch. On Big-Bench Hard, MATH, and MMLU, the resulting 7–9B-scale leader outperforms both single-agent training (GRPO, SCoRe) and multi-agent training (ACC-Collab, SelectLLM, a deferral leader), and the gain persists when the team is removed entirely at inference. If this is right, multi-agent reasoning no longer requires training or deploying a whole team: one cheaply trained model gets most of the benefit, with additional gains available when the team is present.

What carries the argument

The load-bearing mechanism is the MLPO objective: the standard Group Relative Policy Optimization (GRPO) loss, computed on roll-outs whose prompts append the K agent solutions to the task, so the leader's advantage is estimated relative to peers that saw the same team outputs. Around it sits a hierarchical inference loop in which agents propose solutions in round 0, the leader emits a structured <think>/<answer> synthesis, agents revise against the leader's output, and the cycle repeats for T=5 rounds. Training data is built by sampling four solutions per agent, forming four distinct prompts per task, filtering out tasks the team already solves, and optionally continuing with MLPO+ on responses from later rounds; an SFT phase with synthetic backtracking examples teaches natural self-correction, though the paper reports the GRPO phase carries most of the gain.

What would settle it

Train the zero-shot GRPO baseline on the same tasks with the same number of training examples, optimization steps, and compute as MLPO, but without any agent responses in the prompt; if its zero-shot accuracy matches the MLPO leader's accuracy, the multi-agent guidance is not the source of the reported gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that training a single leader LLM to aggregate and evaluate responses from a fixed team of three untrained agents improves performance both in collaboration and alone. Concretely, MLPO produces a leader whose accuracy on BBH, MATH, and MMLU exceeds that of single-agent trained models (zero-shot GRPO, SCoRe, self-refine variants) and of multi-agent trained baselines (ACC-Collab, SelectLLM, Deferral Leader), all at 7–9B parameter scale. The same leader achieves higher zero-shot accuracy without the team than a model trained with standard GRPO, which the paper attributes to a richer training signal from diverse agent solutions. The paper also argues the leader genuinely uses the team at inference: accuracy rises when agent responses are provided, and the trained leader dominates its untrained team and the individual agents across nearly all categories and difficulty levels.

Load-bearing premise

The load-bearing premise is that MLPO's advantage over the single-agent GRPO baseline comes from the agent-conditioned training prompts rather than from the leader simply receiving more training examples or more optimization steps.

Editorial extensions

If this is right

  • Only the leader needs training; the supporting agents stay off-the-shelf, so the approach scales to larger teams without multiplying training cost.
  • Deployed solo with no extra inference compute, the MLPO-trained leader still beats single-agent GRPO, so the training signal transfers a free accuracy gain to ordinary single-model use.
  • Deployed with the team, accuracy rises further, and under a fixed 40-generation inference budget the MLPO leader still wins on all three benchmarks after just two rounds.
  • Continuing training on later-round interactions (MLPO+) adds accuracy on all benchmarks—on BBH from 0.882 to 0.920—at little additional cost because the data is collected offline.
  • The leader keeps high accuracy when few or none of the agents are correct, indicating it learns to verify and override answers rather than copy them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the zero-shot gain is genuine, agent-conditioned prompts are acting as implicit exploration for RL fine-tuning; a direct test would feed the model its own previous samples as pseudo-agents and check whether the gain reproduces without any external team.
  • The easy-task filtering result points to a curriculum-by-agent-difficulty principle—redirecting training away from tasks the team already solves sharpens synthesis and override behavior—and varying the 75% filter threshold would map that trade-off directly.
  • The ablation showing reasoning-only inputs often beat answer-only inputs suggests the leader internalizes agents' reasoning rather than their verdicts; corrupting agent reasoning while leaving final answers correct would quantify how much the leader relies on the argument itself.
  • Because the paper does not report the number of training examples or optimization steps behind the single-agent GRPO baselines in Figure 4 and Table 1, a matched-compute re-run with identical data and step counts is the cleanest way to separate the benefit of agent-conditioned prompts from the benefit of more training, and the paper's own conclusion lists longer context windows and sequential lead
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MLPO (Multi-agent guided Leader Policy Optimization), a hierarchical multi-agent training scheme in which a single leader LLM is trained to synthesize and evaluate responses from a fixed team of off-the-shelf agents. Training proceeds through an SFT phase that generates backtracking-style data and a GRPO-style RL phase whose prompts are augmented with agent solutions; an extension, MLPO+, adds training on later-round agent responses. The authors report accuracy on BBH, MATH, and MMLU for Qwen2.5-7B leaders with a heterogeneous agent team, claiming substantial gains over single-agent baselines (GRPO, SCoRe, Self-Reflect) and multi-agent baselines (ACC-Collab, SelectLLM, Deferral Leader), including a zero-shot gain when the leader is deployed without the team. They also present inference-compute-controlled majority-vote results with a 40-generation budget and extensive ablations on team composition, number of solution sets, filtering, and ordering.

Significance. If the central claims hold, the paper makes a useful contribution: it shows that a single trained leader can coordinate untrained peers, reducing the training cost of multi-agent methods, and that the training signal from agent-conditioned prompts can improve the leader even in zero-shot deployment. The paper has real strengths: Figure 3 explicitly controls inference-time compute by capping all methods at 40 generations, the ablations in Section 4.3 examine several plausible confounds (number of solution sets, easy-task filtering, ordering), and results are reported with error bars and replicated for two additional leader models in Appendix A. However, the most novel claim, the zero-shot improvement of MLPO over single-agent GRPO, is not currently supported by a matched training-budget comparison, and one ablation table contains a label/value inconsistency. These issues are fixable but are load-bearing for the paper's central message.

major comments (3)
  1. [Section 4.2.3; Figure 4; Appendix B] The zero-shot comparison between MLPO and Zero-Shot GRPO is not controlled for training compute or training-data scale. Section 4.2.3 attributes the zero-shot gap to 'the training prompts,' but the MLPO pipeline in Section 3.2 uses four agent-conditioned prompts per task, an SFT phase that queries the leader 16 times per task, and an additional filtering step, while Appendix B lists only learning-rate and KL settings for Zero-Shot GRPO with no number of training examples, optimization steps, batch size, or total compute. Without a matched-budget control, the Figure 4 gain could reflect more training data or more optimization steps rather than the agent-conditioned prompt structure. Please report full training budgets for all methods and include a GRPO baseline trained with the same number of prompt-response pairs and steps, or otherwise demonstrate that the zero-shot advantage survives a matched-compute comparison.
  2. [Section 4.3; Table 3] The row labeled 'GRPO Zeroshot Data' in Table 3 reports 0.747/0.814/0.718, which numerically matches the 'Self-Reflect GRPO' row in Table 1 (0.747/0.814/0.718), not the 'Zero-Shot GRPO' row in Table 1 (0.742/0.791/0.712). This inconsistency makes it impossible to determine which baseline is actually being compared in the zero-shot analysis and prevents reconstruction of the comparison that Section 4.2.3 relies on. Please correct the labels or the values and ensure that all table entries are traceable to the corresponding experimental runs.
  3. [Sections 3.2.2 and 4.3; Table 5] The difficulty-based filtering of 'easy' tasks is applied only to MLPO training (the 75% threshold in Section 3.2.2), and Table 5 shows that this filter improves BBH accuracy from 0.869 to 0.882. The GRPO, SCoRe, and ACC-Collab baselines are not reported with an equivalent filtering step, so a portion of the observed MLPO gain relative to these baselines could be a curriculum/filtering effect rather than an effect of multi-agent-guided conditioning. Add a control in which the same easy-task filter is applied to a single-agent GRPO baseline (or report filtered and unfiltered results for all training-based methods) so that the contribution of the filter can be separated from the contribution of the agent-conditioned prompts.
minor comments (5)
  1. [Throughout] There are several typos and spacing errors that should be corrected: 'Deferal Leader' in Tables 6 and 7, 'retrianing' in Appendix B, '4 our of 7' in Section 4.2.4, 'divers solutions' in Section 4.2.4, 'Alterative Choice of Leader' in Section 4.3, and 'ensue fair comparisons' in Section 4.3.
  2. [Table 1] Table 1 presents the main results without an inference-compute control, while Figure 3 provides that control; adding a footnote to Table 1 indicating that compute is not matched there would prevent readers from over-interpreting the raw accuracy gaps.
  3. [Section 3.2.2] The MLPO loss is described as using 'the modifications to the GRPO loss proposed in Dr. GRPO [40],' but the specific modifications (e.g., which clipping or advantage terms are changed) are not described; please specify them so the objective is reproducible without consulting a separate paper.
  4. [Section 4.2.2; Figure 3] The counting of 'total LLM generations' for the 40-generation budget is not fully specified for iterative methods; for example, it is unclear whether each agent's response and each leader response in every round count as one generation and how majority-vote runs are counted. Please state the exact counting rule in the caption or text.
  5. [Appendix B] The hyperparameter section covers baselines but does not list MLPO's full training configuration (number of steps, batch size, learning-rate schedule, SFT epochs); adding this information would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular construction: MLPO's gains are empirical benchmark comparisons; missing training-budget controls and minor self-citations do not amount to circularity.

full rationale

The paper's central claims are accuracy comparisons on held-out benchmark splits after training, not derivations from fitted parameters. The MLPO loss (Sec. 3.2.2) is a standard clipped-ratio policy-gradient objective whose only difference from GRPO is that the prompt includes agent responses; nothing in the loss defines or 'predicts' the test accuracy. The SFT and MLPO+ data constructions use the leader's own generations and filtered agent responses, but these are training-set building blocks, not predictions alleged to follow from a model. The zero-shot result (Sec. 4.2.3) is an empirical observation; it is not forced by construction, although the paper never reports training steps or examples for the GRPO baseline, and Table 3 labels 'GRPO Zeroshot Data' with the same numbers as 'Self-Reflect GRPO' in Table 1. That is an experimental-control and reporting weakness, not a definitional equivalence. Self-citations ([12], [13]) appear only as related-work motivation and baselines; the central result does not rest on them. No step exhibits the required reduction of output to input.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method is empirical RL and relies on standard assumptions about GRPO stability and the usefulness of agent-generated context. The hand-chosen thresholds (75% filter, 4 solution sets, 16 SFT samples) are training choices that the paper ablates; none of them are new physical or theoretical entities.

free parameters (4)
  • Easy-task filter threshold = 0.75
    Tasks where at least 75% of agent responses are correct are removed from MLPO training data. The threshold is set by hand and is shown to improve accuracy in Table 5.
  • Number of agent solutions per task = 4
    Four sets of agent responses are sampled per task during MLPO data generation; the ablation in Table 4 shows diminishing returns beyond 4.
  • Leader samples per prompt for SFT = 16
    16 leader completions are generated per agent-response set; tasks where all 16 are incorrect are discarded.
  • Inference rounds = 5
    All iterative methods are run for 5 rounds; this choice affects all methods equally.
assumptions (3)
  • domain assumption GRPO with sparse correct or incorrect rewards plus a formatting reward trains effective reasoning without reward hacking
    The method depends on this assumption, inherited from DeepSeekMath and Dr. GRPO, and uses it to train the leader.
  • domain assumption Responses from a fixed, diverse team of off-the-shelf LLMs provide useful supervision to train a leader
    Core premise of MLPO; if the agent responses were not informative, the leader would learn nothing beyond its own outputs.
  • domain assumption Standard benchmark train and test protocol is valid for MMLU and BBH despite unspecified split construction
    The paper says training split but does not describe how training data is obtained for MMLU and BBH, which are typically used for evaluation only.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs." pith.science (2026). https://pith.science/paper/GRN4XRHB

@misc{pith2026250708960,
  author       = {Pith},
  title        = {Pith review of: How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GRN4XRHB}},
  note         = {Machine review of arXiv:2507.08960}
}
read the original abstract

Large Language Models (LLMs) have achieved strong performance on a wide range of complex reasoning tasks, yet further gains are often possible by leveraging the complementary strengths of multiple models. While multi-agent frameworks can improve solution quality by leveraging multiple LLMs, existing methods are often computationally expensive, both at training and inference time. In this work, we introduce a hierarchical multi-agent framework that addresses these challenges by training only a single leader LLM to coordinate a team of untrained peer agents. To this end, we propose Multi-agent guided Leader Policy \textbf{O}ptimization (MLPO), a novel approach which trains the leader to evaluate and synthesize agent responses without auxiliary value networks or explicit agent feedback. Leaders trained with MLPO exhibit improved performance not only when interacting with the agent team at inference time, but also enjoy improved performance when deployed in single-agent settings without the team. Empirical results on Big-Bench Hard (BBH), MATH, and MMLU demonstrate that our framework achieves substantial performance improvements over both single-agent and multi-agent baselines. Our results highlight the effectiveness and efficiency of training a single, flexible leader for collaborative reasoning in multi-agent LLM systems.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mathematical methods of reinforcement learning

    math.OC 2026-07 accept

    A survey unifying the operator-theoretic, probabilistic, and optimization-based mathematical structures underlying modern reinforcement learning algorithms.

Reference graph

Works this paper leans on

81 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682–17690, 2024

    2024

  2. [2]

    Hudson, Ehsan Adeli, Russ Altman, and et al

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, and et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  3. [3]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, and et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, and et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  4. [4]

    Chateval: Towards better llm-based evaluators through multi-agent debate.arXiv preprint arXiv:2308.07201, 2023

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate.arXiv preprint arXiv:2308.07201, 2023

    arXiv 2023

  5. [5]

    Socrasynth: Multi-llm reasoning with conditional statistics.arXiv preprint arXiv:2402.06634, 2024

    Edward Y Chang. Socrasynth: Multi-llm reasoning with conditional statistics.arXiv preprint arXiv:2402.06634, 2024

    arXiv 2024

  6. [6]

    Reconcile: Round-table conference improves reasoning via consensus among diverse llms.arXiv preprint arXiv:2309.13007, 2023

    Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms.arXiv preprint arXiv:2309.13007, 2023

    arXiv 2023

  7. [7]

    Symbolic mixture- of-experts: Adaptive skill-based routing for heterogeneous reasoning.arXiv preprint arXiv:2503.05641, 2025

    Justin Chih-Yao Chen, Sukwon Yun, Elias Stengel-Eskin, Tianlong Chen, and Mohit Bansal. Symbolic mixture- of-experts: Adaptive skill-based routing for heterogeneous reasoning.arXiv preprint arXiv:2503.05641, 2025

    arXiv 2025

  8. [8]

    Universal self-consistency for large language model generation.arXiv preprint arXiv:2311.17311, 2023

    Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation.arXiv preprint arXiv:2311.17311, 2023

    arXiv 2023

Show all 81 references
  1. [9]

    Cost-effective online multi-llm selection with versatile reward models

    Xiangxiang Dai, Jin Li, Xutong Liu, Anqi Yu, and John Lui. Cost-effective online multi-llm selection with versatile reward models. arXiv preprint arXiv:2405.16587, 2024

    2024 arXiv

  2. [10]

    Improving factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. InForty-first International Conference on Machine Learning, 2023

    2023

  3. [11]

    Debate only when necessary: Adaptive multiagent collaboration for efficient llm reasoning

    Sugyeong Eo, Hyeonseok Moon, Evelyn Hayoon Zi, Chanjun Park, and Heuiseok Lim. Debate only when necessary: Adaptive multiagent collaboration for efficient llm reasoning. arXiv preprint arXiv:2504.05047, 2025. URL https://arxiv.org/abs/2504.05047

    2025 arXiv

  4. [12]

    Multi-llm debate: Framework, principals, and interventions.Advancesin Neural Information Processing Systems, 37:28938–28964, 2024

    Andrew Estornell and Yang Liu. Multi-llm debate: Framework, principals, and interventions.Advancesin Neural Information Processing Systems, 37:28938–28964, 2024

    2024

  5. [13]

    Acc-collab: An actor-critic approach to multi-agent llm collaboration.arXiv preprint arXiv:2411.00053, 2024

    Andrew Estornell, Jean-Francois Ton, Yuanshun Yao, and Yang Liu. Acc-collab: An actor-critic approach to multi-agent llm collaboration.arXiv preprint arXiv:2411.00053, 2024. 16

    2024 arXiv

  6. [14]

    Acc-collab: An actor-critic approach to multi-agent llm collaboration, 2024

    Andrew Estornell, Jean-Francois Ton, Yuanshun Yao, and Yang Liu. Acc-collab: An actor-critic approach to multi-agent llm collaboration, 2024

    2024

  7. [15]

    Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration.arXiv preprint arXiv:2402.00367, 2024

    Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration.arXiv preprint arXiv:2402.00367, 2024

    2024 arXiv

  8. [16]

    Modular pluralism: Pluralistic alignment via multi-llm collaboration.arXiv preprint arXiv:2406.15951, 2024

    Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. Modular pluralism: Pluralistic alignment via multi-llm collaboration.arXiv preprint arXiv:2406.15951, 2024

    2024 arXiv

  9. [17]

    When one llm drools, multi-llm collaboration rules.arXiv preprint arXiv:2502.04506, 2025

    Shangbin Feng, Wenxuan Ding, Alisa Liu, Zifeng Wang, Weijia Shi, Yike Wang, Zejiang Shen, Xiaochuang Han, Hunter Lang, Chen-Yu Lee, et al. When one llm drools, multi-llm collaboration rules.arXiv preprint arXiv:2502.04506, 2025

    2025 arXiv

  10. [18]

    Heterogeneous swarms: Jointly optimizing model roles and weights for multi-llm systems

    Shangbin Feng, Zifeng Wang, Palash Goyal, Yike Wang, Weijia Shi, Huang Xia, Hamid Palangi, Luke Zettlemoyer, Yulia Tsvetkov, Chen-Yu Lee, et al. Heterogeneous swarms: Jointly optimizing model roles and weights for multi-llm systems. arXiv preprint arXiv:2502.04510, 2025

    2025

  11. [19]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  12. [20]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  13. [21]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, and et al. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  14. [22]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, and et al. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  15. [23]

    Metagpt: Meta programming for a multi-agent collaborative framework

    Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for a multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2024. URLhttps://arxiv.org/abs/...

    2024 arXiv

  16. [24]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.arXiv preprint arXiv:2311.05232, 2023

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, and et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.arXiv preprint arXiv:2311.05232, 2023

    2023 arXiv

  17. [25]

    Ensemble learning for heterogeneous large language models with deep parallel collaboration.Advancesin Neural Information Processing Systems, 37:119838–119860, 2024

    Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Ting Liu, and Bing Qin. Ensemble learning for heterogeneous large language models with deep parallel collaboration.Advancesin Neural Information Processing Systems, 37:119838–119860, 2024. 18

    2024

  18. [26]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  19. [27]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.arXiv preprint arXiv:2306.02561, 2023

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.arXiv preprint arXiv:2306.02561, 2023

    2023 arXiv

  20. [28]

    Two heads are better than one: Test-time scaling of multi-agent collaborative reasoning.arXiv preprint arXiv:2504.09772, 2025

    Can Jin, Hongwu Peng, Qixin Zhang, Yujin Tang, Dimitris N Metaxas, and Tong Che. Two heads are better than one: Test-time scaling of multi-agent collaborative reasoning.arXiv preprint arXiv:2504.09772, 2025

    2025 arXiv

  21. [29]

    Bowman, Tim Rocktäschel, and Ethan Perez

    Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive llms leads to more truthful answers, 2024. URLhttps://arxiv.org/abs/2402.06782

    2024 arXiv

  22. [30]

    Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024

    2024 arXiv

  23. [31]

    Smoa: Improving multi-agent large language models with sparse mixture-of-agents.arXiv preprint arXiv:2411.03284, 2024

    Dawei Li, Zhen Tan, Peijia Qian, Yifan Li, Kumar Satvik Chaudhary, Lijie Hu, and Jiayi Shen. Smoa: Improving multi-agent large language models with sparse mixture-of-agents.arXiv preprint arXiv:2411.03284, 2024

    2024 arXiv

  24. [32]

    Two heads are better than one: Dual-model verbal reflection at inference-time.arXiv preprint arXiv:2502.19230, 2025

    Jiazheng Li, Yuxiang Zhou, Junru Lu, Gladys Tyen, Lin Gui, Cesare Aloisi, and Yulan He. Two heads are better than one: Dual-model verbal reflection at inference-time.arXiv preprint arXiv:2502.19230, 2025

    2025

  25. [33]

    Prd: Peer rank and discussion improve large language model based evaluations

    Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.02762, 2023

    2023 arXiv

  26. [34]

    From drafts to answers: Unlocking llm potential via aggregation fine-tuning.arXiv preprint arXiv:2501.11877, 2025

    Yafu Li, Zhilin Wang, Tingchen Fu, Ganqu Cui, Sen Yang, and Yu Cheng. From drafts to answers: Unlocking llm potential via aggregation fine-tuning.arXiv preprint arXiv:2501.11877, 2025

    2025 arXiv

  27. [35]

    Improving multi-agent debate with sparse communication topology.arXiv preprint arXiv:2406.11776, 2024

    Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene Ie. Improving multi-agent debate with sparse communication topology.arXiv preprint arXiv:2406.11776, 2024

    2024 arXiv

  28. [36]

    Encouraging divergent thinking in large language models through multi-agent debate.arXiv preprint arXiv:2305.19118, 2023

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate.arXiv preprint arXiv:2305.19118, 2023

    2023 arXiv

  29. [37]

    Marft: Multi-agent reinforcement fine-tuning.arXiv preprint arXiv:2504.16129, 2025

    Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. Marft: Multi-agent reinforcement fine-tuning.arXiv preprint arXiv:2504.16129, 2025

    2025 arXiv

  30. [38]

    Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion

    Tongxuan Liu, Xingyu Wang, Weizhe Huang, Wenjiang Xu, Yuting Zeng, Lei Jiang, Hailong Yang, and Jing Li. Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion. arXiv preprint arXiv:2409.14051, 2024

    2024

  31. [39]

    Towards hierarchical multi-agent workflows for zero-shot prompt optimization.arXiv preprint arXiv:2405.20252, 2024

    Yuchi Liu, Jaskirat Singh, Gaowen Liu, Ali Payani, and Liang Zheng. Towards hierarchical multi-agent workflows for zero-shot prompt optimization.arXiv preprint arXiv:2405.20252, 2024. URLhttps://arxiv.org/abs/2405. 20252

    2024 arXiv

  32. [40]

    Understanding r1-zero-like training: A critical perspective, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025. URLhttps://arxiv.org/abs/2503.20783

    2025 arXiv

  33. [41]

    Hao Ma, Tianyi Hu, Zhiqiang Pu, Liu Boyin, Xiaolin Ai, Yanyan Liang, and Min Chen. Coevolving with the other you: Fine-tuning llm with sequential cooperative multi-agent reinforcement learning.Advances in Neural Information Processing Systems, 37:15497–15525, 2024

    2024

  34. [42]

    Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems, 36:46534–46594, 2023

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems, 36:46534–46594, 2023

    2023

  35. [43]

    Selectllm: Query-aware efficient selection algorithm for large language models.arXiv preprint arXiv:2408.08545, 2024

    Kaushal Kumar Maurya, KV Srivatsa, and Ekaterina Kochmar. Selectllm: Query-aware efficient selection algorithm for large language models.arXiv preprint arXiv:2408.08545, 2024

    2024 arXiv

  36. [44]

    Beyond accuracy: Evaluating the reasoning behavior of large language models

    Philipp Mondorf and Barbara Plank. Beyond accuracy: Evaluating the reasoning behavior of large language models. arXiv preprint arXiv:2407.00000, 2024

    2024

  37. [45]

    Motwani, Chandler Smith, Rocktim J

    Sumeet R. Motwani, Chandler Smith, Rocktim J. Das, Rafael Rafailov, Ivan Laptev, Philip H. S. Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. Malt: Improving reasoning with multi-agent llm training. arXiv preprint arXiv:2412.01928, 2025. URLhttps://arxiv.or...

    2025

  38. [46]

    Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning.arXiv preprint arXiv:2502.18439, 2025

    Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, and Joo-Kyung Kim. Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning.arXiv preprint arXiv:2502.18439, 2025

    2025 arXiv

  39. [47]

    O1 replication journey: A strategic progress report–part 1.arXiv preprint arXiv:2410.18982, 2024

    Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. O1 replication journey: A strategic progress report–part 1.arXiv preprint arXiv:2410.18982, 2024

    2024 arXiv

  40. [48]

    Towards collaborative intelligence: Propagating intentions and reasoning for multi-agent coordination with large language models

    Xihe Qiu, Haoyu Wang, Xiaoyu Tan, Chao Qu, Yujie Xiong, Yuan Cheng, Yinghui Xu, Wei Chu, and Yuan Qi. Towards collaborative intelligence: Propagating intentions and reasoning for multi-agent coordination with large language models. arXiv preprint arXiv:2407.12532, 2024

    2024 arXiv

  41. [49]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  42. [50]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URLhttps://arxiv.org/abs/1707.06347

    2017 arXiv

  43. [52]

    Malmm: Multi-agent large language models for zero-shot robotics manipulation

    Harsh Singh, Rocktim Jyoti Das, Mingfei Han, Preslav Nakov, and Ivan Laptev. Malmm: Multi-agent large language models for zero-shot robotics manipulation. arXiv preprint arXiv:2411.17636, 2024. URL https: //arxiv.org/abs/2411.17636

    2024 arXiv

  44. [53]

    Should we be going mad? a look at multi-agent debate strategies for llms.arXiv preprint arXiv:2311.17371, 2023

    Andries Smit, Paul Duckworth, Nathan Grinsztajn, Thomas D Barrett, and Arnu Pretorius. Should we be going mad? a look at multi-agent debate strategies for llms.arXiv preprint arXiv:2311.17371, 2023

    2023 arXiv

  45. [54]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, and et al. Challenging big-bench tasks and whether chain-of-thought can solve them. InFindings of ACL, 2022. URL https://arxiv.org/abs/ 2210.09261

    2022 arXiv

  46. [55]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

  47. [56]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  48. [57]

    Rema: Learning to meta-think for llms with multi-agent reinforcement learning.arXiv preprint arXiv:2503.09501, 2025

    Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, et al. Rema: Learning to meta-think for llms with multi-agent reinforcement learning.arXiv preprint arXiv:2503.09501, 2025

    2025 arXiv

  49. [58]

    Mixture-of-agents enhances large language model capabilities

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406.04692, 2024

    2024 arXiv

  50. [59]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  51. [60]

    Chain-of-thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, and et al. Chain-of-thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022

    2022 arXiv

  52. [61]

    White, Doug Burger, and Chi Wang

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation.arXiv preprint ...

    2023 arXiv

  53. [62]

    Examining inter-consistency of large language models collaboration: An in-depth analysis via debate.arXiv preprint arXiv:2305.11595, 2023

    Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Examining inter-consistency of large language models collaboration: An in-depth analysis via debate.arXiv preprint arXiv:2305.11595, 2023

    2023 arXiv

  54. [63]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  55. [64]

    Multi-llm collaborative search for complex problem solving.arXiv preprint arXiv:2502.18873, 2025

    Sen Yang, Yafu Li, Wai Lam, and Yu Cheng. Multi-llm collaborative search for complex problem solving.arXiv preprint arXiv:2502.18873, 2025

    2025 arXiv

  56. [65]

    Agentnet: Decentralized evolutionary coordination for llm-based multi-agent systems.arXiv preprint arXiv:2504.00587, 2025

    Yingxuan Yang, Huacan Chai, Shuai Shao, Yuanyi Song, Siyuan Qi, Renting Rui, and Weinan Zhang. Agentnet: Decentralized evolutionary coordination for llm-based multi-agent systems.arXiv preprint arXiv:2504.00587, 2025

    2025 arXiv

  57. [66]

    R1-sharevl: Incentivizing reasoning capability of multimodal large language models via share-grpo

    Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, et al. R1-sharevl: Incentivizing reasoning capability of multimodal large language models via share-grpo. arXiv preprint arXiv:2505.16673, 2025

    2025 arXiv

  58. [67]

    Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023

    2023

  59. [68]

    X-mas: Towards building multi-agent systems with heterogeneous llms.arXiv preprint arXiv:2505.16997, 2025

    Rui Ye, Xiangrui Liu, Qimin Wu, Xianghe Pang, Zhenfei Yin, Lei Bai, and Siheng Chen. X-mas: Towards building multi-agent systems with heterogeneous llms.arXiv preprint arXiv:2505.16997, 2025

    2025 arXiv

  60. [69]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen...

    2025 arXiv

  61. [70]

    Chain of agents: Large language models collaborating on long-context tasks.Advances in Neural Information Processing Systems, 37: 132208–132237, 2024

    Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Arik. Chain of agents: Large language models collaborating on long-context tasks.Advances in Neural Information Processing Systems, 37: 132208–132237, 2024

    2024

  62. [71]

    Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks.arXiv preprint arXiv:2503.15478, 2025

    Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks.arXiv preprint arXiv:2503.15478, 2025. 21

    2025 arXiv

  63. [72]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Qinghong Zhu, Yuchen Liu, Zijian Liu, Yifan Wu, Toni Yu, and et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 22 Appendix A Different Choice of Leader In the main body we use Qwen-2.5-7B-Instruct a...

    2024 arXiv

  64. [74]

    Carefully reflect on the aggregator’s feedback and your previous solution

  65. [75]

    Address each question raised where relevant

    Revise your answer step-by-step to improve its correctness and clarity. Address each question raised where relevant

  66. [76]

    Regardless of the approach, always conclude with: 25 Therefore, the final answer is: $\boxed{[answer]}$

    Double-check for any logical, calculation, or reasoning errors. Regardless of the approach, always conclude with: 25 Therefore, the final answer is: $\boxed{[answer]}$. Where [answer] is just the final number or expression that solves the problem. C.2 Leader Prompts The leader...

  67. [78]

    - End the answer with: Therefore, the final answer is: $\boxed{[answer]}$

    <answer>...</answer>: Your final answer should be aggregated from the best elements of the agents’ responses. - End the answer with: Therefore, the final answer is: $\boxed{[answer]}$. C.3 Generating Backtracking Data for SFT To teach the leader model natural backtracking and ...

  68. [79]

    **Prompt**: The original math question and agent responses

  69. [80]

    These should be carefully reviewed for mistakes

    **Incorrect aggregator response**: Response from a previous aggregator that contains errors in aggregation or reasoning and hence arrives at an incorrect final answer. These should be carefully reviewed for mistakes

  70. [81]

    Wait, that doesn’t seem right

    **Correct aggregator response**: Response from a previous aggregator that contains the correct reasoning, evaluation and aggregation. Here’s the input data for your aggregation: Question: [question text] Agent 1 response: [agent_1 response] Agent 2 response: [agent_2 response]...

  71. [82]

    <think>...</think>: A long, detailed chain-of-thought reasoning process

  72. [83]

    <answer>...</answer>: Your final answer should be aggregated from the best elements of the agents’ responses. - End the answer with: ‘Therefore, the final answer is: $\boxed{[answer]}$’ 30 Your output should follow this format: <think> [detailed thought process of the aggregat...

  73. [2024]

    URLhttps://arxiv.org/abs/2408.00118

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.