Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing a single uniform LLM with role-matched heterogeneous LLMs improves multi-agent performance without changing the workflow.

desk verdict A genuinely useful benchmark and a plausible case for heterogeneous MAS, but the headline AIME numbers are partially in-sample and the statistics are thinner than the abstract implies. read the letter →

arxiv 2505.16997 v1 pith:KEOBPYJS submitted 2025-05-22 cs.AI cs.CLcs.MA

Rui Ye , Xiangrui Liu , Qimin Wu , Xianghe Pang , Zhenfei Yin , Lei Bai , Siheng Chen This is my paper · ORCID
classification cs.AIcs.CLcs.MA
keywords heterogeneousmulti-agentsystemsLLM-basedagentsmodelselectionMASbenchmarkcollectiveintelligencerolespecializationreasoningmodelscollaboration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most multi-agent systems today run every agent on the same large language model, which caps the system at that model's weaknesses. This paper tries to show that the same agent framework can be made smarter simply by powering different agent roles with different models, chosen by how well each model performs that specific function in that domain. To support the claim, it builds a benchmark that scores 27 models on five agent functions—question-answering, revising, aggregating, planning, and evaluating—across five domains, and then uses those scores to assign models to roles in three existing multi-agent methods plus a prototype. On held-out tests, the heterogeneous assignments consistently beat every homogeneous configuration, with competition-math gains of up to 47 percentage points. The payoff, if the claim holds, is that many existing systems can be improved in minutes by better model allocation rather than by redesigning the collaboration.

What carries the argument

The load-bearing machinery is X-MAS-Bench, a controlled benchmark that isolates each of five agent functions—question-answering, revise, aggregation, planning, and evaluation—within five domains, across 27 LLMs and over 1.7 million evaluations, producing a per-function-domain ranking. The companion mechanism, X-MAS-Design, takes an existing homogeneous multi-agent system and replaces the single driver model with the top-ranked model for each role, keeping prompts, roles, and interaction topology fixed so that any measured gain is attributable to model heterogeneity rather than workflow changes. The benchmark's rankings are the engine: they turn the general idea of 'use diverse models' into a concrete, reproducible role-assignment recipe.

What would settle it

On a held-out competition-math set, run the same heterogeneous multi-agent system twice: once assigning each role the top X-MAS-Bench scorer for that function-domain, and once assigning the bottom scorer while keeping all other design choices fixed; if the bottom-scorer configuration matches or beats the top-scorer configuration, the benchmark rankings are not the causal ingredient behind the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that a multi-agent system need not be driven by a single LLM; assigning each agent role a different model chosen from a per-function, per-domain benchmark can improve the whole system's accuracy while keeping the framework's structure, prompts, and interaction logic untouched. The evidence is a two-part construction: X-MAS-Bench evaluates 27 models on five agent functions across five domains, and X-MAS-Design uses those rankings to replace the uniform model in three existing multi-agent methods and one prototype. On held-out tests, the heterogeneous configuration outperforms every homogeneous configuration on average, with the largest gains in competition math: one existing method rises from 20% to 50% on AIME-2024, another from 40% to 63%, and on the entirely new AIME-2025 set the prototype scores 46.67% against 13.33% for the best homogeneous chatbot. The paper also reports that increasing the number of candidate models generally improves performance, reinforcing the claim that heterogeneity itself is a source of value.

Load-bearing premise

The entire design rests on the assumption that a model's isolated score on a single function-domain test predicts how well that model will perform when playing the same role inside a full multi-agent conversation; if that transfer fails, the benchmark-guided role assignments lose their foundation.

Editorial extensions

If this is right

  • Any existing multi-agent system that keeps its role descriptions and interaction topology intact can be upgraded by swapping each role's uniform model for a role-matched model, so gains do not require redesigning the system.
  • A model that scores highest in one function-domain can be only mid-tier in another, so system builders should choose models per role rather than seeking a single overall leader.
  • Mixing instructed chatbots with reasoning models can produce large gains on competition mathematics, even though either type alone underperforms the mixed team.
  • Adding more candidate models to the pool generally improves the heterogeneous team, suggesting that the benefits scale with model diversity rather than with any single best model.
  • Gains appear on held-out and entirely new benchmarks, indicating that the role-assignment recipe transfers beyond the benchmark's own test sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the isolated per-function rankings may not fully capture cross-agent interaction effects, so a direct test would compare top-scorer assignments against random assignments on the same multi-agent task; the paper only compares against arbitrary hand-picked configurations.
  • Editorial inference: because smaller specialized models sometimes beat larger ones in the benchmark, heterogeneous role assignment is also a potential cost lever; the paper reports performance but does not compute cost per accuracy gain.
  • Editorial inference: the monotone improvement with more candidates suggests that gains come substantially from pool diversity, implying that even imperfect role assignments may help as long as the models differ; the paper's arbitrary-selection experiment hints at this but does not fully isolate diversity from ranking quality.
  • Editorial inference: the mixed chatbot-reasoner experiments use one chatbot and one reasoning model, so varying the number and strength of reasoners per role would map out when reasoning models help or hurt different roles.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper argues that LLM-based multi-agent systems need not be homogeneous: replacing the single model that drives all agents with per-role heterogeneous models can improve performance without redesigning the MAS structure. To support this, the authors construct X-MAS-Bench, evaluating 27 LLMs across 5 agent functions (question-answering, revise, aggregation, planning, evaluation) and 5 domains with 21 test sets, totaling over 1.7 million evaluations. They then propose X-MAS-Design, which assigns each role in an existing MAS the highest-scoring available model for that role according to X-MAS-Bench. Experiments on LLM-Debate, AgentVerse, DyLAN, and a custom X-MAS-Proto show consistent gains for heterogeneous configurations over homogeneous baselines, including large reported improvements on AIME-2024 and on the out-of-sample AIME-2025 and MATH-MAS benchmarks.

Significance. If the central claim holds, the paper makes a useful practical contribution: it offers a large, openly released benchmark for MAS-relevant agent functions and a simple, method-agnostic recipe for converting homogeneous MAS frameworks into heterogeneous ones. The breadth of the benchmark, the open-source release of code and data, and the inclusion of genuinely out-of-sample evaluations on AIME-2025 and MATH-MAS are strengths that go beyond many prior studies. The claim that heterogeneity can be exploited by a lightweight lookup-based role assignment, rather than by structural redesign or retraining, is plausible and worth publishing. However, the empirical support is more fragile than the abstract suggests: the headline AIME-2024 numbers are partly in-sample, no uncertainty quantification is provided, and several effect sizes rest on very small counts. The qualitative conclusion survives the current evidence, but the quantitative claims need substantial re-framing and re-analysis.

major comments (3)
  1. [Section C, Section E.1, Table 3, Introduction] AIME-2024 is listed among the 21 test sets of X-MAS-Bench in Section C, and Section E.1 derives the mixed chatbot-reasoner role assignments from X-MAS-Bench observations. The AIME-2024 results in Table 3 and the abstract's '47% performance boost' are therefore partially in-sample evaluations, which contradicts the Introduction's statement that the X-MAS-Design analysis 'covers five domains with no sample overlap compared to X-MAS-Bench.' The out-of-sample AIME-2025 and MATH-MAS results in Table 4 support the qualitative conclusion, but the paper should present those as the primary clean validation and clearly label the AIME-2024 numbers as having been used for model selection rather than as independent evidence.
  2. [Section 3.2, Table 3] The experiments use a single run at temperature 0.5 with no repeated seeds and no error bars or confidence intervals. On the 30-question AIME-2024 set, the AgentVerse improvement from 20% to 50% is a change from 6 to 15 correct answers, and the DyLAN improvement from 40% to 63% is a change from 12 to 19 correct answers. These small counts make the headline differences fragile on their own. Please report bootstrap confidence intervals or multiple-seed results, and temper claims that are not supported by the sample size.
  3. [Abstract, Section 4.3] The paper labels absolute percentage-point gains as percentages: the '47% performance boost' corresponds to a 46.67-percentage-point gain for X-MAS-Proto on AIME-2024, and the '8.4% performance improvement on MATH' corresponds to an 8.4-percentage-point gain in Table 2. Please distinguish percentage points from relative improvements throughout, since this materially changes the magnitude of the reported effect sizes.
minor comments (4)
  1. [Section 1, Section 5] There are typos: 'his paradigm' should be 'this paradigm' in Section 1, and 'genralists' should be 'generalists' in Section 5.
  2. [Figure 4, Section 4.4] The text says 'We follow the setup in Section 2' but the relevant experimental setup is described in Section 4.2; please correct the cross-reference.
  3. [Appendix D, Appendix E] The appendix lists concrete role assignments, but the mapping from MAS roles such as 'role assigner', 'critic', 'ranker', and 'node agent' to the five X-MAS-Bench functions is not stated. Since X-MAS-Design is defined as assigning the top scorer for each function, a documented role-to-function mapping would make the recipe reproducible.
  4. [Tables 1 and 5] The rank symbols in Tables 1 and 5 appear as corrupted characters (e.g., '♂¶edal'); please check the table rendering in the camera-ready version.

Circularity Check

1 steps flagged · score 5.0 of 10

The headline AIME-2024 gains are partially in-sample because AIME-2024 is also a dataset inside X-MAS-Bench; the out-of-sample AIME-2025 and MATH-MAS results keep the core qualitative claim independently supported.

  1. fitted input called prediction [Section 4.3 / Table 3, with Section 4.1 and Appendix C]
    "Specifically, in the competition-level AIME-2024 benchmark, AgentVerse [16] is improved from 20% to 50%, and DyLAN [11] improved from 40% to 63%. ... Our analysis covers five domains with no sample overlap compared to X-MAS-Bench. ... Our datasets cover domains including mathematics (AIME-2024 [53], AQUA-RAT [51], GSM-Hard [52], MATH [27], MMLU-Math [54], MMLU-Pro-Math [55]), ... where each dataset is randomly sampled up to 500 examples without replacement. ..."

    AIME-2024 appears both as an X-MAS-Bench dataset (Appendix C) and as the evaluation set for the headline mixed chatbot-reasoner gains (Section 4.3, Table 3). Section 4.1 defines X-MAS-Design as assigning each role the top X-MAS-Bench scorer, so the role assignments behind Table 3 are informed by the same 30 AIME-2024 problems on which the 20% to 50% and 40% to 63% gains are reported. The claimed 47% boost is therefore partly a selection artifact rather than an independent prediction. The AIME-2025 and MATH-MAS results in Table 4 are genuinely out-of-sample and support the qualitative claim.

full rationale

The paper's derivation chain is mostly self-contained: X-MAS-Bench measures per-function, per-domain scores under controlled prompts; X-MAS-Design takes the top scorer for each role; and the chatbot-only evaluations use held-out splits plus new benchmarks. This is not a fitted-parameter feedback loop into MAS outcomes, so the central idea of heterogeneous agents has independent content. However, the abstract's flagship '47% performance boost' is computed on AIME-2024, which is also one of the X-MAS-Bench datasets. Since the role assignment rule is explicitly 'top performer based on observations from X-MAS-Bench,' the AIME-2024 evaluation is partially in-sample and the Introduction's claim of 'no sample overlap' is contradicted for that benchmark. The out-of-sample AIME-2025 and MATH-MAS results in Table 4 (46.67 and 48.13 for X-MAS-Design versus 13.33/14.18 for the best homogeneous chatbot) show the qualitative conclusion is not an artifact, so the paper should not be scored as fundamentally circular. The main circularity is localized to the headline AIME-2024 numbers and the benchmark overlap, hence a moderate score of 5. Other self-citations (e.g., MAS-GPT [6], EvoMAC [14], MATH-MAS [69]) are not load-bearing for the main derivation and do not raise the score further.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No constants are fitted to data. The central claims rest on four domain assumptions: the five-function decomposition covers MAS behavior; single-run accuracy at temperature 0.5 is stable; per-role benchmark scores transfer to in-MAS performance; and held-out splits of the same benchmarks are a valid generalization test. None of these is independently verified in the paper.

assumptions (4)
  • domain assumption The five chosen functions (QA, revise, aggregation, planning, evaluation) adequately decompose the behaviors required of agents in typical MAS.
    Section 3.1 defines the functions; the X-MAS-Design relies on these being the right decomposition for the evaluated MAS methods (AgentVerse, LLM-Debate, DyLAN).
  • domain assumption Single-run accuracy at temperature 0.5 for up to 500 samples is a stable estimate of model capability for each function-domain pair.
    Section 3.2 sets temperature 0.5 and one pass; no confidence intervals or repeated runs are reported, so ranking stability is assumed. This is load-bearing for model selection in X-MAS-Design.
  • domain assumption Per-role benchmark scores transfer to the same model's effectiveness when embedded in a full multi-agent workflow.
    Section 4.1 replaces each agent role with the top-scoring model from X-MAS-Bench, assuming isolation scores predict in-MAS behavior.
  • domain assumption Held-out splits of the same benchmark families are a valid test of generalization for the selection made on X-MAS-Bench.
    Section 4.2 uses held-out MATH-500, MBPP, SciBench, PubMedQA, FinanceBench; only math has truly new benchmarks (AIME-2025, MATH-MAS).

how reviews work

0 comments
Cite this review

Pith. "Pith review of X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMs." pith.science (2026). https://pith.science/paper/KEOBPYJS

@misc{pith2026250516997,
  author       = {Pith},
  title        = {Pith review of: X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KEOBPYJS}},
  note         = {Machine review of arXiv:2505.16997}
}
read the original abstract

LLM-based multi-agent systems (MAS) extend the capabilities of single LLMs by enabling cooperation among multiple specialized agents. However, most existing MAS frameworks rely on a single LLM to drive all agents, constraining the system's intelligence to the limit of that model. This paper explores the paradigm of heterogeneous LLM-driven MAS (X-MAS), where agents are powered by diverse LLMs, elevating the system's potential to the collective intelligence of diverse LLMs. We introduce X-MAS-Bench, a comprehensive testbed designed to evaluate the performance of various LLMs across different domains and MAS-related functions. As an extensive empirical study, we assess 27 LLMs across 5 domains (encompassing 21 test sets) and 5 functions, conducting over 1.7 million evaluations to identify optimal model selections for each domain-function combination. Building on these findings, we demonstrate that transitioning from homogeneous to heterogeneous LLM-driven MAS can significantly enhance system performance without requiring structural redesign. Specifically, in a chatbot-only MAS scenario, the heterogeneous configuration yields up to 8.4\% performance improvement on the MATH dataset. In a mixed chatbot-reasoner scenario, the heterogeneous MAS could achieve a remarkable 47\% performance boost on the AIME dataset. Our results underscore the transformative potential of heterogeneous LLMs in MAS, highlighting a promising avenue for advancing scalable, collaborative AI systems.

Figures

Figures reproduced from arXiv: 2505.16997 by the authors.

Figure 1
Figure 1. Overview of our X-MAS-Bench and X-MAS-Design. X-MAS-Bench assesses the capa [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Benchmarking chatbot LLMs on 5 MAS-related functions and 5 domains. We see [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Diversity for the win. Experiments are conducted with X-MAS-Proto on three domains. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparing X-MAS with LLM selec￾tion guided by X-MAS-Bench and arbitrary selec￾tion. X-MAS-Design, which is guided by X-MAS￾Bench, significantly performs the best. Arbitrary model selection could lead to sub￾optimal performance: X-MAS-Bench offers critical observations …
Figure 5
Figure 5. Figure 5: Benchmarking LLMs on 5 MAS-related functions and 5 domains. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 2 citations worldwide. Full citation record

  1. How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs

    cs.MA 2025-07 conditional novelty 5.0 of 10

    A leader LLM trained with a GRPO variant that conditions on frozen agent responses improves both collaborative and zero-shot accuracy on BBH, MATH, and MMLU.

Reference graph

Works this paper leans on

72 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  3. [4]

    Siren’s song in the ai ocean: a survey on hallucination in large language models

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023

    arXiv 2023

  4. [5]

    Factscore: Fine-grained atomic evaluation of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, 2023

    work page 2023

  5. [6]

    MAS-GPT: Training LLMs to build LLM-based multi-agent systems

    Rui Ye, Shuo Tang, Rui Ge, Yaxin Du, Zhenfei Yin, Jing Shao, and Siheng Chen. MAS-GPT: Training LLMs to build LLM-based multi-agent systems. In Workshop on Reasoning and Planning for Large Language Models, 2025

    work page 2025

  6. [7]

    Chatdev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15174–15186, 2024

    work page 2024

  7. [8]

    Towards an ai co-scientist.arXiv preprint arXiv:2502.18864, 2025

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist.arXiv preprint arXiv:2502.18864, 2025

    arXiv 2025

  8. [9]

    Metagpt: Meta programming for a multi-agent collaborative framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024

    work page 2024

Show all 72 references
  1. [10]

    Macm: Utilizing a multi-agent system for condition mining in solving complex mathematical problems

    Bin Lei, Yi Zhang, Shan Zuo, Ali Payani, and Caiwen Ding. Macm: Utilizing a multi-agent system for condition mining in solving complex mathematical problems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  2. [11]

    A dynamic llm-powered agent network for task-oriented agent collaboration

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic llm-powered agent network for task-oriented agent collaboration. In First Conference on Language Modeling, 2024

    2024

  3. [12]

    Autonomous chemical research with large language models

    Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570–578, 2023

    2023

  4. [13]

    The ai scientist: Towards fully automated open-ended scientific discovery

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024

    2024 arXiv

  5. [14]

    Self-evolving multi-agent networks for software development

    Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Siheng Chen. Self-evolving multi-agent networks for software development. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  6. [15]

    Improving factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2024

    2024

  7. [16]

    Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representat...

    2024

  8. [17]

    Groups of diverse problem solvers can outperform groups of high-ability problem solvers

    Lu Hong and Scott E Page. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Sciences, 101(46):16385–16389, 2004

    2004

  9. [18]

    Cognitive style as environmentally sensitive individual differences in cognition: A modern synthesis and applications in education, business, and management

    Maria Kozhevnikov, Carol Evans, and Stephen M Kosslyn. Cognitive style as environmentally sensitive individual differences in cognition: A modern synthesis and applications in education, business, and management. Psychological science in the public interest, 15(1):3–33, 2014. 10

    2014

  10. [19]

    Cognitive diversity, collective intelligence, and learning in teams

    Ishani Aggarwal, Anita Williams Woolley, Christopher F Chabris, and Thomas W Malone. Cognitive diversity, collective intelligence, and learning in teams. Proceedings of Collective Intelligence, 1(3.1):3–3, 2015

    2015

  11. [20]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  12. [21]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  13. [22]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024

    2024 arXiv

  14. [23]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024

    2024

  15. [24]

    Scaling large language model-based multi-agent collaboration

    Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large language model-based multi-agent collaboration. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  16. [25]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2...

    2024

  17. [26]

    Mapcoder: Multi-agent code generation for competitive problem solving

    Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. Mapcoder: Multi-agent code generation for competitive problem solving. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4912–4944, 2024

    2024

  18. [27]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  19. [28]

    Automated design of agentic systems

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  20. [29]

    Less is more: Using multiple LLMs for applications with lower costs

    Lingjiao Chen, Matei Zaharia, and James Zou. Less is more: Using multiple LLMs for applications with lower costs. In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023

    2023

  21. [30]

    Collabstory: Multi-llm collaborative story generation and authorship analysis

    Saranya Venkatraman, Nafis Irtiza Tripto, and Dongwon Lee. Collabstory: Multi-llm collaborative story generation and authorship analysis. arXiv preprint arXiv:2406.12665, 2024

    2024 arXiv

  22. [31]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, 2023

    2023

  23. [32]

    Mixture-of-agents enhances large language model capabilities

    Junlin Wang, Jue W ANG, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  24. [33]

    Reconcile: Round-table conference improves reasoning via consensus among diverse llms

    Justin Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085, 2024

    2024

  25. [34]

    Masrouter: Learning to route llms for multi-agent systems

    Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. Masrouter: Learning to route llms for multi-agent systems. arXiv preprint arXiv:2502.11133, 2025

    2025 arXiv

  26. [35]

    Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. 11

    2024

  27. [36]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  28. [37]

    Introducing healthbench

    OpenAI. Introducing healthbench. https://openai.com/index/healthbench/, 2025. Accessed: 2025-05-15

    2025

  29. [38]

    Pixiu: a large language model, instruction data and evaluation benchmark for finance

    Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. Pixiu: a large language model, instruction data and evaluation benchmark for finance. In Proceedings of the 37th International Conference on Neural Information Processing Syst...

    2023

  30. [39]

    Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36:38975–38987, 2023

    2023

  31. [40]

    Judgebench: A benchmark for evaluating LLM-based judges

    Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica. Judgebench: A benchmark for evaluating LLM-based judges. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  32. [41]

    Mistral-7b-instruct-v0.3

    Mistral. Mistral-7b-instruct-v0.3. https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.3 , 2024. Accessed: 2025-05-15

    2024

  33. [42]

    Mistral-small-3.1-24b-instruct-2503

    Mistral. Mistral-small-3.1-24b-instruct-2503. https://huggingface.co/mistralai/ Mistral-Small-3.1-24B-Instruct-2503 , 2025. Accessed: 2025-05-15

    2025

  34. [43]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  35. [44]

    Developing chemdfm as a large language foundation model for chemistry

    Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Bo Chen, Hongshen Xu, Zichen Zhu, Su Zhu, et al. Developing chemdfm as a large language foundation model for chemistry. Cell Reports Physical Science, 6(4), 2025

    2025

  36. [45]

    Sciphi-mistral-7b-32k

    SciPhi. Sciphi-mistral-7b-32k. https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k , 2023. Accessed: 2025-05-15

    2023

  37. [46]

    Llama3-xuanyuan3-70b-chat

    Duxiaoman-DI. Llama3-xuanyuan3-70b-chat. https://huggingface.co/Duxiaoman-DI/ Llama3-XuanYuan3-70B-Chat , 2024. Accessed: 2025-05-15

    2024

  38. [47]

    Zhilu-2-8b-instruct

    SYSU-MUCFC-FinTech-Research-Center. Zhilu-2-8b-instruct. https://huggingface.co/ SYSU-MUCFC-FinTech-Research-Center/ZhiLu-2-8B-Instruct , 2024. Accessed: 2025-05-15

    2024

  39. [48]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  40. [49]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  41. [50]

    Open Thoughts

    OpenThoughts Team. Open Thoughts. https://open-thoughts.ai, January 2025

    2025

  42. [51]

    Program induction by rationale generation: Learning to solve and explain algebraic word problems

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1...

    2017

  43. [52]

    Pal: Program-aided language models

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR, 2023

    2023

  44. [53]

    Aime-2024

    Maxwell-Jia. Aime-2024. https://huggingface.co/datasets/Maxwell-Jia/AIME_2024, 2024. Accessed: 2025-05-15

    2024

  45. [54]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021

    2021

  46. [55]

    MMLU-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language unders...

    2024

  47. [56]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  48. [57]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  49. [58]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  50. [59]

    Scibench: Evaluating college-level scientific problem- solving abilities of large language models

    Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem- solving abilities of large language models. In Forty-first International Conference o...

    2024

  51. [60]

    Scieval: A multi-level large language model evaluation benchmark for scientific research

    Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. Scieval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19053–19061, 2024

    2024

  52. [61]

    Sciknoweval: Evaluating multi-level scientific knowledge of large language models

    Kehua Feng, Keyan Ding, Weijie Wang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Yu Zhao, Jianhua Yao, Qiang Zhang, and Huajun Chen. Sciknoweval: Evaluating multi-level scientific knowledge of large language models. arXiv preprint arXiv:2406.09098, 2024

    2024

  53. [62]

    Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR, 2022

    2022

  54. [63]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021

    2021

  55. [64]

    Pubmedqa: A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natur...

    2019

  56. [65]

    Fi- nancebench: A new benchmark for financial question answering

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Fi- nancebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944 , 2023

    2023 arXiv

  57. [66]

    Finqa: A dataset of numerical reasoning over financial data

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natu...

    2021

  58. [67]

    P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014

    2014

  59. [68]

    Aime2025

    OpenCompass. Aime2025. https://huggingface.co/datasets/opencompass/AIME2025, 2024. Accessed: 2025-05-15

    2024

  60. [69]

    Reso: A reward-driven self-organizing llm-based multi-agent system for reasoning tasks

    Heng Zhou, Hejia Geng, Xiangyuan Xue, Zhenfei Yin, and Lei Bai. Reso: A reward-driven self-organizing llm-based multi-agent system for reasoning tasks. arXiv preprint arXiv:2503.02390, 2025

    2025 arXiv

  61. [70]

    Codestral-22b-v0.1

    Mistral. Codestral-22b-v0.1. https://huggingface.co/mistralai/Codestral-22B-v0.1 , 2024. Accessed: 2025-05-15

    2024

  62. [71]

    Openbiollms: Advancing open-source large language models for healthcare and life sciences

    Malaikannan Sankarasubbu Ankit Pal. Openbiollms: Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B , 2024

    2024

  63. [72]

    Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine

    Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint arXiv:2308.09442, 2023. 13 A Limitations Despite being the most comprehensive evaluation of LLMs f...

    2023 arXiv

  64. [2024]

    Association for Computational Linguistics

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.