ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration

Bingbing Xu; Guo Chen; Haibo Shi; Junjie Huang; Reed Li; Yu Lu; Ziwen Li

REVIEW 5 major objections 6 minor 57 references

ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration

T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that heterogeneous execution traces of LLM multi-agent systems can be scored deterministically against a query-specific forest of verified-success collaboration graphs, yielding a reproducible, inference-free signal that…

desk verdict ForestBench is a serious benchmark contribution with a genuine reproducibility gap in its headline metric: the reference weight q_k is never defined, so Forest Match is not computable from the paper as written. read the letter →

arxiv 2608.08605 v2 pith:4OH7QF4Y submitted 2026-08-09 cs.AI

Guo Chen , Ziwen Li , Reed Li , Yu Lu , Haibo Shi , Bingbing Xu , Junjie Huang This is my paper · ORCID

classification cs.AI

keywords multi-agentsystemsLLMevaluationcollaborationgraphsreferenceforestdeterministicgraphsimilarityprocess-levelBench

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-agent systems built on large language models differ widely in how their agents communicate and cooperate, yet their execution logs are so framework-specific that no common process-level yardstick exists. This paper proposes normalizing every trace into a directed acyclic collaboration graph — nodes are atomic actions, edges are information transfers — and grading a candidate graph against a query-specific forest of verified-success graphs that record many distinct ways the task can be solved. The headline metric, Forest Match, is a deterministic weighted average of role-, edge-, and size-based similarities, so once the forest is built a trace is scored in milliseconds with no further LLM inference. On 844 queries from seven public datasets, this structural signal separates six multi-agent frameworks whose final-answer accuracies differ by only a few percentage points, and the rankings survive backbone swaps, reference-construction changes, and metric-weight perturbations. If the reference forests fairly represent success, the field gains a cheap, reusable, process-level basis for comparing and debugging agent systems.

What carries the argument

The load-bearing object is the collaboration DAG $G(\tau)=(V,E,\phi)$: one node per atomic action, one directed edge per information transfer, and each node labeled by a canonical role from a shared vocabulary, which makes the representation invariant to agent renaming and to reordering of independent actions. Around this sits the query-specific reference forest $\mathcal{F}(q)=\{(G_k,q_k)\}_{k=1}^{K}$, a precomputed set of verified-success graphs drawn from six representative MAS paradigms, each carrying a weight that blends verified correctness, structural parsimony, and token cost. Forest Match aggregates pairwise similarities $\mathrm{sim}(G,G')=\alpha s_{\mathrm{node}}+\beta s_{\mathrm{edge}}+\gamma s_{\mathrm{topo}}$ with $(\alpha,\beta,\gamma)=(0.4,0.4,0.2)$ into a weighted average. The machinery's work is to convert "how well does this method collaborate?" into deterministic arithmetic against a fixed, released asset, so evaluation never needs an LLM and never varies with rubric or judge.

What would settle it

Re-build the reference forest for the same 844 queries from successful traces collected without supplying the gold answer during execution, and compare the two forests' topology distributions (edge counts, role mixes, critique/revision edges); if the structures differ systematically, or if the framework ranking shifts materially under the unconditioned forest, Forest Match is measuring conformity to a target-conditioned generator pool rather than alignment with successful collaboration.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the collaboration process of a multi-agent system can be lifted out of framework-specific logs into a shared directed acyclic graph, and that a candidate should be judged not against one prescribed gold process but against a forest of verified-success structures. The measure that carries this claim, Forest Match, is a weighted average of pairwise similarities between the candidate graph and every reference tree, where each pairwise similarity blends a multiset Jaccard over canonicalized roles, a Jaccard over role-typed edge fingerprints, and a size-ratio term. The authors contend that this single deterministic number, computed with no additional model calls, gives a structurally grounded and reproducible basis for horizontal comparison across heterogeneous MAS traces, and that it exposes collaboration differences — and failure mechanisms — that final-answer accuracy leaves invisible.

Load-bearing premise

Everything rests on the released forest fairly representing successful collaboration for each query: the ten reference graphs per query come from the same six frameworks and the same backbone, and they are produced while the agents already know the gold answer.

Editorial extensions

If this is right

A new MAS framework can be scored on the full 844-query benchmark in minutes with zero inference tokens per trace, once the released forests are downloaded.
Process-level rankings are reproducible: the six candidate methods keep the same aggregate ordering when the reference forest is rebuilt with different backbone LLMs (rank correlation 1.00 in the tested settings).
Accuracy near-ties stop being a roadblock: Forest Match spans 0.316–0.421 while accuracies cluster between 0.531 and 0.569, so methods that look equal on final answers are separated by their collaboration structure.
Failure analysis splits into structural problems (missing roles, wrong motif, truncated pipeline) versus content problems (hallucinated facts, wrong aggregation), each implying a different repair.
Domain specialization becomes visible: a method that ranks last globally can rank first on code-style tasks, so global leaderboards should be read together with source-level analysis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the reference forest truly represents successful collaboration rather than the six generator frameworks, Forest Match could double as a development-time monitor: cheap enough to score every trace during tuning, it would flag drift away from successful collaboration shapes before outcomes regress.
A stress test the paper leaves implicit is to rebuild forests from independent implementations of the same paradigms, or from human-expert traces, and check whether cross-forest rankings agree; where they diverge, the reference construction is the main source of variance.
The role-alias tables are the least guarded step in the pipeline; a sensitivity study that perturbs alias mappings would reveal how much of the score is carried by semantic role alignment rather than by raw topology.
Because Forest Match is reference-relative, it rewards familiar collaboration shapes by construction; a genuinely novel but successful coordination pattern may score low until new reference trees are added, so the benchmark is best treated as a living asset.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper before deciding how to spend time on it. First, the central asset is real: 844 filtered queries, 8,440 precomputed target-conditioned reference graphs, adapters, and a deterministic graph-metric panel that runs in milliseconds per trace. The graph abstraction itself borrows from Gemmas, Magdi, and graph-of-thoughts, but the assembled benchmark with a query-specific reference forest is new and could be genuinely useful to the LLM multi-agent community. Second, the headline metric has a load-bearing definitional gap: Equation (5) defines Forest Match as a weighted average over references with weights w_k = q_k / max q_k', but q_k is never specified anywhere. The text says it combines verified correctness, structural parsimony, and token cost, but gives no formula, scale, or aggregation rule. That means Table 1 cannot be independently reproduced from the paper. This is not a minor presentational issue: the paper's central claim is that Forest Match is deterministic and reproducible, and a key component of the metric is undefined. The sensitivity analysis in Section 5.4 varies the sim weights but never varies q_k, so we do not even know whether the reported rankings are robust to that choice.

What the paper does well is worth emphasizing. The robustness studies are unusually thorough: leave-one-framework-out, leave-one-source-out, gold-conditioning vs open-ended construction, a 36-point TSP threshold sweep, and multiple weight schemes. The results are stable across most of these choices, which is real evidence. The paper is also honest about scope: it repeatedly says Forest Match is reference-relative, that TSP filtering does not prove a query requires multiple agents, and that the LLM failure diagnostics are not part of the benchmark score. That transparency is to its credit.

Soft spots beyond q_k: no comparison with Gemmas, the most directly relevant graph-based metric; Table 1 has no confidence intervals (only 3 seeds per query); and the role alias tables in Section 4.4 are hand-built, so cross-framework role mapping could introduce artifacts that are not tested. The phrase "collaboration-necessary queries" in the abstract overstates what the TSP filter establishes; the paper itself admits this later, but the framing is misleading.

Who is this for? Anyone building or evaluating LLM multi-agent systems, especially those who want a low-cost structural comparison tool rather than an LLM judge. It deserves a serious referee, but the review must require the q_k formula to be specified, a Gemmas comparison, and ideally error bars. I would cite it once those fixes land.

Referee Report

5 major / 6 minor

Summary. The paper introduces ForestBench, an evaluation framework that maps heterogeneous LLM-based multi-agent system (MAS) execution traces into a shared space of directed acyclic collaboration graphs, then compares candidate graphs against a query-specific reference forest of verified-success graphs. The framework defines a deterministic metric panel including the headline Forest Match score, an aggregation of pairwise graph similarities weighted by reference quality scores q_k. The authors instantiate the framework with 844 filtered queries from seven public datasets, precompute ten target-conditioned reference graphs per query using six MAS frameworks plus a chain-of-thought baseline, and evaluate six representative MAS frameworks. Extensive robustness experiments investigate sensitivity to reference backbone, reference generator, gold-answer conditioning, metric weights, query-filtering thresholds, and controlled structural perturbations, and a cost analysis shows that per-trace evaluation is orders of magnitude cheaper than LLM-as-Judge.

Significance. If the central metric is fully specified and the reference forest is accepted as a valid representation of successful collaboration, the paper would make a useful contribution: it provides a deterministic, inference-free evaluation basis for MAS comparison, releases a reusable benchmark asset of 8,440 reference graphs, and shows via controlled experiments that structural signals complement outcome accuracy. The robustness studies are unusually extensive, and the leave-one-out and held-out-generator ablations directly address the most obvious circularity concern. However, the headline Forest Match score is not currently computable from the paper alone because the reference weight q_k is never defined, and several validation gaps (missing confidence intervals, surprisingly small perturbation effects, unvalidated role alias tables) weaken the reproducibility and validity claims. The core idea is sound and the issues appear fixable, so the appropriate path is major revision rather than rejection.

major comments (5)

[Section 3.3, Eq. (5); Section 4.3] The reference weight q_k is never defined. Eq. (5) defines Forest Match as a weighted average with w_k = q_k / max_k' q_k', and Section 4.3 states only that q_k combines 'verified correctness, structural parsimony, and token cost,' but no formula, scale, or aggregation rule is given anywhere in the paper or appendix. Since Forest Match is the headline metric and the paper's central claim is that it is deterministic and reproducible, Table 1 cannot be independently recomputed or audited. The sensitivity analysis in Section 5.4 varies only alpha, beta, gamma in Eq. (2); it does not vary the q_k weighting, so the stability of the reported rankings with respect to the unspecified reference weights is unknown. Please specify q_k explicitly or show that the rankings are robust across a plausible family of q_k definitions.
[Table 1, Section 5.2] The main comparative results are reported as point estimates with no confidence intervals or significance tests. Each framework is run with three seeds per query, yielding roughly 15,192 traces, so sampling variability can be quantified, but Table 1 reports only means. The accuracy range is only 0.038, and several Forest Match differences (e.g., Swarm 0.350 vs. AFlow 0.347) are smaller than the reported perturbation effect of 0.058 in Section 5.4. Without bootstrap intervals or paired tests, the reader cannot tell whether the horizontal ranking is statistically meaningful. Please add confidence intervals or significance tests for the Table 1 metrics, or explicitly state why these are not applicable.
[Section 5.4, Appendix Table 5] The perturbation experiment reports that deleting half the nodes (p=0.5) reduces Forest Match by only 0.058, which is surprisingly small for a structural metric that is supposed to measure alignment of collaboration structure. Since node deletion removes half the actions and many edges, one would expect a substantially larger drop; a 0.058 change is comparable to the smallest inter-framework differences in Table 1. The paper does not report the behavior of the individual components s_node, s_edge, and s_topo under this perturbation, so it is unclear whether Forest Match is insensitive to severe structural damage or whether the effect is diluted by aggregation across the forest. Please report the per-component decomposition and provide a fuller damage-response curve (e.g., p from 0.1 to 0.9) to demonstrate that the metric responds meaningfully to structural degradation.
[Section 4.4] The per-framework alias tables that map raw agent names into the shared role vocabulary R are described as a single conversion routine, but no construction procedure, release location, or validation is described. Because s_node and s_edge in Eqs. (2)-(3) are computed on canonicalized roles, any mis-mapping of semantically distinct roles into the same canonical role (or vice versa) directly changes all reported Forest Match and Best Similarity values. The paper should specify how the alias tables were built, release them with the benchmark, and ideally report sensitivity to alias-table perturbations (e.g., randomly switching a fraction of role mappings) to show that the main rankings do not hinge on hand-built role equivalences.
[Section 4.3, Section 6] The reference forest is generated from the same six MAS frameworks and the same DeepSeek-V4-Flash backbone used in candidate evaluation, with generation conditioned on the gold answer. The leave-one-out and open-ended ablations in Section 5.3 mitigate the same-generator and same-conditioning concerns, but the forest still represents only the successful structures expressible by these six generators under one backbone. A genuinely novel MAS whose collaboration pattern is outside this generator pool will be scored as poorly aligned by construction, which limits the 'generalizable' claim in the title and abstract. Please state this scope explicitly as a limitation, or add a validation with at least one out-of-pool reference generator to show the framework can accommodate unseen collaboration paradigms.

minor comments (6)

[Section 4.2] The Trace-Suitability Pipeline relies on scores from 'a strong filtering LLM' for depth, width, and decomposability, but no agreement measure, human validation, or re-labeling consistency check is reported. The threshold sweep in Appendix A shows ranking stability with respect to thresholds, but it does not validate the labels themselves. Please add at least a small human-annotation or judge-agreement study for the TSP scores.
[Section 3.3, Eq. (7)] Mean Uptake depends on an embedding model and a threshold theta_u that is said to be 'fixed across all methods,' but the embedding model, the value of theta_u, and the normalization procedure are not specified. Please provide these details or a reference to the released code.
[Section 3.3, auxiliary metrics] Efficiency is defined only as 'graph parallelism per active agent'; no formula is given. Please define it precisely so that Table 1 values are reproducible.
[Appendix D, Table 8] The diagnostic robustness results show low pairwise Cohen's kappa for structural failure labels (0.129-0.351). The paper appropriately limits claims to coarse repair directions, but the main-text statement that 'coarse diagnostic directions are stable across judges' (Section 5.5) should explicitly cite these kappa values to avoid overstating the stability of even coarse labels.
[General presentation] The paper still contains ACM template placeholder artifacts (e.g., 'Conference’17, July 2017, Washington, DC, USA' in the header and the placeholder DOI). These should be updated before publication.
[Section 5.1] The leave-one-out processing is described only in a single sentence. Please clarify whether the Table 1 results use LOO references for all six frameworks, and whether the same LOO procedure is applied in the robustness experiments reported in Section 5.3 and Appendix E.

Circularity Check

0 steps flagged · score 0.0 of 10

No definitional circularity found; Forest Match is an explicitly reference-relative metric whose reference construction is disclosed and tested by leave-one-out and held-out-generator ablations.

full rationale

The central metric, FM in Eq. (5), is defined directly as a weighted average of pairwise graph similarities against a declared query-specific reference forest; it is not used to predict an external outcome and no parameter is fitted to the reported rankings. The reference forest is built from six MAS frameworks and a single-agent anchor, but Section 5.1 applies leave-one-out when candidate and reference share a framework source, and Section 5.3 removes whole generators and re-ranks, with Top-3 preserved in 6/7 settings; hence the headline ordering is not forced by comparing a candidate with its own generated references. The paper also explicitly scopes the metric as reference-relative alignment ('how does a candidate's observed collaboration structure relate to the benchmark's set of verified-success structures'), which is a transparent operational definition rather than a disguised prediction. The sensitivity studies vary the weights of Eq. (2), the TSP thresholds, and the reference backbone, all external to the definition of FM. No load-bearing step depends on a self-citation or an imported uniqueness theorem. One genuine specification gap exists: the reference score q_k in Eq. (5) is described only as 'combining verified correctness, structural parsimony, and token cost' with no formula or scale given, so Eq. (5) is not fully reproducible from the text; this is an incomplete-definition issue, not a circular reduction, and it does not raise the circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The framework rests on hand-chosen design parameters and modeling assumptions about trace-to-graph fidelity and role semantics. No new physical or ontological entities are postulated; the reference forest and collaboration graph are methodological constructs without independent falsifiable handles. The most consequential choices are the target-conditioned generation of reference graphs and the unspecified q_k weighting.

free parameters (5)

SIM weights (alpha, beta, gamma) = 0.4, 0.4, 0.2
Default weights in Eq. (2) for node, edge, and topology similarity. Hand-chosen rather than fitted; sensitivity to alternative weights is reported in Section 5.4.
TSP thresholds (theta_d, theta_w, theta_delta) = 5, 0.5, 0.5
Hand-set thresholds in Eq. (9) that determine which queries enter the 844-query pool. The paper sweeps 36 combinations and reports stable rankings, but the benchmark scope depends on this choice.
Reference count K per query = 10
Section 4.3 retains ten verified-success reference graphs per query. The effect of K on the stability of Forest Match is not examined.
Uptake threshold theta_u = not reported
Mean Uptake in Eq. (7) uses a cosine threshold theta_u that is stated to be fixed across methods, but the numeric value is omitted.
Reference weight q_k = not specified
Section 3.3 says each reference graph has a score q_k combining verified correctness, structural parsimony, and token cost, but the functional form is never given, so FM as published cannot be independently recomputed.

assumptions (4)

domain assumption MAS traces can be projected faithfully to a DAG of atomic actions and information-transfer edges.
Section 3.1 defines G(tau) in this way. If the adapter misses implicit context reuse or cross-agent references, the graph and all downstream metrics measure the wrong structure.
ad hoc to paper The per-framework alias tables correctly map semantically identical roles into a shared vocabulary R.
Section 4.4. Node and edge similarities in Eq. (3) depend entirely on this mapping; an incorrect alias table biases s_node, s_edge, and therefore Forest Match.
domain assumption Reference traces conditioned on the gold answer are representatives of successful collaboration.
Section 4.3 constructs references by providing a* as a target. This guarantees outcome correctness but does not guarantee that the trace resembles a natural successful process.
ad hoc to paper The Trace-Suitability Pipeline identifies collaboration-necessary queries.
Section 4.2 admits the filter is heuristic and does not prove multi-agent necessity; the 844-query benchmark scope rests on this premise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration." pith.science (2026). https://pith.science/paper/4OH7QF4Y

@misc{pith2026260808605,
  author       = {Pith},
  title        = {Pith review of: ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4OH7QF4Y}},
  note         = {Machine review of arXiv:2608.08605}
}

abstract

Multi-agent systems (MAS) built on Large Language Models (LLMs) are proliferating rapidly, but their heterogeneous execution traces provide no common basis for evaluation across methods. Outcome-only benchmarks discard collaborations, whereas LLM-as-Judge evaluation requires additional, model-dependent inference and can vary with the LLM and rubric. We introduce a generalizable evaluation framework that maps native MAS traces into a shared space of unified collaboration graphs, enabling different methods to be evaluated under the same representation, reference set, and metric panel. Candidate graphs are compared with a query-specific reference forest. Each forest is a benchmark-provided collection of verified-success graphs: it records diverse ways in which representative MAS methods can complete the task, rather than prescribing a unique optimal process. Instantiating the framework as ForestBench, we filter $844$ collaboration-necessary queries from seven public datasets, precompute ten successful target-conditioned reference graphs per query, and evaluate six representative MAS frameworks. Controlled backbone, reference-construction, and perturbation studies test the stability and scope of evaluation. Once the benchmark forests are built, ForestBench scores a trace in milliseconds without further LLM inference, providing a reusable structural basis for comparing diverse MAS collaboration traces.

Figures

Figures reproduced from arXiv: 2608.08605 by the authors.

**Figure 2.** Overview of ForestBench framework. function of 𝐺(𝜏) alone, the evaluation needs no further LLM calls and runs in milliseconds per trace. Pairwise graph similarity. Given a candidate graph 𝐺 and the reference graph 𝐺 ′ representing one tree in the forest, we measure their pairwise similarity as sim(𝐺,𝐺′ ) = 𝛼 𝑠node + 𝛽 𝑠edge +𝛾 𝑠topo, (2) with default weights (𝛼, 𝛽,𝛾) = (0.4, 0.4, 0.2). Let 𝑅(𝐺) be the multiset of c… view at source ↗

**Figure 3.** Per-source composition of ForestBench after Trace-Suitability filtering. which contains 844 queries distributed across reasoning (55.9%), multi-hop QA (23.7%), code generation (15.8%), and tool use (4.6%). The per-source composition of Q★ is shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Failure-mode diagnostics across the six candidate frameworks on failed traces. Structural-OK denotes traces whose [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Online evaluation cost of ForestBench versus LLMas-Judge for 1,000 traces. ForestBench is ≈ 2,900× faster and uses zero inference tokens per 1k traces. LLM–as–judge remain rubric-sensitive; the complete prompts and sensitivity results are reported in Appendix D. As a …

**Figure 6.** Figure 6: System-prompt variants for the prompt-variant robustness check. The full experiment runs the same DeepSeek judge [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 22 canonical work pages

[1]

Anthropic. 2026. Introducing Sonnet 4.6. https://www.anthropic.com/news/ claude-sonnet-4-6

work page 2026
[2]

Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Guangyuan Piao, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jurgen Müller, et al. 2025. Demystifying chains, trees, and graphs of thoughts.IEEE Transactions on Pattern Analysis and Machine Intelligence(2025)

work page 2025
[3]

Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ram- chandran, et al. 2026. Why do multi-agent llm systems fail?Advances in Neural Information Processing Systems38 (2026)

work page 2026
[4]

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024. Chateval: Towards better llm-based evalu- ators through multi-agent debate. InInternational conference on learning repre- sentations, Vol. 2024. 9079–9093

work page 2024
[5]

Justin Chih-Yao Chen, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal

work page
[6]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

arXiv 2021
[7]

Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. 2024. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. InInternational Conference on Learning Representations, Vol. 2024. 20094–20136

work page 2024
[8]

Yanjun Chen, Yirong Sun, Hanlin Wang, Xinming Zhang, Xiaoyu Shen, Wenjie Li, and Wei Zhang. 2026. Contextual Counterfactual Credit Assignment for Multi-Agent Reinforcement Learning in LLM Collaboration.arXiv preprint arXiv:2603.06859(2026)

arXiv 2026

Show all 57 references

[9]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168(2021)

2021 arXiv
[10]

DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Con- text Intelligence

2026
[11]

Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2024. Self-collaboration code genera- tion via chatgpt.ACM Transactions on Software Engineering and Methodology33, 7 (2024), 1–38

2024
[12]

Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch
[13]

Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges.arXiv preprint arXiv:2402.01680 (2024)

2024 arXiv
[14]

Shanshan Han, Qifan Zhang, Weizhao Jin, and Zhaozhuo Xu. 2024. LLM multi- agent systems: Challenges and open problems.arXiv preprint arXiv:2402.03578 (2024)

2024 arXiv
[15]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing.arXiv preprint arXiv:2009.03300(2020)

2020 arXiv
[16]

Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Steven Yau, Zijuan Lin, Liyang Zhou, et al . 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations, Vol....

2024
[17]

Peizhou Huang, Zixuan Zhong, Zhongwei Wan, Donghao Zhou, Samiul Alam, Xin Wang, Zexin Li, Zhihao Dou, Li Zhu, Jing Xiong, et al. 2026. MMDeepResearch- Bench: A Benchmark for Multimodal Deep Research Agents.arXiv preprint arXiv:2601.12346(2026)

2026
[18]

Yoichi Ishibashi and Yoshimasa Nishimura. 2024. Self-organized agents: A llm multi-agent framework toward ultra large-scale code generation and optimization. arXiv preprint arXiv:2404.02183(2024)

2024 arXiv
[19]

Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers.arXiv preprint arXiv:2402.06782(2024)

2024 arXiv
[20]

Jisoo Lee, Raeyoung Chang, Dongwook Kwon, Harmanpreet Singh, and Nikhil Verma. 2025. Gemmas: Graph-based evaluation metrics for multi agent systems. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. 1522–1532

2025
[21]

Boyi Li, Zhonghan Zhao, Der-Horng Lee, and Gaoang Wang. 2025. Adaptive graph pruning for multi-agent communication.arXiv preprint arXiv:2506.02951 (2025)

2025 arXiv
[22]

Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. 2024. A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges.Vicinagearth1, 1 (2024), 9

2024
[23]

Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene Ie. 2024. Improving multi-agent debate with sparse communication topology. InFindings of the Association for Computational Linguistics: EMNLP

2024
[24]

Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging divergent thinking in large language models through multi-agent debate. InProceedings of the 2024 conference on empirical methods in natural languag...

2024
[25]

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2024. Agentbench: Evaluating llms as agents. InInternational Conference on Learning Representations, Vol. 2024. 52989–53046

2024
[26]

Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhen- zhong Lan, Lingpeng Kong, and Junxian He. 2024. Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in neural information pro- cessing systems37 (2024), 74325–74362

2024
[27]

Panagiotis Michelakis, Yiannis Hadjiyiannis, and Dimitrios Stamoulis. 2025. CORE: Full-Path Evaluation of LLM Agents Beyond Final State.arXiv preprint arXiv:2509.20998(2025)

2025
[28]

OpenAI. 2024. Swarm: An Educational Framework Exploring Ergonomic, Light- weight Multi-Agent Orchestration. https://github.com/openai/swarm. GitHub Conference’17, July 2017, Washington, DC, USA Guo Chen, Ziwen Li, Reed Li, Yu Lu, Haibo Shi, Bingbing Xu, and Junjie Huang repository

2024
[29]

OpenAI. 2026. Introducing GPT-5.5. https://openai.com/index/introducing-gpt- 5-5/

2026
[30]

Tianyue Ou, Wanyao Guo, Apurva Gandhi, Graham Neubig, and Xiang Yue. 2025. AgentDiagnose: An Open Toolkit for Diagnosing LLM Agent Trajectories. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 207–215

2025
[31]

Bo Pan, Jiaying Lu, Ke Wang, Li Zheng, Zhen Wen, Yingchaojie Feng, Minfeng Zhu, and Wei Chen. 2025. Agentcoord: Visually exploring coordination strategy for llm-based multi-agent collaboration.Computers & graphics(2025), 104338

2025
[32]

Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. 2024. Chatdev: Communicative agents for software development. InProceedings of the 62nd annual meeting of the association for computational linguistics (vol...

2024
[33]

Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, et al. 2025. Scaling large language model-based multi-agent collaboration. InInternational Conference on Learning Representations, Vol. 2025. 41488–41505

2025
[34]

Qwen Team. 2026. Qwen3.6-Plus: Towards Real World Agents. https://qwen.ai/ blog?id=qwen3.6

2026
[35]

Xu Shen, Yixin Liu, Yiwei Dai, Yili Wang, Rui Miao, Yue Tan, Shirui Pan, and Xin Wang. 2025. Understanding the information propagation effects of communi- cation topologies in llm-based multi-agent systems. InProceedings of the 2025 Conference on Empirical Methods in Natural L...

2025
[36]

Andries Smit, Paul Duckworth, Nathan Grinsztajn, Thomas D Barrett, and Arnu Pretorius. 2023. Should we be going mad? a look at multi-agent debate strategies for llms.arXiv preprint arXiv:2311.17371(2023)

2023 arXiv
[37]

Yashar Talebirad and Amirhossein Nadiri. 2023. Multi-agent collaboration: Har- nessing the power of intelligent llm agents.arXiv preprint arXiv:2306.03314 (2023)

2023 arXiv
[38]

Jiaxuan Wang, Yulan Hu, Wenjin Yang, Zheng Pan, Xin Li, and Lan-Zhe Guo
[39]

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171(2022)

2022 arXiv
[40]

Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Pei Chen, Ziwei Dong, Jing Huang, Jiri Gesi, Xianfeng Tang, Chen Luo, Qun Liu, et al. 2026. Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents.arXiv preprint arXiv:2601.20144(2026)

2026 arXiv
[41]

Zhexuan Wang, Yutong Wang, Xuebo Liu, Liang Ding, Miao Zhang, Jie Liu, and Min Zhang. 2025. Agentdropout: Dynamic agent elimination for token-efficient and high-performance llm-based multi-agent collaboration. InProceedings of the 63rd Annual Meeting of the Association for Com...

2025
[42]

Ziting Wang, Shize Zhang, Haitao Yuan, Jinwei Zhu, Shifu Li, Wei Dong, and Gao Cong. 2025. FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data.arXiv preprint arXiv:2509.02473(2025)

2025 arXiv
[43]

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. InFirst conference on language modeling

2024
[44]

Yue Wu, Xuan Tang, Tom Mitchell, and Yuanzhi Li. 2024. Smartplay: A benchmark for llms as intelligent agents. InInternational Conference on Learning Representa- tions, Vol. 2024. 1543–1561

2024
[45]

Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See Kiong Ng, and Jiashi Feng. 2024. Magic: Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collaboration. In Proceedings of the 2024 Conference on Empir...

2024
[46]

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems36 (2023), 11809–11822

2023
[47]

Rui Ye, Shuo Tang, Rui Ge, Yaxin Du, Zhenfei Yin, Siheng Chen, and Jing Shao
[48]

Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2025. Survey on evaluation of llm-based agents.arXiv preprint arXiv:2503.16416(2025)

2025 arXiv
[49]

Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. 2024. G-designer: Archi- tecting multi-agent communication topologies via graph neural networks.arXiv preprint arXiv:2410.11782(2024)

2024 arXiv
[50]

Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. 2025. Aflow: Automating agentic workflow generation. InInternational Conference on Learning Representations, Vol. 2025. 34040–34077

2025
[51]

Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ö Arık. 2024. Chain of agents: Large language models collaborating on long- context tasks.Advances in Neural Information Processing Systems37 (2024), 132208–132237

2024
[52]

Heng Zhou, Hejia Geng, Xiangyuan Xue, Li Kang, Yiran Qin, Zhiyong Wang, Zhenfei Yin, and Lei Bai. 2025. Reso: A reward-driven self-organizing llm-based multi-agent system for reasoning tasks. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Process...

2025
[53]

Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. 2024. Gptswarm: Language agents as optimizable graphs. InForty-first International Conference on Machine Learning. ForestBench: A Unified Graph Framework for Evaluating Mult...

2024
[2023]

Improving factuality and reasoning in language models through multiagent debate.arXiv preprint arXiv:2305.14325(2023)

2023 arXiv
[2024]

Magdi: Structured distillation of multi-agent interaction graphs improves reasoning in smaller language models.arXiv preprint arXiv:2402.01620(2024)

2024 arXiv
[2025]

Mas-gpt: Training llms to build llm-based multi-agent systems.arXiv preprint arXiv:2503.03686(2025)

2025 arXiv
[2026]

Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling.arXiv preprint arXiv:2604.08178(2026)

2026 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Anthropic. 2026. Introducing Sonnet 4.6. https://www.anthropic.com/news/ claude-sonnet-4-6

work page 2026

[2] [2]

Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Guangyuan Piao, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jurgen Müller, et al. 2025. Demystifying chains, trees, and graphs of thoughts.IEEE Transactions on Pattern Analysis and Machine Intelligence(2025)

work page 2025

[3] [3]

Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ram- chandran, et al. 2026. Why do multi-agent llm systems fail?Advances in Neural Information Processing Systems38 (2026)

work page 2026

[4] [4]

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024. Chateval: Towards better llm-based evalu- ators through multi-agent debate. InInternational conference on learning repre- sentations, Vol. 2024. 9079–9093

work page 2024

[5] [5]

Justin Chih-Yao Chen, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal

work page

[6] [6]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

arXiv 2021

[7] [7]

Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. 2024. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. InInternational Conference on Learning Representations, Vol. 2024. 20094–20136

work page 2024

[8] [8]

Yanjun Chen, Yirong Sun, Hanlin Wang, Xinming Zhang, Xiaoyu Shen, Wenjie Li, and Wei Zhang. 2026. Contextual Counterfactual Credit Assignment for Multi-Agent Reinforcement Learning in LLM Collaboration.arXiv preprint arXiv:2603.06859(2026)

arXiv 2026