Pith. sign in

REVIEW 5 major objections 5 minor 66 references

Co-Saving: Resource Aware Multi-Agent Collaboration for Software Development

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Co-Saving claims that injecting shortcuts mined from past successful software-development trajectories into a multi-agent LLM system cuts token usage by 50.85% on average while improving composite code quality by 10.06%.

desk verdict A clever shortcut mechanism for cheaper multi-agent coding, but the experimental reporting is too inconsistent to support the headline numbers. read the letter →

arxiv 2505.21898 v1 pith:DEMQAZ3C submitted 2025-05-28 cs.CL cs.AIcs.MAcs.SE

Rennai Qiu , Chen Qian , Ran Li , Yufan Dang , Weize Chen , Cheng Yang , Yingli Zhang , Ye Tian
show 4 more authors
Xuantang Xiong Lei Han Zhiyuan Liu Maosong Sun
This is my paper
classification cs.CLcs.AIcs.MAcs.SE
keywords multi-agentsystemsLLMagentstokenefficiencyresource-awarecollaborationsoftwaredevelopmentshortcutsChatDevbudgetedcompletionrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that multi-agent LLM systems can be made resource-aware by reusing successful reasoning paths from past tasks. It introduces shortcuts, instructional transitions that let an agent jump directly from one solution state to a later state, skipping intermediate reasoning agents. On software development tasks from SRDD, Co-Saving reports a 50.85% average token reduction and a 10.06% quality gain over ChatDev. This matters because the dominant cost of multi-agent systems is exactly the token-heavy dialogue that such shortcuts prune.

What carries the argument

The central object is the shortcut: a stored instructional transition $(n_i, n_j)$ that connects two non-adjacent nodes of a previously successful execution graph, letting the current task jump from solution state $n_i$ to $n_j$ without passing through the intermediate agents. Shortcuts are filtered by a composite score: node value $w(n_j) = \mathrm{sim}(n_j, \mathrm{task}) \times \mathrm{sim}(n_j, s_{|N|}) \times \mathbb{1}[\mathrm{compiles}]$, shortcut value $v(n_i, n_j) = w(n_j) - w(n_i)$, cost $C$ as the harmonic mean of percentile-ranked time and token usage, and an emergency factor $\gamma$ as the harmonic mean of consumed-to-budget ratios for time and tokens. The surviving shortcuts are fed as instructions to the reviewing agent, and the task is forcibly terminated once the current interaction graph reaches the reference graph's edge count. This termination rule, together with the shortcuts, is the mechanism that converts fewer reasoning steps into lower token consumption.

What would settle it

Run Co-Saving with an oracle retriever that always returns a same-category reference task versus a distractor reference from a different category; if quality and budgeted completion rate collapse in the distractor condition, the reference-proxy assumption is the active mechanism. Separately, ablate the forced-termination rule: if token savings persist without it, the shortcuts themselves drive the gain; if they vanish, early stopping is the real cost saver.

Watch

Extended reading notes

Core claim

The paper's central claim is that a multi-agent software-development system can compress its own reasoning chain by reusing successful trajectories from past tasks, and that this compression saves tokens without sacrificing output quality. Each task execution is represented as a directed graph whose nodes are solution states and whose edges are instructions; a shortcut is an edge that skips intermediate nodes. Given a new task, Co-Saving retrieves a similar reference task, scores candidate shortcuts by value and cost, and applies the best ones while an emergency factor shifts priority from value to cost as the budget drains. On the SRDD benchmark, the paper reports a 50.85% average reduction in token usage and a 10.06% improvement in overall code quality over ChatDev, with the budgeted completion rate rising from 0.016 to 0.728.

Load-bearing premise

The load-bearing premise is that the retrieved reference task's execution graph, including its final solution and its number of reasoning steps, is a reliable proxy for what the current task needs; if the reference is not truly similar, retrieved shortcuts will steer agents toward the wrong goal and the forced termination can stop the current task too early.

Editorial extensions

If this is right

  • Multi-agent LLM systems can use historical execution graphs as memory, so token cost can scale with task novelty rather than with a fixed number of dialogue phases.
  • Resource-aware control with forced termination trades a modest completeness drop (0.816 vs. 0.904 for ChatDev) for a large increase in budgeted completion rate (0.728 vs. 0.016).
  • The method depends on reference-task retrieval: shortcuts only help when the retrieved task shares the current task's goal and required reasoning depth.
  • The emergency factor lets the system dynamically re-weight solution value against remaining time and tokens as the budget drains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported token savings may be driven largely by the forced-termination rule rather than by shortcut content; an ablation that disables termination while keeping shortcut filtering would isolate the two effects.
  • Completeness actually falls relative to ChatDev, so the 10.06% 'quality' improvement is a property of the product-of-four-metrics composite, not a claim that every code-quality dimension improves.
  • If retrieval quality improves with more historical data, Co-Saving's savings should grow as the repository expands, but the same growth increases the risk that a poorly matched reference will mislead the current task.
  • The value function inherits the embedding model's blind spots, so tasks where code-text similarity is hard to measure may receive systematically wrong shortcut scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Co-Saving, a resource-aware multi-agent system for software development that mines 'shortcuts' from historically successful execution graphs, scores candidate shortcuts via a value-cost tradeoff, and forcibly terminates execution when the current interaction graph reaches the reference task's edge count. The authors claim an average 50.85% token-usage reduction and a 10.06% quality improvement over ChatDev on a subset of the SRDD dataset.

Significance. The problem addressed—token- and time-inefficiency of multi-agent LLM systems—is real and practically important, and the graph-based formalization of interaction trajectories is a sensible starting point. If the claims were supported, a resource-aware MAS that roughly halves token use without sacrificing quality would be a useful contribution. However, the current evidence does not support the central claims: the main results are internally inconsistent, key quantities are unverifiable, and the shortcut-scoring mechanism appears to optimize directly on the evaluation metric and to rely on an unvalidated reference-solution proxy. The paper's idea is worth further study, but the reported validation is not reliable.

major comments (5)
  1. [Section 3.1, Table 1 vs. Section 3.2, Table 2] The full Co-Saving system reports substantially different metrics in the two tables: Quality is 0.2515 in Table 1 but 0.5453 in Table 2; Completeness is 0.816 vs. 0.925; Executability is 0.688 vs. 0.85; BCR is 0.728 vs. 0.800. The paper gives no explanation for these discrepancies (e.g., different test subsets, different runs, or a typo). Since the abstract's 10.06% quality improvement is computed from one of these numbers (0.2515 - 0.1510 = 0.1005), this inconsistency makes the headline claim unverifiable and undermines the ablations, which use Table 2 as the reference point.
  2. [Section 2.2, Eq. (4) and Implementation Details] The node score w(n_j) = sim(n_j, task) * sim(n_j, s_|N|) * [[s_j]] uses s_|N|, defined as 'the solution at the final node in the graph, representing the ultimate goal.' At shortcut-evaluation time during a current task, no final solution exists; the only implementable reading is that s_|N| is the retrieved reference task's final solution. Shortcut selection therefore measures progress toward the reference task's answer, not toward the current task's requirement. The forced-termination rule—terminating once the current graph reaches the reference graph's edge count—then makes token savings at least partly by construction. Table 1's Completeness drop (0.816 vs. ChatDev's 0.904) is consistent with premature termination on less-similar tasks. No experiment removes or varies the sim(n_j, s_|N|) term, so the paper cannot distinguish genuine transfer from reference-solution leakage.
  3. [Section 3, Metrics vs. Section 2.2, Eq. (4)] The Consistency metric used in the co-learning evaluation is defined as the cosine similarity between the generated code's embedding and the natural-language requirement's embedding. The first factor in Eq. (4), sim(n_j, task), is the same cosine similarity between the candidate node's code and the task requirement. Because Quality is the product of Completeness, Executability, Consistency, and Granularity, Co-Saving's shortcut filter directly optimizes one component of its own evaluation metric. The reported 10.06% quality improvement may therefore partly reflect optimization on the evaluation criterion rather than a genuine improvement in code quality, and an independent or held-out evaluation criterion is needed to rule out circularity.
  4. [Section 3.3 and Abstract] The paper claims an 'average reduction of 50.85% in token usage' but never reports raw token counts, per-task token distributions, or the exact resource budgets used to compute BCR. Figure 3 shows distributions only, without numeric values. Because the forced-termination rule caps the number of edges in the interaction graph, the 50.85% figure is not independently verifiable and may be determined by the stopping policy rather than by the quality of shortcut selection. The paper should report raw token counts and preferably confidence intervals or significance tests.
  5. [Section 2.2, Eqs. (6)-(8)] The composite cost is written as C = F_γ(α, β) = 2αβ/(α+β), but the right-hand side is independent of γ; the 'emergency factor' γ is defined only later in Eq. (8), and no equation shows how γ modifies value, cost, or the final shortcut-selection decision. The ablation that removes γ therefore tests a mechanism that is never formally specified. In addition, the notation is inconsistent: the text preceding Eq. (6) uses T for both the set of normalized time values and the set of normalized token values, and the symbols F_1 and F_γ are introduced without definition.
minor comments (5)
  1. [Section 2.1] The paragraph beginning 'Here, N denotes the set of nodes...' is duplicated verbatim in the text, which should be removed.
  2. [Section 3, Baselines and Table 1] The sentence 'LLaMA 3 70B and are widely adopted...' is ungrammatical, and Table 1 contains formatting issues such as 'GPT-40.9800' missing a space.
  3. [Section 3, Datasets] The paper states that 'a subset of the SRDD' is used but does not report the subset size, the category distribution, or the train/test split; this basic information is needed for reproducibility.
  4. [Section 3.4, Case Study] The case study illustrates the shortcut mechanism but provides no quantitative evidence; adding token counts or path lengths for the specific example would make it more informative.
  5. [General] The footnote marker for 'Co-Saving' in Section 3.1 appears as 'Co-Saving 1' with a footnote separated from the main text; this styling should be cleaned up.

Circularity Check

2 steps flagged · score 6.0 of 10

Eq. 4's shortcut score reuses the same embedding cosine similarity that defines the Consistency component of the reported Quality metric, and the edge-count termination cap makes the token savings structural, so the headline gains are partly by construction.

  1. self definitional [Section 2.2, Eq. (4); Section 3, Metrics; Implementation Details]
    "w(nj) = sim(nj, task) × sim(nj, s|N|) × [[sj]] (4) ... The two similarity terms are computed as the cosine similarity between the embedding vectors of the corresponding texts or code. ... Consistency: Evaluates the semantic alignment between the generated code and the original natural language requirement, computed as the cosine similarity between their respective embedding vectors. ... Quality: ... defined as the product of these four metrics. ... For node evaluation, metric consistency computation, and reference task retrieval, we employ text-embedding-ada-002 as the semantic embedder."

    The shortcut score in Eq. 4 contains sim(nj, task), which is exactly the Consistency metric: cosine similarity between generated code and the natural-language requirement, computed with the same embedder. Quality is defined as the product of Completeness, Executability, Consistency, and Granularity, so Co-Saving filters shortcuts by directly rewarding increases in a quantity that later appears multiplicatively in its own evaluation metric. The same embedder is explicitly reused for node evaluation and consistency computation, making the 10.06% quality improvement at least partly a self-fulfilling optimization of the metric. The compilability indicator [[sj]] similarly targets the Executability component, further coupling the selection rule to the reported quality score.

  2. other [Implementation Details; Section 3.1]
    "To prevent excessive interactions, once the current interaction graph reaches or exceeds the number of edges in the reference task graph, we forcibly terminate the task."

    The reported 50.85% token reduction is a structural consequence of this cap, not an emergent efficiency gain measured independently. Co-Saving bounds the interaction graph by the retrieved reference graph's edge count, while ChatDev continues through its full phase pipeline, so Co-Saving cannot produce longer trajectories and its token count is capped by construction. The paper's own explanation of the completeness drop — 'Co-Saving may opt to terminate reasoning prematurely, prioritizing efficiency over completeness' — confirms that the termination rule, not shortcut quality, is the operative mechanism behind the savings. Thus the headline efficiency result reduces to an input constraint rather than a validated discovery.

full rationale

The paper is an empirical system paper and much of its mechanism is honestly specified. The central concern is that its two headline numbers are not independent of the method's construction. The shortcut value in Eq. 4 reuses the exact embedding-cosine function that defines the Consistency component of the custom Quality metric, and the compile gate [[sj]] targets Executability; the reported 10.06% quality gain is therefore in part the optimizer reporting on its own objective, not an external validation. The 50.85% token reduction is also structurally forced by the edge-count termination cap, which the paper itself connects to the completeness drop (0.816 vs 0.904) in Section 3.1. In addition, the second similarity factor sim(nj, s|N|) is an unvalidated proxy: at shortcut-evaluation time the current graph has no final node, and the case study shows the evaluated nodes (n0,n1), (n0,n2), (n0,n3) belong to the retrieved reference task, so the quality transfer rests on the assumption that the reference final solution matches the current goal. No ablation removes this term. These couplings make the central efficiency/quality claims partially circular, although the framework could be fairly re-evaluated with an independent quality metric and ablations that remove the metric-aligned and reference-final-solution terms.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four unverified premises: the training trajectories are genuinely successful, the reference task's final solution is a valid proxy for the current task's goal, embedding similarity reflects code quality, and the co-learning metric is a valid quality measure. The paper provides no external evidence for these premises beyond a single case study, and the first and fourth premises are tied to the authors' own prior work.

assumptions (4)
  • domain assumption Training trajectories identified as successful are genuinely good solutions and representative of the task distribution.
    Section 2.1 extracts shortcuts from 'historically successful trajectories' without defining the success criterion or trajectory selection, so the validity of the experience base is assumed.
  • ad hoc to paper The final solution of the retrieved reference task is a valid proxy for the goal of the current task.
    Eq. 4 defines w(n_j) using sim(n_j, s_|N|), where s_|N| is the final solution of the reference graph; the paper assumes this proxy is useful without validating task similarity beyond a single case study.
  • domain assumption Embedding cosine similarity between code and text is a meaningful indicator of semantic quality and task alignment.
    Both the node scoring (Eq. 4) and the Consistency metric in evaluation rely on text-embedding-ada-002; the paper assumes these similarities capture code quality.
  • domain assumption The co-learning evaluation metric from the authors' prior work [23] is a valid and unbiased measure of code quality.
    Quality is defined as the product of completeness, executability, consistency, and granularity following [23], authored by the same group; its validity is taken as given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Co-Saving: Resource Aware Multi-Agent Collaboration for Software Development." pith.science (2026). https://pith.science/paper/DEMQAZ3C

@misc{pith2026250521898,
  author       = {Pith},
  title        = {Pith review of: Co-Saving: Resource Aware Multi-Agent Collaboration for Software Development},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DEMQAZ3C}},
  note         = {Machine review of arXiv:2505.21898}
}
read the original abstract

Recent advancements in Large Language Models (LLMs) and autonomous agents have demonstrated remarkable capabilities across various domains. However, standalone agents frequently encounter limitations when handling complex tasks that demand extensive interactions and substantial computational resources. Although Multi-Agent Systems (MAS) alleviate some of these limitations through collaborative mechanisms like task decomposition, iterative communication, and role specialization, they typically remain resource-unaware, incurring significant inefficiencies due to high token consumption and excessive execution time. To address these limitations, we propose a resource-aware multi-agent system -- Co-Saving (meaning that multiple agents collaboratively engage in resource-saving activities), which leverages experiential knowledge to enhance operational efficiency and solution quality. Our key innovation is the introduction of "shortcuts" -- instructional transitions learned from historically successful trajectories -- which allows to bypass redundant reasoning agents and expedite the collective problem-solving process. Experiments for software development tasks demonstrate significant advantages over existing methods. Specifically, compared to the state-of-the-art MAS ChatDev, our method achieves an average reduction of 50.85% in token usage, and improves the overall code quality by 10.06%.

Figures

Figures reproduced from arXiv: 2505.21898 by the authors.

Figure 1
Figure 1. A schematic representation of the executing process, including reference chain and inference [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of reference chain and inference chain to represent the shortcut filtering process. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of path length, time cost and number of tokens. experiments with Co-Saving [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 2
Figure 2. Figure 2: Ultimately, Co-Saving successfully generates an executable program with a correct GUI [PITH_FULL_IMAGE:figures/full_fig_p009_2.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 49 canonical work pages

  1. [1]

    Attention is All You Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is All You Need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017

    work page 2017

  2. [2]

    Language Models are Few-Shot Learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott G...

    work page 1901

  3. [3]

    Sparks of Artificial General Intelligence: Early Experiments with GPT-4

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of Artificial General Intelligence: Early Experiments with GPT-4. InarXiv preprint arXiv:2303.12712, 2023

    arXiv 2023

  4. [4]

    Toran Bruce Richards. AutoGPT. Inhttps://github.com/Significant-Gravitas/AutoGPT, 2023

    work page 2023

  5. [5]

    Generative Agents: Interactive Simulacra of Human Behavior

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), pages 1–22, 2023

    work page 2023

  6. [6]

    Chain-of-thought Prompting Elicits Reasoning in Large Language Models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought Prompting Elicits Reasoning in Large Language Models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24824–24837, 2022

    work page 2022

  7. [7]

    Toolformer: Language Models Can Teach Themselves to Use Tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools. InThirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023

    work page 2023

  8. [8]

    CAMEL: Communicative Agents for ”Mind” Exploration of Large Language Model Society

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative Agents for ”Mind” Exploration of Large Language Model Society. InThirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023

    work page 2023

Show all 66 references
  1. [9]

    ChatDev: Communicative Agents for Software Development

    Chen Qian, Xin Cong, Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative Agents for Software Development. InProceedings of the 62st Annual Meeting of the Association for Computational Lin...

    2023

  2. [10]

    Software development

    Harlan D Mills. Software development. InIEEE Transactions on Software Engineering, number 4, pages 265–273, 1976

    1976

  3. [11]

    GPT-Engineer

    Anton Osika. GPT-Engineer. Inhttps://github.com/AntonOsika/gpt-engineer, 2023

    2023

  4. [12]

    Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration

    Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, and V olker Tresp. Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration. In Toby Walsh, Julie Shah, and Zico Kolter, editors,AAAI-25, Sponsored by the Association for the Adva...

    2025

  5. [13]

    Humanoid Agents: Platform for Simulating Human-like Generative Agents

    Zhilin Wang, Yu Ying Chiu, and Yu Cheung Chiu. Humanoid Agents: Platform for Simulating Human-like Generative Agents. In Yansong Feng and Els Lefever, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions (EMNLP...

    2023

  6. [14]

    War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars

    Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars. InarXiv preprint arXiv:2311.17227, 2023

    2023 arXiv

  7. [15]

    V oyager: An Open-Ended Embodied Agent with Large Language Models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An Open-Ended Embodied Agent with Large Language Models. InIntrinsically-Motivated and Open-Ended Learning Workshop @NeurIPS2023, 2023

    2023

  8. [16]

    Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory

    Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai. Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based ...

    2023 arXiv

  9. [17]

    Avalon’s Game of Thoughts: Battle Against Deception through Recursive Contemplation

    Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon’s Game of Thoughts: Battle Against Deception through Recursive Contemplation. InFindings of the Association for Computational Linguistics: ACL ...

    2024

  10. [18]

    MindAgent: Emergent Gaming Interaction

    Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi V o, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, and Jianfeng Gao. MindAgent: Emergent Gaming Interaction. InarXiv preprint arXiv:2309.09971, 2023

    2023 arXiv

  11. [19]

    Benchmarking Large Language Models As AI Research Agents

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Benchmarking Large Language Models As AI Research Agents. InarXiv preprint arXiv:2310.03302, 2023

    2023 arXiv

  12. [20]

    Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis

    Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas V odrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel McFarland, and James Zou. Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis. InarXiv p...

    arXiv

  13. [21]

    TimeArena: Shaping efficient multitasking language agents in a time-aware simulation

    Yikai Zhang, Siyu Yuan, Caiyu Hu, Kyle Richardson, Yanghua Xiao, and Jiangjie Chen. TimeArena: Shaping efficient multitasking language agents in a time-aware simulation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Ass...

    2024

  14. [22]

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta Programming for A Multi-Agent Collaborative Fr...

    2024

  15. [23]

    Experiential co-learning of software-developing agents

    Chen Qian, Yufan Dang, Jiahao Li, Wei Liu, Zihao Xie, YiFei Wang, Weize Chen, Cheng Yang, Xin Cong, Xiaoyin Che, Zhiyuan Liu, and Maosong Sun. Experiential co-learning of software-developing agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the ...

    2024

  16. [24]

    Training Language Models to Follow Instructions with Human Feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  17. [25]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jef...

    2024

  18. [26]

    Llama: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and Efficient Foundation Language Models. InarXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  19. [27]

    React: Synergizing reasoning and acting in language models, 2023

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023

    2023

  20. [28]

    Language Models are Unsupervised Multitask Learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners. InOpenAI blog, volume 1, page 9, 2019

    2019

  21. [29]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilities of Large Language Models. I...

    2022

  22. [30]

    Role Play with Large Language Models

    Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role Play with Large Language Models. InNature, volume 623, pages 493–498, 2023

    2023

  23. [31]

    Evaluating Large Language Models Trained on Code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating Large Language Models Trained on Code. InarXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  24. [32]

    Popat, Peng Xu, Franz J

    Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. Large Language Models in Machine Translation. In Jason Eisner, editor,Proceedings of the 2007 Joint Con- ference on Empirical Methods in Natural Language Processing and Computational Natural Language Lea...

    2007

  25. [33]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V . Le, Denny Zhou, and Xinyun Chen. Large Language Models as Optimizers. InThe Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  26. [34]

    Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting. InProceedings of the 2024 Conference of the N...

    2024

  27. [35]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. InarXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  28. [36]

    Webarena: A realistic Web Environment for Building Autonomous Agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic Web Environment for Building Autonomous Agents. InThe Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  29. [37]

    Xing, and Zhiting Hu

    Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P. Xing, and Zhiting Hu. PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization. InThe Twelfth International Conference on Learning Representation...

    2024

  30. [38]

    Large Language Models as Tool Makers

    Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large Language Models as Tool Makers. InThe Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  31. [39]

    Toolllm: Facilitating Large Language Models to Master 16000+ Real-World APIs

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating Large Language Models to Master 16000+ Real-World APIs. InThe Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  32. [40]

    TPTU: Task Planning and Tool Usage of Large Language Model-based AI Agents

    Jingqing Ruan, YiHong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, du qing, shi shiwei, Hangyu Mao, Xingyu Zeng, and Rui Zhao. TPTU: Task Planning and Tool Usage of Large Language Model-based AI Agents. InNeurIPS 2023 Foundation Models for Decision Making Workshop, 2023

    2023

  33. [41]

    GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction

    Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction. InThirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023

    2023

  34. [42]

    Cognitive archi- tectures for language agents

    Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive archi- tectures for language agents. InTransactions on Machine Learning Research (TMLR), 2024. Survey Certification

    2024

  35. [43]

    Agentverse: Facilitating Multi-agent Collaboration and Exploring Emergent Behaviors in Agents

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating Multi-agent Collaboration and Exploring Emergent Behaviors in Agents. InThe Twelfth International Conference on Learning Rep...

    2024

  36. [44]

    BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents. ...

    2024

  37. [45]

    ExpeL: LLM Agents Are Experiential Learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM Agents Are Experiential Learners. InProceedings of the AAAI Conference on Artificial Intelligence, number 17, pages 19632–19642, Mar. 2024

    2024

  38. [46]

    LASER: LLM agent with state-space exploration for web navigation

    Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, and Dong Yu. LASER: LLM agent with state-space exploration for web navigation. InNeurIPS 2023 Foundation Models for Decision Making Workshop, 2023

    2023

  39. [47]

    On Generative Agents in Recommendation

    An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, and Tat-Seng Chua. On Generative Agents in Recommendation. InarXiv preprint arXiv:2310.10108, 2023

    2023 arXiv

  40. [48]

    When Large Language Model based Agent Meets User Behavior Analysis: A Novel User Simulation Paradigm

    Lei Wang, Jingsen Zhang, Hao Yang, Zhiyuan Chen, Jiakai Tang, Zeyu Zhang, Xu Chen, Yankai Lin, Ruihua Song, Wayne Xin Zhao, Jun Xu, Zhicheng Dou, Jun Wang, and Ji-Rong Wen. When Large Language Model based Agent Meets User Behavior Analysis: A Novel User Simulation Paradigm. In...

    2023 arXiv

  41. [49]

    DesignGPT: Multi-Agent Collaboration in Design

    Shiying Ding, Xinyi Chen, Yan Fang, Wenrui Liu, Yiwu Qiu, and Chunlei Chai. DesignGPT: Multi-Agent Collaboration in Design. In2023 16th International Symposium on Computational Intelligence and Design (ISCID), pages 204–208, 2023

    2023

  42. [50]

    LLM-powered Autonomous Agents

    Lilian Weng. LLM-powered Autonomous Agents. Inlilianweng.github.io, 2023

    2023

  43. [51]

    Self-evolving multi-agent collaboration networks for software development

    Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Siheng Chen. Self-evolving multi-agent collaboration networks for software development. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  44. [52]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st An...

    2023

  45. [53]

    Agentrefine: Enhancing agent general- ization through refinement tuning

    Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma GongQue, Weihao Zeng, Wei Wang, Jingang Wang, Xunliang Cai, and Weiran Xu. Agentrefine: Enhancing agent general- ization through refinement tuning. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  46. [54]

    Enhancing language model agents using diversity of thoughts

    Vijay Lingam, Behrooz Omidvar Tehrani, Sujay Sanghavi, Gaurav Gupta, Sayan Ghosh, Linbo Liu, Jun Huan, and Anoop Deoras. Enhancing language model agents using diversity of thoughts. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  47. [55]

    Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence

    Weize Chen, Ziming You, Ran Li, yitong guan, Chen Qian, Chenyang Zhao, Cheng Yang, Ruob- ing Xie, Zhiyuan Liu, and Maosong Sun. Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence. InThe Thirteenth International Conference on Learning Repre...

    2025

  48. [56]

    Synergistic multi- agent framework with trajectory learning for knowledge-intensive tasks

    Shengbin Yue, Siyuan Wang, Wei Chen, Xuanjing Huang, and Zhongyu Wei. Synergistic multi- agent framework with trajectory learning for knowledge-intensive tasks. In Toby Walsh, Julie Shah, and Zico Kolter, editors,AAAI-25, Sponsored by the Association for the Advancement of Art...

    2025

  49. [57]

    Scaling large language model- based multi-agent collaboration

    Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large language model- based multi-agent collaboration. InThe Thirteenth International Conference on Learning Representation...

    2025

  50. [58]

    Monte carlo planning with large language model for text- based game agents

    Zijing Shi, Meng Fang, and Ling Chen. Monte carlo planning with large language model for text- based game agents. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  51. [59]

    Benyamin T. Tabarsi. Developing llm-powered trustworthy agents for personalized learning support. In Toby Walsh, Julie Shah, and Zico Kolter, editors,AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025, Philadelphia...

    2025

  52. [60]

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework. InarX...

    2023 arXiv

  53. [61]

    Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. InarXiv preprint arXiv:2305.19118, 2023

    2023 arXiv

  54. [62]

    Chateval: Towards Better LLM-based Evaluators through Multi-agent Debate

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards Better LLM-based Evaluators through Multi-agent Debate. InThe Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  55. [63]

    GameGPT: Multi- agent Collaborative Framework for Game Development

    Dake Chen, Hanbin Wang, Yunhao Huo, Yuzhao Li, and Haoyang Zhang. GameGPT: Multi- agent Collaborative Framework for Game Development. InarXiv preprint arXiv:2310.08067, 2023

    2023 arXiv

  56. [64]

    Chawla, Olaf Wiest, and Xiangliang Zhang

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V . Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. 2024

    2024

  57. [65]

    Agentdropout: Dynamic agent elimination for token-efficient and high-performance llm-based multi-agent collaboration, 2025

    Zhexuan Wang, Yutong Wang, Xuebo Liu, Liang Ding, Miao Zhang, Jie Liu, and Min Zhang. Agentdropout: Dynamic agent elimination for token-efficient and high-performance llm-based multi-agent collaboration, 2025

    2025

  58. [66]

    Budget-constrained tool learning with planning

    Yuanhang Zheng, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. Budget-constrained tool learning with planning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 9039–9052, Bangkok, Thail...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.