Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Cross-Task Experiential Learning on LLM-based Multi-Agent Collaboration

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that reward-annotated step experiences, retrieved as few-shot examples at inference, let LLM multi-agent teams produce higher-quality solutions and converge faster than solving each task from scratch.

desk verdict Plausible step-level experience retrieval for multi-agent LLMs, but the current evidence is too thin to support the SOTA claims. read the letter →

arxiv 2505.23187 v1 pith:X4FQ3F2D submitted 2025-05-29 cs.CL cs.AIcs.MA

Yilong Li , Chen Qian , Yu Xia , Ruijie Shi , Yufan Dang , Zihao Xie , Ziming You , Weize Chen
show 7 more authors
Cheng Yang Weichuan Liu Ye Tian Xuantang Xiong Lei Han Zhiyuan Liu Maosong Sun
This is my paper
classification cs.CLcs.AIcs.MA
keywords multi-agentsystemsexperientiallearningretrieval-augmentedgenerationLLMagentscross-tasktransferreward-guidedretrievaldivide-and-conquerworkflowsoftwaredevelopment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MAEL is a framework that gives LLM-based multi-agent teams a memory: during a learning phase, each agent stores the inputs and outputs of every decision step together with a reward measuring that step's contribution. At inference, each agent retrieves the most similar high-reward past steps and feeds them to the model as few-shot examples. The paper claims this cross-task step-level experience retrieval yields higher-quality solutions and faster convergence than solving each task from scratch, with the largest gains on structurally similar tasks such as repository-scale software development. Across five benchmarks, the MAEL variants beat the single-agent and multi-agent baselines on average, while cutting token consumption and convergence rounds. If true, this gives a way to transfer knowledge across tasks in multi-agent systems without fine-tuning the underlying model.

What carries the argument

The load-bearing objects are the per-agent experience pool and the reward-weighted retrieval score. The pool stores tuples $(s_t,a_t,r_t)$ for five decision-step types -- solve, solvability judge, decompose, critique, and aggregate -- with rewards derived from task-specific metrics such as the quality improvement after critique. The retrieval score $\alpha\cdot\mathrm{similarity}(s_t,s_j)+(1-\alpha)\cdot r_j$ combines embedding cosine similarity with stored reward, and the top-ranked entry is inserted into the agent's prompt as a few-shot example. This mechanism converts episodic experience into prompt-level guidance, which is what carries the claimed gains in quality and convergence speed.

What would settle it

Rerun Table 1 with every method powered by the same base model, the same number of agents, and the same inference budget, and swap MAEL's reward-weighted retrieval for random retrieval from the same experience pool: if the random-retrieval version matches or beats the reward-weighted version, or the experienced version fails to beat its own no-experience ablation, the central claim would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that a multi-agent system can treat its accumulated task-solving experience as a learnable parameter space. The system is an undirected graph of agents, and the workflow runs recursively from the highest-closeness-centrality agent through decompose, solve, critique, and aggregate steps. In the experiential learning phase, each step's input, output, and reward $r_t = g(s_t,a_t)$ are stored in the responsible agent's pool $\mathcal{P}=\{(s_t,a_t,r_t)\}_{t=1}^T$. At inference, the agent scores each stored experience by $\alpha\cdot\mathrm{similarity}(s_t,s_j)+(1-\alpha)\cdot r_j$ and appends the top one to its prompt. The paper's position is that step-wise retrieval, in particular, accumulates locally optimal decisions into globally better solutions, and the mechanism degrades gracefully even when the retrieved experiences are low-reward.

Load-bearing premise

The load-bearing premise is that Table 1's baselines ran under identical conditions to MAEL, with the same base model, the same number of agents, and the same inference budget; if the comparison gave MAEL a stronger model or more agents, the reported gains could come from compute or model choice rather than from experience retrieval.

Editorial extensions

If this is right

  • Step-wise experience retrieval should be the default choice for tasks with overlapping solution traces, since MAEL Step gains 9.2 percentage points over MAEL without experience on the repository-scale benchmark.
  • Because the mechanism only changes prompts and never updates model weights, it can be dropped into any multi-agent system and should keep improving as the experience pool grows over a corpus of past tasks.
  • Retrieval is an efficiency method as well as an accuracy method: the paper reports a 49 percent token reduction on long-form generation with MAEL Task and a 26 percent reduction in convergence rounds on repository-scale development with MAEL Step.
  • On knowledge-heavy or arithmetic tasks where solution traces overlap little, experience retrieval can slightly hurt, so the data point toward a task-adaptive switch that uses retrieval only when traces are likely to overlap.
  • Experience pool size is a real design axis: scaling from 0 to 30 stored tasks produced a 12.0 percentage-point gain on long-form generation and a non-monotonic pattern on code generation, indicating that relevance and diversity matter alongside quantity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper: replacing task-level metrics with process-level reward models for step scoring should sharpen retrieval, because the current rewards are sparse and only as good as the chosen task metric.
  • Not tested in the paper: the authors' explanation of MMLU and GSM-8K degradation implies a measurable precondition -- one could compute the embedding similarity between past and current step inputs and predict in advance whether MAEL will help or hurt.
  • Not tested in the paper: the experience pool is described as team-level learnable weights, so the same pool could in principle be shared across teams or reused with a different base model, although the paper only demonstrates reuse within one system.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces MAEL, a multi-agent LLM framework with an explicit experience-learning stage. In the learning phase, a graph of agents solves training tasks via a decompose-solve-critique-aggregate workflow; each decision step is scored by a task-specific reward, and the (input, output, reward) triple is stored in the acting agent's experience pool. At inference, agents retrieve similar high-reward experiences (weighted combination of embedding similarity and reward) and use them as few-shot examples. Three variants are compared: no experience (MAEL ØExp), task-wise retrieval (MAEL Task), and step-wise retrieval (MAEL Step). Experiments on MMLU, GSM-8K, HumanEval, CommonGen-Hard, and SRDD compare MAEL against GPT-4o-mini, Claude-3.5-sonnet, AutoGen, and EvoMac. The abstract and Section 3.2 claim faster convergence and higher-quality solutions from experience retrieval.

Significance. The underlying idea is attractive and timely: converting stored interaction traces into reward-weighted retrievable exemplars is a simple way to give multi-agent systems a form of cross-task memory, and the graph-based workflow is clearly described. The internal ablations (MAEL Task/Step against MAEL ØExp) and the scaling curve in Figure 3 are the right kind of evidence for the mechanism. If the reported gains were reproducible, MAEL would be a solid contribution to practical LLM-based multi-agent systems. However, the paper as submitted does not establish those gains: the headline comparison is not controlled for model or compute, the experiments are unreplicated and lack error bars, the test-set sizes are not stated, and there are internal numerical inconsistencies. The manuscript also provides no code or data release. Strengths are the clarity of the idea and the presence of targeted ablations; the weaknesses are in the evidence base for the central claims.

major comments (3)
  1. [Section 3.1 and Table 1] The central comparison in Table 1 is not fair as reported. Section 3.1 says MAEL defaults to GPT-3.5 (temperature 0.2), while the two single-agent baselines are GPT-4o-mini and Claude-3.5-sonnet; the text does not state which backbone is used for AutoGen and EvoMac, nor how agent counts and inference budgets are matched. The sentence 'All baselines are rerun under identical settings' does not resolve this, because a single-agent baseline by definition does not have four agents. If the baselines use a stronger or weaker model or a different number of calls, then the averaged gains of MAEL ØExp (4.89 points over GPT-4o-mini) and MAEL Task (69.69 average) could be due to model choice, ensemble size, or compute, rather than to the experience mechanism. Please specify the exact model, temperature, agent count, and total token budget for every row of Table 1, and match them across conditions.
  2. [Section 3.2, Table 1, and Figure 2] The quality and efficiency claims lack statistical support. The number of test items is never stated; the percentages in Table 1 imply 30-item evaluations for MMLU, GSM, and HumanEval. Under that reading, MAEL Task versus MAEL ØExp on HumanEval is 27/30 versus 23/30, and the two-sided exact McNemar p-value is at least 0.125 no matter how the items are paired; the same fragility affects the averaged gains and the composite-metric improvements on CommonGen and SRDD, which are reported without error bars. Figure 2 reports completion tokens and convergence iterations from what appear to be single unreplicated runs, with no confidence intervals or seeds. The text's claim that experience leads to 'significantly improved' performance is therefore not established at the conventional 0.05 level. The authors should report exact test sizes, per-item results, multiple seeds, and variance estimates (or bootstrap intervals) for the headline averages and for Figure 2.
  3. [Section 3.5, Table 4, Section 3.6, and Figure 3] There are internal numerical inconsistencies in the factor analysis and scaling experiments. Section 3.5 states that the no-experience baseline on HumanEval is 73.3%, while Table 1 and Figure 3 report 76.7%; the claimed '+6.7 percentage points' for the Low Reward & Low Similarity condition is consistent only with the 73.3% value, so the analysis in Section 3.5 uses a different baseline than the rest of the paper. Section 3.6 reports CommonGen at 20 stored tasks as 43.4% (+0.7 pp over 42.7%), but Figure 3 plots 44.4%, which is +1.7 pp. These discrepancies must be resolved because the conclusions about robustness to noisy experiences and about scaling rely on exact baseline numbers.
minor comments (5)
  1. [Section 2.1] Contains the placeholder 'Algorithm??'; either include the formal algorithm or remove the cross-reference.
  2. [Section 3.1 and Table 1] The pass@k for HumanEval is never specified; state the value of k and the sampling procedure.
  3. [Section 3.3 and Table 2] The reported '9.2% improvement' and '20.21% gain' are inconsistent with the percentage-point differences in Table 2 (9.2 and 20.4 points); specify whether these are relative changes or percentage points.
  4. [Section 3.4 and Figure 2] The caption says all values are normalized relative to MAEL ØExp, but the axis numbers appear to be raw counts; clarify the normalization and label the series for each method.
  5. [Section 4 and References] The method name is written 'EvoMac' in the text and 'EvoMAC' in the references; standardize the spelling.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reward signal uses the evaluation metric, but the paper's results are empirical, non-monotonic, and not derivable by construction.

full rationale

The reward computation in Section 2.2 (r_t ← g(s_t, a_t)) uses task-specific metrics, and retrieval in Eq. 2 weights similarity against reward, with the same metrics used for evaluation. This is a training-signal/evaluation-alignment loop, not a circular derivation: the retrieved experience is a few-shot exemplar, not the reported score, and the final solution is generated by the LLM workflow rather than computed from the retrieval score. The empirical results refute any claim of forced identity: Table 1 shows performance degradation on MMLU and GSM when experience is added, and Figure 3 shows a non-monotonic dip at 20 accumulated tasks, so high-reward retrieval does not guarantee high evaluation scores. The citations to Qian et al. 2024a,b are background references for linear/graph agent workflows and the SRDD benchmark; they are not used to justify the central claim that MAEL improves quality or convergence, and no uniqueness theorem or ansatz is imported from the authors' prior work. Concerns about baseline model choice, single-run evaluations, and missing significance tests are validity/fairness issues, not circularity. Therefore the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several hand-set hyperparameters, a proxy reward assumption, an embedding relevance assumption, and an unverified comparability assumption. None of these is independently validated in the paper, but none is a brand-new physical or mathematical entity. The method introduces no invented entities in the sense of new particles, forces, dimensions, or conserved quantities.

free parameters (5)
  • alpha = 0.5
    Weight balancing similarity and reward in retrieval_score (Equation 2); set by default, not tuned per task.
  • experience pool size = 30 training tasks
    Each dataset uses 30 training tasks to build pools; scaling experiments vary pool size, but no principled criterion for 30 is given.
  • max refinement rounds = 2 or 3 depending on dataset
    Maximum critique and refinement iterations; hand-set per dataset in Section 3.1, and this changes both experience collection and the convergence measure.
  • max decomposition layers = 1 or 2
    Depth of recursive task decomposition; hand-set per dataset in Section 3.1.
  • number of agents = 4
    Default topology uses four nodes 'aligning with other multi-agent baselines'; this is a structural choice, not justified by data.
assumptions (4)
  • domain assumption Step-level rewards such as Rsolve, Rdecompose, and Rcritique are valid proxies for the quality contribution of each agent action.
    Defined in Section 2.2 without validation that converting final solution quality to per-step rewards assigns credit correctly.
  • domain assumption text-embedding-ada-002 cosine similarity is a reliable measure of task relevance across solution traces.
    Used in Section 2.3 to rank experiences; no evidence is given that semantic embedding distance aligns with helpfulness.
  • domain assumption 30 training tasks per benchmark form a representative experience base for held-out tasks from the same dataset.
    Stated in Section 3.1; train/test split and possible overlap are not reported.
  • ad hoc to paper Baseline conditions are comparable to MAEL in model and compute.
    Implementation Details uses GPT-3.5 for MAEL while Table 1 lists GPT-4o-mini and Claude; the 'identical settings' claim is ambiguous and not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Task Experiential Learning on LLM-based Multi-Agent Collaboration." pith.science (2026). https://pith.science/paper/X4FQ3F2D

@misc{pith2026250523187,
  author       = {Pith},
  title        = {Pith review of: Cross-Task Experiential Learning on LLM-based Multi-Agent Collaboration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X4FQ3F2D}},
  note         = {Machine review of arXiv:2505.23187}
}
read the original abstract

Large Language Model-based multi-agent systems (MAS) have shown remarkable progress in solving complex tasks through collaborative reasoning and inter-agent critique. However, existing approaches typically treat each task in isolation, resulting in redundant computations and limited generalization across structurally similar tasks. To address this, we introduce multi-agent cross-task experiential learning (MAEL), a novel framework that endows LLM-driven agents with explicit cross-task learning and experience accumulation. We model the task-solving workflow on a graph-structured multi-agent collaboration network, where agents propagate information and coordinate via explicit connectivity. During the experiential learning phase, we quantify the quality for each step in the task-solving workflow and store the resulting rewards along with the corresponding inputs and outputs into each agent's individual experience pool. During inference, agents retrieve high-reward, task-relevant experiences as few-shot examples to enhance the effectiveness of each reasoning step, thereby enabling more accurate and efficient multi-agent collaboration. Experimental results on diverse datasets demonstrate that MAEL empowers agents to learn from prior task experiences effectively-achieving faster convergence and producing higher-quality solutions on current tasks.

Figures

Figures reproduced from arXiv: 2505.23187 by the authors.

Figure 1
Figure 1. MAEL casts MAS as a graph of LLM-based agents, each endowed with an experience pool [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison of token consumption (Left) and number of convergence round (Right). Since [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Performance of MAELTask on HumanEval and CommonGen benchmarks with varying experience pool sizes (0, 10, 20, and 30 accumulated tasks). On CommonGen, initial gains are more gradual: performance climbs from 34.5 % with no experience to 42.7 % at 10 tasks (+8.2 pp), 43.4 % at 20 tasks (+0.7 pp), and finally 46.5 % at 30 tasks (+3.1 pp). The gentler slope here reflects CommonGen’s larger solution space and higher varia… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Solution quality comparison across five cases using different experience retrieval methods. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: A case illustrating the assistance of experience (yellow). [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. StackPlanner: A Centralized Hierarchical Multi-Agent System with Task-Experience Memory Management

    cs.AI 2026-01 conditional novelty 5.0 of 10

    A hierarchical multi-agent system whose coordinator actively condenses/prunes task memory and retrieves cross-task experience, trained with GRPO, reports higher F1 than baselines on four benchmarks.

Reference graph

Works this paper leans on

52 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Attention is All You Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is All You Need . In Advances in Neural Information Processing Systems (NeurIPS), 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

    work page 2017

  2. [2]

    Language Models are Unsupervised Multitask Learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners . In OpenAI Blog, 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

    work page 2019

  3. [3]

    ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs . In International Conference on Learning Representations (ICLR), 2024 a . URL https://iclr.cc/virtual/2024/poster/18267

    work page 2024

  4. [4]

    ToolFormer: Language Models Can Teach Themselves to Use Tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. ToolFormer: Language Models Can Teach Themselves to Use Tools . In arXiv preprint arXiv:2302.04761, 2023. URL https://arxiv.org/pdf/2302.04761

    arXiv 2023

  5. [5]

    AgentVerse: Facilitating Multi-agent Collaboration and Exploring Emergent Behaviors in Agents

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. AgentVerse: Facilitating Multi-agent Collaboration and Exploring Emergent Behaviors in Agents . In International Conference on Learning Representations (ICLR), 2024 a . URL https://iclr.cc/virtual/2024/poster/19109

    work page 2024

  6. [6]

    CAMEL: Communicative Agents for ``Mind'' Exploration of Large Language Model Society

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative Agents for ``Mind'' Exploration of Large Language Model Society . In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2303.17760

    arXiv 2023

  7. [7]

    Autogen: Enabling next-gen llm applications via multi-agent conversation

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023. URL https://arxiv.org/abs/2308.08155

    arXiv 2023

  8. [8]

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework . In International Conference on Learning Representations (ICLR), 2024. ...

    work page 2024

Show all 52 references
  1. [9]

    Generative Agents: Interactive Simulacra of Human Behavior

    Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative Agents: Interactive Simulacra of Human Behavior . In Annual ACM Symposium on User Interface Software and Technology (UIST), 2023. URL https://doi.org/10.114...

    2023

  2. [11]

    Self-evolving multi-agent collaboration networks for software development

    Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Siheng Chen. Self-evolving multi-agent collaboration networks for software development. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevi...

    2025

  3. [12]

    Evoflow: Evolving diverse agentic workflows on the fly

    Guibin Zhang, Kaijie Chen, Guancheng Wan, Heng Chang, Hong Cheng, Kun Wang, Shuyue Hu, and Lei Bai. Evoflow: Evolving diverse agentic workflows on the fly. arXiv preprint arXiv:2502.07373, 2025. URL https://arxiv.org/abs/2502.07373

    2025 arXiv

  4. [13]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Łukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems . In arXiv preprint arXiv:2110.14168, 202...

    arXiv

  5. [14]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding . In International Conference on Learning Representations (ICLR), 2021. URL https://api.semanticscholar.org/Corpus...

    2021

  6. [15]

    Evaluating Large Language Models Trained on Code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating Large Language Models Trained on Code . In arXiv preprint arXiv:2107.03374, 2021. URL https://arxiv.or...

    2021 arXiv

  7. [16]

    Self-Refine: Iterative Refinement with Self-Feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterativ...

    2023

  8. [17]

    ChatDev: Communicative Agents for Software Development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative Agents for Software Development . In Annual Meeting of the Association for Computational ...

    2024

  9. [18]

    Experiential Co-Learning of Software-Developing Agents

    Chen Qian, Yufan Dang, Jiahao Li, Wei Liu, Zihao Xie, Yifei Wang, Weize Chen, Cheng Yang, Xin Cong, Xiaoyin Che, Zhiyuan Liu, and Maosong Sun. Experiential Co-Learning of Software-Developing Agents . In Annual Meeting of the Association for Computational Linguistics (ACL), 202...

    2024

  10. [19]

    Gpt-4o mini: Advancing cost-efficient intelligence

    OpenAI . Gpt-4o mini: Advancing cost-efficient intelligence. 2024. URL https://platform.openai.com/docs/models/gpt-4o-mini

    2024

  11. [20]

    Claude 3.5 sonnet: The most intelligent and cost-effective mid-sized model

    Anthropic . Claude 3.5 sonnet: The most intelligent and cost-effective mid-sized model. 2024. URL https://www.anthropic.com/index/claude-3-5-sonnet

    2024

  12. [21]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling . In arXiv preprint arXiv:2407.21787, 2024. URL https://arxiv.org/abs/2407.21787

    2024 arXiv

  13. [22]

    Sparks of Artificial General Intelligence: Early Experiments with GPT-4

    S \'e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of Artificial General Intelligence: Early Experiments with GPT-4 . In arXiv preprint arXiv:2303.12712, 2023. URL...

    2023 doi arXiv

  14. [23]

    Llama: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and Efficient Foundation Language Models . In arXiv preprint arXiv:2302.13971, 2023. URL h...

    2023 arXiv

  15. [24]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilities of Large Language Models . ...

    2022 arXiv

  16. [25]

    Role Play with Large Language Models

    Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role Play with Large Language Models . In Nature, 2023. URL https://www.nature.com/articles/s41586-023-06647-8

    2023

  17. [26]

    Popat, Peng Xu, Franz J

    Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. Large Language Models in Machine Translation . In Empirical Methods in Natural Language Processing (EMNLP), 2007. URL https://aclanthology.org/D07-1090/

    2007

  18. [27]

    Training Language Models to Follow Instructions with Human Feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  19. [28]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large Language Models as Optimizers . In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2309.03409

    2024 arXiv

  20. [29]

    Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting . In North American Chapter of the Association...

    2024

  21. [30]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models . In arXiv preprint arXiv:2001.08361, 2020. URL https://doi.org/10.48550/arXiv.2001.08361

    2001 doi arXiv

  22. [31]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. URL https://arxiv.org/abs/2307.13854

    2023 arXiv

  23. [32]

    RecAgent: A Novel Simulation Paradigm for Recommender Systems

    Lei Wang, Jingsen Zhang, Xu Chen, Yankai Lin, Ruihua Song, Wayne Xin Zhao, and Ji-Rong Wen. RecAgent: A Novel Simulation Paradigm for Recommender Systems . In arXiv preprint arXiv:2306.02552, 2023 a . URL https://arxiv.org/pdf/2306.02552

    2023 arXiv

  24. [33]

    Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models . In Annual Meeting of the Association for Computational Linguistics (ACL), 2023 b . URL ...

    2023

  25. [34]

    Toran Bruce Richards. AutoGPT . In https://github.com/Significant-Gravitas/AutoGPT, 2023. URL https://github.com/Significant-Gravitas/AutoGPT

    2023

  26. [35]

    Learning to Break: Knowledge-Enhanced Reasoning in Multi-Agent Debate System

    Haotian Wang, Xiyuan Du, Weijiang Yu, Qianglong Chen, Kun Zhu, Zheng Chu, Lian Yan, and Yi Guan. Learning to Break: Knowledge-Enhanced Reasoning in Multi-Agent Debate System . In arXiv preprint arXiv:2312.04854, 2024. URL https://arxiv.org/pdf/2312.04854

    2024 arXiv

  27. [36]

    Reflexion: Language Agents with Verbal Reinforcement Learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning . In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://arxiv.org/abs/2303.11366

    2024 arXiv

  28. [37]

    Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L

    Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive Architectures for Language Agents . In arXiv preprint arXiv:2309.02427, 2023. URL https://arxiv.org/pdf/2309.02427

    2023 arXiv

  29. [38]

    Dynamic LLM-Agent Network: An LLM-agent Collaboration Framework with Agent Team Optimization

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic LLM-Agent Network: An LLM-agent Collaboration Framework with Agent Team Optimization . In arXiv preprint arXiv:2310.02170, 2023. URL https://arxiv.org/pdf/2310.02170

    2023 arXiv

  30. [39]

    Large language models as tool makers

    Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint arXiv:2305.17126, 2023. URL https://arxiv.org/abs/2305.17126

    2023 arXiv

  31. [40]

    Maddison, and Tatsunori Hashimoto

    Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational Scaling Laws and the Predictability of Language Model Performance . In arXiv preprint arXiv:2405.10938, 2024. URL https://arxiv.org/pdf/2405.10938

    2024 arXiv

  32. [41]

    Expel: LLM Agents are Experiential Learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: LLM Agents are Experiential Learners . In AAAI Conference on Artificial Intelligence (AAAI), 2024. URL https://doi.org/10.1609/aaai.v38i17.29936

    2024 doi

  33. [42]

    Laser: Llm agent with state-space exploration for web navigation

    Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu. Laser: Llm agent with state-space exploration for web navigation. arXiv preprint arXiv:2309.08172, 2023. URL https://arxiv.org/abs/2309.08172

    2023 arXiv

  34. [43]

    On Generative Agents in Recommendation

    An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, and Tat-Seng Chua. On Generative Agents in Recommendation . In arXiv preprint arXiv:2310.10108, 2023. URL https://arxiv.org/pdf/2310.10108

    2023 arXiv

  35. [44]

    Augmenting Language Models with Long-Term Memory

    Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting Language Models with Long-Term Memory . In Advances in Neural Information Processing Systems (NeurIPS), 2023 c . URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ebd...

    2023

  36. [45]

    Designgpt: Multi-agent collaboration in design

    Shiying Ding, Xinyi Chen, Yan Fang, Wenrui Liu, Yiwu Qiu, and Chunlei Chai. Designgpt: Multi-agent collaboration in design. In 2023 16th International Symposium on Computational Intelligence and Design (ISCID), pages 204--208. IEEE, 2023. URL https://arxiv.org/abs/2311.11591

    2023 arXiv

  37. [46]

    Llm-powered autonomous agents

    Lilian Weng. Llm-powered autonomous agents. https://lilianweng.github.io/posts/2023-06-23-agent/, 2023

    2023

  38. [47]

    Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence

    Weize Chen, Ziming You, Ran Li, Yitong Guan, Chen Qian, Chenyang Zhao, Cheng Yang, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence . In arXiv preprint arXiv:2407.07061, 2024 b . URL https://arx...

    2024 arXiv

  39. [48]

    LM vs LM: Detecting Factual Errors via Cross Examination

    Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. LM vs LM: Detecting Factual Errors via Cross Examination . In Empirical Methods in Natural Language Processing (EMNLP), 2023. URL https://aclanthology.org/2023.emnlp-main.778/

    2023

  40. [49]

    Griffiths, and Mengdi Wang

    Xudong Guo, Kaixuan Huang, Jiale Liu, Wenhui Fan, Natalia Vélez, Qingyun Wu, Huazheng Wang, Thomas L. Griffiths, and Mengdi Wang. Embodied LLM Agents Learn to Cooperate in Organized Teams . In arXiv preprint arXiv:2403.12482, 2024. URL https://arxiv.org/pdf/2403.12482

    2024 arXiv

  41. [50]

    Evoagent: Towards automatic multi-agent generation via evolutionary algorithms

    Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. Evoagent: Towards automatic multi-agent generation via evolutionary algorithms. arXiv preprint arXiv:2406.14228, 2024. URL https://arxiv.org/abs/2406.14228

    2024 arXiv

  42. [51]

    Agentsquare: Automatic llm agent search in modular design space

    Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. Agentsquare: Automatic llm agent search in modular design space. arXiv preprint arXiv:2410.06153, 2024. URL https://arxiv.org/abs/2410.06153

    2024 arXiv

  43. [52]

    Automated design of agentic systems

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint arXiv:2408.08435, 2024. URL https://arxiv.org/abs/2408.08435

    2024 arXiv

  44. [53]

    Heterogeneous swarms: Jointly optimizing model roles and weights for multi-llm systems

    Shangbin Feng, Zifeng Wang, Palash Goyal, Yike Wang, Weijia Shi, Huang Xia, Hamid Palangi, Luke Zettlemoyer, Yulia Tsvetkov, Chen-Yu Lee, et al. Heterogeneous swarms: Jointly optimizing model roles and weights for multi-llm systems. arXiv preprint arXiv:2502.04510, 2025. URL h...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.