Pith. sign in

REVIEW 5 major objections 7 minor 5 cited by

OAgents: An Empirical Study of Building Effective Agents

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OAgents is an open-source framework that reports 73.93 percent Pass@3 on GAIA, the top score in its comparison table, and a systematic study of which agent components drive that score.

desk verdict A useful ablation catalog and open-source framework, but the SOTA claim is unsupported because they compare their Pass@3 to everyone else's Pass@1. read the letter →

arxiv 2506.15741 v2 pith:3S3JRX5Q submitted 2025-06-17 cs.AI cs.CL

He Zhu , Tianrui Qin , King Zhu , Heyuan Huang , Yeyi Guan , Jinxiang Xia , Yi Yao , Hanhao Li
show 16 more authors
Ningning Wang Pai Liu Tianhao Peng Xin Gui Xiaowan Li Yuhui Liu Yuchen Eleanor Jiang Jun Wang Changwang Zhang Xiangru Tang Ge Zhang Jian Yang Minghao Liu Xitong Gao Jiaheng Liu Wangchunshu Zhou
This is my paper
classification cs.AIcs.CL
keywords AIagentsGAIAbenchmarkempiricalstudyagentframeworkreproducibilitytest-timescalingwebsearchmodulardesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Agent research is hard to compare because frameworks differ in prompt details, tool implementations, and evaluation protocols, and this paper tries to fix that. It runs a controlled ablation study on the GAIA benchmark (with a BrowseComp check) to isolate which agent components—search sources, query optimization, document parsing, planning, memory, test-time scaling—drive performance. On that basis it releases OAgents, an open-source modular framework that it reports reaches 73.93 percent Pass@3 on GAIA, the highest number in its own comparison table, ahead of the closed-source systems it benchmarks. The paper also argues that previous open-source results are not reproducible, showing that its own replication of one open-source research agent degraded significantly. If the paper is right, agent teams can compare designs fairly using its protocol, and the field gets a modular open-source baseline.

What carries the argument

The load-bearing instrument is the component-level ablation built on the OAgents modular framework, organized around two axes the paper calls Factual Acquisition Capacity (tools, search, multimodal parsing) and Logical Reasoning Fidelity (plan, memory, test-time scaling). The evaluation protocol—Pass@N with multiple random runs, automatic re-answering on empty outputs, and majority voting when used—is what lets the paper attribute score differences to one component at a time. The individual levers are concrete and named: multi-source retrieval that blends five search sources and routes historical queries to an archival web index; a query pipeline of Reflect and Expand; a three-tool minimalist browser (Search, Visit, Read); plan revision every N steps plus subtask dependency graphs; plan tips injected as soft constraints; memory with summarization, vector retrieval, and long-term fusion; and test-time scaling via best-of-N, reflection, and a process reward model. These mechanisms are the vehicles for the paper's claims.

What would settle it

Run the full OAgents pipeline and its ablations ten times with different random seeds and report the score distribution per configuration; if the spread for a single configuration exceeds the reported gaps—for example the 14.54-point plan-tips gain or the 1-to-4-point memory gains—then the component ordering is not established. A stronger check is to hold out a fresh set of GAIA-style questions, craft plan tips without seeing them, and test whether the 73.93 percent Pass@3 and the ablation ranking survive.

Watch

Extended reading notes

Core claim

The central claim is that the effectiveness of a general-purpose agent is governed by a small set of measurable component choices, and that once those choices are identified under a stable protocol, an open-source framework can outperform proprietary ones on GAIA. Concretely, the paper reports OAgents-Pass@3 (at least one of three independent attempts correct) at 73.93 percent average, with Level 1 at 83.02, Level 2 at 74.42, and Level 3 at 53.85, versus 71.52 for the best closed-source system in its comparison table, and Pass@1 at 66.67 percent. The accompanying ablations attribute the gains to a structured web reader over raw HTML (9.3 percent on Level 2), five search sources over one (up 7.69 percent on Level 3), reflection-expansion query optimization (3.03 percent average), plan tips distilled from error logs (14.54 percent), long-term memory (about 4.2 percent over no memory), and best-of-4 test-time scaling (5.19 percent), while reflection alone hurt Level 3 by 6.62 percent. It further claims that standard evaluation protocols are so inconsistent that open-source systems are non-reproducible, citing its own replication of one open-source deep-research agent showing significant degradation.

Load-bearing premise

That differences of a few percentage points on the 165-question GAIA validation set are stable signals of component quality, not random run-to-run noise, and that the components chosen by inspecting this same set will keep their advantage on new questions.

Editorial extensions

If this is right

  • If the 73.93 percent Pass@3 result holds, open-source agent frameworks can match or exceed closed-source systems on GAIA without proprietary orchestration.
  • The ablation ordering says tooling and search engineering—multi-source retrieval, query optimization, document parsing—matter at least as much as planning and memory for factual tasks.
  • The reported non-reproducibility implies that any published GAIA score should be accompanied by its run count, error-handling rules, and ensembling method before it can be compared with others.
  • The 14.54 percent plan-tips gain suggests that mining an agent's own failure logs for heuristic guidance is a cheap, high-leverage way to improve planning on a specific benchmark.
  • Reflection's 6.62 percent drop on Level 3 tasks indicates test-time scaling methods that help easy questions can hurt hard multi-step reasoning, so agent design should be complexity-aware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the plan tips were derived from GAIA's error logs, the 14.54 percent gain is likely benchmark-specific: applying the same method to BrowseComp or a new benchmark would require mining that benchmark's failures first, so the transferability of this component is untested.
  • If protocol variance is really as large as the paper claims, the GAIA leaderboard may reorder substantially when every entry is re-run under one fixed protocol; an independent re-run of the leading closed-source systems with OAgents' protocol would provide that test.
  • The paper's two-axis decomposition suggests a cheap proxy for agent quality: measure search success per question and reasoning-chain stability separately; a testable extension is predicting GAIA performance from search coverage alone.
  • Multi-source search gains were concentrated in Level 3 tasks, so a natural follow-up is to test whether the benefit comes from source diversity or raw source count, for example by holding k=5 fixed but varying which sources are included.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper presents OAgents, an open-source modular agent framework, and reports a systematic empirical study on the GAIA benchmark and a BrowseComp subset. The central claims are (i) that OAgents achieves state-of-the-art performance among open-source agent frameworks on GAIA, with a Pass@3 average of 73.93% (Table 1), and (ii) that the accompanying ablations identify which design choices in planning, memory, tools, and test-time scaling matter most. The paper also proposes an evaluation protocol intended to reduce run-to-run variance and improve reproducibility, and it provides prompts and code. The main evidence is a set of comparisons against closed- and open-source baselines, sequential ablations on the GAIA validation set, and a small BrowseComp experiment.

Significance. If the claims were fully supported, the paper would be a useful contribution to the agentic AI literature: it is openly available, modular, and draws attention to the often-overlooked issues of evaluation variance and metric reporting. The strongest assets are the release of code and prompts, the explicit discussion of run-to-run variance and empty-answer handling, and the attempt to isolate the effect of individual components. However, the headline SOTA claim rests on a metric mismatch (Pass@3 for OAgents vs. Pass@1 for all other frameworks), several ablations are selected and evaluated on the same GAIA validation set, and the paper does not provide error bars or significance tests for differences of one to three percentage points on 165 questions. These issues mean that the central empirical conclusions are not currently established. The paper's methodology could become valuable if the comparisons were made apples-to-apples and if the evaluation protocol were applied uniformly to all baselines.

major comments (5)
  1. [Section 4.2, Table 1 and Table 9] The headline claim that 'OAgents-Pass@3 achieves the highest overall average score of 73.93%, outperforming all other frameworks' is not supported by the data as presented, because OAgents is the only system evaluated under Pass@3 while all other frameworks report Pass@1. Under the paper's own default metric (Section 4.1, 'we report the average Pass@1 score'), OAgents scores 66.67, below OWL-Workforce at 69.09 in Table 1 and below AWorld's 77.58 in Table 9, where the metric is listed as 'Unknown'. Since Pass@3 is at least as large as Pass@1 by definition, the comparison is internally invalid. The authors need to either evaluate all baselines under Pass@3 or compare OAgents under Pass@1 before claiming state-of-the-art performance.
  2. [Section 4.1, evaluation protocol] The evaluation protocol described in Section 4.1 allows the model to re-answer a question when the answer is empty or contains 'Unable to determine', while other frameworks' published Pass@1 scores almost certainly do not include such a retry mechanism. This protocol can only inflate OAgents' Pass@1 score relative to baselines, and the paper does not report how often re-answering occurs or what fraction of the final accuracy it contributes. Because the headline comparison in Table 1 partly rests on this protocol, the paper needs to either quantify the effect of re-answering or compare against baselines run under the identical protocol.
  3. [Section 4.2.2, Table 7] The 14.54% gain attributed to 'Plan tips' is obtained by adding heuristics that were themselves summarized from analysis of historical error logs on GAIA, and the evaluation is then performed on the same GAIA validation set. This is a selection-on-the-test-set procedure: the tips are hand-crafted from GAIA error analyses and then evaluated on GAIA, so the reported improvement does not provide evidence that the tips generalize. No held-out split, temporal split, or cross-validation is used. The claim that plan tips 'prove' the value of experiential knowledge is therefore not established by the present experiments.
  4. [Section 4.2, replication of Open Deep Research] The paper states, 'We replicate Open Deep Research [10] and note the results as Smolagents,' but Open Deep Research is a LangChain project and Smolagents [19] is a separate HuggingFace framework; these are different codebases. The reported 'significant degradation' of the replication is not accompanied by any description of what was changed relative to the original configuration, what model or budget was used, or how many runs were averaged. This makes the reproducibility claim unverifiable and the corresponding entry in Table 1 unreliable.
  5. [Section 4.2.1, Tables 3-6 and Section 4.2.2, Figures 2-3] Many conclusions rest on differences of less than 3 percentage points on the 165-question GAIA validation set, and the paper reports single runs without error bars, confidence intervals, or significance tests. Examples include Table 4 (multi-source k=3 at 52.12 vs. single-source at 51.52) and Table 7 (subtask decomposition at 53.94 vs. OAgents at 51.52). Given the paper's own emphasis on large run-to-run variance (Section 1), these small differences cannot be interpreted as evidence for the claimed component effects without additional statistical support.
minor comments (7)
  1. [Section 4.2] The phrase 'establishing a new standard for esay task handling' contains a typo: 'esay' should be 'easy'.
  2. [Table 7] The abbreviation 'r.p.' in the row 'r.p. Static workflow' is not defined; please spell out the phrase or add a note to the caption.
  3. [Section 4.2.2, Test-Time Scaling] The text discussing test-time scaling refers to 'Table 3' for the ablation results, but the TTS results appear in Figure 3; the citation should be corrected.
  4. [Figure 2] The caption and figure text contain malformed unicode or rendering artifacts (e.g., '/uni00000024/...'), which obscure the figure contents; please provide a readable version.
  5. [Table 9] The AWorld entry reports an 'Unknown' metric, yet it is used in the comparison table; either identify the metric or remove the row from a table that is otherwise about Pass@1 vs. Pass@3.
  6. [Equations (4) and (12)] The formulation πθ(at|st, H) = softmax(Q(st, at) + β·fH(st, at)) is presented as an augmented policy but is not clearly linked to the actual implementation described in the prompts; please clarify how this equation is operationalized.
  7. [References] AWorld [2] is cited in Table 9 but is not discussed in the Related Work section; please add a sentence describing this system and its evaluation protocol.

Circularity Check

3 steps flagged · score 6.0 of 10

Partial circularity: the Pass@3 SOTA claim is inflated by definition, and the headline GAIA score is an in-sample result obtained after tuning components (including Plan tips) on the same 165 questions, with no held-out validation.

  1. self definitional [Section 4.1 (Evaluation Protocol), Section 4.2 / Table 1, Section 5 / Table 9]
    "The primary metric used is Pass@N, which measures the probability that at least one correct solution is found among N independent model attempts. ... Notably, our method (OAgents-Pass@3) achieves the highest overall average score of 73.93%, outperforming all other frameworks, including both closed-source and open-source systems. ... we reimplemented the state-of-the-art OWL framework to obtain its Pass@1 performance for comparison. Additionally, we evaluated our proposed open-source framework, OAgents, under the Pass@3 setting."

    By the paper's own definition, Pass@3 is 'at least one correct solution found among 3 independent attempts,' so for any model Pass@3 is monotonically no smaller than Pass@1. The paper computes Pass@3 only for OAgents, while all compared baselines are Pass@1 numbers, and Section 5 explicitly confirms this asymmetry. Therefore the claim that OAgents-Pass@3 'outperforms all other frameworks' is not a measured superiority but an artifact of comparing an inflated metric to a stricter one; the margin is built into the metric definition rather than established by the experiment.

  2. fitted input called prediction [Section 4.2.2, Table 7; Appendix A.4 and B.1]
    "The Plan tips are summarized from analysis of historical error logs and incorporate heuristic knowledge gained from past failures. They contribute to a 14.54% performance improvement, proving that leveraging prior experience helps prevent errors and build more robust plans."

    Plan tips are constructed by analyzing error logs from GAIA tasks, i.e., from the very benchmark on which the 14.54% gain is then measured. This is fitting a prompt-level heuristic to a dataset and then reporting its performance on that same dataset as evidence of effectiveness. The stated improvement is an in-sample fit to GAIA failure patterns, not an out-of-sample prediction; without a held-out split or a separate benchmark, the gain shows that the tips encode GAIA-specific answer cues rather than that the planning method generalizes.

1 more flagged steps
  1. fitted input called prediction [Section 1 and Sections 4.2.1–4.2.2, Tables 2–7]
    "Based on our findings, we build and open-source OAgents, a new foundation agent framework that achieves state-of-the-art performance among open-source projects."

    OAgents is assembled from components selected on the same GAIA validation set: Tables 2–7 choose Jina reader, k=5 multi-source retrieval, query optimization, plan tips, memory configurations, and TTS strategies by comparing their GAIA scores, and Plan tips are derived from GAIA error logs. The final configuration is then scored on the same 165 questions to produce the 73.93 headline number in Table 1. No held-out split or significance test is reported, so the 'state-of-the-art' score is the value of the fitted configuration on the fitting set. The claim reduces to 'the GAIA-maximizing configuration scores highest on GAIA,' which is an in-sample selection result rather than an independent validation.

full rationale

The paper is a benchmark-driven empirical study, not a formal derivation, so the circularity is of the empirical kind: components and prompts are selected by observing GAIA scores, and then those same scores are presented as evidence of superiority. The most concrete instance is Plan tips, which are explicitly 'summarized from analysis of historical error logs' and then credited with a 14.54% improvement on the same GAIA set. A second, definitional issue is the Pass@3 versus Pass@1 comparison: since Pass@3 is at least one success in three tries, it is larger than Pass@1 by construction, and the paper only computes Pass@3 for OAgents. Together, these make the headline 'outperforming all other frameworks' partially circular—the margin is either in-sample fit or metric inflation. There is no load-bearing self-citation chain here; the paper's own equations are not used to derive the result, and the ablations on BrowseComp and the model-family comparisons retain independent content. The score is 6 because the central SOTA claim is substantially weakened by same-benchmark selection and metric mismatch, while the underlying component studies still report real, if in-sample, measurements.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central empirical claims rest on benchmark-specific tuning and unquantified variance; no new physical or mathematical entities are introduced.

free parameters (2)
  • Plan tips heuristics (H) = hand-written from GAIA error logs
    Used to improve planning; derived from the same benchmark used for evaluation, so the improvement is not independently validated.
  • Configuration choices (Jina reader, multi-source k=5) = selected by GAIA scores
    Tables 3-4 choose the best browser and search configuration based on GAIA results; this is a model selection step on the evaluation set.
assumptions (3)
  • domain assumption GAIA validation set accuracy at 1-3 point deltas reflects real differences
    The paper treats small score gaps as meaningful without error bars or significance tests.
  • domain assumption Re-answering on empty or 'Unable to determine' outputs is a valid evaluation protocol
    Section 4.1 allows retries under these conditions, which can inflate Pass@1.
  • domain assumption The reported scores of other frameworks are comparable despite mixed Pass@K metrics
    Table 1 mixes Pass@1, Pass@3 and unknown metrics; Section 5 acknowledges this but still uses the numbers for ranking claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OAgents: An Empirical Study of Building Effective Agents." pith.science (2026). https://pith.science/paper/3S3JRX5Q

@misc{pith2026250615741,
  author       = {Pith},
  title        = {Pith review of: OAgents: An Empirical Study of Building Effective Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3S3JRX5Q}},
  note         = {Machine review of arXiv:2506.15741}
}
read the original abstract

Recently, Agentic AI has become an increasingly popular research field. However, we argue that current agent research practices lack standardization and scientific rigor, making it hard to conduct fair comparisons among methods. As a result, it is still unclear how different design choices in agent frameworks affect effectiveness, and measuring their progress remains challenging. In this work, we conduct a systematic empirical study on GAIA benchmark and BrowseComp to examine the impact of popular design choices in key agent components in a fair and rigorous manner. We find that the lack of a standard evaluation protocol makes previous works, even open-sourced ones, non-reproducible, with significant variance between random runs. Therefore, we introduce a more robust evaluation protocol to stabilize comparisons. Our study reveals which components and designs are crucial for effective agents, while others are redundant, despite seeming logical. Based on our findings, we build and open-source OAgents, a new foundation agent framework that achieves state-of-the-art performance among open-source projects. OAgents offers a modular design for various agent components, promoting future research in Agentic AI.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

    cs.SE 2026-07 unverdicted novelty 6.0 of 10

    A tool that injects faults into an LLM agent's MCP tool calls, replays them under mitigation, and verifies the fix—revealing that agents mostly fail silently on stale or bad data.

  2. ToolSelf: Unifying Task Execution and Self-Reconfiguration via Tool-Driven Emergent Adaptation

    cs.AI 2026-02 conditional novelty 6.0 of 10

    An LLM agent that can call a reconfiguration tool to update its sub-goals, toolbox, strategy, and context outperforms static-config agents across FRAMES, xbench, GAIA, and SWE-bench Lite.

  3. Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL

    cs.AI 2025-08 conditional novelty 6.0 of 10

    A single LLM is trained with multi-agent distilled trajectories plus agentic RL, and the resulting Chain-of-Agents models set state-of-the-art Pass@1 scores among tool-integrated reasoning methods on GAIA, BrowseComp,...

  4. WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization

    cs.CL 2025-07 conditional novelty 6.0 of 10

    WebShaper formalizes web information-seeking tasks as set-theoretic queries, synthesizes training questions by layer-wise expansion, and uses the resulting data to train open-source agents that outperform prior open-s...

  5. Deep Research Agents: A Systematic Examination And Roadmap

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A survey that organizes LLM-powered deep research agents into static versus dynamic workflows and single versus multi agent architectures, and reviews their benchmarks and open challenges.

Reference graph

Works this paper leans on

45 extracted references · 22 canonical work pages · cited by 5 Pith papers

  1. [10]

    Open deep research

    LangChain. Open deep research. [Online], 2024.https://github.com/langchain-ai/open_deep_research

    work page 2024

  2. [19]

    ‘smola- gents‘: a smol library to build great agentic systems.https://github.com/huggingface/smolagents, 2025

    Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. ‘smola- gents‘: a smol library to build great agentic systems.https://github.com/huggingface/smolagents, 2025. 15

    work page 2025

  3. [1]

    Desearch, 2024

    Desearch AI. Desearch, 2024. URLhttps://desearch.ai/

    work page 2024

  4. [2]

    Aworld: A unified agent playground for computer and phone use tasks, 2025

    Agent Team at Ant Group. Aworld: A unified agent playground for computer and phone use tasks, 2025. URL https://github.com/inclusionAI/AWorld

    work page 2025

  5. [3]

    Tapeagents: a holistic framework for agent development and optimization, 2024

    Dzmitry Bahdanau, Nicolas Gontier, Gabriel Huang, Ehsan Kamalloo, Rafael Pardinas, Alex Piché, Torsten Scholak, Oleh Shliazhko, Jordan Prince Tremblay, Karam Ghanem, Soham Parikh, Mitul Tiwari, and Quaizar Vohra. Tapeagents: a holistic framework for agent development and optimization, 2024. URLhttps://arxiv. org/abs/2412.08445

    arXiv 2024

  6. [4]

    Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308.10848, 2(4):6, 2023

    arXiv 2023

  7. [5]

    Magentic-one: A generalist multi-agent system for solving complex tasks.arXiv preprint arXiv:2411.04468, 2024

    Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. Magentic-one: A generalist multi-agent system for solving complex tasks.arXiv preprint arXiv:2411.04468, 2024

    arXiv 2024

  8. [6]

    Griffiths, and Mengdi Wang

    Xudong Guo, Kaixuan Huang, Jiale Liu, Wenhui Fan, Natalia Vélez, Qingyun Wu, Huazheng Wang, Thomas L. Griffiths, and Mengdi Wang. Embodied LLM agents learn to cooperate in organized teams. In Language Gamification - NeurIPS 2024 Workshop, 2024. URLhttps://openreview.net/forum?id=VKlrzygQlT

    work page 2024

Show all 45 references
  1. [7]

    Autonomous agentic ai: execute multi-step workflows autonomously

    H2O.ai. Autonomous agentic ai: execute multi-step workflows autonomously. [Online], 2024.https://h2o.ai/ platform/enterprise-h2ogpte/#AgenticAI

    2024

  2. [8]

    MetaGPT: Meta programming for a multi-agent collaborative framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, StevenKaShingYau, ZijuanLin, LiyangZhou, ChenyuRan, LingfengXiao, ChenglinWu, andJürgenSchmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. I...

    2024

  3. [9]

    Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025

    Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Ping Luo, and Guohao Li. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025. URL...

    2025

  4. [11]

    Camel: Communicative agents for" mind" exploration of large language model society.Advances in Neural Information Processing Systems, 36:51991–52008, 2023

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society.Advances in Neural Information Processing Systems, 36:51991–52008, 2023

    2023

  5. [12]

    From llm to conversational agent: A memory enhanced architecture with fine-tuning of large language models, 2024

    Na Liu, Liangyu Chen, Xiaoyu Tian, Wei Zou, Kaijiang Chen, and Ming Cui. From llm to conversational agent: A memory enhanced architecture with fine-tuning of large language models, 2024. URLhttps://arxiv.org/abs/ 2401.02777

    2024 arXiv

  6. [13]

    Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization.arXiv preprint arXiv:2310.02170, 2023

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization.arXiv preprint arXiv:2310.02170, 2023

    2023 arXiv

  7. [14]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InThe TwelfthInternational Conference on Learning Representations, 2023

    2023

  8. [15]

    deepresearch, 2024

    OpenAI. deepresearch, 2024. URL https://openai.com/index/introducing-deep-research/

    2024

  9. [16]

    Langfun, September 2023

    Daiyi Peng. Langfun, September 2023. URLhttps://github.com/google/langfun

    2023

  10. [17]

    Tool learning with foundation models.ACM Computing Surveys, 57(4):1–40, 2024

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang, Chaojun Xiao, et al. Tool learning with foundation models.ACM Computing Surveys, 57(4):1–40, 2024

    2024

  11. [18]

    Alita: Generalist agent enabling scalable agentic reasoning with minimal predefinition and maximal self-evolution

    Jiahao Qiu, Xuan Qi, Tongcheng Zhang, Xinzhe Juan, Jiacheng Guo, Yifu Lu, Yimin Wang, Zixin Yao, Qihan Ren, Xun Jiang, et al. Alita: Generalist agent enabling scalable agentic reasoning with minimal predefinition and maximal self-evolution. arXiv preprint arXiv:2505.20286, 2025

    2025 arXiv

  12. [20]

    Taskcraft: Automated generation of agentic tasks, 2025

    Dingfeng Shi, Jingyi Cao, Qianben Chen, Weichen Sun, Weizhen Li, Hongxuan Lu, Fangchen Dong, Tianrui Qin, King Zhu, Minghao Yang, Jian Yang, Ge Zhang, Jiaheng Liu, Changwang Zhang, Jun Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. Taskcraft: Automated generation of agentic...

    2025 arXiv

  13. [21]

    Reflexion: Language agents with verbal reinforcement learning, 2023

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URLhttps://arxiv.org/abs/2303.11366

    2023 arXiv

  14. [22]

    Significant-Gravitas. Autogpt. [Online], 2023. https://github.com/Significant-Gravitas/AutoGPT

    2023

  15. [23]

    Llm-planner: Few-shot grounded planning for embodied agents with large language models

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. InProceedings of the IEEE/CVF international conference on computer vision, pages 2998–3009, 2023

    2023

  16. [24]

    Autoagent: A fully-automated and zero-code framework for llm agents

    Jiabin Tang, Tianyu Fan, and Chao Huang. Autoagent: A fully-automated and zero-code framework for llm agents. arXiv e-prints, pages arXiv–2502, 2025

    2025

  17. [25]

    Mader: Trajectory planner in multiagent and dynamic environments

    Jesus Tordesillas and Jonathan P How. Mader: Trajectory planner in multiagent and dynamic environments. IEEE Transactions on Robotics, 38(1):463–476, 2021

    2021

  18. [26]

    Meet trase systems

    Trase. Meet trase systems. [Online], 2024.https://www.trasesystems.com/

    2024

  19. [27]

    Executable code actions elicit better llm agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. InForty-firstInternational Conference on Machine Learning, 2024

    2024

  20. [28]

    Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025

    2025 arXiv

  21. [29]

    Autogen: Enabling next-gen llm applications via multi-agent conversation.arXiv preprint arXiv:2308.08155, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation.arXiv preprint arXiv:2308.08155, 2023

    2023 arXiv

  22. [30]

    Os-copilot: Towards generalist computer agents with self-improvement.arXiv preprint arXiv:2402.07456, 2024

    Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement.arXiv preprint arXiv:2402.07456, 2024

    2024 arXiv

  23. [31]

    Openagents: An open platform for language agents in the wild

    Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, et al. Openagents: An open platform for language agents in the wild. arXiv preprint arXiv:2310.10634, 2023

    arXiv

  24. [32]

    A-mem: Agentic memory for llm agents

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110, 2025

    2025 arXiv

  25. [33]

    How2comm: Communication-efficient and collaboration-pragmatic multi-agent perception.Advances in Neural Information Processing Systems, 36:25151–25164, 2023

    Dingkang Yang, Kun Yang, Yuzheng Wang, Jing Liu, Zhi Xu, Rongbin Yin, Peng Zhai, and Lihua Zhang. How2comm: Communication-efficient and collaboration-pragmatic multi-agent perception.Advances in Neural Information Processing Systems, 36:25151–25164, 2023

    2023

  26. [34]

    What2comm: Towards communication-efficient collaborative perception via feature decoupling

    Kun Yang, Dingkang Yang, Jingyu Zhang, Hanqi Wang, Peng Sun, and Liang Song. What2comm: Towards communication-efficient collaborative perception via feature decoupling. In Proceedings of the 31st ACM international conference on multimedia, pages 7686–7695, 2023

    2023

  27. [35]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  28. [36]

    A survey on the memory mechanism of large language model based agents.arXiv preprint arXiv:2404.13501, 2024

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents.arXiv preprint arXiv:2404.13501, 2024

    2024 arXiv

  29. [37]

    Language agent tree search unifies reasoning acting and planning in language models, 2024

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2024. URLhttps://arxiv.org/abs/2310.04406

    2024 arXiv

  30. [38]

    Recurrentgpt: Interactive generation of (arbitrarily) long text, 2023

    Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. Recurrentgpt: Interactive generation of (arbitrarily) long text, 2023. URL https: //arxiv.org/abs/2305.13304. 16

    2023 arXiv

  31. [39]

    Agents: An open-source framework for autonomous language agents

    Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Xiangru Tang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An open-source framework fo...

  32. [40]

    http :// web . archive . org / cdx / search / cdx ? url ={ url }& output = json & from ={ timestamp }

    Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, Huajun Chen, and Yuchen Eleanor Jiang. Symbolic learning enables self-evolving agents. 2024. URL https://arxiv.org/abs/2406.18532. 17 Appendix A Det...

    2024 arXiv

  33. [42]

    ELSE if given task can be broken down into subtasks, you will: • Identify all subtasks needed to complete the overall task

    [Second step] ... ELSE if given task can be broken down into subtasks, you will: • Identify all subtasks needed to complete the overall task. • Provide a single PARALLEL-LIST that ONLY contains subtasks index joined by comma that can start immediately with NO dependencies on o...

  34. [43]

    **ST2:[Subtask Description]

    [Second Step] ... **ST2:[Subtask Description]

  35. [44]

    Wait for ST[X] to complete

    [First Step - If this subtask depends on another, explicitly state: "Wait for ST[X] to complete" as the first step]

  36. [45]

    Problems

    [Second Step] ... **STx:[xxx] ... Here is your task: {Task} List of facts that you know: {Answer_Facts} Please strictly follow the suggestions below: {Experience} Now begin! Write your plan below. 22 B.2 Search Agent Prompts Query-Reflection Prompt You are a highly skilled que...

  37. [2023]

    URL https://arxiv.org/abs/2309.07870

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.