Pith. sign in

REVIEW 5 major objections 3 minor 209 references

Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey

T0 review · 5 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims AI-agent benchmarks are best sorted by a two-axis taxonomy of external environments and internal capabilities.

desk verdict Useful taxonomy survey with a load-bearing weakness: the reference tables contain demonstrable citation errors that must be fixed before the paper is usable. read the letter →

arxiv 2506.11102 v1 pith:JQK5FKKV submitted 2025-06-06 cs.CL cs.AI

Jiachen Zhu , Menghui Zhu , Renting Rui , Rong Shan , Congmin Zheng , Bo Chen , Yunjia Xi , Jianghao Lin
show 4 more authors
Weiwen Liu Ruiming Tang Yong Yu Weinan Zhang
This is my paper
classification cs.CLcs.AI
keywords AIagentevaluationLLMchatbotsbenchmarktaxonomyenvironment-capabilityanalysisselectionevolutionaryperspectivecapabilitiessurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the messy field of LLM-based AI agent evaluation becomes navigable once it is viewed as an evolutionary continuation of LLM chatbots. It defines five dimensions along which agents surpass chatbots—complex environment, multi-source instructor, dynamic feedback, multimodal perception, and advanced capability—and then sorts existing benchmarks along two axes: the external environment the agent operates in (coding, web, OS, mobile, scientific, game) and the internal capability the benchmark tests (planning, self-reflection, interaction, memory, general). The payoff would be practical: a researcher with a new agent can consult the attribute tables and pick benchmarks matched to the agent's environment and capabilities instead of relying on ad hoc choices. The paper also outlines future evaluation trends and a two-stage benchmark-selection method.

What carries the argument

The machinery is the Environment×Capability taxonomy: a two-axis classification that assigns each benchmark a home by where the agent acts and what competence is probed. The paper supplements it with Figure 1 (the five-aspect chatbot-to-agent evolution) and Figure 3 (four evaluation-perspective trends), and with per-category attribute tables that list realism, interactivity, observation type, action space, metrics, and evaluator. The taxonomy does the argument's work: it converts 'which benchmark should I use?' into a lookup problem.

What would settle it

Audit every row of the appendix tables against the cited primary source: if a substantial share misstate the environment, metric, or evaluator—as the duplicate identifier shared by two benchmarks already suggests—the claim that the taxonomy yields reliable selection guidance is refuted. A second check is whether a 2025 agent benchmark fits into some environment-by-capability cell; if it cannot, the taxonomy's completeness claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that agent evaluation is not an unstructured list but a structured two-axis landscape driven by the same forces that produced agents from chatbots. External environments are the driving force; internal capabilities are the result; evaluation benchmarks capture both. The paper claims that every major existing benchmark can be placed into one of the six environment categories and one of the five capability categories, and that the attribute tables in Appendix A make the placement concrete enough to guide selection. The same evolutionary logic yields four forward-looking perspectives—environment, agent, evaluator, and metric—that point to where evaluation is heading.

Load-bearing premise

The taxonomy's practical guidance assumes every row of the benchmark tables correctly describes the cited paper, but the paper's reference list already mixes up two benchmarks under one identifier and points one entry to an unrelated exam-scoring paper, so the tables cannot be used without checking the sources.

Editorial extensions

If this is right

  • If the taxonomy is correct, a researcher can narrow benchmark candidates by first fixing the agent's operating environment (coding, web, OS, mobile, scientific, game) and then the capability it exercises (planning, self-reflection, interaction, memory, general).
  • The five-aspect framework gives a sharp test for whether a task actually requires an agent rather than a chatbot; static single-turn benchmarks belong to the chatbot era and should not be used to claim agent-level performance.
  • The attribute tables supply concrete comparison dimensions—realism, offline/online, observation type, action space, metric, evaluator—so benchmark selection becomes a matter of matching attributes rather than reputation.
  • The four trends predict the near-term shape of evaluation: multimodal inputs, evolving and stateful environments, multi-agent and multi-turn settings, agent judges, personalized tasks, and efficiency and social-good metrics.
  • The two-stage selection workflow lets developers evaluate an agent today against its current environment and capability cell and later re-check the agent against evolving benchmark dimensions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's qualitative categorization, one testable extension is to score every listed benchmark on the five aspects and check whether the resulting scores predict observed performance gaps across current agents.
  • The same taxonomy could be turned into a decision procedure: parse a developer's agent description into environment and capability tags, then rank candidate benchmarks by matching attributes—the paper provides the rows but not the ranking rule.
  • If evaluation keeps moving online and evolving, the attribute tables themselves become dated; an implicit consequence of the paper's 'static to evolving' trend is that benchmark attributes need versioning, since a snapshot-based description may mislead future users.
  • The five-aspect distinction suggests measuring 'agentness' as a spectrum rather than a binary, with benchmarks placed by how many of the five aspects they exercise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. This survey proposes an evolutionary perspective on evaluating LLM-based AI agents. It distinguishes AI agents from LLM chatbots along five dimensions—complex environment, multi-source instructor, dynamic feedback, multi-modal perception, and advanced capability—and organizes existing evaluation benchmarks along two axes: external environments (coding, web, OS, mobile, scientific, game) and internal capabilities (planning, self-reflection, interaction, memory, general). For each category it provides detailed attribute tables in Appendix A, intended as practical reference tables, and concludes with a two-stage benchmark selection methodology. The paper makes no empirical claims; its contribution is a taxonomy and a structured literature review.

Significance. If the attribute tables and citation mapping are reliable, the paper offers a practically useful organization of a rapidly growing and fragmented benchmark landscape. The five-aspect chatbot-to-agent distinction is a reasonable conceptual lens, and the two-axis taxonomy (environment/capability) is intuitive and likely helpful for researchers selecting benchmarks. The survey is broad in coverage and the appendix tables, when accurate, would be a valuable reference artifact. However, the paper's practical value depends entirely on the fidelity of its literature processing; the demonstrated reference errors directly undermine that value until corrected.

major comments (5)
  1. [§IV-C2 and Table X] References [141] and [145] are both given as arXiv:2406.12045, yet the text and Table X treat τ-bench and IntellAgent as distinct benchmarks with different characteristics. Since [141] correctly points to the τ-bench paper, [145] must be a distinct arXiv identifier for IntellAgent; as printed, a reader cannot locate the correct primary source for IntellAgent, and this error propagates into the practical reference table. Please correct the reference and verify all attributes attributed to each benchmark against the primary sources.
  2. [§IV-C3 and Table XI] SmartEval is described as a multi-game benchmark (Rock-Paper-Scissors, Tower of Hanoi, Minecraft) and is cited as [151], but reference [151] is 'SmartEval: Evaluation System for Descriptive Answers in Examinations Using IoT-Enabled Technologies and Artificial Intelligence' published in Sensors (2023), which is about examination-answer evaluation, not game tasks. This is a clear citation mismatch that invalidates the specific entry in Table XI and the surrounding textual claim. The authors must replace the reference with the actual multi-game SmartEval work or correct the description.
  3. [§IV-B and References] In Section IV-B, the text states that 'LLM-Evolve [122] targets the self-reflection ability of AI agents on standard benchmarks such as MMLU [194].' Reference [194] is 'Self-contrast: Better reflection through inconsistent solving perspectives' (arXiv:2401.02009), which is not the MMLU benchmark. This mis-citation appears in the discussion of a concrete benchmark used in an evaluation protocol; please replace it with the correct MMLU citation (e.g., Hendrycks et al., 2020) and audit all other benchmark-name citations in the same section.
  4. [§V-A and References] The text refers to 'the swe-bench [24] series,' but reference [24] is SWE-bench-java, a single benchmark, not the SWE-bench series. Additionally, in the same section, 'VideoWebArena [41]' is cited, but reference [41] is VisualWebArena; VideoWebArena is reference [48] as correctly cited earlier in Section III-B. These errors in high-level discussion reinforce the concern that citation-to-benchmark mappings have not been systematically checked. Please correct these references and ensure internal consistency.
  5. [Appendix A (Overall)] The appendix tables are the paper's central practical deliverable, yet at least three errors (above) have been found in their benchmark-to-reference mappings. Because the selection methodology in Section V-E directs readers to consult these tables for 'essential' attributes, an undetected error in any table row can directly mislead a researcher's benchmark choice. I ask the authors to verify every entry in every table against the primary source (title, arXiv ID, and key attributes), not just the flagged items, before a revised version is considered.
minor comments (3)
  1. [Throughout] There are numerous typos and formatting inconsistencies, including 'disucssions' and 'furhter' in the contributions list, 'Enviornments' in section headings, 'Evovable' in Figure 3, and 'Ecffective' in Section III-F. Please run a careful proofreading pass.
  2. [References] Several references are incomplete, such as [25] (no author or venue), [128] and [130] using only 'G. Team' or 'F. Team', and [169] mixing arXiv ID and URL formats. Please standardize the bibliography according to a single style and provide full author lists where available.
  3. [§V-A and Figure 3] The phrase 'VideoWebArena [41]' in Section V-A is incorrect, and the label 'Evovable' in Figure 3 should read 'Evolving'. These are presentation issues but should be fixed alongside the substantive citation corrections.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the survey organizes external benchmarks and makes no predictive or fitted claims.

full rationale

This is a survey paper with no fitted parameters, no empirical predictions, and no derivational chain of equations. The claimed contributions are an analytical framework distinguishing AI agents from LLM chatbots along five aspects, a two-axis categorization of existing benchmarks by environment and capability, and practical reference tables. None of these is derived from its own outputs: the framework is a proposed viewpoint stated by the authors, and the taxonomy and tables are organized descriptions of externally published benchmarks. The benchmark-selection methodology applies the taxonomy to hypothetical use cases rather than predicting benchmark properties from the taxonomy, so there is no reduction of a 'prediction' to an input. The skeptic's documented reference errors (e.g., references [141] and [145] sharing an arXiv ID, [151] pointing to an unrelated Sensors paper, and [194] being mis-cited for MMLU) are correctness and reliability issues in the literature-processing pipeline, not circularity. Even if the tables contain errors, the errors do not make the framework self-referential or make the survey's conclusions equivalent to its inputs. No load-bearing self-citation chain or imported-uniqueness argument is present, and the self-citations that might exist are not used to justify the central taxonomy. The derivation chain is therefore self-contained and non-circular, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

A survey has no free parameters or invented entities. Its epistemic weight rests on the accuracy of the cited benchmark descriptions and the completeness of the proposed taxonomy.

assumptions (2)
  • domain assumption The surveyed benchmark descriptions are accurate and representative of the field.
    The paper's contribution is the taxonomy and tables, which summarize primary sources; errors in these summaries directly undermine the framework.
  • domain assumption The five aspects (environment, instructor, feedback, perception, capability) are sufficient to distinguish AI agents from chatbots.
    Section II.B defines agents by these five dimensions without arguing that this set is complete or non-overlapping.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey." pith.science (2026). https://pith.science/paper/JQK5FKKV

@misc{pith2026250611102,
  author       = {Pith},
  title        = {Pith review of: Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQK5FKKV}},
  note         = {Machine review of arXiv:2506.11102}
}
read the original abstract

The advent of large language models (LLMs), such as GPT, Gemini, and DeepSeek, has significantly advanced natural language processing, giving rise to sophisticated chatbots capable of diverse language-related tasks. The transition from these traditional LLM chatbots to more advanced AI agents represents a pivotal evolutionary step. However, existing evaluation frameworks often blur the distinctions between LLM chatbots and AI agents, leading to confusion among researchers selecting appropriate benchmarks. To bridge this gap, this paper introduces a systematic analysis of current evaluation approaches, grounded in an evolutionary perspective. We provide a detailed analytical framework that clearly differentiates AI agents from LLM chatbots along five key aspects: complex environment, multi-source instructor, dynamic feedback, multi-modal perception, and advanced capability. Further, we categorize existing evaluation benchmarks based on external environments driving forces, and resulting advanced internal capabilities. For each category, we delineate relevant evaluation attributes, presented comprehensively in practical reference tables. Finally, we synthesize current trends and outline future evaluation methodologies through four critical lenses: environment, agent, evaluator, and metrics. Our findings offer actionable guidance for researchers, facilitating the informed selection and application of benchmarks in AI agent evaluation, thus fostering continued advancement in this rapidly evolving research domain.

Figures

Figures reproduced from arXiv: 2506.11102 by the authors.

Figure 1
Figure 1. Evolution between AI agents and LLM chatbots in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall structure of this paper. Multimodal Perception. To function in real-world settings and respond to complex instructions, AI agents are equipped with multimodal sensing—processing not just text, but also visual, auditory, and even tactile or environmental sensor data. The development of Multimodal Large Language Models (MLLMs) exemplifies this leap, allowing agents to understand and reason across diverse m… view at source ↗
Figure 3
Figure 3. Four perspectives of evolution prospect about AI agent evaluation. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

209 extracted references · 6 canonical work pages

  1. [141]

    Available: https://arxiv.org/abs/2406

    [Online]. Available: https://arxiv.org/abs/2406. 18518

  2. [145]

    Intellagent: A benchmark for evaluating tool-augmented agents in real-world domains,

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “Intellagent: A benchmark for evaluating tool-augmented agents in real-world domains,” 2024. [Online]. Available: https://arxiv.org/abs/2406.12045

    arXiv 2024

  3. [151]

    Smarteval: Evaluation system for descriptive answers in examinations using iot- enabled technologies and artificial intelligence,

    M. Alahiet al., “Smarteval: Evaluation system for descriptive answers in examinations using iot- enabled technologies and artificial intelligence,”Sensors, vol. 23, no. 11, p. 5206, 2023. [Online]. Available: https://doi.org/10.3390/s23115206

    work page doi:10.3390/s23115206 2023

  4. [194]

    P-folio: Evalu- ating and improving logical reasoning with abun- dant human-written reasoning chains,

    S. Han, A. Yu, R. Shenet al., “P-folio: Evalu- ating and improving logical reasoning with abun- dant human-written reasoning chains,”arXiv preprint arXiv:2410.09207, 2024

    arXiv 2024

  5. [41]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks,

    J. Y . Koh, R. Loet al., “Visualwebarena: Evaluating multimodal agents on realistic visual web tasks,” 2024. [Online]. Available: https://arxiv.org/abs/2401.13649

    arXiv 2024

  6. [24]

    Swe-bench-java: A github issue resolving benchmark for java, 2024,

    D. Zan, Z. Huanget al., “Swe-bench-java: A github issue resolving benchmark for java, 2024,”URL https://arxiv. org/abs/2408.14354

    arXiv 2024

  7. [122]

    Realm-bench: A real-world planning benchmark for llms and multi-agent systems,

    L. Geng and E. Y . Chang, “Realm-bench: A real-world planning benchmark for llms and multi-agent systems,” arXiv preprint arXiv:2502.18836, 2025

    arXiv 2025

  8. [48]

    Videowebarena: Evaluating long context multimodal agents with video understanding web tasks,

    L. Jang, Y . Liet al., “Videowebarena: Evaluating long context multimodal agents with video understanding web tasks,” 2025. [Online]. Available: https://arxiv.org/ abs/2410.19100

    arXiv 2025

Show all 209 references
  1. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeeret al., “Attention is all you need,” inAdvances in Neural Information Processing Systems, 2017, pp. 5998–6008. [Online]. Available: http://arxiv.org/abs/1706.03762

    2017 arXiv

  2. [2]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023. [Online]. Available: https: //doi.org/10.48550/arXiv.2303.08774

    2023 doi arXiv

  3. [3]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavrilet al., “Llama: Open and efficient foundation language models,”CoRR, vol. abs/2302.13971, 2023. [Online]. Available: https: //doi.org/10.48550/arXiv.2302.13971

    2023 doi arXiv

  4. [4]

    Gemini: A family of highly capable multimodal models,

    R. Anil, S. Borgeaudet al., “Gemini: A family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023. [Online]. Available: https: //arxiv.org/abs/2312.11805

    2023 arXiv

  5. [5]

    Qwen technical report,

    Y . Bai, Y . Linet al., “Qwen technical report,”arXiv preprint arXiv:2309.16609, 2023. [Online]. Available: https://arxiv.org/abs/2309.16609

    2023 arXiv

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yanget al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025. [Online]. Available: https://arxiv.org/abs/2501.12948

    2025 arXiv

  7. [7]

    Agent-e: From autonomous web navigation to foundational design principles in agentic systems,

    T. Abuelsaad, D. Akkilet al., “Agent-e: From autonomous web navigation to foundational design principles in agentic systems,”arXiv preprint arXiv:2407.13032, 2024. [Online]. Available: https://arxiv.org/abs/2407.13032

    2024 arXiv

  8. [8]

    Beyond browsing: Api-based web agents,

    Y . Song, F. Xuet al., “Beyond browsing: Api-based web agents,”arXiv preprint arXiv:2410.16464, 2024. [Online]. Available: https://arxiv.org/abs/2410.16464

    2024 arXiv

  9. [9]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassanoet al., “Reflexion: Language agents with verbal reinforcement learning,”arXiv preprint arXiv:2303.11366, 2023. [Online]. Available: https://arxiv.org/abs/2303.11366

    2023 arXiv

  10. [10]

    A survey on evaluation of large language models,

    Y . Chang, X. Wanget al., “A survey on evaluation of large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2307.03109

    2023 arXiv

  11. [11]

    A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations,

    M. T. R. Laskar, S. Alqahtaniet al., “A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations,” 2024. [Online]. Available: https://arxiv.org/abs/2407.04069

    2024 arXiv

  12. [12]

    Evaluating large language models: A comprehensive survey,

    Z. Guo, R. Jinet al., “Evaluating large language models: A comprehensive survey,” 2023. [Online]. Available: https://arxiv.org/abs/2310.19736

    2023 arXiv

  13. [13]

    Evaluating large lan- guage models trained on code,

    M. Chen, J. Tworeket al., “Evaluating large lan- guage models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  14. [14]

    Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation,

    X. Du, M. Liuet al., “Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation,”arXiv preprint arXiv:2308.01861, 2023

    2023 arXiv

  15. [15]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shenet al., “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” inProceedings of the 46th IEEE/ACM Inter- national Conference on Software Engineering, 2024, pp. 1–12

    2024

  16. [16]

    Evaluating instruction-tuned large language models on code comprehension and generation,

    Z. Yuan, J. Liuet al., “Evaluating instruction-tuned large language models on code comprehension and generation,” arXiv preprint arXiv:2308.01240, 2023

    2023 arXiv

  17. [17]

    Codecriticbench: A holistic code critique benchmark for large language models,

    A. Zhang, M. Donget al., “Codecriticbench: A holistic code critique benchmark for large language models,” arXiv preprint arXiv:2502.16614, 2025

    2025 arXiv

  18. [18]

    Cruxeval: A benchmark for code reasoning, understanding and execution,

    A. Gu, B. Rozièreet al., “Cruxeval: A benchmark for code reasoning, understanding and execution,”arXiv preprint arXiv:2401.03065, 2024

    2024 arXiv

  19. [19]

    Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions,

    T. Y . Zhuo, M. C. Vuet al., “Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions,”arXiv preprint arXiv:2406.15877, 2024

    2024 arXiv

  20. [20]

    Natural language to code generation in interactive data science notebooks,

    P. Yin, W.-D. Liet al., “Natural language to code generation in interactive data science notebooks,”arXiv preprint arXiv:2212.09248, 2022

    2022 arXiv

  21. [21]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yanget al., “Swe-bench: Can language models resolve real-world github issues?”arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  22. [22]

    Swe-bench multimodal: Do ai systems generalize to visual software domains?

    J. Yang, C. E. Jimenezet al., “Swe-bench multimodal: Do ai systems generalize to visual software domains?” arXiv preprint arXiv:2410.03859, 2024

    2024 arXiv

  23. [23]

    Swe- search: Enhancing software agents with monte carlo tree search and iterative refinement,

    A. Antoniades, A. Örwall, K. Zhanget al., “Swe- search: Enhancing software agents with monte carlo tree search and iterative refinement,”arXiv preprint arXiv:2410.20285, 2024

    2024 arXiv

  24. [25]

    Swe arena: An open evaluation platform for automated software engineering,

    “Swe arena: An open evaluation platform for automated software engineering,” 2024

    2024

  25. [26]

    Large language model critics for execution-free evaluation of code changes,

    A. Yadavally, H. Nguyenet al., “Large language model critics for execution-free evaluation of code changes,” arXiv preprint arXiv:2501.16655, 2025

    2025 arXiv

  26. [27]

    Swe-bench+: En- hanced coding benchmark for llms,

    R. Aleithan, H. Xue, and et al., “Swe-bench+: En- hanced coding benchmark for llms,”arXiv preprint arXiv:2410.06992, 2024

    2024 arXiv

  27. [28]

    Repobench: Bench- marking repository-level code auto-completion systems,

    T. Liu, C. Xu, and J. McAuley, “Repobench: Bench- marking repository-level code auto-completion systems,” arXiv preprint arXiv:2306.03091, 2023

    2023 arXiv

  28. [29]

    Swt- bench: Testing and validating real-world bug-fixes with JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 15 code agents,

    N. Mündler, M. Müller, J. He, and M. Vechev, “Swt- bench: Testing and validating real-world bug-fixes with JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 15 code agents,”Advances in Neural Information Process- ing Systems, vol. 37, pp. 81 857–81 887, 2024

    2015

  29. [30]

    Chatdev: Communicative agents for software development,

    C. Qian, W. Liuet al., “Chatdev: Communicative agents for software development,”arXiv preprint arXiv:2307.07924, 2023

    2023 arXiv

  30. [31]

    Prompting large language models to tackle the full software development lifecycle: A case study,

    B. Li, W. Wuet al., “Prompting large language models to tackle the full software development lifecycle: A case study,”arXiv preprint arXiv:2403.08604, 2024

    2024 arXiv

  31. [32]

    Ml-bench: Evaluating large lan- guage models and agents for machine learning tasks on repository-level code,

    X. Tang, Y . Liuet al., “Ml-bench: Evaluating large lan- guage models and agents for machine learning tasks on repository-level code,”arXiv preprint arXiv:2311.09835, 2023

    2023 arXiv

  32. [33]

    Pybench: Evaluating llm agent on various real-world coding tasks,

    Y . Zhang, Y . Pan, Y . Wang, and J. Cai, “Pybench: Evaluating llm agent on various real-world coding tasks,” 2024. [Online]. Available: https://arxiv.org/abs/ 2407.16732

    2024 arXiv

  33. [34]

    World of bits: An open- domain platform for web-based agents,

    T. Shi, A. Karpathyet al., “World of bits: An open- domain platform for web-based agents,” inProceedings of the 34th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, D. Precup and Y . W. Teh, Eds., vol. 70. PMLR, 06–11 Aug 2017, pp. ...

    2017

  34. [35]

    Reinforcement learning on web interfaces using workflow-guided exploration,

    E. Z. Liu, K. Guuet al., “Reinforcement learning on web interfaces using workflow-guided exploration,” 2018. [Online]. Available: https://arxiv.org/abs/1802.08802

    2018 arXiv

  35. [36]

    Webshop: Towards scalable real-world web interaction with grounded language agents,

    S. Yao, H. Chenet al., “Webshop: Towards scalable real-world web interaction with grounded language agents,” 2023. [Online]. Available: https://arxiv.org/abs/ 2207.01206

    2023 arXiv

  36. [37]

    Mind2web: Towards a generalist agent for the web,

    X. Deng, Y . Guet al., “Mind2web: Towards a generalist agent for the web,” 2023. [Online]. Available: https://arxiv.org/abs/2306.06070

    2023 arXiv

  37. [38]

    Webvoyager: Building an end- to-end web agent with large multimodal models,

    H. He, W. Yaoet al., “Webvoyager: Building an end- to-end web agent with large multimodal models,”arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  38. [39]

    Weblinx: Real-world website navigation with multi-turn dialogue,

    X. H. Lù, Z. Kasner, and S. Reddy, “Weblinx: Real-world website navigation with multi-turn dialogue,”arXiv preprint arXiv:2402.05930, 2024

    2024

  39. [40]

    Webarena: A realistic web environment for building autonomous agents,

    S. Zhou, F. F. Xuet al., “Webarena: A realistic web environment for building autonomous agents,” 2024. [Online]. Available: https://arxiv.org/abs/2307.13854

    2024 arXiv

  40. [42]

    Workarena: How capable are web agents at solving common knowledge work tasks?

    A. Drouin, M. Gasseet al., “Workarena: How capable are web agents at solving common knowledge work tasks?” 2024. [Online]. Available: https://arxiv.org/abs/ 2403.07718

    2024 arXiv

  41. [43]

    Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks,

    L. Boisvert, M. Thakkaret al., “Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks,” 2025. [Online]. Available: https://arxiv.org/abs/2407.05291

    2025 arXiv

  42. [44]

    Mmina: Benchmarking multihop multimodal internet agents,

    Z. Zhang, S. Tian, L. Chen, and Z. Liu, “Mmina: Benchmarking multihop multimodal internet agents,”

  43. [45]

    Assistantbench: Can web agents solve realistic and time-consuming tasks?

    O. Yoran, S. J. Amouyalet al., “Assistantbench: Can web agents solve realistic and time-consuming tasks?” 2024. [Online]. Available: https://arxiv.org/abs/2407.15711

    2024 arXiv

  44. [46]

    Webcanvas: Benchmarking web agents in online environments,

    Y . Pan, D. Konget al., “Webcanvas: Benchmarking web agents in online environments,” 2024. [Online]. Available: https://arxiv.org/abs/2406.12373

    2024 arXiv

  45. [47]

    St-webagentbench: A benchmark for evaluating safety and trustworthiness in web agents,

    I. Levy, B. Wieselet al., “St-webagentbench: A benchmark for evaluating safety and trustworthiness in web agents,” 2024. [Online]. Available: https: //arxiv.org/abs/2410.06703

    2024 arXiv

  46. [49]

    Tur[k]ingbench: A challenge benchmark for web agents,

    K. Xu, Y . Kordiet al., “Tur[k]ingbench: A challenge benchmark for web agents,” 2025. [Online]. Available: https://arxiv.org/abs/2403.11905

    2025 arXiv

  47. [50]

    Bearcubs: A benchmark for computer-using web agents,

    Y . Song, K. Thaiet al., “Bearcubs: A benchmark for computer-using web agents,” 2025. [Online]. Available: https://arxiv.org/abs/2503.07919

    2025 arXiv

  48. [51]

    Theagentcompany: Benchmarking llm agents on consequential real world tasks,

    F. F. Xu, Y . Songet al., “Theagentcompany: Benchmarking llm agents on consequential real world tasks,” 2024. [Online]. Available: https://arxiv.org/abs/ 2412.14161

    2024 arXiv

  49. [52]

    Waber: Evaluating reliability and efficiency of web agents with existing benchmarks,

    S. Kara, F. Faisal, and S. Nath, “Waber: Evaluating reliability and efficiency of web agents with existing benchmarks,” inICLR 2025 Workshop on Foundation Models in the Wild

    2025

  50. [53]

    Visualagentbench: Towards large multimodal models as visual foundation agents,

    X. Liu, T. Zhanget al., “Visualagentbench: Towards large multimodal models as visual foundation agents,”

  51. [54]

    An illusion of progress? assessing the current state of web agents,

    T. Xue, W. Qiet al., “An illusion of progress? assessing the current state of web agents,” 2025. [Online]. Available: https://arxiv.org/abs/2504.01382

    2025

  52. [55]

    Available: https://arxiv.org/abs/2408

    [Online]. Available: https://arxiv.org/abs/2408. 06327

  53. [56]

    Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web,

    R. Kapoor, Y . P. Butalaet al., “Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web,” 2024. [Online]. Available: https://arxiv.org/abs/2402.17553

    2024 arXiv

  54. [57]

    Real: Benchmarking autonomous agents on deterministic simulations of real websites,

    D. Garg, S. VanWeeldenet al., “Real: Benchmarking autonomous agents on deterministic simulations of real websites,” 2025. [Online]. Available: https: //arxiv.org/abs/2504.11543

    2025 arXiv

  55. [58]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real com- puter environments,

    T. Xie, D. Zhanget al., “Osworld: Benchmarking multimodal agents for open-ended tasks in real com- puter environments,”Advances in Neural Information Processing Systems, vol. 37, pp. 52 040–52 094, 2024

    2024

  56. [59]

    Chatshop: Interactive information seeking with language agents,

    S. Chen, S. Wiseman, and B. Dhingra, “Chatshop: Interactive information seeking with language agents,”

  57. [60]

    Available: https://arxiv.org/abs/2404

    [Online]. Available: https://arxiv.org/abs/2404. 09911

  58. [61]

    Omniact: A dataset and benchmark for enabling multimodal generalist JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 16 autonomous agents for desktop and web,

    R. Kapoor, Y . P. Butalaet al., “Omniact: A dataset and benchmark for enabling multimodal generalist JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 16 autonomous agents for desktop and web,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 161–178

    2015

  59. [62]

    Windows agent arena: Eval- uating multi-modal os agents at scale,

    R. Bonatti, D. Zhaoet al., “Windows agent arena: Eval- uating multi-modal os agents at scale,”arXiv preprint arXiv:2409.08264, 2024

    2024 arXiv

  60. [63]

    Agentstudio: A toolkit for building general virtual agents,

    L. Zheng, Z. Huanget al., “Agentstudio: A toolkit for building general virtual agents,”arXiv preprint arXiv:2403.17918, 2024

    2024 arXiv

  61. [64]

    Mapping natural language instruc- tions to mobile ui action sequences,

    Y . Li, J. Heet al., “Mapping natural language instruc- tions to mobile ui action sequences,”arXiv preprint arXiv:2005.03776, 2020

    2005 arXiv

  62. [65]

    Officebench: Benchmarking language agents across multiple applications for office automation,

    Z. Wang, Y . Cuiet al., “Officebench: Benchmarking language agents across multiple applications for office automation,”arXiv preprint arXiv:2407.19056, 2024

    2024 arXiv

  63. [66]

    Pc-agent: A hierarchical multi-agent collaboration framework for complex task automation on pc,

    H. Liu, X. Zhanget al., “Pc-agent: A hierarchical multi-agent collaboration framework for complex task automation on pc,”arXiv preprint arXiv:2502.14282, 2025

    2025 arXiv

  64. [67]

    Androidinthewild: A large-scale dataset for android device control,

    C. Rawles, A. Liet al., “Androidinthewild: A large-scale dataset for android device control,”Advances in Neural Information Processing Systems, vol. 36, pp. 59 708– 59 728, 2023

    2023

  65. [68]

    Ugif: Ui grounded instruction following,

    S. G. Venkatesh, P. Talukdar, and S. Narayanan, “Ugif: Ui grounded instruction following,”arXiv preprint arXiv:2211.07615, 2022

    2022 arXiv

  66. [69]

    Mobile app tasks with iterative feedback (motif): Addressing task feasibil- ity in interactive visual environments,

    A. Burns, D. Arsanet al., “Mobile app tasks with iterative feedback (motif): Addressing task feasibil- ity in interactive visual environments,”arXiv preprint arXiv:2104.08560, 2021

    2021 arXiv

  67. [70]

    Benchmarking mobile device control agents across diverse configurations,

    J. Lee, T. Minet al., “Benchmarking mobile device control agents across diverse configurations,”arXiv preprint arXiv:2404.16660, 2024

    2024 arXiv

  68. [71]

    On the effects of data scale on ui control agents,

    W. Li, W. E. Bishopet al., “On the effects of data scale on ui control agents,”Advances in Neural Information Processing Systems, vol. 37, pp. 92 130–92 154, 2024

    2024

  69. [72]

    Amex: Android multi- annotation expo dataset for mobile gui agents,

    Y . Chai, S. Huanget al., “Amex: Android multi- annotation expo dataset for mobile gui agents,”arXiv preprint arXiv:2407.17490, 2024

    2024 arXiv

  70. [73]

    A3: Android agent arena for mobile gui agents,

    Y . Chai, H. Liet al., “A3: Android agent arena for mobile gui agents,”arXiv preprint arXiv:2501.01149, 2025

    2025

  71. [74]

    Androidworld: A dynamic benchmarking environment for autonomous agents,

    C. Rawles, S. Clinckemaillieet al., “Androidworld: A dynamic benchmarking environment for autonomous agents,”arXiv preprint arXiv:2405.14573, 2024

    2024 arXiv

  72. [75]

    Mobile-bench: An evaluation benchmark for llm-based mobile agents,

    S. Deng, W. Xuet al., “Mobile-bench: An evaluation benchmark for llm-based mobile agents,”arXiv preprint arXiv:2407.00993, 2024

    2024 arXiv

  73. [76]

    Mobilesafetybench: Evaluating safety of autonomous agents in mobile device control,

    J. Lee, D. Hahmet al., “Mobilesafetybench: Evaluating safety of autonomous agents in mobile device control,” arXiv preprint arXiv:2410.17520, 2024

    2024

  74. [77]

    Mobile-env: Building qualified evaluation benchmarks for llm-gui interaction,

    D. Zhang, Z. Shenet al., “Mobile-env: Building qualified evaluation benchmarks for llm-gui interaction,”arXiv preprint arXiv:2305.08144, 2023

    2023 arXiv

  75. [78]

    Understanding the weakness of large language model agents within a complex android environment,

    M. Xing, R. Zhanget al., “Understanding the weakness of large language model agents within a complex android environment,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 6061–6072

    2024

  76. [79]

    Llamatouch: A faithful and scalable testbed for mobile ui task automation,

    L. Zhang, S. Wanget al., “Llamatouch: A faithful and scalable testbed for mobile ui task automation,” in Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, 2024, pp. 1–13

    2024

  77. [80]

    Mobileagentbench: An efficient and user-friendly benchmark for mobile llm agents,

    L. Wang, Y . Denget al., “Mobileagentbench: An efficient and user-friendly benchmark for mobile llm agents,” arXiv preprint arXiv:2406.08184, 2024

    2024 arXiv

  78. [81]

    Spa-bench: A comprehensive benchmark for smartphone agent evaluation,

    J. Chen, D. Yuenet al., “Spa-bench: A comprehensive benchmark for smartphone agent evaluation,” inNeurIPS 2024 Workshop on Open-World Agents, 2024

    2024

  79. [82]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clark, I. Cowheyet al., “Think you have solved question answering? try arc, the ai2 reasoning challenge,” arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  80. [83]

    Androidlab: Training and sys- tematic benchmarking of android autonomous agents,

    Y . Xu, X. Liuet al., “Androidlab: Training and sys- tematic benchmarking of android autonomous agents,” arXiv preprint arXiv:2410.24024, 2024

    2024 arXiv

  81. [84]

    Autoeval: A practical framework for autonomous evaluation of mobile agents,

    J. Sun, Z. Hua, and Y . Xia, “Autoeval: A practical framework for autonomous evaluation of mobile agents,” arXiv preprint arXiv:2503.02403, 2025

    2025

  82. [85]

    Qasa: advanced question answer- ing on scientific articles,

    Y . Lee, K. Leeet al., “Qasa: advanced question answer- ing on scientific articles,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 19 036–19 052

    2023

  83. [86]

    Learn to explain: Multimodal reasoning via thought chains for science question answering,

    P. Lu, S. Mishraet al., “Learn to explain: Multimodal reasoning via thought chains for science question answering,” 2022. [Online]. Available: https://arxiv.org/ abs/2209.09513

    2022 arXiv

  84. [87]

    A dataset of information-seeking questions and answers anchored in research papers,

    P. Dasigi, K. Loet al., “A dataset of information-seeking questions and answers anchored in research papers,” arXiv preprint arXiv:2105.03011, 2021

    2021 arXiv

  85. [88]

    Ms2: Multi-document summarization of medical studies,

    J. DeYoung, I. Beltagyet al., “Ms2: Multi-document summarization of medical studies,”arXiv preprint arXiv:2104.06486, 2021

    2021 arXiv

  86. [89]

    Auto- mated focused feedback generation for scientific writing assistance,

    E. Chamoun, M. Schlichktrull, and A. Vlachos, “Auto- mated focused feedback generation for scientific writing assistance,”arXiv preprint arXiv:2405.20477, 2024

    2024 arXiv

  87. [90]

    Aaar-1.0: Assessing ai’s potential to assist research,

    R. Lou, H. Xuet al., “Aaar-1.0: Assessing ai’s potential to assist research,”arXiv preprint arXiv:2410.22394, 2024

    2024 arXiv

  88. [91]

    Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers,

    C. Si, D. Yang, and T. Hashimoto, “Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers,”arXiv preprint arXiv:2409.04109, 2024

    2024 arXiv

  89. [92]

    Lab-bench: Measur- ing capabilities of language models for biology research,

    J. M. Laurent, J. D. Janizeket al., “Lab-bench: Measur- ing capabilities of language models for biology research,” arXiv preprint arXiv:2407.10362, 2024

    2024 arXiv

  90. [93]

    Researcharena: Benchmarking llms’ ability to collect and organize information as research agents,

    H. Kang and C. Xiong, “Researcharena: Benchmarking llms’ ability to collect and organize information as research agents,”arXiv preprint arXiv:2406.10291, 2024

    2024 arXiv

  91. [94]

    Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents, 2024,

    P. Jansen, M.-A. Côtéet al., “Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents, 2024,”URL https://arxiv. org/abs/2406.06769

    2024 arXiv

  92. [95]

    Paperqa: Retrieval- augmented generative agent for scientific research,

    J. Lála, O. O’Donoghueet al., “Paperqa: Retrieval- augmented generative agent for scientific research,” arXiv preprint arXiv:2312.07559, 2023

    2023 arXiv

  93. [96]

    Scienceworld: Is your agent smarter than a 5th grader?, 2022,

    R. Wang, P. Jansenet al., “Scienceworld: Is your agent smarter than a 5th grader?, 2022,”URL https://arxiv. org/abs/2203.07540, 2022

    2022 arXiv

  94. [97]

    Mlagent- bench: Evaluating language agents on machine learning experimentation, 2024,

    Q. Huang, J. V ora, P. Liang, and J. Leskovec, “Mlagent- bench: Evaluating language agents on machine learning experimentation, 2024,”URL https://arxiv. org/abs/2310, vol. 3302

    2024

  95. [98]

    Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery,

    Z. Chen, S. Chenet al., “Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery,”arXiv preprint arXiv:2410.05080, 2024

    2024 arXiv

  96. [99]

    Mlgym: A new framework JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 17 and benchmark for advancing ai research agents,

    D. Nathani, L. Madaanet al., “Mlgym: A new framework JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 17 and benchmark for advancing ai research agents,”arXiv preprint arXiv:2502.14499, 2025

    2015 arXiv

  97. [100]

    Da-code: Agent data science code generation benchmark for large language models,

    Y . Huang, J. Luoet al., “Da-code: Agent data science code generation benchmark for large language models,” arXiv preprint arXiv:2410.07331, 2024

    2024 arXiv

  98. [101]

    Dsbench: How far are data science agents to becoming data science experts?

    L. Jing, Z. Huanget al., “Dsbench: How far are data science agents to becoming data science experts?”arXiv preprint arXiv:2409.07703, 2024

    2024 arXiv

  99. [102]

    Benchmarking data science agents,

    Y . Zhang, Q. Jianget al., “Benchmarking data science agents,”arXiv preprint arXiv:2402.17168, 2024

    2024 arXiv

  100. [103]

    Balrog: Benchmarking agentic llm and vlm reasoning on games,

    D. Paglieri, B. Cupiałet al., “Balrog: Benchmarking agentic llm and vlm reasoning on games,”arXiv preprint arXiv:2411.13543, 2024

    2024 arXiv

  101. [104]

    Spider2-v: How far are multimodal agents from automating data science and engineering workflows?

    R. Cao, F. Leiet al., “Spider2-v: How far are multimodal agents from automating data science and engineering workflows?”Advances in Neural Information Processing Systems, vol. 37, pp. 107 703–107 744, 2024

    2024

  102. [105]

    Beyond prompts: Dynamic conversational bench- marking of large language models,

    D. Castillo-Bolado, J. Davidson, F. Gray, and M. Rosa, “Beyond prompts: Dynamic conversational bench- marking of large language models,”arXiv preprint arXiv:2409.20222, 2024

    2024 arXiv

  103. [106]

    Vgrp-bench: Visual grid reasoning puzzle benchmark for large vision-language models,

    Y . Ren, K. Tertikaset al., “Vgrp-bench: Visual grid reasoning puzzle benchmark for large vision-language models,”arXiv preprint arXiv:2503.23064, 2025

    2025 arXiv

  104. [107]

    Smartplay: A benchmark for llms as intelligent agents,

    Y . Wu, X. Tang, T. M. Mitchell, and Y . Li, “Smartplay: A benchmark for llms as intelligent agents,”arXiv preprint arXiv:2310.01557, 2023

    2023 arXiv

  105. [108]

    Are large vision language models good game players?

    X. Wang, B. Zhuang, and Q. Wu, “Are large vision language models good game players?”arXiv preprint arXiv:2503.02358, 2025

    2025 arXiv

  106. [109]

    Gamebench: Evaluating strategic reasoning abilities of llm agents,

    A. Costarelli, M. Allenet al., “Gamebench: Evaluating strategic reasoning abilities of llm agents,”arXiv preprint arXiv:2406.06613, 2024

    2024 arXiv

  107. [110]

    Ing-vp: Mllms cannot play easy vision-based games yet,

    H. Zhang, H. Guoet al., “Ing-vp: Mllms cannot play easy vision-based games yet,”arXiv preprint arXiv:2410.06555, 2024

    2024 arXiv

  108. [111]

    Dsgbench: A diverse strategic game benchmark for evaluating llm-based agents in complex decision-making environments,

    W. Tang, Y . Zhouet al., “Dsgbench: A diverse strategic game benchmark for evaluating llm-based agents in complex decision-making environments,”arXiv preprint arXiv:2503.06047, 2025

    2025 arXiv

  109. [112]

    Gamearena: Evaluating llm reasoning through live computer games,

    L. Hu, Q. Liet al., “Gamearena: Evaluating llm reasoning through live computer games,”arXiv preprint arXiv:2412.06394, 2024

    2024 arXiv

  110. [113]

    Competing large language models in multi-agent gaming environments,

    J.-t. Huang, E. J. Liet al., “Competing large language models in multi-agent gaming environments,” inThe Thirteenth International Conference on Learning Repre- sentations, 2025

    2025

  111. [114]

    Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations,

    J. Duan, R. Zhanget al., “Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations,”arXiv preprint arXiv:2402.12348, 2024

    2024 arXiv

  112. [115]

    Autoplanbench: Automatically generating benchmarks for llm planners from pddl,

    K. Stein, D. Fišeret al., “Autoplanbench: Automatically generating benchmarks for llm planners from pddl,” arXiv preprint arXiv:2311.09830, 2023

    2023 arXiv

  113. [116]

    Planbench: An exten- sible benchmark for evaluating large language models on planning and reasoning about change,

    K. Valmeekam, M. Marquezet al., “Planbench: An exten- sible benchmark for evaluating large language models on planning and reasoning about change,”Advances in Neural Information Processing Systems, vol. 36, pp. 38 975–38 987, 2023

    2023

  114. [117]

    On the planning abilities of large language models (a critical investi- gation with a proposed benchmark),

    K. Valmeekam, S. Sreedharanet al., “On the planning abilities of large language models (a critical investi- gation with a proposed benchmark),”arXiv preprint arXiv:2302.06706, 2023

    2023 arXiv

  115. [118]

    Flowbench: Revisiting and benchmarking workflow-guided planning for llm-based agents,

    R. Xiao, W. Maet al., “Flowbench: Revisiting and benchmarking workflow-guided planning for llm-based agents,”arXiv preprint arXiv:2406.14884, 2024

    2024 arXiv

  116. [119]

    Acpbench: Reasoning about action, change, and planning,

    H. Kokel, M. Katzet al., “Acpbench: Reasoning about action, change, and planning,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 25, 2025, pp. 26 559–26 568

    2025

  117. [120]

    Benchmarking agentic workflow generation,

    S. Qiao, R. Fanget al., “Benchmarking agentic workflow generation,”arXiv preprint arXiv:2410.07869, 2024

    2024 arXiv

  118. [121]

    Llf-bench: Benchmark for interactive learning from language feedback,

    C.-A. Cheng, A. Kolobovet al., “Llf-bench: Benchmark for interactive learning from language feedback,” 2023. [Online]. Available: https://arxiv.org/abs/2312.06853

    2023 arXiv

  119. [123]

    Et-plan-bench: Embodied task-level planning benchmark towards spatial- temporal cognition with foundation models,

    L. Zhang, Y . Wang, H. Guet al., “Et-plan-bench: Embodied task-level planning benchmark towards spatial- temporal cognition with foundation models,”arXiv preprint arXiv:2410.14682, 2024

    2024 arXiv

  120. [124]

    When benchmarks talk: Re- evaluating code llms with interactive feedback,

    J. Pan, R. Sharet al., “When benchmarks talk: Re- evaluating code llms with interactive feedback,” 2025. [Online]. Available: https://arxiv.org/abs/2502.18413

    2025 arXiv

  121. [125]

    LLM-evolve: Evaluation for LLM‘s evolving capability on benchmarks,

    J. You, M. Liuet al., “LLM-evolve: Evaluation for LLM‘s evolving capability on benchmarks,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for Computat...

    2024

  122. [126]

    Reflection-bench: probing ai intelligence with reflection,

    L. Li, Y . Wanget al., “Reflection-bench: probing ai intelligence with reflection,” 2024. [Online]. Available: https://arxiv.org/abs/2410.16270

    2024 arXiv

  123. [127]

    Toolllm: Facilitating large language models to master 16000+ real-world apis,

    S. Patilet al., “Toolllm: Facilitating large language models to master 16000+ real-world apis,”arXiv, 2023. [Online]. Available: https://arxiv.org/abs/2305.16504

    2023 arXiv

  124. [128]

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases,

    Q. Tanget al., “Toolalpaca: Generalized tool learning for language models with 3000 simulated cases,”arXiv,

  125. [129]

    Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities,

    J. Luet al., “Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities,”arXiv, 2024. [Online]. Available: https: JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 18 //arxiv.org/abs/2408.04682

    2024 arXiv

  126. [130]

    Revisiting, benchmarking and exploring api recommendation: How far are we?

    Y . Liet al., “Revisiting, benchmarking and exploring api recommendation: How far are we?”arXiv, 2021. [Online]. Available: https://arxiv.org/abs/2112.12653

    2021 arXiv

  127. [131]

    Api-bank: A comprehensive benchmark for tool-augmented llms,

    Y . Liet al., “Api-bank: A comprehensive benchmark for tool-augmented llms,”EMNLP, 2023. [Online]. Avail- able: https://aclanthology.org/2023.emnlp-main.187/

    2023

  128. [132]

    Berkeley function calling leaderboard v3 (aka berkeley tool calling leaderboard v3),

    G. Team, “Berkeley function calling leaderboard v3 (aka berkeley tool calling leaderboard v3),”Gorilla,

  129. [133]

    Available: https://gorilla.cs.berkeley.edu/ leaderboard.html

    [Online]. Available: https://gorilla.cs.berkeley.edu/ leaderboard.html

  130. [134]

    Restgpt: Connecting llms with real-world restful apis,

    R. Team, “Restgpt: Connecting llms with real-world restful apis,”RestGPT, 2023. [Online]. Available: https://restgpt.github.io/

    2023

  131. [135]

    Seal-tools: Self-instruct tool learning dataset for agent tuning and evaluation,

    F. Team, “Seal-tools: Self-instruct tool learning dataset for agent tuning and evaluation,”arXiv, 2024. [Online]. Available: https://arxiv.org/abs/2405.08355

    2024 arXiv

  132. [136]

    Towards stable large-scale benchmarking on tool learning of large language models,

    Z. Guoet al., “Towards stable large-scale benchmarking on tool learning of large language models,”ACL Findings, 2024. [Online]. Available: https://aclanthology. org/2024.findings-acl.664/

    2024

  133. [137]

    Nexusraven-13b, a new sota open-source llm for function calling,

    N. Team, “Nexusraven-13b, a new sota open-source llm for function calling,”Nexusflow, 2024. [Online]. Available: https://github.com/nexusflowai/NexusRaven

    2024

  134. [138]

    A comprehensive corpora for training and benchmarking api llms,

    K. Basuet al., “A comprehensive corpora for training and benchmarking api llms,”ACL, 2024. [Online]. Available: https://aclanthology.org/2024.acl-long.694/

    2024

  135. [139]

    Tooleyes: Fine-grained evaluation for tool learning capabilities of large language models,

    T. Team, “Tooleyes: Fine-grained evaluation for tool learning capabilities of large language models,” COLING, 2025. [Online]. Available: https://aclanthology. org/2025.coling-main.12/

    2025

  136. [140]

    Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets,

    N. Team, “Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets,”NeurIPS,

  137. [142]

    Action- based conversations dataset: A corpus for building more in-depth task-oriented dialogue systems,

    D. Chen, H. Chen, Y . Yang, A. Lin, and Z. Yu, “Action- based conversations dataset: A corpus for building more in-depth task-oriented dialogue systems,” 2021. [Online]. Available: https://arxiv.org/abs/2104.00783

    2021 arXiv

  138. [143]

    Exploring multi-step and constrained function calling under long-context scenario,

    T. Team, “Exploring multi-step and constrained function calling under long-context scenario,”arXiv, 2025. [Online]. Available: https://arxiv.org/abs/2501.10132

    2025 arXiv

  139. [144]

    A benchmark for evaluating llms on nested sequences of api calls,

    K. Basuet al., “A benchmark for evaluating llms on nested sequences of api calls,”OpenReview, 2024. [Online]. Available: https://openreview.net/forum?id= r7staQknbI

    2024

  140. [146]

    Petoolbench: Personalized evaluation benchmark for tool-augmented llms,

    P. Team, “Petoolbench: Personalized evaluation benchmark for tool-augmented llms,”arXiv, 2024. [Online]. Available: https://arxiv.org/abs/2407.02912

    2024 arXiv

  141. [148]

    Collab-overcooked: Benchmarking and evaluating large language models as collaborative agents,

    R. Wanget al., “Collab-overcooked: Benchmarking and evaluating large language models as collaborative agents,” arXiv preprint arXiv:2502.20073, 2025. [Online]. Available: https://arxiv.org/abs/2502.20073

    2025

  142. [149]

    Multiwoz – a large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling,

    P. Budzianowski, T.-H. Wenet al., “Multiwoz – a large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling,” 2020. [Online]. Available: https://arxiv.org/abs/1810.00278

    2020 arXiv

  143. [150]

    Automated test generation to evaluate tool-augmented llms as conversational ai agents,

    S. Arcadinho, D. Aparicio, and M. Almeida, “Automated test generation to evaluate tool-augmented llms as conversational ai agents,” 2024. [Online]. Available: https://arxiv.org/abs/2409.15934

    2024 arXiv

  144. [152]

    Battleagentbench: A benchmark for evaluating cooperation and competition capabilities of language models in multi-agent systems,

    W. Wang, D. Zhanget al., “Battleagentbench: A benchmark for evaluating cooperation and competition capabilities of language models in multi-agent systems,” arXiv preprint arXiv:2408.15971, 2024. [Online]. Available: https://arxiv.org/abs/2408.15971

    2024 arXiv

  145. [153]

    Evaluating the collaboration and competition of llm agents,

    A. Gangradeet al., “Evaluating the collaboration and competition of llm agents,”arXiv preprint arXiv:2503.01935, 2025. [Online]. Available: https: //arxiv.org/abs/2503.01935

    2025 arXiv

  146. [154]

    The narrativeqa reading comprehension challenge,

    T. Koˇcisk`y, J. Schwarzet al., “The narrativeqa reading comprehension challenge,”Transactions of the Associa- tion for Computational Linguistics, vol. 6, pp. 317–328, 2018

    2018

  147. [155]

    Sotopia: Interactive evaluation for social intelligence in language agents,

    ——, “Sotopia: Interactive evaluation for social intelligence in language agents,”arXiv preprint arXiv:2310.11667, 2023. [Online]. Available: https: //arxiv.org/abs/2310.11667

    2023 arXiv

  148. [156]

    Auto-arena: Automating llm evaluations with agent peer battles and committee discussions,

    R. Zhaoet al., “Auto-arena: Automating llm evaluations with agent peer battles and committee discussions,” arXiv preprint arXiv:2405.20267, 2024. [Online]. Available: https://arxiv.org/abs/2405.20267

    2024 arXiv

  149. [157]

    Dialsim: A real-time sim- ulator for evaluating long-term multi-party dialogue understanding of conversational agents,

    J. Kim, W. Chayet al., “Dialsim: A real-time sim- ulator for evaluating long-term multi-party dialogue understanding of conversational agents,”arXiv preprint arXiv:2406.13144, 2024

    2024

  150. [158]

    Mindagent: Emergent gaming interaction,

    H. Yuet al., “Mindagent: Emergent gaming interaction,” inFindings of the Association for Computational Linguistics: NAACL 2024, 2024, pp. 200–210. [Online]. Available: https://aclanthology.org/2024.findings-naacl. 200/

    2024

  151. [159]

    Cooperate or collapse: Emergence of sustainable cooperation in a society of llm agents,

    G. Piattiet al., “Cooperate or collapse: Emergence of sustainable cooperation in a society of llm agents,”arXiv preprint arXiv:2404.16698, 2024. [Online]. Available: https://arxiv.org/abs/2404.16698

    2024 arXiv

  152. [160]

    Episodic memories generation and evaluation benchmark for large language models,

    A. Huet, Z. B. Houidi, and D. Rossi, “Episodic memories generation and evaluation benchmark for large language models,”arXiv preprint arXiv:2501.13121, 2025. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 19

    2025 arXiv

  153. [161]

    Qmsum: A new benchmark for query-based multi-domain meeting summarization,

    M. Zhong, D. Yinet al., “Qmsum: A new benchmark for query-based multi-domain meeting summarization,” arXiv preprint arXiv:2104.05938, 2021

    2021 arXiv

  154. [162]

    Quality: Question answering with long input texts, yes!

    R. Y . Pang, A. Parrishet al., “Quality: Question answering with long input texts, yes!”arXiv preprint arXiv:2112.08608, 2021

    2021 arXiv

  155. [163]

    Agentbench: Evaluating llms as agents,

    X. Liu, H. Yuet al., “Agentbench: Evaluating llms as agents,” 2023. [Online]. Available: https: //arxiv.org/abs/2308.03688

    2023 arXiv

  156. [164]

    Evaluating very long- term conversational memory of llm agents,

    A. Maharana, D.-H. Leeet al., “Evaluating very long- term conversational memory of llm agents,”arXiv preprint arXiv:2402.17753, 2024

    2024 arXiv

  157. [165]

    Longmemeval: Benchmarking chat assistants on long-term interactive memory,

    D. Wu, H. Wanget al., “Longmemeval: Benchmarking chat assistants on long-term interactive memory,”arXiv preprint arXiv:2410.10813, 2024

    2024 arXiv

  158. [166]

    Galileo ai agent leaderboard,

    G. A. Team, “Galileo ai agent leaderboard,”

  159. [167]

    Perltqa: A personal long-term memory dataset for memory classification, retrieval, and synthesis in question answering,

    Y . Du, H. Wanget al., “Perltqa: A personal long-term memory dataset for memory classification, retrieval, and synthesis in question answering,”arXiv preprint arXiv:2402.16288, 2024

    2024 arXiv

  160. [168]

    Streambench: Towards benchmarking continuous improvement of language agents,

    C.-K. Wu, Z. R. Tamet al., “Streambench: Towards benchmarking continuous improvement of language agents,”Advances in Neural Information Processing Systems, vol. 37, pp. 107 039–107 063, 2024

    2024

  161. [169]

    Who’s the mvp? a game- theoretic evaluation benchmark for modular attribution in llm agents,

    Y . Yang, B. Huang, S. Qiet al., “Who’s the mvp? a game- theoretic evaluation benchmark for modular attribution in llm agents,”arXiv preprint arXiv:2502.00510, 2025. [Online]. Available: https://arxiv.org/abs/2502.00510

    2025

  162. [170]

    Gaia: a benchmark for general ai assistants,

    G. Mialon, C. Fourrieret al., “Gaia: a benchmark for general ai assistants,” 2023. [Online]. Available: https://arxiv.org/abs/2311.12983

    2023 arXiv

  163. [171]

    Mmau: A holistic benchmark of agent capabilities across diverse domains,

    G. Yin, H. Bai, S. Maet al., “Mmau: A holistic benchmark of agent capabilities across diverse domains,” arXiv preprint arXiv:2407.18961, 2024. [Online]. Available: https://arxiv.org/pdf/2407.18961

    2024 arXiv

  164. [172]

    Mle-bench: Eval- uating machine learning agents on machine learning engineering,

    J. S. Chan, N. Chowdhuryet al., “Mle-bench: Eval- uating machine learning agents on machine learning engineering,”arXiv preprint arXiv:2410.07095, 2024

    2024 arXiv

  165. [173]

    Available: https://huggingface.co/spaces/ galileo-ai/agent-leaderboard

    [Online]. Available: https://huggingface.co/spaces/ galileo-ai/agent-leaderboard

  166. [174]

    Hal: A holistic agent leaderboard for centralized and reproducible agent evaluation,

    B. Stroebl, S. Kapoor, and A. Narayanan, “Hal: A holistic agent leaderboard for centralized and reproducible agent evaluation,” https://github.com/ princeton-pli/hal-harness, 2025

    2025

  167. [175]

    Agent arena: A platform for evaluating and comparing llm agents,

    N. Yekollu, A. Bohra, A. Chirumamillaet al., “Agent arena: A platform for evaluating and comparing llm agents,” 2025. [Online]. Available: https://www. agent-arena.com/

    2025

  168. [176]

    Rank analysis of incomplete block designs: I. the method of paired comparisons,

    R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,”Biometrika, vol. 39, no. 3-4, pp. 324– 345, 1952

    1952

  169. [177]

    Claude ai,

    Anthropic, “Claude ai,” 2025, accessed: 2025-04-27. [Online]. Available: https://claude.ai/

    2025

  170. [178]

    Repocoder: Repository- level code completion through iterative retrieval and generation,

    F. Zhang, B. Chenet al., “Repocoder: Repository- level code completion through iterative retrieval and generation,”arXiv preprint arXiv:2303.12570, 2023

    2023 arXiv

  171. [179]

    Program induction by rationale generation: Learning to solve and explain alge- braic word problems,

    W. Ling, D. Yogatamaet al., “Program induction by rationale generation: Learning to solve and explain alge- braic word problems,”arXiv preprint arXiv:1705.04146, 2017

    2017 arXiv

  172. [180]

    Grounding large language models in interactive environments with online reinforce- ment learning,

    T. Carta, C. Romacet al., “Grounding large language models in interactive environments with online reinforce- ment learning,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 3676–3713

    2023

  173. [181]

    Benchmarking the spectrum of agent capa- bilities,

    D. Hafner, “Benchmarking the spectrum of agent capa- bilities,”arXiv preprint arXiv:2109.06780, 2021

    2021 arXiv

  174. [182]

    Textworld: A learning environment for text-based games,

    M.-A. Côté, A. Kádáret al., “Textworld: A learning environment for text-based games,” inComputer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected...

    2018

  175. [183]

    Are nlp models really able to solve simple math word problems?

    A. Patel, S. Bhattamishra, and N. Goyal, “Are nlp models really able to solve simple math word problems?”arXiv preprint arXiv:2103.07191, 2021

    2021 arXiv

  176. [184]

    Chain of thought prompting elicits reasoning in large language models,

    J. Wei, X. Wanget al., “Chain of thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 24 824–24 837. [Online]. Available: https://arxiv.org/abs/2201.11903

    2022 arXiv

  177. [185]

    Tree of thoughts: Deliberate problem solving with large language models,

    X. Yao, Z. Yeet al., “Tree of thoughts: Deliberate problem solving with large language models,” in Advances in Neural Information Processing Systems, vol. 36, 2023. [Online]. Available: https://arxiv.org/abs/ 2305.10601

    2023 arXiv

  178. [186]

    Folio: Natural language reasoning with first-order logic,

    S. Han, H. Schoelkopfet al., “Folio: Natural language reasoning with first-order logic,” 2024. [Online]. Available: https://arxiv.org/abs/2209.00840

    2024 arXiv

  179. [187]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosarajuet al., “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  180. [188]

    Measuring mathematical problem solving with the math dataset,

    D. Hendrycks, C. Burnset al., “Measuring mathematical problem solving with the math dataset,”arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  181. [189]

    Cumulative reason- ing with large language models,

    Y . Zhang, J. Yanget al., “Cumulative reason- ing with large language models,”arXiv preprint arXiv:2308.04371, 2023

    2023 arXiv

  182. [190]

    Mint: Evaluating llms in multi- turn interaction with tools and language feedback,

    X. Wang, Z. Wanget al., “Mint: Evaluating llms in multi- turn interaction with tools and language feedback,” 2024. [Online]. Available: https://arxiv.org/abs/2309.10691

    2024 arXiv

  183. [191]

    Hotpotqa: A dataset for diverse, ex- plainable multi-hop question answering,

    Z. Yang, P. Qiet al., “Hotpotqa: A dataset for diverse, ex- plainable multi-hop question answering,”arXiv preprint arXiv:1809.09600, 2018

    2018 arXiv

  184. [192]

    Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies,

    M. Geva, D. Khashabiet al., “Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies,”Transactions of the Association for Computational Linguistics, vol. 9, pp. 346–361, 2021

    2021

  185. [193]

    Large language models cannot self-correct reasoning yet,

    J. Huang, X. Chenet al., “Large language models cannot self-correct reasoning yet,” 2024. [Online]. Available: https://arxiv.org/abs/2310.01798

    2024 arXiv

  186. [195]

    Exploring and benchmarking the planning capabilities of large language models,

    B. Bohnet, A. Novaet al., “Exploring and benchmarking the planning capabilities of large language models,” arXiv preprint arXiv:2406.13094, 2024

    2024 arXiv

  187. [196]

    Natural plan: Benchmark- ing llms on natural language planning,

    H. S. Zheng, S. Mishraet al., “Natural plan: Benchmark- ing llms on natural language planning,”arXiv preprint arXiv:2406.04520, 2024

    2024 arXiv

  188. [197]

    Chatbot arena: An open platform for evaluating llms by human preference,

    W.-L. Chiang, L. Zhenget al., “Chatbot arena: An open platform for evaluating llms by human preference,” 2024

    2024

  189. [198]

    The benefits of a concise chain of thought on problem-solving in large language models,

    M. Renze and E. Guven, “The benefits of a concise chain of thought on problem-solving in large language models,” in2024 2nd International Conference on Foundation and Large Language Models (FLLM). IEEE, Nov. 2024, p. 476–483. [Online]. Available: http://dx.doi.org/10.1109/FLLM...

    2024

  190. [199]

    Self-reflection makes large language models safer, less biased, and ideologically neutral,

    F. Liu, N. AlDahoulet al., “Self-reflection makes large language models safer, less biased, and ideologically neutral,” 2025. [Online]. Available: https://arxiv.org/abs/ 2406.10400 JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 20

    2025 arXiv

  191. [200]

    Agentverse: Facilitating multi- agent collaboration and exploring emergent behaviors,

    W. Chen, Y . Suet al., “Agentverse: Facilitating multi- agent collaboration and exploring emergent behaviors,”

  192. [201]

    Self-contrast: Better reflection through inconsistent solving perspectives,

    W. Zhang, Y . Shenet al., “Self-contrast: Better reflection through inconsistent solving perspectives,” 2024. [Online]. Available: https://arxiv.org/abs/2401.02009

    2024 arXiv

  193. [202]

    Measuring coding challenge competence with apps,

    D. Hendrycks, S. Basartet al., “Measuring coding challenge competence with apps,” 2021. [Online]. Available: https://arxiv.org/abs/2105.09938

    2021 arXiv

  194. [203]

    Livecodebench: Holistic and contamination free evaluation of large language models for code,

    N. Jain, K. Hanet al., “Livecodebench: Holistic and contamination free evaluation of large language models for code,” 2024. [Online]. Available: https: //arxiv.org/abs/2403.07974

    2024 arXiv

  195. [205]

    Agent- as-a-judge: Evaluate agents with agents,

    M. Zhuge, C. Zhao, D. Ashleyet al., “Agent- as-a-judge: Evaluate agents with agents,”arXiv preprint arXiv:2410.10934, 2024. [Online]. Available: https://arxiv.org/abs/2410.10934

    2024 arXiv

  196. [206]

    Deepclaude: Combining deepseek r1’s reasoning with claude’s creativity and code generation,

    Asterisk, “Deepclaude: Combining deepseek r1’s reasoning with claude’s creativity and code generation,” 2025, accessed: 2025-04-27. [Online]. Available: https://deepclaude.com/

    2025

  197. [208]

    Available: https://arxiv.org/abs/2308

    [Online]. Available: https://arxiv.org/abs/2308. 10848

  198. [209]

    Acebench: Who wins the match point in tool usage?

    C. Chen, X. Haoet al., “Acebench: Who wins the match point in tool usage?” 2025. [Online]. Available: https://arxiv.org/abs/2501.12851 APPENDIXA TABLES OFTAXONOMY To assist researchers in efficiently and accurately selecting appropriate evaluation benchmarks for various types ...

    2025

  199. [210]

    Format verification → Execution verification → Semantic verification

    Static Research paper pool Novelty, excitement, feasibility, expected effectiveness Research idea generation PaperQA [92] Static Scientific article pool Retrieval AUC, retrieval probability Scientific article question answering SCIENCEWORLD [93] InteractiveSimulated text envir...

    2015

  200. [2023]

    Available: https://arxiv.org/abs/2306

    [Online]. Available: https://arxiv.org/abs/2306. 05301

  201. [2024]

    Available: https://arxiv.org/abs/2404

    [Online]. Available: https://arxiv.org/abs/2404. 09992

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.