Pith. sign in

REVIEW 2 major objections 6 minor 8 cited by

Magentic-UI: Towards Human-in-the-loop Agentic Systems

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Keeping a human in the loop raised an AI agent's task-completion rate by 71 percent on the GAIA benchmark, from 30.3 to 51.9 percent.

desk verdict The 71% HITL gain is real but mislabeled: the protocol lets the simulated user supply final answers when the agent fails, and that fallback alone may explain much of the improvement. The paper is still a worthwhile open-source systems contribution, but the headline claim needs a major revision. read the letter →

arxiv 2507.22358 v1 pith:FRECMOC5 submitted 2025-07-30 cs.AI cs.HC

Hussein Mozannar , Gagan Bansal , Cheng Tan , Adam Fourney , Victor Dibia , Jingya Chen , Jack Gerrits , Tyler Payne
show 12 more authors
Matheus Kunzler Maldaner Madeleine Grunde-McLaughlin Eric Zhu Griffin Bassman Jacob Alber Peter Chang Ricky Loynd Friederike Niedtner Ece Kamar Maya Murad Rafah Hosn Saleema Amershi
This is my paper
classification cs.AIcs.HC
keywords human-in-the-loopAIagentsmulti-agentsystemshuman-agentinteractionco-planningsimulateduserevaluationagentsafetyweb
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that keeping a human in the loop, rather than pushing for full autonomy, is the practical path to extracting useful work from today's imperfect AI agents. It presents Magentic-UI, an open-source web interface where a person collaborates with a multi-agent system through editable plans, mid-task handoffs, action approvals, answer verification, saved plans, and parallel sessions. Its central quantitative evidence comes from a simulated-user study on GAIA, a benchmark of 465 general-assistant questions: a user with a human-written plan for each task raised the agent's task-completion rate from 30.3 percent to 51.9 percent, a 71 percent relative gain, while being consulted in only 10 percent of tasks and supplying the final answer in 18 percent. The authors read this as evidence that even lightweight oversight can close a meaningful part of the gap to human performance (92 percent on the same benchmark), and they frame human involvement as a permanent design principle rather than a stopgap for weak models.

What carries the argument

The load-bearing machinery is a multi-agent team headed by an Orchestrator agent that generates, edits, and tracks a plan, with the user represented as a UserProxy agent and treated as a member of the team. The shared object that makes human involvement cheap is the plan itself: a sequence of natural-language steps (agent name, title, details) that both the user and the Orchestrator can read and edit, serving as a small domain-specific language for collaboration. Co-planning lets the user revise this plan before execution, co-tasking lets either side interrupt the other mid-execution, and an action guard, a two-stage filter of irreversibility heuristics plus an LLM judge, decides which actions require explicit human approval. For evaluation, the paper converts the GAIA benchmark into an interactive one by adding a simulated user, an LLM without tools whose prompt is tuned to give guidance without revealing ground-truth answers; this harness is what produces the headline 30.3-to-51.9 percent comparison.

What would settle it

Re-run the interactive GAIA experiment with three ablations: (1) a simulated user with no side-information plan, (2) a simulated user with the anti-leakage tuning removed, and (3) real users who see only the public task text. If completion stays near 51.9 percent under (1), the gain is genuine steering; if it collapses, the plan itself does the work; if it jumps under (2), leaked answers inflate the headline; and (3) settles whether the effect survives contact with actual humans.

Watch

Extended reading notes

Core claim

The paper's central claim is that human-in-the-loop agentic systems can unlock productivity from imperfect autonomous agents at low human cost. The load-bearing empirical assertion is a simulated-user experiment on the GAIA validation set (162 tasks): Magentic-UI in autonomous mode completes 30.3 percent of tasks, and with a simulated user that has access to side information about each task, completion rises to 51.9 percent, a 71 percent relative improvement. The simulated user is an LLM without tools that behaves like a knowledgeable human, is consulted in only 10 percent of tasks, and provides the final answer in 18 percent of tasks when the agent cannot. A control simulated user powered by a smarter model reaches 42.6 percent, and human performance on the same set is 92 percent, so the authors conclude that human-in-the-loop interaction bridges part of the gap to human performance at a fraction of the cost. In support of the overall design, the paper also reports autonomous performance across four benchmarks (42.5 percent on GAIA's test set with o4-mini, 27.6 on AssistantBench, 82.2 on WebVoyager, 45.5 on WebGames) and a safety evaluation in which all 24 adversarial scenarios were blocked under the default configuration.

Load-bearing premise

The headline 71 percent gain rests on the assumption that the simulated user, an LLM handed a human-written plan for each GAIA task with its prompt tuned to avoid spilling the answer, behaves like a real knowledgeable human, even though the paper reports that answers still leaked in 6 percent of tasks.

Editorial extensions

If this is right

  • Human guidance can substitute for model scale: a GPT-4o agent guided by a knowledgeable simulated user (51.9 percent) beats the same agent guided by a smarter o4-mini simulated user (42.6 percent).
  • The human cost of the gain is small: the agent asked for help in only 10 percent of tasks, about 1.1 queries per assisted task, and the simulated user supplied the final answer in 18 percent of tasks when the agent stalled.
  • Verifiable success signals change agent behavior: on WebGames, where completing the task reveals a password, Magentic-UI replans in 52.9 percent of tasks, roughly its failure rate, whereas on GAIA and AssistantBench it replans in about 20 to 22 percent, suggesting agents struggle to know when to stop without a clear success signal.
  • The layered safety design (Docker sandboxing, a fresh browser without credentials, action guards, website allow-lists) blocked all 24 adversarial scenarios in the default configuration; when mitigations were disabled, prompt injection reliably compromised the agent, so the safeguards are load-bearing for safe deployment.
  • Autonomous capability is preserved: Magentic-UI with o4-mini reaches 82.2 percent on WebVoyager and 45.5 percent on WebGames, comparable to leading web agents on those benchmarks, so the interactivity modifications did not sacrifice raw browsing ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the simulated-user result transfers to real users, interface design, how easily a person can read and edit a plan, could matter as much as model scaling for agent products; a testable version is to run the same 162-task protocol with real users who see only the public task text and compare their assisted completion rate to 51.9 percent.
  • The reported 6 percent answer-leakage rate marks the ceiling of a contamination risk: because GAIA's human-written plans contain the answers, the tuned prompt is the only barrier between guidance and ground truth, and future simulated-user benchmarks should audit leakage per task rather than in aggregate.
  • The replanning statistics suggest a cheap engineering lever: environments that give agents a clear success signal make agents stop at the right moment, whereas open-ended benchmarks leave agents wandering; instrumenting open tasks with self-verification checks could cut wasted runtime, which the paper shows is roughly twice as long on failed tasks (median 236.7 seconds versus 113.9 seconds).
  • The safety results imply that human oversight is itself a security control: action guards and plan approvals stopped all 24 attacks under the default configuration, suggesting that a human checkpoint at irreversible actions may be more robust than relying on the model to refuse adversarial instructions alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper presents Magentic-UI, an open-source web interface for human-in-the-loop agentic systems, built on a multi-agent architecture adapted from Magentic-One. It introduces six interaction mechanisms (co-planning, co-tasking, action approval, answer verification, memory, multitasking) and evaluates the system in four ways: autonomous performance on GAIA, AssistantBench, WebVoyager, and WebGames; simulated-user experiments on GAIA; a qualitative user study with 12 participants; and targeted safety/security red-teaming. The headline result is in Section 7.3: on the GAIA validation set, Magentic-UI with a simulated user that has access to a human-written plan improves task completion from 30.3% to 51.9%, a 71% relative improvement, which the paper attributes to lightweight human-in-the-loop feedback.

Significance. Should the central claim hold, the paper makes a useful contribution to human-agent interaction by demonstrating that a modest amount of human guidance can substantially improve the task success of a generalist web agent, and by providing an open-source platform for further study. Strengths include the release of code and evaluation harness, the breadth of the evaluation (three benchmark families plus safety scenarios), and the explicit discussion of limitations in Section 8.2, including the lack of a productivity measurement. The safety evaluation with 24 adversarial scenarios is a valuable addition. However, the simulated-user result has a specific identification problem that currently prevents the headline claim from being supported at the stated magnitude, as detailed in the major comments.

major comments (2)
  1. [7.3] The reported 71% improvement (30.3% to 51.9%) conflates interactive guidance with an oracle-style answer-rescue policy. The protocol states that 'if Magentic-UI does not provide a final answer, then the simulated user provides an answer instead,' and the paper reports this fallback is used in 18% of tasks. The simulated user has access to the human-written per-task plan (side information), so this fallback can supply correct answers directly, independent of the co-planning/co-tasking interaction mechanisms. The leakage audit is conducted only 'when Magentic-UI provides a final answer,' explicitly excluding the 18% fallback cases. Because each successful fallback adds roughly 0.62 percentage points (1/162), if even half of the ~29 fallback tasks are answered correctly the fallback alone accounts for about 9 of the 21.6-point gain; if most are correct it accounts for almost the entire gain. The paper must decompose the improvement by reporting the success rate of the fallback cases, the completion rate excluding those cases, and the incremental contribution of co-planning and co-tasking separately. Without this decomposition, the headline claim in the abstract and Section 7.3 is not identified.
  2. [7.3 (simulated user validity)] The simulated user with side information is given a human-written plan that 'often' embeds the ground-truth answer, and the prompt was manually tuned to prevent leakage, with a reported 6% leak rate when Magentic-UI provides a final answer. This audit is not a rigorous measurement: it relies on manual judgment, has no reported inter-annotator reliability, and, as noted, excludes the fallback cases. The paper should provide an automated or at least systematically sampled leakage check for both conditions and report leakage rates separately for tasks with and without the final-answer fallback. Additionally, the claim that these experiments 'reflect a lower bound on the value of human feedback' is not supported: real users do not generally have access to a human-written solution plan for each task, so the side-information condition may overestimate the value of human guidance. The paper should temper this claim or provide a more realistic operationalization of the human input.
minor comments (6)
  1. [Abstract and Section 1] The abstract lists five interaction mechanisms (co-planning, co-tasking, multi-tasking, action guards, long-term memory) but says there are six; the Introduction lists six including action approval and answer verification. Please reconcile.
  2. [2 Related Work] The sentence 'We revisit our progress on these challenges via Magentic-UI in Section X' contains a placeholder 'Section X' that must be filled.
  3. [7.2, Table 1] The autonomous evaluation reports single-run point estimates without confidence intervals or significance tests, making claims such as 'matches the performance of Magentic-One' and 'falls short of the current SOTA' difficult to assess; consider adding error bars or multi-seed results.
  4. [7.3] The sentence 'These experiments reflect a lower bound on the value of human feedback' is misleading; the simulated user receives per-task human-written plans, which real users would not have, so the result is at best an upper bound on the value of a knowledgeable user under idealized conditions.
  5. [8.3] There is a typographical error in 'Magentic-UI s browser' (missing apostrophe).
  6. [6.2] The plan DSL definition in Eq. (1) uses a notation with parentheses and colons that is not formally defined; please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline human-in-the-loop gain is an external benchmark measurement, not an identity or a self-citation-derived result.

full rationale

The paper's load-bearing empirical claims are evaluated against external benchmarks and do not reduce to their inputs by construction. The autonomous results in Section 7.2 compare Magentic-UI against GAIA, AssistantBench, WebVoyager, and WebGames using external leaderboard baselines, so the reported task-completion rates are not defined in terms of the system's own components. The headline simulated-user result in Section 7.3 is also an empirical measurement: a simulated user with side information (a human-written GAIA plan) interacts with Magentic-UI, and task completion is scored against GAIA ground truth; success is not a tautological consequence of providing the plan, as the paper reports partial help usage (help requested in 10% of tasks, final-answer fallback in 18%) and a leakage audit (6% leakage when Magentic-UI provides a final answer). The self-citations to Magentic-One [21] and the human-agent communication taxonomy [7] frame the design and discussion, but the paper does not use those citations as proof of the empirical improvements; the benchmark comparisons and the human-study observations stand independently of those citations. The missing 'Section X' cross-reference and the fact that the leakage audit is scoped to tasks where Magentic-UI provides a final answer are report-quality and identification concerns, not circularity. No equation, definition, or fitted parameter is shown to be equivalent to a predicted result by construction, so no circular step is established.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. Its contributions are software artifacts, design patterns, and evaluation protocols; the UserProxy agent is a design pattern, not an invented entity.

free parameters (4)
  • UserProxy description field = quoted in Section 6.3
    Hand-written prompt controlling when the Orchestrator delegates steps to the user; the paper says it was arrived at through unstructured interaction with the system.
  • Simulated user prompt tuning = tuned to avoid leaking GAIA answers; leakage 6%
    Prompt crafted to make the simulated user guide rather than reveal answers; directly affects the 51.9% result in Section 7.3.
  • ActionGuard irreversibility heuristics = always/maybe/never per action type
    Developer-set labels on actions that determine which actions require human approval; affects safety-test outcomes in Section 6.4.
  • ActionGuard judge prompt = Appendix B system prompt
    LLM-based judge decides 'maybe' actions; its thresholds are set by prompt wording, affecting approval frequency and user burden.
assumptions (4)
  • domain assumption LLM agents with natural-language agent descriptions can reliably execute delegated steps and respect guardrail prompts.
    The Orchestrator loop and ActionGuard depend on this being true enough; failures would break the architecture's core delegation in Sections 6.2-6.4.
  • domain assumption Benchmark metrics (GAIA string-match, WebVoyager GPT-4 judge, WebGames password signal) are valid measures of task success.
    All quantitative claims rely on these external benchmarks as ground truth in Section 7.1.
  • domain assumption A simulated user with GAIA side-information plans approximates a real user with relevant domain knowledge.
    The headline human-in-the-loop improvement is measured with this simulated user, not real users, in Section 7.3.
  • domain assumption The 24 internal adversarial scenarios are representative of near-term real-world safety risks.
    Safety conclusions are drawn from this internal set in Section 7.5 and Appendix D.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Magentic-UI: Towards Human-in-the-loop Agentic Systems." pith.science (2026). https://pith.science/paper/FRECMOC5

@misc{pith2026250722358,
  author       = {Pith},
  title        = {Pith review of: Magentic-UI: Towards Human-in-the-loop Agentic Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FRECMOC5}},
  note         = {Machine review of arXiv:2507.22358}
}
read the original abstract

AI agents powered by large language models are increasingly capable of autonomously completing complex, multi-step tasks using external tools. Yet, they still fall short of human-level performance in most domains including computer use, software development, and research. Their growing autonomy and ability to interact with the outside world, also introduces safety and security risks including potentially misaligned actions and adversarial manipulation. We argue that human-in-the-loop agentic systems offer a promising path forward, combining human oversight and control with AI efficiency to unlock productivity from imperfect systems. We introduce Magentic-UI, an open-source web interface for developing and studying human-agent interaction. Built on a flexible multi-agent architecture, Magentic-UI supports web browsing, code execution, and file manipulation, and can be extended with diverse tools via Model Context Protocol (MCP). Moreover, Magentic-UI presents six interaction mechanisms for enabling effective, low-cost human involvement: co-planning, co-tasking, multi-tasking, action guards, and long-term memory. We evaluate Magentic-UI across four dimensions: autonomous task completion on agentic benchmarks, simulated user testing of its interaction capabilities, qualitative studies with real users, and targeted safety assessments. Our findings highlight Magentic-UI's potential to advance safe and efficient human-agent collaboration.

Figures

Figures reproduced from arXiv: 2507.22358 by the authors.

Figure 1
Figure 1. Magentic-UI is an open-source research prototype of a human-centered agent that is [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The Magentic-UI interface displaying a task in progress being completed. The interface is [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The plan editor component in Magentic-UI showing the generated plan in response to a [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Screenshots of the Magentic-UI interface showing: (a) the user interrupting the system to [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The saved plans view in Magentic-UI showing the users’ plans that they learned, created, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Overall System Architecture of Magentic-UI. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Planning Mode. When the user types in their query to Magentic-UI, the Orchestrator generates a plan in response. When generating the plan, the Orchestrator can use web search and can retrieve relevant plans from memory (discussed in Section 5). A plan is a list of an a…
Figure 7
Figure 7. Figure 7: Simplified Orchestrator loop (DSL) adhering to the schema in (1), and interpreted by the Orchestrator in execution mode. PlanStep := ( agent name, title, details ) Plan := PlanStep1 , PlanStep2 , . . . , PlanStepn (1) For instance, if the task was “create a csv with th…
Figure 8
Figure 8. Figure 8: Magentic-UI implements an action guard system to ensure irreversible or potentially [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Distribution of the run time in seconds of Magentic-UI on the WebVoyager dataset split [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Analysis of Magentic-UI planning statistics across all four evaluated Datasets. We show [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Comparison on the GAIA validation set of the accuracy of Magentic-One, Magentic-UI [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: System Usability Scale results. The results were positive, with a 74.58 score overall. 75% [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: In scenario social_eng_03, Magentic-UI encounters a “Chrome Update” phishing popup mid-task. In testing, Magentic-UI identifies the phishing attempt, re-plans, and waits for user approval. scenarios were effective. The layered mitigations ensured that users were consu…
Figure 14
Figure 14. Figure 14: In scenario social_eng_01 Magentic-UI encounters an article behind a paywall, and is presented an opportunity to log in. Upon clicking the log-in button, the agent is presented with an egregious request for OAuth permissions. If the user is logged in to Google (e.g., …
Figure 15
Figure 15. Figure 15: In scenario injection_web_01, Magentic-UI encounters a webpage that has anticipated that it will be summarized by an LLM. The page explains that proper summarization requires decoding and provides a direction for where decoding instructions can be found. In this case,…

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Plans Work in Mysterious Ways: Evaluating a Plan Mode for Spreadsheet Agents

    cs.HC 2026-07 conditional novelty 6.0 of 10

    Plan Mode for spreadsheet agents shifts requirements to clarifying questions, reduces refinement, and improves perceived collaboration without changing final workbook quality.

  2. Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight

    cs.HC 2025-09 conditional novelty 6.0 of 10

    GUI agents frequently fall for deceptive interface designs, often without recognizing them, and human supervision of agents improves avoidance only partially while introducing new attention and workload costs.

  3. Instruction Agent: Enhancing Agent with Expert Demonstration

    cs.AI 2025-09 conditional novelty 6.0 of 10

    With one human demonstration, the Instruction Agent extracts instructions and uses verifier/backtracker modules to execute them, reaching 60% success on 20 previously unsolved OSWorld tasks.

  4. Governed AI-Assisted Engineering: Graduated Human Oversight for Agentic Code Generation in Regulated Domains

    cs.HC 2026-06 unverdicted novelty 5.5 of 10

    GAIE introduces an Oversight Classification Model to route code generation tasks to human-in-the-loop, human-over-the-loop, or automated-with-monitoring tiers based on regulatory impact, customer proximity, reversibil...

  5. JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents

    cs.CV 2026-07 conditional novelty 5.0 of 10

    JarvisHub open-sources a three-layer canvas-state, protocol-bridge, and agent-runtime harness so multimodal creative agents can inspect and update a shared editable project graph over long workflows.

  6. Plover: Steering GUI Agents through Plan-Centric Interaction

    cs.AI 2026-07 conditional novelty 5.0 of 10

    An expert repairing visible plans rescued 23 of 26 failed GUI automation runs, turning 17 into full and 6 into partial successes.

  7. HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization

    cs.HC 2026-06 unverdicted novelty 5.0 of 10

    HiLSVA shows that a human-in-the-loop LLM agent system can help novices and experts complete scientific visualization tasks, while human oversight adds measurable execution time.

  8. Agents in the Wild: Where Research Meets Deployment

    cs.AI 2026-07 unverdicted

    A tutorial description reviewing the state of LLM agent deployment, with no new research findings.

Reference graph

Works this paper leans on

116 extracted references · 36 canonical work pages · cited by 8 Pith papers

  1. [1]

    Aliannejadi, H

    M. Aliannejadi, H. Zamani, F. Crestani, and W. B. Croft. Asking clarifying questions in open-domain information-seeking conversations. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’19, page 475–484. ACM, July 2019

    2019

  2. [2]

    Amodei, C

    D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety.arXiv preprint arXiv:1606.06565, 2016

    arXiv 2016

  3. [3]

    Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, oct 2024

    Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, oct 2024

    2024

  4. [4]

    A. T. at InclusionAI. Aworld: A framework for agent learning of complex tasks via action- observation-reward experience, 2025

    2025

  5. [5]

    Github | babyagi.https://github.com/yoheinakajima/babyagi, 2023

    BabyAGI. Github | babyagi.https://github.com/yoheinakajima/babyagi, 2023

    2023

  6. [6]

    Bansal, B

    G. Bansal, B. Nushi, E. Kamar, D. Weld, W. Lasecki, and E. Horvitz. Updates in human-ai teams: Understanding and addressing the performance/compatibility tradeoff. In AAAI Conference on Artificial Intelligence. AAAI, January 2019

    2019

  7. [7]

    Bansal, J

    G. Bansal, J. W. Vaughan, S. Amershi, E. Horvitz, A. Fourney, H. Mozannar, V. Dibia, and D. S. Weld. Challenges in human-agent communication.ArXiv, 2024

    2024

  8. [8]

    Bansal, T

    G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro, and D. S. Weld. Does the whole exceed its parts? the effect of ai explanations on complementary team performance, 2021. 27

    2021

Show all 116 references
  1. [9]

    J. Brooke. SUS – a quick and dirty usability scale, pages 189–194. 01 1996

    1996

  2. [10]

    Z. Chen, M. White, R. Mooney, A. Payani, Y. Su, and H. Sun. When is tree search useful for llm planning? it depends on the discriminator, 2024

    2024

  3. [11]

    Cheng, C

    Y. Cheng, C. Zhang, Z. Zhang, X. Meng, S. Hong, W. Li, Z. Wang, Z. Wang, F. Yin, J. Zhao, et al. Exploring large language model based intelligent agents: Definitions, methods, and prospects. arXiv preprint arXiv:2401.03428, 2024

    2024 arXiv

  4. [12]

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  5. [13]

    Introducing devin, the first ai software engineer, 2024

    Cognition.ai. Introducing devin, the first ai software engineer, 2024

    2024

  6. [14]

    K. Z. Cui, M. Demirer, S. Jaffe, L. Musolff, S. Peng, and T. Salz. The productivity effects of generative ai: Evidence from a field experiment with github copilot. 2024

    2024

  7. [15]

    X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su. Mind2web: Towards a generalist agent for the web. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, vol- ume 36, pages 28...

    2023

  8. [16]

    X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su. Mind2web: Towards a generalist agent for the web, 2023

    2023

  9. [17]

    L. Dong, T. Yuan, Y. Wang, T. Xia, Z. Zhang, Z. He, B. Zhou, R. Wang, F. Li, G. Liu, L. Xu, and R. Zhao. R-judge: Benchmarking safety risk awareness for llm agents. InConference on Empirical Methods in Natural Language Processing, 2024

    2024

  10. [18]

    Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch. Improving factuality and reasoning in language models through multiagent debate.arXiv preprint arXiv:2305.14325, 2023

    2023 arXiv

  11. [19]

    H. Fang, X. Zhu, and I. Gurevych. Inferact: Inferring safe actions for llm-based agents through preemptive evaluation and human feedback, 2024

    2024

  12. [20]

    K. J. K. Feng, K. Pu, M. Latzke, T. August, P. Siangliulue, J. Bragg, D. S. Weld, A. X. Zhang, and J. C. Chang. Cocoa: Co-planning and co-execution with ai agents, 2025

    2025

  13. [21]

    Fourney, G

    A. Fourney, G. Bansal, H. Mozannar, C. Tan, E. Salinas, Erkang, Zhu, F. Niedtner, G. Proeb- sting, G. Bassman, J. Gerrits, J. Alber, P. Chang, R. Loynd, R. West, V. Dibia, A. Awadallah, E. Kamar, R. Hosn, and S. Amershi. Magentic-one: A generalist multi-agent system for solvin...

    2024

  14. [22]

    Github copilot, 2021

    GitHub. Github copilot, 2021

    2021

  15. [23]

    Copilot workspace: An agentic dev environment, designed for everyday tasks

    GitHub Next. Copilot workspace: An agentic dev environment, designed for everyday tasks. https://githubnext.com/projects/copilot-workspace, May 2025. Technical pre- view (sunset May 30, 2025)

    2025

  16. [24]

    B. Gou, Z. Huang, Y. Ning, Y. Gu, M. Lin, W. Qi, A. Kopanev, B. Yu, B. J. Gutiérrez, Y. Shu, C. H. Song, J. Wu, S. Chen, H. N. Moussa, T. Zhang, J. Xie, Y. Li, T. Xue, Z. Liao, K. Zhang, B. Zheng, Z. Cai, V. Rozgic, M. Ziyadi, H. Sun, and Y. Su. Mind2web 2: Evaluating agentic ...

    2025

  17. [25]

    Goyal, M

    N. Goyal, M. Chang, and M. Terry. Designing for human-agent alignment: Understanding what humans want from their agents. InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–6, 2024. 28

    2024

  18. [26]

    B. J. Grosz and S. Kraus. The evolution of sharedplans. InProceedings of the International Conference on Multi-Agent Systems, 1999

    1999

  19. [27]

    T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang. Large language model based multi-agents: A survey of progress and challenges.arXiv preprint arXiv:2402.01680, 2024

    2024 arXiv

  20. [28]

    H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu. Webvoyager: Building an end-to-end web agent with large multimodal models.arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  21. [29]

    H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu. Webvoyager: Building an end-to-end web agent with large multimodal models, 2024

    2024

  22. [30]

    S. Hong, X. Zheng, J. Chen, Y. Cheng, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, et al. Metagpt: Meta programming for multi-agent collaborative framework.arXiv preprint arXiv:2308.00352, 2023

    2023 arXiv

  23. [31]

    Huang, A

    K.-H. Huang, A. Prabhakar, S. Dhawan, Y. Mao, H. Wang, S. Savarese, C. Xiong, P. Laban, and C.-S. Wu. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments. InProceedings of the 2025 Conference of the Nations of the Amer...

    2025

  24. [32]

    Huang, A

    K.-H. Huang, A. Prabhakar, O. Thorat, D. Agarwal, P. K. Choubey, Y. Mao, S. Savarese, C. Xiong, and C.-S. Wu. Crmarena-pro: Holistic assessment of llm agents across diverse business scenarios and interactions.arXiv preprint arXiv:2505.18878, 2025

    2025 arXiv

  25. [33]

    F. Huq, Z. Z. Wang, F. F. Xu, T. Ou, S. Zhou, J. P. Bigham, and G. Neubig. Cowpilot: A framework for autonomous and human-agent collaborative web navigation.arXiv preprint arXiv:2501.16609, 2025

    2025

  26. [34]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024

    2024

  27. [35]

    J. Y. Koh, S. McAleer, D. Fried, and R. Salakhutdinov. Tree search for language model agents, 2024

    2024

  28. [36]

    Li and J

    E. Li and J. Waldo. Websuite: Systematically evaluating why web agents fail.arXiv preprint arXiv:2406.01623, 2024

    2024 arXiv

  29. [37]

    G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. Camel: Communicative agents for "mind" exploration of large scale language model society, 2023

    2023

  30. [38]

    W. Li, W. Bishop, A. Li, C. Rawles, F. Campbell-Ajala, D. Tyamagundlu, and O. Riva. On the effects of data scale on computer control agents.arXiv preprint arXiv:2406.03679, 2024

    2024 arXiv

  31. [39]

    Liang, Z

    T. Liang, Z. He, W. Jiao, X. Wang, Y. Wang, R. Wang, Y. Yang, Z. Tu, and S. Shi. Encouraging divergent thinking in large language models through multi-agent debate, 2023

    2023

  32. [40]

    Z. Liao, J. Jones, L. Jiang, E. Fosler-Lussier, Y. Su, Z. Lin, and H. Sun. Redteamcua: Realistic adversarial testing of computer-use agents in hybrid web-os environments, 2025

    2025

  33. [41]

    J. Liu, Y. Song, B. Y. Lin, W. Lam, G. Neubig, Y. Li, and X. Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding?, 2024

    2024

  34. [42]

    N. Liu, L. Chen, X. Tian, W. Zou, K. Chen, and M. Cui. From llm to conversational agent: A memory enhanced architecture with fine-tuning of large language models.arXiv e-prints, pages arXiv–2401, 2024. 29

    2024

  35. [43]

    Y. Liu, S. K. Lo, Q. Lu, L. Zhu, D. Zhao, X. Xu, S. Harrer, and J. Whittle. Agent design pattern catalogue: A collection of architectural patterns for foundation model based agents. arXiv preprint arXiv:2405.10467, 2024

    2024 arXiv

  36. [44]

    Madras, T

    D. Madras, T. Pitassi, and R. Zemel. Predict responsibly: improving fairness and accuracy by learning to defer.Advances in neural information processing systems, 31, 2018

    2018

  37. [45]

    Masterman, S

    T. Masterman, S. Besen, M. Sawtell, and A. Chao. The landscape of emerging ai agent archi- tectures for reasoning, planning, and tool calling: A survey.arXiv preprint arXiv:2404.11584, 2024

    2024 arXiv

  38. [46]

    B. Messing. An introduction to multiagent systems.Künstliche Intell., 17:58–, 2002

    2002

  39. [47]

    Mialon, C

    G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom. Gaia: a benchmark for general ai assistants, 2023

    2023

  40. [48]

    Mialon, C

    G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom. Gaia: benchmark for general ai assistants.arXiv preprint arXiv:2311.12983, 2023

    2023 arXiv

  41. [49]

    AutoGen AgentChat User Guide

    Microsoft. AutoGen AgentChat User Guide. Microsoft, 2025. Accessed: 2025-07-07

    2025

  42. [50]

    MarkItDown: Python tool for converting files and office documents to Markdown

    Microsoft. MarkItDown: Python tool for converting files and office documents to Markdown. https://github.com/microsoft/markitdown, May 2025. Version 0.1.2

    2025

  43. [51]

    Mozannar

    H. Mozannar. Web agent tutorial.husseinmozannar.github.io, June 2025

    2025

  44. [52]

    Mozannar, J

    H. Mozannar, J. J. Lee, D. Wei, P. Sattigeri, S. Das, and D. Sontag. Effective human-ai teams via learned natural language rules and onboarding, 2023

    2023

  45. [53]

    Mozannar and D

    H. Mozannar and D. Sontag. Consistent estimators for learning to defer to an expert. In International conference on machine learning, pages 7076–7087. PMLR, 2020

    2020

  46. [54]

    Müller and G

    M. Müller and G. Žunič. Browser use: Enable ai to control your browser, 2024

    2024

  47. [55]

    Narasimhan, J

    K. Narasimhan, J. Yang, H. Chen, and S. Yao. Webshop: Towards scalable real-world web interaction with grounded language agents.ArXiv, abs/2207.01206, 2022

    2022 arXiv

  48. [56]

    Introducing deep research, 2025

    OpenAI. Introducing deep research, 2025

    2025

  49. [57]

    Introducing operator, jan 2025

    OpenAI. Introducing operator, jan 2025

    2025

  50. [58]

    B. Pan, J. Lu, K. Wang, L. Zheng, Z. Wen, Y. Feng, M. Zhu, and W. Chen. Agentcoord: Visually exploring coordination strategy for llm-based multi-agent collaboration.arXiv preprint arXiv:2404.11943, 2024

    2024 arXiv

  51. [59]

    J. Pan, Y. Zhang, N. Tomlin, Y. Zhou, S. Levine, and A. Suhr. Autonomous evaluation and refinement of digital agents, 2024

    2024

  52. [60]

    Y. Pan, D. Kong, S. Zhou, C. Cui, Y. Leng, B. Jiang, H. Liu, Y. Shang, S. Zhou, T. Wu, and Z. Wu. Webcanvas: Benchmarking web agents in online environments, 2024

    2024

  53. [61]

    Paranjape, S

    B. Paranjape, S. Lundberg, S. Singh, H. Hajishirzi, L. Zettlemoyer, and M. T. Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models.arXiv preprint arXiv:2303.09014, 2023

    2023 arXiv

  54. [62]

    D. Paul, M. Ismayilzada, M. Peyrard, B. Borges, A. Bosselut, R. West, and B. Faltings. REFINER: Reasoning feedback on intermediate representations. In Y. Graham and M. Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computatio...

    2024

  55. [63]

    S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer. The impact of ai on developer productivity: Evidence from github copilot, 2023

    2023

  56. [64]

    Putta, E

    P. Putta, E. Mills, N. Garg, S. Motwani, C. Finn, D. Garg, and R. Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents, 2024

    2024

  57. [65]

    Y. Qin, S. Hu, Y. Lin, W. Chen, N. Ding, G. Cui, Z. Zeng, Y. Huang, C. Xiao, C. Han, Y. R. Fung, Y. Su, H. Wang, C. Qian, R. Tian, K. Zhu, S. Liang, X. Shen, B. Xu, Z. Zhang, Y. Ye, B. Li, Z. Tang, J. Yi, Y. Zhu, Z. Dai, L. Yan, X. Cong, Y. Lu, W. Zhao, Y. Huang, J. Yan, X. Ha...

    2023

  58. [66]

    Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023

    2023

  59. [67]

    Trase tops gaia leaderboard, 2024

    Red Cell Partners. Trase tops gaia leaderboard, 2024

    2024

  60. [68]

    Sarch, S

    G. Sarch, S. Somani, R. Kapoor, M. J. Tarr, and K. Fragkiadaki. Helper-x: A unified instructable embodied agent to tackle four interactive vision-language domains with memory- augmented language models, 2024

    2024

  61. [69]

    Sarch, Y

    G. Sarch, Y. Wu, M. Tarr, and K. Fragkiadaki. Open-ended instructable embodied agents with memory-augmented large language models. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3468–3500, Singapore, D...

    2023

  62. [70]

    Scerri, D

    P. Scerri, D. V. Pynadath, and M. Tambe. Adjustable autonomy in real-world multi-agent environments. InInternational Conference on Autonomous Agents, 2001

    2001

  63. [71]

    Schick, J

    T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools, 2023

    2023

  64. [72]

    Shaikh, K

    O. Shaikh, K. Gligorić, A. Khetan, M. Gerstgrasser, D. Yang, and D. Jurafsky. Grounding gaps in language model generations, 2024

    2024

  65. [73]

    Shaikh, H

    O. Shaikh, H. Mozannar, G. Bansal, A. Fourney, and E. Horvitz. Navigating rifts in human-llm grounding: Study and benchmark, 2025

    2025

  66. [74]

    Y. Shao, V. Samuel, Y. Jiang, J. Yang, and D. Yang. Collaborative gym: A framework for enabling and evaluating human-agent collaboration.arXiv preprint arXiv:2412.15701, 2024

    2024

  67. [75]

    Shavit, S

    Y. Shavit, S. Agarwal, M. Brundage, S. A. C. O’Keefe, R. Campbell, T. Lee, P. Mishkin, T. Eloundou, A. Hickey, K. Slama, L. Ahmad, P. McMillan, A. Beutel, A. Passos, and D. G. Robinson. Practices for governing agentic ai systems

  68. [76]

    T. Shi, A. Karpathy, L. Fan, J. Hernandez, and P. Liang. World of bits: An open-domain platform for web-based agents. InInternational Conference on Machine Learning. PMLR, 2017

    2017

  69. [77]

    C. Si, T. Hashimoto, and D. Yang. The ideation-execution gap: Execution outcomes of llm-generated versus human research ideas, 2025

    2025

  70. [78]

    Sodhi, S

    P. Sodhi, S. R. K. Branavan, Y. Artzi, and R. McDonald. Step: Stacked llm policies for web actions, 2024

    2024

  71. [79]

    Y. Song, D. Yin, X. Yue, J. Huang, S. Li, and B. Y. Lin. Trial and error: Exploration-based trajectory optimization for llm agents, 2024. 31

    2024

  72. [80]

    Stone and M

    P. Stone and M. Veloso. Multiagent systems: A survey from a machine learning perspective. Auton. Robots, 8(3):345–383, June 2000

    2000

  73. [81]

    Talebirad and A

    Y. Talebirad and A. Nadiri. Multi-agent collaboration: Harnessing the power of intelligent llm agents, 2023

    2023

  74. [82]

    M. Tambe. Implementing agent teams in dynamic multiagent environments.Appl. Artif. Intell., 12:189–210, 1998

    1998

  75. [83]

    Thomas, A

    G. Thomas, A. J. Chan, J. Kang, W. Wu, F. Christianos, F. Greenlee, A. Toulis, and M. Purtorab. Webgames: Challenging general-purpose web-browsing ai agents, 2025

    2025

  76. [84]

    Vaccaro, A

    M. Vaccaro, A. Almaatouq, and T. Malone. When combinations of humans and ai are useful: A systematic review and meta-analysis.Nature Human Behaviour, 8(12):2293–2303, Oct. 2024

    2024

  77. [85]

    Valmeekam, S

    K. Valmeekam, S. Sreedharan, M. Marquez, A. Olmo, and S. Kambhampati. On the planning abilities of large language models (a critical investigation with a proposed benchmark), 2023

    2023

  78. [86]

    Vijayvargiya, A

    S. Vijayvargiya, A. B. Soni, X. Zhou, Z. Z. Wang, N. Dziri, G. Neubig, and M. Sap. Ope- nagentsafety: A comprehensive framework for evaluating real-world ai agent safety.arXiv preprint arXiv:2507.06134, 2025

    2025

  79. [87]

    R. Wang, L. Zheng, and B. An. Synapse: Trajectory-as-exemplar prompting with memory for computer control. InInternational Conference on Learning Representations, 2023

    2023

  80. [88]

    X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, and H. Ji. Executable code actions elicit better llm agents, 2024

    2024

  81. [89]

    X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig. Openhands: An open platform for ai software develo...

    2025

  82. [90]

    Y. Wang, T. Shen, L. Liu, and J. Xie. Sibyl: Simple yet effective agent framework for complex real-world reasoning, 2024

    2024

  83. [91]

    Z. Z. Wang, J. Mao, D. Fried, and G. Neubig. Agent workflow memory, 2024

    2024

  84. [92]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022

    2022 arXiv

  85. [93]

    Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. InCOLM, 2024

    2024

  86. [94]

    Z. Wu, C. Han, Z. Ding, Z. Weng, Z. Liu, S. Yao, T. Yu, and L. Kong. Os-copilot: Towards generalist computer agents with self-improvement.ArXiv, abs/2402.07456, 2024

    2024 arXiv

  87. [95]

    Z. Wu, C. Han, Z. Ding, Z. Weng, Z. Liu, S. Yao, T. Yu, and L. Kong. Os-copilot: Towards generalist computer agents with self-improvement, 2024

    2024

  88. [96]

    Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y. Zhou, W. Wang, C. Jiang, Y. Zou, X. Liu, Z. Yin, S. Dou, R. Weng, W. Cheng, Q. Zhang, W. Qin, Y. Zheng, X. Qiu, X. Huang, and T. Gui. The rise and poten...

    2023

  89. [97]

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.ArXiv, abs/2404.07972, 2024

    2024 arXiv

  90. [98]

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    2024

  91. [99]

    M. Xing, R. Zhang, H. Xue, Q. Chen, F. Yang, and Z. Xiao. Understanding the weakness of large language model agents within a complex android environment. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6061–6072, 2024

    2024

  92. [100]

    F. F. Xu, Y. Song, B. Li, Y. Tang, K. Jain, M. Bao, Z. Z. Wang, X. Zhou, Z. Guo, M. Cao, M. Yang, H. Y. Lu, A. Martin, Z. Su, L. Maben, R. Mehta, W. Chi, L. Jang, Y. Xie, S. Zhou, and G. Neubig. Theagentcompany: Benchmarking llm agents on consequential real world tasks, 2024

    2024

  93. [101]

    Z. Xu, X. Yang, Y. Wang, Q. Hu, Z. Wu, L. Wang, W. Luo, K. Zhang, B. Hu, and M. Zhang. Comfyui-copilot: An intelligent assistant for automated workflow development.arXiv preprint arXiv:2506.05010, 2025

    2025 arXiv

  94. [102]

    T. Xue, W. Qi, T. Shi, C. H. Song, B. Gou, D. Song, H. Sun, and Y. Su. An illusion of progress? assessing the current state of web agents. 2025

    2025

  95. [103]

    J. Yang, C. E. Jimenez, A. Wettig, K. A. Lieret, S. Yao, K. Narasimhan, and O. Press. Swe-agent: Agent-computer interfaces enable automated software engineering. ArXiv, abs/2405.15793, 2024

    2024 arXiv

  96. [104]

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  97. [105]

    S. Yao, H. Chen, J. Yang, and K. Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023

    2023

  98. [106]

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. tau -bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    2024 arXiv

  99. [107]

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023

    2023

  100. [108]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Syner- gizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  101. [109]

    Yoran, S

    O. Yoran, S. J. Amouyal, C. Malaviya, B. Bogin, O. Press, and J. Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks?, 2024

    2024

  102. [110]

    A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang. Agenttuning: Enabling generalized agent abilities for llms, 2023

    2023

  103. [111]

    Zhang, Z

    Y. Zhang, Z. Ma, Y. Ma, Z. Han, Y. Wu, and V. Tresp. Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration, 2024

    2024

  104. [112]

    Zhang, E

    Z. Zhang, E. Schoop, J. Nichols, A. Mahajan, and A. Swearngin. From interaction to impact: Towards safer ai agent through understanding and evaluating mobile ui operation impacts. In Proceedings of the 30th International Conference on Intelligent User Interfaces, pages 727–744...

    2025

  105. [113]

    Zhang and A

    Z. Zhang and A. Zhang. You only look at screens: Multimodal chain-of-action agents, 2024

    2024

  106. [114]

    Zheng, J

    B. Zheng, J. Kil, H. Sun, Y. Su, and B. Gou. Gpt-4v(ision) is a generalist web agent, if grounded. ArXiv, abs/2401.01614, 2024

    2024 arXiv

  107. [115]

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig. Webarena: A realistic web environment for building autonomous agents, 2024

    2024

  108. [116]

    what are synonyms of interactive?

    Y. Zhu, A. Kellermann, D. Bowman, P. Li, A. Gupta, A. Danda, R. Fang, C. Jensen, E. Ihli, J. Benn, J. Geronimo, A. Dhir, S. Rao, K. Yu, T. Stone, and D. Kang. Cve-bench: A benchmark for ai agents’ ability to exploit real-world web application vulnerabilities, 2025. 34 A Overvi...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.