Pith. sign in

REVIEW 4 major objections 4 minor 40 references

Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A small open-weights model can power a state-of-the-art web agent at a fraction of the cost, if the benchmark measures what it claims.

desk verdict The 92.2% WebVoyager headline is not a held-out result — the paper's own ablation gives 4.5 points to in-domain training — but the open Holo1 weights and the WebClick benchmark are real, useful contributions that deserve a close look. read the letter →

arxiv 2506.02865 v2 pith:PQ7EZE5D submitted 2025-06-03 cs.AI

Mathieu Andreux , Breno Baldas Skuk , Hamza Benchekroun , Emilien Biré , Antoine Bonnet , Riaz Bordie , Nathan Bout , Matthias Brunel
show 36 more authors
Pierre-Louis Cedoz Antoine Chassang Mickaël Chen Alexandra D. Constantinou Antoine d'Andigné Hubert de La Jonquière Aurélien Delfosse Ludovic Denoyer Alexis Deprez Augustin Derupti Michael Eickenberg Mathïs Federico Charles Kantor Xavier Koegler Yann Labbé Matthew C. H. Lee Erwan Le Jumeau de Kergaradec Amir Mahla Avshalom Manevich Adrien Maret Charles Masson Rafaël Maurin Arturo Mena Philippe Modard Axel Moyal Axel Nguyen Kerbel Julien Revelle Mats L. Richter María Santos Laurent Sifre Maxime Theillard Marc Thibault Louis Thiry Léo Tronchon Nicolas Usunier Tony Wu
This is my paper
classification cs.AI
keywords webagentvision-languagemodelGUIgroundingUIlocalizationbenchmarkopen-weightcostefficiencybehavioralcloning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a web agent can reach state-of-the-art performance on WebVoyager using only screenshots and an open-weights 7-billion-parameter vision-language model, at an average cost of about $0.13 per task. The mechanism is Surfer-H, a three-module agent (policy, localizer, validator) paired with Holo1, a VLM family fine-tuned from Qwen2.5-VL on a large mixture of web-grounding, synthetic, and agent-trajectory data. If true, this would push the Pareto frontier of accuracy versus cost for computer-use agents, beating reported numbers from proprietary counterparts. The paper also introduces WebClick, a web-specific localization benchmark, and releases both the model weights and the benchmark. Its own ablation shows that part of the gain comes from training on WebVoyager traces, so the headline number is not a clean held-out evaluation.

What carries the argument

The load-bearing mechanism is the Surfer-H loop: at each step the policy VLM predicts thought, optional note, and next action from the task, memory, and recent screenshots; if the action is a click or a write, a localizer VLM turns the element description into pixel coordinates; when the policy emits an answer, a validator VLM scores it against the task and screenshots and, on rejection, feeds the feedback into memory for another attempt. Holo1 is a single VLM family trained to serve all three roles, and its training mixture—particularly the filtered behavioral cloning of successful traces and the 5M-triplet coordinate-validation set—is what the paper credits for state-of-the-art localization and policy behavior.

What would settle it

Run Surfer-H with Holo1-7B (trained with the full mixture) on WebVoyager-style tasks drawn from websites the agent never saw in training, or on an independent benchmark such as VisualWebArena; if accuracy falls to the ~87.7% level of the model trained without WebVoyager traces, the 92.2% headline is substantially a result of in-domain training rather than general web competence.

Watch

Extended reading notes

Core claim

Surfer-H is a screenshot-only web agent whose policy emits thoughts and actions, whose localizer converts element descriptions into coordinates, and whose validator checks final answers, with all modules able to share a single VLM. Holo1 is that VLM family: starting from Qwen2.5-VL-Instruct, it is fine-tuned on a 31.5B-token mixture that is 50.8% GUI-grounding data, 32.3% complex-visual-understanding data (including a novel 5M-sample coordinate-validation task and 7M-page UI extraction), and 16.9% behavior data from successful agent traces, plus a 1M-pair validator corpus. The paper's central result is that Surfer-H with Holo1-7B as policy and localizer and GPT-4o as validator attains 92.2% accuracy on all 643 WebVoyager tasks after up to 10 attempts, at an estimated $0.13 per task, outperforming the reported Operator (87.0%), Mariner (83.5%), and BrowserUse (89.1%) figures and matching GPT-4.1-driven Surfer-H (92.0%) at a quarter of the cost. It also reports that Holo1 tops average accuracy on localization benchmarks including the new WebClick set, and that a fully self-hosted Holo1-7B agent reaches 80.4% at $0.06 per task. The authors attribute the results to the training mixture, especially filtered behavioral cloning of successful agent traces, and release WebClick and the Holo1 weights.

Load-bearing premise

The result stands on the assumption that a model trained partly on traces of WebVoyager tasks can be measured on WebVoyager as a test of generalization; the paper's own ablation suggests removing those traces costs 4.5 points of accuracy.

Editorial extensions

If this is right

  • If accurate, the result means a 7B-parameter open model can substitute for frontier API models as the policy of a computer-use agent, cutting per-task cost from $0.54 to $0.13 at matched WebVoyager accuracy.
  • A screenshot-only interface without DOM or accessibility trees is enough for near-state-of-the-art web navigation, which removes a dependency on site-specific integrations and makes the agent portable to any graphical interface.
  • Filtered behavioral cloning of successful traces is a transferable training strategy: it contributes a 9.5-point gain over the base model, and adding in-domain WebVoyager traces contributes another 4.5 points.
  • The WebClick benchmark offers a compact, web-specific measure of localization skill, on which Holo1 models lead their size class; this makes localization an isolatable and optimizable component of agent performance.
  • Fully self-hosted deployment is feasible: Surfer-H with Holo1 in all three roles stays on the Pareto front at $0.06 per task, though with a 12-point accuracy drop, which locates validation as the current bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 4.5-point gap between Holo1-7B and Holo1-7B-WVE is a direct estimate of how much of the 92.2% rests on having trained on the evaluation benchmark's own tasks; I would expect a genuinely unseen set of comparable web tasks to land closer to the 87.7% figure.
  • The drop observed when Holo1 replaces GPT-4o as validator suggests answer verification is harder than localization or action selection for small models, and might respond to larger validation-specific training sets or a stronger validator model.
  • The Pareto-curve methodology (accuracy against average cost per task as attempts vary) is transferable to other agent frameworks as a practical way to pick a deployment point given a budget.
  • A direct testable extension: applying the same training mixture and three-module design to other agent benchmarks such as VisualWebArena or WebArena would show whether the GUI-grounding gains generalize beyond WebVoyager-style tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Surfer-H, a modular web agent composed of a policy, a localizer, and a validator, and Holo1, a family of open-weight VLMs fine-tuned from Qwen2.5-VL. It also introduces WebClick, a new web-localization benchmark. The headline claim is that Surfer-H powered by Holo1-7B reaches 92.2% accuracy on WebVoyager at an average cost of $0.13 per task, presented as state-of-the-art and Pareto-optimal. However, the WebVoyager task corpus is explicitly included in the policy training mixture (Section 3.5), and the paper's own ablation (Table 4) shows that removing those traces lowers accuracy by 4.5 points to 87.7%. The 92.2% result is therefore not a held-out measurement. The localization results on public benchmarks and the open release of model weights and WebClick are useful contributions, but they do not support the WebVoyager SOTA claim as stated.

Significance. If the WebVoyager result were a clean held-out evaluation, demonstrating that a 7B open-weights model can outperform proprietary agents such as OpenAI Operator and BrowserUse at a fraction of the cost would be an important result. The paper also provides publicly released weights and a new benchmark, and the localization improvements on Screenspot and GroundUI are credible evidence of progress on UI grounding. However, the central SOTA claim is compromised by in-domain training on the evaluation corpus, and the paper's own ablation quantifies the contamination. The remaining contributions are genuine but incremental; the headline result as framed is not supported.

major comments (4)
  1. [Section 3.5, Table 4] The headline 92.2% WebVoyager score is not a held-out result. Section 3.5 states that the policy training mixture includes agent traces generated on the WebVoyager task corpus, and Table 4 shows that removing those traces (Holo1-7B-WVE) lowers WebVoyager accuracy from 92.2% to 87.7%. Since the evaluation is run on exactly the WebVoyager task set, this 4.5-point difference constitutes training on the test distribution. Because this difference is larger than the 3.1-point margin over the strongest reported baseline (BrowserUse at 89.1%), the abstract's 'state-of-the-art' and Pareto-optimality claims are not robust once the contamination is accounted for.
  2. [Section 5.2, 'Performance on Unseen Tasks'] This subsection is misnamed. It compares a model trained on WebVoyagerExtended only (Holo1-7B-WVE) with one additionally trained on WebVoyager traces (Holo1-7B), and evaluates both on WebVoyager. There is no evaluation on a task set absent from both training mixtures, so the experiment does not demonstrate generalization to unseen tasks; it demonstrates the effect of adding the evaluation corpus to training.
  3. [Section 4.2, Table 2] The WebClick benchmark's agent subset is built from agent attempts on WebVoyager tasks. Since Holo1 is trained on WebVoyager traces (Section 3.5), the WebClick agent-subset scores (e.g., Holo1-7B at 89.77% vs Qwen2.5-VL-7B at 78.47%) are not an independent localization check. The human and calendar subsets are less exposed to this issue, but the average scores in Table 2 and Figure 2 include the contaminated agent subset, so the claimed localization advantage is also partly confounded.
  4. [Section 5.1, Table 5] The external baselines (BrowserUse, Operator, Mariner) are taken from reported numbers, computed at a different time, with different websites and evaluation functions, as the paper itself notes in Section 5.1. Combined with the in-domain training issue, this does not support the claim that Surfer-H+Holo1 is Pareto-optimal against these systems. A controlled re-evaluation under the same harness would be needed to make such a comparison.
minor comments (4)
  1. [Section 3.5 vs Section 5.1] Section 3.5 describes WebVoyager as comprising 643 tasks on 15 common websites, while Section 5.1 says 'all 643 tasks from 10 different websites.' Please reconcile these numbers.
  2. [Section 4.3 vs Table 2] The reported average localization scores (73.55% for Holo1-3B and 76.16% for Holo1-7B) do not match the arithmetic means of the seven columns in Table 2 (73.00% and 76.19%, respectively). Please correct the text or the table.
  3. [Section 5.1, Table 3] The cost comparison relies on internal estimates for Holo1 and Qwen2.5-VL inference costs; it would be helpful to state the sensitivity of the Pareto-front conclusions to these estimates.
  4. [Figure 3] The plot legend is dense and some points are not individually readable; consider labeling the Surfer-H/Holo1 points directly or adding a companion table with exact values.

Circularity Check

2 steps flagged · score 7.0 of 10
  1. fitted input called prediction [Section 3.5 (Behavior Learning) and Section 5.2 / Table 4]
    ""Agent trajectories for this dataset were generated using two task corpora. The first is WebVoyager [11], which comprises 643 tasks on 15 common websites, mostly consumer-facing." ... "We evaluate Surfer-H and external competitors on the WebVoyager benchmark [11], using all 643 tasks from 10 different websites." ... "The performance difference between Holo1-7B-WVE and Holo1-7B (4.5 percentage points) illustrates the added benefit of in-domain experience.""

    Holo1-7B, which attains the headline 92.2% WebVoyager accuracy, was trained on agent traces generated on the WebVoyager task corpus (Section 3.5), and the headline result is measured on that same task corpus (Section 5.1). Table 4 explicitly attributes 4.5 percentage points to this overlap: with only WebVoyagerExtended traces, accuracy is 87.7%; adding WebVoyager traces yields 92.2%. The claimed 3.1-point margin over BrowserUse is therefore smaller than the in-domain training contribution, so the reported SOTA ordering is not a held-out generalization result.

  2. other [Section 4.2 (WebClick construction) and Table 2]
    ""We carefully curated this benchmark dataset from three sources: (1) data collected by our agents while attempting to solve WebVoyager tasks [11], (2) human interactions with the Web during everyday tasks, and (3) human interactions with calendar interfaces.""

    WebClick's agent-sourced subset is collected from the same WebVoyager task corpus whose traces are used to train Holo1 in Section 3.5. The agent and calendar columns of WebClick are then reported (Table 2) as evidence of Holo1's web localization skill. For the agent subset, this is partly in-distribution evaluation: the same underlying task corpus supplies both training traces and benchmark instances. The human and calendar components are independent, so this is a secondary contamination rather than the central claim, but it weakens the claim that WebClick measures generalization to unseen web interactions.

full rationale

The central WebVoyager claim is partially circular: the model achieving the reported 92.2% was trained on agent trajectories from the WebVoyager task corpus, and the evaluation is performed on that same corpus. The paper's own Table 4 quantifies this: removing WebVoyager traces drops the score by 4.5 points, from 92.2% to 87.7%. Since that 4.5-point in-domain contribution exceeds the 3.1-point margin over BrowserUse, the state-of-the-art comparison is not robust to the training/evaluation overlap. The WebClick benchmark similarly includes an agent subset built from WebVoyager attempts, so it is not fully independent for a model trained on the same traces. I do not find self-citation load-bearing or uniqueness-imported-from-authors circularity; the open release of weights and WebClick, and the independent Screenspot/GroundUI localization results, provide real external content. The score is 7 rather than higher because Holo1-7B-WVE still improves 9.5 points over the Qwen baseline without any WebVoyager traces, showing that a substantial portion of the gain is cross-domain and not simply memorization of the evaluation set.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the validity of the WebVoyager and WebClick benchmarks as held-out tests, on the reliability of the GPT-4o judge, on the internal cost model, and on the comparability of external baseline results. The most important dependence, that the evaluation tasks were included in training, is documented in the paper (Sections 3.5 and 4.2) and quantified by the ablation in Table 4.

free parameters (5)
  • Training mixture proportions = 50.79% GUI grounding, 32.28% complex visual understanding, 16.93% behavior learning
    Hand-chosen token percentages in Table 1; no systematic search over mixture weights is described, and the ablation (Table 4) shows the mixture composition changes WebVoyager accuracy by 4.5 points.
  • Maximum attempts = 1, 2, 5, 10 (10 used for the headline)
    Evaluation configuration; the 92.2% headline uses 10 attempts, and lower attempt counts give lower scores (Table 5).
  • Maximum steps per attempt = 30
    Evaluation configuration that bounds episode length; not varied in the paper.
  • Internal per-token cost estimates = Holo1-3B: $0.1/M in, $0.4/M out; Holo1-7B: $0.15/M in, $0.6/M out
    Internal estimates used in Table 3 and cost-per-task figures; if actual serving cost differs, the Pareto-optimality claim changes.
  • GPT-4o judge majority sampling = 3 samples
    Success is computed as a majority vote of three GPT-4o samples (Section 5.1); the judge prompt and decision threshold are not specified.
assumptions (4)
  • domain assumption WebVoyager success as measured by a GPT-4o majority vote is a valid proxy for real-world web task completion
    Used throughout Section 5 without independent validation of the judge's reliability.
  • domain assumption A screenshot-only interface contains enough information for web navigation, so the DOM/accessibility tree can be ignored
    Stated in Section 1 and assumed by the Surfer-H architecture.
  • domain assumption Reported numbers for Operator, Mariner, and BrowserUse are comparable to numbers measured in this paper, despite different evaluation times and functions
    Section 5.1 acknowledges the differences but still uses these numbers for Pareto-optimality claims in Figure 3.
  • ad hoc to paper Training on a benchmark is a legitimate way to report 'state-of-the-art' on that benchmark
    The paper trains Holo1 on WebVoyager traces (Section 3.5), then reports 92.2% on WebVoyager without presenting it as in-domain performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights." pith.science (2026). https://pith.science/paper/PQ7EZE5D

@misc{pith2026250602865,
  author       = {Pith},
  title        = {Pith review of: Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PQ7EZE5D}},
  note         = {Machine review of arXiv:2506.02865}
}
read the original abstract

We present Surfer-H, a cost-efficient web agent that integrates Vision-Language Models (VLM) to perform user-defined tasks on the web. We pair it with Holo1, a new open-weight collection of VLMs specialized in web navigation and information extraction. Holo1 was trained on carefully curated data sources, including open-access web content, synthetic examples, and self-produced agentic data. Holo1 tops generalist User Interface (UI) benchmarks as well as our new web UI localization benchmark, WebClick. When powered by Holo1, Surfer-H achieves a 92.2% state-of-the-art performance on WebVoyager, striking a Pareto-optimal balance between accuracy and cost-efficiency. To accelerate research advancement in agentic systems, we are open-sourcing both our WebClick evaluation dataset and the Holo1 model weights.

Figures

Figures reproduced from arXiv: 2506.02865 by the authors.

Figure 1
Figure 1. Surfer-H operates via screenshots and a limited action set. It maintains an internal memory with the task, recent screenshots, and thought history. The policy generates thoughts and selects the next action. If necessary, the localizer refines the coordinates for clicks or typing. The validator filters answers, gives feedback, and decides whether a task is complete. Memory The past actions are stored in the agent’s i… view at source ↗
Figure 2
Figure 2. Holo1 as Localizer: comparison against competitors, for external and internal benchmarks. Holo1 models reach state-of-the-art average localization performance at all model scales. over Qwen2.5-VL-3B, UGround-V1-2B and UI-TARS-2B on the human-based elements of WebClick, as well as on all Screenspot variants. Furthermore, Holo1-3B not only achieves the highest average localization performance of the 2B and 3B models, … view at source ↗
Figure 3
Figure 3. Pareto-Optimality of Surfer-H+Holo1. Surfer-H success is plotted against cost for varying maximum allowed attempts before the agent must respond, and different underlying policy modules (Holo1, GPT, Gemini, or Qwen2.5-VL). For BrowserUse [3], OpenAI Operator [23] and Project Mariner [8], we use reported numbers (* superscript). Surfer-H powered by Holo1 models reaches state-of-the-art performance while being the mos… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 28 canonical work pages

  1. [1]

    Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku.https://www

    Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku.https://www. anthropic.com/news/3-5-models-and-computer-use, 2024. Accessed: May 23, 2025

    work page 2024

  2. [2]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-VL Technical Report, 2025

    work page 2025

  3. [3]

    Browser use: Sota technical report.https://browser-use.com/posts/ sota-technical-report, 2024

    Browser Use Team. Browser use: Sota technical report.https://browser-use.com/posts/ sota-technical-report, 2024. Accessed May 23, 2025

    work page 2024

  4. [4]

    Cheng, Q

    K. Cheng, Q. Sun, Y. Chu, F. Xu, L. YanTao, J. Zhang, and Z. Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313–9332, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics

    work page 2024

  5. [5]

    Y. Dong, C. F. Ruan, Y. Cai, R. Lai, Z. Xu, Y. Zhao, and T. Chen. Xgrammar: Flexible and efficient structured generation engine for large language models.arXiv preprint arXiv:2411.15100, 2024

    arXiv 2024

  6. [6]

    Gemini 2.0 Flash Docs.https://ai.google.dev/gemini-api/docs/pricing# gemini-2.0-flash

    Google Deepmind. Gemini 2.0 Flash Docs.https://ai.google.dev/gemini-api/docs/pricing# gemini-2.0-flash. Accessed: May 23, 2025

    work page 2025

  7. [7]

    Introducing gemini 2.0: our new ai model for the agentic era.https://blog.google/ technology/google-deepmind/google-gemini-ai-update-december-2024/, December 2024

    Google DeepMind. Introducing gemini 2.0: our new ai model for the agentic era.https://blog.google/ technology/google-deepmind/google-gemini-ai-update-december-2024/, December 2024. Ac- cessed: May 23, 2025

    work page 2024

  8. [8]

    Google Deepmind. Project mariner: agents that can help you accomplish complex tasks.https: //blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/ #agents-for-developers, 2024. Accessed May 23, 2025

    work page 2024

Show all 40 references
  1. [9]

    B. Gou, R. Wang, B. Zheng, Y. Xie, C. Chang, Y. Shu, H. Sun, and Y. Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  2. [10]

    Hartvigsen, S

    T. Hartvigsen, S. Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar. Toxigen: A large-scale machine- generated dataset for adversarial and implicit hate speech detection.arXiv preprint arXiv:2203.09509, 2022

    2022 arXiv

  3. [11]

    H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu. WebVoyager: Building an end- to-end web agent with large multimodal models. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguist...

    2024

  4. [12]

    D. M. Katz, M. J. Bommarito, S. Gao, and P. Arredondo. Gpt-4 passes the bar exam.Philosophical Transactions of the Royal Society A, 2024

    2024

  5. [13]

    J. Y. Koh, R. Lo, L. Jang, V. Duvvur, M. C. Lim, P.-Y. Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.arXiv preprint arXiv:2401.13649, 2024

    2024 arXiv

  6. [14]

    Lauren¸ con, L

    H. Lauren¸ con, L. Tronchon, M. Cord, and V. Sanh. What matters when building vision-language models?arXiv preprint arXiv:2405.02246, 2024

    2024 arXiv

  7. [15]

    Lavague: Web agent framework for builders.https://docs.lavague.ai/en/latest/,

    LaVagueAI. Lavague: Web agent framework for builders.https://docs.lavague.ai/en/latest/,

  8. [16]

    K. Li, Z. Meng, H. Lin, Z. Luo, Y. Tian, J. Ma, Z. Huang, and T.-S. Chua. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use.arXiv preprint arXiv:2504.07981, 2025

    2025 arXiv

  9. [17]

    M¨ uller and G

    M. M¨ uller and G. ˇZuniˇ c. Browser use: Enable ai to control your browser.https://github.com/ browser-use/browser-use, 2024

    2024

  10. [18]

    Nakano, J

    R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. WebGPT: Browser-assisted question-answering with human feedback.arXiv preprint ...

    2022 arXiv

  11. [19]

    Gpt-4.1 documentation.https://platform.openai.com/docs/models/gpt-4.1

    OpenAI. Gpt-4.1 documentation.https://platform.openai.com/docs/models/gpt-4.1. Accessed: May 23, 2025

    2025

  12. [20]

    Gpt-4.1-mini documentation.https://platform.openai.com/docs/models/gpt-4.1-mini

    OpenAI. Gpt-4.1-mini documentation.https://platform.openai.com/docs/models/gpt-4.1-mini. Accessed: May 23, 2025

    2025

  13. [21]

    Gpt-4o documentation.https://platform.openai.com/docs/models/gpt-4o

    OpenAI. Gpt-4o documentation.https://platform.openai.com/docs/models/gpt-4o. Accessed: May 23, 2025

    2025

  14. [22]

    Gpt-4o-mini documentation.https://platform.openai.com/docs/models/gpt-4o-mini

    OpenAI. Gpt-4o-mini documentation.https://platform.openai.com/docs/models/gpt-4o-mini. Accessed: May 23, 2025

    2025

  15. [23]

    https://openai.com/index/introducing-operator/

    OpenAI. https://openai.com/index/introducing-operator/. OpenAI Blog, Jan. 2025

    2025

  16. [24]

    Introducing deep research.https://openai.com/index/introducing-deep-research/, Feb

    OpenAI. Introducing deep research.https://openai.com/index/introducing-deep-research/, Feb. 2025

    2025

  17. [25]

    OpenAIet al.GPT-4 Technical Report, 2024

    2024

  18. [26]

    Introducing perplexity deep research.https://www.perplexity.ai/hub/blog/ introducing-perplexity-deep-research, 2025

    Perplexity Team. Introducing perplexity deep research.https://www.perplexity.ai/hub/blog/ introducing-perplexity-deep-research, 2025. Accessed May 23, 2025

    2025

  19. [27]

    Y. Qin, Y. Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y. Li, S. Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  20. [28]

    Rawles, S

    C. Rawles, S. Clinckemaillie, Y. Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. E. Bishop, W. Li, F. Campbell-Ajala, D. K. Toyama, R. J. Berry, D. Tyamagundlu, T. P. Lillicrap, and O. Riva. Android- world: A dynamic benchmarking environment for autonomous agents. InThe Thirteenth...

    2025

  21. [29]

    Meet replit ghostwriter, your partner in code.https://blog.replit.com/ghostwriter, October 2022

    Replit Team. Meet replit ghostwriter, your partner in code.https://blog.replit.com/ghostwriter, October 2022. Accessed: May 23, 2025

    2022

  22. [30]

    Rohrbach, L

    A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko. Object hallucination in image captioning. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors,Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045,...

    2018

  23. [31]

    Schick, J

    T. Schick, J. Dwivedi-Yu, R. Dess ` ı, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools.arXiv:2302.04761, 2023

    2023 arXiv

  24. [32]

    Q. Shi, M. Tang, K. Narasimhan, and S. Yao. Can language models solve olympiad programming? arXiv preprint arXiv:2404.10952, 2024

    2024 arXiv

  25. [33]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  26. [34]

    Z. Wu, Z. Wu, F. Xu, Y. Wang, Q. Sun, C. Jia, K. Cheng, Z. Ding, L. Chen, P. P. Liang, and Y. Qiao. OS- ATLAS: Foundation action model for generalist GUI agents. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  27. [35]

    Xiao and W

    Y. Xiao and W. Y. Wang. On hallucination and predictive uncertainty in conditional language gener- ation. In P. Merlo, J. Tiedemann, and R. Tsarfaty, editors,Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volum...

    2021

  28. [36]

    F. F. Xu, Y. Song, B. Li, Y. Tang, K. Jain, M. Bao, Z. Z. Wang, X. Zhou, Z. Guo, M. Cao, M. Yang, H. Y. Lu, A. Martin, Z. Su, L. Maben, R. Mehta, W. Chi, L. Jang, Y. Xie, S. Zhou, and G. Neubig. Theagentcompany: Benchmarking llm agents on consequential real world tasks, 2025

    2025

  29. [37]

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  30. [38]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing Reasoning and Acting in Language Models.arXiv preprint arXiv:2210.03629, 2023

    2023 arXiv

  31. [39]

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig. WebArena: A realistic web environment for building autonomous agents.ICLR, 2024. 14

    2024

  32. [2025]

    Accessed: May 23, 2025

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.