Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that GUI grounding can reach state-of-the-art accuracy using only 20K training images by combining self-generated reasoning, spatial-consistency fine-tuning, and KDE-based spatial search.

desk verdict Solid method paper, but the headline '500x less data' claim conflates training data with test-time search compute and does not survive a controlled comparison. read the letter →

arxiv 2505.15259 v2 pith:PZWNAE57 submitted 2025-05-21 cs.LG cs.CL

Hyunseok Lee , Jeonghoon Kim , Beomjun Kim , Jihoon Tack , Chansong Jo , Jaehong Lee , Cheonbok Park , Sookyo In
show 2 more authors
Jinwoo Shin Kang Min Yoo
This is my paper
classification cs.LGcs.CL
keywords GUIgroundingreinforcementlearningspatialconsistencytest-timescalingkerneldensityestimationdataefficiencymultimodallargelanguagemodelscoordinateprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ReGUIDE is a training and inference framework that lets a multimodal language model localize GUI elements on a screen using a tiny fraction of the data that previous grounding models needed. The paper claims that a 20K image-subset from UGround, about 0.2% of the data used by the strongest open baselines, is enough to reach or beat those baselines, because the model learns to generate its own spatial reasoning rather than memorizing coordinates. Concretely, on ScreenSpot the method lifts Qwen-2.5-VL-3B from 55.5% to 87.3% accuracy, and on the harder high-resolution ScreenSpot-Pro from 23.9% to 44.3%. The authors argue that the gain comes from three mechanisms working together: online reinforcement learning that rewards pointing inside the target box, a consistency fine-tuning stage that aligns predictions on full and cropped views, and a test-time search that crops to a density peak and then votes among several predictions. If correct, this means expensive, million-scale GUI annotation is not an inherent requirement for strong grounding.

What carries the argument

The load-bearing mechanism is a three-part loop: (1) self-generated reasoning through online RL, where GRPO samples multiple (reasoning, coordinate) outputs per image and rewards any point inside the target box, so the model learns to articulate spatial cues without any language annotation; (2) spatial-consistency fine-tuning, which takes a correct global-view sample and trains the same reasoning tokens to predict the transformed center on a random crop, enforcing equivariance under cropping; and (3) test-time spatial search, in which kernel density estimation, a smooth voting rule that sums 2D Gaussian bumps around sampled points, locates the mode of the model's coordinate distribution, crops the image there, and then aggregates a second round of samples into a final coordinate. The KDE step is what turns many noisy token-level guesses into one robust point, and it is the component that produces the larger jump on high-resolution screenshots.

What would settle it

Replicate the two-stage training but replace the reasoning text in the local-view samples with a mismatched or empty string while keeping the coordinates correct; if the consistency stage still yields the reported 84.9% on ScreenSpot (versus 83.3% for RL-only), the shared-reasoning assumption is not what carries the gain. A complementary check is the paper's own comparison: mixing the same crops into the RL stage gives 83.2%, so a replication that finds no gap between RL-with-augmentation and RL-plus-consistency-finetuning would falsify the claim that the consistency stage adds real signal.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that GUI coordinate grounding can be made data-efficient by turning the model into a self-explainer and then a self-critic. Instead of supervising the model with human-written explanations, ReGUIDE first runs GRPO-style reinforcement learning where the reward is simply whether the sampled point lies inside the ground-truth bounding box, plus a small formatting reward for enclosing the reasoning in <think> tags. This makes the model generate diverse reasoning traces and keep the ones that lead to correct points. In a second stage, each correct global-view sample is reused as supervision for a randomly cropped local view that shares the same instruction and reasoning text but has transformed coordinates; the model is trained to predict the center of the target in both views, enforcing spatial consistency. At inference, ReGUIDE samples several coordinates, fits a 2D kernel density estimate, crops the image around the density peak, samples again inside that crop, and returns the highest-density point. The paper reports that this pipeline turns a 20K-sample fine-tune of Qwen-2.5-VL into a grounding model that outperforms UGround-7B, which was trained on 10M samples, on ScreenSpot and ScreenSpot-Pro, and that the same grounding improvements transfer to agentic task success rates on Mind2Web and AndroidControl.

Load-bearing premise

The load-bearing premise is that the reasoning text the model generates for the full screenshot remains valid supervision when the model is shown only a random crop centered on the target, even though the crop changes the coordinate frame and hides the surrounding layout the reasoning refers to.

Editorial extensions

If this is right

  • If the central claim holds, a 20K-image subset of UGround is sufficient to match or beat models trained on 10M images, so the bottleneck shifts from data collection to the training recipe.
  • The consistency stage means a model can be aligned across views using only spatial priors, with no extra human labels, which should extend to other localization tasks where target coordinates transform predictably under cropping.
  • Test-time search behaves like a scaling law: the paper's ablation shows accuracy rises monotonically with the number of sampled predictions N, so more inference compute can be traded for accuracy without retraining.
  • Because grounding accuracy directly improves agent task success on Mind2Web and AndroidControl under a fixed planner, stronger grounding is a practical lever for GUI agents, not just a benchmark metric.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to use the KDE density peak's width as a confidence estimate: if the model's coordinate distribution is roughly Gaussian around the target, the spread of samples should predict whether the final point is correct, enabling abstention or verification.
  • The shared-reasoning design implicitly assumes the reasoning trace is view-invariant; a stronger version of the method might explicitly rewrite the reasoning for the cropped view, which could help when the global rationale refers to page layout that the crop no longer shows.
  • The paper's own data-scaling curve from 3.2K to 20K has not saturated, which suggests the same recipe would keep improving with a moderately larger dataset; extrapolating that trend is an editorial inference, not yet demonstrated.
  • An editorial note on deployment: the paper itself flags the lack of an explicit safety framework against malicious automation; that is a deployment limitation and does not bear on the accuracy claim, but it should shape how the capability is released.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ReGUIDE, a framework for GUI coordinate grounding that combines (i) online RL (GRPO) with an accuracy-plus-formatting reward to make an MLLM self-generate reasoning traces, (ii) a consistency fine-tuning stage that supervises predictions on paired global and cropped views while sharing the same reasoning text, and (iii) a test-time spatial search that crops around a KDE density peak and aggregates repeated samples. Training on a 20K subset of UGround (claimed 0.2% of the data used by open-sourced baselines), ReGUIDE reports improving Qwen-2.5-VL-3B from 55.5% to 87.3% on ScreenSpot and from 23.9% to 44.3% on ScreenSpot-Pro, and similarly large gains for the 7B variant. The paper also reports downstream agentic gains on Multimodal-Mind2Web and AndroidControl, component ablations, and hyperparameter studies for the search stage.

Significance. If the data-efficiency claim holds, ReGUIDE would be a valuable demonstration that reasoning RL, spatial consistency, and test-time search can substitute for millions of supervised GUI examples. The paper has real strengths: the ablations in Tables 4, 5, 10, and 11 isolate each component; Figure 2 provides an interesting empirical observation that coordinate-token likelihoods are approximately Gaussian, which supports the KDE aggregation choice; Table 6 shows grounding gains can transfer to downstream agent tasks; and the training details in Appendix A are sufficiently concrete for reproduction. The main reservation is that the headline comparison is not controlled for inference-time compute, and several secondary issues (baseline prompt mismatch, a numerical inconsistency, and an unexamined assumption in the consistency loss) currently prevent the central claim from being accepted as stated.

major comments (4)
  1. [Section 4, Table 1 and Table 5] The headline data-efficiency comparison is not controlled for inference-time compute. ReGUIDE is evaluated with N=M=16 sampled predictions, an 840x840 RoI crop, and KDE aggregation, while all baselines are evaluated with greedy decoding and no search. Table 5 shows that, for the already-trained ReGUIDE-3B, adding the cropping-plus-voting search raises ScreenSpot from 84.3 to 87.3 and ScreenSpot-Pro from 27.9 to 44.3. Thus a large portion of the margin over UGround-7B (86.3/31.1) comes from test-time search rather than from the 20K training set. The statement that ReGUIDE uses 'only 0.2% samples' should either be accompanied by baselines re-evaluated with the same search procedure, or be explicitly reframed as a system-level train-plus-test-compute claim.
  2. [Section 4, 'Baselines' and Appendix F] The '+SFT 20K' baseline is not directly comparable to ReGUIDE. According to the baseline description and the prompts in Appendix F, SFT uses the UGround prompt with a coordinate-only output, while ReGUIDE uses a different prompt that asks for <think> reasoning and an <answer> coordinate. The fact that SFT drops below the base model on ScreenSpot-V2 and ScreenSpot-Pro (for Qwen-2.5-VL-3B: 70.4 to 56.5 and 23.9 to 11.6) suggests prompt, decoding, or loss differences are confounding the comparison. The controlled comparison should use the same prompt format for SFT and ReGUIDE, and ideally also evaluate both with greedy decoding and with the same search procedure.
  3. [Section 4.1, Table 1] The prose states 'ReGUIDE performs accuracy of 45.1%' on ScreenSpot-Pro, but Table 1 reports 44.3 for ReGUIDE-3B and 44.4 for ReGUIDE-7B. This inconsistency should be corrected; if 45.1 comes from a different configuration (for example, a different search setting or a typo), that configuration should be stated explicitly.
  4. [Section 3.1, Eq. (3)] The consistency stage assumes that a self-generated reasoning trace r is a valid supervision target for both the global view and the randomly cropped local view. Because cropping changes the coordinate frame and can remove the very landmarks the reasoning refers to (e.g., 'first row, first column' of a section, as shown in Table 3), the local-view loss may train the model to reproduce reasoning that is spatially inconsistent with the cropped image. This assumption is load-bearing for the loss in Eq. (3), and the paper currently provides only the indirect evidence in Table 11. The authors should provide a direct check—for instance, measuring the local-view loss or success rate when conditioning on the global-view reasoning r, or ablating against re-generating a local-view-specific reasoning trace—to show that the shared reasoning is not teaching contradictory input-output mappings.
minor comments (6)
  1. [Section 4, Evaluation setup] The evaluation protocol should state explicitly that all baselines are decoded greedily and without search; currently only the ReGUIDE search hyperparameters are reported, so the reader must infer the baseline decoding protocol.
  2. [Section 3.2, Eq. (5)] The notation is inconsistent: the text says M new coordinates are generated inside the RoI, but Eq. (5) defines CRoI := {c(i)_RoI}_{i=1}^N. Since the paper uses N=M=16, this is not currently an error in results, but the formalism should be aligned.
  3. [Section 4.2, Table 5 vs Table 4] Table 5 reports 84.3 for ReGUIDE-3B without cropping or voting, while Table 4 reports 84.9 for the same configuration (Reason + Consistency, no Search). The discrepancy should be explained or corrected.
  4. [Section A.3, Table 8] Table 8 lists 'minimum crop ratio 0.3' while the main text says the local view is a random crop of up to 30% of the original area; please clarify whether 0.3 is a minimum, a maximum, or a range, and state the exact sampling procedure.
  5. [Section 4.2, Figure 2] The typo 'spreaded gaussian' should be fixed, and the figure caption should explain how the likelihood surface is computed from model token probabilities, since this claim is used to justify the KDE kernel choice.
  6. [Section 4, Eq. (4)] The argmax in Eq. (4) is over a continuous rectangle; the paper should specify how the density is numerically evaluated (e.g., grid resolution, tie-breaking), since this affects the reproducibility of the reported accuracy numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training rewards use external ground truth, the consistency loss uses augmented labels, and the test-time search is an inference-time ensemble over genuine model outputs.

full rationale

ReGUIDE's derivation chain is self-contained and does not reduce to its own inputs. The RL stage (Eqs. 1-2) rewards predictions that fall inside the ground-truth box from UGround, so the reasoning objective is anchored by external labels. The consistency stage (Eq. 3) supervises the full image and a crop with the ground-truth-derived center of the target; the shared reasoning token sequence is a training choice, not a label manufactured from the model's final prediction. The test-time search (Eqs. 4-5) is a KDE-based voting procedure over the model's own sampled outputs and introduces no fitted parameter that is later renamed as a prediction. No load-bearing step depends on a self-citation: the only overlapping-author citation is [19] (Revise), which appears in a related-work sentence about test-time scaling and is not used to justify ReGUIDE's cropping/voting mechanism; that mechanism is instead related to the independently cited V* [34]. Component claims are supported by ablations (Tables 4, 5, 11) that vary one stage at a time with fixed model and benchmarks, so the reported gains are not forced by construction. The main validity concern is benchmarking rather than circularity: search hyperparameters (N=16, L=840, T=1.0) are selected on the same ScreenSpot/ScreenSpot-Pro benchmarks used in Table 1 while baselines are evaluated with greedy decoding, and a large part of the ScreenSpot-Pro gain is attributable to the search stage (27.9 to 44.3 in Table 5). This conflates train-time data efficiency with test-time compute and may inflate the reported margins, but it does not make the derivation equivalent to its inputs, and it should be assessed as a correctness/fairness risk rather than a circularity score.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim depends on several manually chosen hyperparameters, most critically the test-time search hyperparameters selected on the same benchmarks used for reporting. The paper introduces no new physical or conceptual entities; the spatial prior and reasoning paths are training signals rather than invented constructs.

free parameters (7)
  • Formatting reward weight lambda = 0.1
    Balances the coordinate-accuracy reward and the format reward in Eq. (1); chosen by hand and used throughout.
  • KDE variance Sigma = 0.01
    Bandwidth for the 2D Gaussian kernels in Eq. (4); selected through ablations on the same ScreenSpot and ScreenSpot-Pro benchmarks reported in Table 1.
  • RoI crop size (W_RoI, H_RoI) = 840 x 840
    Fixed crop size for the test-time zoom stage; tuned by ablation in Figure 3a on the reported benchmarks.
  • Number of search samples N=M = 16
    Number of sampled predictions in the cropping and voting stages; ablated in Figure 3b on the reported benchmarks.
  • Decoding temperature T = 1.0
    Sampling temperature for test-time search; ablated in Figure 3c on the reported benchmarks.
  • Minimum crop ratio = 0.3
    The local view is a random crop of up to 30% of the original area; set in the Section 4 training setup.
  • GRPO KL coefficient = 0.01
    KL regularization coefficient for GRPO in Table 7; chosen by hand.
assumptions (5)
  • domain assumption Any point inside the ground-truth bounding box is a correct grounding prediction, and the box center is the natural target.
    This defines both the RL reward in Eq. (1) and the labels in the consistency stage in Section 3.1; if the metric is too coarse, the reported accuracy gains may not reflect fine-grained localization quality.
  • domain assumption The 20K image-instruction subset randomly drawn from UGround is representative enough to learn general GUI grounding.
    The central data-efficiency claim rests on this small subset; the paper gives no analysis of subset composition, duplicate removal, or domain overlap with the ScreenSpot benchmarks.
  • ad hoc to paper Self-generated reasoning traces from RL rollouts are high-quality enough to be reused as shared reasoning for both the global and cropped views.
    Section 3.1 defines d_global and d_local with the same reasoning text r; if the text is not spatially valid after cropping, the consistency training could teach contradictory cues.
  • standard math KDE with a fixed Gaussian bandwidth is a valid way to aggregate coordinate predictions, assuming the predictions form a unimodal near-Gaussian density.
    Equation (4) uses this assumption and Figure 2 provides a post-hoc empirical justification, but the procedure is not derived from first principles.
  • domain assumption Point-in-box accuracy on ScreenSpot, ScreenSpot-V2, and ScreenSpot-Pro is a valid proxy for GUI-agent usefulness.
    The paper separately evaluates agent benchmarks in Table 6, but the headline data-efficiency and state-of-the-art claims rely on this proxy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search." pith.science (2026). https://pith.science/paper/PZWNAE57

@misc{pith2026250515259,
  author       = {Pith},
  title        = {Pith review of: ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PZWNAE57}},
  note         = {Machine review of arXiv:2505.15259}
}
read the original abstract

Recent advances in Multimodal Large Language Models (MLLMs) have enabled autonomous agents to interact with computers via Graphical User Interfaces (GUIs), where accurately localizing the coordinates of interface elements (e.g., buttons) is often required for fine-grained actions. However, this remains significantly challenging, leading prior works to rely on large-scale web datasets to improve the grounding accuracy. In this work, we propose Reasoning Graphical User Interface Grounding for Data Efficiency (ReGUIDE), a novel and effective framework for web grounding that enables MLLMs to learn data efficiently through self-generated reasoning and spatial-aware criticism. More specifically, ReGUIDE learns to (i) self-generate a language reasoning process for the localization via online reinforcement learning, and (ii) criticize the prediction using spatial priors that enforce equivariance under input transformations. At inference time, ReGUIDE further boosts performance through a test-time scaling strategy, which combines spatial search with coordinate aggregation. Our experiments demonstrate that ReGUIDE significantly advances web grounding performance across multiple benchmarks, outperforming baselines with substantially fewer training data points (e.g., only 0.2% samples compared to the best open-sourced baselines).

Figures

Figures reproduced from arXiv: 2505.15259 by the authors.

Figure 1
Figure 1. Overview of training ReGUIDE. Left: The model rolls out multiple (reasoning, coor￾dinate) pairs; rewards the point that falls inside the ground-truth box. Right: Trains the model on paired full-image and cropped views, sharing the reasoning tokens while adjusting the coordinate, which provides multi-view consistency and improves grounding performance. evaluate and refine these outputs [15]. In the localization task,… view at source ↗
Figure 2
Figure 2. Likelihood of coordinate tokens spreaded gaussian naturally. Natural Gaussian distribution of ground likelihood. As an additional qualitative analysis, we investigate the likeli￾hood distributions of coordinates surrounding the initial pre￾dicted coordinates. By calculating the likelihood of coordi￾nates with same generated reasoning. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Ablation studies on grounding accuracy (%) w.r.t. (a) crop size [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Attention map between text tokens, X-axis is key tokens, Y-axis is query tokens. In the [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Grounding accuracy (%) on SCREENSPOT [7] as the UGround [11] training set grows from 3.2 K to 20 K samples. We trained from Qwen-2.5-VL-3B. C Illustration of Searching Strategy [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Overview of ReGUIDE’s searching strategy.. Starting from the full screenshot, the model samples N coordinate predictions, votes using kernel density estimation, and recenters a crop on the density peak. One searching pass is then run inside the crop, and the point with…
Figure 7
Figure 7. Figure 7: Learning curve during training GRPO of ReGUIDE. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Auxiliary Reasoning Unleashes GUI Grounding in VLMs

    cs.CV 2025-09 conditional novelty 5.0 of 10

    Overlaying labeled grids and axes on screenshots substantially improves zero-shot GUI grounding in most VLMs, with the best variant zooming into grid cells.

  2. SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control

    cs.AI 2025-08 conditional novelty 4.0 of 10

    A multi-agent RL workflow that interleaves single-agent updates, applied to mobile GUI control, achieves SOTA zero-shot performance and a +14.8 MATH500 gain.

  3. DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning

    cs.AI 2025-06 conditional novelty 4.0 of 10

    Separating text and icon grounding with iterative zooming improves GUI-element localization accuracy of existing vision-language models without retraining.

Reference graph

Works this paper leans on

40 extracted references · 10 canonical work pages · cited by 3 Pith papers

  1. [1]

    Ahmadian, C

    A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024

    arXiv 2024

  2. [2]

    Qwen2.5-vl technical report

    Alibaba. Qwen2.5-vl technical report. arXiv preprint arXiv:abs/2502.13923, 2025

    arXiv 2025

  3. [3]

    Anthropic

    T. Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. https://www.anthropic.com/news/3-5-models-and-computer-use , October 2024

    work page 2024

  4. [4]

    Anthropic

    T. Anthropic. Claude 3.7 sonnet and claude code. https://www.anthropic.com/news/ claude-3-7-sonnet , February 2025

    work page 2025

  5. [5]

    Caron, H

    M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021

    2021

  6. [6]

    C. Chen, Z. Zhang, I. Khalilov, B. Guo, S. A. Gebreegziabher, Y . Ye, Z. Xiao, Y . Yao, T. Li, and T. J.-J. Li. Toward a human-centered evaluation framework for trustworthy llm-powered gui agents. arXiv preprint arXiv:2504.17934, 2025

    arXiv 2025

  7. [8]

    Cheng, Q

    K. Cheng, Q. Sun, Y . Chu, F. Xu, Y . Li, J. Zhang, and Z. Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024

    arXiv 2024

  8. [9]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    DeepSeek. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    arXiv 2024

Show all 40 references
  1. [10]

    X. Deng, Y . Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y . Su. Mind2web: Towards a generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=kiYqbO3wqw

    2023

  2. [11]

    B. Gou, R. Wang, B. Zheng, Y . Xie, C. Chang, Y . Shu, H. Sun, and Y . Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024

    2024 arXiv

  3. [12]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [13]

    R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y . Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018

    2018 arXiv

  5. [14]

    W. Hong, W. Wang, Q. Lv, J. Xu, W. Yu, J. Ji, Y . Wang, Z. Wang, Y . Dong, M. Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024

    2024

  6. [15]

    Hosseini, X

    A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457, 2024

    2024 arXiv

  7. [16]

    J. Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025

    2025 arXiv

  8. [17]

    Huang, B

    W. Huang, B. Jia, Z. Zhai, S. Cao, Z. Ye, F. Zhao, Z. Xu, Y . Hu, and S. Lin. Vision-r1: Incentiviz- ing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  9. [18]

    A. K. Lampinen, I. Dasgupta, S. C. Chan, K. Matthewson, M. H. Tessler, A. Creswell, J. L. McClelland, J. X. Wang, and F. Hill. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329, 2022. 10

    2022 arXiv

  10. [19]

    H. Lee, S. Oh, J. Kim, J. Shin, and J. Tack. Revise: Learning to refine at test-time via intrinsic self-verification. arXiv preprint arXiv:2502.14565, 2025

    2025 arXiv

  11. [20]

    K. Li, Z. Meng, H. Lin, Z. Luo, Y . Tian, J. Ma, Z. Huang, and T.-S. Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981, 2025

    2025 arXiv

  12. [21]

    W. Li, W. Bishop, A. Li, C. Rawles, F. Campbell-Ajala, D. Tyamagundlu, and O. Riva. On the effects of data scale on computer control agents. arXiv preprint arXiv:2406.03679, 2024

    2024 arXiv

  13. [22]

    K. Q. Lin, L. Li, D. Gao, Z. Yang, S. Wu, Z. Bai, W. Lei, L. Wang, and M. Z. Shou. Showui: One vision-language-action model for gui visual agent. arXiv preprint arXiv:2411.17465, 2024

    2024 arXiv

  14. [23]

    Z. Lu, Y . Chai, Y . Guo, X. Yin, L. Liu, H. Wang, G. Xiong, and H. Li. Ui-r1: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620, 2025

    2025 arXiv

  15. [24]

    X. Ma, Z. Ding, Z. Luo, C. Chen, Z. Guo, D. F. Wong, X. Feng, and M. Sun. Deeppercep- tion: Advancing r1-like cognitive visual perception in mllms for knowledge-intensive visual grounding. arXiv preprint arXiv:2503.12797, 2025

    2025

  16. [25]

    M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021

    2021 arXiv

  17. [26]

    Gpt-4o system card

    OpenAI. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  18. [27]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  19. [28]

    Y . Qin, Y . Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y . Li, S. Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  20. [29]

    S. J. Rennie, E. Marcheret, Y . Mroueh, J. Ross, and V . Goel. Self-critical sequence training for image captioning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 7008–7024, 2017

    2017

  21. [30]

    Rosenblat

    M. Rosenblat. Remarks on some nonparametric estimates of a density function. Ann. Math. Stat, 27:832–837, 1956

    1956

  22. [31]

    P. Shaw, M. Joshi, J. Cohan, J. Berant, P. Pasupat, H. Hu, U. Khandelwal, K. Lee, and K. N. Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. Advances in Neural Information Processing Systems, 36:34354–34370, 2023

    2023

  23. [32]

    Snell, J

    C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  24. [33]

    S. Wang, W. Liu, J. Chen, Y . Zhou, W. Gan, X. Zeng, Y . Che, S. Yu, X. Hao, K. Shao, et al. Gui agents with foundation models: A comprehensive survey. arXiv preprint arXiv:2411.04890, 2024

    2024 arXiv

  25. [34]

    Wu and S

    P. Wu and S. Xie. V*: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084–13094, 2024

    2024

  26. [35]

    Z. Wu, Z. Wu, F. Xu, Y . Wang, Q. Sun, C. Jia, K. Cheng, Z. Ding, L. Chen, P. P. Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024

    2024 arXiv

  27. [36]

    Xia and R

    X. Xia and R. Luo. Gui-r1: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458, 2025. 11

    2025 arXiv

  28. [37]

    Y . Xu, Z. Wang, J. Wang, D. Lu, T. Xie, A. Saha, D. Sahoo, T. Yu, and C. Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454, 2024

    2024 arXiv

  29. [38]

    T. Yu, Y . Yao, H. Zhang, T. He, Y . Han, G. Cui, J. Hu, Z. Liu, H.-T. Zheng, M. Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  30. [39]

    Zhang, Z

    J. Zhang, Z. Ding, C. Ma, Z. Chen, Q. Sun, Z. Lan, and J. He. Breaking the data barrier–building gui agents through task generalization. arXiv preprint arXiv:2504.10127, 2025

    2025 arXiv

  31. [40]

    H. Zhao, T. Chen, and Z. Wang. On the robustness of gui grounding models against image attacks. arXiv preprint arXiv:2504.04716, 2025

    2025 arXiv

  32. [41]

    golden trajectory

    B. Zheng, B. Gou, J. Kil, H. Sun, and Y . Su. Gpt-4v(ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreview.net/forum?id=piecKJ2DlB. 12 A Experimental Details In this section, we describe the ex...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.