Pith. sign in

REVIEW 3 major objections 5 minor 8 cited by

GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GUI-G2 claims that replacing binary hit-or-miss rewards with Gaussian point and coverage rewards lets a 7B model surpass a 72B model on professional-software GUI grounding by a relative 24.7% on ScreenSpot-Pro.

desk verdict A credible same-pipeline ablation shows Gaussian rewards beat sparse ones for GUI grounding, but the headline numbers are over-sold and the key hyperparameter is tuned on the eval set. read the letter →

arxiv 2507.15846 v3 pith:MGC4C3F3 submitted 2025-07-21 cs.LG cs.AIcs.CLcs.CVcs.HC

Fei Tang , Zhangxuan Gu , Zhengxi Lu , Xuyang Liu , Shuheng Shen , Changhua Meng , Wen Wang , Wenqi Zhang
show 4 more authors
Yongliang Shen Weiming Lu Jun Xiao Yueting Zhuang
This is my paper
classification cs.LGcs.AIcs.CLcs.CVcs.HC
keywords GUIgroundingGaussianrewardsdensecontinuousoptimizationreinforcementlearningGRPOadaptivevarianceScreenSpot-Provision-languagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that GUI grounding underperforms because standard reinforcement-learning rewards are binary: a click inside the target box scores 1, anything outside scores 0, so the model receives no gradient for near-misses. It proposes GUI-G2, which treats each interface element as a two-dimensional Gaussian distribution and rewards predictions with a smooth point term for center alignment plus a coverage term for spatial overlap, with variances scaled to element size. Trained with GRPO on a 7B vision-language model, the system reports 92.0% on ScreenSpot, 93.3% on ScreenSpot-v2, and 47.5% on ScreenSpot-Pro, the last a relative 24.7% improvement over the 72B UI-TARS baseline. The paper's claim is that dense, continuously varying spatial feedback—not model scale—is what drives accurate GUI grounding.

What carries the argument

The load-bearing object is the Gaussian reward surface defined over the whole interface plane. For a ground-truth box $b = [x_1, y_1, x_2, y_2]$, the paper places a 2D Gaussian with mean $\mu = ((x_1+x_2)/2, (y_1+y_2)/2)$ and diagonal covariance with $\sigma_x = \alpha(x_2-x_1)$, $\sigma_y = \alpha(y_2-y_1)$, $\alpha = 0.5$. The point reward evaluates the predicted center by the Gaussian density at that location, so reward decays smoothly with distance; the coverage reward uses the Bhattacharyya coefficient between predicted and ground-truth Gaussians, with a closed form that penalizes both center offset and shape mismatch. The adaptive variance is what lets one formula serve tiny icons and full-screen panels, and the sum of the two components is normalized into advantages inside GRPO. Together these pieces convert a sparse hit-or-miss signal into dense gradients everywhere, which is the mechanism the paper credits for the reported gains.

What would settle it

Train the same 7B model on the same data with identical Gaussian rewards but with $\alpha = 0.25$ and $\alpha = 0.75$, and evaluate on ScreenSpot-Pro: if either setting closes the gap over UI-TARS-72B, the claim that $\alpha = 0.5$ encodes a general '2$\sigma$' principle is not supported. A complementary check is to collect human click positions on ScreenSpot-Pro elements and verify that the Gaussian reward ranks two near-miss predictions in the same order as human modal click distance; any inversion would falsify the behavioral grounding of the reward.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that reward geometry determines GUI grounding quality. Each ground-truth element is converted into a Gaussian $N(x; \mu, \Sigma)$ with mean at the element centroid and diagonal covariance $\Sigma = \mathrm{diag}(\sigma_x^2, \sigma_y^2)$, where $\sigma_x = \alpha \cdot \text{width}$ and $\sigma_y = \alpha \cdot \text{height}$ for a fixed $\alpha = 0.5$. Two rewards are defined: a point reward $R_{\text{point}} = N(\mu_p; \mu_{gt}, \Sigma_{gt})$ measuring localization, and a coverage reward $R_{\text{coverage}}$ equal to the Bhattacharyya coefficient between predicted and ground-truth Gaussians, which has a closed form combining center distance and covariance similarity. The total reward $R_{\text{total}} = \nu R_{\text{point}} + \gamma R_{\text{coverage}}$ with $\nu = \gamma = 1$ feeds GRPO advantage normalization over eight sampled responses. The paper reports that this continuous signal produces monotonic convergence of predictions toward target centers while sparse baselines oscillate, and that cutting the Gaussian feedback off at element boundaries costs 4.9% accuracy, supporting the claim that GUI grounding should be optimized as a dense spatial field rather than a classification task.

Load-bearing premise

The load-bearing premise is that one global scaling constant, $\alpha = 0.5$, chosen by sweeping ScreenSpot-v2, transfers to unseen benchmarks and to every element size; if different interfaces or element types need different click tolerances, the Gaussian signal can misdirect training and the reported advantage shrinks or vanishes.

Editorial extensions

If this is right

  • A 7B model trained with Gaussian rewards can outperform a 72B model trained with sparse or distance rewards on high-resolution professional interfaces, implying reward design can substitute for model scale in GUI grounding.
  • Sparse binary and IoU rewards produce erratic training trajectories, while the Gaussian reward converges monotonically (average distance to target center falling from 290px to 150px), implying near-miss feedback is what stabilizes RL for spatial tasks.
  • Providing Gaussian feedback everywhere, not just inside target boxes, improves accuracy by 4.9%, implying optimization discontinuities at element boundaries are harmful even when the reward is continuous inside.
  • Removing either the point or the coverage component degrades performance (92.1% and 90.2% vs 93.3%), implying precise localization and regional targeting are complementary signals.
  • Explicit reasoning tokens hurt grounding accuracy (88.7% vs 93.3% with 87.7% fewer tokens), implying GUI grounding is primarily a perceptual task that does not benefit from chain-of-thought.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same point-plus-coverage Gaussian decomposition could serve as a dense auxiliary loss during supervised fine-tuning or as an inference-time reranker, since the reward needs no RL to be computed.
  • The paper fixes $\alpha$ by a global sweep; a natural test is to make variance per-element or predicted by the model, which would generalize the mechanism to interfaces where element size does not track click tolerance.
  • The finding that thinking tokens hurt grounding suggests that for other spatial perceptual tasks—medical image localization, map pointing, robotic reach targets—reward shape may matter more than added reasoning, a claim the paper does not test.
  • Because the same benchmark gains accompany a 76.9% share of remaining errors from icon semantics, the Gaussian reward may be near a ceiling for localization while semantic understanding stays the bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes GUI-G2, a reward-shaping method for RL-based GUI grounding. It replaces binary hit/miss rewards with two dense Gaussian terms: a point reward based on the target-element Gaussian density at the predicted center (Eq. 2) and a coverage reward based on the Bhattacharyya coefficient between predicted and target Gaussians (Eq. 4). The covariance is set adaptively to element size via sigma = alpha times the box dimension (Eq. 5), and the combined reward is used in GRPO training of Qwen2.5-VL-7B. On ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro, the trained GUI-G2-7B reaches 92.0%, 93.3%, and 47.5%, respectively, with the headline 24.7% relative gain over UI-TARS-72B on ScreenSpot-Pro. Ablations compare sparse point/IoU rewards, an inside-only Gaussian, component weighting, and thinking vs. no-thinking, and an appendix reports random-reward controls.

Significance. Assuming the reported results are reproducible, the paper makes a useful and low-cost contribution: the closed-form Gaussian rewards are simple, easy to implement, and provide dense optimization signals, and the controlled comparison in Table 2 plus the spurious-reward controls in Appendix A.1 are informative evidence that dense, spatially meaningful rewards matter. The announced code release also supports reproducibility. The broader claims, however, are stronger than the evidence: the gain over the closest distance-based continuous baseline on ScreenSpot-Pro is only 0.2 percentage points (Table 3), and the key parameter alpha is tuned on ScreenSpot-v2 and evaluated on the same benchmark, so the quantitative advantage is partly a selection effect. The 'new paradigm' phrasing also overstates what is essentially a reward-shaping design.

major comments (3)
  1. [Section 4.4 / Figure 5 / Section 4.1 / Table 2] The adaptive variance scale alpha is selected by a hyperparameter sweep that maximizes ScreenSpot-v2 accuracy (Figure 5) and is fixed to 0.5 for all reported models (Section 4.1). The cleanest evidence for the central claim, the sparse-vs-dense reward comparison in Table 2, is then reported on the same ScreenSpot-v2 benchmark (93.3% vs. 87.4%). As a result, part of the +5.9 point gap may be a selection effect rather than a property of Gaussian rewards. Please select alpha on a validation split that is not used for the headline evaluation, or add a sweep on ScreenSpot-Pro and another held-out set showing that the relative ranking of reward shapes is stable across alpha. Without such a check, the claim that the optimal reward geometry transfers across interface distributions is not supported.
  2. [Table 3 / Section 4.2] On ScreenSpot-Pro, GUI-G2-7B (47.5%) exceeds the closest distance-based continuous baseline SE-GUI-7B (47.3%) by only 0.2 points, while the headline 24.7% relative gain is against UI-TARS-72B (38.1%), which is neither the strongest RL baseline nor the strongest continuous baseline in the table. The 'substantially outperforms continuous distance-based methods' claim for the high-resolution regime therefore rests on a very thin margin, and SE-GUI is an external model trained with its own data and recipe. Please provide variance or significance information over multiple runs for the ScreenSpot-Pro comparison, or qualify the claim to reflect the margin.
  3. [Section 4.4 / Eq. (5) / Table 5] The adaptation principle used to justify alpha = 0.5 is internally inconsistent as written. The text defines '2sigma Principle' as sigma_x = width * 2, but then states that alpha = 0.5, which by Eq. (5) gives sigma_x = width / 2, 'aligns with the 2sigma statistical principle'. Moreover, the sigma-ablation results (1sigma, 2sigma, 3sigma, fixed sigma) are said to be 'shown in Table 5', but Table 5 is the nu/gamma weighting ablation, and the sigma results appear only in Figure 5. Please correct the nomenclature and report the complete sigma-ablation table, since this is the central design parameter.
minor comments (5)
  1. [Abstract / Section 4.2] The improvement over UI-TARS-72B is reported as 24.7% in the abstract and as 9.4% in Section 4.2; please label relative versus absolute gains consistently.
  2. [Table 6] The token reduction is listed as 87.7% in the table but as 76.9% in the prose; make the two numbers consistent.
  3. [References] The MacKenzie reference entry reads 'I. Scott MacKenzie and.'; the author field is incomplete.
  4. [Figure 5] The x-axis is not labeled; please specify the alpha values swept and state clearly that the plotted accuracy is on ScreenSpot-v2.
  5. [Figure 10] The caption contains 'Scrrenspot-v2' and 'bad case分析'; please proofread captions for typos and language consistency.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the Gaussian reward derivation is self-contained, with minor alpha-selection-on-eval and a non-load-bearing self-citation.

full rationale

GUI-G2's reward construction (Eqs. 1-6) is not derived from, nor does it redefine, the benchmarks it predicts. The Gaussian point/coverage rewards and adaptive variance (Eq. 5) specify a reward family; alpha, nu, and gamma are hyperparameters, not fitted targets. The human-click Gaussian statistics in Figure 1 are motivation only, and the fitted AITW mu/sigma values do not appear in the reward equations. The one overlapping-author citation (Tang et al. 2025a) is used to define GUI grounding and to situate the method, not to justify the Gaussian reward or to exclude alternatives; no uniqueness theorem is imported. The only validity-adjacent concern is that alpha=0.5 is selected by a sweep on ScreenSpot-v2 (Figure 5) and the same benchmark supplies the headline 93.3% and the dense-vs-sparse comparison in Table 2, so part of the reported gain is a model-selection effect rather than an independent test. That is a benchmark-tuning concern, not a definitional circularity: the trained model's accuracy is not determined by alpha alone, and the ScreenSpot-Pro result (47.5% vs 38.1% for UI-TARS-72B) is not tuned on that distribution. No equation reduces to its own input, so the derivation chain is self-contained; score 2 reflects the minor alpha-on-eval selection and the non-load-bearing self-citation, not a circular derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The math of the rewards is self-contained and standard; the load-bearing inputs are the hand-selected hyperparameters (α, ν, γ), the behavioral assumption that human clicks justify Gaussian rewards, the GRPO framework, and the modeling choice that a predicted box size defines a Gaussian click distribution.

free parameters (3)
  • α (adaptive variance scale) = 0.5
    Scales σ for point and coverage rewards (Eq. 5). Chosen by hyperparameter sensitivity analysis on ScreenSpot-v2 (Figure 5); the main result is then reported on the same benchmark.
  • ν (point reward weight) = 1.0
    Weight of R_point in Eq. 6. Selected as the best of three values in ablation on ScreenSpot-v2 (Table 5).
  • γ (coverage reward weight) = 1.0
    Weight of R_coverage in Eq. 6. Selected as the best of three values in ablation on ScreenSpot-v2 (Table 5).
assumptions (4)
  • standard math Bhattacharyya coefficient for 2D Gaussians has the closed form given in Eq. 4
    Used to define the coverage reward in Section 3.2; standard result, but no proof or citation is given in the paper.
  • domain assumption Human clicking on GUI elements follows Gaussian distributions centered on the element (AITW analysis)
    Used in Section 1 and Figure 1 to motivate the Gaussian reward; the reported μ=0.111, σ=0.429 are not clearly defined and are not used to set the reward parameters.
  • domain assumption GRPO with KL regularization is a suitable and stable optimizer for this task
    The entire RL setup (Eq. 7-8) is inherited from Shao et al. 2024 without modification or re-validation.
  • domain assumption The Gaussian covariance derived from the predicted box (via Eq. 5) is a meaningful representation of the predicted click distribution
    The coverage reward depends on Σp from the predicted box; this modeling choice is asserted, not justified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding." pith.science (2026). https://pith.science/paper/MGC4C3F3

@misc{pith2026250715846,
  author       = {Pith},
  title        = {Pith review of: GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MGC4C3F3}},
  note         = {Machine review of arXiv:2507.15846}
}
abstract

Graphical User Interface (GUI) grounding maps natural language instructions to precise interface locations for autonomous interaction. Current reinforcement learning approaches use binary rewards that treat elements as hit-or-miss targets, creating sparse signals that ignore the continuous nature of spatial interactions. Motivated by human clicking behavior that naturally forms Gaussian distributions centered on target elements, we introduce GUI Gaussian Grounding Rewards (GUI-G$^2$), a principled reward framework that models GUI elements as continuous Gaussian distributions across the interface plane. GUI-G$^2$ incorporates two synergistic mechanisms: Gaussian point rewards model precise localization through exponentially decaying distributions centered on element centroids, while coverage rewards assess spatial alignment by measuring the overlap between predicted Gaussian distributions and target regions. To handle diverse element scales, we develop an adaptive variance mechanism that calibrates reward distributions based on element dimensions. This framework transforms GUI grounding from sparse binary classification to dense continuous optimization, where Gaussian distributions generate rich gradient signals that guide models toward optimal interaction positions. Extensive experiments across ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro benchmarks demonstrate that GUI-G$^2$, substantially outperforms state-of-the-art method UI-TARS-72B, with the most significant improvement of 24.7% on ScreenSpot-Pro. Our analysis reveals that continuous modeling provides superior robustness to interface variations and enhanced generalization to unseen layouts, establishing a new paradigm for spatial reasoning in GUI interaction tasks.

Figures

Figures reproduced from arXiv: 2507.15846 by the authors.

Figure 1
Figure 1. GUI grounding performance and human click behavior. Left: Performance comparison [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of reward modeling strategies. (a-c) Existing methods treat GUI elements [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. GUI Gaussian Grounding Rewards (GUI-G2 ). Our framework transforms GUI grounding through continuous Gaussian modeling. Given a task instruction and screenshot, the policy model generates multiple predictions that are evaluated using our dual reward mechanism. Gaussian Point Rewards assess localization precision while Gaussian Coverage Rewards measure spatial overlap, together providing dense learning signals that gu… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Reward comparison analysis. Left: Training dynamics of sparse reward variants (Point, IoU, Point+IoU) showing reward standard deviation and convergence patterns. Right: Distance to target center over training steps, where Gaussian rewards demonstrate monotonic converge…
Figure 5
Figure 5. Figure 5: Hyperparameter sensitivity analysis for adaptive sigma (σ). Performance peaks at α = 0.5 with 93.3% accuracy on Screenspot-v2. GUI-G 2 (Full) GUI-G 2 w/o GC GUI-G 2 w/o GP GUI-G 2 w/ IG 88 89 90 91 92 93 94 93.3% 92.1% 90.2% 88.4% GUI-G 2 (Full) w/o Gaussian Coverage w…
Figure 7
Figure 7. Figure 7: Hyperparameter settings used in the training experiments. Continuous random rewards decline from 90.6% to 87.9% (-2.7%) and binary random rewards drop from 88.6% to 84.5% (-4.1%) over 3000 steps. These spurious rewards fail to provide effective learning signals for GUI…
Figure 8
Figure 8. Figure 8: Performance comparison between random reward strategies on ScreenSpot-V2. Both [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Reward distribution and standard deviation analysis during training. The reward variance [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Analysis of GUI grounding failure cases. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Grounding Computer Use Agents on Human Demonstrations

    cs.LG 2025-11 conditional novelty 7.0 of 10

    GroundCUA, a 3.56M-element human-annotated desktop grounding dataset, and GroundNext models achieve strong UI grounding with less than one-tenth the SFT data of prior work.

  2. GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

    cs.CV 2025-11 conditional novelty 7.0 of 10

    Supervising an MLLM's intrinsic self-attention with patch-level GUI labels, aggregated via a learnable anchor token and hidden-state-selected query tokens, reaches state-of-the-art 3B-scale GUI grounding accuracy with...

  3. GUI-AC: Enhancing Continual Learning in GUI Agents

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    GUI-AC stabilizes RFT for non-stationary GUI data by down-weighting noisy advantages and relaxing clipping bounds via a grounding certainty term.

  4. GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Action-weighted SFT plus KL-regularized GRPO on a curated 81K reasoning dataset lifts open-source GUI agents by 11–22 points on online task-completion benchmarks.

  5. UItron: Foundational GUI Agent with Advanced Perception and Planning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    UItron, trained with multi-stage SFT and curriculum RL on a new 1M-step Chinese app dataset, beats prior GUI agents on perception, grounding, planning, and especially Chinese mobile app scenarios.

  6. PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editors

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    Multi-turn visual feedback refinement outperforms single-shot coordinate prediction for pixel-precise GUI grounding in complex coding environments.

  7. Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning

    cs.CV 2025-10 conditional novelty 5.0 of 10

    HyperClick trains GUI grounding models with GRPO to output clicks plus confidence scores, jointly rewarding correct clicks and Brier-calibrated confidence, and reports SOTA accuracy on six of seven benchmarks with bet...

  8. SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control

    cs.AI 2025-08 conditional novelty 4.0 of 10

    A multi-agent RL workflow that interleaves single-agent updates, applied to mobile GUI control, achieves SOTA zero-shot performance and a +14.8 MATH500 gain.

Reference graph

Works this paper leans on

55 extracted references · 5 canonical work pages · cited by 8 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Scott MacKenzie and

    I. Scott MacKenzie and. Fitts' law as a research and design tool in human-computer interaction. Human–Computer Interaction, 7 0 (1): 0 91--139, 1992. doi:10.1207/s15327051hci0701\_3. URL https://doi.org/10.1207/s15327051hci0701_3

    work page doi:10.1207/s15327051hci0701 1992

  3. [3]

    Claude computer use

    Anthropic. Claude computer use. Available at: https://www.anthropic.com/news/developing-computer-use, 2024

    work page 2024

  4. [4]

    Uibert: Learning generic multimodal representations for ui understanding, 2021

    Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Aguera y Arcas. Uibert: Learning generic multimodal representations for ui understanding, 2021. URL https://arxiv.org/abs/2107.13731

    arXiv 2021

  5. [5]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2...

    arXiv 2025

  6. [6]

    Seeclick: Harnessing gui grounding for advanced visual gui agents, 2024

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents, 2024. URL https://arxiv.org/abs/2401.10935

    arXiv 2024

  7. [7]

    Le, Sergey Levine, and Yi Ma

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025. URL https://arxiv.org/abs/2501.17161

    arXiv 2025

  8. [8]

    Flashattention-2: Faster attention with better parallelism and work partitioning, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307.08691

    arXiv 2023

Show all 55 references
  1. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  2. [10]

    Pp-ocr: A practical ultra lightweight ocr system, 2020

    Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, and Haoshuang Wang. Pp-ocr: A practical ultra lightweight ocr system, 2020. URL https://arxiv.org/abs/2009.09941

    2020 arXiv

  3. [11]

    Video-r1: Reinforcing video reasoning in mllms, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms, 2025. URL https://arxiv.org/abs/2503.21776

    2025 arXiv

  4. [12]

    P. M. Fitts. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental PSychology, 74: 0 381--391, 1954

    1954

  5. [13]

    Navigating the digital world as humans do: Universal visual grounding for gui agents, 2024

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents, 2024. URL https://arxiv.org/abs/2410.05243

    2024 arXiv

  6. [14]

    Cogagent: A visual language model for gui agents, 2024

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2024. URL https://arxiv.org/abs/2312.08914

    2024 arXiv

  7. [15]

    Appagentx: Evolving gui agents as proficient smartphone users

    Wenjia Jiang, Yangyang Zhuang, Chenxi Song, Xu Yang, Joey Tianyi Zhou, and Chi Zhang. Appagentx: Evolving gui agents as proficient smartphone users. 2025. URL https://arxiv.org/abs/2503.02268

    2025 arXiv

  8. [16]

    Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. 2024. URL https://arxiv.org/abs/2402.17553

    2024 arXiv

  9. [17]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023. URL https://arxiv.org/abs/2304.02643

    2023 arXiv

  10. [18]

    Screenspot-pro: Gui grounding for professional high-resolution computer use, 2025

    Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use, 2025

    2025

  11. [19]

    Appagent v2: Advanced agent for flexible mobile interactions, 2024

    Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Appagent v2: Advanced agent for flexible mobile interactions, 2024. URL https://arxiv.org/abs/2408.11824

    2024

  12. [20]

    Showui: One vision-language-action model for gui visual agent, 2024

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for gui visual agent, 2024. URL https://arxiv.org/abs/2411.17465

    2024 arXiv

  13. [21]

    Autoglm: Autonomous foundation agents for guis

    Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, Junjie Gao, Junjun Shan, Kangning Liu, Shudan Zhang, Shuntian Yao, Siyi Cheng, Wentao Yao, Wenyi Zhao, Xinghan Liu, Xinyi Liu, Xinying Chen, Xinyue Yang, ...

    2024 arXiv

  14. [22]

    Video compression commander: Plug-and-play inference acceleration for video large language models, 2025 a

    Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang. Video compression commander: Plug-and-play inference acceleration for video large language models, 2025 a . URL https://arxiv.org/abs/2505.14454

    2025

  15. [23]

    Global compression commander: Plug-and-play inference acceleration for high-resolution large vision-language models, 2025 b

    Xuyang Liu, Ziming Wang, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Bo Zheng, Linfeng Zhang, Siteng Huang, and Honggang Chen. Global compression commander: Plug-and-play inference acceleration for high-resolution large vision-language models, 2025 b . URL https://arxiv.or...

    2025

  16. [24]

    Shifting ai efficiency from model-centric to data-centric compression, 2025 c

    Xuyang Liu, Zichen Wen, Shaobo Wang, Junjie Chen, Zhishan Tao, Yubo Wang, Xiangqi Jin, Chang Zou, Yiyu Wang, Chenfei Liao, Xu Zheng, Honggang Chen, Weijia Li, Xuming Hu, Conghui He, and Linfeng Zhang. Shifting ai efficiency from model-centric to data-centric compression, 2025 ...

    2025

  17. [25]

    Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners

    Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners. 2025 d . URL https://arxiv.org/abs/2504.14239

    2025 arXiv

  18. [26]

    Omniparser for pure vision based gui agent, 2024

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024. URL https://arxiv.org/abs/2408.00203

    2024 arXiv

  19. [27]

    Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning

    Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning. 2025. URL https://arxiv.org/abs/2503.21620

    2025 arXiv

  20. [28]

    Gui-r1 : A generalist r1-style vision-language action model for gui agents

    Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-r1 : A generalist r1-style vision-language action model for gui agents. 2025. URL https://arxiv.org/abs/2504.10458

    2025 arXiv

  21. [29]

    Introducing gpt-4o

    OpenAI. Introducing gpt-4o. Available at: https://openai.com/index/hello-gpt-4o, 2024

    2024

  22. [30]

    Ui-tars: Pioneering automated gui interaction with native agents, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, C...

    2025 arXiv

  23. [31]

    Android in the wild: A large-scale dataset for android device control

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. 2023. URL https://arxiv.org/abs/2307.10088

    2023 arXiv

  24. [32]

    Spurious rewards: Rethinking training signals in rlvr, 2025

    Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals in rlvr, 2025. URL ht...

    2025 arXiv

  25. [33]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  26. [34]

    Vlm-r1: A stable and generalizable r1-style large vision-language model, 2025

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-r1: A stable and generalizable r1-style large vision-language model, 2025. URL https://arxiv.org/abs/2504.07615

    2025 arXiv

  27. [35]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023. URL https://arxiv.org/abs/2303.17580

    2023 arXiv

  28. [36]

    Os-genesis: Automating gui agent trajectory construction via reverse task synthesis, 2025

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, and Zhiyong Wu. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis, 2025...

    2025 arXiv

  29. [37]

    Think twice, click once: Enhancing gui grounding via fast and slow systems

    Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song, Weiming Lu, and Yueting Zhuang. Think twice, click once: Enhancing gui grounding via fast and slow systems. 2025 a . URL https://arxiv.org/abs/2503.06470

    2025 arXiv

  30. [38]

    A survey on (m)llm-based gui agents

    Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, and Yueting Zhuang. A survey on (m)llm-based gui agents. 2025 b . URL https://arxiv.org/abs/2504.13865

    2025 arXiv

  31. [39]

    Lpo: Towards accurate gui agent interaction via location preference optimization, 2025 c

    Jiaqi Tang, Yu Xia, Yi-Feng Wu, Yuwei Hu, Yuhui Chen, Qing-Guo Chen, Xiaogang Xu, Xiangyu Wu, Hao Lu, Yanqing Ma, Shiyin Lu, and Qifeng Chen. Lpo: Towards accurate gui agent interaction via location preference optimization, 2025 c . URL https://arxiv.org/abs/2506.09373

    2025 arXiv

  32. [40]

    Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration, 2024 a

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration, 2024 a . URL https://arxiv.org/abs/2406.01014

    2024 arXiv

  33. [41]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception, 2024 b

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception, 2024 b . URL https://arxiv.org/abs/2401.16158

    2024 arXiv

  34. [42]

    Mobile-agent-e: Self-evolving mobile assistant for complex tasks, 2025

    Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks, 2025. URL https://arxiv.org/abs/2501.11733

    2025 arXiv

  35. [43]

    Gui-actor: Coordinate-free visual grounding for gui agents

    Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, et al. Gui-actor: Coordinate-free visual grounding for gui agents. arXiv preprint arXiv:2506.03143, 2025

    2025 arXiv

  36. [44]

    Os-atlas: A foundation action model for generalist gui agents, 2024

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. Os-atlas: A foundation action model for generalist gui agents, 2024. URL https://arxiv.org/abs/2410.23218

    2024 arXiv

  37. [45]

    Scaling computer-use grounding via user interface decomposition and synthesis, 2025

    Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong. Scaling computer-use grounding via user interface decomposition and synthesis, 2025. URL ...

    2025

  38. [46]

    Aria-ui: Visual grounding for gui instructions, 2024

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions, 2024. URL https://arxiv.org/abs/2412.16256

    2024 arXiv

  39. [47]

    Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning

    Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, and Bo Li. Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. 2025. URL https://arxiv.org/abs/2505.12370

    2025 arXiv

  40. [48]

    Ufo: A ui-focused agent for windows os interaction, 2024

    Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Ufo: A ui-focused agent for windows os interaction, 2024. URL https://arxiv.org/abs/2402.07939

    2024 arXiv

  41. [49]

    Large language model-brained gui agents: A survey

    Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Large language model-brained gui agents: A survey. 2025 a . URL https://arxiv.org/abs/2411.18279

    2025 arXiv

  42. [50]

    Ufo2: The desktop agentos

    Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, Liqun Li, Yu Kang, Zhao Jiang, Suzhen Zheng, Rujia Wang, Jiaxu Qian, Minghua Ma, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Ufo2: The desktop agento...

    2025 arXiv

  43. [51]

    Appagent: Multimodal agents as smartphone users, 2023

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users, 2023. URL https://arxiv.org/abs/2312.13771

    2023 arXiv

  44. [52]

    Gui-g1: Understanding r1-zero-like training for visual grounding in gui agents

    Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu. Gui-g1: Understanding r1-zero-like training for visual grounding in gui agents. 2025. URL https://arxiv.org/abs/2505.15810

    2025 arXiv

  45. [53]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  46. [54]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  47. [55]

    Pattern Anal

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.