Pith. sign in

REVIEW 4 major objections 5 minor 10 cited by

GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Drop the thinking: a 3B GUI grounding model trained without CoT tops 7B rivals.

desk verdict A solid empirical study of R1-Zero-style RL for GUI grounding, with a sensible recipe that likely works, but the headline SOTA claim outruns the controlled evidence. read the letter →

arxiv 2505.15810 v2 pith:W2YIAKPG submitted 2025-05-21 cs.CL cs.AIcs.CV

Yuqi Zhou , Sunhao Dai , Shuai Wang , Kaiwen Zhou , Qinglin Jia , Jun Xu This is my paper · ORCID
classification cs.CLcs.AIcs.CV
keywords GUIgroundingreinforcementlearningGRPOR1-Zerorewardhackingchain-of-thoughtScreenSpotbenchmarkvisual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the R1-Zero recipe of reinforcement learning plus explicit chain-of-thought reasoning, successful for math and code, is actively counterproductive for GUI grounding. Across three pipeline components it shows: longer reasoning chains hurt grounding accuracy; hit- and IoU-based rewards push models to shrink or inflate predicted boxes; and the standard GRPO objective biases training toward easy examples. The paper's proposed fix combines a no-thinking Fast Thinking Template, a box-size regularization reward, and a difficulty-weighted GRPO objective. The resulting GUI-G1-3B, trained on 17K public samples from Qwen2.5-VL-3B-Instruct, reaches 90.3% on ScreenSpot and 37.1% on ScreenSpot-Pro, surpassing larger models such as UI-TARS-7B. If correct, the paper shows that adapting RL to the geometry of the task, rather than adding reasoning, is what unlocks strong visual grounding.

What carries the argument

The machinery is a three-part training intervention. First, a Fast Thinking Template asks the model to output the bbox coordinates in JSON directly, with no <think> block. Second, the reward is $R_{\text{Hit}} + 0.25 R_{\text{IoU}} + 0.125 R_{\text{Box}}$, where $R_{\text{Box}} = \frac{4}{x_{p1}+x_{p2}+y_{p1}+y_{p2}}$ with each term like $x_{p1}=1/(1-|\hat{x}_1-x_1|/\text{image width})$; this term penalizes box-size mismatch and counters the opposing hacking pressures. Third, the GRPO objective is multiplied by $w_q = 0.5 + \frac{1/\lambda_q - \min_i(1/\lambda_i)}{\max_i(1/\lambda_i)-\min_i(1/\lambda_i)}$, with $\lambda_q$ the relative box size of the target normalized by image width plus height, and the length normalization $1/|o_i|$ is replaced by a constant. The difficulty weight is the mechanism that shifts gradient mass toward small-target, harder examples.

What would settle it

A controlled comparison that keeps the Fast Thinking Template and the box-size reward identical but sets $w_q=1$ for every sample would directly test the difficulty-weighting claim; if ScreenSpot-Pro accuracy does not drop, the gain is not caused by the weighting. A complementary check is to run the method on a benchmark where tiny targets are not harder (for example, high-resolution icons with large clear labels), where the inverse-size proxy should lose its effectiveness if it is truly a difficulty signal.

Watch

Extended reading notes

Core claim

The central claim is that the gains attributed to 'thinking' in R1-style GUI agents actually come from the online RL signal, and that explicit chain-of-thought templates reduce grounding quality because grounding depends more on visual tokens than on text tokens. The paper identifies two opposite failure modes in common rewards: hit-based rewards reward point accuracy and therefore shrink predicted boxes, while IoU-based rewards inflate boxes to raise overlap, so optimizing either alone produces reward hacking. It introduces a box-size reward $R_{\text{Box}}$ that compares each predicted edge to the ground-truth edge scaled by image width, and a modified GRPO objective that replaces the per-response length $|o_i|$ with a constant $\text{Max\_Tokens}$ and multiplies the objective by a difficulty weight $w_q$ built from the inverse of the relative box size, so smaller targets receive larger gradients. The claimed result is that this combination, without any reasoning template and with one training epoch on 17K samples, sets a new state of the art on ScreenSpot and ScreenSpot-Pro for models of this size.

Load-bearing premise

The load-bearing premise is that smaller ground-truth boxes are genuinely harder grounding examples, so the inverse-size weight $w_q$ correctly identifies which samples should receive larger gradients during policy updates.

Editorial extensions

If this is right

  • GUI-G1-3B's numbers on ScreenSpot (90.3%) and ScreenSpot-Pro (37.1%) become the reference points for resource-efficient GUI grounding, beating InfiGUI-R1-3B and the larger UI-TARS-7B.
  • The inverse relation between reasoning length and grounding accuracy implies that GUI grounding should be treated as a fast, perception-driven task, so future R1-style GUI agents should not inherit thinking templates from general RL recipes.
  • Removing the length normalization from GRPO and weighting by difficulty should generalize beyond GUI grounding to other perception tasks where output length is not a proxy for effort.
  • The box-size reward provides a concrete recipe for preventing size-based reward hacking in any grounding RL setup, not just GUI agents.
  • The small training budget (17K samples, one epoch, no KL regularization) suggests that much of the capability is unlocked from pretrained knowledge via the RL signal rather than from new task-specific data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the inverse-size difficulty proxy holds, the same weighting could be applied to other perception tasks with scale-imbalanced targets, such as small-object detection in natural images, where similar easy-example bias is observed.
  • The template comparison suggests that the 'aha moment' celebrated in R1-Zero visual reasoning may not transfer to grounding; a direct comparison on the same base model between a thinking-free and a thinking template would settle whether the template, rather than the data, drives the difference.
  • A testable extension is to replace the hand-scaled $w_q$ with a learned or adaptive difficulty weight, or to derive the weight from model uncertainty, which would decouple the difficulty proxy from box size and show whether the mechanism is difficulty weighting per se or merely upweighting small objects.
  • The reward combination suggests a general principle: when multiple metrics are complementary and competing, add an explicit constraint on the object being optimized (here box size) rather than relying on tuned linear combinations of the metrics alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies R1-Zero-style reinforcement learning for visual grounding in GUI agents, decomposing the pipeline into input template, reward design, and policy update. Through analyses on a 900-sample setup, it reports that longer chain-of-thought reasoning hurts grounding accuracy, that hit-based and IoU-based rewards induce opposite box-size reward hacking, and that GRPO exhibits length and difficulty biases. The authors propose a Fast Thinking Template, a box-size reward RBox added to RHit and RIoU, and a modified GRPO objective with no length normalization and a difficulty-dependent weight. They train GUI-G1-3B, initialized from Qwen2.5-VL-3B-Instruct, on about 17K public samples and report 90.3% on ScreenSpot and 37.1% on ScreenSpot-Pro, claiming state-of-the-art results for models of comparable size and an advantage over the larger UI-TARS-7B.

Significance. If the reported results hold, the paper offers a practical and data-efficient recipe for adapting R1-Zero-style RL to GUI grounding, with valuable negative evidence against reasoning templates for grounding tasks and a documented reward-hacking phenomenon. The use of public external benchmarks, a relatively small fully public training set, and a released repository are strengths that increase the reproducibility value of the work. However, the causal attribution of the final SOTA numbers to the proposed objective changes is not established by the current experimental design, and the difficulty-weight proxy is not validated for this base model; these issues prevent me from treating the recipe claim as fully supported at this stage.

major comments (4)
  1. [§3.3, §4, Table 2] The central attribution of the final gains to the proposed GRPO modifications is not established. Table 2, the only ablation of the RL objective, is run on the 900-sample analysis setup described in Appendix C.1 and reports cumulative gains of 82.3 (standard GRPO), 83.2 (Max_Tokens), and 83.3 (difficulty weighting); the final GUI-G1-3B is trained on the 17K set and changes at least five things simultaneously relative to prior R1-style agents: the no-thinking template, RHit+alpha*RIoU+beta*RBox, replacing |oi| with Max_Tokens, a query-level difficulty weight, and an 8-shot consistency filter. Any of these factors, especially the template and the data filter, could shift ScreenSpot accuracy through output format and training distribution alone. A factorial ablation at the 17K scale, or a deflated claim that the combined recipe is SOTA, is needed to connect the headline numbers to the proposed fixes.
  2. [§3.3, Appendix C.3] The difficulty weight wq is computed from the inverse relative box size, under the claim that smaller boxes are harder. This proxy is plausible for professional GUI datasets but is not validated on this base model or this training distribution; the cited support [17] is about ScreenSpot-Pro statistics, not about Qwen2.5-VL-3B-Instruct's error patterns. Since both RBox and wq depend on box size, the reweighting may simply reinforce output-size regularization rather than capture true per-sample difficulty. Please provide base-model accuracy as a function of lambda_q on the training data, or an ablation with randomized or shuffled wq, to demonstrate that the difficulty weighting is acting for the stated reason.
  3. [§3.2, Table 1] The definition of RBox is inconsistent between Table 1 and the body text. Table 1 lists RBox = 4/(xp1+xp2+yp1+yp2), while the text defines xp1 = 1/(1 - |xhat1 - x1|/image_width), which suggests the reward is based on side-distance matching rather than simply matching box size. As written, the exact functional form and the gradient behavior of RBox are unclear, and this matters because RBox is one of the three core proposed contributions. Please give a single precise formula and state what quantity it is designed to match.
  4. [§4, Tables 4 and 5] The state-of-the-art claims rest on single-run comparisons with numbers taken from prior papers. GUI-G1-3B outperforms InfiGUI-R1-3B by 2.8 points on ScreenSpot and by 1.4 points on ScreenSpot-Pro, margins that are within plausible run-to-run variation for RL-trained models, especially given that the main ablation in Table 2 also reports differences of about one point with no error bars. Please report variance across at least three training or evaluation seeds for the final model and, where possible, for the strongest baselines under the same evaluation protocol, or explicitly hedge the superiority claim.
minor comments (5)
  1. [§3.3, Eq. (2) and Table 2] The notation for the difficulty weight is inconsistent: Section 3.3 and Appendix C.3 use wq, while Table 2 and Section 3.4 use wp. Please unify the notation.
  2. [Figure 3] The legend entry "Golden" appears to be a typo for "Ground Truth".
  3. [Appendix D.1, Table 6] The sum of the listed samples is 17,485, which is consistent with "17K," but the paper does not report the number of samples removed by the eight-shot consistency filter; please clarify whether 17K refers to the retained or the original count.
  4. [§2 and §4] Section 2 presents the GRPO objective with KL-divergence regularization, while Section 4 states that no KL divergence regularization is applied in the final training; the analysis experiments in Appendix C.1 also use no KL by default. Please clarify whether the "Standard GRPO" baseline in Table 2 uses the same no-KL setting, so that the comparison isolates the objective changes rather than the KL term.
  5. [Table 4] For the aggregated rows of Claude Computer Use and Gemini 2.0, the paper reports only an average score; please indicate the evaluation protocol or source for these aggregated values, since the column structure otherwise provides per-domain breakdowns.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the RL recipe is evaluated on external benchmarks and its components are not defined in terms of the target results.

full rationale

The paper's derivation chain is self-contained against external benchmarks. The difficulty weight w_q (Appendix C.3) is a training-time loss reweighting computed from the ground-truth relative box size of each training sample; it is not fitted to ScreenSpot/ScreenSpot-Pro and does not rename those evaluation targets. The RBox reward is a rule-based training signal on the predicted box relative to the ground truth, not a prediction of benchmark accuracy. Length-normalization removal and difficulty weighting are credited to external prior work [25], and the relative-box-size difficulty proxy to [17]; neither is a self-citation. The only self-citation is [45] in a general related-work sentence and is not load-bearing. The paper's own Limitations (B.2) concede that dataset composition and hyperparameter tuning are not fully explored, which is a causal-attribution/robustness concern about the final 17K model rather than a circularity: Table 2 ablations are on a 900-sample setup, but that is a confound, not an equation-level equivalence. Evaluation uses ScreenSpot and ScreenSpot-Pro with results from original papers, so the SOTA claim is externally checkable. No quoted step reduces by construction to its own inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method depends on a few hand-set coefficients and two domain assumptions. RBox and w_q are scoring and weighting functions, not new entities. The main free parameters are the reward weights and the difficulty-weight scaling interval; the Max_Tokens constant is cited but its value is not reported.

free parameters (4)
  • alpha (RIoU reward weight) = 0.25
    Weights the IoU term in RHit + alpha*RIoU + beta*RBox; chosen in Section 4 with no sensitivity analysis reported.
  • beta (RBox reward weight) = 0.125
    Weights the box-size regularization reward; chosen in Section 4 with no sensitivity analysis reported.
  • Difficulty-weight interval bounds = (0.5, 1.5]
    Normalized inverse box-size scores are linearly rescaled to the interval (0.5, 1.5] in Appendix C.3; the bounds are hand-chosen to keep weights stable.
  • Max_Tokens constant = not reported
    Replaces |oi| in the GRPO objective to remove length bias (Section 3.3); the numeric value is cited from [25] but not stated in the paper.
assumptions (3)
  • domain assumption GRPO bias analysis from [25], including length bias and difficulty bias, transfers to multimodal GUI grounding.
    Section 3.3 builds the modified objective on these cited observations without re-deriving them for grounding.
  • domain assumption Ground-truth relative box size is a valid proxy for grounding difficulty.
    Used to define w_q in Section 3.3 and Appendix C.3; the entire difficulty-aware reweighting depends on this assumption.
  • standard math The GRPO update rule (Eq. 2) is a valid policy-gradient estimator.
    Imported from DeepSeekMath [33] and not re-proven; the training results assume the optimizer behaves as specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents." pith.science (2026). https://pith.science/paper/W2YIAKPG

@misc{pith2026250515810,
  author       = {Pith},
  title        = {Pith review of: GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W2YIAKPG}},
  note         = {Machine review of arXiv:2505.15810}
}
read the original abstract

Recent Graphical User Interface (GUI) agents replicate the R1-Zero paradigm, coupling online Reinforcement Learning (RL) with explicit chain-of-thought reasoning prior to object grounding and thereby achieving substantial performance gains. In this paper, we first conduct extensive analysis experiments of three key components of that training pipeline: input design, output evaluation, and policy update-each revealing distinct challenges arising from blindly applying general-purpose RL without adapting to GUI grounding tasks. Input design: Current templates encourage the model to generate chain-of-thought reasoning, but longer chains unexpectedly lead to worse grounding performance. Output evaluation: Reward functions based on hit signals or box area allow models to exploit box size, leading to reward hacking and poor localization quality. Policy update: Online RL tends to overfit easy examples due to biases in length and sample difficulty, leading to under-optimization on harder cases. To address these issues, we propose three targeted solutions. First, we adopt a Fast Thinking Template that encourages direct answer generation, reducing excessive reasoning during training. Second, we incorporate a box size constraint into the reward function to mitigate reward hacking. Third, we revise the RL objective by adjusting length normalization and adding a difficulty-aware scaling factor, enabling better optimization on hard samples. Our GUI-G1-3B, trained on 17K public samples with Qwen2.5-VL-3B-Instruct, achieves 90.3% accuracy on ScreenSpot and 37.1% on ScreenSpot-Pro. This surpasses all prior models of similar size and even outperforms the larger UI-TARS-7B, establishing a new state-of-the-art in GUI agent grounding. The project repository is available at https://github.com/Yuqi-Zhou/GUI-G1.

Figures

Figures reproduced from arXiv: 2505.15810 by the authors.

Figure 1
Figure 1. This framework employs the GRPO algorithm for optimization, emphasizing three critical [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (Left) shows the grounding accuracy under varying numbers of output tokens and image tokens. “Text” refers to cases where the target is a textual element, while “Icon” refers to image targets. (Right) presents the grounding accuracy on the Text and Icon subsets across different image sizes. Within each group, samples are evenly divided based on their text ratio. Longer thinking leads to worse grounding performance. … view at source ↗
Figure 3
Figure 3. Changes in accuracy (left), IoU (middle), and relative box size (right) across policy iterations [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: (Left) Two cases with predicted bounding boxes and golden-truth boxes. (Right) Two examples illustrating why RIoU favors larger boxes, while RHit prefers smaller ones. 3.3 Analysis on GRPO Objective Recent approaches to improving GUI agents [24, 27, 39] have adopted RL…
Figure 5
Figure 5. Figure 5: Illustration of the response-level length biases and query-level difficulty biases in GRPO. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Changes in output length and proportion of extreme samples during policy training. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPO

    cs.CV 2026-07 accept novelty 7.0 of 10

    A single MLLM jointly improves region captioning and localization by rewarding captions that let it reconstruct the original mask, needing only region inputs.

  2. Grounding Computer Use Agents on Human Demonstrations

    cs.LG 2025-11 conditional novelty 7.0 of 10

    GroundCUA, a 3.56M-element human-annotated desktop grounding dataset, and GroundNext models achieve strong UI grounding with less than one-tenth the SFT data of prior work.

  3. GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

    cs.CV 2025-11 conditional novelty 7.0 of 10

    Supervising an MLLM's intrinsic self-attention with patch-level GUI labels, aggregated via a learnable anchor token and hidden-state-selected query tokens, reaches state-of-the-art 3B-scale GUI grounding accuracy with...

  4. GUI-AC: Enhancing Continual Learning in GUI Agents

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    GUI-AC stabilizes RFT for non-stationary GUI data by down-weighting noisy advantages and relaxing clipping bounds via a grounding certainty term.

  5. UItron: Foundational GUI Agent with Advanced Perception and Planning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    UItron, trained with multi-stage SFT and curriculum RL on a new 1M-step Chinese app dataset, beats prior GUI agents on perception, grounding, planning, and especially Chinese mobile app scenarios.

  6. Phi-Ground Tech Report: Advancing Perception in GUI Grounding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Phi-Ground models achieve state-of-the-art click accuracy on five GUI grounding benchmarks for models under 10B parameters using a 40M-sample training recipe with text-first inputs, random-resize augmentation, uniform...

  7. MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment

    cs.LG 2025-07 conditional novelty 5.0 of 10

    MobileGUI-RL applies online RL with self-generated and filtered tasks plus trajectory-level rewards to mobile GUI agents, reporting improved success rates on AndroidWorld and AITW benchmarks.

  8. SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control

    cs.AI 2025-08 conditional novelty 4.0 of 10

    A multi-agent RL workflow that interleaves single-agent updates, applied to mobile GUI control, achieves SOTA zero-shot performance and a +14.8 MATH500 gain.

  9. GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Modeling GUI elements as Gaussian distributions instead of binary targets yields 92.0% (ScreenSpot), 93.3% (ScreenSpot-v2), and 47.5% (ScreenSpot-Pro) for a 7B model, outperforming UI-TARS-72B by a relative 24.7% on t...

  10. DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning

    cs.AI 2025-06 conditional novelty 4.0 of 10

    Separating text and icon grounding with iterative zooming improves GUI-element localization accuracy of existing vision-language models without retraining.

Reference graph

Works this paper leans on

45 extracted references · 12 canonical work pages · cited by 10 Pith papers

  1. [17]

    K. Li, Z. Meng, H. Lin, Z. Luo, Y . Tian, J. Ma, Z. Huang, and T.-S. Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use.arXiv preprint arXiv:2504.07981, 2025

    arXiv 2025

  2. [1]

    Ahmadian, C

    A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248–12267, 2024

    work page 2024

  3. [2]

    Developing a computer use model

    Anthropic. Developing a computer use model. https://www.anthropic.com/news/ developing-computer-use, 2024. Accessed: 2025-04-12

    work page 2024

  4. [3]

    C. Bai, X. Zang, Y . Xu, S. Sunkara, A. Rastogi, J. Chen, et al. Uibert: Learning generic multimodal representations for ui understanding.arXiv preprint arXiv:2107.13731, 2021

    arXiv 2021

  5. [4]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [5]

    Y . Chai, S. Huang, Y . Niu, H. Xiao, L. Liu, D. Zhang, P. Gao, S. Ren, and H. Li. Amex: Android multi-annotation expo dataset for mobile gui agents.arXiv preprint arXiv:2407.17490, 2024

    arXiv 2024

  7. [6]

    Z. Chen, Y . Min, B. Zhang, J. Chen, J. Jiang, D. Cheng, W. X. Zhao, Z. Liu, X. Miao, Y . Lu, et al. An empirical study on eliciting and improving r1-like reasoning models.arXiv preprint arXiv:2503.04548, 2025

    arXiv 2025

  8. [7]

    Cheng, Q

    K. Cheng, Q. Sun, Y . Chu, F. Xu, L. YanTao, J. Zhang, and Z. Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313–9332, 2024

    work page 2024

Show all 45 references
  1. [8]

    DeepMind

    G. DeepMind. Gemini-2.0 (project mariner). https://deepmind.google/technologies/ project-mariner, 2024. Accessed: 2025-04-12

    2024

  2. [9]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo- gie...

    2019

  3. [10]

    B. Gou, R. Wang, B. Zheng, Y . Xie, C. Chang, Y . Shu, H. Sun, and Y . Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. In13th International Conference on Learning Representations, ICLR 2025, 2025

    2025

  4. [11]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [12]

    W. Hong, W. Wang, Q. Lv, J. Xu, W. Yu, J. Ji, Y . Wang, Z. Wang, Y . Dong, M. Ding, et al. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024

    2024

  6. [13]

    Huang, B

    W. Huang, B. Jia, Z. Zhai, S. Cao, Z. Ye, F. Zhao, Z. Xu, Y . Hu, and S. Lin. Vision-r1: Incentiviz- ing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  7. [14]

    Kahneman.Thinking, Fast and Slow

    D. Kahneman.Thinking, Fast and Slow. Farrar, Straus and Giroux, New York, October 2011

    2011

  8. [15]

    W. Kool, H. van Hoof, and M. Welling. Buy 4 reinforce samples, get a baseline for free! 2019

    2019

  9. [16]

    Li and Y

    G. Li and Y . Li. Spotlight: Mobile ui understanding using vision-language models with a focus. In11th International Conference on Learning Representations, ICLR 2023, 2023

    2023

  10. [18]

    M. Li, J. Zhong, S. Zhao, Y . Lai, and K. Zhang. Think or not think: A study of explicit thinking in rule-based visual reinforcement fine-tuning.arXiv e-prints, pages arXiv–2503, 2025. 10

    2025

  11. [19]

    W. Li, W. Bishop, A. Li, C. Rawles, F. Campbell-Ajala, D. Tyamagundlu, and O. Riva. On the effects of data scale on computer control agents.arXiv e-prints, pages arXiv–2406, 2024

    2024

  12. [20]

    Y . Li, G. Li, X. Zhou, M. Dehghani, and A. Gritsenko. Vut: Versatile ui transformer for multi-modal multi-task user interface modeling.arXiv preprint arXiv:2112.05692, 2021

    2021 arXiv

  13. [21]

    Z. Li, K. You, H. Zhang, D. Feng, H. Agrawal, X. Li, M. P. S. Moorthy, J. Nichols, Y . Yang, and Z. Gan. Ferret-ui 2: Mastering universal user interface understanding across platforms.arXiv preprint arXiv:2410.18967, 2024

    2024 arXiv

  14. [22]

    K. Q. Lin, L. Li, D. Gao, Z. Yang, S. Wu, Z. Bai, W. Lei, L. Wang, and M. Z. Shou. Showui: One vision-language-action model for gui visual agent.arXiv preprint arXiv:2411.17465, 2024

    2024 arXiv

  15. [23]

    X. Liu, B. Qin, D. Liang, G. Dong, H. Lai, H. Zhang, H. Zhao, I. L. Iong, J. Sun, J. Wang, et al. Autoglm: Autonomous foundation agents for guis.arXiv preprint arXiv:2411.00820, 2024

    2024 arXiv

  16. [24]

    Y . Liu, P. Li, C. Xie, X. Hu, X. Han, S. Zhang, H. Yang, and F. Wu. Infigui-r1: Advancing multi- modal gui agents from reactive actors to deliberative reasoners.arXiv preprint arXiv:2504.14239, 2025

    2025 arXiv

  17. [25]

    Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  18. [26]

    Y . Lu, J. Yang, Y . Shen, and A. Awadallah. Omniparser for pure vision based gui agent.arXiv preprint arXiv:2408.00203, 2024

    2024 arXiv

  19. [27]

    Z. Lu, Y . Chai, Y . Guo, X. Yin, L. Liu, H. Wang, G. Xiong, and H. Li. Ui-r1: Enhancing action prediction of gui agents by reinforcement learning.arXiv preprint arXiv:2503.21620, 2025

    2025 arXiv

  20. [28]

    F. Meng, L. Du, Z. Liu, Z. Zhou, Q. Lu, D. Fu, T. Han, B. Shi, W. Wang, J. He, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  21. [29]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. urlhttps://openai.com/index/learning-to-reason-with-llms/. Accessed: 15 March 2025

    2025

  22. [30]

    Gpt-4o, 2024

    OpenAI. Gpt-4o, 2024. Accessed: 2025-01-03

    2024

  23. [31]

    Y . Peng, G. Zhang, M. Zhang, Z. You, J. Liu, Q. Zhu, K. Yang, X. Xu, X. Geng, and X. Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025

    2025 arXiv

  24. [32]

    Y . Qin, Y . Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y . Li, S. Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  25. [33]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  26. [34]

    H. Shen, P. Liu, J. Li, C. Fang, Y . Ma, J. Liao, Q. Shen, Z. Zhang, K. Zhao, Q. Zhang, R. Xu, and T. Zhao. Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  27. [35]

    K. Team, A. Du, B. Yin, B. Xing, B. Qu, B. Wang, C. Chen, C. Zhang, C. Du, C. Wei, et al. Kimi-vl technical report.arXiv preprint arXiv:2504.07491, 2025

    2025 arXiv

  28. [36]

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  29. [37]

    S. Wang, W. Liu, J. Chen, Y . Zhou, W. Gan, X. Zeng, Y . Che, S. Yu, X. Hao, K. Shao, et al. Gui agents with foundation models: A comprehensive survey.arXiv preprint arXiv:2411.04890, 2024. 11

    2024 arXiv

  30. [38]

    Z. Wu, Z. Wu, F. Xu, Y . Wang, Q. Sun, C. Jia, K. Cheng, Z. Ding, L. Chen, P. P. Liang, et al. Os-atlas: A foundation action model for generalist gui agents. In13th International Conference on Learning Representations, ICLR 2025, 2025

    2025

  31. [39]

    Xia and R

    X. Xia and R. Luo. Gui-r1: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458, 2025

    2025 arXiv

  32. [40]

    Y . Xu, Z. Wang, J. Wang, D. Lu, T. Xie, A. Saha, D. Sahoo, T. Yu, and C. Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction.arXiv preprint arXiv:2412.04454, 2024

    2024 arXiv

  33. [41]

    Y . Yang, Y . Wang, D. Li, Z. Luo, B. Chen, C. Huang, and J. Li. Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256, 2024

    2024 arXiv

  34. [42]

    Zhang, L

    L. Zhang, L. Gao, and M. Xu. Does chain-of-thought reasoning help mobile gui agent? an empirical study.arXiv preprint arXiv:2503.16788, 2025

    2025 arXiv

  35. [43]

    Zhang, W

    Z. Zhang, W. Xie, X. Zhang, and Y . Lu. Reinforced ui instruction grounding: Towards a generic ui task automation api.arXiv preprint arXiv:2310.04716, 2023

    2023 arXiv

  36. [44]

    aha moment

    H. Zhou, X. Li, R. Wang, M. Cheng, T. Zhou, and C.-J. Hsieh. R1-zero’s" aha moment" in visual reasoning on a 2b non-sft model.arXiv preprint arXiv:2503.05132, 2025

    2025 arXiv

  37. [45]

    Y . Zhou, S. Wang, S. Dai, Q. Jia, Z. Du, Z. Dong, and J. Xu. Chop: Mobile operating assistant with constrained high-frequency optimized subtask planning.arXiv preprint arXiv:2503.03743, 2025. 12 A Acknowledgements We thank Yuchong Sun for helpful discussions and insights duri...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.