REVIEW 3 major objections 5 minor 5 cited by
Phi-Ground Tech Report: Advancing Perception in GUI Grounding
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Phi-Ground, a sub-10B model family trained on 40M+ samples, achieves state-of-the-art click accuracy on all five GUI grounding benchmarks when paired with a planner, and on three in the end-to-end setting.
desk verdict Useful training recipe for GUI grounding, but the reported SOTA is inflated by selecting the best checkpoint on the test benchmarks. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the two-stage grounding pipeline with reference-expression expansion. A planner, run with a fixed prompt, rewrites a short UI instruction into a long description covering function, position, and appearance; the trained Phi-Ground model then reads that description together with the screenshot and emits a click point directly as ordinary text coordinates, with positions represented relative to the image and scaled by 1000. Training data are produced by a 'Long-gold' variant in which the annotating multimodal model receives the ground-truth box and a crop, so the generated descriptions are precise and separable. The paper also identifies the scaling-friendly ingredients: plain next-token prediction on text coordinates outperforms tokenized coordinate tokens, label smoothing, and loss reweighting once training data exceed about a million samples; random resize of screenshots onto a white canvas is the augmentation that most helps high-resolution benchmarks; and resampling web-crawl data so element center points are spread uniformly over a 50x50 grid prevents distribution bias from common web layouts. Model variants share the same backbone but vary the number of image crops, which lets the paper treat image-token count and parameter count together as the compute axis of a scaling law.
What would settle it
Release the planner outputs and rerun every sub-10B baseline in a shared harness using identical long reference expressions; if a prior model such as UGround-v1 or UI-TARS-1.5 then scores near 55.0 on ScreenSpot-pro, the gains attributed to Phi-Ground's recipe would vanish. Independently, re-annotate a sample of the proprietary Gold set from public screenshots and check whether its element positions and reference styles resemble Phi-Ground's training data more closely than do ScreenSpot-pro or UI-Vision; strong resemblance would undercut the claimed generalization.
Extended reading notes
Core claim
The paper's central claim is that a grounding-focused multimodal model under 10B parameters, trained with a specific data-first recipe, can outperform both the listed open-source GUI agents and commercial computer-use systems on click localization. In the two-stage 'agent' setting, an advanced multimodal planner turns a short instruction into a combined functional, positional, and appearance reference expression, and the small Phi-Ground model maps that expression to a click coordinate; there the paper reports state-of-the-art results on all five benchmarks, including 55.0 on ScreenSpot-pro and 36.2 on UI-Vision. In the single-model 'end-to-end' setting, where the model receives only the short instruction and screenshot, it reports state-of-the-art results on three benchmarks, including 43.2 on ScreenSpot-pro and 27.2 on UI-Vision. The paper attributes the gains to scale combined with several non-obvious choices: text-before-image input order, random resize augmentation, resampling training elements so their centers cover the canvas uniformly, and multi-round DPO after supervised training has saturated.
Load-bearing premise
The comparison stands or falls on whether the evaluation protocol treats all models alike: baselines are re-run with the paper's own planner prompts and measured on the paper's proprietary Gold dataset, with no shared harness, released planner outputs, or independent replication, so if those choices happen to favor Phi-Ground's training distribution the SOTA claim would weaken.
Editorial extensions
If this is right
- Computer-use agents can be built as a small clicker model plus a general planner, lowering deployment cost and latency while keeping high click accuracy.
- Training data quantity in the tens of millions makes simple text-coordinate prediction sufficient, so future work can drop tokenized-coordinate schemes, label smoothing, and loss reweighting.
- Random resize augmentation and uniform element-position resampling should become standard in GUI grounding data pipelines, especially for high-resolution or small-target benchmarks.
- Preference optimization such as multi-round DPO can push a perception model past its supervised-training plateau by improving robustness to data distribution and noisy labels, not by adding reasoning.
- Evaluation across five benchmarks rather than one or two changes conclusions: a model can top ScreenSpot-V2 while losing on harder high-resolution and Windows-native sets, so multi-benchmark reporting is necessary.
Reading between the lines
- A testable extension follows from the two-stage decomposition: benchmark grounding models with a fixed planner and with released planner outputs, so the perception component can be measured separately from planning quality; the paper's error analysis already suggests that a large share of ScreenSpot-pro failures is caused by the planner rather than the model.
- The random-resize result is likely transferable to other high-resolution perception tasks where targets are tiny, such as satellite imagery or medical imaging: presenting the model with randomly scaled canvases may be cheaper than simply adding more resolution.
- The uniform-center resampling result implies that benchmark scores can be sensitive to where interactive elements sit in a screenshot; future datasets may want to report center-position distributions alongside accuracy.
- If multi-round DPO on the model's own rollouts improves a saturated perception model, preference-style objectives may work as a general clean-label regularizer in perception, a hypothesis the paper leaves untested outside GUI grounding.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Phi-Ground, a family of GUI grounding models under 10 billion parameters, and an empirical study of the full training pipeline from data collection to post-training. The authors introduce a two-stage agent setting in which a large MLLM planner expands short instructions into long reference expressions, and a fine-tuned grounding model produces click coordinates in text form. Using up to 40M training samples drawn from open-source, CommonCrawl, Bing search, and human-labeled sources, they study input modality order, output coordinate formats, data augmentation, center-point resampling, in-domain post-training with SFT/DPO/curriculum/reject-sampling, and scaling with respect to parameter count and image tokens. The paper claims state-of-the-art results on all five benchmarks in the agent setting (e.g., 55.0 on ScreenSpot-Pro and 36.2 on UI-Vision) and on three of five benchmarks in the end-to-end setting (43.2 and 27.2 on the same two benchmarks).
Significance. If the results hold, the paper provides a detailed and useful recipe for GUI grounding, with several valuable negative results (tokenized coordinates, label smoothing, loss reweighting) and a credible scaling analysis that accounts for image tokens as a computational axis. The authors deserve credit for evaluating on multiple benchmarks, for disclosing failed approaches, and for the careful error decomposition in Section 6.2. However, the central SOTA claim currently rests on an evaluation protocol that allows the authors to select the best checkpoint using the test benchmarks themselves, and on a proprietary evaluation set and self-run baseline comparisons that cannot be independently checked. These issues directly affect the validity of the headline numbers, so the paper needs substantial revision before the claims can be accepted.
major comments (3)
- [§5.5] Section 5.5 states: 'We also observed some fluctuations in the performance across different benchmarks during training, which led us to conduct tests at regular intervals and ultimately select the checkpoint that performed best across all benchmarks.' This means the scores reported for Phi-Ground in Table 10 are maxima over checkpoints evaluated on the same test benchmarks that are used for final comparison, not the scores of a single fixed model. The baselines (UI-TARS, UGround, OS-Atlas, SeeClick) are fixed released checkpoints that receive no such benefit. The comparison is therefore asymmetric, and because several SOTA margins in Table 10 are small (55.0 vs 48.8 on ScreenSpot-Pro, 36.2 vs 29.4 on UI-Vision, 73.87 vs 71.58 on ShowDown), the reported 'SOTA on all five benchmarks' may be an artifact of checkpoint selection rather than a property of a deployable checkpoint. The authors should re-evaluate all results with a single fixed checkpoint chosen using a held-out validation split, or explicitly report the variance over training checkpoints and demonstrate that the SOTA claims survive without test-set-based selection.
- [§4.1, §6.1, Table 10] The evaluation of baselines and of the proposed models is not provided as a shared, reproducible harness. All baselines are re-run by the authors using their own planner prompts and their own pipeline for generating long reference expressions, and the Gold dataset is proprietary. Since the training data also uses long reference expressions generated by the same planner style (Section 4.2), there is a risk that the comparison is biased toward Phi-Ground's training distribution. The paper should release the evaluation code, the exact prompts and planner outputs, or at least a detailed protocol that would let others reproduce the baseline numbers; otherwise the relative improvements over UGround-v1 and UI-TARS-1.5 cannot be verified.
- [Tables 2-7, 10] The paper reports only single accuracy values without confidence intervals, error bars, or statistical significance tests. Given the small differences that underlie several SOTA claims, and the checkpoint-selection issue above, the reported margins (e.g., 55.0 vs 48.8 on ScreenSpot-Pro) may be within run-to-run or checkpoint-to-checkpoint variability. The authors should report confidence intervals (e.g., Wilson intervals for binary accuracy, or bootstrap over samples) and, where possible, multiple training runs or at least a fixed final checkpoint selected on a validation split.
minor comments (5)
- [Table 8, Figures 12-14] Several tables and figures contain garbled sequences such as '/uni00000013/uni00000011/uni00000014...' in the rendered text. As presented, these parts of Table 8 and the captions of Figures 12-14 are unreadable and need to be regenerated with correct encoding.
- [Figure 1 and Section 4.1] The evaluation dataset is referred to as 'Gold' in most places but as 'Golden' in Figure 7 and in the text near Figure 7; please unify the naming.
- [Section 5.2, Algorithms 1-2] The notation in Algorithms 1 and 2 is used before the variables are fully explained. In particular, the box representation (x1, y1, x2, y2) should be stated explicitly at the start of Section 5.2 so the augmentation pseudocode is self-contained.
- [Section 5.5, Figure 13] The text says 'F LOP s= 6N D' in Figure 13, but the surrounding discussion in the main text uses 'N D' as the product of image tokens and parameters; please clarify the exact formula and whether the constant 6 is intended.
- [Section 6.1] In the paragraph after Table 10, the authors discuss ScreenSpot V1 and V2 and state that they did not include mobile data or balance icon/text types; this is an interesting statement, but the numbers for ScreenSpot V1 are not reported in any table. Please add the V1 results or remove the explicit reference to 'V1' in the discussion.
Circularity Check
Reported SOTA scores are the best test-checkpoint scores from §5.5, so the headline comparison reduces to test-set model selection.
-
fitted input called prediction
[Section 5.5 (Scaling) and Table 10]
"We also observed some fluctuations in the performance across different benchmarks during training, which led us to conduct tests at regular intervals and ultimately select the checkpoint that performed best across all benchmarks."
The reported Phi-Ground accuracies in Table 10 are produced by selecting, among checkpoints evaluated during training, the one with the best accuracy on the same five test benchmarks used for the comparison. By construction, each reported number is the maximum over the checkpoints evaluated on the test set, rather than the accuracy of a single fixed model. The fixed released baselines (UI-TARS, UGround, SeeClick, OS-Atlas) cannot receive this selection benefit, and no held-out validation split is described. The margins on key benchmarks are small (55.0 vs 48.8 on ScreenSpot-pro; 36.2 vs 29.4 on UI-Vision; 73.87 vs 71.58 on ShowDown), so the central 'SOTA on all five benchmarks' claim is statistically forced by the selection rule.
full rationale
The paper is an empirical study; coordinate representations, loss variants, data augmentation, and data mixing are tested in ablations against multiple benchmarks, and the final training recipe is not derived from the benchmarks by an equation. There is no self-citation chain or imported uniqueness theorem. The one load-bearing circular step is checkpoint selection on the test benchmarks (Section 5.5), which makes the headline scores, by construction, the best of a set of checkpoints evaluated on the same benchmarks used for the SOTA comparison, while baselines are fixed releases. The Gold dataset shares application domains with the human-labeled training data, but the paper discloses different annotator teams and settings; that is a mild domain-overlap limitation rather than an equation-level circularity. Because the central numerical claims reduce to test-set model selection, the circularity score is 6 rather than 0-2.
Assumptions & free parameters
free parameters (4)
- Random resize augmentation (probability, max screen size) =
100%, 4096 px
- Center-point re-sampling factor R =
R=0.25 (RES=36%) in Figure 10, used for CommonCrawl
- Image token configuration (num_crops) =
6/15/28 crops -> 7/16/29 image tokens
- Training data proportions per dataset =
e.g., CommonCrawl 60%, Fineweb 21%, Windows 10% for 40M
assumptions (3)
- domain assumption Click accuracy (predicted point inside ground-truth bounding box) is a valid and sufficient measure of GUI grounding quality.
- domain assumption GPT-4O and O4-mini produce reference expressions of sufficient quality to evaluate the grounding model fairly.
- domain assumption The Gold dataset, annotated by a separate team, does not systematically favor the authors' in-domain training data.
Cite this review
Pith. "Pith review of Phi-Ground Tech Report: Advancing Perception in GUI Grounding." pith.science (2026). https://pith.science/paper/NKTL3DF3
@misc{pith2026250723779,
author = {Pith},
title = {Pith review of: Phi-Ground Tech Report: Advancing Perception in GUI Grounding},
year = {2026},
howpublished = {\url{https://pith.science/paper/NKTL3DF3}},
note = {Machine review of arXiv:2507.23779}
}
abstract
With the development of multimodal reasoning models, Computer Use Agents (CUAs), akin to Jarvis from \textit{"Iron Man"}, are becoming a reality. GUI grounding is a core component for CUAs to execute actual actions, similar to mechanical control in robotics, and it directly leads to the success or failure of the system. It determines actions such as clicking and typing, as well as related parameters like the coordinates for clicks. Current end-to-end grounding models still achieve less than 65\% accuracy on challenging benchmarks like ScreenSpot-pro and UI-Vision, indicating they are far from being ready for deployment. % , as a single misclick can result in unacceptable consequences. In this work, we conduct an empirical study on the training of grounding models, examining details from data collection to model training. Ultimately, we developed the \textbf{Phi-Ground} model family, which achieves state-of-the-art performance across all five grounding benchmarks for models under $10B$ parameters in agent settings. In the end-to-end model setting, our model still achieves SOTA results with scores of \textit{\textbf{43.2}} on ScreenSpot-pro and \textit{\textbf{27.2}} on UI-Vision. We believe that the various details discussed in this paper, along with our successes and failures, not only clarify the construction of grounding models but also benefit other perception tasks. Project homepage: \href{https://zhangmiaosen2000.github.io/Phi-Ground/}{https://zhangmiaosen2000.github.io/Phi-Ground/}
Figures
Figures from the paper (13 more)
Forward citations
Cited by 5 Pith papers
-
Grounding Computer Use Agents on Human Demonstrations
GroundCUA, a 3.56M-element human-annotated desktop grounding dataset, and GroundNext models achieve strong UI grounding with less than one-tenth the SFT data of prior work.
-
MissClick: Exploiting Digit-Serialized Coordinates to Attack GUI Grounding Models
Place-value-aware attack objectives raise untargeted click-misdirection to 73-75% and targeted hijacking to 45-63% on OS-Atlas and UGround, 17-47 percentage points above prior attacks.
-
GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL
Action-weighted SFT plus KL-regularized GRPO on a curated 81K reasoning dataset lifts open-source GUI agents by 11–22 points on online task-completion benchmarks.
-
Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents
A real-device-centric foundation GUI agent with hybrid GUI+CLI batched actions, AutoResearch data flywheel, online RL, and a proactive harness reaches SOTA mobile and competitive desktop/web scores.
-
HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents
A 3B-scale GUI agent reportedly scores 82.6% on AndroidWorld and 42% on real-device tasks, but the evidence is not independently verified and may overlap with its RL training.
Reference graph
Works this paper leans on
-
[1]
A survey on large language model based autonomous agents.Frontiers of Computer Science, 2024
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents.Frontiers of Computer Science, 2024
2024
-
[2]
The rise and potential of large language model based agents: A survey.Science China Information Sciences, 2025
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey.Science China Information Sciences, 2025
2025
-
[3]
Yuheng Cheng, Ceyao Zhang, Zhengwen Zhang, Xiangrui Meng, Sirui Hong, Wenhao Li, Zihao Wang, Zekai Wang, Feng Yin, Junhua Zhao, et al. Exploring large language model based intelligent agents: Definitions, methods, and prospects.arXiv preprint arXiv:2401.03428, 2024
arXiv 2024
-
[4]
Large language model-brained gui agents: A survey.arXiv preprint arXiv:2411.18279, 2024
Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, et al. Large language model-brained gui agents: A survey.arXiv preprint arXiv:2411.18279, 2024
arXiv 2024
-
[5]
Pascal J Sager, Benjamin Meyer, Peng Yan, Rebekka von Wartburg-Kottler, Layan Etaiwi, Aref Enayati, Gabriel Nobel, Ahmed Abdulkadir, Benjamin F Grewe, and Thilo Stadelmann. A comprehensive survey of agents for computer use: Foundations, challenges, and future directions.arXiv preprint arXiv:2501.16150, 2025
arXiv 2025
-
[6]
Artificial intelligence, machine learning and deep learning in advanced robotics, a review.Cognitive Robotics, 2023
Mohsen Soori, Behrooz Arezoo, and Roza Dastres. Artificial intelligence, machine learning and deep learning in advanced robotics, a review.Cognitive Robotics, 2023
2023
-
[7]
Demetris Vrontis, Michael Christofi, Vijay Pereira, Shlomo Tarba, Anna Makrides, and Eleni Trichina. Artificial intelligence, robotics, advanced technologies and human resource management: a systematic review.Artificial intelligence and international HRM, 2023
work page 2023
-
[8]
Introducing openai o3 and o4-mini, 2025
OpenAI. Introducing openai o3 and o4-mini, 2025
2025
Show all 83 references
-
[9]
Claude sonnet 4, 2025
Anthropic. Claude sonnet 4, 2025
2025
-
[10]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025
2025 arXiv
-
[11]
Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025
2025 arXiv
-
[12]
Operator system card, 2025
OpenAI. Operator system card, 2025
2025
-
[13]
Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024
Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024
2024
-
[14]
Mobile agents and security.IEEE Communica- tions magazine, 1998
Michael S Greenberg, Jennifer C Byington, and David G Harper. Mobile agents and security.IEEE Communica- tions magazine, 1998
1998
-
[15]
A survey of privacy in multi-agent systems.The Knowledge Engineering Review, 2014
Jose M Such, Agustín Espinosa, and Ana García-Fornes. A survey of privacy in multi-agent systems.The Knowledge Engineering Review, 2014
2014
-
[16]
No secrets between the two of us: Privacy concerns over using ai agents
Sohye Lim and Hongjin Shim. No secrets between the two of us: Privacy concerns over using ai agents. Cyberpsychology: Journal of Psychosocial Research on Cyberspace, 2022
2022
-
[17]
Privacy in mobile agents
K Cartrysse and JCA Van Der Lubbe. Privacy in mobile agents. InIEEE First Symposium onMulti-Agent Security and Survivability, 2004. IEEE, 2004. 20 Phi-Ground Tech Report
2004
-
[18]
Reinforcement learning on web interfaces using workflow-guided exploration.ICLR, 2018
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration.ICLR, 2018
2018
-
[19]
Mind2web: Towards a generalist agent for the web.NIPS, 2023
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web.NIPS, 2023
2023
-
[20]
Aguvis: Unified pure vision agents for autonomous gui interaction.ICML, 2025
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction.ICML, 2025
2025
-
[21]
Guicourse: From general vision language models to versatile gui agents.arXiv preprint arXiv:2406.11317, 2024
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. Guicourse: From general vision language models to versatile gui agents.arXiv preprint arXiv:2406.11317, 2024
2024 arXiv
-
[22]
From grounding to planning: Bench- marking bottlenecks in web agents.arXiv preprint arXiv:2409.01927, 2024
Segev Shlomov, Aviad Sela, Ido Levy, Liane Galanti, Roy Abitbol, et al. From grounding to planning: Bench- marking bottlenecks in web agents.arXiv preprint arXiv:2409.01927, 2024
2024
-
[23]
Realwebassist: A benchmark for long-horizon web assistance with real-world users.arXiv preprint arXiv:2504.10445, 2025
Suyu Ye, Haojun Shi, Darren Shih, Hyokun Yun, Tanya Roosta, and Tianmin Shu. Realwebassist: A benchmark for long-horizon web assistance with real-world users.arXiv preprint arXiv:2504.10445, 2025
2025
-
[24]
Ui-vision: A desktop-centric gui benchmark for visual perception and interaction.arXiv preprint arXiv:2503.15661, 2025
Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M Tamer Özsu, Aishwarya Agrawal, David Vazquez, et al. Ui-vision: A desktop-centric gui benchmark for visual perception and interaction.arXiv preprint arXiv:2503.15661, 2025
2025 arXiv
-
[25]
Screenspot-pro: Gui grounding for professional high-resolution computer use.arXiv preprint arXiv:2504.07981, 2025
Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use.arXiv preprint arXiv:2504.07981, 2025
2025 arXiv
-
[26]
Seeclick: Harnessing gui grounding for advanced visual gui agents
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024
2024
-
[27]
Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners.arXiv preprint arXiv:2504.14239, 2025
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners.arXiv preprint arXiv:2504.14239, 2025
2025 arXiv
-
[28]
Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025
2025 arXiv
-
[29]
Ui-tars-1.5.https://seed-tars.com/1.5, 2025
ByteDance Seed. Ui-tars-1.5.https://seed-tars.com/1.5, 2025
2025
-
[30]
Navigating the digital world as humans do: Universal visual grounding for gui agents.ICLR, 2025
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents.ICLR, 2025
2025
-
[31]
Os-atlas: A foundation action model for generalist gui agents.ICLR, 2025
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents.ICLR, 2025
2025
-
[32]
Unitab: Unifying text and box outputs for grounded vision-language modeling
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. InECCV. Springer, 2022
2022
-
[33]
Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.NIPS, 2023
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.NIPS, 2023
2023
-
[34]
Ferret: Refer and ground anything anywhere at any granularity.ICLR, 2024
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity.ICLR, 2024
2024
-
[35]
Unified-io: A unified model for vision, language, and multi-modal tasks.ICLR, 2023
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks.ICLR, 2023
2023
-
[36]
Git: A generative image-to-text transformer for vision and language.arXiv preprint arXiv:2205.14100, 2022
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language.arXiv preprint arXiv:2205.14100, 2022
2022 arXiv
-
[37]
Data augmentation for object detection: A review
Parvinder Kaur, Baljit Singh Khehra, and Er Bhupinder Singh Mavi. Data augmentation for object detection: A review. In2021 IEEE International Midwest Symposium on Circuits and Systems (MWSCAS). IEEE, 2021
2021
-
[38]
Deep learning scaling is predictable, empirically
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. 21 Phi-Ground Tech Report
2017 arXiv
-
[39]
Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020
2001 arXiv
-
[40]
Training compute-optimal large language models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. InNIPS, 2022
2022
-
[41]
Ui-e2i-synth: Advancing gui grounding with large-scale instruction synthesis.arXiv preprint arXiv:2504.11257, 2025
Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, and Yan Lu. Ui-e2i-synth: Advancing gui grounding with large-scale instruction synthesis.arXiv preprint arXiv:2504.11257, 2025
2025 arXiv
-
[42]
A careful examination of large language model performance on grade school arithmetic.NIPS, 2024
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, et al. A careful examination of large language model performance on grade school arithmetic.NIPS, 2024
2024
-
[43]
Skywork: A more open bilingual foundation model.arXiv preprint arXiv:2310.19341, 2023
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, et al. Skywork: A more open bilingual foundation model.arXiv preprint arXiv:2310.19341, 2023
2023 arXiv
-
[44]
The showdown computer control evaluation suite, 2025
General Agents Team. The showdown computer control evaluation suite, 2025
2025
-
[45]
Visualcritic: Making lmms perceive visual quality like humans.arXiv preprint arXiv:2403.12806, 2024
Zhipeng Huang, Zhizheng Zhang, Yiting Lu, Zheng-Jun Zha, Zhibo Chen, and Baining Guo. Visualcritic: Making lmms perceive visual quality like humans.arXiv preprint arXiv:2403.12806, 2024
2024 arXiv
-
[46]
Benchlmm: Benchmarking cross-style visual capability of large multimodal models
Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Yaohang Li, Xing Luo, Chenyu Yi, and Alex Kot. Benchlmm: Benchmarking cross-style visual capability of large multimodal models. InECCV. Springer, 2024
2024
-
[47]
Blink: Multimodal large language models can see but not perceive
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. InECCV. Springer, 2024
2024
-
[48]
Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023
2023 arXiv
-
[49]
Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. InICML. PMLR, 2022
2022
-
[50]
Common crawl - open repository of web crawl data, 2025
2025
-
[51]
Omniparser for pure vision based gui agent
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203, 2024
2024 arXiv
-
[52]
Winclick: Gui grounding with multimodal large language models.arXiv preprint arXiv:2503.04730, 2025
Zheng Hui, Yinheng Li, Tianyi Chen, Colby Banbury, Kazuhito Koishida, et al. Winclick: Gui grounding with multimodal large language models.arXiv preprint arXiv:2503.04730, 2025
2025 arXiv
-
[53]
Hello gpt-4o, 2024
OpenAI. Hello gpt-4o, 2024
2024
-
[54]
Modeling caption diversity in contrastive vision-language pretraining.arXiv preprint arXiv:2405.00740, 2024
Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mahmoud Assran, Andrew Gordon Wilson, Aaron Courville, and Nicolas Ballas. Modeling caption diversity in contrastive vision-language pretraining.arXiv preprint arXiv:2405.00740, 2024
2024 arXiv
-
[55]
Viarl: Adaptive temporal grounding via visual iterated amplification reinforcement learning.arXiv preprint arXiv:2505.15447, 2025
Ziqiang Xu, Qi Dai, Tian Xie, Yifan Yang, Kai Qiu, DongDong Chen, Zuxuan Wu, and Chong Luo. Viarl: Adaptive temporal grounding via visual iterated amplification reinforcement learning.arXiv preprint arXiv:2505.15447, 2025
2025 arXiv
-
[56]
Emma: Efficient visual alignment in multi-modal llms.arXiv preprint arXiv:2410.02080, 2024
Sara Ghazanfari, Alexandre Araujo, Prashanth Krishnamurthy, Siddharth Garg, and Farshad Khorrami. Emma: Efficient visual alignment in multi-modal llms.arXiv preprint arXiv:2410.02080, 2024
2024 arXiv
-
[57]
Llama 2: Open foundation and fine-tuned chat models
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023
2023 arXiv
-
[58]
Adaptive rejection sampling for gibbs sampling.Journal of the Royal Statistical Society: Series C (Applied Statistics), 41(2), 1992
Walter R Gilks and Pascal Wild. Adaptive rejection sampling for gibbs sampling.Journal of the Royal Statistical Society: Series C (Applied Statistics), 41(2), 1992
1992
-
[59]
Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021
2021 arXiv
-
[60]
Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022. 22 Ph...
2022 arXiv
-
[61]
Direct preference optimization: Your language model is secretly a reward model.NIPS, 2023
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.NIPS, 2023
2023
-
[62]
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3), 1992
1992
-
[63]
Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017
2017 arXiv
-
[64]
Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262, 2025
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262, 2025
2025 arXiv
-
[65]
Deepseekmath: Pushing the limits of mathematical reasoning in open language models
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024
2024 arXiv
-
[66]
Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning.arXiv preprint arXiv:2503.21620, 2025
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning.arXiv preprint arXiv:2503.21620, 2025
2025 arXiv
-
[67]
Gui-g1: Understanding r1-zero-like training for visual grounding in gui agents.arXiv preprint arXiv:2505.15810, 2025
Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu. Gui-g1: Understanding r1-zero-like training for visual grounding in gui agents.arXiv preprint arXiv:2505.15810, 2025
2025 arXiv
-
[68]
Gui-r1: A generalist r1-style vision-language action model for gui agents.arXiv preprint arXiv:2504.10458, 2025
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-r1: A generalist r1-style vision-language action model for gui agents.arXiv preprint arXiv:2504.10458, 2025
2025 arXiv
-
[69]
Trl: Transformer reinforcement learning
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https: //github.com/huggingface/trl, 2020
2020
-
[70]
Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023
2023 arXiv
-
[71]
A general theoretical paradigm to understand learning from human preferences
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics. PMLR, 2024
2024
-
[72]
Towards efficient exact optimization of language model alignment
Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient exact optimization of language model alignment. InICML, 2024
2024
-
[73]
Noise contrastive alignment of language models with explicit rewards.Advances in Neural Information Processing Systems, 37:117784–117812, 2024
Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards.Advances in Neural Information Processing Systems, 37:117784–117812, 2024
2024
-
[74]
Provably robust dpo: Aligning language models with noisy feedback
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. InInternational Conference on Machine Learning, pages 42258–42274. PMLR, 2024
2024
-
[75]
Self-play preference optimization for language model alignment.arXiv preprint arXiv:2405.00675, 2024
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment.arXiv preprint arXiv:2405.00675, 2024
2024 arXiv
-
[76]
Distributional preference alignment of llms via optimal transport
Igor Melnyk, Youssef Mroueh, Brian Belgodere, Mattia Rigotti, Apoorva Nitsure, Mikhail Yurochkin, Kristjan Greenewald, Jiri Navratil, and Jarret Ross. Distributional preference alignment of llms via optimal transport. NIPS, 2024
2024
-
[77]
Discovering preference optimization algorithms with and for large language models.NIPS, 2024
Chris Lu, Samuel Holt, Claudio Fanconi, Alex Chan, Jakob Foerster, Mihaela van der Schaar, and Robert Lange. Discovering preference optimization algorithms with and for large language models.NIPS, 2024
2024
-
[78]
Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment.ACL, 2025
Karel D’Oosterlinck, Winnie Xu, Chris Develder, Thomas Demeester, Amanpreet Singh, Christopher Potts, Douwe Kiela, and Shikib Mehri. Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment.ACL, 2025
2025
-
[79]
Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.arXiv preprint arXiv:2503.01743, 2025
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.arXiv preprint ar...
2025 arXiv
-
[80]
Online dpo: Online direct preference optimization with fast-slow chasing.arXiv preprint arXiv:2406.05534, 2024
Biqing Qi, Pengfei Li, Fangyuan Li, Junqi Gao, Kaiyan Zhang, and Bowen Zhou. Online dpo: Online direct preference optimization with fast-slow chasing.arXiv preprint arXiv:2406.05534, 2024
2024 arXiv
-
[81]
Learning dynamics of llm finetuning.ICLR, 2024
Yi Ren and Danica J Sutherland. Learning dynamics of llm finetuning.ICLR, 2024
2024
-
[82]
Visual test-time scaling for gui agent grounding.arXiv preprint arXiv:2505.00684, 2025
Tiange Luo, Lajanugen Logeswaran, Justin Johnson, and Honglak Lee. Visual test-time scaling for gui agent grounding.arXiv preprint arXiv:2505.00684, 2025
2025 arXiv
-
[83]
19" and
Jungjae Lee, Dongjae Lee, Chihun Choi, Youngmin Im, Jaeyoung Wi, Kihong Heo, Sangeun Oh, Sunjae Lee, and Insik Shin. Safeguarding mobile gui agent via logic-based action verification.arXiv preprint arXiv:2503.18492, 2025. 23 Phi-Ground Tech Report A Experiment Settings Due to ...
1935 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.