Pith. sign in

REVIEW 4 major objections 6 minor 43 references

Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that replacing coordinate regression with candidate generation plus frozen cross-modal matching removes GUI grounding hallucinations, yielding more than 20% accuracy gains on ScreenSpot-Pro.

desk verdict The decoupling idea is real and worth testing, but the 'regression-free' claim is contradicted by the paper's own Eq. (5), and missing code/ablation data make the headline numbers unverifiable. read the letter →

arxiv 2608.09654 v1 pith:YDHHMONR submitted 2026-08-10 cs.AI

Yuke Li , Xuehan Hou This is my paper · ORCID
classification cs.AI
keywords GUIgroundingregression-freelocalizationlayout-awarematchingcoordinatehallucinationCLIPmultimodallargelanguagemodelsScreenSpot-ProMind2Web
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the usual way of making GUI agents point at screen elements—training a multimodal model to output coordinates—is what causes them to hallucinate, and that a different division of labor avoids the problem. A frozen MLLM first rewrites the user instruction into a visual description; a detector adapted to GUI layouts proposes candidate text and icon regions; a frozen CLIP model then selects the best matching region. No component learns coordinate regression, so the final coordinates come from choosing an existing candidate. The consequence, if right, is that precise GUI grounding can be achieved with only Text/Icon labels and a small amount of spatial annotation, and on ScreenSpot-Pro the paper reports a gain of more than 20 percentage points over end-to-end systems while Mind2Web element accuracy and step success rate rise by more than 15%.

What carries the argument

The load-bearing object is the Layout-Aware GUI Grounding Model, a two-step matcher. First, a GUI-domain-adapted DINO detector, fine-tuned on 200k screenshots labeled only with Text and Icon, outputs candidate bounding boxes plus a six-dimensional geometric vector encoding center, width, height, aspect ratio, and area. Then CLIP's frozen image encoder extracts a feature for each candidate crop, CLIP's text encoder encodes the MLLM's elaborated instruction, and cosine similarity selects the winner. For spatial queries, a learnable linear projection maps the geometric vector into the visual feature space and adds it to the candidate feature; this projection is trained with a contrastive loss on a small set of spatial query pairs and stays near zero when spatial cues are absent.

What would settle it

Run the DINO candidate stage alone on ScreenSpot-Pro and measure recall: for each ground-truth target, does at least one proposed candidate overlap it? If recall is far below the reported 41.3% average accuracy, the matching stage cannot be doing the claimed work.

Watch

Extended reading notes

Core claim

The paper's central claim is that coordinate regression is the root cause of GUI grounding hallucinations, and that replacing it with candidate generation plus frozen cross-modal matching removes that failure mode. The Layout-Aware GUI Grounding Model first lets a GUI-adapted DINO detector propose text and icon regions, then uses frozen CLIP to match each candidate's visual feature against a description produced by a frozen MLLM. A single learnable linear projection adds geometric bias for spatial queries without reintroducing regression. On ScreenSpot-Pro the paper reports a gain of more than 20 percentage points over the strongest end-to-end baseline, and on Mind2Web it reports element accuracy and step success rate gains above 15%.

Load-bearing premise

The one load-bearing premise, introduced where the detector is described, is that the Text/Icon candidate generator almost always proposes the target element, because the frozen matcher can only select among those proposals and cannot recover a missed target.

Editorial extensions

If this is right

  • GUI agents can operate from screenshots alone, with no DOM or view hierarchy, while avoiding coordinate regression.
  • Only Text/Icon binary labels plus a small set of spatial query pairs are needed to train the grounding module, cutting annotation cost.
  • Spatially qualified instructions like 'upper-right close button' can be handled by a single learnable linear projection that stays near-zero for plain queries.
  • Accuracy gains on ScreenSpot-Pro and Mind2Web suggest the decoupling transfers across desktop, mobile, and web interfaces.
  • Coordinate hallucinations are structurally prevented, because coordinates are selected from detected candidates rather than generated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the DINO candidate generator ever misses the target, the CLIP matcher cannot recover it, so the method's true ceiling is candidate recall rather than matching quality.
  • Replacing the frozen MLLM with a stronger or cheaper instruction parser should shift accuracy without retraining the grounding module, although the paper does not test this.
  • The same Text/Icon candidate-plus-frozen-matching design could transfer to other text-and-icon dominated visual domains, such as document or diagram grounding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a two-stage GUI grounding framework. A frozen MLLM first rewrites an abstract user instruction into a structured visual description rich in layout cues; a GUI-domain-adapted DINO detector then proposes candidate regions using only Text/Icon binary labels; finally, a frozen CLIP model selects the best-matching candidate, with a lightweight learned geometric fusion layer that injects spatial information. The authors report large improvements over end-to-end baselines on ScreenSpot, ScreenSpot-Pro, AITW, and Mind2Web, and claim that the pipeline is "regression-free" and hence avoids coordinate hallucinations. The manuscript argues that decoupling semantic understanding from layout-aware localization resolves the core difficulties of GUI grounding without expensive MLLM fine-tuning.

Significance. If the reported results are reproducible, the decoupled architecture is a practically attractive alternative to end-to-end MLLM grounding: it avoids full MLLM fine-tuning, uses only cheap Text/Icon labels for the candidate detector, and leverages frozen CLIP for open-vocabulary matching. The core idea is clear and easy to communicate, and the reported margins on ScreenSpot-Pro (41.3% vs. 18.9% for the best baseline) are striking. However, the central "regression-free" claim is currently not supported as stated, because the DINO candidate generator is trained with L1 and GIoU regression losses (Eq. 5), and the paper does not report candidate recall, ablations, error bars, or code/data. These gaps leave the attribution of the gains to the proposed mechanisms unresolved, though they are addressable with additional experiments and a more careful formulation of the claim.

major comments (4)
  1. [§3.2.1 and Eq. (5)] The paper's central "regression-free" claim is internally contradicted by the training of the candidate generator. Eq. (5) includes L_reg, an L1 bounding-box regression loss, and L_giou, a generalized IoU loss, applied to the DINO Transformer decoder, and Section 3.2.1 states that the detector is fine-tuned to output predicted boxes. Since the final output box is the box produced by this detector and CLIP only selects among these regressed candidates, the pipeline as a whole does learn coordinate regression. The abstract and Contribution (1) claim that the model avoids "learning any coordinate regression" and "requiring no coordinate regression parameters," which is not supported by the described method. At best the matching stage is regression-free, not the full grounding pipeline. The authors should either narrow the claim to the matching stage or provide a control experiment that removes or ablates the regression head and demonstrates that the reported gains do not depend on it.
  2. [§3.2.1 and §3.2.2] No candidate-recall metric is reported. The method's accuracy is upper-bounded by the recall of the DINO Text/Icon candidate generator: if the target element is not among the proposals, the frozen CLIP matcher cannot recover it. The reported gains over baselines could therefore come from improved DINO candidate quality rather than from "regression-free" matching. Please report candidate recall (e.g., Recall@k at the relevant IoU threshold) on ScreenSpot-Pro and Mind2Web, separately for Text and Icon candidates, and also report an oracle accuracy where the correct candidate is selected by ground-truth information. This would separate the contribution of candidate generation from the contribution of cross-modal matching.
  3. [§4 (Experiments)] The Introduction and Conclusion state that "Controlled ablation studies" confirm the contributions of layout-prior candidate generation and lightweight geometric fusion, but no ablation table or experiment appears in Section 4. Without ablations, the attribution of the reported gains to the two proposed mechanisms is unverified. Please add an ablation study that removes or disables each component (e.g., no geometric fusion, no layout-prior candidates, and a direct CLIP matching baseline on raw image patches) and reports the resulting accuracy on the same benchmarks.
  4. [§3.3 and §4] The experimental section lacks essential reproducibility details and statistical support. The number of spatial training pairs for Stage 2 is not given, the sampling and annotation procedure for those pairs is not described, and no code, data, or trained checkpoints are provided. Moreover, the benchmark tables report no error bars or significance tests, even though the claimed margins are large. At minimum, please specify the size and construction of the spatial-pair set, report standard errors or confidence intervals over multiple runs, and release the code and candidate-generator checkpoints.
minor comments (6)
  1. [Table 1] The rows for Fuyu and CogAgent are numerically identical (8.4, 6.6, 6.2, 2.9, 6.5, 3.4, 5.7); this is likely a copy-and-paste error and should be corrected.
  2. [References] References [8] and [9] are the same paper (arXiv:2410.05243); the duplicate entry should be removed and the citation numbering adjusted.
  3. [Abstract and §3.3] The abstract says the model is trained "with only Text/Icon binary labels," but Section 3.3 states that Stage 2 uses "manually annotated spatial queries" and Section 3.2.2 describes training the geometric fusion on spatial query-region pairs; please reconcile these statements.
  4. [§3.1] The Preliminary paragraph calls the MLLM "training-free" while the rest of the paper says it is "frozen-parameter"; the wording should be consistent, since the MLLM is not trained but is used with in-context prompting.
  5. [Figure 4] The caption says "after pre-training" and "fine-tuning with visual captions," which conflicts with the paper's claim that the approach avoids fine-tuning; the caption should use terminology consistent with the method description.
  6. [§3.2.2] Equation (2) defines cosine similarity with explicit norm normalization, but Eq. (6) uses sim(·,·) without defining it again; please state explicitly that sim is the same normalized cosine similarity from Eq. (2).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark results are externally evaluated and no load-bearing claim reduces to its own inputs.

full rationale

I found no load-bearing circular step in the derivation chain. The two learned components are trained on auxiliary signals (Text/Icon bounding-box labels and region-caption pairs) and then evaluated on held-out benchmarks (ScreenSpot, ScreenSpot-Pro, Mind2Web, AITW); the reported numbers are not obtained by fitting a parameter to the benchmark outputs, and no uniqueness theorem or equivalence from the authors' prior work is used to force a conclusion. The one author self-citation ([13]) is an unrelated MLLM-listing reference and carries no argumentative weight. The paper's 'regression-free' wording is internally inconsistent with Eq. (5), where the DINO proposer is trained with L1 and GIoU box-regression losses, and the paper never reports candidate recall separately; those are correctness and attribution-of-gain concerns, not circularity, because the empirical claims still rest on external evaluation rather than on the paper's own equations being rewound into its own assumptions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central result rests on three off-the-shelf components (MLLM description, DINO proposals, CLIP matching) plus a small trained projection. The key unverified premises are candidate recall and cross-modal description compatibility. No circular definition of the benchmark result is involved, and no new physical or formal entities are postulated.

free parameters (3)
  • Wg and bg (geometric fusion layer) = Not reported; Wg in R^(dc x 6), bg in R^(dc) trained on spatial query-region pairs
    The only learnable parameters in the matching stage are trained with InfoNCE (Eq. 6) to shift CLIP features by a spatial offset; their final values and validation impact are not reported.
  • DINO fine-tuning hyperparameters = lr 1e-4 (backbone 1e-5), weight decay 1e-4, 50 epochs, batch 8, random scaling augmentation
    These hand-set values are used to adapt the detector to Text/Icon labels; no sensitivity analysis is provided.
  • Loss weights and temperature = lambda_reg=5.0, lambda_giou=2.0, lambda_cls=2.0, tau=0.07
    Standard constants chosen by hand; no ablation on these values is shown.
assumptions (4)
  • domain assumption The DINO Text/Icon detector recalls every target element among its candidates.
    Section 3.2.1 builds candidate regions with this detector; if the target is absent, the CLIP matcher has no chance to select it. No recall metric is reported.
  • domain assumption MLLM-generated visual descriptions are semantically compatible with CLIP text embeddings.
    Sections 3.1 and 3.2.2: the elaborated query Q is embedded by frozen CLIP; if Q describes attributes or spatial relations in a way CLIP was not trained on, matching degrades.
  • domain assumption Automated parsing tools and Qwen3-VL captions provide correct Text/Icon regions and descriptions for training.
    Section 3.3 builds region-caption pairs with automated tools and Qwen3-VL; errors in either tool propagate directly into the contrastive training signal.
  • domain assumption The 200k sampled screenshots are representative of the test benchmarks (ScreenSpot, ScreenSpot-Pro, AITW, Mind2Web).
    Section 3.3 names no training datasets, so distribution shift from training screenshots to professional software test interfaces is an unvalidated premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching." pith.science (2026). https://pith.science/paper/YDHHMONR

@misc{pith2026260809654,
  author       = {Pith},
  title        = {Pith review of: Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YDHHMONR}},
  note         = {Machine review of arXiv:2608.09654}
}
read the original abstract

GUI agents are shifting from metadata-dependent large language models to purely visual multimodal large language models (MLLMs) that operate directly on screenshots. The core task, GUI grounding, requires translating abstract user instructions into precise element coordinates. This task faces a persistent dual obstacle: conventional grounding models lack the semantic richness to interpret abstract instructions, while end-to-end MLLMs suffer from coordinate hallucinations caused by deficient fine-grained perception. We propose a regression-free framework where a frozen MLLM performs instruction parsing and a dedicated grounding model handles precise localization without learning any coordinate regression. A frozen MLLM first elaborates the abstract instruction into a structured visual description rich in layout cues. These descriptions are then fed to a novel Layout-Aware GUI Grounding Model, which performs regression-free localization by matching against layout-prior candidates, inherently suppressing hallucinations and avoiding expensive fine-tuning. The grounding model is trained with only Text/Icon binary labels, requiring no coordinate regression parameters. On ScreenSpot-Pro, our method achieves over 20% improvement in grounding accuracy over end-to-end systems; on Mind2Web, it raises success rate and element selection rate by more than 15%. These results demonstrate that decoupling instruction understanding from layout-aware localization effectively resolves the core challenges of GUI interaction.

Figures

Figures reproduced from arXiv: 2608.09654 by the authors.

Figure 1
Figure 1. Framework of our proposed method. This approach integrates a Multimodal Large Language [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overall architecture of the proposed layout-aware grounding model for GUI grounding, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Training Dataset Construction Pipeline. We first sampled 200k images from common GUI [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: (a) Instruction elaboration: abstract user commands are transformed into structured visual [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 12 canonical work pages

  1. [9]

    Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243, 2024

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243, 2024

    arXiv 2024

  2. [29]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  3. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [2]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  5. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [4]

    Introducing our multimodal models, 2023

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa˘gnak Ta¸ sırlar. Introducing our multimodal models, 2023

    work page 2023

  7. [5]

    Seeclick: Harnessing gui grounding for advanced visual gui agents.arXiv preprint arXiv:2401.10935, 2024

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents.arXiv preprint arXiv:2401.10935, 2024

    arXiv 2024

  8. [6]

    Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    2023

Show all 43 references
  1. [7]

    Multimodal web navigation with instruction-finetuned foundation models

    Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetuned foundation models. InICLR, 2024

    2024

  2. [10]

    Learning to navigate the web

    Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur. Learning to navigate the web. InInternational Conference on Learning Representations

  3. [11]

    A real-world webagent with planning, long context understanding, and program synthesis

    Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. InICLR, 2024

    2024

  4. [12]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024

    2024

  5. [13]

    Anchor Frame Bridging for Coherent First-Last Frame Video Generation

    Xuehan Hou, Meng Fan, Pengchong Qiao, Zesen Cheng, Yian Zhao, Lei Zhu, Kaiwen Cheng, Chang Liu, and Jie Chen. Anchor Frame Bridging for Coherent First-Last Frame Video Generation. InInternational Conference on Learning Representations (ICLR), 2026

    2026

  6. [14]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  7. [15]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InInternational conference on machine learning, pages 4904–4916. ...

    2021

  8. [16]

    Language models can solve computer tasks.Advances in Neural Information Processing Systems, 36:39648–39677, 2023

    Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks.Advances in Neural Information Processing Systems, 36:39648–39677, 2023

    2023

  9. [17]

    Spotlight: Mobile ui understanding using vision-language models with a focus

    Gang Li and Yang Li. Spotlight: Mobile ui understanding using vision-language models with a focus. In The Eleventh International Conference on Learning Representations

  10. [18]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  11. [19]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023. 10

    2023

  12. [20]

    Screenspot-pro: Gui grounding for professional high-resolution computer use

    Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. InProceedings of the 33rd ACM International Conference on Multimedia, pages 8778–8786, 2025

    2025

  13. [21]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages ...

    2022

  14. [22]

    Mapping natural language instructions to mobile ui action sequences.arXiv preprint arXiv:2005.03776, 2020

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences.arXiv preprint arXiv:2005.03776, 2020

    2005 arXiv

  15. [23]

    Reinforcement learning on web interfaces using workflow-guided exploration.arXiv preprint arXiv:1802.08802, 2018

    Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration.arXiv preprint arXiv:1802.08802, 2018

    2018 arXiv

  16. [24]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  17. [25]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean conference on computer vision, pages 38–55. Springer, 2024

    2024

  18. [26]

    Autoglm: Autonomous foundation agents for guis.arXiv preprint arXiv:2411.00820, 2024

    Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis.arXiv preprint arXiv:2411.00820, 2024

    2024 arXiv

  19. [27]

    Omniparser for pure vision based gui agent.arXiv preprint arXiv:2408.00203, 2024

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent.arXiv preprint arXiv:2408.00203, 2024

    2024 arXiv

  20. [30]

    Androidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36: 59708–59728, 2023

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Androidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36: 59708–59728, 2023

    2023

  21. [31]

    Grounding dino 1.5: Advance the" edge" of open-set object detection

    Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al. Grounding dino 1.5: Advance the" edge" of open-set object detection. arXiv preprint arXiv:2405.10300, 2024

    2024 arXiv

  22. [32]

    From pixels to ui actions: Learning to follow instructions via graphical user interfaces.Advances in Neural Information Processing Systems, 36: 34354–34370, 2023

    Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina N Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces.Advances in Neural Information Processing Sy...

    2023

  23. [33]

    World of bits: An open-domain platform for web-based agents

    Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. InInternational Conference on Machine Learning, pages 3135–3144. PMLR, 2017

    2017

  24. [34]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  25. [35]

    Enabling conversational interaction with mobile ui using large language models

    Bryan Wang, Gang Li, and Yang Li. Enabling conversational interaction with mobile ui using large language models. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–17, 2023

    2023

  26. [36]

    Autodroid: Llm-powered task automation in android

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. InProceedings of the 30th Annual International Conference on Mobile Computing and Networking, page...

    2024

  27. [37]

    Mobilevlm: A vision-language model for better intra-and inter-ui understanding

    Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, and Shuo Shang. Mobilevlm: A vision-language model for better intra-and inter-ui understanding. InEMNLP (Findings), 2024

    2024

  28. [38]

    Os-atlas: A foundation action model for generalist gui agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024

    arXiv

  29. [39]

    Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256, 2024

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256, 2024

    2024 arXiv

  30. [40]

    Dino: Detr with improved denoising anchor boxes for end-to-end object detection

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. InThe Eleventh International Conference on Learning Representations

  31. [41]

    You only look at screens: Multimodal chain-of-action agents.arXiv preprint arXiv:2309.11436, 2023

    Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents.arXiv preprint arXiv:2309.11436, 2023

    2023 arXiv

  32. [42]

    Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    2024 arXiv

  33. [43]

    Synapse: Trajectory-as-exemplar prompting with memory for computer control

    Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control. InNeurIPS 2023 F oundation Models for Decision Making Workshop

    2023

  34. [44]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023

    2023 arXiv

  35. [45]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 12

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.