Pith. sign in

REVIEW 4 cited by

WinClick: GUI Grounding with Multimodal Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.04730 v1 pith:MX74S6UK submitted 2025-01-27 cs.CL cs.HC

Zheng Hui , Yinheng Li , Dan zhao , Tianyi Chen , Colby Banbury , Kazuhito Koishida This is my paper
classification cs.CLcs.HC
keywords groundingwinclickwindowswinspotagentschallengedatadesktop
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Graphical User Interface (GUI) tasks are vital for automating workflows such as software testing, user interface navigation. For users, the GUI is the most intuitive platform for interacting with a computer. Previous work identified a key challenge in developing visual GUI agents: GUI grounding - the ability to accurately locate screen elements based on instructions. However, most existing GUI agents rely on structured data formats like DOM or HTML files in training or inferencing, which are inaccessible across all applications, particular in a general desktop environments such as Windows OS. To address this, we introduce WinClick, a novel visual GUI agent developed in Windows platform. WinClick leverages screenshots to detect actionable regions. To overcome the challenge of GUI grounding, we enhance WinClick with GUI grounding pre-training and propose an LLM-based method for aligning GUI grounding data. Additionally, we introduce WinSpot, the first comprehensive benchmark for GUI grounding on Windows. Our experiments demonstrate that WinClick, combined with GUI grounding pre-training, significantly outperforms existing baselines, offering a scalable solution for GUI automation in desktop environments. WinSpot is publicly available at https://github.com/zackhuiiiii/WinSpot.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TargetFinder: Detecting Widgets from Pixels on Desktop Interfaces

    cs.HC 2026-07 conditional novelty 6.0 of 10

    A fine-tuned YOLO pipeline on a new 520-screenshot, 38,000-widget dataset detects desktop GUI widgets from pixels and drives system-wide Bubble Cursor and Semantic Pointing.

  2. Instruction Agent: Enhancing Agent with Expert Demonstration

    cs.AI 2025-09 conditional novelty 6.0 of 10

    With one human demonstration, the Instruction Agent extracts instructions and uses verifier/backtracker modules to execute them, reaching 60% success on 20 previously unsolved OSWorld tasks.

  3. Phi-Ground Tech Report: Advancing Perception in GUI Grounding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Phi-Ground models achieve state-of-the-art click accuracy on five GUI grounding benchmarks for models under 10B parameters using a 40M-sample training recipe with text-first inputs, random-resize augmentation, uniform...

  4. FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow

    cs.CL 2025-05 conditional novelty 6.0 of 10

    FullFront adds a three-task benchmark for webpage design, perception, and code generation, and finds top MLLMs still fail at fine-grained layout and interaction implementation.

Pith tools