REVIEW 22 cited by
ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent advancements in Multi-modal Large Language Models (MLLMs) have led to significant progress in developing GUI agents for general tasks such as web browsing and mobile phone use. However, their application in professional domains remains under-explored. These specialized workflows introduce unique challenges for GUI perception models, including high-resolution displays, smaller target sizes, and complex environments. In this paper, we introduce ScreenSpot-Pro, a new benchmark designed to rigorously evaluate the grounding capabilities of MLLMs in high-resolution professional settings. The benchmark comprises authentic high-resolution images from a variety of professional domains with expert annotations. It spans 23 applications across five industries and three operating systems. Existing GUI grounding models perform poorly on this dataset, with the best model achieving only 18.9%. Our experiments reveal that strategically reducing the search area enhances accuracy. Based on this insight, we propose ScreenSeekeR, a visual search method that utilizes the GUI knowledge of a strong planner to guide a cascaded search, achieving state-of-the-art performance with 48.1% without any additional training. We hope that our benchmark and findings will advance the development of GUI agents for professional applications. Code, data and leaderboard can be found at https://gui-agent.github.io/grounding-leaderboard.
Forward citations
Cited by 22 Pith papers
-
Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure
Across four open-weight models and three OpenAI models, textual state beliefs of GUI agents follow a conflicting structured value on 30–75% of probes where the same model reads the pixels correctly, so agent state bel...
-
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
OSWorld 2.0 is a benchmark of 108 realistic long-horizon computer-use tasks where current agents achieve only 20.6% binary completion, struggling with state inference and constraint tracking.
-
GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
GUI-Perturbed shows that GUI grounding models suffer systematic accuracy collapse under relational instructions and visual changes such as 70% zoom, with even augmented fine-tuning worsening results.
-
GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding
Supervising an MLLM's intrinsic self-attention with patch-level GUI labels, aggregated via a learnable anchor token and hidden-state-selected query tokens, reaches state-of-the-art 3B-scale GUI grounding accuracy with...
-
ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search
ReGUIDE reaches state-of-the-art GUI grounding accuracy using 0.2% of the usual training data by adding self-generated reasoning, spatial-consistency training, and test-time KDE coordinate search.
-
GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
GUI-Lens improves GUI grounding by having a VLM iteratively select crops to zoom into, guided by OCR and UI detector references, and verifies its proposed clicks before outputting a coordinate.
-
Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?
A new 2,013-sample Linux desktop benchmark shows frontier VLMs are far from solving temporal ordering and single-action reconstruction, with best ordering exact-match near 65%.
-
Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning
CrossAgent learns step-level action-interface selection via a three-stage SFT + single-turn GRPO + multi-turn GRPO pipeline, reporting 54.6% mean success on 800+ Minecraft tasks after RL on only 30 tasks.
-
Learning Active Perception via Self-Evolving Preference Optimization for GUI Grounding
LASER uses self-evolving preference optimization to teach GUI models to crop and zoom before clicking, reaching 55.7% on ScreenSpot-Pro with a 7B model.
-
MobiAgent: A Systematic Framework for Customizable Mobile Agents
A full-stack mobile agent framework reports state-of-the-art task completion on its own DAG-based benchmark and 2-3x speedups from replaying recorded trajectories.
-
SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
A self-evolving computer-use agent trained with full-trajectory state judging and curriculum task generation goes from 11.3% to 34.5% average success on five OSWorld apps, and a specialist-to-generalist variant beats ...
-
Phi-Ground Tech Report: Advancing Perception in GUI Grounding
Phi-Ground models achieve state-of-the-art click accuracy on five GUI grounding benchmarks for models under 10B parameters using a 40M-sample training recipe with text-first inputs, random-resize augmentation, uniform...
-
MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
MMBench-GUI provides a multi-platform, four-level benchmark and an efficiency-aware metric, and its experiments indicate visual grounding is the main bottleneck for current GUI agents.
-
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
An attention-based action head with multi-patch supervision outperforms coordinate-generation baselines on GUI grounding, and a verifier further improves accuracy.
-
PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editors
Multi-turn visual feedback refinement outperforms single-shot coordinate prediction for pixel-precise GUI grounding in complex coding environments.
-
Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning
HyperClick trains GUI grounding models with GRPO to output clicks plus confidence scores, jointly rewarding correct clicks and Brier-calibrated confidence, and reports SOTA accuracy on six of seven benchmarks with bet...
-
MiMo-VL Technical Report
MiMo-VL-7B-RL, a 7B open-source vision-language model, reports state-of-the-art results on 35 of 40 benchmarks and a 59.4 OlympiadBench score, with the report crediting long-CoT pretraining data and mixed on-policy RL.
-
Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights
The paper claims a state-of-the-art, low-cost open-weight web agent, but its WebVoyager result is inflated by training on the evaluation benchmark.
-
MMTABREAL: Real-World Benchmark for Multimodal Table Understanding
The paper releases a 500-table, 4,021-question benchmark of real-world multimodal tables and shows that leading vision-language models drop 20-40% in accuracy relative to earlier synthetic benchmarks.
-
GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents
GUI-G1-3B shows that for GUI grounding, removing reasoning chains, regularizing box size in the reward, and reweighting GRPO by difficulty yields 90.3% on ScreenSpot and 37.1% on ScreenSpot-Pro.
-
A Lightweight Incentive-Based Privacy-Preserving Smart Metering Protocol for Value-Added Services
A layered protocol of local differential privacy, blind signatures, pseudonyms, temporal aggregation, and anonymous routing is claimed to keep smart-meter readings private while still enabling reward token redemption.
-
ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding
A 3B vision-language model fine-tuned with LoRA on a 24K example dataset with a two-stage schedule reaches the best reported accuracy among sub-4B models on GUI grounding benchmarks.
Discussion (0). Sign in to comment.