REVIEW 7 cited by
Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-UI 2 introduces three key innovations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT-4o with set-of-mark visual prompting. These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empirical experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks $\times$ 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platform benchmark demonstrate that Ferret-UI 2 significantly outperforms Ferret-UI, and also shows strong cross-platform transfer capabilities.
Forward citations
Cited by 7 Pith papers
-
Routing Is Least Learnable Where It Is Most Valuable: Bounds on Representation Routing for Web Agents
Per-task routing between text, image, and hybrid observations of a browser page does not currently beat one fixed choice, because the labels needed to learn routing exist only where the agent already succeeds; only a ...
-
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
OSWorld 2.0 is a benchmark of 108 realistic long-horizon computer-use tasks where current agents achieve only 20.6% binary completion, struggling with state inference and constraint tracking.
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
Page-granular Flex-Mem and switchable Flex-NPU cut TrustZone LLM TTFT by ~10× vs a CMA strawman and ~2.4× vs a pipelined secure-NPU strawman on RK3588.
-
UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
Training computer-use agents on a hybrid GUI-plus-programmatic-tool action space, with SFT and online RL, improves OSWorld success by ~23% relative over GUI-only baselines and transfers to Windows without Windows-spec...
-
SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsing
An end-to-end GUI-perception MLLM that uses a token router and continuous coordinate decoder to simultaneously ground and parse interface elements, achieving SOTA on several grounding benchmarks and a new parsing benchmark.
-
Universal Retrieval for Multimodal Trajectory Modeling
Introduces multimodal trajectory retrieval, constructs the GAE-Bench benchmark from five GUI agent datasets, and trains GAE-Retriever, which outperforms zero-shot retrieval baselines.
-
GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents
GUI-G1-3B shows that for GUI grounding, removing reasoning chains, regularizing box size in the reward, and reweighting GRPO by difficulty yields 90.3% on ScreenSpot and 37.1% on ScreenSpot-Pro.
Discussion (0). Continue with ORCID to comment.