Pith. sign in

REVIEW 10 cited by

GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.07562 v1 pith:SWB3SLGE submitted 2023-11-13 cs.CV cs.AI

An Yan , Zhengyuan Yang , Wanrong Zhu , Kevin Lin , Linjie Li , Jianfeng Wang , Jianwei Yang , Yiwu Zhong
show 4 more authors
Julian McAuley Jianfeng Gao Zicheng Liu Lijuan Wang
This is my paper · ORCID
classification cs.CVcs.AI
keywords navigationmm-navigatorscreenactionsmartphonezero-shotaccuracyactions
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical user interface (GUI) navigation task. MM-Navigator can interact with a smartphone screen as human users, and determine subsequent actions to fulfill given instructions. Our findings demonstrate that large multimodal models (LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its advanced screen interpretation, action reasoning, and precise action localization capabilities. We first benchmark MM-Navigator on our collected iOS screen dataset. According to human assessments, the system exhibited a 91\% accuracy rate in generating reasonable action descriptions and a 75\% accuracy rate in executing the correct actions for single-step instructions on iOS. Additionally, we evaluate the model on a subset of an Android screen navigation dataset, where the model outperforms previous GUI navigators in a zero-shot fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for future research into the GUI navigation task. The project page is at https://github.com/zzxslp/MM-Navigator.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Comprehensive Study of Implementation Bugs in Multi-modal Agents

    cs.SE 2026-07 accept novelty 7.0 of 10

    First systematic taxonomy of 158 multi-modal agent bugs plus a runtime analyzer that recovers most open issues and surfaces 31 new ones.

  2. SEE: Structure-aware Exploring \& Exploiting for Long-horizon GUI Agent Trajectory Synthesis

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A graph-based pipeline automatically synthesizes long, multi-step mobile-app interaction trajectories, and fine-tuning GUI agents on them improves task success and cross-benchmark transfer.

  3. UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action

    cs.CV 2025-10 conditional novelty 6.0 of 10

    Training computer-use agents on a hybrid GUI-plus-programmatic-tool action space, with SFT and online RL, improves OSWorld success by ~23% relative over GUI-only baselines and transfers to Windows without Windows-spec...

  4. CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A decoupled planner-executor GUI agent, trained by per-app reinforcement learning followed by specialist-to-generalist distillation, lifts ScienceBoard success from about 7.6% to 21.0% average and 40% pass@8.

  5. SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience

    cs.AI 2025-08 conditional novelty 6.0 of 10

    A self-evolving computer-use agent trained with full-trajectory state judging and curriculum task generation goes from 11.3% to 34.5% average success on five OSWorld apps, and a specialist-to-generalist variant beats ...

  6. Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents

    cs.LG 2025-09 conditional novelty 5.0 of 10

    EMPG re-weights policy-gradient updates by step-level token entropy, amplifying confident correct actions and muting uncertain ones, and adds a future-clarity bonus.

  7. Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System

    cs.CL 2025-06 conditional novelty 5.0 of 10

    AI phone agents succeed on single-step commands but fail on multi-step compositions; a new benchmark measures this gap and a scheduler that decomposes tasks recovers most of it.

  8. Digi-Q: Learning Q-Value Functions for Training Device-Control Agents

    cs.LG 2025-02 conditional novelty 5.0 of 10

    An offline RL method learns a Q-function from frozen VLM features and extracts a device-control policy by imitating the best of several actions ranked by that Q-function.

  9. DPO Learning with LLMs-Judge Signal for Computer Use Agents

    cs.AI 2025-06 conditional novelty 4.0 of 10

    An LLM-as-Judge pipeline that scores synthetic GUI interaction trajectories and fine-tunes a 2B model with DPO yields a local computer-use agent that beats its base model on 15-step OSWorld tasks.

  10. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools