REVIEW 13 cited by
Android in the Zoo: Chain-of-Action-Thought for GUI Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language model (LLM) leads to a surge of autonomous GUI agents for smartphone, which completes a task triggered by natural language through predicting a sequence of actions of API. Even though the task highly relies on past actions and visual observations, existing studies typically consider little semantic information carried out by intermediate screenshots and screen operations. To address this, this work presents Chain-of-Action-Thought (dubbed CoAT), which takes the description of the previous actions, the current screen, and more importantly the action thinking of what actions should be performed and the outcomes led by the chosen action. We demonstrate that, in a zero-shot setting upon three off-the-shelf LMMs, CoAT significantly improves the action prediction compared to previous proposed context modeling. To further facilitate the research in this line, we construct a dataset Android-In-The-Zoo (AitZ), which contains 18,643 screen-action pairs together with chain-of-action-thought annotations. Experiments show that fine-tuning a 1B model (i.e. AUTO-UI-base) on our AitZ dataset achieves on-par performance with CogAgent-Chat-18B.
Forward citations
Cited by 13 Pith papers
-
MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents
MobiBench reaches near-human offline evaluation fidelity for mobile GUI agents by accepting any valid action at each step, and enables modular attribution of performance to agent components.
-
CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning
A decoupled planner-executor GUI agent, trained by per-app reinforcement learning followed by specialist-to-generalist distillation, lifts ScienceBoard success from about 7.6% to 21.0% average and 40% pass@8.
-
SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
A self-evolving computer-use agent trained with full-trajectory state judging and curriculum task generation goes from 11.3% to 34.5% average success on five OSWorld apps, and a specialist-to-generalist variant beats ...
-
Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation
Explicitly storing short-term and long-term text memories, instead of raw screenshots, improves GUI agent accuracy on cross-app tasks, and a new annotated dataset helps 7B models approach 72B-level memory use.
-
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
An attention-based action head with multi-patch supervision outperforms coordinate-generation baselines on GUI grounding, and a verifier further improves accuracy.
-
ZeroGUI: Automating Online GUI Learning at Zero Human Cost
ZeroGUI uses VLM-generated tasks and VLM-estimated rewards with two-stage GRPO to improve GUI agent success rates on OSWorld and AndroidLab without human annotations.
-
MedBrowseComp: Benchmarking Medical Deep Research and Computer Use
A new benchmark of more than 1,000 multi-hop medical browsing questions shows that even the best deep-research and computer-use AI agents answer fewer than half correctly.
-
Qwen-Audio-VAE Technical Report
A 12.5 Hz continuous audio VAE reconstructs speech, music, and sound well while encoding 64×30s clips in 541 ms after latency-aware encoder pruning.
-
Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System
AI phone agents succeed on single-step commands but fail on multi-step compositions; a new benchmark measures this gap and a scheduler that decomposes tasks recovers most of it.
-
MMTABREAL: Real-World Benchmark for Multimodal Table Understanding
The paper releases a 500-table, 4,021-question benchmark of real-world multimodal tables and shows that leading vision-language models drop 20-40% in accuracy relative to earlier synthetic benchmarks.
-
Building a Stable Planner: An Extended Finite State Machine Based Planning Module for Mobile GUI Agent
A hand-authored EFSM planning module boosts Qwen2.5-VL-72B on AndroidWorld from 35.0% to 63.8% task success.
-
SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control
A multi-agent RL workflow that interleaves single-agent updates, applied to mobile GUI control, achieves SOTA zero-shot performance and a +14.8 MATH500 gain.
-
VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception
The paper promises VLM-3D but the body text is entirely the MVISU-Bench mobile-agent benchmark paper, so the stated result is unsupported.
Discussion (0). Sign in to comment.