Pith. sign in

REVIEW 13 cited by

AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.17490 v2 pith:2QFRJVZ2 submitted 2024-07-03 cs.HC cs.AIcs.MM

Yuxiang Chai , Siyuan Huang , Yazhe Niu , Han Xiao , Liang Liu , Dingyu Zhang , Shuai Ren , Hongsheng Li This is my paper · ORCID
classification cs.HCcs.AIcs.MM
keywords amexmobileagentsdatasetandroiddatasetselementexisting
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

AI agents have drawn increasing attention mostly on their ability to perceive environments, understand tasks, and autonomously achieve goals. To advance research on AI agents in mobile scenarios, we introduce the Android Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for generalist mobile GUI-control agents which are capable of completing tasks by directly interacting with the graphical user interface (GUI) on mobile devices. AMEX comprises over 104K high-resolution screenshots from popular mobile applications, which are annotated at multiple levels. Unlike existing GUI-related datasets, e.g., Rico, AitW, etc., AMEX includes three levels of annotations: GUI interactive element grounding, GUI screen and element functionality descriptions, and complex natural language instructions with stepwise GUI-action chains. We develop this dataset from a more instructive and detailed perspective, complementing the general settings of existing datasets. Additionally, we finetune a baseline model SPHINX Agent and illustrate the effectiveness of AMEX.The project is available at https://yxchai.com/AMEX/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents

    cs.AI 2025-12 conditional novelty 8.0 of 10

    MobiBench reaches near-human offline evaluation fidelity for mobile GUI agents by accepting any valid action at each step, and enables modular attribution of performance to agent components.

  2. Grounding Computer Use Agents on Human Demonstrations

    cs.LG 2025-11 conditional novelty 7.0 of 10

    GroundCUA, a 3.56M-element human-annotated desktop grounding dataset, and GroundNext models achieve strong UI grounding with less than one-tenth the SFT data of prior work.

  3. GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

    cs.CV 2025-11 conditional novelty 7.0 of 10

    Supervising an MLLM's intrinsic self-attention with patch-level GUI labels, aggregated via a learnable anchor token and hidden-state-selected query tokens, reaches state-of-the-art 3B-scale GUI grounding accuracy with...

  4. GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents

    cs.CL 2025-06 conditional novelty 6.0 of 10

    An attention-based action head with multi-patch supervision outperforms coordinate-generation baselines on GUI grounding, and a verifier further improves accuracy.

  5. ZeroGUI: Automating Online GUI Learning at Zero Human Cost

    cs.AI 2025-05 conditional novelty 6.0 of 10

    ZeroGUI uses VLM-generated tasks and VLM-estimated rewards with two-stage GRPO to improve GUI agent success rates on OSWorld and AndroidLab without human annotations.

  6. UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents

    cs.CL 2025-05 conditional novelty 6.0 of 10

    UI-Genie uses a specialized reward model and iterative self-improvement to generate synthetic training trajectories, achieving state-of-the-art results for mobile GUI agents on AndroidControl, AndroidLab, and Android Arena.

  7. StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A co-designed 0.9B UI vision-language model tops small-model benchmarks on screen Q&A and Chinese OCR and runs on a Snapdragon 8 Gen5 at 98 tokens/second.

  8. GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior

    cs.AI 2025-06 conditional novelty 5.0 of 10

    GUI-Reflection trains an 8B multimodal GUI agent to recognize mistakes, undo incorrect actions, and retry, improving AndroidWorld success rate from 14.58% (filtered BC baseline) to 34.72% with reflection data and onli...

  9. FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents

    cs.CL 2025-06 reject novelty 5.0 of 10

    A new interactive benchmark for form-filling agents reports that current multimodal models fail at the task, but the evaluation protocol is internally inconsistent and may not measure form-filling ability fairly.

  10. GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents

    cs.CL 2025-05 conditional novelty 5.0 of 10

    GUI-G1-3B shows that for GUI grounding, removing reasoning chains, regularizing box size in the reward, and reweighting GRPO by difficulty yields 90.3% on ScreenSpot and 37.1% on ScreenSpot-Pro.

  11. SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control

    cs.AI 2025-08 conditional novelty 4.0 of 10

    A multi-agent RL workflow that interleaves single-agent updates, applied to mobile GUI control, achieves SOTA zero-shot performance and a +14.8 MATH500 gain.

  12. ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A 3B vision-language model fine-tuned with LoRA on a 24K example dataset with a two-stage schedule reaches the best reported accuracy among sub-4B models on GUI grounding benchmarks.

  13. Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.

Pith tools