Pith. sign in

REVIEW 9 cited by

Android in the Wild: A Large-Scale Dataset for Android Device Control

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.10088 v2 pith:JCMQVKHD submitted 2023-07-19 cs.LG cs.CLcs.HC

Christopher Rawles , Alice Li , Daniel Rodriguez , Oriana Riva , Timothy Lillicrap This is my paper · ORCID
classification cs.LGcs.CLcs.HC
keywords datasetandroiddeviceactionsdevice-controlinstructionslanguagewild
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

There is a growing interest in device-control systems that can interpret human natural language instructions and execute them on a digital device by directly controlling its user interface. We present a dataset for device-control research, Android in the Wild (AITW), which is orders of magnitude larger than current datasets. The dataset contains human demonstrations of device interactions, including the screens and actions, and corresponding natural language instructions. It consists of 715k episodes spanning 30k unique instructions, four versions of Android (v10-13),and eight device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It contains multi-step tasks that require semantic understanding of language and visual context. This dataset poses a new challenge: actions available through the user interface must be inferred from their visual appearance. And, instead of simple UI element-based actions, the action space consists of precise gestures (e.g., horizontal scrolls to operate carousel widgets). We organize our dataset to encourage robustness analysis of device-control systems, i.e., how well a system performs in the presence of new task descriptions, new applications, or new platform versions. We develop two agents and report performance across the dataset. The dataset is available at https://github.com/google-research/google-research/tree/master/android_in_the_wild.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    OSWorld 2.0 is a benchmark of 108 realistic long-horizon computer-use tasks where current agents achieve only 20.6% binary completion, struggling with state inference and constraint tracking.

  2. PG-Agent: An Agent Powered by Page Graph

    cs.AI 2025-08 conditional novelty 6.0 of 10

    An MLLM GUI agent that stores past episodes as a page graph and retrieves action guidelines from it improves step success on three benchmarks.

  3. Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Screen2AX generates hierarchical macOS accessibility metadata from a screenshot and reports improved GPT-4 UI task success compared with native accessibility and OmniParser V2.

  4. Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Explicitly storing short-term and long-term text memories, instead of raw screenshots, improves GUI agent accuracy on cross-app tasks, and a new annotated dataset helps 7B models approach 72B-level memory use.

  5. GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies

    cs.AI 2025-06 conditional novelty 6.0 of 10

    A new benchmark with 5,318 GUI tasks, including 200 abnormal ones, shows that state-of-the-art GUI agents degrade sharply when real-world anomalies appear.

  6. TextAtari: 100K Frames Game Playing with Language Agents

    cs.CL 2025-06 conditional novelty 5.0 of 10

    TextAtari is a text-based Atari benchmark for language agents; 7-8B LLMs stay below 10% of human scores in over 90% of tested conditions, and knowledge injection helps more than chain-of-thought.

  7. Digi-Q: Learning Q-Value Functions for Training Device-Control Agents

    cs.LG 2025-02 conditional novelty 5.0 of 10

    An offline RL method learns a Q-function from frozen VLM features and extracts a device-control policy by imitating the best of several actions ranked by that Q-function.

  8. SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control

    cs.AI 2025-08 conditional novelty 4.0 of 10

    A multi-agent RL workflow that interleaves single-agent updates, applied to mobile GUI control, achieves SOTA zero-shot performance and a +14.8 MATH500 gain.

  9. GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Modeling GUI elements as Gaussian distributions instead of binary targets yields 92.0% (ScreenSpot), 93.3% (ScreenSpot-v2), and 47.5% (ScreenSpot-Pro) for a 7B model, outperforming UI-TARS-72B by a relative 24.7% on t...

Pith tools