Pith. sign in

REVIEW 12 cited by

Mapping Natural Language Instructions to Mobile UI Action Sequences

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2005.03776 v2 pith:BJJGV4P6 submitted 2020-05-07 cs.CL cs.LG

Yang Li , Jiacong He , Xin Zhou , Yuan Zhang , Jason Baldridge This is my paper · ORCID
classification cs.CLcs.LG
keywords actioninstructionslanguagemobileactionsnaturalcreatedescriptions
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present a new problem: grounding natural language instructions to mobile user interface actions, and create three new datasets for it. For full task evaluation, we create PIXELHELP, a corpus that pairs English instructions with actions performed by people on a mobile UI emulator. To scale training, we decouple the language and action data by (a) annotating action phrase spans in HowTo instructions and (b) synthesizing grounded descriptions of actions for mobile user interfaces. We use a Transformer to extract action phrase tuples from long-range natural language instructions. A grounding Transformer then contextually represents UI objects using both their content and screen position and connects them to object descriptions. Given a starting screen and instruction, our model achieves 70.59% accuracy on predicting complete ground-truth action sequences in PIXELHELP.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Grounding Computer Use Agents on Human Demonstrations

    cs.LG 2025-11 conditional novelty 7.0 of 10

    GroundCUA, a 3.56M-element human-annotated desktop grounding dataset, and GroundNext models achieve strong UI grounding with less than one-tenth the SFT data of prior work.

  2. LogiDroid: Individual Functional Test Generation via Business Logic Extraction and Adaptation

    cs.SE 2026-02 conditional novelty 6.0 of 10

    LogiDroid generates functional Android test cases with verification assertions by retrieving similar test cases, fusing their business logic, and adapting it to the target app's real-time GUI state.

  3. TextOnly: A Unified Function Portal for Text-Related Functions on Smartphones

    cs.HC 2025-08 conditional novelty 6.0 of 10

    A unified text portal predicts users' intended text-related smartphone functions from raw input using a hybrid LLM+BERT model, reaching 71.35% Hit@1 in a real-world study.

  4. Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Explicitly storing short-term and long-term text memories, instead of raw screenshots, improves GUI agent accuracy on cross-app tasks, and a new annotated dataset helps 7B models approach 72B-level memory use.

  5. What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A self-generating graph benchmark produces 36k GUI agent tasks with controllable complexity and ten capability scores, and fine-tuning on its trajectories gives small gains on AndroidControl and OmniAct.

  6. GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents

    cs.CL 2025-06 conditional novelty 6.0 of 10

    An attention-based action head with multi-patch supervision outperforms coordinate-generation baselines on GUI grounding, and a verifier further improves accuracy.

  7. ZeroGUI: Automating Online GUI Learning at Zero Human Cost

    cs.AI 2025-05 conditional novelty 6.0 of 10

    ZeroGUI uses VLM-generated tasks and VLM-estimated rewards with two-stage GRPO to improve GUI agent success rates on OSWorld and AndroidLab without human annotations.

  8. TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments

    cs.HC 2025-05 conditional novelty 6.0 of 10

    TransBench is a new benchmark of 1,459 screenshots and 22,000 grounding instructions for measuring how well GUI agents transfer across app versions, platforms, and applications.

  9. Cybernaut: Towards Reliable Web Automation

    cs.SE 2025-08 reject novelty 4.0 of 10

    A demonstration-to-SOP framework plus robust element identification and a trace similarity metric improves enterprise web automation success rates on an internal benchmark, with a fine-tuned consistency classifier rea...

  10. VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception

    cs.CV 2025-08 reject novelty 4.0 of 10

    The paper promises VLM-3D but the body text is entirely the MVISU-Bench mobile-agent benchmark paper, so the stated result is unsupported.

  11. Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.

  12. Large Language Models for Planning: A Comprehensive and Systematic Survey

    cs.AI 2025-05 conditional novelty 3.0 of 10

    A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.

Pith tools