REVIEW 9 cited by
DroidBot-GPT: GPT-powered UI Automation for Android
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This paper introduces DroidBot-GPT, a tool that utilizes GPT-like large language models (LLMs) to automate the interactions with Android mobile applications. Given a natural language description of a desired task, DroidBot-GPT can automatically generate and execute actions that navigate the app to complete the task. It works by translating the app GUI state information and the available actions on the smartphone screen to natural language prompts and asking the LLM to make a choice of actions. Since the LLM is typically trained on a large amount of data including the how-to manuals of diverse software applications, it has the ability to make reasonable choices of actions based on the provided information. We evaluate DroidBot-GPT with a self-created dataset that contains 33 tasks collected from 17 Android applications spanning 10 categories. It can successfully complete 39.39% of the tasks, and the average partial completion progress is about 66.76%. Given the fact that our method is fully unsupervised (no modification required from both the app and the LLM), we believe there is great potential to enhance automation performance with better app development paradigms and/or custom model training.
Forward citations
Cited by 9 Pith papers
-
A Comprehensive Study of Implementation Bugs in Multi-modal Agents
First systematic taxonomy of 158 multi-modal agent bugs plus a runtime analyzer that recovers most open issues and surfaces 31 new ones.
-
DragonCrawl: A Generative, Intent-Based Framework for Scalable Mobile End-to-End Testing
DragonCrawl uses GPT-4o-based intent reasoning, multimodal end-state detection, and backend tool calling to run 1,013 mobile regression tests in CI/CD, achieving over 91% pass rates and cutting test maintenance effort.
-
LogiDroid: Individual Functional Test Generation via Business Logic Extraction and Adaptation
LogiDroid generates functional Android test cases with verification assertions by retrieving similar test cases, fusing their business logic, and adapting it to the target app's real-time GUI state.
-
FuncDroid: Towards Inter-Functional Flows for Comprehensive Mobile App GUI Testing
An LLM-guided Android GUI tester that explicitly models inter-functional flows finds more bugs than coverage- or single-functionality-oriented baselines.
-
Measuring the Security of Mobile LLM Agents under Adversarial Prompts from Untrusted Third-Party Channels
Indirect prompt injection through ads, webviews, and notifications reliably diverts mobile LLM agents into leaking data and installing malware across eight evaluated agents.
-
SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents
A history-aware guard model with an LLM judge is reported to cut jailbreak success on mobile agent tasks from 86.1% to 8.4% while keeping task completion unchanged at 77.8%.
-
Cybernaut: Towards Reliable Web Automation
A demonstration-to-SOP framework plus robust element identification and a trace similarity metric improves enterprise web automation success rates on an internal benchmark, with a fine-tuned consistency classifier rea...
-
VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception
The paper promises VLM-3D but the body text is entirely the MVISU-Bench mobile-agent benchmark paper, so the stated result is unsupported.
-
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.
Discussion (0). Sign in to comment.