REVIEW 10 cited by
ScreenAgent: A Vision Language Model-driven Computer Control Agent
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Existing Large Language Models (LLM) can invoke a variety of tools and APIs to complete complex tasks. The computer, as the most powerful and universal tool, could potentially be controlled directly by a trained LLM agent. Powered by the computer, we can hopefully build a more generalized agent to assist humans in various daily digital works. In this paper, we construct an environment for a Vision Language Model (VLM) agent to interact with a real computer screen. Within this environment, the agent can observe screenshots and manipulate the Graphics User Interface (GUI) by outputting mouse and keyboard actions. We also design an automated control pipeline that includes planning, acting, and reflecting phases, guiding the agent to continuously interact with the environment and complete multi-step tasks. Additionally, we construct the ScreenAgent Dataset, which collects screenshots and action sequences when completing a variety of daily computer tasks. Finally, we trained a model, ScreenAgent, which achieved computer control capabilities comparable to GPT-4V and demonstrated more precise UI positioning capabilities. Our attempts could inspire further research on building a generalist LLM agent. The code is available at \url{https://github.com/niuzaisheng/ScreenAgent}.
Forward citations
Cited by 10 Pith papers
-
STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training
Normalized entropy locates trajectory-neglect steps; STAPO selectively optimizes them with a trajectory-aware reward and trajectory-independent penalty, beating group-based RL baselines on agent tasks.
-
Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation
Screen2AX generates hierarchical macOS accessibility metadata from a screenshot and reports improved GPT-4 UI task success compared with native accessibility and OmniParser V2.
-
Exploring the Potential of Metacognitive Support Agents for Human-AI Co-Creation
Metacognitive support agents, simulated by human wizards, improved the feasibility of AI-generated mechanical designs in a 20-participant formative study.
-
Tactile: Giving Computer-Using Agents Hands and Feet
Adding Tactile, an MCP tool layer that grounds agent actions in macOS accessibility semantics, OCR, and visual fallback, raised Codex Success@100 from 41.1% to 50.0% on macOSWorld-style tasks.
-
Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory
Branch-and-Browse, a tree-structured web agent with page-level action memory, reports 35.8% success and up to 40.4% less time than the cited Tree Search baseline on WebArena.
-
Initial Steps in Integrating Large Reasoning and Action Models for Service Composition
A conceptual framework integrating LRMs and LAMs for end-to-end automated service composition, with no empirical validation.
-
Augmented Vision-Language Models: A Systematic Review
A structured taxonomy of inference-time augmentation techniques that connect vision-language models to external symbolic systems, tools, and knowledge sources.
-
Vision-Language Models Can't See the Obvious
On a new benchmark of odd-one-out images, state-of-the-art vision-language models, including GPT-4o, often fail to identify which low-level feature makes the odd object stand out.
-
ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay
ARPO combines GRPO reinforcement learning with a replay buffer of successful GUI trajectories and task filtering, improving UI-Tars to 29.9% on OSWorld.
-
SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control
A multi-agent RL workflow that interleaves single-agent updates, applied to mobile GUI control, achieves SOTA zero-shot performance and a +14.8 MATH500 gain.
Discussion (0). Continue with ORCID to comment.