REVIEW 21 cited by
OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Graphical User Interface (GUI) agents powered by Vision-Language Models (VLMs) have demonstrated human-like computer control capability. Despite their utility in advancing digital automation, a critical bottleneck persists: collecting high-quality trajectory data for training. Common practices for collecting such data rely on human supervision or synthetic data generation through executing pre-defined tasks, which are either resource-intensive or unable to guarantee data quality. Moreover, these methods suffer from limited data diversity and significant gaps between synthetic data and real-world environments. To address these challenges, we propose OS-Genesis, a novel GUI data synthesis pipeline that reverses the conventional trajectory collection process. Instead of relying on pre-defined tasks, OS-Genesis enables agents first to perceive environments and perform step-wise interactions, then retrospectively derive high-quality tasks to enable trajectory-level exploration. A trajectory reward model is then employed to ensure the quality of the generated trajectories. We demonstrate that training GUI agents with OS-Genesis significantly improves their performance on highly challenging online benchmarks. In-depth analysis further validates OS-Genesis's efficiency and its superior data quality and diversity compared to existing synthesis methods. Our codes, data, and checkpoints are available at https://qiushisun.github.io/OS-Genesis-Home/.
Forward citations
Cited by 21 Pith papers
-
ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories
ISE creates 23,132 execution-grounded multi-turn OS agent trajectories via intent simulation and live execution, improving agent performance on ClawEval from 19.3 to 37.7 pass@1 with Qwen3-8B.
-
Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations
Relabeling LLM-agent trajectories with all goals actually achieved, plus action masking and reweighting, yields sample-efficient gains over SFT and DPO on ALFWorld, PlanCraft, and WebShop.
-
GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL
Action-weighted SFT plus KL-regularized GRPO on a curated 81K reasoning dataset lifts open-source GUI agents by 11–22 points on online task-completion benchmarks.
-
Instruction Agent: Enhancing Agent with Expert Demonstration
With one human demonstration, the Instruction Agent extracts instructions and uses verifier/backtracker modules to execute them, reaching 60% success on 20 previously unsolved OSWorld tasks.
-
CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning
A decoupled planner-executor GUI agent, trained by per-app reinforcement learning followed by specialist-to-generalist distillation, lifts ScienceBoard success from about 7.6% to 21.0% average and 40% pass@8.
-
SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
A self-evolving computer-use agent trained with full-trajectory state judging and curriculum task generation goes from 11.3% to 34.5% average success on five OSWorld apps, and a specialist-to-generalist variant beats ...
-
MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
MMBench-GUI provides a multi-platform, four-level benchmark and an efficiency-aware metric, and its experiments indicate visual grounding is the main bottleneck for current GUI agents.
-
CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback
CodeEvo uses two interacting LLM agents with keyword-guided instruction evolution and hybrid compiler-plus-LLM feedback to synthesize high-quality instruction-code pairs for fine-tuning code models.
-
WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis
A world-model-guided MCTS pipeline synthesizes 4k web navigation trajectories and yields a WebArena Pass@3 success rate of 20.15%, above OS-Genesis (18.66%) and AgentTrek (11.94%).
-
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
An attention-based action head with multi-patch supervision outperforms coordinate-generation baselines on GUI grounding, and a verifier further improves accuracy.
-
ZeroGUI: Automating Online GUI Learning at Zero Human Cost
ZeroGUI uses VLM-generated tasks and VLM-estimated rewards with two-stage GRPO to improve GUI agent success rates on OSWorld and AndroidLab without human annotations.
-
UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents
UI-Genie uses a specialized reward model and iterative self-improvement to generate synthetic training trajectories, achieving state-of-the-art results for mobile GUI agents on AndroidControl, AndroidLab, and Android Arena.
-
Agent-Environment Alignment via Automated Interface Generation
ALIGN automatically generates and verifies an interface wrapper that enriches environment observations, improving LLM agent success rates by up to 45.67 percentage points on ALFWorld without modifying the agent or env...
-
ProgRM: Build Better GUI Agents with Progress Rewards
ProgRM, a per-step progress reward model trained with LCS-based self-annotated labels, improves RL-trained GUI agent success rates on WikiHow relative to outcome reward models.
-
Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning
HyperClick trains GUI grounding models with GRPO to output clicks plus confidence scores, jointly rewarding correct clicks and Brier-calibrated confidence, and reports SOTA accuracy on six of seven benchmarks with bet...
-
Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory
Branch-and-Browse, a tree-structured web agent with page-level action memory, reports 35.8% success and up to 40.4% less time than the cited Tree Search baseline on WebArena.
-
MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment
MobileGUI-RL applies online RL with self-generated and filtered tasks plus trajectory-level rewards to mobile GUI agents, reporting improved success rates on AndroidWorld and AITW benchmarks.
-
PresentAgent: Multimodal Agent for Presentation Video Generation
PresentAgent chains LLM segmentation, slide rendering, TTS, and ffmpeg to turn documents into narrated presentation videos, but the human-level claim rests on five documents and an unvalidated VLM judge.
-
Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents
A new step-level reward-modeling benchmark for multimodal agents shows current MLLMs reach at most 61.6 percent accuracy, and benchmark score correlates strongly (r=0.981 across five models) with downstream A* search ...
-
GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior
GUI-Reflection trains an 8B multimodal GUI agent to recognize mistakes, undo incorrect actions, and retry, improving AndroidWorld success rate from 14.58% (filtered BC baseline) to 34.72% with reflection data and onli...
-
GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding
Modeling GUI elements as Gaussian distributions instead of binary targets yields 92.0% (ScreenSpot), 93.3% (ScreenSpot-v2), and 47.5% (ScreenSpot-Pro) for a 7B model, outperforming UI-TARS-72B by a relative 24.7% on t...
Discussion (0). Continue with ORCID to comment.