Pith. sign in

REVIEW 9 cited by

UFO2: The Desktop AgentOS

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2504.14603 v2 pith:BONUVSQV submitted 2025-04-20 cs.AI cs.HCcs.OS

Chaoyun Zhang , He Huang , Chiming Ni , Jian Mu , Si Qin , Shilin He , Lu Wang , Fangkai Yang
show 13 more authors
Pu Zhao Chao Du Liqun Li Yu Kang Zhao Jiang Suzhen Zheng Rujia Wang Jiaxu Qian Minghua Ma Jian-Guang Lou Qingwei Lin Saravan Rajmohan Dongmei Zhang
This is my paper · ORCID
classification cs.AIcs.HCcs.OS
keywords automationcuasdesktopufo2executionwindowsagentosagents
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent Computer-Using Agents (CUAs), powered by multimodal large language models (LLMs), offer a promising direction for automating complex desktop workflows through natural language. However, most existing CUAs remain conceptual prototypes, hindered by shallow OS integration, fragile screenshot-based interaction, and disruptive execution. We present UFO2, a multiagent AgentOS for Windows desktops that elevates CUAs into practical, system-level automation. UFO2 features a centralized HostAgent for task decomposition and coordination, alongside a collection of application-specialized AppAgent equipped with native APIs, domain-specific knowledge, and a unified GUI--API action layer. This architecture enables robust task execution while preserving modularity and extensibility. A hybrid control detection pipeline fuses Windows UI Automation (UIA) with vision-based parsing to support diverse interface styles. Runtime efficiency is further enhanced through speculative multi-action planning, reducing per-step LLM overhead. Finally, a Picture-in-Picture (PiP) interface enables automation within an isolated virtual desktop, allowing agents and users to operate concurrently without interference. We evaluate UFO2 across over 20 real-world Windows applications, demonstrating substantial improvements in robustness and execution accuracy over prior CUAs. Our results show that deep OS integration unlocks a scalable path toward reliable, user-aligned desktop automation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SWE-bench Goes Live!

    cs.SE 2025-05 conditional novelty 7.0 of 10

    SWE-bench-Live provides a live, automatically curated, Docker-backed benchmark of 1,319 fresh GitHub issue-fixing tasks, on which leading agents score around 19%, well below their SWE-bench Verified results.

  2. StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

    cs.SE 2026-07 conditional novelty 6.0 of 10

    StateAct, a code-first multi-agent harness that acts on program state instead of pixels, improves Claude Opus 4.8 on OSWorld 2.0 from 20.6% to 26.9% binary success at ~9x lower cost.

  3. Long-Term Memory for VLA-based Agents in Open-World Task Execution

    cs.RO 2026-04 unverdicted novelty 6.0 of 10

    ChemBot adds dual-layer memory and future-state asynchronous inference to VLA models, enabling better long-horizon success in chemical lab automation on collaborative robots.

  4. BIMgent: Towards Autonomous Building Modeling via Computer-use Agents

    cs.AI 2025-06 conditional novelty 6.0 of 10

    BIMgent, a GUI-controlling LLM agent, completes 32% of BIM building modeling tasks end-to-end, outperforming baseline computer-use agents that complete none.

  5. GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents

    cs.CL 2025-06 conditional novelty 6.0 of 10

    An attention-based action head with multi-patch supervision outperforms coordinate-generation baselines on GUI grounding, and a verifier further improves accuracy.

  6. ScreenExplorer: Training a Vision-Language Model for Diverse Exploration in Open GUI World

    cs.AI 2025-05 reject novelty 6.0 of 10

    A VLM trained with GRPO and a world-model curiosity reward explores a real desktop GUI more diversely than larger frozen models, but the diversity metric is nearly identical to its training reward.

  7. HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization

    cs.HC 2026-06 unverdicted novelty 5.0 of 10

    HiLSVA shows that a human-in-the-loop LLM agent system can help novices and experts complete scientific visualization tasks, while human oversight adds measurable execution time.

  8. GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Modeling GUI elements as Gaussian distributions instead of binary targets yields 92.0% (ScreenSpot), 93.3% (ScreenSpot-v2), and 47.5% (ScreenSpot-Pro) for a 7B model, outperforming UI-TARS-72B by a relative 24.7% on t...

  9. Get Experience from Practice: LLM Agents with Record & Replay

    cs.LG 2025-05 reject novelty 4.0 of 10

    AgentRR is a proposed paradigm that records agent traces, generalizes them into multi-level experiences, and replays them under safety checks to make LLM agents cheaper, faster, and more reliable.

Pith tools