REVIEW 25 cited by
Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena. Webpage: https://microsoft.github.io/WindowsAgentArena Code: https://github.com/microsoft/WindowsAgentArena
Forward citations
Cited by 25 Pith papers
-
SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents
SciVisAgentBench provides 108 expert-crafted tasks and a mixed LLM-plus-deterministic evaluation pipeline for benchmarking AI agents that perform scientific visualization workflows.
-
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
VLM judges of CUA trajectories are systematically lenient; OSReward measures this with human gold, and OS-Shepherd open models close most of the reliability gap cheaply.
-
Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs
On OSWorld, inference-time scaling for local CUAs mostly yields diminishing returns and shifts failures toward premature false successes rather than raising task success.
-
StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents
StateAct, a code-first multi-agent harness that acts on program state instead of pixels, improves Claude Opus 4.8 on OSWorld 2.0 from 20.6% to 26.9% binary success at ~9x lower cost.
-
SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL
Verifiable task synthesis plus frontier sampling and sliding-window visual training yields open-source SOTA computer-use agents on OSWorld and ScienceBoard.
-
UI-MOPD: Multi-Platform On-Policy Distillation for Continual GUI Agent Learning
Platform-conditioned multi-teacher on-policy distillation lets a shared 8B GUI policy adapt across desktop and mobile without collapsing platform-specific action conventions.
-
ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
A program-aware scheduler and tool-lifecycle manager for LLM agent workflows raises serving throughput by 1.5–3.6x and RL rollout throughput by 1.8–3.9x over vLLM/Continuum baselines.
-
UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
Training computer-use agents on a hybrid GUI-plus-programmatic-tool action space, with SFT and online RL, improves OSWorld success by ~23% relative over GUI-only baselines and transfers to Windows without Windows-spec...
-
Instruction Agent: Enhancing Agent with Expert Demonstration
With one human demonstration, the Instruction Agent extracts instructions and uses verifier/backtracker modules to execute them, reaching 60% success on 20 previously unsolved OSWorld tasks.
-
MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
MMBench-GUI provides a multi-platform, four-level benchmark and an efficiency-aware metric, and its experiments indicate visual grounding is the main bottleneck for current GUI agents.
-
BIMgent: Towards Autonomous Building Modeling via Computer-use Agents
BIMgent, a GUI-controlling LLM agent, completes 32% of BIM building modeling tasks end-to-end, outperforming baseline computer-use agents that complete none.
-
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
An attention-based action head with multi-patch supervision outperforms coordinate-generation baselines on GUI grounding, and a verifier further improves accuracy.
-
ZeroGUI: Automating Online GUI Learning at Zero Human Cost
ZeroGUI uses VLM-generated tasks and VLM-estimated rewards with two-stage GRPO to improve GUI agent success rates on OSWorld and AndroidLab without human annotations.
-
AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery
Fake 'Close AD' ads make VLM web agents click them over 60% of the time, and near 100% in some settings.
-
Rethinking Agent Design: From Top-Down Workflows to Bottom-Up Skill Evolution
Agents that start with no game knowledge can build a reusable skill library through trial-and-error and visual feedback, then progress further in two complex games than baseline agents given extra hints.
-
RT-SHCUA: Real-Time Self-Hosted Computer-Use Agent for UAV Control
An architecture that mediates LLM computer-use agents for UAV control by compiling agent decisions into validated, time-bounded, evidence-logged skill invocations, with a prototype on OpenClaw/PX4/OP-TEE.
-
Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents
Sidekick's multimodal feedback significantly improves multitasking performance with computer-use agents compared to chat-only or peripheral text feedback.
-
EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents
Online RL with STEPO, policy-aware task filtering, DTAC curriculum, and async rollout infrastructure lifts a 32B computer-use agent to 63.2% on OSWorld-Verified.
-
Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System
AI phone agents succeed on single-step commands but fail on multi-step compositions; a new benchmark measures this gap and a scheduler that decomposes tasks recovers most of it.
-
FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents
A new interactive benchmark for form-filling agents reports that current multimodal models fail at the task, but the evaluation protocol is internally inconsistent and may not measure form-filling ability fairly.
-
A Lightweight Incentive-Based Privacy-Preserving Smart Metering Protocol for Value-Added Services
A layered protocol of local differential privacy, blind signatures, pseudonyms, temporal aggregation, and anonymous routing is claimed to keep smart-meter readings private while still enabling reward token redemption.
-
Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.
-
DPO Learning with LLMs-Judge Signal for Computer Use Agents
An LLM-as-Judge pipeline that scores synthetic GUI interaction trajectories and fine-tunes a 2B model with DPO yields a local computer-use agent that beats its base model on 15-step OSWorld tasks.
-
A Red Teaming Roadmap Towards System-Level Safety
A position paper from Scale AI argues that red teaming research should prioritize product-level safety specifications, realistic attacker models, and system-level monitoring over abstract model-level harm benchmarks.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Sign in to comment.