REVIEW 21 cited by
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Autonomous interaction with the computer has been a longstanding challenge with great potential, and the recent proliferation of large language models (LLMs) has markedly accelerated progress in building digital agents. However, most of these agents are designed to interact with a narrow domain, such as a specific software or website. This narrow focus constrains their applicability for general computer tasks. To this end, we introduce OS-Copilot, a framework to build generalist agents capable of interfacing with comprehensive elements in an operating system (OS), including the web, code terminals, files, multimedia, and various third-party applications. We use OS-Copilot to create FRIDAY, a self-improving embodied agent for automating general computer tasks. On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods by 35%, showcasing strong generalization to unseen applications via accumulated skills from previous tasks. We also present numerical and quantitative evidence that FRIDAY learns to control and self-improve on Excel and Powerpoint with minimal supervision. Our OS-Copilot framework and empirical findings provide infrastructure and insights for future research toward more capable and general-purpose computer agents.
Forward citations
Cited by 21 Pith papers
-
FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks
A new benchmark dataset and evaluation framework for testing multimodal AI agents on real field work tasks derived from on-site data and worker interviews.
-
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
VLM judges of CUA trajectories are systematically lenient; OSReward measures this with human gold, and OS-Shepherd open models close most of the reliability gap cheaply.
-
CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning
A decoupled planner-executor GUI agent, trained by per-app reinforcement learning followed by specialist-to-generalist distillation, lifts ScienceBoard success from about 7.6% to 21.0% average and 40% pass@8.
-
PG-Agent: An Agent Powered by Page Graph
An MLLM GUI agent that stores past episodes as a page graph and retrieves action guidelines from it improves step success on three benchmarks.
-
SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
A self-evolving computer-use agent trained with full-trajectory state judging and curriculum task generation goes from 11.3% to 34.5% average success on five OSWorld apps, and a specialist-to-generalist variant beats ...
-
Magentic-UI: Towards Human-in-the-loop Agentic Systems
Magentic-UI, an open-source human-in-the-loop agent interface, reports that lightweight simulated-user input raises GAIA task completion from 30.3% to 51.9%.
-
MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them
A new benchmark with a three-way hallucination taxonomy, snapshot-based test cases, and an LLM judge shows LLM agents hallucinate at over 30% of risky decision points, with open and closed models closer than expected.
-
MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
MMBench-GUI provides a multi-platform, four-level benchmark and an efficiency-aware metric, and its experiments indicate visual grounding is the main bottleneck for current GUI agents.
-
StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley
StarDojo is a 1,000-task benchmark in Stardew Valley combining production and social activities, and the best tested MLLM (GPT-4.1) achieves only 12.7% success on its 100-task subset.
-
BIMgent: Towards Autonomous Building Modeling via Computer-use Agents
BIMgent, a GUI-controlling LLM agent, completes 32% of BIM building modeling tasks end-to-end, outperforming baseline computer-use agents that complete none.
-
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
An attention-based action head with multi-patch supervision outperforms coordinate-generation baselines on GUI grounding, and a verifier further improves accuracy.
-
FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow
FullFront adds a three-task benchmark for webpage design, perception, and code generation, and finds top MLLMs still fail at fine-grained layout and interaction implementation.
-
HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization
HiLSVA shows that a human-in-the-loop LLM agent system can help novices and experts complete scientific visualization tasks, while human oversight adds measurable execution time.
-
OAgents: An Empirical Study of Building Effective Agents
OAgents reports a framework and ablation study claiming open-source state-of-the-art on GAIA, but the empirical evidence is weakened by benchmark fitting and missing statistics.
-
Scaling Test-time Compute for LLM Agents
On the GAIA benchmark, Best-of-N sampling with list-wise answer selection gives the largest agent gains, and selective (score-triggered) reflection beats reflection at every step.
-
Truly Self-Improving Agents Require Intrinsic Metacognitive Learning
The paper proposes that self-improving agents must learn to manage their own learning processes, framing this as intrinsic metacognitive learning, and argues it is necessary for sustained and generalized improvement.
-
Unifying Language Agent Algorithms with Graph-based Orchestration Engine for Reproducible Agent Research
AGORA is a graph-based agent framework that standardizes ten reasoning algorithms; its evaluations show simple Chain-of-Thought prompting is often the most cost-effective, though without statistical rigor.
-
OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation
A hierarchical multi-agent framework with a domain-agnostic planner trained by SFT and DPO achieves 69.70% on GAIA and lifts a Qwen2.5-32B planner by 16.37 points.
-
Agents Require Metacognitive and Strategic Reasoning to Succeed in the Coming Labor Markets
AI agents in future labor markets will need metacognitive and strategic reasoning because incomplete information creates adverse selection, moral hazard, and reputation effects.
-
AppAgent-Claw: CLI Is All You Need for GUI Automation
A record-once, replay-many system converts demonstrated GUI workflows into reliable OpenClaw skills via layered visual localization and post-action validation, without runtime LLM inference.
-
DPO Learning with LLMs-Judge Signal for Computer Use Agents
An LLM-as-Judge pipeline that scores synthetic GUI interaction trajectories and fine-tunes a 2B model with DPO yields a local computer-use agent that beats its base model on 15-step OSWorld tasks.
Discussion (0). Sign in to comment.