REVIEW 18 cited by
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Autonomous agents that address day-to-day digital tasks (e.g., ordering groceries for a household), must not only operate multiple apps (e.g., notes, messaging, shopping app) via APIs, but also generate rich code with complex control flow in an iterative manner based on their interaction with the environment. However, existing benchmarks for tool use are inadequate, as they only cover tasks that require a simple sequence of API calls. To remedy this gap, we built $\textbf{AppWorld Engine}$, a high-quality execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs and populated with realistic digital activities simulating the lives of ~100 fictitious users. We then created $\textbf{AppWorld Benchmark}$ (40K lines of code), a suite of 750 natural, diverse, and challenging autonomous agent tasks requiring rich and interactive code generation. It supports robust programmatic evaluation with state-based unit tests, allowing for different ways of completing a task while also checking for unexpected changes, i.e., collateral damage. The state-of-the-art LLM, GPT-4o, solves only ~49% of our 'normal' tasks and ~30% of 'challenge' tasks, while other models solve at least 16% fewer. This highlights the benchmark's difficulty and AppWorld's potential to push the frontiers of interactive coding agents. The project website is available at https://appworld.dev/.
Forward citations
Cited by 18 Pith papers
-
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
OSWorld 2.0 is a benchmark of 108 realistic long-horizon computer-use tasks where current agents achieve only 20.6% binary completion, struggling with state inference and constraint tracking.
-
Autonomous Repair for Multi-Agent Systems via Monte-Carlo Tree Search
An MCTS-based repair loop with partial rollouts and an LLM-as-a-judge reward raises post-repair pass rates by 3–12% over DoVer, ReAct, and Reflexion baselines, backed by a new replayable benchmark StateMAS.
-
WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing
WorkSurface-Bench measures surface routing separately from answer correctness and finds near-perfect routing still leaves 25–44% answer errors across four LLM backbones.
-
HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
A new 65-task benchmark measures whether AI agents obey long company handbooks across multi-tool workflows; the best model passes 36.2% under strict grading.
-
RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback
Hindsight dual intrinsic feedback (progress rewards + distilled lessons with SimUtil-UCB) lets LLM agents evolve across episodes and sets new SOTA success rates on ALFWorld, WebShop, Sokoban and Minesweeper.
-
The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios
Trainee-Bench evaluates LLM agents on scheduling, active exploration, and continual learning in randomized workplace scenarios; the best current model completes only 35% of tasks.
-
SpecifyUI: Supporting Iterative UI Design Intent Expression through Structured Specifications and Generative AI
Structured, parameterized UI specifications extracted from screenshots improve LLM-based generation fidelity and iterative control compared with text prompting.
-
VehicleWorld: A Highly Integrated Multi-Device Environment for Intelligent Vehicle Interaction
Agents that directly set the car's state in a new vehicle simulator outperformed API-calling agents on both accuracy and speed.
-
Doc2Agent: Scalable Generation of Tool-Using Agents from API Documentation
Doc2Agent automatically converts unstructured REST API documentation into validated, Python-based tools for AI agents, reporting a 55% relative WebArena improvement over direct API calling.
-
Self-Challenging Language Model Agents
A language model agent can generate its own verifiable training tasks and improve its tool-use success rate by about 2x without human-annotated data.
-
lmgame-Bench: How Good are LLMs at Playing Games?
lmgame-Bench turns six classic games into a scaffolded LLM evaluation suite, ranks 13 models, detects contamination, and reports RL transfer from Sokoban or Tetris to unseen games and planning tasks.
-
ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision
Partial LLM-agent evaluations can support a promote/reject decision after 15–25% of tasks on some benchmarks, but not others; reports should state the decision rule and unresolved count.
-
AgentOmnia: Scaling Agentic Models for Full-Scenario Applications
Post-training an open Qwen3-30B agent on 52,361 verifiable tasks in 5,018 synthesized stateful environments lifts its average across four agent benchmarks from 22.9% to 41.7%.
-
OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation
A hierarchical multi-agent framework with a domain-agnostic planner trained by SFT and DPO achieves 69.70% on GAIA and lifts a Qwen2.5-32B planner by 16.37 points.
-
Evaluation and Benchmarking of LLM Agents: A Survey
A review that proposes a two-dimensional taxonomy for evaluating LLM agents and highlights enterprise-specific evaluation gaps.
-
Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
In a new three-game benchmark tracking planning, revision, and budget use across 12 LLMs, ChatGPT-o3-mini ranked highest, while overcorrecting models such as Qwen-Plus won few matches.
-
A Red Teaming Roadmap Towards System-Level Safety
A position paper from Scale AI argues that red teaming research should prioritize product-level safety specifications, realistic attacker models, and system-level monitoring over abstract model-level harm benchmarks.
-
Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.
Discussion (0). Continue with ORCID to comment.