REVIEW 15 cited by
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Existing benchmarks for grounding language in interactive environments either lack real-world linguistic elements, or prove difficult to scale up due to substantial human involvement in the collection of data or feedback signals. To bridge this gap, we develop WebShop -- a simulated e-commerce website environment with $1.18$ million real-world products and $12,087$ crowd-sourced text instructions. Given a text instruction specifying a product requirement, an agent needs to navigate multiple types of webpages and issue diverse actions to find, customize, and purchase an item. WebShop provides several challenges for language grounding including understanding compositional instructions, query (re-)formulation, comprehending and acting on noisy text in webpages, and performing strategic exploration. We collect over $1,600$ human demonstrations for the task, and train and evaluate a diverse range of agents using reinforcement learning, imitation learning, and pre-trained image and language models. Our best model achieves a task success rate of $29\%$, which outperforms rule-based heuristics ($9.6\%$) but is far lower than human expert performance ($59\%$). We also analyze agent and human trajectories and ablate various model components to provide insights for developing future agents with stronger language understanding and decision making abilities. Finally, we show that agents trained on WebShop exhibit non-trivial sim-to-real transfer when evaluated on amazon.com and ebay.com, indicating the potential value of WebShop in developing practical web-based agents that can operate in the wild.
Forward citations
Cited by 15 Pith papers
-
Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce
LLM merchant agents in a new dynamic auction benchmark capture at most 32% of hindsight-optimal profit; profit tracks margin per win more than win rate, and fast pre-shock learners adapt poorly to preference shocks.
-
Interactive Task Alignment as a POMDP
Under ambiguous user requests, current LLMs recover the intended task only 22–32% of the time, well below human accuracy of 48%, and post-training only partially closes the gap.
-
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
OSWorld 2.0 is a benchmark of 108 realistic long-horizon computer-use tasks where current agents achieve only 20.6% binary completion, struggling with state inference and constraint tracking.
-
Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories
Offline DPO on expert states with one-step student negatives and schema-preserving augmentation improves LLM agents beyond SFT and can match online GRPO without environment interaction.
-
From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents
Reward-hack activations flag latent policy states in LLM agents but require added entropy and context features to better predict when those states lead to exploit actions.
-
Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History
Persona2Web is a new open-web benchmark where agents must infer a user's preferences from synthetic browsing history to solve intentionally ambiguous queries; current best agents score 13% success.
-
LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services
LocalSearchBench—1.3M merchant records and 900 multi-hop local-life QA tasks across 9 Chinese cities—shows the best reasoning agent reaches only 35.6% correctness.
-
Graph-Enhanced Policy Optimization in LLM Agent Training
GEPO adds graph-centrality-based intrinsic rewards, dynamic discounts, and two-level advantage shaping to group-based RL, improving LLM agent success on ALFWorld, WebShop, and a private Workbench benchmark.
-
Magentic-UI: Towards Human-in-the-loop Agentic Systems
Magentic-UI, an open-source human-in-the-loop agent interface, reports that lightweight simulated-user input raises GAIA task completion from 30.3% to 51.9%.
-
A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning
Multi-turn RL with only unary 'try again' feedback improves multi-turn reasoning accuracy by up to 14% while preserving single-turn performance.
-
Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework
A framework paper that adapts AI safety case methodology to the specific threat of manipulation attacks by internally deployed misaligned AI.
-
OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems
OPT-BENCH is a 30-task benchmark showing that LLM agents generally improve optimization results when they are given their own past solutions and feedback.
-
Data-Driven Boundary Control of Distributed Port-Hamiltonian Systems
GP-dPHS learning plus interconnection boundary control yields probabilistic boundedness conditions for closed-loop trajectories under Hamiltonian model mismatch.
-
AgentFly: Extensible and Scalable Reinforcement Learning for LM Agents
AgentFly enables multi-turn RL training for LM agents with token masking, asynchronous tool calls, and pooled environment management, demonstrated on six tasks.
-
Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.
Discussion (0). Sign in to comment.