REVIEW 7 cited by
ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. Most of these agents rely on general-purpose, proprietary models like GPT-4 and focus on designing better prompts to improve their planning abilities. However, general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. This simple yet effective approach shows substantial gains over prompting-based agents on existing benchmarks -- ScribeAgent achieves state-of-the-art direct generation performance on Mind2Web and improves the task success rate by 7.3% over the previous best text-only web agents on WebArena. We further perform detailed ablation studies on various fine-tuning design choices and provide insights into LLM selection, training recipes, context window optimization, and effect of dataset sizes.
Forward citations
Cited by 7 Pith papers
-
Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories
Offline DPO on expert states with one-step student negatives and schema-preserving augmentation improves LLM agents beyond SFT and can match online GRPO without environment interaction.
-
Agent Learning via Early Experience
Training agents to predict consequences of their own alternative actions (implicit world modeling) or to reflect on why expert actions were better (self-reflection) consistently improves task success, OOD generalizati...
-
Context manipulation attacks : Web agents are susceptible to corrupted memory
Web agents with protected prompts can still be hijacked by injecting malicious steps into their stored task plans, reaching up to 63% success on privacy leaks.
-
What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
A self-generating graph benchmark produces 36k GUI agent tasks with controllable complexity and ten capability scores, and fine-tuning on its trajectories gives small gains on AndroidControl and OmniAct.
-
Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
Scaling the number of interaction steps, trained via a curriculum over rollout horizon, improves web-agent task success and outperforms scaling per-step reasoning under fixed token budgets.
-
InSTA: Towards Internet-Scale Training For Agents
Automated LLM task generation, agent execution, and judge filtering at 150k-site scale lets a 1.7B model match much larger web agents.
-
Reflection-Based Memory For Web navigation Agents
Reflection-Augmented Planning (ReAP) retrieves short self-reflections from past web navigation tasks and lifts held-out task success by 11 points on WebArena.
Discussion (0). Continue with ORCID to comment.