REVIEW 14 cited by
Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Autonomous agents powered by large language models (LLMs) have the potential to enhance human capabilities, assisting with digital tasks from sending emails to performing data analysis. The abilities of existing LLMs at such tasks are often hindered by the lack of high-quality agent data from the corresponding environments they interact with. We propose Learn-by-interact, a data-centric framework to adapt LLM agents to any given environments without human annotations. Learn-by-interact synthesizes trajectories of agent-environment interactions based on documentations, and constructs instructions by summarizing or abstracting the interaction histories, a process called backward construction. We assess the quality of our synthetic data by using them in both training-based scenarios and training-free in-context learning (ICL), where we craft innovative retrieval approaches optimized for agents. Extensive experiments on SWE-bench, WebArena, OSWorld and Spider2-V spanning across realistic coding, web, and desktop environments show the effectiveness of Learn-by-interact in various downstream agentic tasks -- baseline results are improved by up to 12.2\% for ICL with Claude-3.5 and 19.5\% for training with Codestral-22B. We further demonstrate the critical role of backward construction, which provides up to 14.0\% improvement for training. Our ablation studies demonstrate the efficiency provided by our synthesized data in ICL and the superiority of our retrieval pipeline over alternative approaches like conventional retrieval-augmented generation (RAG). We expect that Learn-by-interact will serve as a foundation for agent data synthesis as LLMs are increasingly deployed at real-world environments.
Forward citations
Cited by 14 Pith papers
-
WeClawArena: An Auditable Sandbox and Benchmark for Cross-User Agents Collaboration and Security in Human-Centered Agent Networks
A benchmark and Docker sandbox that tests whether owner-scoped AI agents can collaborate on real tasks without being manipulated into privacy leaks, poisoned evidence, or invalid approvals.
-
RMSWeb: Reflection, Failure-Mode Mining, and Salvage-DS for Web Agent Reinforcement Learning
RMSWeb combines reflection-conditioned collection, failure-mode mining, and a competence-gated offline RL objective to improve compact web agents' success and step efficiency on live-web benchmarks.
-
Agent Learning via Early Experience
Training agents to predict consequences of their own alternative actions (implicit world modeling) or to reflect on why expert actions were better (self-reflection) consistently improves task success, OOD generalizati...
-
WebGuard: Building a Generalizable Guardrail for Web Agents
WebGuard introduces an action-level risk dataset for web agents and shows that a fine-tuned 7B model improves risk-prediction accuracy from about 38% to 80% and high-risk recall from 20% to 76%, still below deployment...
-
Context manipulation attacks : Web agents are susceptible to corrupted memory
Web agents with protected prompts can still be hijacked by injecting malicious steps into their stored task plans, reaching up to 63% success on privacy leaks.
-
Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
Scaling the number of interaction steps, trained via a curriculum over rollout horizon, improves web-agent task success and outperforms scaling per-step reasoning under fixed token budgets.
-
Contextual Experience Replay for Self-Improvement of Language Agents
Contextual Experience Replay distills trajectories into retrievable dynamics and skills, improving WebArena success from 24.3% to 36.7% and VisualWebArena from 26.2% to 31.9% without any training.
-
LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback
Self-generated agent trajectories from LAM SIMULATOR improved fine-tuned model pass rates by up to 49.3% on ToolBench and CRMArena.
-
Self-Challenging Language Model Agents
A language model agent can generate its own verifiable training tasks and improve its tool-use success rate by about 2x without human-annotated data.
-
AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery
Fake 'Close AD' ads make VLM web agents click them over 60% of the time, and near 100% in some settings.
-
ProgRM: Build Better GUI Agents with Progress Rewards
ProgRM, a per-step progress reward model trained with LCS-based self-annotated labels, improves RL-trained GUI agent success rates on WikiHow relative to outcome reward models.
-
Rethinking Agent Design: From Top-Down Workflows to Bottom-Up Skill Evolution
Agents that start with no game knowledge can build a reusable skill library through trial-and-error and visual feedback, then progress further in two complex games than baseline agents given extra hints.
-
Cognitive Duality for Adaptive Web Agents
A web agent called CogniWeb reportedly reaches 43.96% success on WebArena with 75% fewer tokens by switching between fast and slow reasoning, but the supplied body is a different paper and the claims could not be checked.
-
Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback
EXIF repeatedly has a teacher agent explore an environment, relabel the exploration as tasks, train a student agent on it, and use the student's failures to guide the next round, improving 7B-8B agents in Webshop and Crafter.
Discussion (0). Continue with ORCID to comment.