REVIEW 9 cited by
Reinforcement Learning for Long-Horizon Interactive LLM Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Interactive digital agents (IDAs) leverage APIs of stateful digital environments to perform tasks in response to user requests. While IDAs powered by instruction-tuned large language models (LLMs) can react to feedback from interface invocations in multi-step exchanges, they have not been trained in their respective digital environments. Prior methods accomplish less than half of tasks in sophisticated benchmarks such as AppWorld. We present a reinforcement learning (RL) approach that trains IDAs directly in their target environments. We formalize this training as a partially observable Markov decision process and derive LOOP, a data- and memory-efficient variant of proximal policy optimization. LOOP uses no value network and maintains exactly one copy of the underlying LLM in memory, making its implementation straightforward and as memory-efficient as fine-tuning a single LLM. A 32-billion-parameter agent trained with LOOP in the AppWorld environment outperforms the much larger OpenAI o1 agent by 9 percentage points (15% relative). To our knowledge, this is the first reported application of RL to IDAs that interact with a stateful, multi-domain, multi-app environment via direct API calls. Our analysis sheds light on the effectiveness of RL in this area, showing that the agent learns to consult the API documentation, avoid unwarranted assumptions, minimize confabulation, and recover from setbacks.
Forward citations
Cited by 9 Pith papers
-
WeClawArena: An Auditable Sandbox and Benchmark for Cross-User Agents Collaboration and Security in Human-Centered Agent Networks
A benchmark and Docker sandbox that tests whether owner-scoped AI agents can collaborate on real tasks without being manipulated into privacy leaks, poisoned evidence, or invalid approvals.
-
Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL
A self-speculating agent trained by alternating agent and speculator RL raises its own next-tool-call Hit@1 from 44 to 61 (Qwen3-4B) and from 49 to 66 (Qwen3.5-4B) without task-success loss.
-
TCPO: Turn-Level Credit Policy Optimization
TCPO improves multi-turn LLM RL by converting verifier scores into turn-level credit via three reference-based comparisons, beating baseline methods on math, code, and agent tasks.
-
STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training
Normalized entropy locates trajectory-neglect steps; STAPO selectively optimizes them with a trajectory-aware reward and trajectory-independent penalty, beating group-based RL baselines on agent tasks.
-
Human-Centric Reflective Architecture for Human-AI Collaborative Decision-Making
HCRA turns collaborative recommendation into a stochastic game whose termination condition is driven by human-calibrated acceptance probability and linguistic self-reflection, yielding higher success rates than uncali...
-
SOD: Step-wise On-policy Distillation for Small Language Model Agents
SOD reweights on-policy distillation strength step-by-step using divergence to stabilize tool use in small language model agents, yielding up to 20.86% gains and 26.13% on AIME 2025 for a 0.6B model.
-
Kinetics: Rethinking Test-Time Scaling Laws
A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.
-
Agentic Reinforcement Learning with Self-Distilled Reward Shaping
ADRS converts a frozen model's privileged skill-conditioned token scores into calibrated, return-gated reward shaping for agentic reinforcement learning, reporting consistent long-horizon gains.
-
Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization
BAO, a behavior-enhanced SFT plus regularized RL pipeline, improves proactive agents' task performance while lowering user-involvement rate, beating UserRL baselines on three UserRL gym tasks.
Discussion (0). Sign in to comment.