REVIEW 5 cited by
From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The outstanding capabilities of large language models (LLMs) render them a crucial component in various autonomous agent systems. While traditional methods depend on the inherent knowledge of LLMs without fine-tuning, more recent approaches have shifted toward the reinforcement learning strategy to further enhance agents' ability to solve complex interactive tasks with environments and tools. However, previous approaches are constrained by the sparse reward issue, where existing datasets solely provide a final scalar reward for each multi-step reasoning chain, potentially leading to ineffectiveness and inefficiency in policy learning. In this paper, we introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. Inheriting the spirit of novice-to-expert theory, we first compare the actions of the expert and the agent to automatically generate intermediate rewards for fine-grained optimization. Additionally, we propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment. Further theoretical analysis demonstrates that the action distribution of the agent can converge toward the expert action distribution over multiple training cycles. Experimental results across various datasets indicate that StepAgent outperforms existing baseline methods.
Forward citations
Cited by 5 Pith papers
-
RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents
Reward-Swap Policy Optimization uses dense-reward exploration to generate diverse trajectories that then train the final policy strictly under true outcome rewards, lifting multi-turn agent performance.
-
Process Reward Informed Tree Rollout for Effective Multi-Turn RL
Using process feedback to guide which partial trajectories to branch on, PATR replaces uniform independent rollouts with adaptive tree rollouts for multi-turn agent RL, reporting +5.0 points on SWE-Bench and +9.3 on F...
-
Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
EMPG re-weights policy-gradient updates by step-level token entropy, amplifying confident correct actions and muting uncertain ones, and adds a future-clarity bonus.
-
ODYSSE: Episode-wise Policy Optimization for Personalized Agentic Reasoning
An episode-level variant of GRPO (ESPO) improves personalized GUI-agent reasoning on the 102-episode SmartSpot benchmark, outperforming step-wise and outcome-only training baselines.
-
SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution
SPA-RL attributes a single final task reward to individual agent steps and uses the attributions plus an executability signal as dense rewards for PPO, yielding modest success-rate gains on WebShop, ALFWorld, and VirtualHome.
Discussion (0). Sign in to comment.