REVIEW 14 cited by
ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Answering complex natural language questions often necessitates multi-step reasoning and integrating external information. Several systems have combined knowledge retrieval with a large language model (LLM) to answer such questions. These systems, however, suffer from various failure cases, and we cannot directly train them end-to-end to fix such failures, as interaction with external knowledge is non-differentiable. To address these deficiencies, we define a ReAct-style LLM agent with the ability to reason and act upon external knowledge. We further refine the agent through a ReST-like method that iteratively trains on previous trajectories, employing growing-batch reinforcement learning with AI feedback for continuous self-improvement and self-distillation. Starting from a prompted large model and after just two iterations of the algorithm, we can produce a fine-tuned small model that achieves comparable performance on challenging compositional question-answering benchmarks with two orders of magnitude fewer parameters.
Forward citations
Cited by 14 Pith papers
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
LLMs for Agentic Home Energy Management
Tool-calling LLM agents can make near-optimal home appliance schedules on ordinary tariff days, but they regularly fail safety constraints and therefore need a deterministic feasibility validator before actuation.
-
Estimating the Empowerment of Language Model Agents
EELMA estimates the mutual information between an LM agent's actions and future text states, and this 'empowerment' is shown to correlate with task performance across toy games and WebArena.
-
UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making
UProp estimates extrinsic uncertainty in multi-step LLM decision-making by approximating pointwise mutual information over trajectory samples, improving AUROC for identifying incorrect agent outputs by 2.3 to 11% over...
-
WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model
A co-evolving world model that simulates web page transitions improves both training-data generation and inference-time planning for self-improving web agents.
-
Exploring Expert Failures Improves LLM Agent Tuning
EEF improves LLM agent fine-tuning by extracting beneficial action segments from failed expert trajectories via rollouts from intermediate states, reporting a 62% WebShop win rate and a ScienceWorld score above 81.
-
Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments
A data-centric method that relabels agent trajectories with new instructions, called backward construction, improves LLM agent performance on web, code, desktop, and data-science tasks without human labeling.
-
Engineering Trustworthy Agentic AI for Critical Systems
A survey claiming that agentic AI trustworthiness is a single cross-domain problem and outlining a framework for graded, certifiable assurance.
-
Truly Self-Improving Agents Require Intrinsic Metacognitive Learning
The paper proposes that self-improving agents must learn to manage their own learning processes, framing this as intrinsic metacognitive learning, and argues it is necessary for sustained and generalized improvement.
-
Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
A trained critique model providing step-level feedback improves an LLM actor's mathematical reasoning at test time and during self-improvement, with gains concentrated on hard problems.
-
AI Reasoning for Wireless Communications and Networking: A Survey and Perspectives
A survey that organizes LLM and AI reasoning methods into a taxonomy and maps them onto the physical, link, network, transport, and application layers of wireless networks.
-
METEOR: Evolutionary Journey of Large Language Models from Guidance to Self-Growth
METEOR combines weak-to-strong distillation, iterative GPT-4 feedback, and contrastive self-training to adapt 7B-8B LLMs to a domain, with gains measured only by GPT-4 as judge.
-
Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks
A hybrid LLM-plus-RL agent is described, but its performance table is explicitly illustrative and no code or data is provided, so the claimed gains are not established.
-
Toward Efficient Agents: Memory, Tool learning, and Planning
A survey that organizes efficiency techniques for LLM agents into memory, tool learning, and planning, and consolidates benchmarks and metrics for measuring cost-performance trade-offs.
Discussion (0). Continue with ORCID to comment.