REVIEW 23 cited by
Teaching Large Language Models to Reason with Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Reinforcement Learning from Human Feedback (\textbf{RLHF}) has emerged as a dominant approach for aligning LLM outputs with human preferences. Inspired by the success of RLHF, we study the performance of multiple algorithms that learn from feedback (Expert Iteration, Proximal Policy Optimization (\textbf{PPO}), Return-Conditioned RL) on improving LLM reasoning capabilities. We investigate both sparse and dense rewards provided to the LLM both heuristically and via a learned reward model. We additionally start from multiple model sizes and initializations both with and without supervised fine-tuning (\textbf{SFT}) data. Overall, we find all algorithms perform comparably, with Expert Iteration performing best in most cases. Surprisingly, we find the sample complexity of Expert Iteration is similar to that of PPO, requiring at most on the order of $10^6$ samples to converge from a pretrained checkpoint. We investigate why this is the case, concluding that during RL training models fail to explore significantly beyond solutions already produced by SFT models. Additionally, we discuss a trade off between maj@1 and pass@96 metric performance during SFT training and how conversely RL training improves both simultaneously. We then conclude by discussing the implications of our findings for RLHF and the future role of RL in LLM fine-tuning.
Forward citations
Cited by 23 Pith papers
-
RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models
RepoReasoner is a repository-level code-reasoning benchmark with output-prediction and call-chain tasks; the best LLM reaches only 69.1% Pass@1 even with oracle context, with low recall in dependency tracing.
-
LeAct: Learning to Reason from Expert Actions
An AI can learn to reason by sampling explanations for an expert's actions and keeping only the ones that help it predict those actions.
-
AdaKP: Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learning
Online re-selection of atomic hint subsets during RL training improves competition-math accuracy by +1.85 average over a fixed-hint baseline, concentrated on reward-sparse benchmarks.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Med-R$^3$: Enhancing Medical Retrieval-Augmented Reasoning of LLMs via Progressive Reinforcement Learning
A progressive three-stage RL framework with medical-specific rewards improves retrieval-augmented reasoning on medical QA benchmarks, reportedly surpassing GPT-4o-mini with an 8B model.
-
Learning Efficient Robotic Garment Manipulation with Standardization
APS-Net combines fling and pick-and-place actions to unfold and standardize garments, achieving better coverage, alignment, and real-world folding success than earlier methods.
-
Intent Factored Generation: Unleashing the Diversity in Your Language Model
Intent Factored Generation samples a high-temperature intent, such as keywords or a summary, and then samples the final response at lower temperature conditioned on that intent, increasing semantic diversity while kee...
-
Learning to Select In-Context Demonstration Preferred by Large Language Model
A generative preference-learning method trains a latent demonstration selector from LLM feedback and improves few-shot in-context learning performance on most of 19 benchmark datasets.
-
Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
A generative reward model trained with long chain-of-thought and rule-based RL outperforms standard and vertically scaled reward baselines on RM-Bench and RewardBench.
-
Process Reward Models for LLM Agents: Practical Framework and Directions
A lightweight actor-critic loop trains 3B LLM agents to surpass GPT-4o on ALFWorld by learning step-level reward models from rollouts or demonstrations.
-
Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details
For Other-Play in Yokai, agents trained with different implementation details coordinate across implementations about as well as across seeds, supporting inter-seed cross-play as a proxy for cross-implementation evaluation.
-
RePO: Replay-Enhanced Policy Optimization
RePO adds a replay buffer to GRPO so each prompt is optimized on both fresh and stored samples, yielding higher math benchmark scores than GRPO across several small language models.
-
Truly Self-Improving Agents Require Intrinsic Metacognitive Learning
The paper proposes that self-improving agents must learn to manage their own learning processes, framing this as intrinsic metacognitive learning, and argues it is necessary for sustained and generalized improvement.
-
MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning
Multi-domain RLVR data mixing, guided by a quadratic surrogate fitted to 11 pilot runs, improves a Qwen2-VL-2B model's out-of-distribution accuracy by about 5 points over uniform mixing.
-
RAST: Reasoning Activation in LLMs via Small-model Transfer
Injecting the logit gap between a small RL-trained model and its base into a larger base model at decoding time improves reasoning accuracy on math and code benchmarks, sometimes matching or exceeding RL training of t...
-
HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving
The HCRMP planner feeds LLM semantic hints into state representation and critic weighting instead of letting the LLM decide actions, reporting better CARLA driving metrics.
-
Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
A training-free method that feeds probability-weighted token embeddings back into LLMs during reasoning, improving accuracy and token efficiency on math and coding tasks.
-
From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI
Autonomous AI becomes dependable when tool use is embedded in persistent workspaces with reusable skills, shifting evaluation from answers to task closure.
-
Structured Synthetic Reasoning Data for Arithmetic Fine-Tuning of Small Language Models
A 21,250-example GSM8K-derived synthetic dataset with natural-language traces, Socratic cues, and distractors improved GSM8K exact-match accuracy of LoRA-tuned Qwen3-0.6B/1.7B from 36.5/53.5% to 49.1/66.5%.
-
TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance
TaoSR-AGRL improves e-commerce search relevance by combining dense rule-aware reward shaping with adaptive ground-truth-guided replay in GRPO, reporting gains on Taobao's private offline and online evaluations.
-
When LLMs Copy to Think: Uncovering Copy-Guided Attacks in Reasoning LLMs
Hidden strings in code exploit a reasoning model's tendency to copy tokens into its own thinking, enabling output length and result manipulation.
-
SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
SCOUT combines progressive distillation with a cross-attention module to make recursive latent reasoning work through fine-tuning, yielding up to 1.8% accuracy gains over standard fine-tuning.
-
Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models
A survey of LLM hallucination research that formalizes hallucination types and argues, via incompleteness and undecidability arguments, that hallucinations cannot be fully eliminated.
Discussion (0). Continue with ORCID to comment.