REVIEW 14 cited by
SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent advances of reasoning models, exemplified by OpenAI's o1 and DeepSeek's R1, highlight the significant potential of Reinforcement Learning (RL) to enhance the reasoning capabilities of Large Language Models (LLMs). However, replicating these advancements across diverse domains remains challenging due to limited methodological transparency. In this work, we present two-Staged history-Resampling Policy Optimization (SRPO), which surpasses the performance of DeepSeek-R1-Zero-32B on the AIME24 and LiveCodeBench benchmarks. SRPO achieves this using the same base model as DeepSeek (i.e. Qwen2.5-32B), using only about 1/10 of the training steps required by DeepSeek-R1-Zero-32B, demonstrating superior efficiency. Building upon Group Relative Policy Optimization (GRPO), we introduce two key methodological innovations: (1) a two-stage cross-domain training paradigm designed to balance the development of mathematical reasoning and coding proficiency, and (2) History Resampling (HR), a technique to address ineffective samples. Our comprehensive experiments validate the effectiveness of our approach, offering valuable insights into scaling LLM reasoning capabilities across diverse tasks.
Forward citations
Cited by 14 Pith papers
-
ReCo: Reweighting GRPO Against Distributional Concentration
ReCo fixes GRPO’s collapse onto high-probability base-model answers by normalizing response weights by expected occurrence and scaling token updates by Bernoulli variance, improving large-k Pass@k on math reasoning.
-
Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
SARA uses a Bayesian stopping rule on a prompt's first few responses to abandon saturated response groups early and reallocate rollouts, matching accuracy with up to 67% fewer rollouts than dynamic sampling.
-
CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learning
CodeReasoner combines a concise execution-focused dataset, instruction tuning, and GRPO RL to make 7B/14B models match or beat GPT-4o on code reasoning benchmarks.
-
The Challenge of Teaching Reasoning to LLMs Without RL or Distillation
Twenty high-quality chain-of-thought examples from a reasoning model are enough to activate strong math reasoning in a 32B base model with lightweight fine-tuning.
-
AdapThink: Adaptive Thinking Preferences for Reasoning Language Model
AdapThink is an RL post-training framework that adaptively reduces overthinking and underthinking in reasoning language models by rewarding confidence-appropriate reasoning depth and diverse training samples.
-
Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
An online pre-rollout filter that predicts and skips zero-variance prompts from reward dynamics reduces GRPO rollout time by up to 2.4x with comparable math reasoning accuracy.
-
rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset
A large verified dataset of 418K competition code problems built by synthesis, test-input utility functions, and majority-vote output labeling improves small model code reasoning to frontier level.
-
Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
A plan-then-reason SFT plus a plan-quality reward in GRPO improves math-reasoning accuracy by small but consistent margins over GRPO and DAPO.
-
KAT-V1: Kwai-AutoThink Technical Report
KAT-V1-40B is a 40B language model that switches between deep reasoning and direct answering per query, reporting matching-or-better benchmark scores with lower token use.
-
From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
A sample-centric RLVR method (LPPO), combining hint prefixes and learning-progress weighting, improves math reasoning pass@1 by 2 to 4 points over a GRPO baseline.
-
Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs
A 2.75B-active-parameter MoE model, trained with a token-budget-stabilized RL method (C3PO), matches or exceeds several 7-8B dense reasoning models on AIME, LiveCodeBench, and GPQA benchmarks.
-
FinLMM-R1: Enhancing Financial Reasoning in LMM through Scalable Data and Reward Design
A two-stage RL framework with length, image-selection, and adversarial rewards, trained on 89,378 ASP-built financial image-question pairs, improves multimodal reasoning over LMM-R1.
-
First Return, Entropy-Eliciting Explore
FR3E improves LLM math reasoning by finding high-entropy tokens in a trajectory and doing extra rollouts from those points to shape rewards.
-
Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.
Discussion (0). Continue with ORCID to comment.