REVIEW 13 cited by
Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent advances in reinforcement learning (RL) using numerical rewards have significantly enhanced the complex reasoning capabilities of large language models (LLMs). However, we identify three fundamental limitations of purely numerical feedback: performance plateaus, ineffective spontaneous self-reflection, and persistent failures. We show that plateaued RL models can successfully refine failed solutions when given natural language critiques. Motivated by this, we propose Critique-GRPO, an online RL framework that integrates both natural language and numerical feedback for policy optimization. This approach enables LLMs to learn simultaneously from initial responses and critique-guided refinements, effectively internalizing the exploration benefits of both stages. Extensive experiments show that Critique-GRPO outperforms all compared supervised and RL-based fine-tuning methods, achieving average Pass@1 improvements of approximately +15.0-21.6% on various Qwen models and +7.3% on Llama-3.2-3B-Instruct across eight challenging reasoning tasks. Notably, Critique-GRPO facilitates effective self-improvement through self-critiquing, achieving substantial gains over GRPO, e.g., a +16.7% Pass@1 improvement on AIME 2024. The code and models are released at: https://github.com/zhangxy-2019/critique-GRPO
Forward citations
Cited by 13 Pith papers
-
LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks
Training a language model by distilling a coach's written experiential knowledge beats training on a scalar rubric score for open-ended tasks, with better out-of-distribution transfer.
-
Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry
Local retry from the earliest erroneous turn, with pivotal credit isolation and reflection rewards, improves Pass@k and reduces interaction cost over MetaRL and GiGPO on agent and search tasks.
-
REVES: REvision and VErification--Augmented Training for Test-Time Scaling
REVES augments LLM post-training by decoupling revision and verification signals from successful multi-step trajectories, reporting +6.5 point gains on LiveCodeBench over RL baselines.
-
Credit Assignment with Resets in Language Model Reasoning
The paper introduces Random-Reset Policy Optimization (RRPO) and Self-Reset Policy Optimization (SRPO) that use resets to enable more precise credit assignment in RL for language model reasoning, with SRPO outperformi...
-
PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization
PAIR combines a hidden-state probe with an attention correction to deliver robust step-level rewards for GRPO-based optimization of multi-turn LLM agents, achieving high AUROC on contaminated trajectories at low cost.
-
CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning
CPMobius uses iterative coach-player reinforcement learning to improve mathematical reasoning in LLMs without external training data, yielding +4.9 average accuracy gains on Qwen2.5-Math-7B-Instruct.
-
AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens
Short low-perplexity tokens sampled from a judge's own next-token distribution flip binary "No" judgments to "Yes" at high rates on most tested LLM judges.
-
Trust Region On-Policy Distillation
TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.
-
RL with Learnable Textual Feedback: A Bilevel Approach
Bi-NAC frames RL with textual feedback as a Stackelberg bilevel program and reports that 2B and 6B models trained this way outperform larger GRPO baselines on MATH-500 and GPQA.
-
XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
XRPO extends GRPO with adaptive rollout allocation, in-context example seeding for unsolved prompts, and novelty-weighted advantages, reporting roughly 1-4% higher accuracy and faster convergence.
-
CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
CLPO: an online curriculum that rewrites each problem according to the model's own current accuracy and scales the KL penalty by difficulty, reporting strong but internally inconsistent gains over GRPO and DAPO.
-
TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance
TaoSR-AGRL improves e-commerce search relevance by combining dense rule-aware reward shaping with adaptive ground-truth-guided replay in GRPO, reporting gains on Taobao's private offline and online evaluations.
-
MoL-RL: Distilling Multi-Step Environmental Feedback into LLMs for Feedback-Independent Reasoning
MoL-RL combines MoL continual training on multi-step environmental feedback with GRPO post-training to improve LLM math and code reasoning without runtime feedback loops.
Discussion (0). Sign in to comment.