REVIEW 7 cited by
SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) exhibit varying levels of confidence across input prompts (questions): some lead to consistent, semantically similar answers, while others yield diverse or contradictory outputs. This variation reflects LLM's uncertainty about the input prompt, a signal of how confidently the model understands a given problem. However, vanilla Group Relative Policy Optimization (GRPO) treats all prompts equally during policy updates, ignoring this important information about the model's knowledge boundaries. To address this limitation, we propose SEED-GRPO (Semantic Entropy EnhanceD GRPO), which explicitly measures LLMs' uncertainty of the input prompts semantic entropy. Semantic entropy measures the diversity of meaning in multiple generated answers given a prompt and uses this to modulate the magnitude of policy updates. This uncertainty-aware training mechanism enables dynamic adjustment of policy update magnitudes based on question uncertainty. It allows more conservative updates on high-uncertainty questions while maintaining the original learning signal on confident ones. Experimental results on five mathematical reasoning benchmarks (AIME24 56.7, AMC 68.7, MATH 83.4, Minerva 34.2, and OlympiadBench 48.0) demonstrate that SEED-GRPO achieves new state-of-the-art performance in average accuracy, validating the effectiveness of uncertainty-aware policy optimization.
Forward citations
Cited by 7 Pith papers
-
Mimic Human Cognition, Master Multi-Image Reasoning: A Meta-Action Framework for Enhanced Visual Understanding
A structured five-step reasoning template plus diverse-trajectory cold start and diversity-preserving two-stage RL lifts a 7B multimodal model to state-of-the-art multi-image reasoning on several benchmarks.
-
EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models
EMO-R3, which combines a three-step emotional reasoning prompt with a reward for the model agreeing with its own image–emotion judgments, raises visual emotion-recognition accuracy by about one point over plain GRPO.
-
AQA-TTRL: Self-Adaptation in Audio Question Answering with Test-Time Reinforcement Learning
Test-time reinforcement learning against self-generated majority-vote pseudo-labels improves audio question answering accuracy on MMAU, MMAR, and MMSU.
-
Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
EMPG re-weights policy-gradient updates by step-level token entropy, amplifying confident correct actions and muting uncertain ones, and adds a future-clarity bonus.
-
Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
S-GRPO reweights GRPO advantages by an assumed noise level p to down-weight unbalanced groups, claiming improved and more noise-robust math reasoning training.
-
EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity
EDGE-GRPO reduces advantage collapse in GRPO by injecting reference solutions into response groups and scaling advantages by policy entropy, achieving competitive math reasoning with only 1K training samples.
-
Adaptive Termination for Multi-round Parallel Reasoning: An Universal Semantic Entropy-Guided Framework
A semantic entropy-guided stopping rule for multi-round parallel LLM reasoning improves accuracy while reducing inference steps on five benchmarks.
Discussion (0). Sign in to comment.