REVIEW 18 cited by
DCPO: Dynamic Clipping Policy Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
DCPO: Dynamic Clipping Policy Optimization
read the original abstract
Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a promising framework for enhancing the reasoning capabilities of large language models. However, existing approaches such as GRPO often suffer from zero gradients. This problem arises primarily due to fixed clipping bounds for token-level probability ratios and the standardization of identical rewards, which can lead to ineffective gradient updates and underutilization of generated responses. In this work, we propose Dynamic Clipping Policy Optimization(DCPO), which introduces a dynamic clipping strategy that adaptively adjusts clipping bounds based on token-specific prior probabilities to enhance token-level exploration, and a smooth advantage standardization technique that standardizes rewards across cumulative training steps to improve the response-level effective utilization of generated responses. DCPO achieved state-of-the-art performance on four benchmarks based on four different models. In particular, DCPO achieved an Avg@1 of 46.7 under greedy decoding and an Avg@32 of 38.8 under 32 times sampling on the AIME24 benchmark, surpassing DAPO (36.7/31.6), GRPO (36.7/32.1) and GSPO (40.0/34.9) on the Qwen2.5-Math-7B model. On the AIME25 benchmark based on Qwen2.5-14B, DCPO achieves a performance of (23.3/19.0), surpassing GRPO (13.3/10.5), DAPO (20.0/15.3) and GSPO (16.7/9.9). Furthermore, DCPO achieved an average 28% improvement in the nonzero advantage over GRPO in four models, doubled the training efficiency over DAPO, and significantly reduced the token clipping ratio by an order of magnitude compared to both GRPO and DAPO, while achieving superior performance. These results highlight DCPO's effectiveness in leveraging generated data more efficiently for reinforcement learning in large language models.
Forward citations
Cited by 18 Pith papers
-
Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning
Replacing GRPO's fixed clipping range with a task-wise entropy-aware adaptive bound stabilizes multi-task agentic LLM training by synchronizing exploration-exploitation paces.
-
GUI-AC: Enhancing Continual Learning in GUI Agents
Grounding-certainty-guided Adaptive Advantage and Dynamic Clipping improve continual GUI grounding over RFT baselines on ScreenSpot-V1/V2/Pro.
-
Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
GRPO suffers advantage collapse on uniform-reward groups; ACR quantifies it and AVSPO adds virtual samples to restore gradients, yielding 4-6% accuracy gains on math benchmarks across 0.5B-14B models.
-
Revisiting DAgger in the Era of LLM-Agents
DAgger-style training with turn-level policy interpolation raises 4B and 8B LLM agents to 27.3% and 29.8% on SWE-bench Verified, beating several larger published systems.
-
Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
OPEFO prevents entropy collapse in RLVR by rescaling token updates according to their entropy change contributions, yielding more stable optimization and better results on math benchmarks.
-
Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance
Span-level Wasserstein distances between hidden-state distributions of correct and incorrect rollouts provide a self-supervised signal to reweight advantages in GRPO, improving fine-grained credit assignment on math a...
-
Policy Improvement Reinforcement Learning
PIRL maximizes cumulative policy improvement across iterations instead of surrogate rewards and is proven aligned with final performance; PIPO implements it via retrospective verification for stable closed-loop optimization.
-
Chart Specification: Structural Representations for Incentivizing VLM Reasoning in Chart-to-Code Generation
A 7B VLM trained with a structured chart-specification reward beats larger and commercial models on chart-to-code benchmarks using only 3K-4K training samples.
-
SSPO: Subsentence-level Policy Optimization
SSPO computes policy importance ratios at the subsentence level with entropy-adjusted clipping bounds, yielding higher average scores than GRPO and GSPO on math reasoning benchmarks with Qwen models.
-
VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation
VOLD transfers text-only LLM reasoning to a 3B VLM via teacher-trace SFT followed by GRPO plus on-policy distillation, improving visual-math accuracy over GRPO-alone and prior text-only baselines (e.g., MathVision 28....
-
Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
Sample difficulty in RLVR shows non-monotonic effects on LLM reasoning, with easy/medium problems strengthening computation and reasoning features while hard problems often yield weak or harmful signals.
-
Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
The paper shows that advantage collapse in GRPO causes training stagnation on math reasoning benchmarks and proposes AVSPO, which uses real-time monitoring to inject virtual reward samples and reduces collapse while i...
-
MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Models
MCPO fixes vanishing training signals and shrinking weights in GRPO by using a hinge-KL regularizer on mastered prompts and prioritizing majority-correct prompts, yielding higher pass@1 and pass@k on math tasks.
-
Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation
Intra-group objectives in sparse-reward RL must maintain token gradient exchangeability to enable cancellation on weak-credit tokens and stabilize training.
-
Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation
IBPO converts sparse terminal rewards into step-sensitive advantages by treating differences among sampled reasoning trajectories as counterfactual alternatives, improving RL stability and ceilings on math and code be...
-
Policy Improvement Reinforcement Learning
PIPO adds closed-loop policy-improvement feedback to RL post-training so updates that raise measured performance are reinforced and those that drop it are suppressed.
-
GUI-AC: Enhancing Continual Learning in GUI Agents
GUI-AC stabilizes RFT for non-stationary GUI data by down-weighting noisy advantages and relaxing clipping bounds via a grounding certainty term.
-
Baichuan-M4: A Clinical-Grade Medical Agent System for Continuous Care
The paper describes Baichuan-M4, a coordinated medical agent system that reports leading scores across static knowledge, dynamic consultation, long-context memory, retrieval, OCR, and multimodal tasks with a 3.3% hall...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.