Pith. sign in

REVIEW 5 major objections 5 minor 18 cited by

GRPO-style RLVR wastes most generated data through fixed clipping bounds and per-step reward standardization; DCPO replaces both, nearly doubling response utilization and lifting AIME24 to 38.8.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 11:35 UTC pith:GLFHKLWW

load-bearing objection Genuinely new mechanisms and a plausible efficiency story, but the headline SOTA claim rests on 30-problem benchmarks without error bars and should not be taken at face value. the 5 major comments →

arxiv 2509.02333 v2 pith:GLFHKLWW submitted 2025-09-02 cs.CL cs.AIcs.LG

DCPO: Dynamic Clipping Policy Optimization

Shihui Yang , Chengfeng Dou , Peidong Guo , Kai Lu , Qiang Ju , Fei Deng , Rihui Xin This is my paper
classification cs.CL cs.AIcs.LG
keywords reinforcement learning from verifiable rewardsGRPOdynamic clippingadvantage standardizationsample efficiencymathematical reasoninglarge language modelsresponse utilization ratio
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Reinforcement learning from verifiable rewards (RLVR) trains reasoning models, but the standard GRPO update wastes data: fixed ratio-clipping bounds and step-level reward standardization routinely send the gradient to zero, so more than half of all generated responses never contribute to learning. DCPO claims to fix both defects. Its dynamic clipping rule widens the allowed update interval for low-probability tokens, letting the model explore rare but informative tokens, and its smooth advantage standardization mixes each step's reward statistics with statistics accumulated over all earlier responses, so responses with tied rewards still produce a learning signal. On four benchmarks across four Qwen models, the paper reports that this raises the response utilization ratio by about 28 percentage points (to roughly 72%), cuts the token clipping ratio by an order of magnitude, halves training time relative to DAPO, and improves competition-math accuracy — AIME24 Avg@32 reaches 38.8 with Qwen2.5-Math-7B versus 32.1 for GRPO. If these results hold, DCPO is the most sample-efficient of the compared RLVR methods while also improving performance on hard competition math.

Core claim

The central claim is that GRPO-family RLVR suffers two self-inflicted gradient deaths — fixed symmetric clipping of the probability ratio and standardization of rewards within a single sampling step — and that both can be removed without destabilizing training. DCPO replaces |r-1| ≤ ε with a probability-weighted constraint |(r-1)·p| ≤ ε, whose closed-form solution yields per-token clipping bounds that widen as the old probability q(x) shrinks; tokens with prior probability below about 1/(1+ε) get a much larger admissible ratio, capped at r_max = 10. It then standardizes advantages using a smoothed blend of the current-step and cumulative reward distributions, and averages loss over tokens wi

What carries the argument

Two mechanisms carry the argument. Dynamic adaptive clipping (DAC): instead of a fixed interval around ratio 1, the bound solves |(r(x)-1)p(x)| ≤ ε, giving a closed-form per-token interval whose width grows as 1/√q(x), so rare tokens get a wider admissible update range while common tokens keep GRPO's tight interval; the ratio is hard-capped at 10 following the dual-clipping idea. Smooth advantage standardization (SAS): per-step advantage and cumulative advantage (mean and variance of all responses to the prompt so far) are blended with weights that shift as 1/i, and the smaller-magnitude blend is used, so a step where all rewards tie still contributes a signal of (1/i)·A_total. A third compo

Load-bearing premise

For a fixed problem, the distribution of rewards over sampled responses is treated as unchanged across all training steps, so means and variances collected from every past response still describe the responses generated today; if the model's behavior shifts that distribution, the cumulative statistics mis-scale current advantages.

What would settle it

Measure whether the SAS premise holds: pick a set of prompts and, at each training step, record the current-step reward mean (μ_new) and the cumulative reward mean (μ_total) for that prompt. If the standardized gap (μ_new − μ_total)/σ_total grows systematically beyond sampling noise in the second half of training, the reward distribution is drifting and the cumulative statistics mis-calibrate current advantages. The decisive experiment is to ablate SAS with a rolling window (e.g., the last 50 steps of rewards instead of all history): if rolling-window standardization beats full-history on AIME

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • RLVR training becomes markedly cheaper per useful update: roughly 72% of generated responses carry a nonzero learning signal instead of 44%, and DAPO's dynamic sampling filter — which discards tied-reward responses and regenerates — becomes unnecessary, cutting training time roughly in half.
  • Under sampling-based decoding (Avg@32), competition-math accuracy improves most: AIME24 goes from 32.1 (GRPO) to 38.8 on the 7B model, and AIME25 from 10.5 to 19.0 on the 14B model, suggesting the trained policy's sampled distribution, not just its greedy output, contains more correct solutions.
  • The token clipping ratio falls by an order of magnitude, meaning nearly the whole response contributes to each update; combined with the reported entropy curves, DCPO keeps policy entropy in a middle band rather than collapsing (GRPO) or fluctuating heavily (DAPO).
  • Ablations show each component helps alone: OTM, SAS, and DAC each beat GRPO on average Avg@32 when swapped in individually, and their combination beats all baselines, so the gains are additive rather than resting on a single trick.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The stationarity premise behind SAS — that a prompt's reward distribution is the same at every training step — is directly testable: if reward distributions drift as the policy sharpens (which the paper's own entropy and clipping curves hint at), a rolling-window version of the cumulative statistics should outperform the full-history version; that experiment would show how far the assumption can b
  • The dynamic clipping rule encodes a concrete claim about the loss landscape: gradients from rare, high-entropy tokens are the valuable ones. If that is right, the same probability-dependent widening should transfer beyond math to code generation and other RLVR domains where outcome rewards are sparse and within-step ties are common.
  • The 28-point response-utilization gain implies DCPO's advantages estimate a different baseline object than GRPO's — a per-prompt historical baseline rather than a within-batch baseline — so a natural check is whether the smoothed advantages still behave as an unbiased variance-reduction baseline when responses come from rapidly changing policies.
  • The largest benchmark gains cluster on AIME (hard competition problems) rather than MATH500; if the mechanism is genuinely about rare-token exploration, the method should show smaller gains on easy items and larger gains on hard subsets of any benchmark — a prediction the paper's data supports but does not isolate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DCPO (Dynamic Clipping Policy Optimization) for RLVR. It combines three modifications to GRPO: (i) token-level dynamic-adaptive clipping bounds derived from an importance-sampling variance argument (Section 2), (ii) Smooth Advantage Standardization (SAS) that mixes per-step and cumulative reward standardization (Section 3), and (iii) an 'Only Token Mean' loss that averages the policy loss over tokens within a response but not over responses. The authors report evaluations on MATH500, AMC23, AIME24, and AIME25 with four Qwen models, comparing against GRPO, DAPO, and GSPO. The paper claims state-of-the-art performance on the four benchmarks, plus a 28 pp higher response utilization ratio than GRPO, roughly doubled training efficiency over DAPO, and an order-of-magnitude lower token clipping ratio.

Significance. If the empirical claims held, DCPO would be a useful and broadly applicable contribution to RLVR: the underlying ideas—probability-dependent clipping and cumulative standardization—are plausible, and the paper provides a public repository, ablations, and a detailed appendix. The method is simple enough to adopt in existing GRPO-style frameworks. However, the current manuscript does not substantiate the headline claims. The reported benchmark gains on AIME24/AIME25 are within sampling noise for 30-problem sets, the 'state-of-the-art' wording is contradicted by several rows of Table 1, and the RUR improvement is largely a direct algebraic consequence of the SAS formula rather than an independent empirical finding. The theoretical derivation of the dynamic clipping bounds is also not fully sound as written. These issues are fixable in principle, but they are load-bearing for the paper's central claims.

major comments (5)
  1. [§4, Table 1, Abstract] The claim of 'state-of-the-art performance on four benchmarks based on four models' is not supported by Table 1. DCPO does not win every benchmark or every model. Examples: Qwen2.5-3B AIME24 Avg@1 is 3.3 vs 10.0 for GRPO and DAPO; Qwen2.5-Math-7B MATH500 is 82.5 vs 84.0 for GSPO; Qwen2.5-14B AMC23 Avg@32 is 79.9 vs 85.1 for DAPO and AIME24 Avg@1 is 20.0 vs 23.3 for GSPO. The strongest defensible statement is that DCPO has the best or near-best average across the four benchmarks on some models, and even that is not true for Qwen2.5-3B Avg@32, where DCPO (22.7) is below DAPO (23.1). Please qualify the claim and report a proper aggregate with error bars.
  2. [§4-5.1, Table 1 (AIME24/AIME25)] The headline accuracy gains on AIME24 and AIME25 are not shown to be statistically distinguishable from noise. These sets contain 30 problems each; Avg@32 is a per-problem proportion (fraction of 32 samples answered correctly) averaged over 30 problems. Under a clustered interpretation, the standard error of the 7.2 pp gap on AIME24-7B (38.8 vs 31.6) is roughly 8-12 pp, and the 8.5 pp gap on AIME25-14B (19.0 vs 10.5) has a similar standard error. No seeds, no per-problem confidence intervals, and no within-problem variances are reported. The abstract and Section 6 rest the central claim on these specific AIME gaps; as presented, the differences are within one standard error and cannot be taken as evidence of superiority.
  3. [§3, Eq. (6)-(7), Table 2] The reported 28 pp improvement in response utilization ratio (RUR) over GRPO is by construction. When all rewards for a prompt are identical in a step, the GRPO advantage is zero. Under SAS, if A_new = 0, Eq. (6)-(7) select a nonzero advantage of size (1/i) A_total (the paper states this explicitly). Therefore the high RUR does not independently show that generated responses are used more 'effectively' in the sense of improved learning; it is a direct consequence of the chosen smoothing. To support the sample-efficiency claim, the paper needs a metric that is not definitionally aligned with the method, e.g., accuracy per generated response, gradient signal quality, or a controlled comparison of final performance as a function of the number of training samples.
  4. [§2 and Appendix A.5, Eq. (4), (24)-(27)] The derivation of the dynamic clipping bounds is not a valid closed-form solution. The constraint |(r-1)p| ≤ ε, with p = r q, leads to the quadratic inequality -ε ≤ q(r²-r) ≤ ε. The feasible set is a union of two intervals, approximately [0, r_low] ∪ [r_high, r_high'] for q ≥ 4ε. The paper silently selects the interval containing r = 1, namely [0.5+0.5√(1-4ε/q), 0.5+0.5√(1+4ε/q)], and does so without justification for dropping the lower interval. Appendix A.5 Eq. (25) further conflates variables ('lowge ≤ 0 ≤ r ≤ highle') and Eq. (24)-(26) contain inconsistent notation. Since dynamic clipping is one of the two core contributions, the theoretical justification must either be corrected or explicitly recharacterized as an empirically motivated heuristic.
  5. [§3 (stationarity assumption)] SAS assumes that, for a fixed prompt, 'the overall reward distribution ... can be considered as drawn from the same global distribution throughout the course of training.' This assumption is not justified and is implicitly contradicted by the paper's own observations: Figure 7 shows substantial entropy dynamics over training, and the paper discusses entropy collapse and model-dependent trajectories. If the reward distribution drifts as the policy changes, using accumulated mean/σ from all past steps can systematically mis-calibrate current advantages. Provide either theoretical conditions under which the cumulative normalization remains valid, or empirical sensitivity checks (e.g., comparing against a moving-window baseline).
minor comments (5)
  1. [§5.2, Eq. (9)] Typo: 'microm' should be 'microbatch'. Also, the definition of N and how microbatch size relates to the batch size of 512 and minibatch of 32 should be stated explicitly.
  2. [Appendix A.9] The GSPO hyperparameters are given as clipping thresholds (3e-4, 4e-4). These seem unusually small; please clarify whether these are the values used in the original GSPO paper and how they were chosen for the comparisons.
  3. [Table 2] The formatting of Table 2 is broken: values and model names run together. Please reformat and include the omitted DAPO row or state clearly that DAPO is excluded because its dynamic sampling makes RUR incomparable.
  4. [Figure 4] The caption and axis labels are mislabeled: panel (b) says r(x) bounds for q∈[0,1], but the x-axis is q; the text '∅' in the y-axis is unclear. Please redraw the figure with proper labels and consistent notation.
  5. [§5.4, Figure 3] The ablation curves are presented without error bars or repeated seeds. Given that the final performance differences are small on several benchmarks, the claim that 'each component contributes positively' and 'their combination yields substantial gains' would be more convincing with standard deviations or at least two seeds.

Circularity Check

1 steps flagged

RUR improvement is by construction; benchmark accuracy comparisons remain independent evidence.

specific steps
  1. self definitional [Section 3 (Eqs. 6-7), Section 5.3 (Eq. 10, Table 2)]
    "Once the prompt participates in the model optimization, the responses generated in the subsequent steps will participate in the model optimization regardless of whether the current advantage is 0 or not. Consequently, when identical rewards occur within a step, these responses will also update the model with the advantage of 1/i Âi total,j, preserving useful learning signals and improving data efficiency. ... RUR= Number of responses with non-zero advantage / Total number of generated responses ×100%"

    RUR is defined as the proportion of responses with nonzero advantage (Eq. 10). The SAS advantage in Eqs. (6)-(7) is constructed so that, when all rewards at one step are equal, the final advantage is (1/i) A_total, which is nonzero whenever the cumulative statistic is nonzero. The paper states this explicitly. Consequently, the reported 28-point RUR increase over GRPO (Table 2) follows from the definition of the advantage and the RUR metric, rather than being an independently measured property of sample efficiency. The claim is a designed consequence of the method, not an empirical discovery.

full rationale

DCPO's benchmark accuracy results (AIME24/AIME25 Avg@32, etc.) are compared against external baselines and are not circular: they do not follow from the paper's equations alone. The dynamic clipping derivation is a mathematical consequence of the proposed |(r-1)p(x)| ≤ ε constraint, and the paper does not rely on self-citations to force its main accuracy conclusions. However, one prominent efficiency claim is circular by construction: the Response Utilization Ratio is defined as the fraction of responses with nonzero advantage, and DCPO's smoothed advantage is specifically designed to give identical-reward responses a nonzero (1/i) A_total advantage. Hence the 28% RUR improvement over GRPO is guaranteed by the definition of the advantage and the metric, not by an independent empirical mechanism. The token clipping ratio reduction is explained by the concentration of tokens at high q(x) and is not a definitional identity, so it does not rise to the same level of circularity. Overall, the central accuracy contribution remains independent, but the headline RUR efficiency gain reduces to a definitional artifact, warranting a partial circularity score of 6.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The method rests on two new mechanisms (dynamic clipping and cumulative advantage standardization) plus a modified loss. The dynamic clipping derivation introduces an ad hoc constraint and selects a non-unique interval; the cumulative standardization assumes stationary per-prompt reward statistics; and the entropy-driven exploration rationale relies on a prior empirical claim.

free parameters (3)
  • epsilon_low = 0.16
    Lower clipping threshold for dynamic bounds; hand-set to match GRPO's fixed clipping at the (q=1, r=1-epsilon) point (Appendix A.6).
  • epsilon_high = 0.2
    Upper clipping threshold; hand-set to match GRPO at (q=1/(1+epsilon), r=1+epsilon) point.
  • r_max = 10
    Hard cap on the probability ratio for both positive and negative advantages, inspired by dual clipping (Ye et al. 2020); chosen by the authors.
axioms (4)
  • domain assumption The reward distribution for a fixed prompt is stationary across training steps.
    Section 3 states this to justify cumulative standardization; no evidence provided that rewards do not drift as the policy changes.
  • ad hoc to paper The variance inflation of importance sampling is best controlled by bounding |(r(x)-1)p(x)| <= epsilon.
    Eq (3) introduces this constraint without derivation; the final clipping bound is one of several feasible intervals from Eq (22), chosen without justification.
  • domain assumption High-entropy (low-probability) tokens are the primary drivers of reasoning capability in RLVR.
    Section 5.2 cites Wang et al. (2025) as support; this is an empirical claim about LLM training used to motivate widening clipping for rare tokens.
  • ad hoc to paper The min-absolute-value combination of smoothed advantages (Eq 7) improves stability.
    Heuristic design choice; no theoretical or empirical ablation isolating this specific selection rule.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 19809 in / 15278 out tokens · 148891 ms · 2026-08-05T11:35:24.396373+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of DCPO: Dynamic Clipping Policy Optimization." pith.science (2026). https://pith.science/paper/GLFHKLWW

@misc{pith2026250902333,
  author       = {Pith},
  title        = {Pith review of: DCPO: Dynamic Clipping Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GLFHKLWW}},
  note         = {Machine review of arXiv:2509.02333}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a promising framework for enhancing the reasoning capabilities of large language models. However, existing approaches such as GRPO often suffer from zero gradients. This problem arises primarily due to fixed clipping bounds for token-level probability ratios and the standardization of identical rewards, which can lead to ineffective gradient updates and underutilization of generated responses. In this work, we propose Dynamic Clipping Policy Optimization(DCPO), which introduces a dynamic clipping strategy that adaptively adjusts clipping bounds based on token-specific prior probabilities to enhance token-level exploration, and a smooth advantage standardization technique that standardizes rewards across cumulative training steps to improve the response-level effective utilization of generated responses. DCPO achieved state-of-the-art performance on four benchmarks based on four different models. In particular, DCPO achieved an Avg@1 of 46.7 under greedy decoding and an Avg@32 of 38.8 under 32 times sampling on the AIME24 benchmark, surpassing DAPO (36.7/31.6), GRPO (36.7/32.1) and GSPO (40.0/34.9) on the Qwen2.5-Math-7B model. On the AIME25 benchmark based on Qwen2.5-14B, DCPO achieves a performance of (23.3/19.0), surpassing GRPO (13.3/10.5), DAPO (20.0/15.3) and GSPO (16.7/9.9). Furthermore, DCPO achieved an average 28% improvement in the nonzero advantage over GRPO in four models, doubled the training efficiency over DAPO, and significantly reduced the token clipping ratio by an order of magnitude compared to both GRPO and DAPO, while achieving superior performance. These results highlight DCPO's effectiveness in leveraging generated data more efficiently for reinforcement learning in large language models.

Figures

Figures reproduced from arXiv: 2509.02333 by Chengfeng Dou, Fei Deng, Kai Lu, Peidong Guo, Qiang Ju, Rihui Xin, Shihui Yang.

Figure 1
Figure 1. Figure 1: TCR across models and methods. 5.2 THE TOKEN CLIPPING RATIO (TCR) We use the Token Clipping Ratio (TCR) as the proportion of tokens excluded from policy updates during back-propagation due to clipping. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: demonstrates that the RUR for GRPO training decreases significantly over time, regardless of the size or type of model. Notably, RUR drops sharply from above 90% to below 50%. For the Qwen2.5-Math-7B, for instance, the RUR plummets to as low as 30% after approximately 200 training steps. This finding suggests that GRPO’s reliance on current advantage standardization results in a significant fraction of zer… view at source ↗
Figure 4
Figure 4. Figure 4: Clipping bound comparisons. Lines show bounds for fixed clipping ( [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Avg@1 performance across benchmarks and outperformed GSPO on the two models where GSPO was evaluated. Furthermore, its overall performance was more stable than the baseline, without the significant performance drop seen in GRPO or the significant fluctuations seen in DAPO. This suggests that DCPO can stimulate stronger model reasoning capabilities during training, providing greater potential for the model … view at source ↗
Figure 6
Figure 6. Figure 6: Avg@32 performance across benchmarks. 0 100 200 300 400 0.0 0.5 1.0 1.5 Qwen2.5-Math-1.5B-Instruct GRPO DAPO DCPO 0 100 200 300 400 0 1 2 3 4 Qwen2.5-3B GRPO DAPO DCPO 0 100 200 300 400 0.0 0.5 1.0 1.5 Qwen2.5-Math-7B GRPO DAPO DCPO GSPO 0 100 200 300 400 0.0 0.5 1.0 1.5 2.0 Qwen2.5-14B GRPO DAPO DCPO GSPO Actor Entropy Training Steps [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The Entropy of the models during Training. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning

    cs.LG 2026-07 accept novelty 6.0

    Replacing GRPO's fixed clipping range with a task-wise entropy-aware adaptive bound stabilizes multi-task agentic LLM training by synchronizing exploration-exploitation paces.

  2. GUI-AC: Enhancing Continual Learning in GUI Agents

    cs.CV 2026-06 conditional novelty 6.0

    Grounding-certainty-guided Adaptive Advantage and Dynamic Clipping improve continual GUI grounding over RFT baselines on ScreenSpot-V1/V2/Pro.

  3. Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

    cs.LG 2026-05 unverdicted novelty 6.0

    GRPO suffers advantage collapse on uniform-reward groups; ACR quantifies it and AVSPO adds virtual samples to restore gradients, yielding 4-6% accuracy gains on math benchmarks across 0.5B-14B models.

  4. Revisiting DAgger in the Era of LLM-Agents

    cs.LG 2026-05 conditional novelty 6.0

    DAgger-style training with turn-level policy interpolation raises 4B and 8B LLM agents to 27.3% and 29.8% on SWE-bench Verified, beating several larger published systems.

  5. Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization

    cs.LG 2026-05 unverdicted novelty 6.0

    OPEFO prevents entropy collapse in RLVR by rescaling token updates according to their entropy change contributions, yielding more stable optimization and better results on math benchmarks.

  6. Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance

    cs.CL 2026-04 unverdicted novelty 6.0

    Span-level Wasserstein distances between hidden-state distributions of correct and incorrect rollouts provide a self-supervised signal to reweight advantages in GRPO, improving fine-grained credit assignment on math a...

  7. Policy Improvement Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 6.0

    PIRL maximizes cumulative policy improvement across iterations instead of surrogate rewards and is proven aligned with final performance; PIPO implements it via retrospective verification for stable closed-loop optimization.

  8. Chart Specification: Structural Representations for Incentivizing VLM Reasoning in Chart-to-Code Generation

    cs.CV 2026-02 conditional novelty 6.0

    A 7B VLM trained with a structured chart-specification reward beats larger and commercial models on chart-to-code benchmarks using only 3K-4K training samples.

  9. SSPO: Subsentence-level Policy Optimization

    cs.CL 2025-11 unverdicted novelty 6.0

    SSPO computes policy importance ratios at the subsentence level with entropy-adjusted clipping bounds, yielding higher average scores than GRPO and GSPO on math reasoning benchmarks with Qwen models.

  10. VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation

    cs.CV 2025-10 conditional novelty 6.0

    VOLD transfers text-only LLM reasoning to a 3B VLM via teacher-trace SFT followed by GRPO plus on-policy distillation, improving visual-math accuracy over GRPO-alone and prior text-only baselines (e.g., MathVision 28....

  11. Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs

    cs.AI 2026-05 unverdicted novelty 5.0

    Sample difficulty in RLVR shows non-monotonic effects on LLM reasoning, with easy/medium problems strengthening computation and reasoning features while hard problems often yield weak or harmful signals.

  12. Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

    cs.LG 2026-05 unverdicted novelty 5.0

    The paper shows that advantage collapse in GRPO causes training stagnation on math reasoning benchmarks and proposes AVSPO, which uses real-time monitoring to inject virtual reward samples and reduces collapse while i...

  13. MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Models

    cs.AI 2026-04 unverdicted novelty 5.0

    MCPO fixes vanishing training signals and shrinking weights in GRPO by using a hinge-KL regularizer on mastered prompts and prioritizing majority-correct prompts, yielding higher pass@1 and pass@k on math tasks.

  14. Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation

    cs.LG 2026-04 unverdicted novelty 5.0

    Intra-group objectives in sparse-reward RL must maintain token gradient exchangeability to enable cancellation on weak-credit tokens and stabilize training.

  15. Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation

    cs.LG 2026-04 unverdicted novelty 5.0

    IBPO converts sparse terminal rewards into step-sensitive advantages by treating differences among sampled reasoning trajectories as counterfactual alternatives, improving RL stability and ceilings on math and code be...

  16. Policy Improvement Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 5.0

    PIPO adds closed-loop policy-improvement feedback to RL post-training so updates that raise measured performance are reinforced and those that drop it are suppressed.

  17. GUI-AC: Enhancing Continual Learning in GUI Agents

    cs.CV 2026-06 unverdicted novelty 4.0

    GUI-AC stabilizes RFT for non-stationary GUI data by down-weighting noisy advantages and relaxing clipping bounds via a grounding certainty term.

  18. Baichuan-M4: A Clinical-Grade Medical Agent System for Continuous Care

    cs.AI 2026-06 unverdicted novelty 3.0

    The paper describes Baichuan-M4, a coordinated medical agent system that reports leading scores across static knowledge, dynamic consultation, long-context memory, retrieval, OCR, and multimodal tasks with a 3.3% hall...

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages · cited by 14 Pith papers

  1. [1]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261,

    Pith/arXiv arXiv

  2. [4]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    Pith/arXiv arXiv

  3. [7]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256,

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256,

    Pith/arXiv arXiv

  4. [8]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.arXiv preprint arXiv:2506.01939,

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.arXiv preprint arXiv:2506.01939,

    Pith/arXiv arXiv

  5. [9]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024a

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024a. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical repor...

    Pith/arXiv arXiv

  6. [11]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118,

    Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118,

    Pith/arXiv arXiv

  7. [12]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Pith/arXiv arXiv

  8. [13]

    Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    Pith/arXiv arXiv

  9. [14]

    θParameters of the actor model

    11 DCPO: Dynamic Clipping Policy Optimization A APPENDIX A.1 MAINSYMBOLDEFINITIONS GNumber of responses generated per prompt at each step. θParameters of the actor model. πθ(oi|q)Probability of responseo i given promptqunder parametersθ. πθold (oi|q)Probability of responseo i given promptqunder previous parametersθ old. πref Reference policy for KL diverg...

    work page 2024

  10. [15]

    Consequently, the probability ratio r(x) must also be non-negative, and the clipping hyper-parameter ϵ must be non-negative.    0≤p(x)≤1 0≤q(x)≤1 0≤r(x) 0≤ϵ (23) To ensure the validity and practical applicability of the clipping bound derived from Equation (22), we define the necessary conditions for its bounds as outlined in Equation (24).  ...

    work page 2020

  11. [2017]

    Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  12. [2020]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  13. [2022]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  14. [2024]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al

    URLhttps://arxiv.org/abs/2412.19437. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  15. [2025]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021a

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021a. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical...

    Pith/arXiv arXiv