Pith. sign in

REVIEW 3 major objections 6 minor 33 references

Self-distillation lets rubric-based RL fix two hidden failure modes and train twice as fast.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 04:09 UTC pith:ZOORX4VP

load-bearing objection CriPO gives a clear on-policy fix for two real rubric-RL failure modes, but the suppressed-criteria mechanism is only qualitatively validated and the headline numbers lack runs. the 3 major comments →

arxiv 2607.18082 v3 pith:ZOORX4VP submitted 2026-07-20 cs.LG cs.AI

CriPO: Enhancing Rubric-based RL via Self-Distillation

Mingxuan Xia , Yuhang Yang , Chao Ye , Shuai Zhu , Shenzhi Yang , Guangcheng Zhu , Yuhang Zhang , Cheng Peng
show 2 more authors
Haobo Wang Siqing Wang
This is my paper
classification cs.LG cs.AI
keywords rubric-based reinforcement learningself-distillationGRPOunexplored criteriasuppressed criteriatoken-level advantageopen-ended tasksLLM post-training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that rubric-based reinforcement learning for open-ended tasks is held back by two distinct, prevalent failure modes: criteria that no sampled response satisfies (unexplored criteria) and criteria that are satisfied but receive negative or zero aggregate advantage, so their learning signal is lost (suppressed criteria). It proposes CriPO, which keeps GRPO as the stable reward-grounded backbone and adds on-policy self-distillation through two self-teachers—one injects missing criterion-specific behaviors via localized forward-KL, the other identifies and flips token-level advantages for suppressed criteria. The paper claims CriPO consistently beats plain GRPO and rubric-guided baselines on medicine and science benchmarks, reaching GRPO's final performance in about half the optimization steps. A sympathetic reader would care because this offers a way to make rubric-based RL more reliable and efficient for domains where verifiable rewards are unavailable.

Core claim

CriPO's central claim is that the two common failure modes of rubric-based RL—unexplored criteria (UC) and suppressed criteria (SC)—can be fixed without introducing a train-inference mismatch by using on-policy self-distillation. For UC, a criterion-injection self-teacher revises the best rollout conditioned on the unmet criteria, and a forward-KL loss is applied only on tokens that contribute most of the divergence. For SC, a counterfactual self-teacher is asked to weaken the satisfied criteria, and token-level probability shifts localize the criterion-relevant spans in negative-advantage rollouts; these tokens get their advantages flipped to a small positive value within the GRPO loss. The

What carries the argument

The core mechanism is a pair of on-policy self-teachers built from the policy itself. For unexplored criteria, a criterion-injection self-teacher (the same model given the previous best response plus the unmet criteria) produces revised distributions, and a cumulative-KL token filter restricts distillation to the informative positions. For suppressed criteria, a counterfactual self-teacher (the same model told to weaken the satisfied criteria) is compared with the student; token positions where the original token's log-probability drops and the teacher confidently prefers an alternative are selected as criterion-relevant, and the GRPO advantage at those tokens is flipped from negative to a s

Load-bearing premise

The suppressed-criteria branch assumes that the counterfactual self-teacher's token-level probability drops reliably identify the tokens encoding the criterion; if that localization is noisy, flipping advantages to positive values reinforces arbitrary or incidental tokens rather than the desired behavior.

What would settle it

A ground-truth evaluation of token localization: annotate the criterion-relevant spans in a set of negative-advantage rollouts, then measure precision/recall of the tokens selected by the counterfactual teacher (Eq. 8). If the selected tokens frequently fall outside the annotated spans, the advantage-flipping benefit should disappear or reverse when compared with random token selection in a controlled experiment.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the claims hold, rubric-based RL can be made more sample-efficient, hitting the same performance in roughly half the optimization steps, which directly cuts training compute.
  • The two-module design suggests that unexplored and suppressed criteria are separable failure modes, so future rubric-RL methods can target each independently.
  • The localized forward-KL approach may generalize beyond rubrics: any sparse, criterion-like feedback could be injected via self-teaching without off-policy rollouts.
  • The counterfactual advantage-flipping mechanism could be applied to any scalar-reward RL where certain desirable behaviors are masked by aggregate rewards, not just rubric-based settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The suppressed-criteria token localization is the most fragile component; the paper only tests random-token selection and one qualitative case, not a ground-truth measure of whether the located tokens truly encode the criterion, so the benefit of CriPO-S could come from indiscriminate positive pressure rather than precise localization.
  • Editorial inference: The 2x step-speedup claim could be sensitive to the training budget and judge model; the paper uses 200 steps and GPT-4o-mini for evaluation, so a longer-horizon or different-judge setting might change the efficiency comparison.
  • Editorial inference: The method's reliance on a separate judge model (Qwen3-32B) for training means the reported improvements may partly reflect the judge's biases; using a different judge during training might shrink the gap to GRPO.
  • Editorial inference: A natural testable extension is to apply CriPO to other open-ended domains such as long-form writing or dialogue, where rubric criteria are equally common and the same failure modes likely appear.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper identifies two failure modes in rubric-based GRPO for open-ended LLM tasks: Unexplored Criteria (UC), which receive no optimization signal because no rollout satisfies them, and Suppressed Criteria (SC), where criterion-satisfying rollouts receive non-positive aggregate advantages due to scalar reward aggregation. The authors propose CriPO, which keeps GRPO as the backbone and adds two on-policy self-distillation corrections: (1) a criterion-injection self-teacher that distills missing UC behaviors into the highest-advantage rollout via localized forward-KL, and (2) a counterfactual self-teacher that localizes SC-relevant tokens in negative-advantage rollouts (Eq. 8) and flips their advantages to a small positive value τ_flip. Experiments on RaR-Medicine/RaR-Science and cross-domain benchmarks with Qwen3-1.7B/4B claim consistent improvements over GRPO and HeRL, with roughly 2× fewer optimization steps to reach GRPO's converged performance.

Significance. If the empirical claims hold, CriPO is a useful contribution: it offers a principled, on-policy way to address two real limitations of rubric-based GRPO without the train-inference mismatch of external-guidance rollouts. The problem formulation is clear, the failure-mode statistics (57% SC, 1.8 per sample) are informative, and the ablations show that both the token-filtering and the token-localization choices matter. The paper also includes cross-judge evaluation (GPT-4o-mini vs. Qwen3-32B) and an OOD instruction-following check, which strengthen confidence in the general claims. However, the central empirical support is weakened by the absence of multiple seeds, standard deviations, and significance tests, and the SC branch's mechanism is validated only by a single qualitative case and a random-token ablation, not by a direct measure of token-localization accuracy.

major comments (3)
  1. [§4.2, Table 1; §4.4, Table 2; §A.2, Table 4] All reported accuracies are from single training runs, with no standard deviations, no multiple seeds, and no significance tests. The main claim 'CriPO improves the average score from 59.2 to 62.4' (+3.2 on Qwen3-1.7B) and the SC-specific gain on Qwen3-4B (+0.8 over CriPO-U, 69.6 vs. 68.8) could be run-to-run noise under typical RL training variance in this setting. The '2× fewer steps' claim in Figure 5 is also based on one trajectory. Please report mean±std over at least 3 seeds, or provide a clear justification for why single-run comparisons are sufficient here (e.g., low variance observed in practice).
  2. [Eq. (8), §3.2; §4.3 'Case Study'; §4.4 Table 2] The load-bearing premise of the SC branch is that the heuristic in Eq. (8)—tokens where Δ>0 and p_T(y_t) < α·max_v p_T(v)—correctly locates tokens that encode the suppressed criterion. The evidence is one qualitative case (Figure 8) and a random-token ablation (Table 2). The random-token ablation only shows that nonrandom selection matters; it does not show that the selected tokens are criterion-relevant. A heuristic selecting high-entropy tokens, or tokens where the teacher confidently prefers any alternative, could also beat random without isolating the target criterion. Moreover, Appendix B's observation that boundary tokens have large KL and must be masked (Figure 10) indicates that the teacher distribution is contaminated by prompt-revision artifacts; the same contamination could affect the counterfactual comparison in Eq. (8). Please provide a quantitative localization evaluation,
  3. [§4.4 Table 2; Appendix B (β values)] The method introduces several free hyperparameters (γ=0.95, α=0.1, τ_flip=0.1, β=0.05/0.03, K=3, boundary mask fraction). The paper reports no sensitivity analysis for α, τ_flip, or β, and the β values differ between model sizes with only a one-line mention. Since the SC branch's advantage flipping value τ_flip directly controls how strongly the localized tokens are reinforced, and the UC branch's loss weight β controls the distillation strength, the absence of any robustness check leaves the reader unable to judge whether the reported gains are stable or cherry-picked. Please add at least a small sensitivity sweep for the most critical hyperparameters.
minor comments (6)
  1. [Figure 1 title] The left panel's legend reads 'Unexplorable Criteria' but the text and main body use 'Unexplored Criteria' (and 'Suppressed Critieria' has a typo). Please unify terminology.
  2. [Figure 8 caption] The caption text is garbled: 'The teacher primarily shifts its prediction at the token “Avoid”, correctly attributing the blind criterion to the relevant span in the rollout.' The word 'blind' should be 'suppressed' (or similar), and the sentence flow is broken. Please fix.
  3. [Appendix B, token masking] The first/last 1% token masking is an important design choice that affects the KL-based token selection, but it is not mentioned in the main text's description of Eq. (6) or Eq. (8). Please state this explicitly in Section 3, since it changes the definition of T_u and T_s_i.
  4. [§3.1, 'Best-Rollout Selection'] The choice to apply UC behavior injection only to the highest-advantage rollout is justified conceptually, but the ablation in Table 2 shows a small drop when removing it (66.4→65.6). The text says this 'suggests' the benefit is real, but the magnitude is close to the single-run noise level. Please temper the wording or add seed variance.
  5. [§2, Eq. (4)] The zero-advantage suppression case (ΣA_i = 0 ∧ |S_j| < G/2) is defined but never explicitly analyzed in the experiments. Figure 7 reports total SC counts but does not separate negative-advantage from zero-advantage suppression. Reporting this breakdown would make the failure-mode analysis more precise.
  6. [Appendix A.1, Table 3] OOD instruction-following results show small fluctuations on individual benchmarks (e.g., -0.6 on IFEval for Qwen3-4B CriPO-RM). The conclusion 'does not substantially degrade' is fair, but the differences are within what single-run noise would produce; please avoid over-interpreting these numbers.

Circularity Check

0 steps flagged

No significant circularity: CriPO is an empirical method evaluated against external benchmarks; no derived quantity reduces to its inputs by construction.

full rationale

This paper is an empirical methods paper rather than a formal derivation, and its central claims are benchmark evaluations. The definitions of unexplored and suppressed criteria (Eqs. 3-4) are descriptive failure-mode characterizations, not quantities fitted to the outcome and then renamed as predictions. The two CriPO interventions—forward-KL behavior injection and counterfactual token-level advantage flipping—are explicitly designed to address those failure modes, but whether they improve final performance is tested on held-out medicine and science benchmarks, including a second judge model, cross-domain sets, and ablations. The self-teacher is the same policy, so the supervision is self-referential in mechanism, which raises a soundness question about whether Eq. 8 accurately localizes criterion-relevant tokens; however, this is an empirical validation concern, not a logical circularity. Nothing in the paper equates the prediction metric to the training signal by construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no renamed known result. The lack of error bars and the qualitative validation of token localization are experimental robustness issues, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 2 invented entities

The central claims rest on the reliability of the rubric judge, the behavioral fidelity of the self-teacher prompts, and the heuristic validity of locally flipping GRPO advantages. The method-specific constants (γ, α, τ_flip, β, K, boundary masks) are hand-set rather than derived and are not accompanied by sensitivity analyses. No new physical entities or external falsifiable handles are introduced beyond the two self-teacher constructs, which are only internally validated.

free parameters (6)
  • γ (cumulative KL threshold) = 0.95
    Controls which tokens are included in the behavior-injection OPSD loss (Eq. 6). Chosen by hand; no sensitivity analysis is reported.
  • α (teacher confidence threshold) = 0.1
    Controls whether the counterfactual teacher 'actively replaced' the sampled token in Eq. (8). Chosen by hand; the paper shows no ablation over α.
  • τ_flip (flipped advantage value) = 0.1
    The positive constant substituted for negative rollout advantages at localized tokens in Eq. (9). No theoretical or empirical justification for this magnitude.
  • β (OPSD loss weight) = 0.05 for Qwen3-1.7B, 0.03 for Qwen3-4B
    Trade-off between GRPO and OPSD losses in Eq. (11); model-size-dependent values were chosen without reported tuning curves.
  • K (max criteria in teacher prompt) = 3
    When more than three UC/SC criteria exist, only top-K by weight are used to construct teacher prompts. Hand-set in Appendix B.
  • boundary mask fraction = first and last 1% of response tokens
    Tokens near both ends are masked before token filtering to reduce boundary noise. This is an additional hand-set engineering choice.
axioms (4)
  • domain assumption The rubric judge (Qwen3-32B) produces reliable binary criterion scores; UC/SC statistics and rewards rest on this.
    The natural-language judge prompt in Figure 11 is assumed to faithfully map responses to PRESENT/NOT_PRESENT for each criterion. If judge errors are correlated with the failure modes, the measured UC/SC statistics and rewards are distorted.
  • ad hoc to paper Conditioning the policy on 'revise to satisfy missing criteria' prompts produces token-level distributions that reflect the missing (or covered) criterion behavior.
    Eqs. (7) and (8) assume the self-teacher's next-token shifts are meaningful supervision signals. No ground-truth annotation or controlled study demonstrates that the teacher revisions actually satisfy the intended criteria rather than just changing phrasing.
  • ad hoc to paper Locally flipping token advantages inside the GRPO clipped surrogate objective is a valid optimization direction.
    Eqs. (9)–(10) replace rollout-level advantages with token-level positives at selected positions. This is a heuristic intervention; the paper provides no proof that such local flips preserve the policy-improvement properties of GRPO.
  • domain assumption Group-relative normalization of aggregated rubric rewards is an appropriate backbone for open-ended QA.
    The paper inherits GRPO's assumption that within-prompt group normalization yields useful learning signals for long, open-ended responses, despite the reward ambiguity it identifies.
invented entities (2)
  • Criterion-injection self-teacher distribution p_T^t = πθ(· | x, C_u, y*_<t) no independent evidence
    purpose: Provides token-level target distributions for injecting unexplored-criterion behaviors into the student policy.
    The paper does not verify that the teacher's revised distributions actually encode the missing criterion; only downstream benchmark scores are measured.
  • Counterfactual self-teacher distribution p_T^t = πθ(· | C_s, y_i,<t) no independent evidence
    purpose: Used to locate criterion-relevant tokens through probability drops after the criterion is removed.
    No ground-truth token localization labels are provided. Validation is limited to a random-token ablation and one qualitative medical case.

pith-pipeline@v1.3.0-alltime-deepseek · 18548 in / 13443 out tokens · 163747 ms · 2026-08-04T04:09:51.703483+00:00 · methodology

0 comments
read the original abstract

Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (Unexplored Criteria, UC) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout, yet they introduce a train-inference mismatch: the policy is optimized on rollouts produced under external guidance while this guidance is absent at inference time, causing error accumulation through autoregressive decoding. Moreover, these exploration-focused approaches overlook a fundamentally different failure mode that we term Suppressed Criteria (SC) -- criteria that are satisfied by some rollouts yet whose learning signals are lost during optimization because scalar reward aggregation assigns them non-positive aggregate advantages. Our analysis reveals that SC are remarkably prevalent: over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 SC per sample. To simultaneously address both UC and SC without introducing training-inference mismatch, we propose Criterion-Distilled Policy Optimization (CriPO), which enhances rubric-based RL via on-policy self-distillation. For UC, CriPO constructs a criterion-injection self-teacher and computes a localized forward-KL loss to inject missing behaviors into the policy. For SC, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values, preserving useful patterns that would otherwise be suppressed. Experiments on medicine and science benchmarks demonstrate that CriPO consistently outperforms rubric-based RL, achieving stronger final performance with approximately $2\times$ fewer optimization steps.

Figures

Figures reproduced from arXiv: 2607.18082 by Chao Ye, Cheng Peng, Guangcheng Zhu, Haobo Wang, Mingxuan Xia, Shenzhi Yang, Shuai Zhu, Siqing Wang, Yuhang Yang, Yuhang Zhang.

Figure 1
Figure 1. Figure 1: Left: Two failure modes in GRPO: Unexplored Criteria (UC), which are never satisfied by any rollout in the group (i.e., C4), and Suppressed Criteria (SC), for which criterion-satisfying rollouts receive non-positive aggregate advantages (i.e., C3). Right: The ratio of samples with UC/SC and the average number of UC/SC per sample when training Qwen3-4B on RaR-Medicine, showing both UC and SC are prevalent a… view at source ↗
Figure 2
Figure 2. Figure 2: Reward dynamics when train￾ing Qwen3-4B on RaR-Medicine using OPSD alone. Different KL variants all result in performance degradation. In rubric-based RL, rubric criteria can naturally serve as the privileged information ξ. By conditioning the self￾teacher on rubrics or criteria feedback, OPSD can expose criterion-level supervision, which appears to offer a uni￾fied solution to the two limitations of GRPO … view at source ↗
Figure 3
Figure 3. Figure 3: Overview of CriPO. CriPO first identifies unexplored and suppressed criteria from rubric￾evaluated rollouts. For unexplored criteria, CriPO injects the missing criterion-specific information through OPSD Loss. For suppressed criteria, CriPO compares the student with a counterfactual teacher to locate criterion-relevant tokens, and then flips their advantages to modify GRPO Loss. The two objectives are comb… view at source ↗
Figure 4
Figure 4. Figure 4: Cumulative KL contribution across tokens sorted by magnitude. Only 34.6% of tokens contribute 95% of the total KL. • Contribution-Guided Token Filtering. Since the self-teacher’s revision is localized—only a few token positions are actually modified to satisfy the missing criteria—the teacher distribution shifts substantially only at those positions. The remaining tokens ex￾hibit small KL values that prima… view at source ↗
Figure 5
Figure 5. Figure 5: Compute efficiency of CriPO. Left: CriPO surpasses the best GRPO performance at step 175 with about 2.0× fewer optimization steps. Right: CriPO reaches better-than-GRPO performance before completing its full training budget across model scales and domains. 40 80 120 160 200 Training Step 0.48 0.51 0.54 0.57 0.60 Reward 40 80 120 160 200 Training Step 0.16 0.24 0.32 0.40 Entropy 40 80 120 160 200 Training S… view at source ↗
Figure 6
Figure 6. Figure 6: Training dynamics of CriPO. We compare CriPO and GRPO across reward, entropy, and response length over 200 training steps. CriPO achieves higher rewards, maintains greater entropy for exploration, and produces longer responses that better support rubric-relevant content. domains: despite the extra per-step computation, CriPO surpasses GRPO before GRPO training completes. These results show that CriPO trade… view at source ↗
Figure 7
Figure 7. Figure 7: Statistics of Unexplored and Suppressed Criteria. We report their occurrence ratio and average count across methods. CriPO reduces both types of criteria compared with GRPO, while CriPO-U and CriPO-S show targeted effects on their corresponding failure modes. A 35-week pregnant woman with painless vaginal bleeding is most likely to have placenta previa . Key evidence : \n - Painless third-trimester bleedin… view at source ↗
Figure 8
Figure 8. Figure 8: Case Study of Token Selection for Suppressed Criteria. The counterfactual self-teacher correctly identifies the token “Avoid” as criterion-relevant for localized advantage correction. is most effective at reducing suppressed criteria. These results verify that each component targets its intended failure mode, and their combination further improves overall performance. Case Study of Token Selection for Supp… view at source ↗
Figure 9
Figure 9. Figure 9: Case study on medical reasoning. CriPO produces a more complete reasoning process than GRPO by exploring rubric-relevant clinical cues and diagnostic rationales, suggesting that its longer responses reflect deeper reasoning rather than superficial length expansion. B Detailed Experimental Settings Dataset Details. For RaR-Medicine and RaR-Science [Gunjal et al., 2025], we first filter out overly easy sampl… view at source ↗
Figure 11
Figure 11. Figure 11: Detailed training configurations of CriPO and all baselines are provided in Table 5. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 10
Figure 10. Figure 10: Token-wise forward KL across positions. Forward KL is higher near response bound￾aries, so we mask the first and last 1% of tokens to reduce boundary noise during localized OPSD. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Rubric judge prompt. The judge evaluates model responses according to predefined criteria and produces criterion-wise scores for reward computation. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 23 linked inside Pith

  1. [4]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. CoRR, abs/2501.12948,

    Pith/arXiv arXiv

  2. [5]

    Deepresearch bench: A comprehensive benchmark for deep research agents.CoRR, abs/2506.11763,

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents.CoRR, abs/2506.11763,

    Pith/arXiv arXiv

  3. [6]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. A survey on llm-as-a-judge. CoRR, abs/2411.15594, 2024a. Siyi Gu, Jialin Chen, Sophia Zhou, Arman Cohan, and Rex Ying. Rethinking reward supervision: Rubric-conditioned self-distillation.arXiv preprint ar...

    Pith/arXiv arXiv

  4. [7]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InICLR, 2024b. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains.CoRR, abs/2507.17746,

    Pith/arXiv arXiv

  5. [8]

    Reinforcement learning via self-distillation.CoRR, abs/2601.20802,

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation.CoRR, abs/2601.20802,

    Pith/arXiv arXiv

  6. [9]

    Rebellious student: Reversing teacher signals for reasoning exploration with self-distilled RLVR.CoRR, abs/2605.10781, 2026a

    Jeonghye Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Rebellious student: Reversing teacher signals for reasoning exploration with self-distilled RLVR.CoRR, abs/2605.10781, 2026a. Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capa...

    Pith/arXiv arXiv

  7. [10]

    Unifying group-relative and self-distillation policy optimization via sample routing.CoRR, abs/2604.02288, 2026a

    12 Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. Unifying group-relative and self-distillation policy optimization via sample routing.CoRR, abs/2604.02288, 2026a. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan- ang Gao, Wenkai Yang, Z...

    arXiv

  8. [11]

    La, Duy M

    Phuc Minh Nguyen, Chinh D. La, Duy M. H. Nguyen, Nitesh V . Chawla, Binh T. Nguyen, and Khoa D. Doan. The reasoning boundary paradox: How reinforcement learning constrains language models.CoRR, abs/2510.02230,

    arXiv

  9. [12]

    GPT-4 technical report.CoRR, abs/2303.08774,

    OpenAI. GPT-4 technical report.CoRR, abs/2303.08774,

    Pith/arXiv arXiv

  10. [14]

    Rubric-guided self-distillation: Post-training without rubric verifiers.arXiv preprint arXiv:2606.12507,

    MohammadHossein Rezaei, Anas Mahmoud, Zihao Wang, Utkarsh Tyagi, Advait Gosai, Razvan- Gabriel Dumitru, Aakash Sabharwal, Bing Liu, and Yunzhong He. Rubric-guided self-distillation: Post-training without rubric verifiers.arXiv preprint arXiv:2606.12507,

    Pith/arXiv arXiv

  11. [15]

    Proximal policy optimization algorithms.CoRR, abs/1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.CoRR, abs/1707.06347,

    Pith/arXiv arXiv

  12. [17]

    Hybridflow: A flexible and efficient RLHF framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. InProceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 - 3 April 2025, pages 1279–1297,

    2025

  13. [18]

    A survey of on-policy distillation for large language models.CoRR, abs/2604.00626,

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.CoRR, abs/2604.00626,

    Pith/arXiv arXiv

  14. [19]

    Qwen3 technical report.CoRR, abs/2505.09388,

    Qwen Team. Qwen3 technical report.CoRR, abs/2505.09388,

    Pith/arXiv arXiv

  15. [20]

    Checklists are better than reward models for aligning language models.CoRR, abs/2507.18624,

    Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. Checklists are better than reward models for aligning language models.CoRR, abs/2507.18624,

    arXiv

  16. [21]

    TRACE: distilling where it matters via token-routed self on-policy alignment.CoRR, abs/2605.10194,

    Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Zheng Pan, Xin Li, and Lan-Zhe Guo. TRACE: distilling where it matters via token-routed self on-policy alignment.CoRR, abs/2605.10194,

    Pith/arXiv arXiv

  17. [22]

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. SWE-RL: advancing LLM reasoning via reinforcement learning on open software evolution.CoRR, abs/2502.18449,

    Pith/arXiv arXiv

  18. [23]

    Writingbench: A comprehensive benchmark for generative writing.CoRR, abs/2503.05244,

    Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. Writingbench: A comprehensive benchmark for generative writing.CoRR, abs/2503.05244,

    arXiv

  19. [24]

    Rubrics to tokens: Bridging response-level rubrics and token-level rewards in instruction following tasks.CoRR, abs/2604.02795,

    13 Tianze Xu, Yanzhao Zheng, Pengrui Lu, Lyumanshan Ye, Yong Wu, ZhenTao Zhang, YuanQiang Yu, Chao Ma, JiHuai Zhu, Pengfei Liu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, and Gang Yu. Rubrics to tokens: Bridging response-level rubrics and token-level rewards in instruction following tasks.CoRR, abs/2604.02795,

    Pith/arXiv arXiv

  20. [25]

    Self-distilled RLVR.CoRR, abs/2604.03128,

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled RLVR.CoRR, abs/2604.03128,

    Pith/arXiv arXiv

  21. [26]

    On-policy context distillation for language models.CoRR, abs/2602.12275,

    Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models.CoRR, abs/2602.12275,

    Pith/arXiv arXiv

  22. [27]

    Yifei, Allen Chang, Chaitanya Malaviya, and Mark Yatskar

    Li S. Yifei, Allen Chang, Chaitanya Malaviya, and Mark Yatskar. Researchqa: Evaluating scholarly question answering at scale across 75 fields with survey-mined questions and rubrics.CoRR, abs/2509.00496,

    arXiv

  23. [28]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? CoRR, abs/2504.13837,

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? CoRR, abs/2504.13837,

    Pith/arXiv arXiv

  24. [29]

    Experience is the best teacher: Motivating effective exploration in reinforcement learning for llms.CoRR, abs/2603.20046,

    Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi, Baisheng Lai, and Jianqiang Huang. Experience is the best teacher: Motivating effective exploration in reinforcement learning for llms.CoRR, abs/2603.20046,

    arXiv

  25. [30]

    Self- distilled reasoner: On-policy self-distillation for large language models.CoRR, abs/2601.18734,

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self- distilled reasoner: On-policy self-distillation for large language models.CoRR, abs/2601.18734,

    Pith/arXiv arXiv

  26. [31]

    Breaking the explo- ration bottleneck: Rubric-scaffolded reinforcement learning for general LLM reasoning.CoRR, abs/2508.16949,

    Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Jiale Zhao, Jingwen Yang, Jianwei Lv, Kongcheng Zhang, Yihe Zhou, Hengtong Lu, Wei Chen, Yan Xie, and Mingli Song. Breaking the explo- ration bottleneck: Rubric-scaffolded reinforcement learning for general LLM reasoning.CoRR, abs/2508.16949,

    Pith/arXiv arXiv

  27. [32]

    The many faces of on-policy distillation: Pitfalls, mechanisms, and fixes.CoRR, abs/2605.11182,

    Siqi Zhu, Xuyan Ye, Hongyu Lu, Weiye Shi, and Ge Liu. The many faces of on-policy distillation: Pitfalls, mechanisms, and fixes.CoRR, abs/2605.11182,

    Pith/arXiv arXiv

  28. [33]

    Table 3:Supplementary OOD evaluation.We report instruction-following performance on IFE- val, IFBench, and MulDimIF

    A Supplementary Experiments A.1 OOD Generalization on Instruction Following. Table 3:Supplementary OOD evaluation.We report instruction-following performance on IFE- val, IFBench, and MulDimIF. CriPO-RM and CriPO-RS are trained on RaR-Medicine and RaR-Science, respectively. Model IFEval IFBench MulDimIF Qwen3-1.7B 69.7 18.7 14.9 + CriPO-RM 69.5↓0.2 19.0↑0...

    2000

  29. [2017]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.CoRR, abs/2402.03300,

    Pith/arXiv arXiv

  30. [2023]

    Rlcsd: Reinforcement learning with contrastive on-policy self-distillation.arXiv preprint arXiv:2606.11709,

    Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, and Lijie Wen. Rlcsd: Reinforcement learning with contrastive on-policy self-distillation.arXiv preprint arXiv:2606.11709,

    Pith/arXiv arXiv

  31. [2024]

    Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Healthbench: Evaluating large language models towards improved human health.CoRR, abs/2505.08775,

    Pith/arXiv arXiv

  32. [2025]

    Rucl: Strati- fied rubric-based curriculum learning for multimodal large language model reasoning.CoRR, abs/2602.21628,

    Yukun Chen, Jiaming Li, Longze Chen, Ze Gong, Jingpeng Li, Zhen Qin, Hengyu Chang, Ancheng Xu, Zhihao Yang, Hamid Alinejad-Rokny, Qiang Qu, Bo Zheng, and Min Yang. Rucl: Strati- fied rubric-based curriculum learning for multimodal large language model reasoning.CoRR, abs/2602.21628,

    arXiv

  33. [2026]

    Process reinforcement through implicit rewards.Trans

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards.Trans. Mach...

    2026