Pith. sign in

REVIEW 3 major objections 7 minor 73 references

ACPO improves LLM reasoning RL by assigning token credit with mode-local surrogate entropy, boosting uncertain successful steps and penalizing overconfident failures.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 04:42 UTC pith:OIRBMQKP

load-bearing objection Solid RLVR methods paper: mode-local surrogate entropy plus asymmetric pos/neg credit works better than SAPO/GTPO/DAPO on three 7–8B models; theory is local and honest, not oversold. the 3 major comments →

arxiv 2607.03126 v2 pith:OIRBMQKP submitted 2026-07-03 cs.LG cs.AI

ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy

Zijun Xie , Yuyang You , Yongzhi Li , Enlei Gong , Zeyu Chen , Quan Chen , Yanhua Cheng , Peng Jiang
show 1 more author
Yadong Mu
This is my paper
classification cs.LG cs.AI
keywords reinforcement learninglarge language modelscredit assignmentsurrogate entropyRLVRpolicy optimizationmathematical reasoningtoken-level weighting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Sparse end-of-trajectory rewards make it hard for reinforcement learning to tell which tokens in a long reasoning chain actually caused success or failure. This paper argues that a simple mode-local uncertainty score—one minus the probability of the model’s single most likely next token—can serve as a practical credit signal without the gradient mismatch that comes from optimizing full-vocabulary entropy. ACPO multiplies that score into the advantage asymmetrically: it amplifies uncertain tokens on winning rollouts and damps or redirects pressure on overconfident tokens on losing ones. The authors show the surrogate is bounded relative to true entropy under concentrated next-token distributions, and that under modal alignment and proximal updates the leading-order gradient still points the intended policy direction. On math and coding benchmarks across three base models, the method consistently beats strong outcome-based and entropy-aware baselines, suggesting denser, model-internal credit can raise both learning speed and final accuracy when only verifiers supply rewards.

Core claim

The paper claims that replacing true next-token entropy with the mode-local surrogate δ = 1 − π(v*), then using it to form the asymmetric advantages (2+δ)·A for positive trajectories and (1−δ)·A otherwise, yields a token-level credit assignment rule that improves RLVR training. Under the stated modal-alignment and proximal-update conditions this modulation preserves the intended policy-gradient direction to leading order, while empirically outperforming GRPO, DAPO, SAPO, GTPO, and 80/20 on AIME, MATH500, HMMT, BeyondAIME, and HumanEvalPro.

What carries the argument

Mode-local surrogate entropy δ_i,t(θ) = 1 − π_θ(v*_i,t), the complementary probability of the dominant next-token mode, used inside the ACPO advantage (2+δ)·A when A>0 and (1−δ)·A otherwise, so that uncertainty scales reinforcement of successful forks and reduces pressure on noisy post-error tokens while avoiding full-distribution entropy gradients.

Load-bearing premise

The direction-preserving argument assumes that the token the model actually sampled is usually the same as its single most likely token, and that policy updates stay close enough that importance weights are near one.

What would settle it

Measure modal alignment (fraction of steps where the sampled token equals the argmax) on a held-out training distribution; if it falls well below the reported ~84% while ACPO’s advantage over SAPO/DAPO vanishes or reverses under the same compute and data, the leading-order claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Adaptive Credit Policy Optimization (ACPO), a token-level credit assignment method for RL with verifiable rewards on LLM reasoning. Building on SAPO’s soft gating, ACPO reweights sequence-level advantages with a mode-local surrogate entropy δ=1−π(v*), asymmetrically: (2+δ)·Â for positive-advantage trajectories and (1−δ)·Â otherwise, so that uncertain successful steps are reinforced and overconfident failed steps are penalized. The authors derive deterministic Shannon-entropy bounds for δ, argue that under modal alignment and proximal updates the leading-order gradient preserves the intended policy-gradient direction, and report consistent gains over GRPO, DAPO, SAPO, GTPO, and 80/20 on AIME24/25, MATH500, HMMT25, BeyondAIME, and HumanEvalPro across three 7–8B base models.

Significance. If the empirical gains hold under broader evaluation, ACPO is a practical, annotation-free improvement to sparse-outcome RLVR credit assignment that is easy to drop into existing GRPO/SAPO-style pipelines. Strengths include: (i) a carefully protocolled first-error entropy analysis (Appendix A) that motivates the asymmetric design; (ii) explicit ablations of both channels, the offset form, and surrogate vs detached true entropy (Table 2, Fig. 3) plus a top-k study (Fig. 6); (iii) closed-form entropy bounds (Eqs. 13–14, Appendix B) and a scoped leading-order gradient characterization (Eqs. 16–17, Appendix C) rather than an overclaimed global guarantee. The contribution is incremental relative to concurrent entropy-aware RLVR work, but the combination of mode-local proxy, asymmetric modulation, and multi-model multi-benchmark evidence is useful for the community.

major comments (3)
  1. [Table 1, §6.2] Table 1 reports single-run mean@8 / pass@1 with no multi-seed standard errors or statistical tests, despite RLVR training being known to be high-variance. Gains are consistent across three base models, which partially mitigates this, but for a load-bearing empirical claim the paper should either report ≥3 seeds with error bars on the main averages or, at minimum, provide seed-level numbers for AIME24/25 and the overall average so that the magnitude of improvement (e.g., +2.5 on Qwen2.5-Math-7B average vs SAPO) can be assessed against run-to-run noise.
  2. [§5.2, Eqs. (16)–(17), Appendix C, Table 4] The direction-preservation claim in §5.2 / Eq. (16)–(17) is correctly scoped as a leading-order approximation under modal alignment (Table 4: 83.9% greedy) and proximal updates (w≈1), and §7 states this limitation. However, the hybrid factor M_i,t is derived under π(y)≈π(v*) and w≈1; the paper does not quantify how often training steps violate these conditions (e.g., fraction of tokens with |π(y)−π(v*)|>0.1 or |w−1|>0.5) nor how the actual gradient direction behaves on those tokens. A short empirical check—e.g., cosine similarity between the ACPO token gradient and the pure policy-gradient term stratified by alignment/off-policy bins—would make the theory–practice link load-bearing rather than only asymptotic.
  3. [Eq. (12), Table 2, §4.5] The positive-channel offset “2” in Eq. (12) is a free design constant. Table 2 shows (2+δ,1−δ) beats several alternatives, but the search is small and the constant is not derived from the bounds or the M_i,t analysis (where positivity only requires 2+δ>0 for δ∈[0,1]). Either motivate 2 from a concrete positivity/margin argument tied to Eq. (17), or report a short sensitivity sweep over the offset (e.g., {1,1.5,2,3}) so readers know the result is not tuned to a single lucky value.
minor comments (7)
  1. [Figure 2] Figure 2 y-axis labels and panel titles are hard to read in the compiled layout; Acc@aime24, Training Reward, Entropy, and Response Length should be clearly labeled per panel with shared step axis.
  2. [§4.3, Eq. (9)] Eq. (9) decomposes the true-entropy gradient mismatch clearly; a one-line statement that detaching H removes the second term but retains tail sensitivity would help readers who skip to the method.
  3. [Table 1 caption] Table 1 “Average” is an unweighted mean of heterogeneous metrics (mean@8 math + pass@1 code). State this explicitly in the caption and consider reporting math-only average separately.
  4. [§6.1] τ_pos=1.0 and τ_neg=1.05 are given in §6.1 without sensitivity; a brief note that results are stable under small τ perturbations (or a pointer to SAPO’s defaults) would reduce reproducibility friction.
  5. [Appendix A.1, Figure 1] Appendix A.1 filtering leaves 8 prompts / 91 pairs; state whether the qualitative pattern in Fig. 1b holds if the first-error set is expanded or if only automatic (non-LLM-assisted) error markers are used.
  6. [§4.1, §4.4] Minor typos / style: “introducesurrogate” missing space (§4.4); “alatent reward” (§4.1); arXiv ID and “Preprint” header are fine for review but should be cleaned for camera-ready.
  7. [§6] Code and exact training configs are not stated as released. For an empirical RL methods paper this should be clarified (public repo or supplementary configs).

Circularity Check

0 steps flagged

No significant circularity: ACPO is a design choice with characterizing theory and external benchmark tests, not a result forced by its inputs.

full rationale

The paper’s load-bearing chain is: (i) define mode-local surrogate entropy δ=1−π(v*) as a credit proxy; (ii) plug it into an asymmetric advantage reweighting (2+δ for positive advantage, 1−δ otherwise) inside a SAPO-style objective; (iii) prove deterministic entropy bounds and a leading-order gradient form under stated approximations (modal alignment, proximal updates); (iv) evaluate accuracy against held-out math/code benchmarks and ablations. None of these steps equates a claimed prediction to a fitted input by construction. δ is an explicit design definition, not a quantity fitted to the target metrics and then re-reported as a forecast. The bounds L(δ)≤H≤U(δ) and the hybrid factor M_i,t are mathematical characterizations of that design under assumptions the authors state and measure (Table 4 greedy ratio ~83.9%, w≈1); they do not tautologically force Table 1 gains. Ablations (Table 2, Fig. 3, Fig. 6) show alternative weightings and true-entropy variants perform differently, which is inconsistent with a by-construction result. Baselines (GRPO, DAPO, SAPO, GTPO, 80/20) and datasets are external; there is no load-bearing uniqueness theorem or ansatz imported from overlapping-author prior work that forbids alternatives. Standard empirical-methods structure with independent experimental content—score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 2 invented entities

Central empirical claim rests on standard RLVR machinery plus a small set of design choices (surrogate definition, asymmetric offsets, temperatures) and two modeling assumptions used for theory (modal alignment, proximal updates). No new physical entities; the invented objects are algorithmic proxies.

free parameters (4)
  • positive/negative soft-gate temperatures τpos, τneg = 1.0 / 1.05
    Set by hand to 1.0 and 1.05; control SAPO-style gating asymmetry and enter every update.
  • positive-channel offset constant (the '2' in 2+δ) = 2
    Hand-chosen positivity margin in Eq. 12; selected via ablation among several (W+,W−) forms on AIME24.
  • actor learning rate = 1e-6
    Fixed at 1e-6 for all reported runs; affects whether proximal-update regime holds.
  • group size G and batch configuration = G=8, batch 256/16
    G=8 samples/prompt, global batch 256, mini-batch 16; standard but free design choices that affect advantage standardization.
axioms (6)
  • domain assumption Latent dense reward decomposition R(y)=∑_t r*_t exists and token credit should track E[r*_t R]/E[R²] (Eq. 8).
    Unobservable decomposition used to motivate any token-level weight α_t; not verified, only posited in §4.1.
  • domain assumption Step-wise predictive uncertainty (entropy or mode-local δ) is a useful proxy for latent token importance in reasoning CoTs.
    Core modeling hypothesis; supported by first-error entropy analysis (§4.2) but not proven generally.
  • domain assumption Modal alignment: sampled token coincides with argmax next-token often enough that π(y)≈π(v*) for leading-order gradient analysis.
    Empirically ~83.9% (Table 4); required for Appendix C simplification of ∇J_ACPO.
  • domain assumption Proximal-update regime: importance weights w_i,t ≈ 1 during training so higher-order off-policy terms can be dropped.
    Stated in §5.2; soft gating and small LR encourage it but it is not guaranteed.
  • domain assumption LLM next-token distributions are sufficiently mass-concentrated that δ bounds true entropy tightly under top-k effective support.
    Used in §5.1 / Appendix B.4 with citations to Holtzman/Ethayarajh; justifies preferring mode-local over full entropy.
  • standard math Standard policy-gradient / importance-sampling identities and Shannon entropy calculus.
    Background math for Eqs. 1–3, 9, 15–17.
invented entities (2)
  • mode-local surrogate entropy δ_i,t = 1 − π_θ(v*_i,t) independent evidence
    purpose: Cheap, tail-invariant uncertainty proxy used as the token-level credit modulator instead of full Shannon entropy.
    Defined in §4.4; independent of how residual mass is spread; bounds and correlation with entropy are paper-internal plus mass-concentration literature.
  • hybrid modulation factor M_i,t combining current and old-policy δ no independent evidence
    purpose: Leading-order scalar that multiplies the policy gradient under modal alignment; claimed to stay non-negative and damp off-policy updates.
    Derived in §5.2 / Appendix C–D as an approximation, not an independently measured physical quantity.

pith-pipeline@v1.1.0-grok45 · 26157 in / 3597 out tokens · 35841 ms · 2026-07-12T04:42:42.991322+00:00 · methodology

0 comments
read the original abstract

Reinforcement Learning (RL) has substantially improved the reasoning ability of large language models (LLMs), but sparse outcome rewards still make token-level credit assignment difficult. Existing scalable RL methods typically assign trajectory-level rewards uniformly across tokens, while recent entropy-aware approaches either rely on coarse detached heuristics or directly optimize true entropy, which can introduce non-local gradient components misaligned with sampled-token policy updates. We propose Adaptive Credit Policy Optimization (ACPO), a token-level credit assignment framework based on a mode-local surrogate entropy. ACPO asymmetrically modulates policy updates by emphasizing uncertain decisions in successful rollouts and overconfident tokens in failed rollouts. We show that the surrogate admits deterministic entropy bounds and, under modal alignment and proximal updates, preserves the policy-gradient direction to leading order. Experiments on mathematical reasoning and coding benchmarks, including AIME 2025 and HumanEvalPro, show that ACPO consistently improves over strong RL baselines such as DAPO, GTPO, and SAPO.

Figures

Figures reproduced from arXiv: 2607.03126 by Enlei Gong, Peng Jiang, Quan Chen, Yadong Mu, Yanhua Cheng, Yongzhi Li, Yuyang You, Zeyu Chen, Zijun Xie.

Figure 1
Figure 1. Figure 1: Empirical analysis of token entropy around the first reasoning error. (a) Token-level entropy visualization. At the divergence point highlighted by the boxes, the incorrect trajectory introduces an ungrounded variable (a), after which subsequent tokens exhibit substantially higher entropy, shown in red/orange. In contrast, the correct trajectory derives the solution by factoring a cubic polynomial and main… view at source ↗
Figure 2
Figure 2. Figure 2: Training dynamics of SAPO and ACPO on Qwen2.5-Math-7B. ACPO shows faster early [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablation study. This figure shows the evolution of validation accuracy and entropy on [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Per-token entropy along full reasoning trajectories (Correct vs. Incorrect). For each of the 8 AIME prompts, we plot the smoothed step-wise entropy of a correct CoT (green) and an incorrect CoT (red). The vertical dotted lines annotate the exact position of the first fatal reasoning error and its logical counterpart. The global profiles illustrate that a single reasoning flaw typically induces a persistent… view at source ↗
Figure 5
Figure 5. Figure 5: Surrogate entropy tracks normalized true entropy. Along a real rollout trajectory from Qwen2.5-Math-7B on the AIME25 benchmark, the proposed surrogate entropy closely follows the macroscopic trend of normalized true entropy while remaining mode-local and invariant to tail dispersion. Substituting Eq. 23 back into the expression for g(δi,t(θ)): max δ g(δ) = δi,t(θ) (log(|V| − 1) − log t) = δi,t(θ)(t + 1) = … view at source ↗
Figure 6
Figure 6. Figure 6: Impact of different k values on training dynamics. The figure illustrates the learning curves of mean@8 accuracy on the AIME24 dataset during training, comparing the effects of setting k ∈ {1, 5, 20, 100}. As theoretically derived in Appendix B.4, the approximation error of our surrogate entropy δi,t(θ) is tightly bounded when the next-token probability mass is highly concentrated. To empirically investiga… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 33 linked inside Pith

  1. [1]

    arXiv preprint arXiv:1502.05477 , year=

    Trust Region Policy Optimization , author=. arXiv preprint arXiv:1502.05477 , year=

    Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  3. [3]

    arXiv preprint arXiv:2511.20347 , year=

    Soft adaptive policy optimization , author=. arXiv preprint arXiv:2511.20347 , year=

    Pith/arXiv arXiv

  4. [4]

    arXiv preprint arXiv:2506.01939 , year=

    Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning , author=. arXiv preprint arXiv:2506.01939 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2507.15778 , year=

    Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR , author=. arXiv preprint arXiv:2507.15778 , year=

    Pith/arXiv arXiv

  6. [6]

    American Invitational Mathematics Examination (AIME) 2024 , author=

    2024

  7. [7]

    American Invitational Mathematics Examination (AIME) 2025 , author=

    2025

  8. [8]

    BeyondAIME: Advancing Math Reasoning Evaluation Beyond High School Olympiads , author=

  9. [9]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  10. [10]

    arXiv preprint arXiv:2409.12122 , year=

    Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement , author=. arXiv preprint arXiv:2409.12122 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2103.03874 , year=

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2512.00499 , year=

    ESPO: Entropy Importance Sampling Policy Optimization , author=. arXiv preprint arXiv:2512.00499 , year=

    arXiv

  13. [13]

    arXiv preprint arXiv:2512.02556 , year=

    DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models , author=. arXiv preprint arXiv:2512.02556 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2410.01679 , year=

    VinePPO: Refining Credit Assignment in RL Training of LLMs , author=. arXiv preprint arXiv:2410.01679 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:1904.09751 , year=

    THE CURIOUS CASE OF NEURAL TEXT DeGENERATION , author=. arXiv preprint arXiv:1904.09751 , year=

    Pith/arXiv arXiv 1904

  16. [16]

    arXiv preprint arXiv:1711.03953 , year=

    Breaking the Softmax Bottleneck: A High-Rank RNN Language Model , author=. arXiv preprint arXiv:1711.03953 , year=

    Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:1909.00512 , year=

    How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings , author=. arXiv preprint arXiv:1909.00512 , year=

    Pith/arXiv arXiv 1909

  18. [18]

    arXiv preprint arXiv:2508.03772 , year=

    GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control , author=. arXiv preprint arXiv:2508.03772 , year=

    arXiv

  19. [19]

    arXiv preprint arXiv:1506.02438 , year=

    HIGH-DIMENSIONAL CONTINUOUS CONTROL USING GENERALIZED ADVANTAGE ESTIMATION , author=. arXiv preprint arXiv:1506.02438 , year=

    Pith/arXiv arXiv

  20. [20]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , journal =

    DeepSeek. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , journal =

  21. [21]

    Yifan Bai and Yiping Bao and Guanduo Chen and Jiahao Chen and Ningxin Chen and Ruijue Chen and Yanru Chen and Yuankun Chen and Yutian Chen and Zhuofu Chen and Jialei Cui and Hao Ding and Mengnan Dong and Angang Du and Chenzhuang Du and Dikang Du and Yulun Du and Yu Fan and Yichen Feng and Kelin Fu and Bofei Gao and Hongcheng Gao and Peizhong Gao and Tong ...

  22. [22]

    CoRR , volume =

    Shengyu Zhang and Linfeng Dong and Xiaoya Li and Sen Zhang and Xiaofei Sun and Shuhe Wang and Jiwei Li and Runyi Hu and Tianwei Zhang and Fei Wu and Guoyin Wang , title =. CoRR , volume =

  23. [23]

    CoRR , volume =

    Yu Yue and Yufeng Yuan and Qiying Yu and Xiaochen Zuo and Ruofei Zhu and Wenyuan Xu and Jiaze Chen and Cheng. CoRR , volume =

  24. [24]

    CoRR , volume =

    Ganqu Cui and Lifan Yuan and Zefan Wang and Hanbin Wang and Wendi Li and Bingxiang He and Yuchen Fan and Tianyu Yu and Qixin Xu and Weize Chen and Jiarui Yuan and Huayu Chen and Kaiyan Zhang and Xingtai Lv and Shuo Wang and Yuan Yao and Xu Han and Hao Peng and Yu Cheng and Zhiyuan Liu and Maosong Sun and Bowen Zhou and Ning Ding , title =. CoRR , volume =

  25. [25]

    CoRR , volume =

    Aili Chen and Aonian Li and Bangwei Gong and Binyang Jiang and Bo Fei and Bo Yang and Boji Shan and Changqing Yu and Chao Wang and Cheng Zhu and Chengjun Xiao and Chengyu Du and Chi Zhang and Chu Qiao and Chunhao Zhang and Chunhui Du and Congchao Guo and Da Chen and Deming Ding and Dianjun Sun and Dong Li and Enwei Jiao and Haigang Zhou and Haimo Zhang an...

  26. [26]

    CoRR , volume =

    Yuzhong Zhao and Yue Liu and Junpeng Liu and Jingye Chen and Xun Wu and Yaru Hao and Tengchao Lv and Shaohan Huang and Lei Cui and Qixiang Ye and Fang Wan and Furu Wei , title =. CoRR , volume =

  27. [27]

    CoRR , volume =

    Qiying Yu and Zheng Zhang and Ruofei Zhu and Yufeng Yuan and Xiaochen Zuo and Yu Yue and Tiantian Fan and Gaohong Liu and Lingjun Liu and Xin Liu and Haibin Lin and Zhiqi Lin and Bole Ma and Guangming Sheng and Yuxuan Tong and Chi Zhang and Mofan Zhang and Wang Zhang and Hang Zhu and Jinhua Zhu and Jiaze Chen and Jiangjie Chen and Chengyi Wang and Hongli ...

  28. [28]

    Group Sequence Policy Optimization , journal =

    Chujie Zheng and Shixuan Liu and Mingze Li and Xiong. Group Sequence Policy Optimization , journal =

  29. [29]

    CoRR , volume =

    Yiran Guo and Lijie Xu and Jie Liu and Dan Ye and Shuang Qiu , title =. CoRR , volume =

  30. [30]

    CoRR , volume =

    Yizhi Li and Qingshui Gu and Zhoufutu Wen and Ziniu Li and Tianshun Xing and Shuyue Guo and Tianyu Zheng and Xin Zhou and Xingwei Qu and Wangchunshu Zhou and Zheng Zhang and Wei Shen and Qian Liu and Chenghua Lin and Jian Yang and Ge Zhang and Wenhao Huang , title =. CoRR , volume =

  31. [31]

    The Surprising Effectiveness of Negative Reinforcement in

    Xinyu Zhu and Mengzhou Xia and Zhepei Wei and Wei. The Surprising Effectiveness of Negative Reinforcement in. CoRR , volume =

  32. [32]

    2025 , eprint=

    ASPO: Asymmetric Importance Sampling Policy Optimization , author=. 2025 , eprint=

    2025

  33. [33]

    2025 , eprint=

    Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective , author=. 2025 , eprint=

    2025

  34. [34]

    2025 , eprint=

    BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping , author=. 2025 , eprint=

    2025

  35. [35]

    CoRR , volume =

    An Yang and Beichen Zhang and Binyuan Hui and Bofei Gao and Bowen Yu and Chengpeng Li and Dayiheng Liu and Jianhong Tu and Jingren Zhou and Junyang Lin and Keming Lu and Mingfeng Xue and Runji Lin and Tianyu Liu and Xingzhang Ren and Zhenru Zhang , title =. CoRR , volume =

  36. [36]

    CoRR , volume =

    An Yang and Anfeng Li and Baosong Yang and Beichen Zhang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Gao and Chengen Huang and Chenxu Lv and Chujie Zheng and Dayiheng Liu and Fan Zhou and Fei Huang and Feng Hu and Hao Ge and Haoran Wei and Huan Lin and Jialong Tang and Jian Yang and Jianhong Tu and Jianwei Zhang and Jian Yang and Jiaxi Yang and Ji...

  37. [37]

    CoRR , volume =

    Kaiyan Zhang and Yuxin Zuo and Bingxiang He and Youbang Sun and Runze Liu and Che Jiang and Yuchen Fan and Kai Tian and Guoli Jia and Pengfei Li and Yu Fu and Xingtai Lv and Yuchen Zhang and Sihang Zeng and Shang Qu and Haozhan Li and Shijie Wang and Yuru Wang and Xinwei Long and Fangfu Liu and Xiang Xu and Jiaze Ma and Xuekai Zhu and Ermo Hua and Yihao L...

  38. [38]

    CoRR , volume =

    Yang Yue and Zhiqi Chen and Rui Lu and Andrew Zhao and Zhaokai Wang and Yang Yue and Shiji Song and Gao Huang , title =. CoRR , volume =

  39. [39]

    CoRR , volume =

    Mingjie Liu and Shizhe Diao and Ximing Lu and Jian Hu and Xin Dong and Yejin Choi and Jan Kautz and Yi Dong , title =. CoRR , volume =

  40. [40]

    CoRR , volume =

    Shenzhi Wang and Le Yu and Chang Gao and Chujie Zheng and Shixuan Liu and Rui Lu and Kai Dang and Xionghui Chen and Jianxin Yang and Zhenru Zhang and Yuqiong Liu and An Yang and Andrew Zhao and Yang Yue and Shiji Song and Bowen Yu and Gao Huang and Junyang Lin , title =. CoRR , volume =

  41. [41]

    CoRR , volume =

    Guanhua Huang and Tingqiang Xu and Mingze Wang and Qi Yi and Xue Gong and Siheng Li and Ruibin Xiong and Kejiao Li and Yuhao Jiang and Bo Zhou , title =. CoRR , volume =

  42. [42]

    EuroSys , pages =

    Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu , title =. EuroSys , pages =

  43. [43]

    Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph Gonzalez and Hao Zhang and Ion Stoica , title =

  44. [44]

    Ilya Loshchilov and Frank Hutter , title =

  45. [45]

    Yanli Zhao and Andrew Gu and Rohan Varma and Liang Luo and Chien. PyTorch. Proc

  46. [46]

    NeurIPS Datasets and Benchmarks , year =

    Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt , title =. NeurIPS Datasets and Benchmarks , year =

  47. [47]

    Bowman , title =

    David Rein and Betty Li Hou and Asa Cooper Stickland and Jackson Petty and Richard Yuanzhe Pang and Julien Dirani and Julian Michael and Samuel R. Bowman , title =. CoRR , volume =

  48. [48]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , booktitle =

    Naman Jain and King Han and Alex Gu and Wen. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , booktitle =

  49. [49]

    arXiv preprint arXiv:2305.18290 , year=

    Direct preference optimization: Your language model is secretly a reward model , author=. arXiv preprint arXiv:2305.18290 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating Large Language Models Trained on Code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  51. [51]

    arXiv preprint arXiv:2303.08774 , year=

    GPT-4 Technical Report , author=. arXiv preprint arXiv:2303.08774 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv preprint arXiv:2203.11171 , year=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. arXiv preprint arXiv:2203.11171 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2403.07691 , year=

    ORPO: Monolithic Preference Optimization without Reference Model , author=. arXiv preprint arXiv:2403.07691 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2406.08464 , year=

    Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing , author=. arXiv preprint arXiv:2406.08464 , year=

    Pith/arXiv arXiv

  55. [55]

    arXiv preprint arXiv:2403.08295 , year=

    Gemma: Open Models Based on Gemini Research and Technology , author=. arXiv preprint arXiv:2403.08295 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv preprint arXiv:2310.06825 , year=

    Mistral 7B , author=. arXiv preprint arXiv:2310.06825 , year=

    Pith/arXiv arXiv

  57. [57]

    arXiv preprint arXiv:2404.14219 , year=

    Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone , author=. arXiv preprint arXiv:2404.14219 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2403.04652 , year=

    Yi: Open Foundation Models by 01.AI , author=. arXiv preprint arXiv:2403.04652 , year=

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2308.12950 , year=

    Code Llama: Open Foundation Models for Code , author=. arXiv preprint arXiv:2308.12950 , year=

    Pith/arXiv arXiv

  60. [60]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  61. [61]

    arXiv preprint arXiv:2402.03300 , year=

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  62. [62]

    Advances in Neural Information Processing Systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in Neural Information Processing Systems , volume=

  63. [63]

    Advances in Neural Information Processing Systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in Neural Information Processing Systems , volume=

  64. [64]

    arXiv preprint arXiv:2407.21783 , year=

    The Llama 3 Herd of Models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  65. [65]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  66. [66]

    Advances in Neural Information Processing Systems , volume=

    STaR: Bootstrapping Reasoning With Reasoning , author=. Advances in Neural Information Processing Systems , volume=

  67. [67]

    arXiv preprint arXiv:2310.12036 , year=

    A General Theoretical Paradigm to Understand Learning from Human Preferences , author=. arXiv preprint arXiv:2310.12036 , year=

    Pith/arXiv arXiv

  68. [68]

    arXiv preprint arXiv:2402.01306 , year=

    KTO: Model Alignment as Prospect Theoretic Optimization , author=. arXiv preprint arXiv:2402.01306 , year=

    Pith/arXiv arXiv

  69. [69]

    arXiv preprint arXiv:2407.10670 , year=

    Qwen2 Technical Report , author=. arXiv preprint arXiv:2407.10670 , year=

    Pith/arXiv arXiv

  70. [70]

    arXiv preprint arXiv:2305.20050 , year=

    Let's Verify Step by Step , author=. arXiv preprint arXiv:2305.20050 , year=

    Pith/arXiv arXiv

  71. [71]

    arXiv preprint arXiv:2512.21625 , year=

    Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards , author=. arXiv preprint arXiv:2512.21625 , year=

    arXiv

  72. [72]

    arXiv preprint arXiv:2412.21199 , year=

    Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation , author=. arXiv preprint arXiv:2412.21199 , year=

    Pith/arXiv arXiv

  73. [73]

    International Conference on Machine Learning (ICML) , pages=

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor , author=. International Conference on Machine Learning (ICML) , pages=. 2018 , organization=

    2018