Pith. sign in

REVIEW 4 major objections 5 minor 82 references

Parameter Exploration for RLVR via Variational Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sampling rollout policies from a learned weight posterior improves LLM reinforcement learning, and the multi-policy chunked variant C3PO performs best.

desk verdict C3PO is a genuinely new and plausible weight-space exploration method for RLVR, but its headline results lean on an under-derived Seq-MIS correction that the paper itself shows is necessary. read the letter →

arxiv 2608.09805 v1 pith:BUQQ2SEM submitted 2026-08-10 cs.LG cs.AIcs.CL

Vatsal Venkatkrishna , Nico Daheim , Iryna Gurevych This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords parameter-spaceexplorationRLVRreinforcementlearningwithverifiablerewardsvariationalIVONGRPOweightperturbationLLMreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that exploration in large-language-model reinforcement learning should happen in weight space, not only in token space. The authors introduce 3PO (Perturbed Parameter Policy Optimization), a family of methods that samples rollout policies from a learned approximate posterior over the model's weights, using the IVON variational optimizer, instead of only raising the sampling temperature during generation. Across two 7B models and math and code benchmarks, all 3PO variants beat standard GRPO (group-relative policy optimization) on average at near-identical FLOPs cost, and the best variant, C3PO, mixes rollouts from several perturbed policies inside each GRPO advantage group. The gains concentrate on harder benchmarks and come with fewer zero-advantage groups and fewer malformed rollouts, which the paper reads as evidence that parameter perturbations produce genuinely more useful exploration than action-space noise. If the claim holds, parameter-space exploration is a cheap, orthogonal lever that existing temperature-based RLVR recipes can adopt without retraining.

What carries the argument

The engine is the IVON optimizer, which maintains a diagonal Gaussian posterior over the model weights: at each step a perturbed point $\hat{\theta} = m + \sigma \odot z$ with $z \sim \mathcal{N}(0, I)$ is used for rollout generation, and the per-parameter variance $\sigma^2 = 1/(\lambda(h + \delta))$ is inversely proportional to a Hessian $h$ scaled by an effective sample size $\lambda$, so the amount of exploration is learned rather than fixed. The 3PO family wraps this in three rollout-grouping strategies, and the variant that carries the paper's best results is C3PO's chunked noising: $N$ independent weight samples each generate $G/N$ rollouts, and the GRPO advantage is computed over the full group of $G$ rollouts, maximizing diversity within the advantage calculation. The third load-bearing piece is the Seq-MIS correction, a sequence-level importance-sampling and masking fix applied to the GRPO ratio because the $N$ rollout generators differ from the policy being updated; Appendix I shows the training reward stays flat without it.

What would settle it

Run C3PO on the Olmo3 math setup with Seq-MIS replaced by a provably unbiased estimator for groups generated from $N$ policies (for example, caching the $N$ weight samples and recomputing exact per-token ratios). If the training reward stays flat under that unbiased estimator, the stability carrying the main results is not attributable to parameter-space exploration; if training recovers, the correction is validated and the mechanism confirmed.

Watch

Extended reading notes

Core claim

The paper's central claim is that sampling weights from a learned posterior at rollout time provides an additional exploration lever for RLVR that action-space controls like temperature cannot provide, because weight noise can reorder the token distribution rather than merely flatten it. Concretely, the authors claim that drawing one or more policy samples from the IVON posterior and generating rollouts with them improves downstream pass@1 over GRPO across both model families, with C3PO — which partitions each GRPO group of $G$ rollouts across $N$ independently sampled weights and computes advantages over the aggregated group — having the highest average performance in both model families. They further claim that multi-sample parameter exploration rescues zero-advantage groups throughout training and produces fewer degenerate and incorrect rollouts than GRPO and the action-space baselines tested (temperature-based Polaris, entropy regularization, and KL-covariance weighting), and that the benefit is largest on harder benchmarks such as AIME and LiveCodeBench.

Load-bearing premise

The load-bearing premise is that the Seq-MIS correction is the right fix for C3PO's training-inference mismatch: the paper adopts it from a blog citation without deriving or validating it as a proper estimator when one GRPO group mixes $N$ different rollout policies, and without it C3PO's training reward stays flat.

Editorial extensions

If this is right

  • C3PO-style chunked noising is a drop-in change to the rollout side of an existing RLVR pipeline: it lifts average pass@1 over GRPO at near-identical per-step FLOPs, so practitioners can adopt it without touching the rest of the recipe.
  • Parameter-space exploration helps most where the pre-RL model is weakest, with the largest gains on AIME and LiveCodeBench; the same rollout budget yields more high-reward trajectories on hard tasks.
  • The learned noise matters, not noise itself: a matched isotropic-noise run plateaus like GRPO, so the Hessian-scaled variance and its effective sample size $\lambda$ are what carry the improvement.
  • Multi-sample variants (M3PO, C3PO) keep rescuing zero-advantage groups into late training, while single-sample B3PO and all action-space baselines lose this ability, indicating that across-policy group diversity sustains the learning signal.
  • The roughly 1.5$\times$ wall-clock overhead is a systems artifact of inefficient multi-model sampling; since all 3PO variants converge faster, their effective cost to reach a target score could fall below GRPO once that infrastructure improves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If rescuing dead groups is the mechanism, the benefit should grow with model scale: larger pretrained models have denser neighborhoods of competent solutions around their weights, so a fixed noise budget should yield more diverse valid rollouts — a testable prediction on larger checkpoints.
  • The stability that carries the main results depends on the Seq-MIS correction being a proper estimator when a GRPO group mixes $N$ policies, which the paper neither derives nor validates; a correct unbiased estimator for that setting would settle whether the gains are genuinely due to parameter-space exploration.
  • The paper's analysis suggests a practical monitoring habit: track zero-advantage group rescue and malformed-rollout rates rather than entropy, since entropy falls both when a policy focuses on high-reward tokens and when it collapses.
  • Adaptive noise schedules are left for future work, and the paper finds opposite $\lambda$ preferences for M3PO and C3PO, so annealing the noise scale during training could recover gains that a fixed $\lambda$ leaves on the table.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Perturbed Parameter Policy Optimization (3PO), a family of methods that perform parameter-space exploration for reinforcement learning with verifiable rewards (RLVR) by sampling rollout policies from an approximate posterior learned with the IVON optimizer. Three variants are studied: B3PO (one weight perturbation per gradient step), M3PO (multiple perturbations with per-perturbation advantage groups), and C3PO (chunking each GRPO group across N perturbed policies and computing advantages over the mixed group). Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation report that 3PO methods, especially C3PO, improve average Pass@1 over GRPO, rescue more zero-advantage groups, and produce fewer malformed or incorrect rollouts. The paper includes ablations of the effective sample size lambda, the number of Monte Carlo samples M, the chunk size N, and the initial Hessian, and it identifies a Seq-MIS correction as necessary for C3PO training stability.

Significance. If the reported gains are robust, the paper makes a useful contribution: it demonstrates a compute-efficient exploration lever that is orthogonal to action-space methods and can be applied to off-the-shelf checkpoints. The manuscript has several concrete strengths: the 3-seed paired comparison of C3PO versus GRPO on Olmo3 (Appendix G), the consistent code-generation results (Section 4.4), and the isotropic-noise control (Appendix H) that indicates the learned Hessian, not merely added noise, is responsible for the late-training improvement. The public release of the implementation is also a strength. However, the central C3PO claim depends on a stabilization heuristic whose estimator is never specified, and much of the evidence consists of single runs with hyperparameters selected on the evaluation benchmarks themselves. These issues are load-bearing for the main claim that parameter-space exploration, rather than the stabilization heuristic or selection artifact, drives the improvements.

major comments (4)
  1. [Section 3.2, Eq. (1), Algorithm 2, Appendix I] The Seq-MIS correction is load-bearing but is never defined as an estimator. In C3PO, each rollout i in a group is generated by a different perturbed policy pi_{theta_hat_{n(i)}}, so the importance ratio in Eq. (1) must be defined relative to that rollout's own generator, e.g., R_{i,t} = pi_theta(y_t^i | x, y_{<t}^i) / pi_{theta_hat_{n(i)}}(y_t^i | x, y_{<t}^i). The paper does not state what pi_old is in Eq. (1) under C3PO, and Algorithm 2 only says 'cf. Eq. 1 with Seq-MIS correction' while Appendix A describes masking sequence-level importance weights outside [0.5, 2.0]. This is not a presentation quibble: Appendix I reports that training reward stays flat without the correction, so all C3PO results depend on it. Please provide the derivation of the corrected loss, state its target objective, and validate that the masked sequence-level correction is a controlled estimator, for example by comparing against per-token importance weighting with the correct per-rollout denominator, noise replay, or ablation of the clip bounds.
  2. [Section 4.1, Section 5, Table 1] The main hyperparameters are tuned on the same benchmarks that form Table 1. The effective sample size lambda, the chunk size N, and the Monte Carlo count M are swept in Section 5.1-5.3 on the Olmo3 math benchmarks, and the selected values (lambda=1e9, N=4, M=4) are then used in the headline comparison. For C3PO, lambda=1e9 is chosen after observing collapse at 1e8 and a lower ceiling at 1e10, so part of the reported advantage over GRPO is a selection artifact. The manuscript should either evaluate on held-out development benchmarks, report the default-configuration result, or quantify the selection effect by comparing the best-of-sweep result with the median or a prespecified configuration.
  3. [Appendix G, Table 1] Most entries in Table 1 are single runs; the only multi-seed comparison is C3PO versus GRPO on Olmo3 with three seeds (Appendix G). The claims that 'all 3PO variants outperform action-space baselines on average' and that 'C3PO has the highest average performance in both model families' therefore rest on single runs for Qwen2.5-Math and for the B3PO/M3PO comparisons. Please add at least three seeds for the key pairwise comparisons on both models, or clearly mark single-run cells and restrict the strength of the conclusions. The current paired test in Appendix G is positive but covers only one model and one method pair.
  4. [Section 4.3, Figure 3] The rescue and degeneracy comparison for M3PO is not matched on rollout budget: the footnote to Figure 3 states that GRPO's group is subsampled to 4 rollouts for its pairing with M3PO, while M3PO computes advantages on groups of 4 rollouts per perturbation and aggregates over M=4 perturbations, for a total of 16 rollouts. A GRPO group of 4 is more likely to contain no correct rollout than a group of 16, so the conclusion that 3PO 'rescues more zero-advantage groups than baselines' may be confounded by group size. Please report the comparison at matched total rollout counts or justify the subsampling.
minor comments (5)
  1. [Appendix G] There are typos in the first sentence: 'Hoewever, bcause' should be 'However, because'.
  2. [Section 4.3] The sentence 'Using IVON further improves the quality of these trajectories by learning the distribution over policies jointly during training (Eq. 4)' appears twice in the same paragraph; please remove the duplicate.
  3. [Appendix I, Figure 12] The text describes the no-Seq-MIS run as 'essentially flat' and the corrected run as 'steadily rising,' but it does not report numeric reward values or the number of steps. Please include quantitative endpoints so the reader can judge the magnitude of the effect.
  4. [Reference [35]] The Seq-MIS correction is cited only to a personal blog post ([35]). Since the correction is load-bearing for C3PO, the manuscript should either provide a self-contained description and derivation or cite a peer-reviewed or archival source that defines the method.
  5. [Abstract and Section 7] The abstract and conclusion emphasize 'near-identical FLOPs cost,' but Section 7 and Table 3 report a 1.5x wall-clock overhead for M3PO and C3PO. This is acknowledged as a systems artifact, but the abstract would be more precise if it stated 'near-identical FLOPs' rather than implying comparable running time.

Circularity Check

1 steps flagged · score 4.0 of 10

No equation-level circularity, but the headline 3PO gains are partially a selection artifact: main-config hyperparameters were chosen by sweeping on the same evaluation benchmarks whose scores are then presented as the method's predicted performance.

  1. fitted input called prediction [Section 4.1, Section 5.1, Table 2 vs. Table 1]
    "We use λ= 10^9 for Olmo3 and λ= 10^10 for Qwen2.5-Math. We sweep these hyperparameters in Section 5 and Appendices C and D... Through our experiments, we found that λ= 10^9 is a good default"

    The sweep in Section 5.1/Table 2 is evaluated on exactly the six benchmarks whose averages are reported in Table 1. For C3PO it selects λ=10^9 (average 44.04) over λ=10^10 (41.42) and λ=10^8 (0.00), and Section 4.1 then fixes λ=10^9 for Olmo3 and λ=10^10 for Qwen2.5-Math; Sections 5.2-5.3 similarly choose M=4 and N=4 after sweeping on the same task family. The headline claim that C3PO has the highest average performance is therefore partly an in-sample report of the best hyperparameter configuration found by looking at the target benchmarks, not an independent prediction.

full rationale

There is no construction-level circularity in the derivation chain: Eq. (1) is the GRPO objective, Algorithm 2 applies it to a rollout buffer generated from N sampled weight perturbations, and Eqs. (2)-(4) are the standard variational-learning posterior update. None of the 3PO definitions is stated in terms of the benchmark scores it is used to predict, and the comparisons are made against external benchmarks and baselines rather than against the paper's own outputs. The Seq-MIS correction in Section 3.2 is load-bearing and is asserted via an unpublished blog citation [35] rather than derived; Appendix I shows training stalls without it. That is a missing-support and correctness risk, not a circular reduction, so it does not by itself raise the circularity score. The IVON self-citations [10, 55] are also not load-bearing circularity: IVON is an externally released optimizer, and the main empirical contrast is GRPO versus 3PO on external benchmarks. The concrete circularity-adjacent weakness is the selection of λ, M, and N on the same six math benchmarks whose averages are then featured in Table 1, which makes part of the reported gain a selection artifact. Because the central claim still has independent empirical content, the score is 4 rather than 0-2.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on several hand-set hyperparameters (lambda, N, M) that were tuned on the same benchmarks used for the main results, on the IVON approximate posterior, and on the Seq-MIS correction borrowed from a non-archival source. No new theoretical entities are introduced.

free parameters (8)
  • ESS lambda (Olmo3) = 1e9
    Controls the learned noise scale in Eq. 4; selected by sweeping on the six evaluation benchmarks (Table 2, Section 5.1).
  • ESS lambda (Qwen2.5-Math) = 1e10
    Same as above; Qwen2.5-Math needed a higher lambda for stability (Section 5.1, Appendix C).
  • C3PO chunk size N = 4
    Chosen from N in {1,2,4,8,16} on the evaluation benchmarks (Section 5.3, Figure 5c).
  • M3PO MC samples M = 4 with G=4
    Chosen from M in {1,2,4,8} under equal-compute constraint (Section 5.2, Figure 5a-b).
  • Initial Hessian h0 = 0.001
    Hand-set constant for IVON at the start of RLVR (Appendix A); ablation shows learned prior adds little (Section 5.4).
  • IVON learning rates = 50.0 (SFT), 1.0 (RLVR)
    Hand-set; choices not ablated, could affect absolute performance.
  • Seq-MIS clip bounds = [0.5, 2.0]
    Masking range for sequence-level importance weights in C3PO (Appendix A), necessary for stability.
  • Batch size and group size = 32 prompts, G=16
    Fixed across methods; not ablated.
assumptions (4)
  • domain assumption The GRPO group-relative advantage (Eq. 1) is a valid learning signal for RLVR and zero-advantage groups stall learning.
    Section 2.1; this motivates the whole exploration approach.
  • domain assumption IVON's diagonal Hessian (Eq. 4) provides a meaningful per-parameter noise scale for exploration.
    Used throughout; Appendix H with isotropic matched noise suggests the Hessian structure matters, but the Hessian itself is an approximation.
  • domain assumption The Seq-MIS correction makes GRPO valid when C3PO mixes N rollout policies in one group.
    Section 3.2; the paper relies on reference [35] and does not derive the correction.
  • domain assumption The evaluation protocol (Pass@1 over 8 samples at temperature 0.6, top-p 0.95, top-k 50) is a reliable measure of downstream performance.
    Appendix A; all headline claims depend on this protocol.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Parameter Exploration for RLVR via Variational Learning." pith.science (2026). https://pith.science/paper/BUQQ2SEM

@misc{pith2026260809805,
  author       = {Pith},
  title        = {Pith review of: Parameter Exploration for RLVR via Variational Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BUQQ2SEM}},
  note         = {Machine review of arXiv:2608.09805}
}
read the original abstract

Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significantly impact downstream performance. Many existing methods control exploration in the action-space, for example, using temperature scaling. However, these methods cannot reorder tokens but only influence the variance in the output distribution. This limits exploration and can lead to divergence or stalled training. Here, we investigate parameter-space exploration, where rollouts are generated by sampling different policies from a posterior that may each explore different rollouts. Sampling less or more diverse policies is then a complementary control lever over exploration. We introduce a family of methods called Perturbed Parameter Policy Optimization (3PO) which use different sampling strategies and different rollout grouping for reward estimation. Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks show that these approaches consistently improve average downstream performance over standard GRPO at a near-identical FLOPs cost. Moreover, using multiple parameter samples consistently produces fewer zero-advantage groups and malformed or incorrect rollouts during training than GRPO and action-space baselines. Overall, our work presents evidence that parameter-space exploration can improve reinforcement learning for LLMs.

Figures

Figures reproduced from arXiv: 2608.09805 by the authors.

Figure 1
Figure 1. Temperature versus Weight noise. Temperature scaling controls the entropy of the token distribution but preserves relative token ordering. Adding noise to the model’s weights can completely alter the token distribution, helping the model explore new regions of the solution space. of the token distribution and not the relative token order, which limits the trajectories that can be explored. Moreover, it uniformly alt… view at source ↗
Figure 2
Figure 2. Overview of noising strategies. Squircles denote model weights (base θ or perturbed θˆ), while squares denote rollouts; each column corresponds to a rollout group for a different prompt. Color intensity denotes the amount of noise added. GRPO generates all G rollouts using a single shared model θ, with diversity arising only from sampling. B3PO samples a single perturbation θˆ per batch and reuses it across all prom… view at source ↗
Figure 3
Figure 3. (Left) Cumulative zero-advantage groups rescued over training. 3PO rescues more groups than GRPO and action-space baselines, with multiple perturbations yielding consistent improvements. (Middle) Paired degeneracy rates vs GRPO. Polaris causes high degeneration due to uniform token reordering at high temperature. (Right) Paired incorrect rollout rates vs GRPO. While EntReg escapes degeneracy in late training, it buy… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Reward and accuracy curves for code genera￾tion. C3PO outperforms all other methods on reward and accuracy. GRPO retains higher entropy longest but converts it into neither reward nor accuracy gains. Method ESS (λ) Avg. Math Pass@1 B3PO 108 41.90 ± 0.51 109 42.73 ± 0.4…
Figure 5
Figure 5. Figure 5: (a) Equal-compute comparison: Benefits of increasing M are offset by reducing G. (b) Scaling M: The advantages of more MC samples are fully realized without an equal-compute constraint. (c) Scaling N: N > 1 improves performance, further scaling is incremental. (d) Lear…
Figure 6
Figure 6. Figure 6: Reward and entropy curves for both model families. All 3PO methods converge faster than vanilla GRPO. Olmo3-C3PO’s entropy declines rapidly, but all methods maintain largely similar entropy profiles for Qwen2.5-Math. 0 100 200 300 400 500 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Re…
Figure 7
Figure 7. Figure 7: λ scaling curves across all 3PO methods. Adding too much noise with small λ hurts performance, while large λ values can make the sampled models too similar. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Effects of scaling λ for Qwen2.5-Math. Qwen2.5-Math is more sensitive to small λ, where overexploration causes the curves to oscillate. Tuning λ is therefore important. This suggests that scaling M without shrinking the rollout group could be a viable strategy for impr…
Figure 9
Figure 9. Figure 9: Detailed results for ablating MC samples and chunk size. Using M > 1 at a constant rollout budget lowers entropy, but average reward and downstream pass@1 are largely unchanged. 0 200 400 Step 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Reward 0 200 400 Step 0.1 0.2 0.3 0.4 0.5 0.6 0.…
Figure 10
Figure 10. Figure 10: G–M tradeoff on Qwen2.5-Math. The variance reduction from larger M does not compensate for the loss in group diversity from smaller G, and vice versa. from the learned prior implies that 3PO can be applied to any off-the-shelf checkpoint without first running an IVON-…
Figure 11
Figure 11. Figure 11: Effects of a learned noise prior on other algorithms. All three 3PO variants respond similarly to a learned noise prior, possibly due to a relatively isotropic Hessian even after SFT. 0 100 200 300 400 500 Step 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Training reward IVON vs. …
Figure 12
Figure 12. Figure 12: (Left) IVON vs isotropic noise Isotropic noise converges early but plateaus similar to GRPO. (Middle) Effect of Seq-MIS correction. Training completely stalls without the correction due to training-inference mismatch. (Right) Effect of Thompson sampling. Sampling a fr…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 44 canonical work pages

  1. [1]

    A survey of exploration methods in reinforcement learning.CoRR, abs/2109.00157, 2021

    Susan Amin, Maziar Gomrokchi, Harsh Satija, Herke van Hoof, and Doina Precup. A survey of exploration methods in reinforcement learning.CoRR, abs/2109.00157, 2021. URL https: //arxiv.org/abs/2109.00157

    arXiv 2021

  2. [2]

    Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025

    Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang et al. Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. URL https://hkunlp.github.io/blog/ 2025/Polaris

    2025

  3. [3]

    Learning to explore with parameter-space noise: A deep dive into parameter-space noise for reinforcement learning with verifiable rewards.CoRR, abs/2602.02555, 2026

    Bizhe Bai, Xinyue Wang, Peng Ye, and Tao Chen. Learning to explore with parameter-space noise: A deep dive into parameter-space noise for reinforcement learning with verifiable rewards.CoRR, abs/2602.02555, 2026. doi: 10.48550/ARXIV .2602.02555. URL https: //doi.org/10.48550/arXiv.2602.02555

    work page doi:10.48550/arxiv.2602.02555 2026

  4. [4]

    Llama-nemotron: Efficient reasoning models, 2025

    Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani et al. Llama-nemotron: Efficient reasoning models, 2025. URLhttps://arxiv.org/abs/2505.00949

    arXiv 2025

  5. [5]

    Weight un- certainty in neural network

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight un- certainty in neural network. In Francis Bach and David Blei, editors,Proceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of Ma- chine Learning Research, pages 1613–1622, Lille, France, 07–09 Jul 2015. PMLR. URL https://proceedings...

    2015

  6. [6]

    Fullstack bench: Evaluating llms as full stack coders, 2025

    Bytedance-Seed-Foundation-Code-Team, :, Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, Shijie Geng et al. Fullstack bench: Evaluating llms as full stack coders, 2025. URLhttps://arxiv.org/abs/2412.00535

    arXiv 2025

  7. [7]

    Skyrl-v0: Train real-world long-horizon agents via reinforcement learning, 2025

    Shiyi Cao, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang, Akshay Malik, Graham Neubig et al. Skyrl-v0: Train real-world long-horizon agents via reinforcement learning, 2025. URLhttps://github.com/NovaSky-AI/SkyRL

    work page 2025

  8. [8]

    Evaluating large language models trained on code.CoRR, abs/2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman et al. Evaluating large language models trained on code.CoRR, abs/2107.03374, 2021. URL https://arxiv.org/ abs/2107.03374

    arXiv 2021

Show all 82 references
  1. [9]

    Exploration vs exploitation: Rethinking RLVR through clipping, entropy, and spurious reward

    Peter Chen, Xiaopeng Li, Ziniu Li, Wotao Yin, Xi Chen, and Tianyi Lin. Exploration vs exploitation: Rethinking RLVR through clipping, entropy, and spurious reward. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?i...

    2026

  2. [10]

    Improving lora with variational learning, 2025

    Bai Cong, Nico Daheim, Yuesong Shen, Rio Yokota, Mohammad Emtiyaz Khan, and Thomas Möllenhoff. Improving lora with variational learning, 2025. URL https://arxiv.org/abs/ 2506.14280

    2025 arXiv

  3. [11]

    The entropy mechanism of reinforcement learning for reasoning language models.CoRR, abs/2505.22617, 2025

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen et al. The entropy mechanism of reinforcement learning for reasoning language models.CoRR, abs/2505.22617, 2025. doi: 10.48550/ARXIV .2505. 22617. URLhttps:/...

    2025 doi arXiv

  4. [12]

    Uncertainty-aware decoding with minimum bayes risk

    Nico Daheim, Clara Meister, Thomas Möllenhoff, and Iryna Gurevych. Uncertainty-aware decoding with minimum bayes risk. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=hPpyUv1XyQ

    2025

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URLhttps://arxiv.org/abs/2501.12948

    2025 arXiv

  6. [14]

    A survey on policy search for robotics.Found

    Marc Peter Deisenroth, Gerhard Neumann, and Jan Peters. A survey on policy search for robotics.Found. Trends Robotics, 2(1-2):1–142, 2013. doi: 10.1561/2300000021. URL https://doi.org/10.1561/2300000021

    2013 doi

  7. [15]

    Sharpness-aware min- imization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. InInternational Conference on Learning Representations, 2021. URLhttps://openreview.net/forum?id=6Tm1mposlrM

    2021

  8. [16]

    Noisy networks for exploration

    Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, V olodymyr Mnih, Rémi Munos, Demis Hassabis et al. Noisy networks for exploration. In6th International Conference on Learning Representations, ICLR 2018, Van- couver, B...

    2018

  9. [17]

    Neural thickets: Diverse task experts are dense around pretrained weights.CoRR, abs/2603.12228, 2026

    Yulu Gan and Phillip Isola. Neural thickets: Diverse task experts are dense around pretrained weights.CoRR, abs/2603.12228, 2026. doi: 10.48550/ARXIV .2603.12228. URL https: //doi.org/10.48550/arXiv.2603.12228

    2026 doi

  10. [18]

    Practical variational inference for neural networks

    Alex Graves. Practical variational inference for neural networks. In J. Shawe- Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, editors,Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc.,

  11. [19]

    Skywork open reasoner 1 technical report

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen et al. Skywork open reasoner 1 technical report. CoRR, abs/2505.22312, 2025. doi: 10.48550/ARXIV .2505.22312. URL https://doi.org/ 10.48550/arXiv.2505.22312

    2025 doi arXiv

  12. [20]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung, editors,Pro- ceedings of the Neural Information Processin...

    2021

  13. [21]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. InInternational Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rygGQyrFvH

    2020

  14. [22]

    Brorl: Scaling reinforcement learning via broadened exploration.CoRR, abs/2510.01180, 2025

    Jian Hu, Mingjie Liu, Ximing Lu, Fang Wu, Zaïd Harchaoui, Shizhe Diao, Yejin Choi, Pavlo Molchanov, June Yang, Jan Kautz et al. Brorl: Scaling reinforcement learning via broadened exploration.CoRR, abs/2510.01180, 2025. doi: 10.48550/ARXIV .2510.01180. URL https: //doi.org/10....

    2025 doi

  15. [23]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. 2025. URL https://openreview.net/ forum?id=chfJJYC3iL

    2025

  16. [24]

    Near-optimal regret bounds for reinforcement learning.J

    Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning.J. Mach. Learn. Res., 11:1563–1600, 2010. doi: 10.5555/1756006.1859902. URL https://dl.acm.org/doi/10.5555/1756006.1859902

    2010

  17. [25]

    Rethinking entropy regularization in large reasoning models.CoRR, abs/2509.25133, 2025

    Yuxian Jiang, Yafu Li, Guanxu Chen, Dongrui Liu, Yu Cheng, and Jing Shao. Rethinking entropy regularization in large reasoning models.CoRR, abs/2509.25133, 2025. doi: 10.48550/ ARXIV .2509.25133. URLhttps://doi.org/10.48550/arXiv.2509.25133

    2025 doi

  18. [26]

    The bayesian learning rule.Journal of Machine Learning Research, 24(281):1–46, 2023

    Mohammad Emtiyaz Khan and Håvard Rue. The bayesian learning rule.Journal of Machine Learning Research, 24(281):1–46, 2023. URL http://jmlr.org/papers/v24/22-0291. html

    2023

  19. [27]

    Fast and scalable bayesian deep learning by weight-perturbation in adam

    Mohammad Emtiyaz Khan, Didrik Nielsen, V oot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable bayesian deep learning by weight-perturbation in adam. In Jennifer G. Dy and Andreas Krause, editors,Proceedings of the 35th International Conference on Machine ...

    2018

  20. [28]

    Generalized variational inference: Three arguments for deriving new posteriors

    Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. Generalized variational inference: Three arguments for deriving new posteriors. 2019. URL https://arxiv.org/abs/1904. 02063

    2019

  21. [29]

    Policy gradient reinforcement learning for fast quadrupedal locomotion

    Nate Kohl and Peter Stone. Policy gradient reinforcement learning for fast quadrupedal locomotion. InProceedings of the 2004 IEEE International Conference on Robotics and Automation, ICRA 2004, April 26 - May 1, 2004, New Orleans, LA, USA, pages 2619–2624. IEEE, 2004. doi: 10....

    2004 arXiv

  22. [30]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and...

    2023

  23. [31]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu et al

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu et al. Tülu 3: Pushing frontiers in open language model post-training.CoRR, abs/2411.15124, 2024. doi: 10.48550/ ARXIV .2...

    2024 doi arXiv

  24. [32]

    Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo et al

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V . Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo et al. Solving quantitative reasoning problems with language models. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Daniell...

    2022

  25. [33]

    Verified taco problems

    Kaixin Li. Verified taco problems. https://huggingface.co/datasets/likaixin/ TACO-verified, 2024. URL https://huggingface.co/datasets/likaixin/TACO- verified

    2024

  26. [34]

    Handling the positive-definite con- straint in the bayesian learning rule

    Wu Lin, Mark Schmidt, and Mohammad Emtiyaz Khan. Handling the positive-definite con- straint in the bayesian learning rule. InProceedings of the 37th International Conference on 13 Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine Learn- ing R...

    2020

  27. [35]

    When speed kills stability: Demystifying RL collapse from the training-inference mismatch

    Jiacai Liu, Yingru Li, Yuqian Fu, Jiawei Wang, Qian Liu, and Zhuo Jiang. When speed kills stability: Demystifying RL collapse from the training-inference mismatch. https:// richardli.xyz/rl-collapse, September 2025

    2025

  28. [36]

    Code-r1: Reproducing r1 for code with reliable rewards

    Jiawei Liu and Lingming Zhang. Code-r1: Reproducing r1 for code with reliable rewards. 2025

    2025

  29. [37]

    Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.CoRR, abs/2505.24864, 2025

    Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.CoRR, abs/2505.24864, 2025. doi: 10.48550/ARXIV .2505.24864. URL https: //doi.org/10.48550/...

    2025 doi arXiv

  30. [38]

    Regularization matters in policy optimization - an empirical study on continuous control

    Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell. Regularization matters in policy optimization - an empirical study on continuous control. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,

    2021

  31. [39]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. InSecond Conference on Language Modeling, 2025. URLhttps://openreview.net/forum?id=5PAF7PAY2Y

    2025

  32. [40]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URLhttps://openreview.net/forum?id=Bkg6RiCqY7

    2019

  33. [41]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa et al

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y . Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa et al. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://pretty-radio- b75.notion.site/DeepScaleR-Surpassing-O1-Pr...

    2025

  34. [42]

    American Invitational Mathematics Examination, 2026

    Mathematical Association of America. American Invitational Mathematics Examination, 2026. URLhttps://maa.org/maa-invitational-competitions/

    2026

  35. [43]

    Soap-bubbles: Structured weight uncertainty for neural networks, 2026

    Adrian Robert Minut, Nico Daheim, Marco Miani, Mohammad Emtiyaz Khan, Wu Lin, and Thomas Möllenhoff. Soap-bubbles: Structured weight uncertainty for neural networks, 2026. URLhttps://arxiv.org/abs/2606.23357

    2026 arXiv

  36. [44]

    SAM as an optimal relaxation of bayes

    Thomas Möllenhoff and Mohammad Emtiyaz Khan. SAM as an optimal relaxation of bayes. InThe Eleventh International Conference on Learning Representations, 2023. URL https: //openreview.net/forum?id=k4fevFqSQcX

    2023

  37. [45]

    Faster, more efficient RLHF through off-policy asynchronous learning

    Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Faster, more efficient RLHF through off-policy asynchronous learning. InThe Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview...

    2025

  38. [46]

    Olmo 3, 2025

    Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison et al. Olmo 3, 2025. URL https://arxiv.org/abs/2512.13961

    2025 arXiv

  39. [47]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray et al

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray et al. Training language models to follow instructions with human feedback. 2022. URL http://papers.nips.cc/ paper_files/paper/2022/...

    2022

  40. [48]

    Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz

    Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y . Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Can...

    2018

  41. [49]

    Exploring parameter space in reinforcement learning.Paladyn J

    Thomas Rückstieß, Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, and Jürgen Schmidhuber. Exploring parameter space in reinforcement learning.Paladyn J. Behav. Robotics, 1(1):14–24,

  42. [50]

    Evolution strategies as a scalable alternative to reinforcement learning.CoRR, abs/1703.03864, 2017

    Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning.CoRR, abs/1703.03864, 2017. URL http://arxiv. org/abs/1703.03864

    2017 arXiv

  43. [51]

    Parameter-exploring policy gradients.Neural Networks, 23(4):551–559, 2010

    Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients.Neural Networks, 23(4):551–559, 2010. doi: 10.1016/J.NEUNET.2009.12.004. URL https://doi.org/10.1016/j.neunet.2009. 12.004

    2010 doi

  44. [52]

    Amrith Setlur, Matthew Y . R. Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for llms.CoRR, abs/2506.09026, 2025. doi: 10.48550/ARXIV .2506.09026. URL https://doi.o...

    2025 doi arXiv

  45. [53]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.CoRR, abs/2402.03300, 2024. doi: 10.48550/ARXIV .2402.03300. URL https://doi.org/1...

    2024 doi arXiv

  46. [54]

    On entropy control in LLM-RL algorithms.CoRR, abs/2509.03493, 2025

    Han Shen. On entropy control in LLM-RL algorithms.CoRR, abs/2509.03493, 2025. doi: 10.48550/ARXIV .2509.03493. URLhttps://doi.org/10.48550/arXiv.2509.03493

    2025 doi

  47. [55]

    Variational learning is effective for large deep networks

    Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan et al. Variational learning is effective for large deep networks. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, A...

    2024

  48. [56]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024. URLhttps://github.com/verl-project/verl

    2024 arXiv

  49. [57]

    Strehl and Michael L

    Alexander L. Strehl and Michael L. Littman. A theoretical analysis of model-based interval estimation. InProceedings of the 22nd International Conference on Machine Learning, ICML ’05, page 856–863, New York, NY , USA, 2005. Association for Computing Machinery. ISBN 1595931805...

    2005

  50. [58]

    Path integral policy improvement with covariance matrix adaptation

    Freek Stulp and Olivier Sigaud. Path integral policy improvement with covariance matrix adaptation. InProceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26 - July 1, 2012. icml.cc / Omnipress, 2012. URL http://icml.cc...

    2012

  51. [59]

    RL grokking recipe: How does RL unlock and transfer new algorithms in LLMs? InThe Fourteenth International Conference on Learning Representations, 2026

    Yiyou Sun, Yuhan Cao, Pohao Huang, Haoyue Bai, Hannaneh Hajishirzi, Nouha Dziri, and Dawn Song. RL grokking recipe: How does RL unlock and transfer new algorithms in LLMs? InThe Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/f...

    2026

  52. [60]

    Richard S. Sutton. Generalization in reinforcement learning: Successful exam- ples using sparse coarse coding. In David S. Touretzky, Michael Mozer, and Michael E. Hasselmo, editors,Advances in Neural Information Processing Sys- tems 8, NIPS, Denver, CO, USA, November 27-30, 1...

    1995

  53. [61]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement learning - an introduction. Adaptive computation and machine learning. MIT Press, 1998. ISBN 978-0-262-19398-6. URL http: //www.incompleteideas.net/book/first/the-book.html

    1998

  54. [62]

    Theodorou, Jonas Buchli, and Stefan Schaal

    Evangelos A. Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to reinforcement learning.J. Mach. Learn. Res., 11:3137–3181, 2010. doi: 10.5555/ 1756006.1953033. URLhttps://dl.acm.org/doi/10.5555/1756006.1953033

    2010

  55. [63]

    Generalized exploration in policy search

    Herke van Hoof, Daniel Tanneberg, and Jan Peters. Generalized exploration in policy search. Mach. Learn., 106(9-10):1705–1724, 2017. doi: 10.1007/S10994-017-5657-1. URL https: //doi.org/10.1007/s10994-017-5657-1

    2017 doi

  56. [64]

    Aletheia: What makes RLVR for code verifiers tick?CoRR, abs/2601.12186, 2026

    Vatsal Venkatkrishna, Indraneil Paul, and Iryna Gurevych. Aletheia: What makes RLVR for code verifiers tick?CoRR, abs/2601.12186, 2026. doi: 10.48550/ARXIV .2601.12186. URL https://doi.org/10.48550/arXiv.2601.12186

    2026 doi arXiv

  57. [65]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning.CoRR, abs/2506.01939, 2025

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning.CoRR, abs/2506.01939, 2025. doi: 10.48550/...

    2025 doi arXiv

  58. [66]

    Learning from delayed rewards

    Christopher John Cornish Hellaby Watkins et al. Learning from delayed rewards. 1989

    1989

  59. [67]

    Adversarial weight perturbation helps robust generalization

    Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors,Advances in Neural Information Processing Systems 33: Annual Confere...

    2020

  60. [68]

    The invisible leash: Why RLVR may not escape its origin.CoRR, abs/2507.14843, 2025

    Fang Wu, Weihao Xuan, Ximing Lu, Zaïd Harchaoui, and Yejin Choi. The invisible leash: Why RLVR may not escape its origin.CoRR, abs/2507.14843, 2025. doi: 10.48550/ARXIV .2507. 14843. URLhttps://doi.org/10.48550/arXiv.2507.14843

    2025 doi

  61. [69]

    Reasoning or memorization? unreliable results of reinforcement learning due to data contamination

    Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang et al. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. In Sven Koenig, Chad Jenkins, and Matthew E. Taylor,...

    2026

  62. [70]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, 2024

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, 2024. URL https://arxiv.org/abs/ 2409.12122

    2024 arXiv

  63. [71]

    Your efficient rl framework secretly brings you off-policy rl training, August 2025

    Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training, August 2025. URL https://fengyao.notion.site/off-policy-rl

    2025

  64. [72]

    The debate on RLVR reasoning capability boundary: Shrinkage, expansion, or both? A two-stage dynamic view.CoRR, abs/2510.04028, 2025

    Xinhao Yao, Lu Yu, Xiaolin Hu, Fengwei Teng, Qing Cui, Jun Zhou, and Yong Liu. The debate on RLVR reasoning capability boundary: Shrinkage, expansion, or both? A two-stage dynamic view.CoRR, abs/2510.04028, 2025. doi: 10.48550/ARXIV .2510.04028. URL https://doi.org/10.48550/ar...

    2025 doi

  65. [73]

    DAPO: An open-source LLM reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu et al. DAPO: An open-source LLM reinforcement learning system at scale. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. ...

    2025

  66. [74]

    Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach

    Rosie Zhao, Alexandru Meterez, Sham M. Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RL post-training amplifies behaviors learned in pretraining.CoRR, abs/2504.07912, 2025. doi: 10.48550/ARXIV .2504.07912. URL https://doi.org/10. 48550/arXiv.2504.07912

    2025 doi arXiv

  67. [75]

    Group sequence policy optimization, 2025

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang et al. Group sequence policy optimization, 2025. URL https://arxiv.org/abs/2507.18071

    2025 arXiv

  68. [76]

    The surprising effectiveness of negative reinforcement in LLM reasoning

    Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in LLM reasoning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview. net/forum?id=ftVlLG9cks

    2025

  69. [77]

    Exploring multi-temperature strategies for token- and rollout-level control in RLVR

    Haomin Zhuang, Yujun Zhou, Taicheng Guo, Yue Huang, Fangxu Liu, Kai Song, and Xiangliang Zhang. Exploring multi-temperature strategies for token- and rollout-level control in RLVR. CoRR, abs/2510.08892, 2025. doi: 10.48550/ARXIV .2510.08892. URL https://doi.org/ 10.48550/arXiv...

    2025 doi

  70. [2010]

    URL https://doi.org/10.2478/s13230-010- 0002-4

    doi: 10.2478/S13230-010-0002-4. URL https://doi.org/10.2478/s13230-010- 0002-4

    doi

  71. [2011]

    URL https://proceedings.neurips.cc/paper_files/paper/2011/file/ 7eb3c8be3d411e8ebfab08eba5f49632-Paper.pdf

    2011

  72. [2018]

    URLhttps://openreview.net/forum?id=rywHCPkAW

  73. [2021]

    URLhttps://openreview.net/forum?id=yr1mzrH3IC

  74. [2024]

    URLhttps://proceedings.mlr.press/v235/shen24b.html

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.