Pith. sign in

REVIEW 4 major objections 4 minor 6 cited by

Outcome-based Exploration for LLM Reasoning

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Outcome-based RL for LLM reasoning degrades answer diversity even on training questions the model has never solved, via a transfer from solved to unsolved questions; UCB-style bonuses over final answers improve accuracy while mitigating the

desk verdict Useful empirical study of diversity collapse with modest but real gains; theory is conditional on an unmeasured learner-capability assumption. read the letter →

arxiv 2509.06941 v1 pith:TRBKOHBT submitted 2025-09-08 cs.LG cs.CL

Yuda Song , Julia Kempe , Remi Munos This is my paper
classification cs.LGcs.CL MSC 68Q3262L05
keywords diversitycollapseoutcome-basedexplorationLLMreasoningreinforcementlearningUCBpass@kGRPObandits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the standard recipe for improving LLM math reasoning — reinforcement learning that rewards only the final answer — destroys generation diversity even before deployment, and that the damage can be repaired cheaply by exploring in the space of final answers rather than the space of reasoning traces. Treating RL training as a sampling process, the authors compare what an RL run has produced on the training questions against what the base model would produce with the same sampling budget, and find that RL ends up with fewer distinct answers, even on questions it has never solved. They call this the transfer of diversity degradation: once the model concentrates on the correct answers of solved questions, the collapse propagates to unsolved ones through the shared policy. Because verifiable reasoning tasks admit only a handful of distinct final answers, they propose outcome-based exploration — UCB-style bonuses attached to answers — and two concrete algorithms, UCB-Con and Batch, that improve test accuracy on competition benchmarks while preserving pass@k diversity. A companion bandit analysis shows that outcome-level exploration enjoys regret that scales with the number of answers, not the number of traces, provided training on one trace generalizes to other traces with the same answer.

What carries the argument

Four objects carry the argument. (1) RL-as-sampling: each epoch of n trajectories per question is treated as a sample draw, so the RL run's cumulative pass@k and diff@k curves can be overlaid on base-model sampling with k = nT epochs; this is what exposes the transfer of diversity degradation. (2) The outcome space: for verifiable tasks the final answer acts as a coarse-grained label of the reasoning trace, and the paper measures that per-question answer supports stay under roughly 50, making outcome-space exploration tractable. (3) The outcome-based bonus: b_ucb(x,a) = min(1, 1/sqrt(N(x,a))) over final answers, inserted into the GRPO advantage, with variants UCB-Mean (batch-mean baseline) a

What would settle it

Hold out a set of training questions and freeze the policy's updates on them (zero gradient) while training normally on the rest; if the frozen copy's diff@k on those questions does not fall below the base model's at equal sampling budget, the claimed transfer of diversity degradation is not caused by cross-question propagation and the paper's mechanism is wrong. Separately, measure the soft-generalization fraction rho_o directly: train on a single trace ending in answer o and compare the probability lift on other traces ending in o against traces ending in other answers; if the lift is near z

Watch

Extended reading notes

Core claim

The paper's central finding is that diversity collapse under outcome-based RL is a training-set phenomenon with a cross-question mechanism: as training proceeds, the policy concentrates probability on correct answers of questions it has already solved, and because the policy is shared, this concentration transfers to unsolved questions, where the model now explores fewer distinct answers than the base model would at equal sampling cost. The paper's second finding is that the damage is addressable at the outcome level, because the final-answer space of a reasoning task is small — on average under 50 distinct answers per question. Adding the bonus b_ucb(x,a) = min(1, 1/sqrt(N(x,a))) to the GRP

Load-bearing premise

The load-bearing premise is Assumption A.3 (soft generalization): after seeing one reasoning trace that ends in a particular final answer, the model can route back to that answer and can remove a substantial fraction of all other traces ending in the same answer from future exploration — an assumed property of how the model generalizes that the paper does not measure; if that fraction is zero, the theoretical regret benefit disappears.

Editorial extensions

If this is right

  • UCB-Con and Batch both beat GRPO in pass@1 and pass@32 on MATH-500, AIME, and AMC23 with Llama-3.1-8B and Qwen-2.5-7B, and they mitigate the late-training overoptimization that makes vanilla RL's final checkpoint worse than its early one.
  • The exploration bonus attaches to final answers, so it is agnostic to the underlying RL algorithm and can be layered onto any verifiable-reward training pipeline.
  • Under soft generalization, outcome-level UCB converts a problem as hard as a K-armed bandit into one with O(sqrt(mT log T)) regret: the effective difficulty of RL reasoning post-training is set by the number of distinct answers, not the number of reasoning traces.
  • Batch exploration produces higher per-generation entropy on incorrect traces and more distinct answers per batch, making it the better choice when test-time pass@k at large k is the deployment goal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the transfer mechanism is real, exploration effort should be concentrated on questions the policy has not solved — a direct test would apply outcome bonuses only below a per-question solve threshold and compare with the paper's global bonuses.
  • The soft-generalization fraction rho_o that Theorem A.4 depends on is measurable: train on a single trace with answer o and measure the probability lift on other traces with answer o versus traces with other answers; models with a larger measured lift should show proportionally larger gains from outcome-based exploration.
  • The bandit view suggests a data-curation lever the paper does not explore: because regret scales with the answer count m, rebalancing training data to keep per-question answer diversity high should improve RL sample efficiency independently of the bonus.
  • Batch exploration is defined purely at the batch level, so its penalty could be applied at inference time when sampling many answers for test-time scaling, without any retraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies diversity collapse in outcome-based RL for LLM reasoning. It frames RL post-training as a sampling process and reports that RL reduces effective diversity on the training set relative to sampling from the base model, including on unsolved questions (the 'transfer of diversity degradation'), and that final-answer spaces are sufficiently small to be tractable. It then proposes outcome-based exploration bonuses: historical UCB-style variants (UCB, UCB-Mean, UCB-Con) and a batch-level repetition penalty (Batch). Experiments with GRPO on Llama-3.1-8B-Instruct and Qwen-2.5-7B-Base on MATH/DAPO splits report improved pass@1 and pass@k over vanilla RL while mitigating diversity collapse. The paper also introduces an outcome-based bandit abstraction and proves a lower bound (Theorem A.1) and conditional upper bounds (Theorems A.3 and A.4).

Significance. The empirical core is plausible and useful: the proposed bonuses are simple, algorithm-agnostic, and the evaluation covers two models, several datasets, three seeds, and Appendix D error bars. The paper honestly acknowledges limitations (verifiable domains, single-turn tasks). The lower bound in Theorem A.1 is a clean negative result, and the empirical finding that RL can reduce training-set diversity on a per-sample basis is noteworthy. However, the formal upper-bound contribution is currently not established as written: the proof of Theorem A.4 has a gap, and the key generalization assumption is unmeasured and not implemented by the proposed algorithms. The 'transfer' explanation in Section 2.2 is also presented with causal language despite only correlational evidence. The paper's central empirical claims are defensible, but the theoretical and explanatory claims need substantial reworking.

major comments (4)
  1. [Appendix A.5, Theorem A.4 and Algorithm 3] The proof decomposes regret at the first hit of the optimal outcome tau_star and then applies Lemma A.4 to the tail. But Algorithm 3 does not enter outcome-level UCB until all m outcomes have been discovered; the tail from tau_star to tau_disc still consists of fresh draws from U, not pulls of representatives. Lemma A.4 does not apply to those rounds, and no bound on tau_disc is given under Assumption A.3. Concretely, if m=2, s_o* = K/2, and rho_o=0 for all o, then after hitting the optimal outcome the algorithm must still discover the rare outcome, which requires about K fresh draws and incurs about K*Delta regret, while the claimed bound is O(2 + sqrt(T log T)), independent of K. The reduction to an m-armed bandit therefore does not follow from the stated algorithm and proof. Additionally, the pseudocode does not remove the drawn arm A_t from U in the discovery phase, unlike Algorithms
  2. [Assumption A.3 and Section 3] Theorem A.4's improvement over the K-arm lower bound is entirely driven by Assumption A.3: the learner can route perfectly to a previously observed outcome and exclude a fraction rho_o of its outcome class from fresh probes. The paper itself calls this assumption 'rather unconventional' and notes it is an assumption on the learner. Yet no experiment measures rho_o, and the proposed UCB-Con and Batch algorithms (Section 3) do not implement routing or exclusion; they only add count-based bonuses. If rho_o=0, the first term becomes K/s_o*, recovering the K-arm regime. The main text's informal Theorem 4.2 is even stronger, relying on Assumption A.2 (strong generalization). The conclusion that outcome-based exploration is 'principled' is therefore conditional on an unverified capability that is not used by the proposed methods. The authors should either measure/justify rho_o from LLM data, or
  3. [Section 2.2, transfer of diversity degradation] The paper claims that reduced diversity on solved questions 'propagates' to unsolved questions, and calls this a central finding. The evidence is that RL training yields lower diff@k on unsolved questions than sampling from the base model. Since RL updates one shared network, unsolved-question behavior can change through many mechanisms unrelated to collapse on solved questions (e.g., general reasoning drift, reward hacking, format shift). No intervention isolates the proposed causal chain. The sentence 'The observed diversity degradation can therefore be explained as follows' overstates the evidence. Please reframe this as a hypothesis consistent with the observations, or add a controlled experiment (e.g., training only on solved questions and measuring unsolved-question diversity).
  4. [Table 3 and Table 4] Table 3 reports pass@1 and pass@32 'at the best checkpoint over three random seeds' without specifying how the best checkpoint is selected. If the selection is based on the test metrics being reported, the comparison is optimistic and can favor the method with more variance. The final-checkpoint results in Table 4 show materially smaller or different effects (e.g., UCB-Con on Qwen DAPO pass@32: 0.642 in Table 3 vs. 0.589 final; Batch final pass@32 is often the best). Please report a fixed protocol (e.g., validation-based checkpoint selection, or average over a fixed training schedule) and apply the same criterion to all methods.
minor comments (4)
  1. [Table 5] The caption of Table 5 appears to be copied from Table 2: it says 'Comparison of different exploration strategies based on the number of different answers sampled in a batch,' but the table lists hyperparameters. Please correct the caption.
  2. [Section 4.2 / Theorem 4.2] The main text states Theorem 4.2 under 'Assumption Assumption A.2' (strong generalization), while the more realistic soft model appears only in the appendix. Please make explicit in the main text which assumption the informal claim uses, since the paper's algorithms do not implement either assumption.
  3. [Figure 1 caption] The parenthetical 'except Qwen-2.5-7B-Base with UCB-Con on pass@1 on the early checkpoint due to exploration, but it has much higher pass@32 rate' is grammatically awkward and should be split into clearer sentences.
  4. [Section 2.2 notation] The notation A^{pi_RL}(x) and A^{pi_base}(x) is not formally defined before use; please define the answer-support notation in the preliminaries.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's central empirical findings are self-contained, and its theoretical claim is an explicitly conditional result rather than a reduction to its own inputs.

full rationale

The paper's main contributions are empirical: it documents diversity collapse during RL training (Section 2.2, Figure 2) and proposes UCB-Con and Batch exploration, evaluating them against the GRPO baseline on external benchmarks (MATH-500, AIME, AMC) with Llama and Qwen models. These results stand on their own and are not constructed from the paper's own assumptions. The theoretical analysis in Appendix A is explicitly conditional: Theorem A.4 derives a regret bound under Assumption A.3 (soft generalization), which the paper itself labels 'rather unconventional' and 'an assumption on the learner instead of the environment' (Appendix A.5). This is a stated modeling assumption, not a hidden circular input, and the paper is transparent that if rho_o = 0 the bound degrades toward the lower bound of Theorem A.1. The absence of direct validation of Assumption A.3 is a limitation of the theory's applicability, not a circular derivation. Self-citations (Song et al. 2024b, Tang and Munos 2025, Arnal et al. 2025) are contextual and do not carry the load of the paper's main claims. No equation is reused as a prediction, no fitted parameter is renamed as a discovery, and no uniqueness theorem from prior work is imported to force the proposed methods. The only notable gap is that the theoretical justification is not directly connected to the exact implemented algorithms, but that is a scope limitation, not circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper's empirical contributions rest on domain assumptions about the verifiable-reward setting and on an unverified causal explanation for the transfer effect. The theoretical contribution (Section 4.2, Appendix A) rests primarily on Assumption A.3, a learner-capability assumption that is not independently grounded. No new physical or ontological entities are introduced.

free parameters (2)
  • UCB bonus coefficient c = 0.1 (Llama-3.1-8B), 0.2 (Qwen-2.5-7B)
    Scales the exploration bonus in Eq. (2); chosen per model without a described selection procedure; exploration strength directly affects the trade-off between training and test performance.
  • UCB-Con constant baseline b0 = 1.0 (easy dataset), 0.5 (medium dataset)
    Controls the sign and magnitude of the exploration signal in UCB-Con; tuned per dataset (Section E); the paper notes b0=0.5 means an answer visited fewer than 4 times gets a positive signal.
assumptions (5)
  • domain assumption Rewards depend only on the final answer and are checked by math_verify equivalence (r(x,a) in {0,1}).
    This is the setting the algorithms target; it restricts the method to verifiable domains, which the paper acknowledges as a limitation in Section 6.
  • domain assumption GRPO objective (Eq. 1) is a reasonable proxy for the KL-regularized expected reward, despite known gradient bias.
    The paper cites Tang and Munos (2025) for the bias and uses GRPO as the 'vanilla RL' baseline; the 'RL as sampling' analysis assumes the training trajectories are comparable to i.i.d. base-model samples.
  • ad hoc to paper Assumption A.3 (soft generalization): after observing one arm with outcome o, the learner can route perfectly to o and exclude a fraction rho_o of the arms in the same outcome class.
    This is the load-bearing assumption for Theorem A.4, the main theoretical justification. It is stated as 'rather unconventional' by the authors and is not measured or validated in the LLM experiments.
  • ad hoc to paper The 'transfer of diversity degradation' explanation: reduced diversity on solved questions propagates to unsolved questions.
    The observed correlation between training progress and reduced unsolved-question diversity is interpreted as a causal transfer effect without an ablation; other mechanisms (shared weights, KL drift) are not ruled out (Section 2.2).
  • standard math Standard UCB regret bound for m-armed bandits with sub-Gaussian rewards (Lemma A.4).
    Cited from Auer et al. (2002); used to bound the post-discovery phase in Theorems A.2-A.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Outcome-based Exploration for LLM Reasoning." pith.science (2026). https://pith.science/paper/TRBKOHBT

@misc{pith2026250906941,
  author       = {Pith},
  title        = {Pith review of: Outcome-based Exploration for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TRBKOHBT}},
  note         = {Machine review of arXiv:2509.06941}
}
read the original abstract

Reinforcement learning (RL) has emerged as a powerful method for improving the reasoning abilities of large language models (LLMs). Outcome-based RL, which rewards policies solely for the correctness of the final answer, yields substantial accuracy gains but also induces a systematic loss in generation diversity. This collapse undermines real-world performance, where diversity is critical for test-time scaling. We analyze this phenomenon by viewing RL post-training as a sampling process and show that, strikingly, RL can reduce effective diversity even on the training set relative to the base model. Our study highlights two central findings: (i) a transfer of diversity degradation, where reduced diversity on solved problems propagates to unsolved ones, and (ii) the tractability of the outcome space, since reasoning tasks admit only a limited set of distinct answers. Motivated by these insights, we propose outcome-based exploration, which assigns exploration bonuses according to final outcomes. We introduce two complementary algorithms: historical exploration, which encourages rarely observed answers via UCB-style bonuses, and batch exploration, which penalizes within-batch repetition to promote test-time diversity. Experiments on standard competition math with Llama and Qwen models demonstrate that both methods improve accuracy while mitigating diversity collapse. On the theoretical side, we formalize the benefit of outcome-based exploration through a new model of outcome-based bandits. Together, these contributions chart a practical path toward RL methods that enhance reasoning without sacrificing the diversity essential for scalable deployment.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bridging Compute- and Data-Optimal Pretraining

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.

  2. Depth-Entropy Guided Sampling for Training-Free LLM Reasoning

    cs.LG 2026-06 conditional novelty 7.0 of 10

    Layer-wise entropy collapse depth is a weak per-candidate signal that, compounded inside MCMC power sampling, yields state-of-the-art training-free LLM reasoning accuracy.

  3. Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

    cs.AI 2025-10 conditional novelty 7.0 of 10

    MENTOR selectively mixes expert and policy token distributions at high-entropy decision points, improving RLVR accuracy and diversity on math and out-of-domain benchmarks compared with full-trajectory imitation.

  4. Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Training with appended behavioral instructions plus correctness-filtered self-distillation improves held-out math pass@1 over DAPO for a 1.7B model, but not for 4B at 4K context.

  5. It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

    cs.AI 2026-05 conditional novelty 6.0 of 10

    W2SPO trains a reasoning LLM by inserting short 8-token branches proposed by a weaker model into its own rollouts and updating only those branch tokens, improving Pass@1 over vanilla GRPO (64.2% vs 62.3%) with a 3.55x...

  6. Beyond the Sampled Token: Preserving Candidate Support in RLVR

    cs.AI 2025-10 conditional novelty 6.0 of 10

    SimKO, an asymmetric top-K token-level regularizer for RLVR, improves pass@K on math/logic benchmarks across K=1..256 without lowering pass@1 relative to GRPO.

Reference graph

Works this paper leans on

38 extracted references · 3 canonical work pages · cited by 6 Pith papers

  1. [1]

    2:fort= 1,2,

    We will bound the regret in theseT ′ rounds by a standard UCB bound (Lemma A.4): E h TX t=τdisc (µ⋆ −µ(O t)) τdisc i ≤O p m T′ logT 18 Algorithm 2Partition-Aware UCB under Strong Generalization 1: Initialize: R (set of discovered outcomes, initially∅); rep[o](representative arm for o, undefined until discovery); no ∈N,ˆµ o ∈R for o∈ R(both0initially); U (...

    work page 1996

  2. [6]

    Dataset reset policy optimization for rlhf.arXiv preprint arXiv:2404.08495,

    Jonathan D Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D Lee, and Wen Sun. Dataset reset policy optimization for rlhf.arXiv preprint arXiv:2404.08495,

    arXiv

  3. [7]

    Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models.arXiv preprint arXiv:2508.10751,

    Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models.arXiv preprint arXiv:2508.10751,

    arXiv

  4. [8]

    Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,

    arXiv

  5. [9]

    Weight ensembling improves reasoning in language models.arXiv preprint arXiv:2504.10478,

    Xingyu Dang, Christina Baek, Kaiyue Wen, Zico Kolter, and Aditi Raghunathan. Weight ensembling improves reasoning in language models.arXiv preprint arXiv:2504.10478,

    arXiv

  6. [10]

    The statistical complexity of interactive decision making.arXiv:2112.13487,

    Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making.arXiv:2112.13487,

    arXiv

  7. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  8. [13]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    arXiv

Show all 38 references
  1. [14]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    arXiv

  2. [15]

    Understanding the effects of rlhf on llm generalisation and diversity.arXiv preprint arXiv:2310.06452,

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity.arXiv preprint arXiv:2310.06452,

    arXiv

  3. [17]

    Jointly reinforcing diversity and quality in language model generations.arXiv preprint arXiv:2509.02534,

    Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, and Tianlu Wang. Jointly reinforcing diversity and quality in language model generations.arXiv preprint arXiv:2509.02534,

    arXiv

  4. [20]

    Approximating kl divergence, 2020.URL http://joschu

    John Schulman. Approximating kl divergence, 2020.URL http://joschu. net/blog/kl-approx. html,

    2020

  5. [22]

    Can large reasoning models self-train?arXiv preprint arXiv:2505.21444,

    Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train?arXiv preprint arXiv:2505.21444,

  6. [23]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    arXiv

  7. [24]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    arXiv

  8. [25]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    arXiv

  9. [26]

    The importance of online data: Understanding preference fine-tuning via coverage.Advances in Neural Information Processing Systems, 37:12243–12270, 2024a

    Yuda Song, Gokul Swamy, Aarti Singh, J Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage.Advances in Neural Information Processing Systems, 37:12243–12270, 2024a. Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade, Dean F...

    arXiv

  10. [27]

    On a few pitfalls in kl divergence gradient estimation for rl.arXiv preprint arXiv:2506.09477,

    Yunhao Tang and Rémi Munos. On a few pitfalls in kl divergence gradient estimation for rl.arXiv preprint arXiv:2506.09477,

    arXiv

  11. [28]

    Optimizing language models for inference time objectives using reinforcement learning.arXiv preprint arXiv:2503.19595,

    Yunhao Tang, Kunhao Zheng, Gabriel Synnaeve, and Rémi Munos. Optimizing language models for inference time objectives using reinforcement learning.arXiv preprint arXiv:2503.19595,

    arXiv

  12. [29]

    The invisible leash: Why rlvr may not escape its origin.arXiv preprint arXiv:2507.14843,

    Fang Wu, Weihao Xuan, Ximing Lu, Zaid Harchaoui, and Yejin Choi. The invisible leash: Why rlvr may not escape its origin.arXiv preprint arXiv:2507.14843,

  13. [30]

    Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf.arXiv preprint arXiv:2405.21046,

    Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf.arXiv preprint arXiv:2405.21046,

    arXiv

  14. [31]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint.arXiv preprint arXiv:2312.11456,

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint.arXiv preprint arXiv:2312.11456,

    arXiv

  15. [33]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    arXiv

  16. [34]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    arXiv

  17. [35]

    The price of format: Diversity collapse in llms.arXiv preprint arXiv:2505.18949,

    Longfei Yun, Chenyang An, Zilong Wang, Letian Peng, and Jingbo Shang. The price of format: Diversity collapse in llms.arXiv preprint arXiv:2505.18949,

    arXiv

  18. [36]

    Self-exploring language models: Active preference elicitation for online alignment.arXiv preprint arXiv:2405.19332, 2024a

    Shenao Zhang, Donghan Yu, Hiteshi Sharma, Han Zhong, Zhihan Liu, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment.arXiv preprint arXiv:2405.19332, 2024a. Zihan Zhang, Yuxin Chen, Jason D...

    arXiv

  19. [37]

    First return, entropy-eliciting explore.arXiv preprint arXiv:2507.07017,

    Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, et al. First return, entropy-eliciting explore.arXiv preprint arXiv:2507.07017,

    arXiv

  20. [1996]

    Optimistically optimistic exploration for provably efficient infinite- horizon reinforcement and imitation learning.arXiv preprint arXiv:2502.13900,

    Antoine Moulin, Gergely Neu, and Luca Viano. Optimistically optimistic exploration for provably efficient infinite- horizon reinforcement and imitation learning.arXiv preprint arXiv:2502.13900,

  21. [2002]

    Exploration by random network distillation.arXiv preprint arXiv:1810.12894,

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation.arXiv preprint arXiv:1810.12894,

    arXiv

  22. [2016]

    Diverse preference optimization.arXiv preprint arXiv:2501.18101,

    Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Diverse preference optimization.arXiv preprint arXiv:2501.18101,

    arXiv

  23. [2018]

    Value-incentivized preference optimization: A unified approach to online and offline rlhf.arXiv preprint arXiv:2405.19320,

    Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline rlhf.arXiv preprint arXiv:2405.19320,

    arXiv

  24. [2019]

    Qwen2.5 technical report.ArXiv, abs/2412.15115,

    14 Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Ba...

    arXiv

  25. [2020]

    e3: Learning to explore enables extrapolation of test-time compute for llms.arXiv preprint arXiv:2506.09026,

    Amrith Setlur, Matthew YR Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for llms.arXiv preprint arXiv:2506.09026,

    arXiv

  26. [2021]

    Navigate the unknown: Enhancing llm reasoning with intrinsic motivation guided exploration.arXiv preprint arXiv:2505.17621,

    Jingtong Gao, Ling Pan, Yejing Wang, Rui Zhong, Chi Lu, Qingpeng Cai, Peng Jiang, and Xiangyu Zhao. Navigate the unknown: Enhancing llm reasoning with intrinsic motivation guided exploration.arXiv preprint arXiv:2505.17621,

  27. [2022]

    Attributing mode collapse in the fine-tuning of large language models

    13 Laura O’Mahony, Leo Grinsztajn, Hailey Schoelkopf, and Stella Biderman. Attributing mode collapse in the fine-tuning of large language models. InICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, volume 2,

    2024

  28. [2023]

    Asymmetric rein- force for off-policy reinforcement learning: Balancing positive and negative rewards.arXiv preprint arXiv:2506.20520,

    Charles Arnal, GaĂŤtan Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, and Remi Munos. Asymmetric rein- force for off-policy reinforcement learning: Balancing positive and negative rewards.arXiv preprint arXiv:2506.20520,

  29. [2024]

    Online preference alignment for language models via count-based exploration.arXiv preprint arXiv:2501.12735,

    Chenjia Bai, Yang Zhang, Shuang Qiu, Qiaosheng Zhang, Kang Xu, and Xuelong Li. Online preference alignment for language models via count-based exploration.arXiv preprint arXiv:2501.12735,

    arXiv

  30. [2025]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.