Pith. sign in

REVIEW 3 major objections 5 minor 9 cited by

Hidden-state diversity bonus improves LLM reasoning and triples sample efficiency.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Representation-based elliptical bonuses improve inference-time and post-training pass@k for LLM reasoning, but the headline AIME result is tainted by validation/test overlap.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Inference-time selection with representation-based bonuses is a genuine contribution, but the post-training headline on AIME leaks test labels through validation and relies on an unablated no-bonus rule, so the efficiency claims are not yet attributable. the 3 major comments →

arxiv 2510.11686 v2 pith:CJZXT6KR submitted 2025-10-13 cs.LG cs.AI

Representation-Based Exploration for Language Models: From Test-Time to Post-Training

Jens Tuyls , Dylan J. Foster , Akshay Krishnamurthy , Jordan T. Ash This is my paper
classification cs.LG cs.AI
keywords representation-based explorationelliptical bonuseslanguage model reasoningpass@kGRPOinference-time selectionRL post-trainingdiversity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that deliberate exploration—explicitly steering a language model toward novel and diverse responses—can do more than sharpen behaviors the base model already has. It argues that a simple, principled diversity bonus computed from the model's own hidden states, an elliptical bonus of the form h^T Σ^{-1}h, substantially improves both inference-time selection and reinforcement-learning post-training. If true, this would mean the knowledge encoded in pre-trained representations can guide the search for new reasoning behaviors, and that exploration with the right notion of diversity is a practical path beyond standard RL sharpening. A sympathetic reader would care because the method is parameter-light, needs no auxiliary networks, and directly targets the verifier-efficiency bottleneck in reasoning tasks.

Core claim

The central claim is that representation-based exploration delivers consistent gains in both test-time and post-training settings. At inference time, the algorithm samples a large pool of responses to a prompt, embeds each response by mean-pooling the model's last-layer hidden states, and iteratively selects responses that maximize an elliptical bonus h^T Σ^{-1}h, which is large for responses that are poorly explained by previously selected ones. Across a range of models and tasks, this selection improves verifier efficiency—the expected number of verifier queries before a correct answer is found—by over 50% for a 14-billion-parameter instruction-tuned model on several math and coding benchm

What carries the argument

The central object is the elliptical bonus div(h | h_{1:i-1}) = h^T Σ^{-1} h, with Σ = λI + Σ_{j<i} h_j h_j^T. The paper's RepExp algorithm iteratively selects, from a large candidate pool, the response whose mean-pooled last-layer hidden state maximizes this bonus, then updates Σ^{-1} via a rank-one Woodbury update. The feature vector h is the pre-trained model's last-layer hidden states averaged over tokens, mean-centered, and projected to 512 dimensions via a sparse random projection. The bonus is history-aware: it penalizes responses whose representations are redundant with those already selected, grounding novelty in the model's own internal geometry rather than in surface text statisti

Load-bearing premise

The load-bearing premise is that the mean-pooled last-layer hidden states, after random projection, faithfully reflect behavioral diversity: responses that are far apart in this representation space are assumed to correspond to distinct solution strategies with non-overlapping chances of being correct.

What would settle it

Take a strong model on a hard reasoning task, generate a large candidate pool, and compute the correctness rate within clusters of hidden-state representations. If clusters with very different embedding positions have nearly identical correctness rates—or if a high-temperature pool produces responses that look novel but are all wrong—then the elliptical bonus cannot beat random selection; a direct head-to-head of RepExp versus random selection with the same verification budget, measured as samples-to-correct per difficulty bin, would settle the claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Representation-based diversity selection improves verifier efficiency over random sampling across models and tasks, with the largest gains on the hardest questions and on stronger models.
  • Adding the same bonus to GRPO rewards preserves or improves pass@k at large k, eliminating the 'diversity collapse' where standard RL degrades pass@k relative to the base model.
  • On AIME 2024, post-training with the bonus yields pass@80 matching pass@256 of standard GRPO, a 3.2x sample-efficiency improvement.
  • A token-level variant of the bonus, applied to logits during autoregressive generation, improves solve rates for large budgets on the hardest math questions, though the current implementation is not wall-clock competitive.
  • Weak models can see no benefit or degradation, so gains depend on the representation quality of the base model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If hidden-state distance tracks solution-strategy diversity, the same selection scheme could cut annotation costs in domains where verifiers are expensive or human, by choosing which candidate responses to show an expert.
  • The method's reliance on base-model representations suggests a testable extension: representation fine-tuning or stronger base models should extend the sample-efficiency gains to regimes where the paper finds no benefit.
  • The inference-time selection framework could be used as a cheap, confound-free screening test for other diversity bonuses (entropy, unlikeliness, determinantal point processes) before committing to expensive RL runs.
  • Combining representation bonuses with prolonged RL or reference-model resetting, which the paper lists as complementary, may compound the diversity gains rather than saturate them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Representation-Based Exploration (RepExp), which uses an elliptical bonus h^T Σ^{-1} h computed from mean-pooled, randomly projected last-layer hidden states of a pretrained language model to select diverse responses (inference-time setting) and to augment GRPO rewards during RL post-training. In the inference-time setting, RepExp filters a large candidate pool down to k responses and is compared with random sampling in terms of verifier efficiency. In post-training, RepExp adds a within-batch diversity bonus to verifier rewards. The main empirical claims are consistent verifier-efficiency improvements across many models and tasks for inference-time exploration, and, for post-training, that RepExp preserves or improves pass@k at large k where standard GRPO collapses, with a headline 3.2x sample-efficiency improvement over GRPO on AIME 2024.

Significance. If the results hold, the paper makes a useful contribution: a simple, scalable exploration bonus that leverages pretrained representations, an inference-time testbed for isolating exploration from optimization, and evidence that diversity bonuses can mitigate 'diversity collapse' in RL post-training. Strengths include the breadth of models and tasks, the public code/website, the clean inference-time protocol, and the explicit comparison with an Unlikeliness baseline. However, the post-training AIME result—the paper's headline efficiency claim—is compromised by the use of AIME 2024 as both validation and test for checkpoint selection, and by an unablated post-hoc rule that disables the bonus exactly when all rollouts in a batch are incorrect. These issues must be resolved before the central post-training claims can be accepted.

major comments (3)
  1. [Section C, Dataset splits / Checkpoint picking] For AIME 2024, the paper states: 'we use the full AIME 2024 dataset both for validation and testing' (Section C, Dataset splits), and checkpoint selection is described as picking 'the checkpoint during training that achieves the highest pass@1 on the respective task's validation set' (Section C, Checkpoint picking). Since the validation set for AIME 2024 is the test set, the reported pass@k curves and the headline 3.2x sample-efficiency gain over GRPO on AIME 2024 are not from a held-out evaluation. This is a load-bearing leak: the comparison is effectively selecting checkpoints on test labels. The authors need to use a disjoint validation set (e.g., AIME 2025 or a held-out subset) and re-report the AIME results.
  2. [Section C, Algorithm details / Table 3] The post-training implementation includes a rule not present in Algorithm 1 or the Section 3 reward definition: 'we do not add a bonus for questions where all rollouts in the batch are incorrect, as we found this to empirically hurt performance' (Section C, Algorithm details; Table 3). This is an unablated, post-hoc modification that suppresses the exploration bonus precisely in the regime where deliberate exploration is claimed to matter. As written, the evaluated algorithm is not 'representation-based exploration' but 'representation-based diversity bonus only when at least one rollout in the batch is correct.' The authors must ablate this rule and show how much of the reported pass@k improvement is attributable to the representation bonus rather than to this exception.
  3. [Section 3 vs. Section 5] Section 3 motivates elliptical bonuses through cumulative uncertainty and lists 'history-aware' as a desirable property, but Section 5 states that in post-training 'the covariance matrix Σ used to compute bonuses is re-initialized for each batch of RL training' and that previously generated sequences for a prompt are not considered. The post-training implementation is therefore a batch-local diversity term, not the cumulative exploration bonus described and motivated in Section 3. This discrepancy should be addressed either by aligning the description with the implementation or by providing an ablation/justification for the batch-local choice, since it changes the interpretation of the central mechanism.
minor comments (5)
  1. [Section 5 vs. Section C] Section 5 says for AIME 2024 the authors 'use the DAPO-Math-17K dataset for training, leaving AIME 2024 for evaluation only,' but Section C says the full AIME 2024 dataset is used 'both for validation and testing.' These statements are inconsistent and should be reconciled.
  2. [Table 3] The Table 3 header reads 'Score' but the column contains hyperparameter values; this should be 'Value' or similar. Also, the 'No bonus if all rollouts incorrect' row should be clearly marked as a post-hoc modification and, ideally, accompanied by an ablation (see major comment).
  3. [Section 3 vs. Section C] Section 3 says dimensionality is reduced to 512 for the representation features, while post-training uses a sparse projection dimension of 32 (and 128 for AIME, per Section C). The paper should clarify that the projection dimension differs between the inference-time and post-training settings.
  4. [Section 4] The text says 'We detail the experimental setup in Section 4,' but the detailed setup appears in Section B; the cross-reference should be fixed.
  5. [Figure 7/8] In the right-hand panels, the 'Solve rate lift' is plotted relative to vanilla generation, but the caption does not state the reference baseline explicitly. A brief mention in the caption would improve clarity.

Circularity Check

0 steps flagged

No significant circularity: pass@k and verifier-efficiency results are independently measured; the unablated no-bonus rule and AIME validation/test overlap are attribution risks, not circular reductions.

full rationale

The claimed derivation chain is: featurize responses with mean-pooled last-layer hidden states, define an elliptical leverage-score bonus h^T Sigma^{-1} h, use it for inference-time coreset selection or as an additive GRPO reward term, then measure pass@k / samples-to-correct with an external verifier. The target quantities in the evaluation (Eq. (1), Section B.1) are defined from the verifier reward r*(x,y), not from the representation h_theta(x,y), so the central results do not reduce to the method's own definitions. The bonus is not fitted to the pass@k labels, and no fitted parameter is renamed as a predicted quantity. The theoretical grounding in Section 3 is a standard linear-regression uncertainty bound, and the main evidence is empirical comparison against random selection, GRPO, and Unlikeliness. I considered the main flagged weaknesses: Section C states 'we do not add a bonus for questions where all rollouts in the batch are incorrect, as we found this to empirically hurt performance,' and AIME 2024 is used for both validation and testing (Section C, Dataset splits). These are unablated post-hoc/selection-on-test issues that weaken attribution of the post-training gains to the representation bonus, but they are overfitting/validity concerns, not circular reductions: no equation is equivalent to an input by construction, and no prediction is statistically forced by the fitted rule. Self-citations (Foster et al. 2025; Huang et al. 2025) are motivational or corroborated by independent external work, and they are not load-bearing for the empirical pass@k claims.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The method relies on a handful of hyperparameters (λ, β, projection dimension) and a post-hoc rule (no bonus for zero-correct batches). No new theoretical entities are introduced.

free parameters (4)
  • λ (regularization) = 1.0 (inference-time), 0.1 (token-level)
    Controls inverse-covariance initialization; chosen by hand without systematic tuning, affects diversity selection.
  • β (bonus coefficient) = 0.01 (post-training); 0.5, 1.0 (token-level)
    Scales the representation bonus; chosen to balance extrinsic reward and bonus, not swept.
  • Sparse projection dimension = 512 (inference-time), 32 (post-training), 128 (AIME post-training)
    Dimensionality reduction for hidden states; chosen for computational efficiency and affects diversity geometry.
  • No-bonus rule for all-incorrect batches = True
    Authors disable the bonus when all rollouts in a batch are incorrect, saying this 'empirically hurt performance'; a post-hoc rule that improves reported results.
axioms (5)
  • domain assumption Last-layer hidden states of the pretrained LM are informative features for measuring behavioral diversity.
    Central to RepExp; stated in Section 3 and tested only via pooling ablations (Figure 4), not against alternative feature spaces.
  • domain assumption Elliptical bonus h^T Σ^{-1} h correctly quantifies novelty in representation space.
    Borrowed from linear bandits (Section 3); for nonlinear LLM representations this is heuristic, and the paper itself notes it can fail for high-temperature samples (Section 4.1, RF4).
  • domain assumption Mean-pooling token representations preserves enough information for diversity.
    Chosen without theoretical justification; ablation in Figure 4 compares pooling schemes but not the pooling assumption itself.
  • standard math Sparse random projection preserves the geometry needed for selection.
    Justified by JL-type guarantees (Li et al., 2006), but projection dimension is chosen by hand.
  • ad hoc to paper Within-batch covariance is sufficient for post-training exploration; ignoring history across batches does not hurt.
    Section 5: covariance is re-initialized per batch, so bonuses only measure novelty relative to the current group of rollouts; this practical choice has no supporting analysis.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Representation-Based Exploration for Language Models: From Test-Time to Post-Training." pith.science (2026). https://pith.science/paper/CJZXT6KR

@misc{pith2026251011686,
  author       = {Pith},
  title        = {Pith review of: Representation-Based Exploration for Language Models: From Test-Time to Post-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CJZXT6KR}},
  note         = {Machine review of arXiv:2510.11686}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement learning (RL) promises to expand the capabilities of language models, but it is unclear if current RL techniques promote the discovery of novel behaviors, or simply sharpen those already present in the base model. In this paper, we investigate the value of deliberate exploration -- explicitly incentivizing the model to discover novel and diverse behaviors -- and aim to understand how the knowledge in pre-trained models can guide this search. Our main finding is that exploration with a simple, principled, representation-based bonus derived from the pre-trained language model's hidden states significantly improves diversity and pass@k rates -- both for post-training, and in a novel inference-time scaling setting we introduce. For inference-time, exploration with representation-based diversity improves efficiency, consistently improving pass@k rates across a variety of models and reasoning tasks. For example, for Qwen-2.5-14b-Instruct we obtain over 50% improvement in verifier efficiency on almost all tasks. For post-training, we show that integrating this exploration strategy into an RL pipeline improves reasoning performance over that of the initial model and over standard RL post-training. For example, on AIME 2024, our post-trained Qwen-2.5-7b-Instruct's pass@80 matches the pass@256 of GRPO on the same model, demonstrating a 3x improvement in test-time sample efficiency. Overall, our findings suggest that deliberate exploration -- with the right notion of diversity -- is a practical path toward discovery of new behaviors beyond sharpening.

Figures

Figures reproduced from arXiv: 2510.11686 by Akshay Krishnamurthy, Dylan J. Foster, Jens Tuyls, Jordan T. Ash.

Figure 1
Figure 1. Figure 1: Representation-based inference-time exploration improves verifier efficiency. (Left) We plot the samples-to-correct, the average number of samples until a correct response is selected, for a wide range of tasks and models. We compare two inference-time exploration methods: representation￾based exploration (Section 3) and naive (random) sampling from the base model. (Right) We display samples-to-correct, di… view at source ↗
Figure 2
Figure 2. Figure 2: Pass@k for RL post-training with exploration. We find that RL generally increases the pass@k for small values of k compared to the base model, but that exploration is required to improve or even preserve base model pass rates for larger values of k. For MATH and GSM8K, RepExp roughly matches or improves upon Unlikeliness for k ≥ 2. For AIME 2024, RepExp is slightly worse than Unlikeliness until k = 64, aft… view at source ↗
Figure 3
Figure 3. Figure 3: RepExp for inference-time exploration. Given a prompt, RepExp selects a diverse set of responses from a large pool by optimizing elliptical bonuses computed using representations from the language model. 2.2 Reinforcement learning post-training As described earlier, RL post-training (e.g., with GRPO or PPO) proceeds by iteratively sampling batches of responses, querying the verifier, and using the feedback… view at source ↗
Figure 4
Figure 4. Figure 4: Representation ablation. We compare averaging all token representations to using those from the penultimate or final token. Averaging is over 2x more sample efficient. representation derived from the language model’s hidden states.3 This approach is arguably the simplest principled strategy that is appropriate for language models, and already yields significant performance improvements in our experiments. … view at source ↗
Figure 5
Figure 5. Figure 5: A closer look into when RepExp provides improvement. (Left) For each task, we rank models according to their pass@1 rate (the weakest model has rank 0, and the strongest has rank 8). We then plot relative improvement (%) of RepExp over random sampling, sorting by rank on the x-axis. While RepExp can hurt weaker models (e.g., Qwen-2.5-0.5B-Instruct), we find stronger models almost always benefit from explor… view at source ↗
Figure 6
Figure 6. Figure 6: Benefits of RepExp across data pools, for the inference-time exploration setup in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Representation-based exploration at the token level, compared to naive autoregressive generation (β = 0) for inference-time exploration. (Left) We plot pass@k for different exploration methods on Qwen-2.5-7B-Instruct, for the 200 hardest (but solvable) questions in MATH as judged by GPT-4o-mini. RepExp improves pass@k for large k over naïve sampling (for both choices of β). (Right) When binning the questio… view at source ↗
Figure 8
Figure 8. Figure 8: Representation-based exploration at the token level, compared to naive autoregressive generation (β = 0) for inference-time exploration. (Left) We added shaded areas indicating one standard error to the left side of [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Pass@k for RL post-training with exploration. We plot [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models

    cs.CL 2026-07 conditional novelty 7.0

    LLM probability estimates violate the law of total probability across partitions, and subgroup-aggregated estimates often beat direct population-level estimates (the macro fallacy).

  2. OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation

    cs.LG 2026-06 unverdicted novelty 7.0

    OrderGrad supplies unbiased likelihood-ratio and reparameterization gradient estimators for finite-sample L-statistics by applying a rank-based reward transformation usable in standard policy-gradient updates.

  3. The tractability landscape of diffusion alignment: regularization, rewards, and computational primitives

    cs.LG 2026-05 unverdicted novelty 7.0

    The choice of closeness measure in diffusion reward alignment determines the computational primitives and tractable reward classes, with linear exponential tilts sufficing for KL with convex rewards and proximal oracl...

  4. On Advantage Estimates for Max@K Policy Gradients

    cs.LG 2026-06 unverdicted novelty 6.0

    Proposes MaxPO using a Leave-Two-Out baseline for centered unbiased advantages in max@K policy gradients, with a unified derivation of finite-batch estimators.

  5. Data-dependent Exploration for Online Reinforcement Learning from Human Feedback

    cs.LG 2026-05 unverdicted novelty 6.0

    DEPO uses historical data to build a data-dependent uncertainty bonus for exploration in online RLHF, yielding an adaptive regret bound and stronger empirical performance than baselines.

  6. Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution

    cs.LG 2026-02 unverdicted novelty 6.0

    PEPO uses pessimistic ensembling of DPO policies on data subsets to achieve single-policy concentrability sample bounds and avoid over-optimization in tabular settings.

  7. Data-dependent Exploration for Online Reinforcement Learning from Human Feedback

    cs.LG 2026-05 unverdicted novelty 5.0

    DEPO constructs uncertainty bonuses from historical data for exploration in online RLHF and provides a data-dependent regret bound that adapts to task hardness.

  8. The Role of Generator Access in Autoregressive Post-Training

    cs.LG 2026-04 unverdicted novelty 5.0

    Limited generator access in autoregressive post-training confines learners to root-start rollouts whose value is bounded by on-policy prefix probabilities, while weak prefix control unlocks richer observations and pro...

  9. Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution

    cs.LG 2026-02 unverdicted novelty 5.0

    PEPO is a single-step pessimistic ensemble algorithm for direct preference optimization that provably avoids over-optimization by depending only on single-policy concentrability without knowing the data distribution o...

Reference graph

Works this paper leans on

49 extracted references · 33 linked inside Pith · cited by 7 Pith papers

  1. [3]

    Note that forAIME 2024 , we adjusted the maximum prompt length to 2048, the maximum response length to 8192, the train batch size to 512, the ppo mini batch size to 128, and the ppo micro batch size per gpu to

    2024

  2. [4]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    Pith/arXiv arXiv

  3. [5]

    Online preference alignment for language models via count-based exploration.arXiv preprint arXiv:2501.12735,

    Chenjia Bai, Yang Zhang, Shuang Qiu, Qiaosheng Zhang, Kang Xu, and Xuelong Li. Online preference alignment for language models via count-based exploration.arXiv preprint arXiv:2501.12735,

    Pith/arXiv arXiv

  4. [6]

    Infalign: Inference-aware language model alignment.arXiv preprint arXiv:2412.19792,

    Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. Infalign: Inference-aware language model alignment.arXiv preprint arXiv:2412.19792,

    Pith/arXiv arXiv

  5. [8]

    Also, forRepExp on AIME 2024 , we increased the sparse projection dimension from 32 to

    2024

  6. [9]

    Pad: Personalized alignment at decoding-time.arXiv:2410.04070,

    Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. Pad: Personalized alignment at decoding-time.arXiv:2410.04070,

    Pith/arXiv arXiv

  7. [10]

    Enhancing diversity in large language models via determinantal point processes.arXiv preprint arXiv:2509.04784, 2025a

    12 Yilei Chen, Souradip Chakraborty, Lorenz Wolf, Ioannis Ch Paschalidis, and Aldo Pacchiano. Enhancing diversity in large language models via determinantal point processes.arXiv preprint arXiv:2509.04784, 2025a. Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@k training for adaptively balancing exploration a...

    arXiv

  8. [12]

    Uniform sampling for matrix approximation

    Michael B Cohen, Yin Tat Lee, Cameron Musco, Christopher Musco, Richard Peng, and Aaron Sidford. Uniform sampling for matrix approximation. InProceedings of the 2015 conference on innovations in theoretical computer science, pages 181–190,

    2015

  9. [13]

    Weight ensembling improves reasoning in language models.arXiv:2504.10478,

    Xingyu Dang, Christina Baek, Kaiyue Wen, Zico Kolter, and Aditi Raghunathan. Weight ensembling improves reasoning in language models.arXiv:2504.10478,

    arXiv

  10. [15]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307,

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307,

    Pith/arXiv arXiv

  11. [16]

    Navigate the unknown: Enhancing llm reasoning with intrinsic motivation guided exploration.arXiv preprint arXiv:2505.17621,

    Jingtong Gao, Ling Pan, Yejing Wang, Rui Zhong, Chi Lu, Qingpeng Cai, Peng Jiang, and Xiangyu Zhao. Navigate the unknown: Enhancing llm reasoning with intrinsic motivation guided exploration.arXiv preprint arXiv:2505.17621,

    arXiv

  12. [18]

    Large-scale data selection for instruction tuning.arXiv preprint arXiv:2503.01807,

    Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning.arXiv preprint arXiv:2503.01807,

    Pith/arXiv arXiv

  13. [21]

    Regularized best-of-n sampling to mitigate reward hacking for language model alignment.arXiv:2404.01054,

    Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, and Kenshi Abe. Regularized best-of-n sampling to mitigate reward hacking for language model alignment.arXiv:2404.01054,

    Pith/arXiv arXiv

  14. [22]

    Args: Alignment as reward-guided search

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. arXiv:2402.01694,

    Pith/arXiv arXiv

  15. [23]

    Diverse preference optimization.arXiv preprint arXiv:2501.18101,

    Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Diverse preference optimization.arXiv preprint arXiv:2501.18101,

    Pith/arXiv arXiv

  16. [24]

    Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, and Tianlu Wang

    URLhttps://api.semanticscholar.org/CorpusID:7995734. Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, and Tianlu Wang. Jointly reinforcing diversity and quality in language model generations.arXiv preprint arXiv:2509.02534,

    Pith/arXiv arXiv

  17. [25]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    Pith/arXiv arXiv

  18. [26]

    Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.arXiv preprint arXiv:2505.24864,

    14 Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.arXiv preprint arXiv:2505.24864,

    Pith/arXiv arXiv

  19. [27]

    Decoding-time realignment of language models.arXiv:2402.02992, 2024a

    Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-time realignment of language models.arXiv:2402.02992, 2024a. Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin. Sample-efficient alignment for llms.arXiv preprint arXiv:241...

    Pith/arXiv arXiv

  20. [28]

    Proximal policy optimization algorithms.arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv:1707.06347,

    Pith/arXiv arXiv

  21. [30]

    e3: Learning to explore enables extrapolation of test-time compute for llms.arXiv preprint arXiv:2506.09026,

    Amrith Setlur, Matthew YR Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for llms.arXiv preprint arXiv:2506.09026,

    Pith/arXiv arXiv

  22. [31]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  23. [32]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Pith/arXiv arXiv

  24. [33]

    Decoding- time language model alignment with multiple objectives.arXiv:2406.18853, 2024a

    Ruizhe Shi, Yifang Chen, Yushi Hu, ALisa Liu, Noah Smith, Hannaneh Hajishirzi, and Simon Du. Decoding- time language model alignment with multiple objectives.arXiv:2406.18853, 2024a. Ruizhe Shi, Runlong Zhou, and Simon S Du. The crucial role of samplers in online direct preference optimization.arXiv preprint arXiv:2409.19605, 2024b. 15 Yuda Song, Julia Ke...

    Pith/arXiv arXiv

  25. [35]

    The invisible leash: Why rlvr may not escape its origin.arXiv preprint arXiv:2507.14843,

    Fang Wu, Weihao Xuan, Ximing Lu, Zaid Harchaoui, and Yejin Choi. The invisible leash: Why rlvr may not escape its origin.arXiv preprint arXiv:2507.14843,

    arXiv

  26. [36]

    Exploratory preference optimization: Harnessing implicit Q*-approximation for sample-efficient RLHF.arXiv:2405.21046,

    Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit Q*-approximation for sample-efficient RLHF.arXiv:2405.21046,

    Pith/arXiv arXiv

  27. [37]

    Provably learning from language feedback.arXiv preprint arXiv:2506.10341,

    Wanqiao Xu, Allen Nie, Ruijie Zheng, Aditya Modi, Adith Swaminathan, and Ching-An Cheng. Provably learning from language feedback.arXiv preprint arXiv:2506.10341,

    Pith/arXiv arXiv

  28. [38]

    Depth-breadth synergy in rlvr: Unlocking llm reasoning gains with adaptive exploration

    Zhicheng Yang, Zhijiang Guo, Yinya Huang, Yongxin Wang, Dongchun Xie, Yiwei Wang, Xiaodan Liang, and Jing Tang. Depth-breadth synergy in rlvr: Unlocking llm reasoning gains with adaptive exploration. arXiv preprint arXiv:2508.13755,

    Pith/arXiv arXiv

  29. [39]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  30. [40]

    Does rein- forcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does rein- forcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Pith/arXiv arXiv

  31. [41]

    Expo: Unlocking hard reasoning with self-explanation- guided reinforcement learning.arXiv preprint arXiv:2507.02834,

    Ruiyang Zhou, Shuozhe Li, Amy Zhang, and Liu Leqi. Expo: Unlocking hard reasoning with self-explanation- guided reinforcement learning.arXiv preprint arXiv:2507.02834,

    arXiv

  32. [42]

    Most closely related to our work, Setlur et al

    16 A Additional Related Work Exploration at test time.Test-time alignment techniques for language models are an active area of research with many complementary threads (Khanov et al., 2024; Chen et al., 2024; Shi et al., 2024a; Liu et al., 2024a; Jinnai et al., 2024; Shi et al., 2024b), but exploration has not typically been the focus of this line of work...

    2024

  33. [43]

    breadth" (batch size) and “depth

    formulate the problem of directly post-training to maximize the pass@N objective, deriving approximate gradient estimators and using them for policy optimization. As discussed in Chow et al. (2025); Chen et al. (2025b), these gradient estimatorsimplicitlyencourage exploration, since they allow the model to distribute probability mass across a more diverse...

    2025

  34. [45]

    Since the dataset does not come with any train or test splits, we use the full set of questions for our experiments

    dataset with more test cases. Since the dataset does not come with any train or test splits, we use the full set of questions for our experiments. For each problem, we generate 6400responses using vanilla settings, except that we settop-p = 0.95. We set the maximum response length per generation to768tokens. •Game of 24 . This dataset contains 1.36k quest...

    2025

  35. [46]

    1− n−c k n k # = 1 |D| |D|X i=1

    on1− 2(depending on the size of the model) NVIDIA A100 40GB GPUs per model-task pair to generate the data pools for all questions in the dataset. Estimating samples-to-correct.For random sampling, we estimate the average number of samples to take (without replacement) from the data pool to find the first correct one as: samples-to-correct= N+ 1 c+ 1 , whe...

    2015

  36. [47]

    Hyperparameters.We use verl for training (Sheng et al., 2024), and provide a full overview of all common hyperparameters in Table 1, all hyperparameters specific to unlikeliness in Table 2, and all hyperpameters specific to RepExp in Table

    2024

  37. [50]

    for inference-time exploration.(Left)We added shaded areas indicating one standard error to the left side of Figure 7.(Right)Compared to the right side of Figure 7, we also addedβ= 0.5 . 1 32 64 96 128 160 192 224 256 k 0.70 0.75 0.80 0.85 0.90 0.95Pass@k ×5.6 ×4.1 MATH 1 32 64 96 128 160 192 224 256 k 0.93 0.94 0.95 0.96 0.97 0.98 0.99 ×13.4 ×3.0 GSM8K 1...

    2024

  38. [512]

    5See also Arumugam and Griffiths (2025), which uses a pre-trained model to simulate posterior sampling in-context for multi-turn sequential decision making tasks

    Preprocessing.In Algorithm 1, after obtaining the representations ¯hθ for every generationy for a fixed prompt x, we sparse project all representations down and then mean-center where the mean is taken across the response-level representations. 5See also Arumugam and Griffiths (2025), which uses a pre-trained model to simulate posterior sampling in-contex...

    2025

  39. [1992]

    Pass@ k policy optimization: Solving harder reinforcement learning problems.arXiv preprint arXiv:2505.15201,

    Christian Walder and Deep Karkhanis. Pass@ k policy optimization: Solving harder reinforcement learning problems.arXiv preprint arXiv:2505.15201,

    Pith/arXiv arXiv

  40. [2006]

    The llama 3 herd of models.arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv:2407.21783,

    Pith/arXiv arXiv

  41. [2010]

    Training verifiers to solve math word problems.arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv:2110.14168,

    Pith/arXiv arXiv

  42. [2011]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024a

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024a. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, ...

    Pith/arXiv arXiv

  43. [2014]

    Rewarding the unlikely: Lifting grpo beyond distribution sharpening.arXiv preprint arXiv:2506.02355,

    Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting grpo beyond distribution sharpening.arXiv preprint arXiv:2506.02355,

    Pith/arXiv arXiv

  44. [2015]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Pith/arXiv arXiv

  45. [2017]

    Rewarding progress: Scaling automated process verifiers for llm reasoning.arXiv preprint arXiv:2410.08146,

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning.arXiv preprint arXiv:2410.08146,

    Pith/arXiv arXiv

  46. [2021]

    Toward efficient exploration by large language model agents.arXiv preprint arXiv:2504.20997,

    Dilip Arumugam and Thomas L Griffiths. Toward efficient exploration by large language model agents.arXiv preprint arXiv:2504.20997,

    arXiv

  47. [2023]

    doi: 10.48550.arXiv preprint ARXIV.2310.06825, 10,

    Pith/arXiv arXiv

  48. [2024]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894,

    Pith/arXiv arXiv

  49. [2025]

    Anti-concentrated confidence bonuses for scalable exploration.arXiv preprint arXiv:2110.11202,

    Jordan T Ash, Cyril Zhang, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Anti-concentrated confidence bonuses for scalable exploration.arXiv preprint arXiv:2110.11202,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.