{"id":"0d115195-607c-4343-ab52-e8fb1e4397be","arxiv_id":"2510.11686","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Representation-based elliptical bonuses improve inference-time and post-training pass@k for LLM reasoning, but the headline AIME result is tainted by validation/test overlap.","lead":"This paper adds a diversity bonus based on a language model's internal representations to exploration, both when choosing answers to check and during reinforcement learning. It reports large gains in sample efficiency for math and code tasks, including a 3x claim on AIME.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-training RepExp disables its exploration bonus whenever all rollouts in a batch are incorrect (Sec. C, Table 3), an unablated post-hoc rule that targets exactly the regime where exploration is claimed to matter; reported pass@k gains may not be attributable to the representation bonus.","rationale":"The concern is load-bearing but does not change the overall conditional verdict. The inference-time experiments provide substantial independent evidence that representation-based selection improves verifier efficiency across many models and tasks, and the post-training MATH/GSM8K results use proper validation/test splits. However, the strongest post-training claim — especially the AIME 3.2x efficiency result — rests on a pipeline with an unablated no-bonus rule and with AIME 2024 used for both validation and test. The no-bonus rule is particularly troubling because it disables exploration exactly where the paper says exploration is needed, so the apparent \"discovery of novel behaviors\" may be an artifact of the rule rather than the representation bonus. The reader's weakest_assumption focused on whether representations faithfully capture behavioral diversity; the manuscript's broad inference-time evidence makes that assumption empirically plausible, so I flag the no-bonus rule instead. A clean ablation and a held-out AIME evaluation are required before the central claim can be accepted; until then the conditional verdict stands.","tokens_in":21624,"tokens_out":9207,"duration_ms":80084,"concrete_test":"Re-run the post-training RepExp comparison on MATH, GSM8K, and AIME 2024 with the \"no bonus if all rollouts incorrect\" rule removed, so the bonus is applied to every prompt, and with checkpoint selection done on a different held-out AIME set (e.g., AIME 2025) while evaluating on AIME 2024, holding all other hyperparameters fixed. If the pass@k curves no longer improve over GRPO or the base model, the central claim is unsupported; if they do, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is an unablated post-hoc rule in the RL post-training pipeline. Section 3 defines RepExp's reward as r*(x,y) + beta * h^T Sigma^{-1} h, but Section C and Table 3 state that in the actual experiments \"we do not add a bonus for questions where all rollouts in the batch are incorrect, as we found this to empirically hurt performance.\" This rule is not part of Algorithm 1, is not motivated by the theory in Section 3, and is not ablated. It suppresses the exploration bonus precisely when all k=8 sampled rollouts are wrong — the regime in which deliberate exploration is supposed to discover novel behaviors beyond the base model. If the bonus is harmful in this regime, the evaluated algorithm is not \"representation-based exploration\" but \"diversity bonus only when a correct answer already appears in the batch.\" The central post-training claims (eliminating diversity collapse; the 3.2x AIME sample-efficiency gain) could then be driven by this rule or its interaction with GRPO, not by the representation bonus. The concern is compounded by AIME 2024 being used both for validation-based checkpoint selection and for test evaluation (Section C, Dataset splits).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Representation-Based Exploration (RepExp), which uses an elliptical bonus h^T Σ^{-1} h computed from mean-pooled, randomly projected last-layer hidden states of a pretrained language model to select diverse responses (inference-time setting) and to augment GRPO rewards during RL post-training. In the inference-time setting, RepExp filters a large candidate pool down to k responses and is compared with random sampling in terms of verifier efficiency. In post-training, RepExp adds a within-batch diversity bonus to verifier rewards. The main empirical claims are consistent verifier-efficiency improvements across many models and tasks for inference-time exploration, and, for post-training, that RepExp preserves or improves pass@k at large k where standard GRPO collapses, with a headline 3.2x sample-efficiency improvement over GRPO on AIME 2024.","tokens_in":21975,"tokens_out":5685,"duration_ms":46971,"significance":"If the results hold, the paper makes a useful contribution: a simple, scalable exploration bonus that leverages pretrained representations, an inference-time testbed for isolating exploration from optimization, and evidence that diversity bonuses can mitigate 'diversity collapse' in RL post-training. Strengths include the breadth of models and tasks, the public code/website, the clean inference-time protocol, and the explicit comparison with an Unlikeliness baseline. However, the post-training AIME result—the paper's headline efficiency claim—is compromised by the use of AIME 2024 as both validation and test for checkpoint selection, and by an unablated post-hoc rule that disables the bonus exactly when all rollouts in a batch are incorrect. These issues must be resolved before the central post-training claims can be accepted.","major_comments":[{"comment":"For AIME 2024, the paper states: 'we use the full AIME 2024 dataset both for validation and testing' (Section C, Dataset splits), and checkpoint selection is described as picking 'the checkpoint during training that achieves the highest pass@1 on the respective task's validation set' (Section C, Checkpoint picking). Since the validation set for AIME 2024 is the test set, the reported pass@k curves and the headline 3.2x sample-efficiency gain over GRPO on AIME 2024 are not from a held-out evaluation. This is a load-bearing leak: the comparison is effectively selecting checkpoints on test labels. The authors need to use a disjoint validation set (e.g., AIME 2025 or a held-out subset) and re-report the AIME results.","section":"Section C, Dataset splits / Checkpoint picking"},{"comment":"The post-training implementation includes a rule not present in Algorithm 1 or the Section 3 reward definition: 'we do not add a bonus for questions where all rollouts in the batch are incorrect, as we found this to empirically hurt performance' (Section C, Algorithm details; Table 3). This is an unablated, post-hoc modification that suppresses the exploration bonus precisely in the regime where deliberate exploration is claimed to matter. As written, the evaluated algorithm is not 'representation-based exploration' but 'representation-based diversity bonus only when at least one rollout in the batch is correct.' The authors must ablate this rule and show how much of the reported pass@k improvement is attributable to the representation bonus rather than to this exception.","section":"Section C, Algorithm details / Table 3"},{"comment":"Section 3 motivates elliptical bonuses through cumulative uncertainty and lists 'history-aware' as a desirable property, but Section 5 states that in post-training 'the covariance matrix Σ used to compute bonuses is re-initialized for each batch of RL training' and that previously generated sequences for a prompt are not considered. The post-training implementation is therefore a batch-local diversity term, not the cumulative exploration bonus described and motivated in Section 3. This discrepancy should be addressed either by aligning the description with the implementation or by providing an ablation/justification for the batch-local choice, since it changes the interpretation of the central mechanism.","section":"Section 3 vs. Section 5"}],"minor_comments":[{"comment":"Section 5 says for AIME 2024 the authors 'use the DAPO-Math-17K dataset for training, leaving AIME 2024 for evaluation only,' but Section C says the full AIME 2024 dataset is used 'both for validation and testing.' These statements are inconsistent and should be reconciled.","section":"Section 5 vs. Section C"},{"comment":"The Table 3 header reads 'Score' but the column contains hyperparameter values; this should be 'Value' or similar. Also, the 'No bonus if all rollouts incorrect' row should be clearly marked as a post-hoc modification and, ideally, accompanied by an ablation (see major comment).","section":"Table 3"},{"comment":"Section 3 says dimensionality is reduced to 512 for the representation features, while post-training uses a sparse projection dimension of 32 (and 128 for AIME, per Section C). The paper should clarify that the projection dimension differs between the inference-time and post-training settings.","section":"Section 3 vs. Section C"},{"comment":"The text says 'We detail the experimental setup in Section 4,' but the detailed setup appears in Section B; the cross-reference should be fixed.","section":"Section 4"},{"comment":"In the right-hand panels, the 'Solve rate lift' is plotted relative to vanilla generation, but the caption does not state the reference baseline explicitly. A brief mention in the caption would improve clarity.","section":"Figure 7/8"}],"recommendation":"major_revision","confidential_remarks":"The inference-time results are broad and appear to support the paper's representation-based diversity mechanism. The post-training evaluation, however, has two serious problems: (1) AIME 2024 is used for both validation and test in checkpoint selection, invalidating the headline efficiency comparison on that task; and (2) the unablated 'no bonus if all rollouts are incorrect' rule changes the algorithm relative to the paper's own description. Both are fixable within the manuscript's scope, but the authors should rerun the AIME experiment with a disjoint validation set and ablate the no-bonus rule before the central post-training claims can be considered reliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, the inference-time selection results look genuine and useful. RepExp, an elliptical bonus built from mean-pooled hidden states with a sparse random projection, consistently improves pass@k and samples-to-correct over random selection for strong models on MATH, GSM8K, MBPP+, and Game-of-24. That is the real contribution. Second, the post-training headline—pass@80 matching GRPO's pass@256 on AIME 2024—does not hold up as evidence for the method, because the evaluation leaks test information and the reward rule suppresses the bonus precisely in the regime where exploration is supposed to matter.\n\nThe two-pronged protocol—validating exploration in an inference-time coreset setting before plugging it into RL—is a nice way to isolate the diversity mechanism from optimization. The token-level generation bonus is a plausible extension, though explicitly preliminary. The pooling ablation (mean vs. last token) is honest and aligns with intuition.\n\nNow the soft spots, both in the RL post-training section. Section C says AIME 2024 is used for both validation and testing. Since checkpoints are selected on validation pass@1, this is test-label leakage; over 30 questions, the reported 3.2x efficiency gain is not reliable. A held-out split, e.g., AIME 2025, would settle it. Also, the same section states: \"we do not add a bonus for questions where all rollouts in the batch are incorrect, as we found this to empirically hurt performance.\" This rule is not in Algorithm 1, not motivated by the theory, and not ablated. It disables exploration when all eight rollouts are wrong—exactly when novel behaviors are needed. The evaluated algorithm is therefore closer to \"diversity bonus only when a correct answer already appears in the batch\" than to representation-based exploration. Both issues are fixable, but they mean the post-training gains cannot be cleanly attributed to the bonus.\n\nThe representational premise is also under-tested. The paper claims hidden-state distance tracks behavioral diversity, but only ablates pooling schemes; it never checks whether distant responses correspond to distinct solution strategies. The weak-model and high-temperature failures in Figures 5–6 are consistent with the premise but also bound its scope.\n\nOverall, this is a thoughtful paper with a solid inference-time contribution and a post-training section that needs rework before its strongest claims are accepted. It deserves serious referee time, not desk rejection. I would send it to peer review with a request for a clean AIME evaluation and an ablation of the no-bonus rule. The core direction is promising enough to warrant that effort.","headline":"Inference-time selection with representation-based bonuses is a genuine contribution, but the post-training headline on AIME leaks test labels through validation and relies on an unablated no-bonus rule, so the efficiency claims are not yet attributable.","tokens_in":22405,"tokens_out":2911,"would_cite":true,"duration_ms":26448,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hidden-state diversity bonus improves LLM reasoning and triples sample efficiency.","keywords":["representation-based exploration","elliptical bonuses","language model reasoning","pass@k","GRPO","inference-time selection","RL post-training","diversity"],"falsifier":"Take a strong model on a hard reasoning task, generate a large candidate pool, and compute the correctness rate within clusters of hidden-state representations. If clusters with very different embedding positions have nearly identical correctness rates—or if a high-temperature pool produces responses that look novel but are all wrong—then the elliptical bonus cannot beat random selection; a direct head-to-head of RepExp versus random selection with the same verification budget, measured as samples-to-correct per difficulty bin, would settle the claim.","tokens_in":21534,"feed_emoji":"🧠","tokens_out":4417,"duration_ms":37033,"temperature":0.7,"pith_summary":"The paper tries to establish that deliberate exploration—explicitly steering a language model toward novel and diverse responses—can do more than sharpen behaviors the base model already has. It argues that a simple, principled diversity bonus computed from the model's own hidden states, an elliptical bonus of the form h^T Σ^{-1}h, substantially improves both inference-time selection and reinforcement-learning post-training. If true, this would mean the knowledge encoded in pre-trained representations can guide the search for new reasoning behaviors, and that exploration with the right notion of diversity is a practical path beyond standard RL sharpening. A sympathetic reader would care because the method is parameter-light, needs no auxiliary networks, and directly targets the verifier-efficiency bottleneck in reasoning tasks.","feed_headline":"Hidden-state diversity bonus triples LLM reasoning efficiency","feed_subtitle":"Selecting responses that look different in the model's own representation space beats random sampling—and fixes RL's collapse in pass@k.","key_machinery":"The central object is the elliptical bonus div(h | h_{1:i-1}) = h^T Σ^{-1} h, with Σ = λI + Σ_{j<i} h_j h_j^T. The paper's RepExp algorithm iteratively selects, from a large candidate pool, the response whose mean-pooled last-layer hidden state maximizes this bonus, then updates Σ^{-1} via a rank-one Woodbury update. The feature vector h is the pre-trained model's last-layer hidden states averaged over tokens, mean-centered, and projected to 512 dimensions via a sparse random projection. The bonus is history-aware: it penalizes responses whose representations are redundant with those already selected, grounding novelty in the model's own internal geometry rather than in surface text statisti","core_discovery":"The central claim is that representation-based exploration delivers consistent gains in both test-time and post-training settings. At inference time, the algorithm samples a large pool of responses to a prompt, embeds each response by mean-pooling the model's last-layer hidden states, and iteratively selects responses that maximize an elliptical bonus h^T Σ^{-1}h, which is large for responses that are poorly explained by previously selected ones. Across a range of models and tasks, this selection improves verifier efficiency—the expected number of verifier queries before a correct answer is found—by over 50% for a 14-billion-parameter instruction-tuned model on several math and coding benchm","pith_inferences":["If hidden-state distance tracks solution-strategy diversity, the same selection scheme could cut annotation costs in domains where verifiers are expensive or human, by choosing which candidate responses to show an expert.","The method's reliance on base-model representations suggests a testable extension: representation fine-tuning or stronger base models should extend the sample-efficiency gains to regimes where the paper finds no benefit.","The inference-time selection framework could be used as a cheap, confound-free screening test for other diversity bonuses (entropy, unlikeliness, determinantal point processes) before committing to expensive RL runs.","Combining representation bonuses with prolonged RL or reference-model resetting, which the paper lists as complementary, may compound the diversity gains rather than saturate them."],"forward_implications":["Representation-based diversity selection improves verifier efficiency over random sampling across models and tasks, with the largest gains on the hardest questions and on stronger models.","Adding the same bonus to GRPO rewards preserves or improves pass@k at large k, eliminating the 'diversity collapse' where standard RL degrades pass@k relative to the base model.","On AIME 2024, post-training with the bonus yields pass@80 matching pass@256 of standard GRPO, a 3.2x sample-efficiency improvement.","A token-level variant of the bonus, applied to logits during autoregressive generation, improves solve rates for large budgets on the hardest math questions, though the current implementation is not wall-clock competitive.","Weak models can see no benefit or degradation, so gains depend on the representation quality of the base model."],"fun_headline_variants":["Hidden-state diversity gives 3x sampling efficiency","Explore with model's own states to boost reasoning","Representation-space diversity improves LLM response selection","Diverse responses from hidden states beat random sampling"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the mean-pooled last-layer hidden states, after random projection, faithfully reflect behavioral diversity: responses that are far apart in this representation space are assumed to correspond to distinct solution strategies with non-overlapping chances of being correct.","fun_headline_variants_meta":{"raw":{"variants":["Hidden-state diversity gives 3x sampling efficiency","Explore with model's own states to boost reasoning","Representation-space diversity improves LLM response selection","Diverse responses from hidden states beat random sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001591,"raw_usage":{"total_tokens":6210,"prompt_tokens":805,"completion_tokens":5405,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":5355}},"tokens_in":549,"tokens_out":5405,"duration_ms":35338,"temperature":1.0,"reasoning_tokens":5355,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:03:37.852609+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a strong model on a hard reasoning task, generate a large candidate pool, and compute the correctness rate within clusters of hidden-state representations. If clusters with very different embedding positions have nearly identical correctness rates—or if a high-temperature pool produces responses that look novel but are all wrong—then the elliptical bonus cannot beat random selection; a direct head-to-head of RepExp versus random selection with the same verification budget, measured as samples-to-correct per difficulty bin, would settle the claim.","supporting_citations":[],"review_version":1}