{"id":"bdaa42ad-5f6a-4835-bbfd-820cb0dd89c9","arxiv_id":"2508.11016","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CURE re-generates responses from prefixes ending at high-entropy tokens to maintain exploration, then continues DAPO training to consolidate gains, achieving 54.3% average accuracy on six math benchmarks.","lead":"CURE is a two-stage training method that prevents policy entropy collapse during reinforcement learning for math reasoning by re-sampling answers at high-uncertainty \"critical token\" points, then fine-tuning with standard static sampling. On Qwen-2.5-Math-7B it reports roughly a 5% average improvement over existing RLVR methods across six math benchmarks while keeping training entropy higher.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CURE's group-relative advantage (Eq. 10) mixes correlated original and re-prompted rollouts, and Eq. 1's importance weights condition on q rather than q'; without an unbiasedness analysis the claimed 5% gain is not attributable to correct policy optimization.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the group-relative advantage estimator in Eq. 10 is used on groups of correlated, mixed-prompt rollouts, whereas GRPO's unbiasedness argument assumes independent rollouts per prompt. I agree with this concern and with the conditional verdict. I add one concrete internal inconsistency that makes the concern sharper: Eq. 1's importance weight conditions on q, not on the re-prompted state q'_i that actually generated the re-prompted rollouts. Appendix A implicitly corrects this by introducing q̃_{o_i}, but the main-text objective is never updated to match. This is an internal correctness issue, not a disagreement with external consensus. The critical-token ablation in Table 2 is genuine supporting evidence for the mechanism, and the reported entropy preservation is consistent with the method's intent, but neither addresses the validity of the policy-gradient estimator. Since the central claim is an empirical performance gain, a biased objective could still produce those numbers, but the paper would then not support its claimed mechanism. A conditional accept is appropriate: the authors should either prove unbiasedness (or bound the bias) of Eq. 10 under their correlated group construction, release the code so the actual likelihood-ratio computation can be inspected, and report multiple seeds. Without these, the 5% gain cannot be confidently attributed to correct policy optimization rather than to the biased baseline or sampling noise.","tokens_in":16814,"tokens_out":8830,"duration_ms":101421,"concrete_test":"Build a small token-level MDP with known rewards, compute the exact expected policy gradient under π_old, then sample G(q) exactly as in Eq. 9 (N1 original and N1·N2 re-prompted rollouts), compute the gradient of the CURE objective Eq. 10 using Eq. 1 and Eq. 2, and average over many groups. If the averaged gradient differs from the exact gradient by more than Monte Carlo error, the estimator is biased. Then re-run the AIME24/AMC23 comparison with a corrected advantage estimator that uses separate group-relative baselines for q and for each q'_i; if the 54.3% average drops materially, the headline gain depends on the biased estimator.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on the CURE objective in Eq. 10, which computes GRPO-style group-relative advantages (Eq. 2) over the group G(q) = G(q,N1) ∪ ⋃_i G(q'_i,N2) defined in Eq. 9. This group contains N1 original rollouts from q and N1·N2 rollouts from re-prompted states q'_i = q ∥ p_i, where each q'_i is derived from one of the original rollouts. Standard GRPO unbiasedness assumes G independent rollouts sampled from the same prompt; here the group mixes different initial states, and the re-prompted rewards are correlated with the original rollouts through the shared prefix p_i. The paper gives no bias analysis or correction for this deviation. The issue is compounded by Eq. 1: the importance weight is written as πθ(o_{i,t} | q, o_{i,<t}) / πθold(o_{i,t} | q, o_{i,<t}), but for re-prompted trajectories the actual conditioning context is q'_i, not q. Appendix A implicitly acknowledges this by writing q̃_{o_i} ∈ {q} ∪ {q'_i}, yet the main-text objective never states the corrected likelihood ratio. Consequently, even if the implementation uses the q'_i conditioning, the gradient for the original rollouts' prefix tokens uses a baseline that includes re-prompted rewards dependent on those same prefix tokens, so the estimator in Eq. 10 is not an unbiased policy-gradient estimator. If the implementation instead follows Eq. 1 literally, then Eq. 10 is not even the correct objective for the re-prompted rollouts. Either way, the reported gains—especially the 35.5 vs. 26.6 on AIME24 and 89.7 vs. 52.2 on AMC23—could be an artifact of a biased advantage baseline rather than of the entropy-preservation mechanism. The paper provides no formal derivation, no bias bound, and no seed variance to rule out noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CURE, a two-stage reinforcement-learning-with-verified-reward (RLVR) method for mathematical reasoning. In Stage 1, for each prompt the policy samples N1 rollouts, token-level entropy identifies a high-entropy 'critical' position, the prefix before that position is appended to the original query to form a refined prompt, and N2 further rollouts are sampled from that refined prompt. All original and re-prompted trajectories are pooled into one group and optimized with a GRPO-like clipped objective whose advantage is computed by standardizing rewards within that mixed group. In Stage 2, the method reverts to static DAPO-style training on the original prompts. Experiments on Qwen2.5-Math-7B with the DAPO-Math-17K dataset report an average of 54.3% on six math benchmarks, compared with 51.7% for the best prior baseline NFT, together with entropy and length curves, an ablation against random truncation, and a qualitative analysis of high-entropy connector tokens.","tokens_in":17171,"tokens_out":6584,"duration_ms":72940,"significance":"If the reported gains are reproducible, CURE offers a conceptually simple and mechanism-level contrast to entropy regularizers: instead of shaping the loss, it injects exploration by re-prompting at points of high policy uncertainty. A notable strength is the ablation against random truncation, which supports the claim that the entropy-based choice of the truncation point, not mere intervention frequency, drives the benefit. The use of external math benchmarks for the final accuracy numbers means the central performance claim is not obtained by fitting parameters to the test sets. The paper is also refreshingly explicit about its limited model scale and about the computational constraints. However, the method's core objective is not fully specified at the level of the likelihood ratios and group-relative advantages, and the empirical comparison lacks variance estimates and a fully consistent baseline protocol. The central claim is defensible but currently rests on an under-analyzed estimator and single-run results.","major_comments":[{"comment":"The importance-weight definition in Eq. (1) conditions on q, but for the re-prompted trajectories in G(q'_i,N2) the actual sampling distribution is pi_theta_old(·|q'_i), so the likelihood ratio must be pi_theta(o_t|q'_i,o_<t)/pi_theta_old(o_t|q'_i,o_<t). Appendix A writes q~_{o_i} in {q} union {q'_i}, but the main-text objective in Eq. (10) never states the corrected ratio. Either the implementation uses the corrected conditioning, in which case Eq. (10) is mis-specified, or it follows Eq. (1) literally, in which case the objective is not the likelihood ratio for the re-prompted rollouts. In addition, the group G(q) contains N1 rollouts from q and N1*N2 rollouts from q'_i, and the re-prompted rewards are correlated with the original rollouts through the shared prefix p_i; the paper provides no analysis of whether the group-relative advantage in Eq. (2) remains an unbiased policy-gradient estimator under this non-i.i.d. group composition. This is not a cosmetic issue because the reported benchmark gains are the only evidence that the update is a correct policy-gradient step. Please provide the exact implemented objective with the corrected likelihood ratios and an unbiasedness or consistency analysis of the advantage estimator.","section":"Sec. 3.2, Eq. (1), Eq. (10), Appendix A"},{"comment":"Eq. (13) states that the gradient is proportional to sum over tokens of (nabla_theta r_i,t(theta)) * A^grp_i,t, but this is not the gradient of the clipped objective in Eq. (10). The clipped surrogate contains a min over the unclipped and clipped likelihood ratios, and when the clipping is active the gradient is zero or is multiplied by a different coefficient. As written, Eq. (13) describes the gradient of an unclipped policy-gradient objective, not of the objective the paper claims to optimize. Please state the implemented gradient (or correct Eq. (13)) and clarify whether clipping is applied in the same way as in DAPO/GRPO.","section":"Sec. 3.3, Eq. (13)"},{"comment":"The central empirical claim is a 5% relative gain over the best prior baseline, but the paper reports a single run for each method and gives no variance or seed information. Differences such as 35.5 vs. 33.4 on AIME24 or 89.7 vs. 88.5 on AMC23 are small relative to typical run-to-run variability in RLVR training, so the absence of error bars leaves the headline gain within plausible noise. Please report results over at least three seeds with mean and standard deviation, or provide a clear argument for why a single run is sufficient in this setting.","section":"Sec. 4.1 and Table 1"},{"comment":"Because CURE is explicitly a modification of DAPO, the main comparison table should include DAPO and GRPO under the same training protocol; currently Table 1 omits both, and the only DAPO numbers appear in the four-benchmark ablation of Table 2 with a different average. Moreover, the baseline protocol is heterogeneous: NFT results are quoted from the original paper, several other baselines are evaluated from their released code, and Clip-Cov and KL-Cov are retrained in-house with a different initial model replacement. This makes the 'best prior baseline' comparison less direct than the prose implies. Please add six-benchmark DAPO/GRPO rows and clearly separate results obtained under identical local conditions from those copied from prior publications.","section":"Table 1 and Appendix B.3"},{"comment":"Eq. (10) normalizes the objective by the total number of tokens in the group, 1/sum|o_i|, whereas the GRPO objective in Eq. (3) normalizes each trajectory by G and by |o_i|. This is a substantive design choice: with total-token normalization, longer trajectories receive more aggregate weight, which interacts with the method's own response-length dynamics and with the group construction. The paper does not discuss this deviation or its effect on the length-regularization behavior shown in Fig. 3. Please justify the normalization or analyze its consequences.","section":"Sec. 3.2.1, Eq. (10)"}],"minor_comments":[{"comment":"The statement that the N1 original rollouts 'constitute only a small fraction of the batch' is inaccurate for the reported hyperparameters: with N1=4 and N2=3, the original rollouts are 4 of 16 group members, i.e., 25% of the group, not a small fraction.","section":"Appendix A"},{"comment":"The word 'Evulation' in the experiment section is a typo and should be 'Evaluation'.","section":"Sec. 4.1"},{"comment":"The table notation 'CURERandom' and 'CUREEntropy' is inconsistent with the prose 'CURE Random' and 'CURE Entropy'; please unify the notation.","section":"Table 2"},{"comment":"The sentence 'representing a 107% improvement over the base model' refers to the AMC23 score (89.7 vs. 52.2), but the reader may read it as applying to the average; please clarify which quantity the percentage refers to.","section":"Sec. 4.2"},{"comment":"The constraint in Eq. (10) uses the symbol gt before it is defined in the text; please move the definition of gt before the equation or add a parenthetical reference.","section":"Sec. 3.2.1, Eq. (10)"}],"recommendation":"major_revision","confidential_remarks":"The paper is suitable in scope for an ML conference or workshop on reasoning and RL. The main obstacle to acceptance is the under-specified objective and the lack of an unbiasedness analysis for the group-relative advantage, not the exploration idea itself. I would not reject on the basis of the entropy-preservation measure being partly self-fulfilling, because the random-truncation ablation addresses the most direct confound. The missing DAPO/GRPO rows in the main table should be added before the paper can credibly claim state-of-the-art status."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth your time. It tackles entropy collapse in RLVR by doing something simple: find the token where the policy is most uncertain, cut the trajectory there, append the prefix to the original question, re-prompt, and train on the original and branched rollouts together. The two-stage scheme (explore then consolidate with DAPO) is clean, and the critical-token ablation is the right experiment—random truncation doesn't help, entropy-guided truncation does.\n\nWhat's actually new: previous entropy fixes reweight losses, clip gradients, or reset the reference policy. None re-concatenate high-entropy prefixes into new prompts. The connector-token analysis is a nice qualitative check that the method is doing what it claims.\n\nThe soft spots, in order of severity.\n\nFirst, the objective in Eq. 10 is written sloppily. The importance weight in Eq. 1 conditions on q, but for re-prompted rollouts the real context is q'_i = q + prefix. Appendix A quietly switches to a set of contexts q_tilde, so the main text either has a typo or the implementation uses a different likelihood ratio than the one defined. That's not fatal by itself, but it raises a real question: the group-relative advantage in Eq. 2 is computed over a group that mixes original rollouts and re-prompted rollouts that share prefixes. Standard GRPO's unbiasedness assumes independent rollouts from the same prompt. Here the baseline for original-rollout tokens can depend on rewards from re-prompted trajectories that were generated from those same tokens. No bias analysis or correction is given. This could be a minor issue in practice, but the paper doesn't show it. A clean fix would be to normalize original and re-prompted rewards separately, or at least present an ablation.\n\nSecond, the evidence is single-run. No seeds, no error bars. Some baselines (NFT, Eurus-2) are numbers lifted from other papers, while Clip-Cov and KL-Cov were trained in-house. The comparison is plausible but not fully controlled.\n\nThird, the abstract says code is available at github.com/bytedance/CURE, but Appendix B.5 says it 'will be released in the near future.' Those two statements contradict each other. The reader should not have to chase this.\n\nIf those three issues were fixed, I'd take the 54.3% average much more seriously. As it stands, the core mechanism has support, but the exact gains are not yet established. I still think this deserves a real peer review—the idea is genuinely useful and the community needs more work on data-level exploration. I just wouldn't want it accepted without the authors addressing the objective formulation and giving us seeds or code.\n\nFor you: read the ablation and the appendix, skip the word clouds. I would want the authors to release code and seeds before I rely on the numbers.","headline":"CURE's core idea—branching at high-entropy tokens and retraining on the mix—is novel and the ablation is convincing, but single-run numbers, a code-availability mismatch, and an under-specified objective for re-prompted rollouts keep me from trusting the 5% gain yet.","tokens_in":17820,"tokens_out":6484,"would_cite":false,"duration_ms":61726,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CURE claims that re-prompting a language model from its own high-entropy reasoning prefixes prevents entropy collapse and lifts six-benchmark math average to 54.3%.","keywords":["reinforcement learning with verified reward","entropy collapse","critical tokens","token-level policy entropy","exploration-exploitation balance","math reasoning","re-concatenation"],"falsifier":"Train CURE with the same compute and hyperparameters but draw every rollout in the group independently from the original question. If the reported 5-point average gain over prior baselines and the entropy lift disappear, then the prefix-sharing re-prompted branches themselves are the active ingredient, and the uncorrected correlation in the advantage estimator is the likely carrier; if the gains survive, the critical-token mechanism is doing independent work.","tokens_in":16604,"feed_emoji":"🧮","tokens_out":9257,"duration_ms":93203,"temperature":0.7,"pith_summary":"This paper argues that entropy collapse in math-reasoning reinforcement learning is caused by repeatedly drawing training prompts from a fixed dataset distribution, and that the fix should be made at the data level rather than only in the update rule. Its proposed method, CURE (Critical-token-guided Re-concatenation for Entropy-collapse prevention), locates the tokens where the policy is most uncertain in each rollout, truncates the response just before one such token, splices the retained prefix onto the original question, and re-samples continuation rollouts from that refined prompt. The original and branched rollouts are optimized together as one group with group-relative advantage, and then training continues on the original static prompts to consolidate gains. On Qwen2.5-Math-7B, CURE reports an average of 54.3% across six math benchmarks, compared with 51.7% for the strongest prior baseline, while keeping policy entropy higher than the compared methods.","feed_headline":"High-entropy forks keep math RL exploring: +5% over prior baselines","feed_subtitle":"CURE re-prompts at uncertain decision tokens, then retrains on static prompts, reaching 54.3% average on six math benchmarks.","key_machinery":"The load-bearing mechanism is critical-token-guided re-concatenation. For each rollout, CURE computes the token-level policy entropy $H_{i,t}=-\\sum_v \\pi_{\\theta_{\\text{old}}}(v\\mid q,o_{i,<t})\\log \\pi_{\\theta_{\\text{old}}}(v\\mid q,o_{i,<t})$, samples one position $t_i^\\star$ uniformly from the top-$K$ highest-entropy positions, truncates just before it to get the prefix $p_i=o_{i,1:t_i^\\star-1}$, forms the refined prompt $q'_i=q\\parallel p_i$, and samples $N_2$ continuation rollouts from each refined prompt. All $N_1+N_1N_2$ rollouts are pooled into a single group $G(q)$ and optimized with a clipped, group-relative advantage objective. A stop-gradient on the inserted prefix means no gradient explains how to reach the re-prompted state, so the refined prompt acts as an unfamiliar initial state rather than as a reinforced action sequence.","core_discovery":"CURE's central claim is that policy entropy collapse in RLVR is a state-distribution problem, not only an objective or update-rule problem. When initial states are always drawn from the fixed training set, the policy quickly concentrates probability mass on a few response patterns and training plateaus; CURE prevents this by re-prompting from the model's own high-entropy decision points. Concretely, for each response the method computes token-level policy entropy, samples one position among the top-K highest-entropy positions, takes the prefix before that position, appends it to the original question, and generates new continuations from that refined prompt. All original and branched rollouts are then optimized together in one group with a clipped group-relative advantage objective. After this exploration phase, CURE switches back to static initial-state sampling and continues training, which the paper reports converts exploration into accuracy gains while keeping the policy more diverse than baselines. The paper's ablation, in which random truncation yields 51.4% average accuracy while entropy-guided truncation yields 53.7%, is presented as evidence that the location of the intervention, not re-prompting by itself, carries the improvement.","pith_inferences":["Going beyond the reported experiments, if the correlation between original and re-prompted rollouts in the shared group is benign at practical batch sizes, the same re-prompting trick could be added to any policy-gradient LLM trainer, not only math-oriented RLVR pipelines.","The paper's observation that high-entropy connector tokens such as 'check', 'verify', and 'hence' are amplified suggests a testable extension: the same critical-token criterion could be applied to code or multimodal reasoning, where branch points and verification steps play a similar role; the paper lists such tasks as future work.","A cheaper variant would follow from the stop-gradient argument: generate only one re-prompted continuation per critical token and exclude the original rollouts from the advantage group, turning the objective into a pure off-policy update; the paper does not test this configuration."],"forward_implications":["RLVR pipelines that keep the prompt distribution fixed will tend to lose policy entropy and plateau, so reshaping the training-state distribution at uncertain decision points delays that plateau.","The two-stage schedule converts exploration into accuracy: the first stage gives 52.1% average accuracy and the second stage raises it to 54.3%, so continuing training on static prompts after exploration does not undo the exploration gains.","The location of the intervention matters: entropy-guided truncation (53.7%) clearly beats random truncation (51.4%) in the paper's ablation, so the critical-token criterion is doing real work.","Because the method changes the sampler and the prompt rather than the reward or the KL term, it is presented as complementary to reward shaping and KL-based regularization.","The re-prompting stage adds extra rollouts per prompt, but the paper reports that the method is data-efficient enough to beat prior methods when trained on only 17K examples."],"supporting_citations":[{"why":"Defines the DAPO update rule, the dynamic sampling filter, and the 17K training set that CURE reuses and compares against.","marker":"Yu et al. 2025"},{"why":"Supplies the entropy-control baselines that CURE contrasts as update-level fixes still operating on a fixed prompt distribution.","marker":"Cui et al. 2025b"},{"why":"Provides the evidence that a small subset of high-entropy forking tokens drives exploration, which motivates selecting the truncation point by token entropy.","marker":"Wang et al. 2025"},{"why":"NFT is the strongest prior baseline at 51.7% average that CURE claims to surpass with 54.3%.","marker":"Chen et al. 2025"},{"why":"Qwen2.5-Math-7B is the base model on which all training runs and all six benchmark evaluations are performed.","marker":"Yang et al. 2024"},{"why":"Supplies the SimpleRL evaluation and verification conventions and one of the comparison baselines whose protocol CURE follows.","marker":"Zeng et al. 2025"}],"fun_headline_variants":["Fork at critical tokens to stop entropy collapse, +5% math","High-entropy re-prompts then static retraining: +5% math","CURE: regenerate at unsure tokens, then exploit for +5%","Branch at policy uncertainty, retrain static: math gains 5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ordinary group-relative advantage normalization stays valid when the group contains correlated rollouts—original trajectories plus re-prompted continuations that share prefixes—and the paper offers no correction or analysis of the bias this correlation could introduce.","fun_headline_variants_meta":{"raw":{"variants":["Fork at critical tokens to stop entropy collapse, +5% math","High-entropy re-prompts then static retraining: +5% math","CURE: regenerate at unsure tokens, then exploit for +5%","Branch at policy uncertainty, retrain static: math gains 5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1438,"prompt_tokens":1067,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":291}},"tokens_in":683,"tokens_out":371,"duration_ms":4664,"temperature":1.0,"reasoning_tokens":291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:29:34.534901+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train CURE with the same compute and hyperparameters but draw every rollout in the group independently from the original question. If the reported 5-point average gain over prior baselines and the entropy lift disappear, then the prefix-sharing re-prompted branches themselves are the active ingredient, and the uncorrected correlation in the advantage estimator is the likely carrier; if the gains survive, the critical-token mechanism is doing independent work.","supporting_citations":[],"review_version":2}