REVIEW 5 major objections 5 minor 4 cited by
CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention
T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read CURE claims that re-prompting a language model from its own high-entropy reasoning prefixes prevents entropy collapse and lifts six-benchmark math average to 54.3%.
desk verdict CURE's core idea—branching at high-entropy tokens and retraining on the mix—is novel and the ablation is convincing, but single-run numbers, a code-availability mismatch, and an under-specified objective for re-prompted rollouts keep me from trusting the 5% gain yet. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is critical-token-guided re-concatenation. For each rollout, CURE computes the token-level policy entropy $H_{i,t}=-\sum_v \pi_{\theta_{\text{old}}}(v\mid q,o_{i,<t})\log \pi_{\theta_{\text{old}}}(v\mid q,o_{i,<t})$, samples one position $t_i^\star$ uniformly from the top-$K$ highest-entropy positions, truncates just before it to get the prefix $p_i=o_{i,1:t_i^\star-1}$, forms the refined prompt $q'_i=q\parallel p_i$, and samples $N_2$ continuation rollouts from each refined prompt. All $N_1+N_1N_2$ rollouts are pooled into a single group $G(q)$ and optimized with a clipped, group-relative advantage objective. A stop-gradient on the inserted prefix means no gradient explains how to reach the re-prompted state, so the refined prompt acts as an unfamiliar initial state rather than as a reinforced action sequence.
What would settle it
Train CURE with the same compute and hyperparameters but draw every rollout in the group independently from the original question. If the reported 5-point average gain over prior baselines and the entropy lift disappear, then the prefix-sharing re-prompted branches themselves are the active ingredient, and the uncorrected correlation in the advantage estimator is the likely carrier; if the gains survive, the critical-token mechanism is doing independent work.
Extended reading notes
Core claim
CURE's central claim is that policy entropy collapse in RLVR is a state-distribution problem, not only an objective or update-rule problem. When initial states are always drawn from the fixed training set, the policy quickly concentrates probability mass on a few response patterns and training plateaus; CURE prevents this by re-prompting from the model's own high-entropy decision points. Concretely, for each response the method computes token-level policy entropy, samples one position among the top-K highest-entropy positions, takes the prefix before that position, appends it to the original question, and generates new continuations from that refined prompt. All original and branched rollouts are then optimized together in one group with a clipped group-relative advantage objective. After this exploration phase, CURE switches back to static initial-state sampling and continues training, which the paper reports converts exploration into accuracy gains while keeping the policy more diverse than baselines. The paper's ablation, in which random truncation yields 51.4% average accuracy while entropy-guided truncation yields 53.7%, is presented as evidence that the location of the intervention, not re-prompting by itself, carries the improvement.
Load-bearing premise
The load-bearing premise is that ordinary group-relative advantage normalization stays valid when the group contains correlated rollouts—original trajectories plus re-prompted continuations that share prefixes—and the paper offers no correction or analysis of the bias this correlation could introduce.
Editorial extensions
If this is right
- RLVR pipelines that keep the prompt distribution fixed will tend to lose policy entropy and plateau, so reshaping the training-state distribution at uncertain decision points delays that plateau.
- The two-stage schedule converts exploration into accuracy: the first stage gives 52.1% average accuracy and the second stage raises it to 54.3%, so continuing training on static prompts after exploration does not undo the exploration gains.
- The location of the intervention matters: entropy-guided truncation (53.7%) clearly beats random truncation (51.4%) in the paper's ablation, so the critical-token criterion is doing real work.
- Because the method changes the sampler and the prompt rather than the reward or the KL term, it is presented as complementary to reward shaping and KL-based regularization.
- The re-prompting stage adds extra rollouts per prompt, but the paper reports that the method is data-efficient enough to beat prior methods when trained on only 17K examples.
Reading between the lines
- Going beyond the reported experiments, if the correlation between original and re-prompted rollouts in the shared group is benign at practical batch sizes, the same re-prompting trick could be added to any policy-gradient LLM trainer, not only math-oriented RLVR pipelines.
- The paper's observation that high-entropy connector tokens such as 'check', 'verify', and 'hence' are amplified suggests a testable extension: the same critical-token criterion could be applied to code or multimodal reasoning, where branch points and verification steps play a similar role; the paper lists such tasks as future work.
- A cheaper variant would follow from the stop-gradient argument: generate only one re-prompted continuation per critical token and exclude the original rollouts from the advantage group, turning the objective into a pure off-policy update; the paper does not test this configuration.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CURE, a two-stage reinforcement-learning-with-verified-reward (RLVR) method for mathematical reasoning. In Stage 1, for each prompt the policy samples N1 rollouts, token-level entropy identifies a high-entropy 'critical' position, the prefix before that position is appended to the original query to form a refined prompt, and N2 further rollouts are sampled from that refined prompt. All original and re-prompted trajectories are pooled into one group and optimized with a GRPO-like clipped objective whose advantage is computed by standardizing rewards within that mixed group. In Stage 2, the method reverts to static DAPO-style training on the original prompts. Experiments on Qwen2.5-Math-7B with the DAPO-Math-17K dataset report an average of 54.3% on six math benchmarks, compared with 51.7% for the best prior baseline NFT, together with entropy and length curves, an ablation against random truncation, and a qualitative analysis of high-entropy connector tokens.
Significance. If the reported gains are reproducible, CURE offers a conceptually simple and mechanism-level contrast to entropy regularizers: instead of shaping the loss, it injects exploration by re-prompting at points of high policy uncertainty. A notable strength is the ablation against random truncation, which supports the claim that the entropy-based choice of the truncation point, not mere intervention frequency, drives the benefit. The use of external math benchmarks for the final accuracy numbers means the central performance claim is not obtained by fitting parameters to the test sets. The paper is also refreshingly explicit about its limited model scale and about the computational constraints. However, the method's core objective is not fully specified at the level of the likelihood ratios and group-relative advantages, and the empirical comparison lacks variance estimates and a fully consistent baseline protocol. The central claim is defensible but currently rests on an under-analyzed estimator and single-run results.
major comments (5)
- [Sec. 3.2, Eq. (1), Eq. (10), Appendix A] The importance-weight definition in Eq. (1) conditions on q, but for the re-prompted trajectories in G(q'_i,N2) the actual sampling distribution is pi_theta_old(·|q'_i), so the likelihood ratio must be pi_theta(o_t|q'_i,o_<t)/pi_theta_old(o_t|q'_i,o_<t). Appendix A writes q~_{o_i} in {q} union {q'_i}, but the main-text objective in Eq. (10) never states the corrected ratio. Either the implementation uses the corrected conditioning, in which case Eq. (10) is mis-specified, or it follows Eq. (1) literally, in which case the objective is not the likelihood ratio for the re-prompted rollouts. In addition, the group G(q) contains N1 rollouts from q and N1*N2 rollouts from q'_i, and the re-prompted rewards are correlated with the original rollouts through the shared prefix p_i; the paper provides no analysis of whether the group-relative advantage in Eq. (2) remains an unbiased policy-gradient estimator under this non-i.i.d. group composition. This is not a cosmetic issue because the reported benchmark gains are the only evidence that the update is a correct policy-gradient step. Please provide the exact implemented objective with the corrected likelihood ratios and an unbiasedness or consistency analysis of the advantage estimator.
- [Sec. 3.3, Eq. (13)] Eq. (13) states that the gradient is proportional to sum over tokens of (nabla_theta r_i,t(theta)) * A^grp_i,t, but this is not the gradient of the clipped objective in Eq. (10). The clipped surrogate contains a min over the unclipped and clipped likelihood ratios, and when the clipping is active the gradient is zero or is multiplied by a different coefficient. As written, Eq. (13) describes the gradient of an unclipped policy-gradient objective, not of the objective the paper claims to optimize. Please state the implemented gradient (or correct Eq. (13)) and clarify whether clipping is applied in the same way as in DAPO/GRPO.
- [Sec. 4.1 and Table 1] The central empirical claim is a 5% relative gain over the best prior baseline, but the paper reports a single run for each method and gives no variance or seed information. Differences such as 35.5 vs. 33.4 on AIME24 or 89.7 vs. 88.5 on AMC23 are small relative to typical run-to-run variability in RLVR training, so the absence of error bars leaves the headline gain within plausible noise. Please report results over at least three seeds with mean and standard deviation, or provide a clear argument for why a single run is sufficient in this setting.
- [Table 1 and Appendix B.3] Because CURE is explicitly a modification of DAPO, the main comparison table should include DAPO and GRPO under the same training protocol; currently Table 1 omits both, and the only DAPO numbers appear in the four-benchmark ablation of Table 2 with a different average. Moreover, the baseline protocol is heterogeneous: NFT results are quoted from the original paper, several other baselines are evaluated from their released code, and Clip-Cov and KL-Cov are retrained in-house with a different initial model replacement. This makes the 'best prior baseline' comparison less direct than the prose implies. Please add six-benchmark DAPO/GRPO rows and clearly separate results obtained under identical local conditions from those copied from prior publications.
- [Sec. 3.2.1, Eq. (10)] Eq. (10) normalizes the objective by the total number of tokens in the group, 1/sum|o_i|, whereas the GRPO objective in Eq. (3) normalizes each trajectory by G and by |o_i|. This is a substantive design choice: with total-token normalization, longer trajectories receive more aggregate weight, which interacts with the method's own response-length dynamics and with the group construction. The paper does not discuss this deviation or its effect on the length-regularization behavior shown in Fig. 3. Please justify the normalization or analyze its consequences.
minor comments (5)
- [Appendix A] The statement that the N1 original rollouts 'constitute only a small fraction of the batch' is inaccurate for the reported hyperparameters: with N1=4 and N2=3, the original rollouts are 4 of 16 group members, i.e., 25% of the group, not a small fraction.
- [Sec. 4.1] The word 'Evulation' in the experiment section is a typo and should be 'Evaluation'.
- [Table 2] The table notation 'CURERandom' and 'CUREEntropy' is inconsistent with the prose 'CURE Random' and 'CURE Entropy'; please unify the notation.
- [Sec. 4.2] The sentence 'representing a 107% improvement over the base model' refers to the AMC23 score (89.7 vs. 52.2), but the reader may read it as applying to the average; please clarify which quantity the percentage refers to.
- [Sec. 3.2.1, Eq. (10)] The constraint in Eq. (10) uses the symbol gt before it is defined in the text; please move the definition of gt before the equation or add a parenthetical reference.
Circularity Check
No significant circularity: CURE's accuracy claims rest on external benchmarks and the entropy mechanism is a design choice, not a fitted outcome.
full rationale
Walking the derivation chain, CURE's two central claims are (1) critical-token-guided re-concatenation sustains policy entropy, and (2) the two-stage procedure improves accuracy on math benchmarks. Neither claim reduces to its own inputs by construction. The critical-token selection in Eqs. 5-8 is a data-collection mechanism, not a fitted parameter: entropy is used to choose where to branch, but the reported entropy-preservation result is measured on test sets (Fig. 2, Tab. 1), not at the selected high-entropy positions, so it is not a tautology. The benchmark gains in Tab. 1 are measured against external math benchmarks under a fixed evaluation protocol; DAPO, Clip-Cov and KL-Cov are independent code-run baselines, and NFT is reported from its original paper as a normal baseline citation. No load-bearing self-citation chain forces the conclusions. The most substantive concern is the correctness or bias of the group-relative advantage estimator when the group mixes correlated original and re-prompted rollouts (Eqs. 9-10), and the mismatch between Eq. 1's likelihood ratio conditioned on q and the actual re-prompted context q'. That is a statistical validity issue, not circularity: the reported predictions are not equivalent to the model's inputs by definition or by fitted construction. Therefore the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (3)
- N1 (Initial Rollouts) =
4
- N2 (Re-Prompting Rollouts) =
3
- Top-K entropy selection =
20
assumptions (3)
- domain assumption Group-relative advantages computed in Eq. 10 remain valid when group members are not independent (original and re-prompted trajectories share prefixes).
- domain assumption High-entropy tokens are the correct intervention points for meaningful exploration.
- domain assumption A second stage of DAPO static-sampling training, without KL or entropy regularization, will convert exploration gains into accuracy without causing entropy collapse or overfitting.
Cite this review
Pith. "Pith review of CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention." pith.science (2026). https://pith.science/paper/VU7T5FVT
@misc{pith2026250811016,
author = {Pith},
title = {Pith review of: CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention},
year = {2026},
howpublished = {\url{https://pith.science/paper/VU7T5FVT}},
note = {Machine review of arXiv:2508.11016}
}
read the original abstract
Recent advances in Reinforcement Learning with Verified Reward (RLVR) have driven the emergence of more sophisticated cognitive behaviors in large language models (LLMs), thereby enhancing their reasoning capabilities. However, in prior RLVR pipelines, the repeated use of static initial-state sampling drawn exactly from the dataset distribution during each sampling phase produced overly deterministic, low diversity model behavior, which manifested as rapid entropy collapse and hindered sustained performance gains during prolonged training. To address this issue, we introduce CURE (Critical-token-gUided Re concatenation for Entropy-collapse prevention), a two-stage framework that balances exploration and exploitation. Specifically, in the first stage, to deliberately steer the model toward novel yet coherent contexts, we re-generate at high-entropy critical tokens and jointly optimize the original and the branched trajectories. The further comparison with vanilla DAPO shows that the regeneration process achieves a better performance on math reasoning tasks while sustaining a high-level entropy degree for exploration. In the second stage, we continue training with static initial-state sampling by DAPO, intentionally placing the model in a familiar state to gradually strengthen exploitation. Extensive experiments on Qwen-2.5-Math-7B show that, compared to other RLVR methods, CURE achieves a 5% performance gain across six math benchmarks, establishing state-of-the-art performance in both entropy and accuracy. A series of experiments further validate the effectiveness of our approach. Code is available at https://github.com/bytedance/CURE.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 4 Pith papers
-
Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoning
CEEH selectively applies entropy regularization to hard questions and a shortest-correct-length penalty to easy ones, reducing reasoning length while preserving accuracy across six math benchmarks.
-
Mimic Human Cognition, Master Multi-Image Reasoning: A Meta-Action Framework for Enhanced Visual Understanding
A structured five-step reasoning template plus diverse-trajectory cold start and diversity-preserving two-stage RL lifts a 7B multimodal model to state-of-the-art multi-image reasoning on several benchmarks.
-
SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
Filtering out trajectories with 'void turns' (no code, no answer) stabilizes multi-turn tool-integrated RL and substantially improves math reasoning benchmarks.
-
Entropy-Guided Loop: Achieving Reasoning through Uncertainty-Aware Generation
A lightweight entropy-triggered refinement loop improves a small LLM's answer quality to roughly 95% of a reasoning model's, at about one-third the cost.
Reference graph
Works this paper leans on
-
[2]
Crucially, there are no log-probability (hence no gradient) terms for tokens inside pi. The mapping q7→ q′ i is therefore never reinforced, ren- dering q′ i a genuinely novel initial state that helps delay pre- mature entropy collapse. B Additional Implementation Details B.1 Baselines Details of Baselines • Eurus-2-7B-PRIME-Zero (Cui et al. 2025a) is a re...
work page 2025
-
[3]
arXiv preprint arXiv:2506.14758
Reasoning with exploration: An en- tropy perspective. arXiv preprint arXiv:2506.14758. Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D
-
[5]
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. Ecoffet, A.; Huizinga, J.; Lehman, J.; Stanley, K. O.; and Clune, J
-
[7]
arXiv preprint arXiv:2505.22312
Sky- work open reasoner 1 technical report. arXiv preprint arXiv:2505.22312. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J
-
[9]
arXiv preprint arXiv:2503.24290
Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290. Jaech, A.; Kalai, A.; Lerer, A.; Richardson, A.; El-Kishky, A.; Low, A.; Helyar, A.; Madry, A.; Beutel, A.; Carney, A.; et al
-
[10]
arXiv preprint arXiv:2412.16720
Openai o1 system card. arXiv preprint arXiv:2412.16720. Kydl´ıˇcek, H. ???? Math-Verify: Math Verification Library. Ladosz, P.; Weng, L.; Kim, M.; and Oh, H
-
[11]
Numinamath: The largest public dataset in AI4Maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9. Liu, M.; Diao, S.; Lu, X.; Hu, J.; Dong, X.; Choi, Y .; Kautz, J.; and Dong, Y . 2025a. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24864....
-
[12]
arXiv preprint arXiv:2412.15084
AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling. arXiv preprint arXiv:2412.15084. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al
Show all 21 references
-
[13]
5: Scaling Reinforcement Learning with LLMs
Kimi k1. 5: Scaling Reinforcement Learning with LLMs. arXiv preprint arXiv:2501.12599. Wang, S.; Yu, L.; Gao, C.; Zheng, C.; Liu, S.; Lu, R.; Dang, K.; Chen, X.; Yang, J.; Zhang, Z.; et al
-
[14]
Wei, Y .; Wang, Z.; Liu, J.; Ding, Y .; and Zhang, L
Beyond the 80/20 rule: High-entropy minority tokens drive effec- tive reinforcement learning for llm reasoning.arXiv preprint arXiv:2506.01939. Wei, Y .; Wang, Z.; Liu, J.; Ding, Y .; and Zhang, L
-
[15]
arXiv preprint arXiv:2504.14945
Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945. Yang, A.; Zhang, B.; Hui, B.; Gao, B.; Yu, B.; Li, C.; Liu, D.; Tu, J.; Zhou, J.; Lin, J.; et al
-
[16]
5-math tech- nical report: Toward mathematical expert model via self- improvement
Qwen2. 5-math tech- nical report: Toward mathematical expert model via self- improvement. arXiv preprint arXiv:2409.12122. Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y .; Zuo, X.; Yue, Y .; Dai, W.; Fan, T.; Liu, G.; Liu, L.; et al
-
[17]
arXiv preprint arXiv:2503.14476
Dapo: An open- source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Yuan, L.; Cui, G.; Wang, H.; Ding, N.; Wang, X.; Deng, J.; Shan, B.; Chen, H.; Xie, R.; Lin, Y .; Liu, Z.; Zhou, B.; Peng, H.; Liu, Z.; and Sun, M
-
[18]
arXiv preprint arXiv:2504.14286
Srpo: A cross-domain implementation of large-scale reinforce- ment learning on llm. arXiv preprint arXiv:2504.14286. Zhou, H.; Li, X.; Wang, R.; Cheng, M.; Zhou, T.; and Hsieh, C.-J
-
[19]
arXiv preprint arXiv:2503.05132
R1-Zero’s” Aha Moment” in Visual Reasoning on a 2B Non-SFT Model. arXiv preprint arXiv:2503.05132. Appendix A Why CURE can prevent entropy collapse? From the perspective of traditional RL. A common line of work in exploration treats the (state, policy) uncertainty via the poli...
2022 arXiv
-
[21]
is an off-policy RLVR frame- work that augments on-policy learning with external rea- soning traces, allowing models to acquire abilities be- yond their own outputs. It mixes off-policy demonstra- tions with on-policy rollouts, combining Mixed-Policy GRPO whose convergence rat...
2025
-
[2018]
arXiv preprint arXiv:1810.12894
Exploration by random network distillation. arXiv preprint arXiv:1810.12894. Chen, H.; Zheng, K.; Zhang, Q.; Cui, G.; Cui, Y .; Ye, H.; Lin, T.-Y .; Liu, M.-Y .; Zhu, J.; and Wang, H
-
[2019]
arXiv preprint arXiv:1901.10995
Go-explore: a new approach for hard- exploration problems. arXiv preprint arXiv:1901.10995. He, C.; Luo, R.; Bai, Y .; Hu, S.; Thai, Z.; Shen, J.; Hu, J.; Han, X.; Huang, Y .; Zhang, Y .; et al
1901 arXiv
-
[2021]
arXiv preprint arXiv:2103.03874
Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. hiyouga
-
[2024]
UL- TRAFEEDBACK: Boosting Language Models with Scaled AI Feedback. In ICML. Cui, G.; Yuan, L.; Wang, Z.; Wang, H.; Li, W.; He, B.; Fan, Y .; Yu, T.; Xu, Q.; Chen, W.; et al. 2025a. Pro- cess reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. Cui, G.; Zhan...
-
[2025]
arXiv preprint arXiv:2505.18116
Bridg- ing supervised learning and reinforcement learning in math reasoning. arXiv preprint arXiv:2505.18116. Cheng, D.; Huang, S.; Zhu, X.; Dai, B.; Zhao, W. X.; Zhang, Z.; and Wei, F
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.