Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

This paper claims that the accuracy loss caused by RL-based reasoning compression stems from entropy collapse, and that applying entropy regularization only to hard questions while aggressively compressing easy ones—with a length penalty an

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:37 UTC pith:RVWQJCQY

load-bearing objection Useful empirical recipe for reasoning compression, but the entropy schedule as written would do the opposite of what the paper claims in the second half of training—needs a fix before the story holds together. the 5 major comments →

arxiv 2602.22642 v2 pith:RVWQJCQY submitted 2026-02-26 cs.LG

Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoning

Qin-Wen Luo , Sheng Ren , Xiang Chen , Rui Liu , Jun Fang , Naiqiang Tan , Sheng-Jun Huang This is my paper
classification cs.LG
keywords reasoning compressionentropy collapseentropy regularizationreinforcement learningchain-of-thoughtlength penaltydifficulty estimationPass@k
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to resolve the accuracy-length trade-off in chain-of-thought reasoning compression. Its diagnosis: existing RL methods that reward brevity trigger rapid entropy collapse, shrinking the policy's exploration space and hurting hard questions that need diverse reasoning paths. Its remedy, CEEH, labels each question hard or easy from a smoothed per-question accuracy estimate, applies entropy regularization only to hard questions, and penalizes correct responses that exceed the historically shortest correct length for that question. On six mathematical benchmarks the method reduces response tokens by roughly a third while keeping accuracy at or above the base model, and improves Pass@k compared with length-only optimization. If right, it shows compression and exploration can be managed separately rather than traded off.

Core claim

The central claim is that explicitly optimizing LLM policies for shorter chains of thought collapses policy entropy, prematurely shrinking the space of reasoning paths the model will try, and that this is why length-only RL compression sacrifices accuracy. The proposed framework, CEEH, therefore splits the objective: for questions the model currently solves below the dataset-average rate, it adds entropy regularization (a maximum-entropy loss with a fivefold stronger coefficient, or an entropy-based advantage term) to keep exploration alive; for questions above average, it lets the policy compress freely. A companion dynamic optimal-length penalty tracks, for each question, the shortest corr

What carries the argument

The load-bearing mechanism is a pair of question-level signals computed from the RL rollouts. Difficulty is estimated per question by an asymmetric exponential moving average of rollout accuracy (updated quickly when accuracy rises, slowly when it falls), thresholded against the dataset-average accuracy; only questions below the threshold receive entropy regularization, with the maximum-entropy coefficient multiplied by five or an entropy-based advantage added. The length signal is each question's historically shortest correct response length, which anchors a penalty—correct-response length minus anchor over anchor, clipped to [-0.9, 1)—applied only when the question's current rollout accura

Load-bearing premise

The load-bearing premise is that the per-question accuracy estimate—derived from a small number of noisy rollouts, smoothed by an asymmetric EMA, and compared against the dataset average—correctly identifies which questions are hard for the model; if it mislabels questions, entropy is applied to the wrong instances and the 'shortest correct response' anchor becomes an unrepresentative target.

What would settle it

Take the trained CEEH pipeline and replace the difficulty labels with a same-size random hard/easy split; if the random split matches CEEH's length-accuracy trade-off, then per-question difficulty labeling is not what carries the result. Alternatively, on a held-out question set, corrupt the difficulty labels (for example, flip hard/easy for half the questions) and check whether Pass@k and accuracy stay the same—if they do, the selective entropy mechanism is not load-bearing.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Length-only RL compression is not just suboptimal but actively harmful to reasoning diversity; the same training data supports higher training accuracy when entropy is preserved on hard questions.
  • Compression and exploration do not have to be opposing forces: per-question allocation lets a single training run shorten easy responses while keeping hard ones long.
  • Because the length anchor is per question and updates with training, the compression signal remains calibrated as response-length distributions shift, removing the need to hand-tune a target length per dataset.
  • Pass@k gains over length-only optimization imply that compressed models retain, and in some cases widen, the set of viable reasoning paths, so sampling multiple answers still yields correct ones.
  • The recipe transfers across model scales (7B and 1.5B) and six benchmarks, suggesting the difficulty-aware split generalizes rather than overfitting one dataset.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: the asymmetric EMA (fast up, slow down) biases the difficulty label toward 'hard,' so CEEH behaves like a conservative curriculum—questions keep exploration longer than they strictly need; a faster decay on improvement would compress more aggressively at some risk to accuracy.
  • Inference: the mechanism is not math-specific; nothing in the difficulty estimate or length anchor depends on the answer being a number, so the same recipe should transfer to code generation, symbolic reasoning, or visual question answering, and comparing CEEH to length-only RL on those domains would test that.
  • Inference: the dynamic shortest-correct anchor invites a specific failure—a single lucky short correct response can become the target even if it is brittle; tracking the diversity of compressed solutions, not just their length, would reveal whether the anchor is consolidating robust short reasoning or overfitting one trajectory.
  • Inference: the paper frames entropy collapse as a training-time pathology, but the same logic suggests an inference-time extension: run a cheap difficulty probe at serving time and allocate a longer reasoning budget only when the probe says hard, putting the compress-easy/explore-hard split into deployment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes CEEH, an RL fine-tuning method for compressing LLM reasoning. It estimates per-question difficulty from an asymmetric EMA of rollout accuracy, applies entropy regularization selectively to currently hard questions (either via a maximum-entropy loss with a cosine-annealed coefficient amplified 5x on hard examples, or via an entropy-based advantage), and introduces a dynamic length penalty anchored to the historically shortest correct response. Experiments on six mathematical reasoning benchmarks with R1-Distill-Qwen-2.5-7B and 1.5B report reduced response length while maintaining or slightly improving accuracy relative to a reproduced length-penalty baseline and to published baselines. The paper also presents training-dynamics analyses of policy entropy, Pass@k, and ablations of the length-penalty coefficient.

Significance. If the claims held, the paper would make a useful practical contribution: separating 'compress easy' from 'explore hard' is a principled way to address the accuracy-efficiency trade-off in RL-based reasoning compression, and the dynamic optimal-length anchor is a plausible mechanism for stabilizing compression pressure. The empirical sweep is broad, and evaluating with avg@16 and Pass@k is appropriate. However, the current evidence does not fully establish the central claims: no seed variation or confidence intervals are reported, only one online RL baseline is reproduced under the same training data, and the formal definition of the entropy schedule in Eq. (11) is inconsistent with the paper's stated exploration-preservation mechanism. That inconsistency is load-bearing because the entire method is framed around maintaining exploration on hard questions. The NAG metric in Table 1 also does not match the stated formula. These issues are fixable, but they require more than local editing.

major comments (5)
  1. [Section 3.2, Eq. (11)] The schedule λ(x,t)=5λ0·cos(πt/T) for hard questions becomes negative for t>T/2, since λ0>0. Because Eq. (5) adds -λH to the loss, a negative λ turns the 'entropy regularization' into entropy minimization, which is the opposite of the claimed exploration-preserving mechanism. Section 4.3 explicitly states that policy entropy decreases in the later stage of training, which is consistent with this reading. If the implementation instead uses a nonnegative schedule, then Eq. (11) is wrong; if it uses Eq. (11), the method contradicts its core rationale. Please correct the schedule (e.g., λ0·(1+cos(πt/T))/2) and re-run, or state and justify the actual schedule used.
  2. [Table 1 vs Eq. (17)] Several NAG entries are inconsistent with the stated formula. For GSM8K, with base Acc_b=91.2 and L_b=1479, the Length-Penalty* row (ACC=91.6, LEN=931) gives NAG ≈ -0.72, not the reported -0.27; CEEH-EA (ACC=91.3, LEN=723) gives NAG ≈ -0.15, not -0.08. Similar mismatches occur in other rows, e.g., MATH500 CEEH-EA. Please verify the table, the formula, or the definition of the reported score; as written, the NAG column does not support the quantitative comparison in Table 1.
  3. [Section 4.1, Tables 1-2] No seed variation, standard deviations, or confidence intervals are reported for any experiment. Table 2 shows Pass@k differences of at most 0.5 points on most benchmarks (e.g., GSM8K 97.6-98.3, MATH500 all 97.2, AIME24 all 80), which is likely within sampling noise for 16 rollouts per question and typical test-set sizes. The claim that CEEH 'improves Pass@k relative to length-only optimization' is not supported without a significance test. Please report multiple seeds or bootstrap CIs for ACC, LEN, and Pass@k.
  4. [Section 4.1, Baseline] Only Length-Penalty* is reproduced under the same training data and setup. All other online RL baselines (AutoThink, LC-R1, etc.) are cited from their original papers, which may use different training data mixtures, compute budgets, and hyperparameters. Since CEEH is an online RL method, the claimed improvements over 'strong baselines' need at least same-data comparisons for the main online baselines, or a clear statement of which experimental conditions are matched.
  5. [Section 3.2, Eqs. (9)-(10), (15)-(16)] The difficulty signal is derived from K=12 rollouts per question (Table 4) and an asymmetric EMA with update rates 0.2/0.05. This is a noisy estimate, and the global-average threshold can misclassify a large fraction of questions. The method's behavior depends critically on this classification, yet no sensitivity analysis is provided. If questions are mislabeled, entropy regularization is applied to the wrong instances and the 'historically shortest correct response' in Eq. (13) becomes an unrepresentative compression target. Please ablate K and the EMA rates, or at least report classification stability over training.
minor comments (7)
  1. [Section 3.3 vs Table 3] Table 3 uses η for the length-penalty coefficient, while Eq. (14) uses β and Eq. (15) uses η for the EMA update rate. Please disambiguate the notation.
  2. [Figures 3 and 6] The legend labels '=0.1' and '=0.2' are missing the symbol, presumably β. Please include it.
  3. [Eq. (7)] Typo: 'defiend' should be 'defined'.
  4. [Eq. (17)] Typo: 'large than' should be 'larger than'. Also state the convention for L = L_b or L > L_b, where the metric is undefined.
  5. [Eq. (13) and surrounding text] Typo: 'penlty' should be 'penalty'.
  6. [Section 2 vs Section 3.4] Eq. (2) presents advantage standardization using the group standard deviation, but Section 3.4 says 'we do not apply advantage standardization; instead, we only subtract a reward-mean baseline.' Please reconcile this in the preliminaries or make the actual advantage formula explicit.
  7. [Table 4] The table lists 'entropy coeff 0.001 for ME (0 for EA)', but Section 3.2 introduces both Eq. (5) and Eq. (7) as components of one framework. Clarify the relationship between the two variants and report the values of α and κ used for the entropy-based advantage.

Circularity Check

0 steps flagged

No circularity: CEEH's difficulty labels and length anchor are training signals, and the reported accuracy/Pass@k gains are not definitionally implied by the loss.

full rationale

The paper's load-bearing claims (accuracy maintenance, Pass@k improvement, length reduction) are supported by RL training with Eqs. 5, 11-14. None of these claims is equivalent to the method's inputs by construction: the difficulty classifier (Eqs. 9-10, 15-16) is a dynamic training signal derived from the model's rollout accuracy, not a fitted parameter used to predict the evaluation metrics; entropy regularization on the hard subset modifies the objective but does not force the observed accuracy/Pass@k outcomes; and the optimal-length penalty (Eq. 13) directly rewards brevity, so the length reduction is an expected property of the optimization objective rather than a circular prediction. The paper cites external prior work for the entropy-based advantage ([7,40]) and for the no-standardization choice ([1]), so no self-citation chain is load-bearing. The only notable concern is an internal sign issue: Eq. 11's cosine schedule makes λ negative for t > T/2, which would turn the max-entropy term into entropy minimization; this is a consistency/correctness question, not a circular reduction of the results to the inputs, and therefore does not affect the circularity score.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The method introduces several hand-tuned hyperparameters (entropy coefficients, EMA rates, penalty clip, clip-high) and rests on two domain assumptions: accuracy-derived difficulty is a reliable per-question signal, and the shortest correct response length is a safe compression target. These are not derived from theory and are the main places where CEEH could fail.

free parameters (7)
  • entropy coefficient lambda_0 = 0.001 (ME); 0 (EA)
    Chosen by hand (Table 4); the 5x multiplier for hard questions is also hand-set in Eq 11.
  • entropy advantage scale alpha and exponent kappa = not reported; adopted from Cheng et al. [7]
    Eq 7 parameters are not tuned or listed in this paper; they are imported from prior work.
  • length penalty coefficient beta = 0.1 or 0.2 (ablation Table 3; main setup not explicitly stated)
    Controls the weight of the dynamic length penalty; different values give different accuracy-length trade-offs (Fig 6, Table 3).
  • EMA update rates eta_improve / eta_decay = 0.2 / 0.05
    Asymmetric EMA coefficients in Eq 16, hand-picked to stabilize difficulty estimates.
  • difficulty multiplier 5x = 5
    Eq 11: hard questions get five times the entropy regularization coefficient; no sensitivity analysis is provided.
  • length penalty clip floor = -0.9
    Section 3.3: clips length penalty to [-0.9, 1) to preserve reward ordering between correct and incorrect answers.
  • GRPO clip-high threshold = 0.28
    Upper PPO clipping threshold raised per DAPO; adopted in Section 3.4.
axioms (4)
  • domain assumption Question-level difficulty can be estimated from per-question reward accuracy over K=12 stochastic rollouts and is stable under asymmetric EMA.
    Section 3.2 Eqs 9-10, 15-16: the whole selective regularization depends on this proxy; no evidence is given that accuracy is a reliable difficulty signal.
  • domain assumption Historically shortest correct response length is a valid compression target; compressing correct responses toward it does not remove reasoning steps needed for correctness.
    Section 3.3 Eq 13: if the shortest-correct trajectory is an outlier or a shortcut, the penalty could reward brittle reasoning.
  • domain assumption Entropy regularization improves Pass@k / preserves reasoning capability by sustaining exploration rather than merely increasing output diversity.
    Cited from [6,7,40,46]; the paper relies on this connection to interpret its Pass@k improvements as capability gains.
  • standard math Standard GRPO/RLVR framework with verifiable rewards is a correct optimization setup for these benchmarks.
    Section 2 Eqs 1-4; assumes benchmark answer matching is correct and that the KL penalty and other standard hyperparameters are adequate.

pith-pipeline@v1.3.0-alltime-deepseek · 18371 in / 14053 out tokens · 123896 ms · 2026-08-02T20:37:48.237981+00:00 · methodology

0 comments
read the original abstract

Chain-of-Thought (CoT) has substantially empowered Large Language Models (LLMs) to tackle complex reasoning tasks, yet the verbose nature of explicit reasoning steps incurs prohibitive inference latency and computational costs, limiting real-world deployment. While existing compression methods - ranging from self-training to Reinforcement Learning (RL) with length constraints - attempt to mitigate this, they often sacrifice reasoning capability for brevity. We identify a critical failure mode in these approaches: explicitly optimizing for shorter trajectories triggers rapid entropy collapse, which prematurely shrinks the exploration space and stifles the discovery of valid reasoning paths, particularly for challenging questions requiring extensive deduction. To address this issue, we propose Compress responses for Easy questions and Explore Hard ones (CEEH), a difficulty-aware approach to RL-based efficient reasoning. CEEH dynamically assesses instance difficulty to apply selective entropy regularization: it preserves a diverse search space for currently hard questions to ensure robustness, while permitting aggressive compression on easier instances where the reasoning path is well-established. In addition, we introduce a dynamic optimal-length penalty anchored to the historically shortest correct response, which effectively counteracts entropy-induced length inflation and stabilizes the reward signal. Across six reasoning benchmarks, CEEH consistently reduces response length while maintaining accuracy comparable to the base model, and improves Pass@k relative to length-only optimization.

Figures

Figures reproduced from arXiv: 2602.22642 by Jun Fang, Naiqiang Tan, Qin-Wen Luo, Rui Liu, Sheng-Jun Huang, Sheng Ren, Xiang Chen.

Figure 1
Figure 1. Figure 1: Accuracy–length trade-off in reasoning compres [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The pipeline of our method. The model accuracy is evaluated via GRPO rollouts, and the optimal length is obtained [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics of policy entropy for R1-Distill [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training accuracy on the same dataset, with R1- [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of response token counts on AMC23, [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The dynamics of response length during train [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Implicit Compression Regularization: Concise Reasoning via Internal Shorter Distributions in RL Post-Training

    cs.AI 2026-05 unverdicted novelty 6.0

    ICR creates a virtual shorter distribution from shortest correct on-policy responses to regularize RL post-training toward concise yet accurate reasoning, improving the accuracy-length Pareto frontier on math and know...

Reference graph

Works this paper leans on

54 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Daman Arora and Andrea Zanette. 2025. Training Language Models to Reason Efficiently.CoRRabs/2502.04463 (2025). arXiv:2502.04463 doi:10.48550/ARXIV. 2502.04463

    doi:10.48550/arxiv.2502.04463 2025

  2. [2]

    Hongye Cao, Zhixin Bai, Ziyue Peng, Boyan Wang, Tianpei Yang, Jing Huo, Yuyao Zhang, and Yang Gao. 2025. Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning.CoRRabs/2512.04359 (2025). arXiv:2512.04359 doi:10.48550/ARXIV.2512.04359

    doi:10.48550/arxiv.2512.04359 2025

  3. [3]

    Qiguang Chen, Dengyun Peng, Jinhao Liu, HuiKang Su, Jiannan Guan, Libo Qin, and Wanxiang Che. 2025. Aware First, Think Less: Dynamic Boundary Self-Awareness Drives Extreme Reasoning Efficiency in Large Language Models. CoRRabs/2508.11582 (2025). arXiv:2508.11582 doi:10.48550/ARXIV.2508.11582

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.11582 2025

  4. [4]

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025. Towards Reason- ing Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Mod- els.CoRRabs/2503.09567 (2025). arXiv:2503.09567 doi:10.48550/ARXIV.2503.09567

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.09567 2025

  5. [5]

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2024. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs.CoRRabs/2412.21187 (2024). arXiv:2412.21187 doi:10.48550/ARXIV.2412.21187

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.21187 2024

  6. [6]

    Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. 2025. Pass@k Training for Adaptively Balancing Ex- ploration and Exploitation of Large Reasoning Models.CoRRabs/2508.10751 (2025). arXiv:2508.10751 doi:10.48550/ARXIV.2508.10751

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.10751 2025

  7. [7]

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhen- liang Zhang, and Furu Wei. 2025. Reasoning with Exploration: An Entropy Perspective.CoRRabs/2506.14758 (2025). arXiv:2506.14758 doi:10.48550/ARXIV. 2506.14758

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.14758 2025

  8. [8]

    Zhengxiang Cheng, Dongping Chen, Mingyang Fu, and Tianyi Zhou. 2025. Opti- mizing Length Compression in Large Reasoning Models.CoRRabs/2506.14755 (2025). arXiv:2506.14755 doi:10.48550/ARXIV.2506.14755

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.14755 2025

  9. [9]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems.CoRRabs/2110.14168 (2021). arXiv:2110.14168 https://arxiv.org/ abs/2110.14168

    Pith/arXiv arXiv 2021

  10. [10]

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. 2025. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models.CoRR abs/2505.22617 (2025). arXiv:2505.22617 doi:10.48550/ARXIV.2505.22617

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.22617 2025

  11. [11]

    Yingqian Cui, Pengfei He, Jingying Zeng, Hui Liu, Xianfeng Tang, Zhenwei Dai, Yan Han, Chen Luo, Jing Huang, Zhen Li, Suhang Wang, Yue Xing, Jiliang Tang, and Qi He. 2025. Stepwise Perplexity-Guided Refinement for Efficient Chain- of-Thought Reasoning in Large Language Models. InFindings of the Association for Computational Linguistics, ACL 2025, Vienna, ...

    2025

  12. [12]

    DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.CoRRabs/2501.12948 (2025). arXiv:2501.12948 doi:10.48550/ARXIV.2501.12948

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.12948 2025

  13. [13]

    Gongfan Fang, Xinyin Ma, and Xinchao Wang. 2025. Thinkless: LLM Learns When to Think.CoRRabs/2505.13379 (2025). arXiv:2505.13379 doi:10.48550/ ARXIV.2505.13379

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.13379 2025

  14. [14]

    Yichao Fu, Junda Chen, Yonghao Zhuang, Zheyu Fu, Ion Stoica, and Hao Zhang

  15. [15]

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. InProceedings of the 62nd Annual Meeting of the Association ...

    2024

  16. [16]

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Lin- feng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025. DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning.CoRRabs/2504.11456 (2025). arXiv:2504.11456 d...

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.11456 2025

  17. [17]

    Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong. 2025. TreeRL: LLM Reinforcement Learning with On-Policy Tree Search. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and...

    2025

  18. [18]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. https://openreview.net/forum?id=nZeVKeeFYf9 Conference acrony...

    2022

  19. [19]

    Kerui Huang, Shuhan Liu, Xing Hu, Tongtong Xu, Lingfeng Bao, and Xin Xia

  20. [20]

    Yuxian Jiang, Yafu Li, Guanxu Chen, Dongrui Liu, Yu Cheng, and Jing Shao

  21. [21]

    arXiv:2509.14093 doi:10.48550/ARXIV.2509.14093

    Reasoning Efficiently Through Adaptive Chain-of-Thought Compression: A Self-Optimizing Framework.CoRRabs/2509.14093 (2025). arXiv:2509.14093 doi:10.48550/ARXIV.2509.14093

    doi:10.48550/arxiv.2509.14093 2025

  22. [22]

    Guosheng Liang, Longguang Zhong, Ziyi Yang, and Xiaojun Quan. 2025. ThinkSwitcher: When to Think Hard, When to Think Fast.CoRRabs/2505.14183 (2025). arXiv:2505.14183 doi:10.48550/ARXIV.2505.14183

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.14183 2025

  23. [23]

    arXiv:2509.25133 doi:10.48550/ARXIV.2509.25133

    Rethinking Entropy Regularization in Large Reasoning Models.CoRR abs/2509.25133 (2025). arXiv:2509.25133 doi:10.48550/ARXIV.2509.25133

    doi:10.48550/arxiv.2509.25133 2025

  24. [24]

    Qingbin Li, Rongkun Xue, Jie Wang, Ming Zhou, Zhi Li, Xiaofeng Ji, Yongqi Wang, Miao Liu, Zheming Yang, Minghui Qiu, and Jing Yang. 2025. CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention.CoRR abs/2508.11016 (2025). arXiv:2508.11016 doi:10.48550/ARXIV.2508.11016

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.11016 2025

  25. [25]

    Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang, Yujin Kim, and Se-Young Yun. 2025. Self-Training Elicits Concise Reasoning in Large Language Models. InFindings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025 (Findings of ACL, Vol. ACL 2025), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and M...

    2025

  26. [26]

    Zehui Ling, Deshu Chen, Hongwei Zhang, Yifeng Jiao, Xin Guo, and Yuan Cheng

  27. [27]

    arXiv:2506.10446 doi:10.48550/ARXIV.2506

    Fast on the Easy, Deep on the Hard: Efficient Reasoning via Powered Length Penalty.CoRRabs/2506.10446 (2025). arXiv:2506.10446 doi:10.48550/ARXIV.2506. 10446

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.10446 2025

  28. [28]

    Hanbing Liu, Lang Cao, Yuanyi Ren, Mengyu Zhou, Haoyu Dong, Xiaojun Ma, Shi Han, and Dongmei Zhang. 2025. Bingo: Boosting Efficient Reasoning of LLMs via Dynamic and Significance-based Reinforcement Learning.CoRRabs/2506.08125 (2025). arXiv:2506.08125 doi:10.48550/ARXIV.2506.08125

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.08125 2025

  29. [29]

    Manning, Ste- fano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Ste- fano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. InAdvances in Neural Infor- mation Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Dece...

    2023

  30. [30]

    Thi Nguyen, Linhao Luo, Fatemeh Shiri, Dinh Phung, Yuan-Fang Li, Thuy-Trang Vu, and Gholamreza Haffari. 2024. Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs. InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL, Vol. ACL 2024), ...

    work page doi:10.18653/v1/2024.findings-acl.1 2024

  31. [31]

    Park, Junsu Kim, Gyeongman Kim, Jinyoung Jo, Sean Choi, Jaewoong Cho, and Ernest K

    Jaesung R. Park, Junsu Kim, Gyeongman Kim, Jinyoung Jo, Sean Choi, Jaewoong Cho, and Ernest K. Ryu. 2025. Clip-Low Increases Entropy and Clip-High De- creases Entropy in Reinforcement Learning of Large Language Models.CoRR abs/2509.26114 (2025). arXiv:2509.26114 doi:10.48550/ARXIV.2509.26114

    doi:10.48550/arxiv.2509.26114 2025

  32. [32]

    Ziqing Qiao, Yongheng Deng, Jiali Zeng, Dong Wang, Lai Wei, Guanbo Wang, Fandong Meng, Jie Zhou, Ju Ren, and Yaoxue Zhang. 2025. ConCISE: Confidence- guided Compression in Step-by-step Efficient Reasoning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, Christos Chris...

    doi:10.18653/v1/ 2025

  33. [33]

    Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2024. Chain of Thoughtlessness? An Analysis of CoT in Planning. InAd- vances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave,...

    2024

  34. [34]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.CoRRabs/2402.03300 (2024). arXiv:2402.03300 doi:10.48550/ARXIV.2402.03300

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.03300 2024

  35. [35]

    Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, and Shiguo Lian. 2025. DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models.CoRRabs/2503.04472 (2025). arXiv:2503.04472 doi:10.48550/ARXIV.2503.04472

    doi:10.48550/arxiv.2503.04472 2025

  36. [36]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. HybridFlow: A Flexible and Efficient RLHF Framework. InProceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 - 3 April 2025. ACM, 1279–1297. doi:10.1145/3689031.3696075

    arXiv 2025

  37. [37]

    Pulkit Verma, Ngoc La, Anthony Favier, Swaroop Mishra, and Julie A. Shah

  38. [38]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-Bench Tasks and Whether Chain-of- Thought Can Solve Them. InFindings of the Association for Computational Linguis- tics: ACL 2023, Toronto, Canada, July 9-14, 2023 (Findin...

    2023

  39. [39]

    Xinyu Tang, Yuliang Zhan, Zhixun Li, Wayne Xin Zhao, Zhenduo Zhang, Zujie Wen, Zhiqiang Zhang, and Jun Zhou. 2025. Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards.CoRRabs/2512.21625 (2025). arXiv:2512.21625 doi:10.48550/ARXIV.2512.21625

    doi:10.48550/arxiv.2512.21625 2025

  40. [40]

    Songjun Tu, Jiahao Lin, Qichao Zhang, Xiangyu Tian, Linjing Li, Xiangyuan Lan, and Dongbin Zhao. 2025. Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL.CoRRabs/2505.10832 (2025). arXiv:2505.10832 doi:10.48550/ARXIV.2505.10832

    doi:10.48550/arxiv.2505.10832 2025

  41. [41]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompt- ing Elicits Reasoning in Large Language Models. InAdvances in Neural Infor- mation Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA,...

    2022

  42. [42]

    arXiv:2509.13351 doi:10.48550/ ARXIV.2509.13351

    Teaching LLMs to Plan: Logical Chain-of-Thought Instruction Tuning for Symbolic Planning.CoRRabs/2509.13351 (2025). arXiv:2509.13351 doi:10.48550/ ARXIV.2509.13351

    doi:10.48550/arxiv.2509.13351 2025

  43. [43]

    Chen Wang, Zhaochun Li, Jionghao Bai, Yuzhi Zhang, Shisheng Cui, Zhou Zhao, and Yue Wang. 2025. Arbitrary Entropy Policy Optimization: Entropy Is Control- lable in Reinforcement Fine-tuning.CoRRabs/2510.08141 (2025). arXiv:2510.08141 doi:10.48550/ARXIV.2510.08141

    Pith/arXiv arXiv doi:10.48550/arxiv.2510.08141 2025

  44. [44]

    Jinyuan Wang, Junlong Li, and Hai Zhao. 2023. Self-prompted Chain-of-Thought on Large Language Models for Open-domain Multi-hop Reasoning. InFindings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 (Findings of ACL, Vol. EMNLP 2023), Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computationa...

    2023

  45. [45]

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. 2025. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning.CoRRabs/2506.01939 (2025). ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.01939 2025

  46. [46]

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2025. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?CoRRabs/2504.13837 (2025). arXiv:2504.13837 doi:10.48550/ARXIV.2504.13837

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.13837 2025

  47. [47]

    Heming Xia, Chak Tou Leong, Wenjie Wang, Yongqi Li, and Wenjie Li. 2025. TokenSkip: Controllable Chain-of-Thought Compression in LLMs. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.)...

    doi:10.18653/v1/2025.emnlp-main.165 2025

  48. [48]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement. CoRRabs/2409.12122 (2024). arXiv:2409.12122 doi:10.48550/A...

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.12122 2024

  49. [49]

    Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Zheng Lin, Li Cao, and Weiping Wang. 2025. Dynamic Early Exit in Reasoning Models.CoRR abs/2504.15895 (2025). arXiv:2504.15895 doi:10.48550/ARXIV.2504.15895

    doi:10.48550/arxiv.2504.15895 2025

  50. [50]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.14476 2025

  51. [52]

    Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. 2025. EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity. CoRRabs/2507.21848 (2025). arXiv:2507.21848 doi:10.48550/ARXIV.2507.21848

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.21848 2025

  52. [53]

    Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng

  53. [54]

    CoRRabs/2506.01347 (2025)

    The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning. CoRRabs/2506.01347 (2025). arXiv:2506.01347 doi:10.48550/ARXIV.2506.01347 Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoning Conference acronym ’XX, June 03–05, 2018, Woodstock, NY A Experimental Setup We utilize the verl framewor...

    doi:10.48550/arxiv.2506.01347 2025

  54. [2025]

    InICLR 2025 Workshop on Foundation Models in the Wild

    Reasoning without self-doubt: More efficient chain-of-thought through certainty probing. InICLR 2025 Workshop on Foundation Models in the Wild

    2025