Pith. sign in

REVIEW 4 major objections 5 minor 8 cited by

Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a difficulty-aware length penalty to RL training cuts a reasoning model's token use by about half while holding accuracy roughly flat.

desk verdict ALP is a simple, plausible length-penalty trick with a strong controlled before/after result, but the cross-method comparisons are confounded and the adaptation story is partly built into the objective. read the letter →

arxiv 2506.05256 v2 pith:T5QTFBVQ submitted 2025-06-05 cs.AI cs.LG

Violet Xiang , Chase Blagden , Rafael Rafailov , Nathan Lile , Sang Truong , Chelsea Finn , Nick Haber This is my paper
classification cs.AIcs.LG
keywords adaptivelengthpenaltyreinforcementlearningreasoningefficiencychain-of-thoughttokenreductionGRPOdifficultyestimationinference-timecompute
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that large reasoning models waste inference compute overthinking easy problems, and that the waste can be cut by changing one term in the training reward rather than by curating data, setting user budgets, or imposing fixed length limits. The Adaptive Length Penalty (ALP) estimates each prompt's difficulty during RL training from how often the model solves it across multiple rollouts, then charges a per-token penalty whose weight grows as the prompt becomes easier. Post-training a 1.5B reasoning model this way cuts average token use by about 50% while keeping accuracy roughly unchanged on three math benchmarks, with the savings concentrated on easy problems and reinvested in hard ones. If this holds, efficiency becomes a trainable property of the reward signal rather than a manual configuration, which matters for any deployment where problem difficulty is unknown in advance.

What carries the argument

The central object is the ALP reward term: a length penalty whose coefficient is proportional to the online solve rate $p_{\text{solved}}(q)$, the fraction of $K$ rollouts that answer a prompt correctly. Because group-based advantage estimators such as GRPO already sample many rollouts per prompt, the difficulty estimate adds no extra training cost. The penalty term $\beta\,\frac{|y|}{N}\max(p_{\text{solved}}(q),1/K)$ makes easy prompts, with high solve rate, pay a large per-token tax, while hard prompts are nearly unpenalized; the model is thereby pushed to internalize difficulty and allocate just enough reasoning.

What would settle it

Retrain L1-Exact, L1-Max, ThinkPrune, and R1-Alpha from the same DeepScaleR-1.5B base with the same dataset, batch size, rollouts, and 100 gradient steps as the ALP run, then plot Pass@1 against average tokens per problem; if ALP does not dominate the resulting curves, the comparative claim collapses. A second check is applying ALP to a non-math benchmark with checkable answers: if the roughly 50% token reduction does not transfer, the claim that models internalize difficulty generally is unsupported.

Watch

Extended reading notes

Core claim

ALP's discovery is that problem difficulty can be read off the training signal and turned directly into a length penalty. With $K$ rollouts per prompt, the empirical solve rate $p_{\text{solved}}(q)$ separates easy from hard prompts; the composite reward $r(y,q)=\mathbf{1}[\mathrm{answer}(y)=y^*]-\beta\,\frac{|y|}{N}\max(p_{\text{solved}}(q),1/K)$ charges a per-token cost whose weight grows as the prompt becomes easier, with the clip ensuring even unsolved prompts pay a small floor penalty. Applied to DeepScaleR-1.5B, this yields roughly 50% fewer tokens at comparable Pass@1 across AIME, MATH-500, and OlympiadBench; on MATH-500 the model spends about 500 tokens on its easiest problems and nearly 3000 on its hardest, a roughly 6x span. The same mechanism works on two different 1.5B base models and survives stress tests with up to 60% competition-level problems, and the behavioral analysis shows the compression is selective: repetition, exploration, verification, and backtracking drop sharply while planning language is partly retained.

Load-bearing premise

The load-bearing premise is that the public checkpoints of L1, ThinkPrune, and R1-Alpha, trained on different base models, datasets, hyperparameters, and training steps, are comparable enough to judge method quality; the paper states it could not run head-to-head training under identical conditions, so if those training disparities explain the gap, the claim that ALP beats existing length-control methods gives way.

Editorial extensions

If this is right

  • At every inference budget tested (512 to 4096 tokens), the ALP-trained model keeps Pass@1 within a small margin of the base model while using roughly half the tokens overall.
  • On easy problems the savings are concentrated: ALP spends only 21% of its token budget on the easiest 50% of problems, freeing a surplus it redirects to hard problems, where it spends 5.35x more tokens than on easy ones.
  • The adaptive allocation persists when the difficulty mix changes: with up to 60% competition-level AIME problems, ALP roughly quadruples its token spend while degrading less in accuracy than fixed-budget baselines.
  • Token usage rises monotonically with the model's own perceived difficulty (one minus solve rate) on all three benchmarks, showing the model calibrates to problem difficulty rather than applying blanket compression.
  • Compression is selective: repetition, exploration, verification, and backtracking markers drop by half or more, while planning markers are partially preserved, meaning efficiency comes from removing redundant reasoning rather than uniformly shortening traces.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's math-only scope, the same penalty should transfer to code generation or tool use, since it needs only a checkable correctness signal and multiple rollouts per prompt.
  • A testable prediction the paper does not run: an ALP model's accuracy should degrade more gracefully than fixed-budget baselines when deployment budgets shrink below training budgets, because its allocation is already difficulty-aware.
  • Extending the method, one could treat total token budget as a constraint and solve for the penalty coefficient per batch rather than hand-setting beta, removing the one remaining hyperparameter.
  • A stress test implied by the behavioral analysis: on problems with a deliberately misleading first step, ALP's reduced backtracking should cost accuracy, and if it does not, the claim that the pruning is selective and safe would need revision.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Adaptive Length Penalty (ALP), a reward modification for RL training of reasoning models. ALP estimates each prompt's empirical solve rate from K rollouts and adds a length penalty whose weight grows with solve rate, so easy prompts are penalized more for verbose outputs. The authors post-train DeepScaleR-1.5B and R1-Distill-Qwen-1.5B on math problems and report roughly 50% lower average token use at comparable Pass@1 on MATH-500, AIME, and OlympiadBench. They also report Pareto-style analyses of token allocation, robustness to MATH/AIME mixtures, hyperparameter sweeps, and a keyword-based study of reasoning behavior changes.

Significance. If the controlled before/after result is reliable, ALP is an inexpensive and broadly applicable intervention: it requires only multiple rollouts per prompt (already present in GRPO/Reinforce++), adds no inference-time overhead, and does not require user-specified budgets. The paper includes a controlled comparison on the same DeepScaleR checkpoint, three benchmarks, ablations over beta and context length, two base models, and a reproducible training setup. The main quantitative claim, however, is currently reported without error bars, and the cross-method comparisons are confounded by unmatched training setups. The adaptation analysis is partly circular because difficulty is defined by the same solve rate used in the training penalty. These issues do not invalidate the core idea but substantially temper the breadth of the claims as stated.

major comments (4)
  1. [Section 4, Comparison paragraph and Figure 1] The claim that ALP 'outperforms existing length-control methods' is not supported by the experimental design: L1-Exact, L1-Max, ThinkPrune, and R1-Alpha are public checkpoints trained on different base models, datasets, hyperparameters, and compute budgets, as the text itself acknowledges. Differences in Figure 1 and the efficiency numbers in Section 5.2 could therefore be caused by these training disparities rather than by ALP. Because this comparative claim appears in the contributions list, it is load-bearing; please either add matched training for at least the most important baselines (e.g., a uniform length penalty and an L1-style prompt budget on the same DeepScaleR-1.5B recipe) or explicitly downgrade these results to 'suggestive, uncontrolled comparisons'.
  2. [Section 5.1 and Table 1] The central 50% token-reduction figure comes from a single training run and a single evaluation setting, with no error bars, seed variation, or significance tests. RL fine-tuning is noisy, and the differences in Pass@1 between ALP and the base model (e.g., 0.80 vs. 0.81 on MATH-500 in Table 1) are within the range that could shift with another seed. Please report at least three training seeds (or a bootstrap over evaluation samples) for the main DeepScaleR-1.5B comparison, and give the standard deviation of the token counts and Pass@1 values.
  3. [Section 3, Eq. (2)] Equation (2) as printed does not contain the generated length |y|: the penalty term reads '-beta N max(p_solved(q), K^{-1})', which is independent of the response length. The surrounding text says the penalty is a per-token cost scaled by 1/N, implying the intended formula is something like r = 1[correct] - beta (|y|/N) max(p_solved(q), 1/K). Please state the exact reward formula unambiguously; the current equation is not reproducible and, taken literally, would not penalize verbosity at all.
  4. [Sections 5.2 and 5.4] The analyses of adaptive allocation and 'internalized difficulty' define difficulty as (1 - empirical solve rate) using the evaluated model's own rollouts, and ALP's training reward is a direct increasing function of that same solve rate. The result that ALP uses fewer tokens on high-solve-rate prompts is therefore partly a consequence of the objective by construction, not an independent discovery about learned difficulty calibration. To support the interpretation in Section 5.4, please validate with an external difficulty signal (e.g., problem source/competition tier, or solve rates from a held-out model) or explicitly frame these figures as descriptive of the trained policy rather than evidence of a learned internal difficulty model.
minor comments (5)
  1. [Section 4 / Section 5.5 / Appendix A.1] The context window is inconsistent: Section 4 says 16384, Section 5.5 and Table 1 describe 8K and 4K context windows, and Appendix A.1 says '16324'; please reconcile these values and state which context window was used for the main ALP model.
  2. [Table 2 and Section 5.5] The number of rollouts per prompt is given as 32 in Table 2 and as K=16 in Section 5.5; please make these consistent and state K for every reported run.
  3. [Throughout] Please fix typos and inconsistencies in notation: 'raccuray' in Eq. (2), 'hyper-param' and 'chante' in Section 5.5, 'We introduces' in Section 6, and the model name 'R1-Distill-Qwen' versus 'DeepSeek-R1-Distilled-Qwen-1.5B'.
  4. [Section 5.2] The 'efficiency score' and 'adaptation ratio' are only described verbally; please give explicit formulas so the Pareto analysis can be reproduced.
  5. [Section 5.6] The keyword-based behavior analysis uses overlapping patterns (e.g., 'alternatively' appears in both Exploration and Backtracking reconsideration), which may double-count; please note this limitation or refine the categories.

Circularity Check

1 steps flagged · score 5.0 of 10

ALP's difficulty-adaptation analysis re-measures the solve-rate signal that defines its own training reward; the 50% token-reduction result is empirically independent.

  1. self definitional [Section 5.4 (and Section 5.2) vs. Eq. (2), with difficulty defined in Section 5.4]
    "r(y, q) =1[answer(y) =y∗]| {z } raccuray −βNmax psolved(q), K−1 ... We analyze token allocation as a function of difficulty, defined as one minus the empirical solve rate (computed using 64 rollouts for AIME, 16 for MATH-500 and OlympiadBench). ... Across all three datasets, ALP shows a consistent pattern: token usage increases monotonically with difficulty."

    The training reward in Eq. (2) charges a per-token penalty scaled by max(p_solved(q), 1/K), so prompts with high empirical solve rate receive the largest length penalty by construction. Section 5.4 then defines difficulty as 1 minus the same empirical solve rate and reports that ALP uses fewer tokens on easy (high-p_solved) problems and more on hard ones. The direction of the token-versus-difficulty relationship is entailed by the objective; observing it mainly verifies that RL optimized the reward, rather than independently showing that ALP 'internalizes problem difficulty.' Section 5.2's Pareto ordering by solve rate and its adaptation ratio use the same construct.

full rationale

The central efficiency claim—ALP post-trained on DeepScaleR-1.5B uses roughly 50% fewer tokens with comparable Pass@1 (Section 5.1, Table 1)—is an empirical before/after result on held-out math benchmarks and does not reduce to the training objective. The comparisons against L1, ThinkPrune, and R1-Alpha are confounded by unmatched base models and training budgets, as the paper itself acknowledges in Section 4, but that is an experimental validity threat, not circularity. Self-citations to prior work by the same authors (Sections 1 and 2) are contextual and not load-bearing. However, the paper's mechanism claims (Sections 5.2 and 5.4) define evaluation difficulty as one minus the empirical solve rate, the exact quantity whose inverse scales the length penalty in Eq. (2). Consequently, showing that ALP allocates fewer tokens to easy problems and more to hard problems is in part a restatement of the objective, not an independent discovery; only the specific magnitudes are empirical. This partial self-definitional circularity in the analysis leaves the main efficiency result intact but warrants a non-trivial circularity score of 5.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

ALP introduces no new physical or mathematical entities. Its load-bearing assumptions are that online solve rate is a good difficulty signal, that the penalty shaping is benign, and that learned difficulty calibration transfers at inference. The method has several tuned hyperparameters and an unspecified normalization constant, so the quantitative claims are outcomes of these choices rather than parameter-free derivations.

free parameters (4)
  • beta (length penalty weight) = 1e-7, 1e-8 (Table 1)
    Global coefficient in Eq. (2) chosen by the authors to balance accuracy and token usage; Table 1 sweeps two values. The reported 50 percent reduction depends on this choice.
  • K (rollout count per prompt for solve rate) = 16 (Section 5.5), 32 (Appendix Table 2)
    Number of rollouts per prompt used to estimate solve rate; affects the variance of the difficulty estimate and the penalty clip at 1/K. The inconsistency means the exact training recipe is ambiguous.
  • Context window during training = 16384 (Section 4), 8192 and 4096 (Table 1, Section 5.5)
    Determines the maximum trace length and the normalization scale of the penalty; the paper uses different values across runs and does not clarify which model corresponds to which.
  • Normalization constant N = not specified, 'e.g. the maximum trace length' (Section 3)
    Scales the per-token cost in Eq. (2); the exact value is not reported, which affects the penalty magnitude and the comparability of beta across runs.
assumptions (4)
  • domain assumption The empirical solve rate psolved(q) computed from K rollouts is a reliable per-prompt difficulty estimate.
    Eq. (1) in Section 3; no analysis of the estimate's variance is provided, yet the training penalty and all difficulty analyses depend on it.
  • domain assumption The clipped penalty max(psolved, 1/K) does not distort the accuracy objective on hard prompts.
    Eq. (2); the clip guarantees unsolved prompts still incur a penalty, but the effect of that uniform floor on hard prompts is assumed benign.
  • domain assumption Difficulty inferred from the prompt text at inference transfers to unseen problems.
    Sections 5.3 and 5.4 interpret token allocation on held-out math benchmarks as evidence of internalized difficulty; this assumes the model can identify difficulty without access to psolved.
  • domain assumption The GRPO framework (Shao et al., 2024) used for policy updates is assumed correct and compatible with the extra length reward term.
    Section 4 and Algorithm 1 rely on GRPO for advantage estimation; the paper does not analyze whether the adaptive penalty interacts adversely with the group-based baseline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning." pith.science (2026). https://pith.science/paper/T5QTFBVQ

@misc{pith2026250605256,
  author       = {Pith},
  title        = {Pith review of: Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T5QTFBVQ}},
  note         = {Machine review of arXiv:2506.05256}
}
read the original abstract

Large reasoning models (LRMs) achieve higher performance on challenging reasoning tasks by generating more tokens at inference time, but this verbosity often wastes computation on easy problems. Existing solutions, including supervised finetuning on shorter traces, user-controlled budgets, or RL with uniform penalties, either require data curation, manual configuration, or treat all problems alike regardless of difficulty. We introduce Adaptive Length Penalty (ALP), a reinforcement learning objective tailoring generation length to per-prompt solve rate. During training, ALP monitors each prompt's online solve rate through multiple rollouts and adds a differentiable penalty whose magnitude scales inversely with that rate, so confident (easy) prompts incur a high cost for extra tokens while hard prompts remain unhindered. Posttraining DeepScaleR-1.5B with ALP cuts average token usage by 50\% without significantly dropping performance. Relative to fixed-budget and uniform penalty baselines, ALP redistributes its reduced budget more intelligently by cutting compute on easy prompts and reallocating saved tokens to difficult ones, delivering higher accuracy on the hardest problems with higher cost.

Figures

Figures reproduced from arXiv: 2506.05256 by the authors.

Figure 1
Figure 1. Pass@1 Performance with different inference budgets (512, 1024, 2048, 4096). Inference [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Pareto efficiency analysis reveals how models distribute computational resources across problems of varying difficulty (inference budget 4096). (Left) Cumulative token allocation curves for problems ordered from easiest to hardest, aggregated across MATH-500, OlympiadBench, and AIME datasets. Shaded regions indicate easy (0-50%) and hard (80-100%) problem ranges. (Right) Adaptation ratio is computed as tokens used f… view at source ↗
Figure 3
Figure 3. Performance-efficiency trade-offs under varying problem distributions (inference budget 4096). Each curve shows model behavior as MATH/AIME mixture changes. (Left) N=500 with 0-12% AIME content (typical deployment). (Right) N=100 with 0-60% AIME content (stress test). ALP maintains strong performance across all distributions through adaptive token allocation. 5.4 How Models Internalize Problem Difficulty While previ… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Token allocation reveals how models internally perceive problem difficulty (inference budget 4096). Average tokens used versus difficulty (1 - solve rate) across three datasets. 5.5 Model and Hyperparameter Sensitivity To assess ALP’s robustness and sensitivity, we con…
Figure 5
Figure 5. Figure 5: Token usage by MATH-500 diffi￾culty levels. Next, keeping model and context size fixed, we swept β over {10−7 , 10−8}. The larger penalty (β = 10−7 ) drives faster reductions in average token usage, but only yields a modest drop in pass@1: the β = 10−8 run uses more to…
Figure 6
Figure 6. Figure 6: Reasoning behavior changes reveal selective compression strategies. Average keyword occurrences per response comparing base DeepScaleR-1.5B (blue) and ALP-trained model (orange) across all evaluation datasets. The average length ration between ALP and the base model is…

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Length Penalties Make Chain-of-Thought Less Monitorable

    cs.AI 2026-07 conditional novelty 6.5 of 10

    Length-penalized RL shortens chain-of-thought while preserving accuracy and hint influence, but preferentially removes the cues that let a monitor detect that influence.

  2. Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    LenVM trains a token-level value head to predict discounted remaining length, enabling length control and efficiency steering on LLMs and VLMs.

  3. On the Optimal Reasoning Length for RL-Trained Language Models

    cs.CL 2026-02 conditional novelty 6.0 of 10

    For RL-trained language models, reasoning accuracy peaks at an intermediate output length; longer outputs mainly add dispersion around an improving mode, while too-short outputs under-think.

  4. CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregation

    cs.CV 2026-01 conditional novelty 6.0 of 10

    Iteratively aggregating multiple reasoning trajectories with Grounding-DINO-verified visual evidence improved VLM accuracy on 13 multimodal benchmarks, and an RL-trained version internalizes the aggregation.

  5. Learning to Reason Efficiently with Discounted Reinforcement Learning

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Discounted correctness rewards in GRPO post-training shorten LLM reasoning traces by roughly 13-40% while matching pass@1 accuracy, supported by Blackwell-optimality analysis for restricted policy classes.

  6. ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution

    cs.LG 2026-07 conditional novelty 5.0 of 10

    ZUNA1.1, an open-source 380M EEG diffusion autoencoder, reconstructs variable-length, flexibly masked EEG at least as well as its predecessor and far better than spherical spline interpolation.

  7. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    cs.SE 2026-06 unverdicted novelty 3.0 of 10

    The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.

  8. Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.

Reference graph

Works this paper leans on

17 extracted references · 1 canonical work pages · cited by 8 Pith papers

  1. [2]

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu

    URL https://arxiv.org/abs/2502.04463. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of o1-like llms,

    arXiv

  2. [3]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

    URL https: //arxiv.org/abs/2412.21187. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi ...

    arXiv

  3. [4]

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al

    URLhttps://arxiv.org/abs/2501.12948. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models.arXiv preprint arXiv:2410.07985,

    arXiv

  4. [6]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    arXiv

  5. [7]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al

    URL https://arxiv.org/abs/2504.01296. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    arXiv

  6. [8]

    Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al

    Notion Blog. Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems.arXiv preprint arXiv:2412.09413,

    arXiv

  7. [9]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393,

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393,

    arXiv

  8. [10]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    arXiv

Show all 17 references
  1. [11]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256,

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256,

    arXiv

  2. [12]

    URL https://arxiv.org/abs/ 2408.03314. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guoku...

    arXiv

  3. [13]

    Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al

    URLhttps://arxiv.org/abs/2501.12599. Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of o1-like llms.arXiv preprint arXiv:2501.18585,

    arXiv

  4. [14]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724,

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724,

    arXiv

  5. [15]

    Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though.arXiv preprint arXiv:2501.04682,

    Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though.arXiv preprint arXiv:2501.04682,

    arXiv

  6. [16]

    Monte carlo tree search boosts reasoning via iterative preference learning.arXiv preprint arXiv:2405.00451,

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning.arXiv preprint arXiv:2405.00451,

    arXiv

  7. [2023]

    Backtracking

    12 A Appendix A.1 Implementation details We provide the exact prompts used for evaluating each model in our experiments. All models use the same mathematical problem as input, with model-specific formatting. A.1.1 ThinkPrune Models For both DeepSeek-R1-Distill-Qwen-1.5B-thinkp...

  8. [2024]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008,

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008,

    arXiv

  9. [2025]

    10 Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker

    URLhttps://arxiv.org/abs/2503.04697. 10 Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.