On the Optimal Reasoning Length for RL-Trained Language Models

Daisuke Nohara; Rio Yokota; Taishi Nakamura

REVIEW 3 major objections 4 minor 1 cited by

For RL-trained reasoning models, accuracy is non-monotonic in output length, peaking at an intermediate value, with longer reasoning increasing answer dispersion even as the most common answer improves.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

For RL-trained language models, reasoning accuracy peaks at an intermediate output length; longer outputs mainly add dispersion around an improving mode, while too-short outputs under-think.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection Useful mode-accuracy/dispersion decomposition for RL-trained reasoning models, but the headline non-monotonicity is partly a between-method artifact: length is confounded with training objective. the 3 major comments →

arxiv 2602.09591 v3 pith:3XZMI2FZ submitted 2026-02-10 cs.CL cs.AIcs.LG

On the Optimal Reasoning Length for RL-Trained Language Models

Daisuke Nohara , Taishi Nakamura , Rio Yokota This is my paper

classification cs.CL cs.AIcs.LG

keywords reinforcement learninglength controlreasoning modelschain-of-thoughtdispersionmode accuracyunder-thinkingoverthinking

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the relationship between output length and accuracy in reinforcement-trained reasoning models is not a simple 'longer is better.' Across mathematical reasoning benchmarks, the authors find that for a model that already reasons well (a distilled model), accuracy rises with length up to a peak and then falls, while a base model that must acquire reasoning keeps improving with length. They decompose accuracy into two parts: whether the most frequent answer is correct (mode accuracy) and how spread out the answers are (dispersion). The key claim is that the fall at long lengths comes from dispersion—the model spreads its answers around an increasingly correct center—while the fall at short lengths comes from under-thinking, where the center itself is wrong. This matters because it tells practitioners that length penalties are not uniformly good: they can help a strong reasoner but harm a model still learning to reason.

Core claim

The paper's central discovery is that the non-monotonic length-accuracy curve observed in prior test-time scaling work also arises in RL-trained policies, and that the mechanism is dispersion. Specifically, in the long-output regime, mode accuracy (the frequency of the most common answer being correct) continues to rise or stays flat while average accuracy declines; the decline coincides with rising answer entropy and falling mode share. In the short-output regime, both mode accuracy and mode share are low, indicating under-thinking. This two-sided failure mode is present across mathematical reasoning benchmarks, and robustness checks at equal wall-clock training and longer evaluation contex

What carries the argument

The analysis rests on a decomposition of a policy's sample accuracy into a 'center' and a 'spread.' The center is captured by mode accuracy—whether the most frequent sampled answer is correct—and the spread by answer entropy and mode share. This extends a prior theoretical model in which a Gaussian policy with variance σ² maximizes expected reward at an intermediate variance; the paper transfers that variance-based explanation from test-time interventions to policies actually trained with different length-penalty objectives (RLOO-LP, ALP, DRPO, GRPO, DAPO). The length-control methods themselves are the experimental lever: each induces a different average output length, and the metrics are pl

Load-bearing premise

The paper treats average output length as the causal axis, but each point on the curve is produced by a different training method with its own penalty hyperparameter; performance differences could in principle be caused by the algorithm rather than by the resulting length.

What would settle it

A sweep within a single length-control method that varies only the penalty coefficient, producing many points at closely spaced lengths, and shows accuracy rising monotonically with length for the distilled model would refute the non-monotonic claim. Alternatively, evaluating a fixed RL-trained policy with a decoding constraint that forces longer or shorter outputs while keeping the policy parameters fixed would test whether length itself, rather than the training objective, drives the dispersion.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

If the dispersion mechanism is real, then for a given model there is a computable optimal length range, and length-control hyperparameters should be tuned to land inside it.
For models with strong pre-existing reasoning (e.g., distilled), moderate length penalties can improve efficiency without sacrificing accuracy; for base models, imposing length penalties during RL can block reasoning acquisition.
RL training without explicit length control tends to push policies into the long-output, high-dispersion regime, so naive reward shaping may systematically miss the optimum.
The persistence of the non-monotonic pattern under fixed wall-clock and longer context suggests that simply giving more training compute or longer evaluation will not fix the short-output or long-output failure modes.
A practical target is an automatic controller that adjusts length per problem difficulty, since easy problems need short reasoning and hard problems need long reasoning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper does not test this, but if mode accuracy keeps improving with length even as sample accuracy falls, then majority-voting or self-consistency decoding could rescue the lost performance at long lengths.
The qualitative split between base and distilled models suggests that the 'optimal length' is not a fixed number but a property of the model's prior; a base model's monotonic curve may become non-monotonic after further RL, meaning the same model can move between regimes during training.
The dispersion-driven degradation implies that answer-entropy at inference time could serve as a cheap early-warning signal for over-thinking, allowing early stopping per problem.
Because different length-control methods occupy different points on the same curve, algorithm choice and length are entangled; untangling them would require a within-method sweep with finer control over length.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful mode-accuracy/dispersion decomposition for RL-trained reasoning models, but the headline non-monotonicity is partly a between-method artifact: length is confounded with training objective.

read the letter

The paper has a genuinely useful observation: for a distilled reasoning model, mode accuracy keeps rising with output length even when sample accuracy saturates or falls, and this decomposition—central tendency vs. dispersion—is a nice extension of Ghosal et al. to RL-trained policies. The practical distinction between under-thinking (short outputs, low mode accuracy) and over-thinking (long outputs, high entropy, low mode share) is worth having.

But the stress-test note is on target. The central non-monotonic curve in Figure 1 pools five different RL objectives with hand-tuned penalties. Within RLOO-LP or ALP, decreasing the penalty appears to increase both length and accuracy monotonically; the descending right-hand limb is mostly supplied by the GRPO/DAPO baselines, which produce the longest outputs. So the "optimal intermediate length" is partly a between-method pattern, not a clean causal claim about length. The wall-clock and 64K-context checks rule out compute and truncation, but they do not rule out the algorithm/length conflation. The dispersion analysis inherits the same confound.

What the paper does well: the experimental setup is careful—fixed training budget, equal wall-clock comparison, extended context, truncation-rate analysis, and an honest report of a failed GFPO reproduction. There is no circularity: mode accuracy, entropy, and mode share are computed from sampled outputs, not fitted to the model's predictions. The Qwen3-Base vs. Distill contrast is a plausible and useful finding even if the mechanism is not fully identified.

Soft spots, in proportion: the abstract claims results for code generation, but the body only reports math benchmarks—that claim should be removed or supported. There are no error bars or multiple seeds; each point is a single run, and one restart was needed for an ALP configuration. For a claim that rests on the shape of a curve, that is a real limitation. The main weakness, though, is the confound between length and method. A within-method sweep with more penalty levels, or better, a direct length intervention (e.g., constrained decoding or truncation at inference) would make the causal story credible.

Who is this for? Practitioners choosing length penalties for RL-trained reasoners, and researchers working on overthinking. It deserves a serious referee—the question is relevant and the mode-accuracy/dispersion framing is a good conceptual tool—but the revision should address the confound, add variance, and align the abstract with the actual scope.

Referee Report

3 major / 4 minor

Summary. The paper trains Qwen3-1.7B-Base and DeepSeek-R1-Distill-Qwen-1.5B with several RL length-control objectives (RLOO-LP, ALP, DRPO) and baselines (GRPO, DAPO) on DAPO-Math-17K. It plots accuracy versus average output length, reporting a monotonically increasing trend for Qwen3-1.7B-Base and an inverted-U (non-monotonic) trend for DeepSeek-R1-Distill-Qwen-1.5B. The authors decompose accuracy into mode accuracy, answer entropy, and mode share, and argue that long outputs degrade performance through increased dispersion while short outputs cause under-thinking. Appendix D provides wall-clock and 64K-context robustness checks. The abstract additionally claims the results hold across both mathematical reasoning and code generation.

Significance. If the non-monotonic length-accuracy relationship is genuine, the paper would meaningfully extend Ghosal et al.'s dispersion-based account from test-time interventions to RL-trained policies and give practical guidance for length-control methods. The training setup is carefully engineered, with detailed analyses of batch-size, precision, and truncated importance sampling; the inclusion of multiple methods and penalty sweeps is a strength. The paper also rules out some natural artifacts (wall-clock compute, truncation). However, the central claim is currently supported only by pooled data across distinct RL objectives, and the abstract overclaims code-generation results absent in the body. The significance is therefore conditional on resolving the confounds below.

major comments (3)

[§3.2.1, Figure 1; Appendix D.1] The non-monotonic relationship for DeepSeek-R1-Distill is obtained by pooling points from different RL objectives (GRPO Eq. 9, DAPO Eq. 10, RLOO-LP Eqs. 1–3, ALP Eq. 4, DRPO Eqs. 7–8) at hand-selected penalty strengths. Output length is an endogenous outcome of training, not an independent variable. The declining right-hand limb may be supplied by the GRPO/DAPO baselines rather than by a within-method length–accuracy tradeoff. The robustness checks in D.1 rule out wall-clock compute and truncation, but they do not address this algorithm/length conflation. Please provide within-method analyses (e.g., for each algorithm, sweep the penalty and show whether length versus accuracy is monotonic or non-monotonic internally) or a controlled length perturbation on a fixed policy. Without this, the causal 'dispersion under long outputs' interpretation is not supported. In addition, each plotted po
[Abstract] The abstract states the results hold 'across both mathematical reasoning and code generation,' but the full paper presents only mathematical benchmarks (AIME 2024, AIME 2025, AMC, MATH-500). No code-generation experiments appear in Section 3, the appendices, or elsewhere. This overclaim is load-bearing for the abstract's scope claim. Either add the missing code-generation experiments or narrow the claim to mathematical reasoning.
[§3.2.2, Figure 2; Appendix D.2] The mode-accuracy/entropy/mode-share analysis is used to support the key mechanistic claim that long outputs increase dispersion while mode accuracy stays high. These metrics are computed from 16 or 64 sampled responses per problem and plotted as point clouds without confidence intervals; the observational claim is made by inspection of a small number of points. More importantly, the same pooled-method confound applies: mode accuracy and entropy are averaged over the same set of methods and hyperparameters, so differences could reflect algorithm identity rather than the effect of output length. Quantify the uncertainty (e.g., bootstrap or repeated evaluation) and test whether the mode-accuracy/entropy pattern holds within individual method families (RLOO-LP/ALP sweeps), not just across the pooled cloud.

minor comments (4)

[Figure captions and legends] The legend entries display values like '=1.0' without the hyperparameter symbols (α, β, λ) used in the text (e.g., Figure 1 and Figure 2). This makes it hard to map points to methods. Please include the symbols in all legends.
[§3.1 / Appendix A.1] The reported step counts differ between models (640 vs 480). The rationale is stated in the appendix but should appear in the main text, especially since wall-clock equality is later used as a robustness check.
[Appendix A.4] The ALP β=1e-4 run that diverged and was restarted is reported in the appendix. This training-instability caveat should be mentioned where ALP results are first used, since it indicates run-to-run variability that is otherwise not quantified in the figures.
[General] The figures are the only source for the exact accuracy and length values. A supplementary table with per-method, per-benchmark accuracy, average length, and mode metrics would aid reproducibility and make the claimed non-monotonic shape easier to assess quantitatively.

Circularity Check

0 steps flagged

No circularity: the paper's claims are direct empirical measurements with no fitted parameters, self-citation chain, or definitional reduction.

full rationale

The paper's central claims are empirical. Accuracy, mode accuracy, answer entropy, and mode share are defined operationally and computed directly from sampled responses (Section 3.2.2, Appendix D.2); they are not derived by fitting any parameter to the outcome and then calling that fit a prediction. The Ghosal et al. Gaussian model is used only as an interpretive analogy, not fitted to this paper's data and not used to define the metrics. The length-control methods (RLOO-LP, ALP, DRPO) define training rewards via length penalties (Eqs. 2, 4, 7-8), but those penalties are training objectives, not equivalent to or derived from the evaluation accuracy plotted in Figure 1. There are no load-bearing self-citations: the authors do not cite their own prior work as the source of any central premise, uniqueness theorem, or ansatz. The reader-flagged issue that output length is endogenous and confounded with the choice of RL objective is a genuine validity threat, but it is not circularity-by-construction: the plotted length-accuracy relationship is an observed correlation, not a quantity forced to equal its inputs by definition. No specific reduction of the form 'Eq. X = Eq. Y by construction' or 'fitted parameter renamed as prediction' can be exhibited from the manuscript. The paper is self-contained as an empirical study, so the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim is empirical and does not introduce new mathematical entities. The main load-bearing choices are the hand-selected penalty hyperparameter sweeps that generate the length axis, plus the interpretive assumptions that sampled answer distributions reflect the policy and that the Ghosal Gaussian mechanism carries over to trained policies. No new particles, mediators, or conserved quantities are introduced.

free parameters (4)

RLOO-LP penalty strength α = 0.1, 0.2, 0.4, 0.8, 1.0
Hand-chosen sweep across penalty strengths; each value produces a different average output length, so the central length-accuracy curve depends on these choices.
ALP penalty strength β = 1e-6, 1e-5, 1e-4, 3e-4, 1e-3
Hand-chosen sweep used to vary output length; the non-monotonic trend is assembled from points at these β values.
DRPO length weight λ = 0.02, 0.05, 0.1, 0.2, 0.5
Hand-chosen regularization sweep controlling how strongly shorter responses are favored inside the DRPO objective.
Maximum response length caps = 8K tokens (Qwen3-1.7B), 16K tokens (DeepSeek-R1-Distill)
Chosen so fewer than 5% of rollouts exceed the cap at training start; this bounds which output lengths are observable and affects the long-output tail.

axioms (4)

domain assumption Ghosal et al.'s one-dimensional Gaussian model for length-performance applies to RL-trained policies and not just test-time interventions.
The paper uses the Gaussian-variance story as the explanatory mechanism for the observed training-policy results without directly measuring the policy's distributional family. Invoked in Section 3.2.2.
domain assumption Sampling 16 or 64 responses per problem at temperature 0.6 yields a reliable estimate of the answer distribution, mode, and entropy.
Mode accuracy and mode share are computed from these samples; if the sample size is too small, mode estimates become noisy. Stated in Appendix A.2.
domain assumption DAPO-Math-17K training and AIME/AMC/MATH-500 evaluation are representative of mathematical reasoning behavior.
The central conclusions about length-accuracy trade-offs are drawn from this single training distribution and these four math benchmarks.
domain assumption The RL training instability controls (batch-size matching, precision/TIS) do not bias the reported length-accuracy relationship.
The paper shows that unstable runs can occur and reports a restarted run; the assumption is that the reported successful runs are representative. Discussed in Appendix A.3-A.4.

reviewed 2026-08-03 · how reviews work

0 comments

Cite this review

Pith. "Pith review of On the Optimal Reasoning Length for RL-Trained Language Models." pith.science (2026). https://pith.science/paper/3XZMI2FZ

@misc{pith2026260209591,
  author       = {Pith},
  title        = {Pith review of: On the Optimal Reasoning Length for RL-Trained Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3XZMI2FZ}},
  note         = {Machine review of arXiv:2602.09591}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

Reinforcement learning substantially improves reasoning in large language models, but it also tends to lengthen chain-of-thought outputs and increase computational cost. Although length-control methods have been proposed, the length-accuracy relationship they induce remains unclear. We train policies with several length-control methods on multiple base models in a controlled setup and find that, across both mathematical reasoning and code generation, accuracy is non-monotonic in output length, peaking at an intermediate value. Mode accuracy, however, continues to improve with length even in settings where sample accuracy plateaus or declines, indicating that the non-monotonic length-accuracy relationship is driven by dispersion around an increasingly correct center.

Figures

Figures reproduced from arXiv: 2602.09591 by Daisuke Nohara, Rio Yokota, Taishi Nakamura.

**Figure 1.** Figure 1: Score vs. average output length. Qwen3-1.7B-Base (top) shows a monotonically increasing trend, while DeepSeek-R1-Distill-Qwen-1.5B (bottom) exhibits a non-monotonic relationship with optimal performance at intermediate lengths. We also observe that Sample Avg (GRPO) and DRPO perform poorly on Qwen3-1.7B-Base compared to other methods, while achieving comparable performance on DeepSeek-R1-Distill (Appen… view at source ↗

**Figure 2.** Figure 2: Decomposition of accuracy into mode accuracy and dispersion metrics for DeepSeekR1-Distill-Qwen-1.5B on AMC and MATH-500. In the long-output regime, degradation is driven by increased dispersion; in the short-output regime, both central tendency and dispersion are affected. pendix D.2. In the long-output regime beyond the performance peak, we observe that mode accuracy remains stable or even improves, ye… view at source ↗

**Figure 3.** Figure 3: Effect of batch size configuration on training dynamics. Top left: response length during training. Top right: absolute difference between rollout and training token probabilities. Bottom: validation scores on MATH-500 and AIME 2024. The 512/32 setting (generation batch size 512, mini-batch size 32) leads to decreasing response length and validation performance, while the 64/64 setting maintains stable tra… view at source ↗

**Figure 4.** Figure 4: Effect of precision and TIS on training dynamics for DeepSeek-R1-Distill-Qwen1.5B. Comparison of BF16 with TIS, FP16 with TIS, FP16 without TIS, and ALP (β = 1e−4) in FP16 without TIS. Note that the maximum response length differs between the ALP experiment and the precision ablation experiments, so response lengths should not be directly compared across these settings. Arora & Zanette (2025) define a rew… view at source ↗

**Figure 5.** Figure 5: GFPO reproduction attempt on two base models. Average output length during training for GFPO (G = 16, k = 8) compared to the DAPO baseline. Both Qwen3-1.7B-Base and DeepSeek-R1-Distill-Qwen-1.5B show increasing output length under GFPO, particularly in later training stages. C GRPO AND DAPO OBJECTIVES Group Relative Policy Optimization (GRPO) (Shao et al., 2024) eliminates the need for a separate value mod… view at source ↗

**Figure 6.** Figure 6: Evolution of output length during training (20-step moving average). Left: average output length. Right: length bias (normalized difference between correct and incorrect response lengths). On Qwen3-1.7B-Base, Sample Avg (GRPO) and DRPO exhibit large negative length bias, indicating that incorrect responses tend to be longer than correct ones [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

**Figure 7.** Figure 7: Score vs. average output length for DeepSeek-R1-Distill-Qwen-1.5B at equal wallclock training time The non-monotonic relationship persists, indicating that performance degradation with short outputs is not due to reduced training tokens. D.1.2 EXTENDED CONTEXT LENGTH EVALUATION Another potential explanation for performance degradation at long output lengths is that responses may exceed the evaluation con… view at source ↗

**Figure 8.** Figure 8: Score vs. average output length for DeepSeek-R1-Distill-Qwen-1.5B with 64K evaluation context length. The non-monotonic pattern is consistent with the 32K evaluation ( [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

**Figure 9.** Figure 9: Truncation rate at 32K vs. 64K evaluation context length for DeepSeek-R1-DistillQwen-1.5B. Each point represents a method with a specific hyperparameter configuration. Points close to the diagonal indicate that extending the context length does not substantially reduce truncation. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗

**Figure 10.** Figure 10: Mode accuracy, answer entropy, and mode share vs. average output length for DeepSeek-R1-Distill-Qwen-1.5B across all benchmarks after 480 training steps. This figure extends [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Implicit Compression Regularization: Concise Reasoning via Internal Shorter Distributions in RL Post-Training
cs.AI 2026-05 unverdicted novelty 6.0

ICR creates a virtual shorter distribution from shortest correct on-policy responses to regularize RL post-training toward concise yet accurate reasoning, improving the accuracy-length Pareto frontier on math and know...

Reference graph

Works this paper leans on

11 extracted references · 3 linked inside Pith · cited by 1 Pith paper

[2]

net/forum?id=tKPqbamNb9

URLhttps://openreview. net/forum?id=tKPqbamNb9. Gang Li, Yan Chen, Ming Lin, and Tianbao Yang. Drpo: Efficient reasoning via decoupled reward policy optimization, 2025a. URLhttps://arxiv.org/abs/2510.04474. Gang Li, Ming Lin, Tomer Galanti, Zhengzhong Tu, and Tianbao Yang. DisCO: Reinforcing large reasoning models with discriminative constrained optimizat...

arXiv
[3]

Qwen Team

URLhttps://arxiv.org/abs/ 2510.26788. Qwen Team. Qwen3 technical report. arXiv:2505.09388,

arXiv
[4]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H

URLhttps://arxiv.org/abs/2508.09726. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.),Ad- vances in Neural Information Processing Systems,

Pith/arXiv arXiv
[5]

Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao

URLhttps://arxiv.org/abs/2506.05256. Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training, August

Pith/arXiv arXiv
[6]

net/forum?id=2a36EMSSTp

URLhttps://openreview. net/forum?id=2a36EMSSTp. 6 On the Optimal Reasoning Length for RL-Trained Language Models A EXPERIMENTALDETAILS A.1 TRAINING All models are trained on the DAPO-Math-17K dataset using the verl framework. Our configura- tion adopts GRPO with the KL penalty removed (β= 0), computing group-relative advantages by normalizing rewards acro...

2024
[8]

RLOO samplesnresponses{y 1,

applies length-based reward shaping to correct responses with per-prompt normalization with RLOO (REINFORCE Leave-One-Out) advantage estimation. RLOO samplesnresponses{y 1, . . . , yn}for a promptxand estimates the advantage for each re- sponsey i as: ˆA(yi, x) =R(yi, x)− 1 n−1 X j̸=i R(yj, x)(1) 8 On the Optimal Reasoning Length for RL-Trained Language M...

2000
[9]

Figure 4:Effect of precision and TIS on training dynamics for DeepSeek-R1-Distill-Qwen- 1.5B.Comparison of BF16 with TIS, FP16 with TIS, FP16 without TIS, and ALP (β= 1e−4) in FP16 without TIS. Note that the maximum response length differs between the ALP experiment and the precision ablation experiments, so response lengths should not be directly compare...

2025
[10]

However, this is an artifact of our logging implementation, which computes the average lengthafter filtering for the shortest outputs rather than across all sampled responses. 10 On the Optimal Reasoning Length for RL-Trained Language Models We hypothesize that GFPO’s filtering mechanism, which excludes longer outputs from training, also prevents the mode...

2000
[11]

C.1 ANALYSIS OFOUTPUTLENGTHVARIANCE We analyze why Sample Avg (GRPO) and DRPO exhibit unstable training on Qwen3-1.7B-Base compared to DeepSeek-R1-Distill-Qwen-1.5B

modifies this by normalizing by the total token count: JDAPO(θ) =E   1PG i=1 |yi| GX i=1 |yi|X t=1 min ρi,t ˆAi,t,clip(ρ i,t,1−ϵ,1 +ϵ) ˆAi,t   (10) These normalization schemes can affect training stability depending on the variance of output lengths, as analyzed in the next section. C.1 ANALYSIS OFOUTPUTLENGTHVARIANCE We analyze why Sample Avg (GRPO) ...

2000
[32]

For DeepSeek-R1-Distill-Qwen-1.5B, we initially attempted to train in BF16 with TIS, but encoun- tered training instability despite using TIS

leads to decreasing response length and validation performance, while the 64/64 setting maintains stable training. For DeepSeek-R1-Distill-Qwen-1.5B, we initially attempted to train in BF16 with TIS, but encoun- tered training instability despite using TIS. Following Qi et al. (2025), we switched to FP16, which substantially reduced the probability differ...

2025
[2025]

DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning

DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948,

Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.

[1] [2]

net/forum?id=tKPqbamNb9

URLhttps://openreview. net/forum?id=tKPqbamNb9. Gang Li, Yan Chen, Ming Lin, and Tianbao Yang. Drpo: Efficient reasoning via decoupled reward policy optimization, 2025a. URLhttps://arxiv.org/abs/2510.04474. Gang Li, Ming Lin, Tomer Galanti, Zhengzhong Tu, and Tianbao Yang. DisCO: Reinforcing large reasoning models with discriminative constrained optimizat...

arXiv

[2] [3]

Qwen Team

URLhttps://arxiv.org/abs/ 2510.26788. Qwen Team. Qwen3 technical report. arXiv:2505.09388,

arXiv

[3] [4]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H

URLhttps://arxiv.org/abs/2508.09726. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.),Ad- vances in Neural Information Processing Systems,

Pith/arXiv arXiv

[4] [5]

Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao

URLhttps://arxiv.org/abs/2506.05256. Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training, August

Pith/arXiv arXiv

[5] [6]

net/forum?id=2a36EMSSTp

URLhttps://openreview. net/forum?id=2a36EMSSTp. 6 On the Optimal Reasoning Length for RL-Trained Language Models A EXPERIMENTALDETAILS A.1 TRAINING All models are trained on the DAPO-Math-17K dataset using the verl framework. Our configura- tion adopts GRPO with the KL penalty removed (β= 0), computing group-relative advantages by normalizing rewards acro...

2024

[6] [8]

RLOO samplesnresponses{y 1,

applies length-based reward shaping to correct responses with per-prompt normalization with RLOO (REINFORCE Leave-One-Out) advantage estimation. RLOO samplesnresponses{y 1, . . . , yn}for a promptxand estimates the advantage for each re- sponsey i as: ˆA(yi, x) =R(yi, x)− 1 n−1 X j̸=i R(yj, x)(1) 8 On the Optimal Reasoning Length for RL-Trained Language M...

2000

[7] [9]

Figure 4:Effect of precision and TIS on training dynamics for DeepSeek-R1-Distill-Qwen- 1.5B.Comparison of BF16 with TIS, FP16 with TIS, FP16 without TIS, and ALP (β= 1e−4) in FP16 without TIS. Note that the maximum response length differs between the ALP experiment and the precision ablation experiments, so response lengths should not be directly compare...

2025

[8] [10]

However, this is an artifact of our logging implementation, which computes the average lengthafter filtering for the shortest outputs rather than across all sampled responses. 10 On the Optimal Reasoning Length for RL-Trained Language Models We hypothesize that GFPO’s filtering mechanism, which excludes longer outputs from training, also prevents the mode...

2000

[9] [11]

C.1 ANALYSIS OFOUTPUTLENGTHVARIANCE We analyze why Sample Avg (GRPO) and DRPO exhibit unstable training on Qwen3-1.7B-Base compared to DeepSeek-R1-Distill-Qwen-1.5B

modifies this by normalizing by the total token count: JDAPO(θ) =E   1PG i=1 |yi| GX i=1 |yi|X t=1 min ρi,t ˆAi,t,clip(ρ i,t,1−ϵ,1 +ϵ) ˆAi,t   (10) These normalization schemes can affect training stability depending on the variance of output lengths, as analyzed in the next section. C.1 ANALYSIS OFOUTPUTLENGTHVARIANCE We analyze why Sample Avg (GRPO) ...

2000

[10] [32]

For DeepSeek-R1-Distill-Qwen-1.5B, we initially attempted to train in BF16 with TIS, but encoun- tered training instability despite using TIS

leads to decreasing response length and validation performance, while the 64/64 setting maintains stable training. For DeepSeek-R1-Distill-Qwen-1.5B, we initially attempted to train in BF16 with TIS, but encoun- tered training instability despite using TIS. Following Qi et al. (2025), we switched to FP16, which substantially reduced the probability differ...

2025

[11] [2025]

DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning

DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948,

Pith/arXiv arXiv

On the Optimal Reasoning Length for RL-Trained Language Models

The reading

Core claim §

What carries the argument

Load-bearing premise §

What would settle it §

If this is right

Where Pith is reading between the lines

discussion (0)

Core claim

Load-bearing premise

What would settle it