{"id":"b4ee31ab-46b4-44d6-804f-8c36d1615f1b","arxiv_id":"2505.07961","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A length-penalized reinforcement learning method called TLDR reduces reasoning token use by about 50 percent in small math language models without hurting accuracy, and temperature scaling of the end-of-sequence token is a training-free alternative.","lead":"This paper proposes two ways to make small language models reason with fewer tokens: scaling the temperature of the end-of-sequence token to stop earlier, and a reinforcement learning reward that penalizes long answers. Across four math benchmarks, the RL method cuts response length by about half while keeping or improving accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data-overlap audit needed: RL training set is drawn from the same benchmark families as the evaluation, so reported accuracy gains could be memorization rather than generalizable efficient reasoning.","rationale":"The paper's headline is an empirical claim: TLDR shortens responses by roughly 50% with minimal accuracy loss versus the SFT baseline. That claim is only meaningful if the evaluation benchmarks measure generalization. Section 6.2 explicitly says the RL training set combines MATH, AIME, AMC, STILL, and OlympiadBench training sets, and the evaluation uses MATH500, AMC, AIME24, and OlympiadBench. This is a textbook contamination risk: AIME24 is a small fixed contest set of 30 problems, so even a handful of memorized problems shifts accuracy by several points; AMC and OlympiadBench have similar public fixed questions. The paper gives no overlap analysis. The reader's weakest-assumption flags this, and I agree it is the most load-bearing concern. The length-reduction part of the claim is less threatened by leakage, since the reward explicitly penalizes length and the model is evaluated with the same prompts used in training; but the 'without accuracy loss' part and the four-benchmark generalization are directly threatened. A concrete overlap audit is the natural check. If overlap is zero, the central claim stands as a conditional empirical finding; if overlap is nonzero, the accuracy comparisons must be redone on clean subsets. I do not see a more fundamental internal inconsistency that would require rejection: the TS mechanism, the reward design, and the controlled RL-versus-RL comparisons are plausible. Thus the reader's CONDITIONAL verdict remains appropriate.","tokens_in":21991,"tokens_out":7612,"duration_ms":74383,"concrete_test":"Obtain the exact RL training files used (the DeepScaleR-1.5B-Preview training set) and the evaluation files for MATH500, AMC, AIME24, and OlympiadBench. Normalize each problem (strip whitespace/punctuation, normalize LaTeX, optionally use an LLM-based near-duplicate detector) and compute exact and near-duplicate overlap between training and each evaluation set. Report overlap counts per benchmark. If any overlap is found, re-run the TLDR versus SFT baseline comparison on the disjoint subset of each evaluation set (or on MATH500 alone) and check whether the roughly 50% length reduction and 'minimal to no accuracy loss' still hold. Also check the DeepSeek-R1-Distill base model's SFT data for the same evaluation problems, since overlap there would also confound the baseline comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.2 states that the RL training data is 'a combination of the training set of MATH, AIME, AMC, STILL, OlympiadBench, which is same as the training set of DeepScaleR-1.5B-Preview,' while evaluation is on 'MATH500 dataset and the test sets of AMC, AIME24 and OlympiadBench datasets.' The central claim—that TLDR yields roughly 50% token-efficiency gains with minimal accuracy loss—is an empirical generalization across these benchmarks. If any AIME24, AMC test, or OlympiadBench test problems appear in the corresponding training sets, the reported accuracies and the accuracy-parity component of the token-efficiency claim are partly attributable to memorization, not to efficient reasoning. The paper provides no overlap audit, no exact-match or near-duplicate check, and no explicit exclusion statement. Because the training set is explicitly assembled from the same benchmark families as the evaluation, contamination is a concrete risk rather than a hypothetical one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses token-efficient reasoning for small language models (SLMs). It documents that SFT-distilled models produce verbose and repetitive chain-of-thought outputs, and that this verbosity is concentrated in incorrect answers. It then proposes two methods: TS, a training-free temperature scaling of the EOS logit, and TLDR, a length-penalized GRPO variant with prompt-conditioned multi-level penalties. Experiments on MATH500, AMC, AIME24, and OlympiadBench are used to argue that TS is more effective than budget forcing and that TLDR yields roughly 50% token reduction with minimal or no accuracy loss relative to SFT baselines, while allowing prompt-controlled short, moderate, and long responses.","tokens_in":22151,"tokens_out":5045,"duration_ms":47499,"significance":"If correct, the contribution is practically valuable: it offers simple, model-agnostic recipes for reducing inference cost of small reasoning models and highlights a real failure mode of SFT distillation. The paper also provides a useful comparison against contemporaneous length-control methods, including L1 and s1's budget forcing. The significance is conditional, however, because the evaluation infrastructure (single runs, no error bars, small AIME24 test set, and unresolved train/test overlap in Section 6.2) does not yet substantiate the strength of the claimed efficiency-accuracy improvements.","major_comments":[{"comment":"The RL training set is stated to be 'a combination of the training set of MATH, AIME, AMC, STILL, OlympiadBench, which is same as the training set of DeepScaleR-1.5B-Preview,' while evaluation uses MATH500 and the test sets of AMC, AIME24, and OlympiadBench. Since the train and eval sets come from the same benchmark families, the paper needs an exact-match and near-duplicate overlap audit, or an explicit exclusion statement, before the accuracy and accuracy-parity components of the token-efficiency claim can be interpreted as evidence of generalizable efficient reasoning; without it, memorization is a concrete alternative explanation.","section":"§6.2 and Table 4"},{"comment":"All reported accuracies and lengths are single-run numbers with no seeds, error bars, or significance tests. On AIME24 this is especially problematic because the benchmark contains only about 30 problems, so a 3.33% difference is one question; claims such as 'TLDR significantly improves token efficiency by about 50% with minimal to no accuracy loss' need at least a few seeds and a variance-aware comparison.","section":"Tables 2 and 4"},{"comment":"The multi-level length-control result is substantially built into the objective: Eq. (1) penalizes correct answers exactly above Lmax/4 for SHORT and above Lmax/2 for MODERATE, and the prompts for those levels are paired with those penalties during training. To establish 'flexible control' as an empirical finding, the paper should report the achieved length distributions (e.g., calibration of mean and median lengths against the desired quartiles) rather than only the average lengths that the penalty directly shapes; otherwise the claim is partly a restatement of the reward design.","section":"Eq. (1), §6.2"},{"comment":"The comparison between TS and budget forcing is not fully specified: BF is reported after sweeping budgets and selecting the configuration with the shortest output among those beating the base, while the TS result does not state how T was chosen or whether it was also selected with the same information. If T is tuned on the evaluation set, the headline comparison is an oracle-selected intervention; the paper should state the T values used and evaluate sensitivity to that choice.","section":"§4 and Table 1"}],"minor_comments":[{"comment":"The text uses 'L > L/2' and 'L > L/4' where Lmax/2 and Lmax/4 are intended, and the logit notation switches from l to z in Section 4.","section":"Eq. (1) and §4"},{"comment":"The y-axis is labeled 'Repeat rate' but the plotted values are token lengths (0 to 4000), which contradicts the caption and the surrounding text.","section":"Figure 9"},{"comment":"There are several typos, including 'widly', 'thrshold', 'maitaining', 'natually', 'suitbale', 'preciously' in the Appendix D heading, and 'Additioned Experiment Results'.","section":"Throughout"},{"comment":"The layout is confusing: rows for the base models under 'No Length Control' are repeated across SHORT, MODERATE, and LONG, and the relationship between the 'No Length Control' and 'With Length Control' column groups should be clarified in the caption.","section":"Table 4"},{"comment":"No statement about code or checkpoint release is provided; adding one would improve reproducibility.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would need a data-overlap audit and multi-seed evaluation to meet the evidentiary standard of a journal publication; the current version is closer to a strong workshop paper. I do not see authorship or scope issues beyond the mismatch between the strength of the claims and the statistical evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper shows two things worth knowing: temperature scaling of the EOS logit is a cheap, training-free way to shorten SFT-distilled reasoning traces, and a length-penalized GRPO variant (TLDR) can cut average tokens roughly in half while holding or slightly improving accuracy on four math benchmarks. The core observation that SFT models generate long, repetitive traces and keep EOS in the top-five but rarely select it is solid, and the TS result is the cleanest contribution.\n\nThe TLDR design is also reasonable: it adds a prompt-conditioned length penalty to GRPO so the user can choose short/medium/long at inference. Comparisons against s1 budget forcing, prompt control, and L1 are extensive, and the pareto-front plots are informative.\n\nThe soft spots are real. Section 6.2 says the RL training set is the combined training sets of MATH, AIME, AMC, STILL, OlympiadBench—the same families as the evaluation. No overlap audit, no exact-match check. On AIME24 and AMC, where problems are public and reused, contamination risk is concrete, not hypothetical. That alone makes the 'minimal accuracy loss' part of the claim conditional. Second, all numbers are single-run; AIME24 is ~30 problems, so a 3-4 point accuracy gap there is within noise. Third, no code or checkpoints are released, which is frustrating given how cheap the intervention is to verify.\n\nI also want to note that the multi-level length control is partly built into the reward: Equation (1) penalizes correct answers above Lmax/4 and Lmax/2. So the SHORT/MODERATE lengths are expected outcomes of optimization, not emergent behavior. That is fine as engineering, but it should not be presented as a surprise that the model hits those length levels.\n\nThe central token-efficiency comparison—TLDR vs SFT on MATH500—does not depend on the reward design; shorter answers plus equal accuracy is the claim, and that is empirical. So the paper deserves a careful referee, but the reviewer needs to demand a data audit, seeds, and error bars, or at minimum a clear statement that AIME/AMC/OlympiadBench train/test splits were verified disjoint. If the authors supply that, this is a useful recipe for efficient small-model reasoning.\n\nFor you: worth flipping through for the TS result and the repetition analysis, but I wouldn't build on the 50% number until it is independently replicated. I'd send it to peer review with a request for major revision on reproducibility and leakage checks.","headline":"Plausible recipe for cutting small-model reasoning tokens in half, but the headline numbers need an overlap audit and error bars before I'd trust them.","tokens_in":22718,"tokens_out":2677,"would_cite":false,"duration_ms":26089,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Small reasoning models waste tokens because they cannot choose a stopping point; EOS temperature scaling and a length-penalized RL objective cut length by half at little accuracy cost.","keywords":["token-efficient reasoning","small language models","chain-of-thought","GRPO","length penalty","temperature scaling","stopping-point control","supervised fine-tuning"],"falsifier":"Normalize and compare the question text of every training example used for the TLDR reinforcement run against the test questions of MATH500, AMC, AIME24, and OlympiadBench; if any test question appears in training, re-run the four benchmarks after removing overlapping questions and check whether the roughly 50% token reduction and accuracy parity persist.","tokens_in":21764,"feed_emoji":"⚡","tokens_out":8595,"duration_ms":78392,"temperature":0.7,"pith_summary":"This paper argues that small language models distilled by supervised fine-tuning reason wastefully: they cannot choose a good moment to stop, so they emit long, repetitive chains of thought, and wrong answers tend to be the most verbose. To fix this, the authors propose two controls. The first, temperature scaling, raises the sampling probability of the end-of-sequence token at inference time, letting the model stop earlier without retraining. The second, TLDR, adds a length penalty to GRPO reinforcement learning, with different penalty levels bound to prompt phrases such as SHORT, MODERATE, and LONG. On four math reasoning benchmarks, TLDR roughly halves response length while keeping accuracy at or near the supervised baseline, and TS tracks the accuracy-length Pareto front more cleanly than budget forcing.","feed_headline":"Small reasoners can cut thinking tokens ~50% without accuracy loss","feed_subtitle":"EOS temperature scaling plus length-penalized RL teaches small models when to stop, saving compute.","key_machinery":"The load-bearing mechanism is the response-length penalty $\\zeta(L)$ added to the GRPO reward, together with the EOS-logit temperature scaling $z'_{i_{\\mathrm{eos}}} = z_{i_{\\mathrm{eos}}}/T$ for training-free control. The penalty is a piecewise function of response length $L$ with thresholds set by the chosen level ($L_{\\max}/4$ for SHORT, $L_{\\max}/2$ for MODERATE, $L_{\\max}$ for LONG), and it is paired with a prompt token such as \"[Response Length: SHORT]\", letting the same model switch length modes at inference time. The temperature-scaling mechanism performs stopping control at sampling time by making the EOS token more likely when $T<1$.","core_discovery":"The central discovery is that stopping time, not reasoning content, is the main wasted resource in distilled small reasoners. The authors find that the end-of-sequence token is treated by supervised fine-tuning like any ordinary token, so the model never learns an efficient stopping policy; in repeated failure cases the EOS token is among the top-five likely next tokens yet is not selected. Reinforcement learning, by contrast, treats stopping as a decision that affects reward. Their TLDR reward subtracts a length penalty $\\eta(L)=\\alpha L/L_{\\max}$ from the accuracy reward, and a multi-level variant uses thresholds at $L_{\\max}/4$ and $L_{\\max}/2$ so that the prompt can request short, moderate, or long reasoning. With mild penalties the trained models produce responses about 50% shorter than the SFT baseline with minimal-to-no accuracy loss, and the wrong-answer repetition rate drops sharply. A training-free alternative, temperature scaling, modifies only the EOS logit by dividing it by $T<1$, which the authors show achieves a better accuracy-efficiency Pareto front than budget-forcing.","pith_inferences":["Because the paper reports that wrong answers are systematically longer, an untested corollary is that an early-stopping or confidence-gated decoder could allocate tokens adaptively and save additional compute beyond the uniform length penalty.","The same prompt-conditioned reward shaping should transfer to non-mathematical reasoning tasks such as planning or code, where trace length is not tied to difficulty; testing TLDR there would show whether stopping-time control is a general property of RL rather than a math-benchmark artifact.","Without a data-contamination audit, the headline roughly 50% token saving is established primarily on benchmarks that may overlap the training mixture, so a clean evaluation on newly written problems is needed before treating the saving as a property of the method rather than of the data.","Combining TS with TLDR in one pipeline could give both a trained stopping policy and a fine-grained inference-time dial, allowing per-query length targets without additional gradient steps."],"forward_implications":["A single small model trained once with TLDR can serve short, moderate, and long reasoning modes selected by prompt, so deployment can adjust compute per query without retraining.","Token cost at inference can drop by about half at roughly constant accuracy, which lowers serving cost and latency for math-style reasoning workloads.","Budget-forcing and prompt-based test-time controls are shown to miss exact length targets; TS gives a finer, model-agnostic dial, so training-free systems can use it.","RL-based length penalties can reduce both correct and wrong response lengths, implying that the model learns to stop early on problems it cannot solve rather than burning context.","SFT distillation from long teacher traces can hurt small students by inducing repetition; length-aware reward shaping is a more direct way to obtain efficient reasoning."],"supporting_citations":[{"why":"Supplies the supervised-fine-tuned distilled base models whose overly long and repetitive outputs motivate the stopping-time analysis.","marker":"[8]"},{"why":"Provides the budget-forcing baseline and the models used to show that test-time length control is imprecise.","marker":"[19]"},{"why":"Provides the RL-trained base model and the combined training-data recipe that the TLDR experiments use for warm-up.","marker":"[17]"},{"why":"Provides the length-controlled RL baseline that TLDR matches under a maximum-length constraint.","marker":"[1]"},{"why":"Defines the MATH500 evaluation set used for the accuracy-length measurements.","marker":"[15]"},{"why":"Supplies the base model family used for size ablations in the SFT and RL comparisons.","marker":"[26]"},{"why":"Supplies the math-specialized base checkpoints used as short-chain-of-thought starting points.","marker":"[27]"}],"fun_headline_variants":["Stopping control cuts small model tokens ~50%","RL teaches reasoners when to stop, saving half","Small models: smarter stopping, 50% fewer tokens","Temperature scaling and RL boost token efficiency","Efficient reasoning: stopping beats SFT verbosity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy and token-efficiency results transfer to new problems only if the four evaluation benchmarks were not part of the distillation or reinforcement training data; the paper does not show that the sets are disjoint.","fun_headline_variants_meta":{"raw":{"variants":["Stopping control cuts small model tokens ~50%","RL teaches reasoners when to stop, saving half","Small models: smarter stopping, 50% fewer tokens","Temperature scaling and RL boost token efficiency","Efficient reasoning: stopping beats SFT verbosity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1344,"prompt_tokens":1025,"completion_tokens":319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":245}},"tokens_in":641,"tokens_out":319,"duration_ms":3910,"temperature":1.0,"reasoning_tokens":245,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:07:04.997427+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Normalize and compare the question text of every training example used for the TLDR reinforcement run against the test questions of MATH500, AMC, AIME24, and OlympiadBench; if any test question appears in training, re-run the four benchmarks after removing overlapping questions and check whether the roughly 50% token reduction and accuracy parity persist.","supporting_citations":[{"cited_title":"Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica","cited_arxiv_id":null,"evidence_quote":"Provides the RL-trained base model and the combined training-data recipe that the TLDR experiments use for warm-up."}],"review_version":1}