Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Temporal Sampling for Forgotten Reasoning in LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned LLMs forget answers they once knew, and sampling earlier checkpoints recovers them.

desk verdict A clean, useful paper with a simple inference-time trick; the forgetting numbers are suggestive but the greedy-to-temperature gap and missing best-checkpoint baseline need fixing. read the letter →

arxiv 2505.20196 v1 pith:D73LVBUG submitted 2025-05-26 cs.AI cs.LG

Yuetai Li , Zhangchen Xu , Fengqing Jiang , Bhaskar Ramasubramanian , Luyao Niu , Bill Yuchen Lin , Xiang Yue , Radha Poovendran This is my paper
classification cs.AIcs.LG
keywords temporalforgettingsamplingreasoningbenchmarkscheckpointensemblereinforcementlearningfine-tuningsupervisedPass@kLoRA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fine-tuning is supposed to make reasoning models better, but this paper argues that the standard final-checkpoint view hides much of what training produces. Across RL and SFT runs on Qwen2.5 models and several math benchmarks, 6.4% to 56.1% of the answers wrong at the final checkpoint were correct at some earlier checkpoint. The paper introduces Temporal Sampling, which spends the same inference budget drawing answers from several saved checkpoints instead of only the last one, and reports 4 to 19 point Pass@k gains plus consistent Majority@k and Best-of-N gains at equal compute. If this holds, evaluating a model only by its final weights systematically underestimates the reasoning ability a training run actually contains.

What carries the argument

The load-bearing machinery is a checkpoint-level accounting plus a decoding allocation. Ever Correct and Temporal Forgetting Scores turn a training run into per-question correctness trajectories across the eight saved checkpoints. Temporal Sampling then distributes the inference budget according to a balanced integer partition of k across t checkpoints in round-robin order, so each checkpoint contributes either floor(k/t) or ceil(k/t) samples; the paper defines Pass@k|t this way, shows it reduces to standard Pass@k when t=1, and proves an unbiased hypergeometric estimator for it. The same allocation is applied to majority voting as Maj@k|t and to Best-of-N selection.

What would settle it

Compare each checkpoint's correctness under greedy decoding with its Pass@1 under temperature-0.6 sampling using the same number of samples; if checkpoints judged correct greedily are no more likely to yield correct sampled answers than checkpoints judged wrong, the Temporal Forgetting Score would not translate into Temporal Sampling gains.

Watch

Extended reading notes

Core claim

The paper's central discovery is Temporal Forgetting: individual questions oscillate between correct and incorrect across training checkpoints, even when overall benchmark scores improve. It defines Ever Correct Score (PECS), the share answered correctly by at least one checkpoint, and Temporal Forgetting Score (PTFS = PECS - P_FT), the share correct somewhere but wrong at the final checkpoint; across five benchmarks this ranges from 6.4% to 56.1%, averaging about 25 points. Because correct reasoning paths are transient, final-checkpoint-only evaluation is a misleading lens. The companion method, Temporal Sampling, allocates k samples round-robin across t checkpoints and recovers many of those transient solutions without retraining or ensembling; at k=64, Pass@k with t=8 beats final-only sampling by 19, 13, and 4 points on AIME2024, AMC, and AIME2025, and Majority@k and Best-of-N also improve. Storing LoRA adapters instead of full checkpoints preserves most of the benefit.

Load-bearing premise

The load-bearing premise is that a checkpoint marked correct by a single greedy decoding actually contains a recoverable correct answer under the sampling used for evaluation; if the correct answer at a checkpoint is mostly decoding luck, the claimed recovered ability would shrink.

Editorial extensions

If this is right

  • Final-checkpoint-only Pass@k, Majority@k, and Best-of-N understate the reasoning competence stored in a fine-tuning run.
  • Using t=8 checkpoints instead of the final one yields 4 to 19 point Pass@k gains and 7 to 8 point Majority@k gains on AIME2024, AMC, and AIME2025 at the same sample budget.
  • Temporal Sampling also strengthens test-time scaling: more checkpoints generally improves Best-of-N selection with process reward models, at no extra training cost.
  • Checkpoint diversity can be stored cheaply: with LoRA adapters, Temporal Sampling beats final-checkpoint sampling for both Pass@k and Maj@k while storing only adapter weights.
  • The new Pass@k|t estimator gives a statistically valid way to compare sampling strategies across checkpoints, with t=1 recovering standard Pass@k.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same temporal-diversity argument should apply to other post-training families such as DPO, PPO variants, and rejection sampling, but the paper only tests GRPO and SFT; if the mechanism is general, Temporal Sampling could boost code generation or theorem proving without retraining.
  • Editorial inference: because gains come from checkpoints that were correct under greedy decoding, the checkpoint pool could be chosen adaptively—selecting the t checkpoints with the highest per-question Ever Correct rate rather than the last t—which would likely reduce storage and compute further.
  • Editorial inference: the paper's own conclusion that competence lives in training dynamics suggests a natural follow-up of compressing the trajectory into a single parameter set, so that checkpoint merging or trajectory distillation might transfer the recovered ability into Pass@1.
  • Editorial inference: the Pass@k|t framing implies that checkpoint frequency is an inference-time hyperparameter, not just a training-log detail, so training runs should be saved with deployment in mind.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports that fine-tuned LLMs frequently answer benchmark questions correctly at intermediate training checkpoints but incorrectly at the final checkpoint, a phenomenon the authors call Temporal Forgetting. They quantify this with two metrics, P_ECS and P_TFS, based on greedy decoding of 8 saved checkpoints from GRPO and SFT runs, and report P_TFS values between 6.4% and 56.1%. They then propose Temporal Sampling, which allocates an inference-time sampling budget across the t most recent checkpoints in a round-robin manner, and introduce a Pass@k|t metric with an unbiased estimator (Appendix C). Experiments on AIME24, AMC, and AIME25 report 4-19 point Pass@k gains, consistent Majority@k and Best-of-N gains, and a LoRA variant that reduces storage cost.

Significance. If the central claims hold, the paper would make a useful empirical contribution: it challenges the common practice of evaluating only the final checkpoint and shows that intermediate parameter snapshots can be exploited at inference time without retraining. The paper has genuine strengths: the unbiased estimator proof in Appendix C is correct, the main comparisons hold the total number of generated samples fixed, the LoRA extension addresses a practical storage concern, and the comparison against a Mixture-of-Models baseline is a good first step. However, the current evidence does not yet establish the recovery claim, because the forgetting metrics are computed with greedy decoding while Temporal Sampling is evaluated with temperature-0.6 sampling, and because no baseline using the best single checkpoint is reported. The statistical basis is also thin on the smallest benchmarks. These gaps are fixable, but they are load-bearing for the paper's central narrative.

major comments (3)
  1. [Section 2.2 vs Section 3.3] The Temporal Forgetting measurements in Section 2.2 use one greedy rollout per checkpoint, explicitly 'to minimize variability,' while Temporal Sampling in Section 3.3 is evaluated with temperature-0.6, top-p 0.95 sampling. These protocols are never connected. A question that is greedy-correct at a checkpoint need not be recoverable under the sampling distribution actually used by Temporal Sampling, so the 6.4%-56.1% P_TFS values are not yet shown to represent recoverable forgotten ability. The paper should add a per-question analysis: for each problem, determine whether the correct samples obtained by Pass@k|t come from checkpoints that were greedy-ever-correct, and report the fraction of P_TFS questions that Temporal Sampling actually recovers. Without this, the claim in the abstract that Temporal Sampling 'recovers forgotten solutions' is not directly supported.
  2. [Section 3.4, Figure 5] The comparison against t=1 (final checkpoint only) does not isolate the benefit of temporal diversity. With round-robin allocation, the gain could come simply from including a stronger earlier checkpoint in the pool rather than from sampling across multiple checkpoints. The paper should add a baseline that spends the entire budget k on a single best checkpoint (e.g., the checkpoint with the highest Pass@1 on a validation split, or an oracle upper bound using the empirically best checkpoint) and show that Temporal Sampling still beats this baseline. Without this, 'decoding across checkpoints' is not distinguished from 'using a better checkpoint.'
  3. [Table 7 and Figure 5] The main quantitative claims rest partly on very small benchmarks: AIME24 has 30 questions and AMC has 40. A 19-point gain on AIME24 at k=64 corresponds to roughly 6 questions, and no confidence intervals, standard errors, or significance tests are reported anywhere in the paper. The paper should provide bootstrap confidence intervals for the Pass@k, Maj@k, and Best-of-N differences, and for the P_TFS percentages, so that the reported 4-19 point gains and the 6.4%-56.1% range can be interpreted as more than sampling noise.
minor comments (5)
  1. [Section 3.3] The text says 'For each problem, we generated 64 responses' but does not state whether N=64 candidate samples are generated per checkpoint or per problem across all checkpoints; please clarify the value of N used in the Pass@k|t estimator.
  2. [Figure 3(b)] The term 'Ever Forgotten' is used in the figure but is not defined in the text; please define it (e.g., the percentage of questions that experience at least one correct-to-incorrect transition) and specify whether it is a per-question or per-transition count.
  3. [References] References [38] and [39] are duplicate citations of the same DeepSeekMath paper, and references [41] and [42] appear to be the same arXiv paper duplicated; please consolidate.
  4. [Section 4 and Figure 9] Section 4 and Figure 9 refer to 'AIME' without specifying the year, while the appendix (Figure 13) specifies AIME24 and AMC; please make the benchmark names consistent.
  5. [Table 1] The text says the analysis shows 'in spite of the improvement of overall performance,' but some model entries, such as S1.1-1.5B on OlympiadBench, have P_FT < P_Base; please qualify the summary statement accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's metrics are definitions, its estimator is a standard unbiased hypergeometric calculation, and its gains are empirical comparisons against a t=1 baseline.

full rationale

The derivation chain is not circular. The Temporal Forgetting Score is defined as P_TFS = P_ECS - P_FT (Section 2.2), which is a direct measurement-based definition, not a derived result that assumes what it claims to show. Pass@k|t is defined as a probability over per-checkpoint pass rates under round-robin allocation, and the Appendix C proof is a standard hypergeometric/binomial unbiasedness argument; it does not assume the conclusion it is proving. The reported gains (Pass@k|t, Maj@k|t, BoN) are empirical comparisons between sampling across saved checkpoints and sampling only from the final checkpoint under identical compute budgets; they are not forced by the definitions of the metrics. No parameter is fitted to the evaluation data and then renamed as a prediction. The greedy-sampling measurement of Ever Correct and Temporal Forgetting in Section 2.2 and the temperature-0.6 sampling for Temporal Sampling in Section 3.3 use different decoding protocols, which is a potential validity gap about whether greedy-correct checkpoints contain recoverable sampled solutions, but it is not circularity: the paper never defines P_TFS in terms of Pass@k|t or vice versa. There is also no load-bearing self-citation or imported uniqueness/ansatz; the cited works provide training settings, evaluation settings, and baselines, not the core claim. Therefore the paper is self-contained with respect to its main empirical and probabilistic claims.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No physical parameters are fitted. The empirical claim depends on two hand-chosen evaluation quantities (t=8 checkpoints, N=64 samples per checkpoint) and on the assumption that greedy-decoding correctness is a stable proxy for temperature-sampling correctness. The Pass@k|t estimator relies on independence across checkpoints, which is stated but not empirically tested.

free parameters (2)
  • t (number of checkpoints used in Temporal Sampling) = 8
    Checkpoint pool size in all main experiments; the paper saves 8 checkpoints and ablates t=1,2,4,8, but does not justify 8 as a tuned or cross-validated choice. Performance grows with t, so results depend on this choice.
  • N (candidate samples per checkpoint for Pass@k|t estimation) = 64
    The unbiased estimator's variance decreases with N, and the paper fixes N=64 for evaluation without reporting sensitivity to this choice.
assumptions (3)
  • domain assumption Samples from different checkpoints are independent, with fixed per-checkpoint correctness probabilities per problem.
    Used in Appendix C, Equation 7, to factor the expectation of the product in the unbiased estimator proof.
  • domain assumption Greedy decoding correctness at a checkpoint is a reliable indicator of that checkpoint's ability to produce a correct answer under temperature sampling.
    P_ECS and P_TFS in Section 2.2 are measured with greedy sampling, while Temporal Sampling in Section 3.3 uses temperature 0.6 samples; the link between the two is not tested.
  • ad hoc to paper The saved checkpoints are representative snapshots of the training trajectory, and the round-robin allocation starting from the latest checkpoint is a sensible budget split.
    No criterion for checkpoint selection is given; the paper uses the t most recent checkpoints without comparing alternative checkpoint ranking strategies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temporal Sampling for Forgotten Reasoning in LLMs." pith.science (2026). https://pith.science/paper/D73LVBUG

@misc{pith2026250520196,
  author       = {Pith},
  title        = {Pith review of: Temporal Sampling for Forgotten Reasoning in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D73LVBUG}},
  note         = {Machine review of arXiv:2505.20196}
}
read the original abstract

Fine-tuning large language models (LLMs) is intended to improve their reasoning capabilities, yet we uncover a counterintuitive effect: models often forget how to solve problems they previously answered correctly during training. We term this phenomenon temporal forgetting and show that it is widespread across model sizes, fine-tuning methods (both Reinforcement Learning and Supervised Fine-Tuning), and multiple reasoning benchmarks. To address this gap, we introduce Temporal Sampling, a simple decoding strategy that draws outputs from multiple checkpoints along the training trajectory. This approach recovers forgotten solutions without retraining or ensembling, and leads to substantial improvements in reasoning performance, gains from 4 to 19 points in Pass@k and consistent gains in Majority@k across several benchmarks. We further extend our method to LoRA-adapted models, demonstrating that storing only adapter weights across checkpoints achieves similar benefits with minimal storage cost. By leveraging the temporal diversity inherent in training, Temporal Sampling offers a practical, compute-efficient way to surface hidden reasoning ability and rethink how we evaluate LLMs.

Figures

Figures reproduced from arXiv: 2505.20196 by the authors.

Figure 1
Figure 1. (a) We observed that during RL training process of Deepseek-R1-1.5B model, 76.7% of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Fine-tuned models like DeepscaleR-1.5B [ [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Forgetting dynamics of Qwen2.5-7B during RL training. (a) Answer correctness tra [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Pass rate distribution across training checkpoints on AIME24. Individual problems show varying pass rates over time. Temporal Sam￾pling exploits these dynamics to improve answer diversity at inference. To better measure the performance of Tem￾poral Sampling, we introdu…
Figure 5
Figure 5. Figure 5: Pass@k for different numbers of checkpoints t on the AIME2024, AMC, and AIME2025 benchmarks when using Temporal Sampling. The case t = 1 represents the baseline of standard P ass@k sampling on the final checkpoint. Our proposed Temporal Sampling for Qwen2.5-7B with t =…
Figure 6
Figure 6. Figure 6: Maj@k (Majority voting) for different numbers of checkpoints t on the AIME2024, AMC, and AIME2025 benchmarks using Temporal Sampling. The case t = 1 represents the baseline of standard majority voting sampling on the final checkpoint. Our proposed Temporal Sampling wit…
Figure 7
Figure 7. Figure 7: Best-of-N decoding on the AIME2024, AMC, and AIME2025 benchmarks using Tem [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Maj@k comparison between Temporal Sampling (t = 3) and a Mixture of Models (MoM) approach on the AIME2024, AMC, and AIME2025 benchmarks. For MoM, the model pool in￾cluded the Qwen2.5-7B-Base final RL checkpoint, Deepseek-Math-7B-Instruct, and Llama-3.1-8B￾Instruct. Tem…
Figure 9
Figure 9. Figure 9: Performance of Temporal Sampling using 8 [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: BoN (Best-of-N) decoding on the AIME2024, AMC, and AIME2025 benchmarks using [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: BoN (Best-of-N) decoding on the AIME2024, AMC, and AIME2025 benchmarks using [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Performance of the base model (PBase ↑), the fine-tuned model (PFT ↑) and the Lost Score (PLost ↓) for Pass@1 sampling and Pass@8 sampling. Fine-tuned models like DeepscaleR￾1.5B [24] and Still-3-1.5B [8] outperform the base model overall but also forget many question…
Figure 13
Figure 13. Figure 13: Performance of Temporal Sampling using 8 checkpoints from LoRA SFT of Qwen2.5- [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. STAR-R1: Spatial TrAnsformation Reasoning by Reinforcing Multimodal LLMs

    cs.CV 2025-05 conditional novelty 6.0 of 10

    STAR-R1 uses single-stage reinforcement learning with fine-grained rewards to improve spatial transformation reasoning in multimodal LLMs, outperforming supervised fine-tuning on cross-view TVR tasks.

  2. First Return, Entropy-Eliciting Explore

    cs.AI 2025-07 conditional novelty 4.0 of 10

    FR3E improves LLM math reasoning by finding high-entropy tokens in a trajectory and doing extra rollouts from those points to shape rewards.

Reference graph

Works this paper leans on

61 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Integer partitions

    George E Andrews and Kimmo Eriksson. Integer partitions. Cambridge University Press, 2004

    work page 2004

  2. [2]

    Training data attribution via ap- proximate unrolled differentiation, 2024

    Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. Training data attribution via ap- proximate unrolled differentiation, 2024

    work page 2024

  3. [3]

    Scaling test-time compute with open models, 2024

    Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models, 2024

    work page 2024

  4. [4]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  6. [6]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  7. [7]

    Raft: Reward ranked finetuning for generative foundation model alignment, 2023

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment, 2023

    2023

  8. [8]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025

    2025

Show all 61 references
  1. [9]

    Alphazero-like tree-search can guide large language model decoding and train- ing, 2023

    Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and train- ing, 2023

    2023

  2. [10]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scienti...

    2024

  3. [11]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems T...

    2021

  4. [12]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021

    2021

  5. [13]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

    2021

  6. [14]

    Putting rl back in rlhf

    Shengyi Costa Huang and Arash Ahmadian. Putting rl back in rlhf. https://huggingface. co/blog/putting_rl_back_in_rlhf_with_rloo, June 12 2024. Hugging Face Blog

    2024

  7. [15]

    Ii-thought : A large-scale, high-quality reasoning dataset, 2025

    Intelligent Internet. Ii-thought : A large-scale, high-quality reasoning dataset, 2025

    2025

  8. [16]

    Disentangling memory and reasoning ability in large language models, 2025

    Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, and Yongfeng Zhang. Disentangling memory and reasoning ability in large language models, 2025

    2025

  9. [17]

    MindStar: Enhancing math reasoning in pre-trained llms at inference time

    Jikun Kang, Xin Zhe Li, Xi Chen, Amirreza Kazemi, Qianyi Sun, Boxing Chen, Dong Li, Xu He, Quan He, Feng Wen, et al. MindStar: Enhancing math reasoning in pre-trained llms at inference time. arXiv preprint arXiv:2405.16265, 2024

    2024 arXiv

  10. [18]

    Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, Franc ¸ois Halbach, Alex Rocha, and Joe Stubbs

    Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, Franc ¸ois Halbach, Alex Rocha, and Joe Stubbs. Lessons learned from the chameleon testbed. In Proceed...

    2020

  11. [19]

    ARGS: Alignment as reward-guided search

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. ARGS: Alignment as reward-guided search. In International Conference on Learning Representations (ICLR), 2024

    2024

  12. [20]

    Kimi k1.5: Scaling reinforcement learning with llms, 2025

    Kimi Team. Kimi k1.5: Scaling reinforcement learning with llms, 2025

    2025

  13. [21]

    Fine-tuning can distort pretrained features and underperform out-of-distribution, 2022

    Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution, 2022

    2022

  14. [22]

    Med-r1: Reinforcement learning for generalizable medical reasoning in vision-language models, 2025

    Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-r1: Reinforcement learning for generalizable medical reasoning in vision-language models, 2025. 11

    2025

  15. [23]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Lu...

    2024

  16. [24]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y . Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://pretty-radio-b75.notion.site/ DeepScaleR-Surpas...

  17. [25]

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning.arXiv preprint arXiv:2308.08747, 2023

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning.arXiv preprint arXiv:2308.08747, 2023

    2023 arXiv

  18. [26]

    Adaptive inference-time compute: Llms can predict if they can do better, even mid-generation

    Rohin Manvi, Anikait Singh, and Stefano Ermon. Adaptive inference-time compute: Llms can predict if they can do better, even mid-generation. arXiv preprint arXiv:2410.02725, 2024

    2024 arXiv

  19. [27]

    s1: Simple test-time scaling, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand `es, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025

    2025

  20. [28]

    Sky-T1: Train your own o1 preview model within $450, 2025

    NovaSky. Sky-T1: Train your own o1 preview model within $450, 2025. Accessed: 2025-01- 09

    2025

  21. [29]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024

    2024

  22. [30]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  23. [31]

    Qwq: Reflect deeply on the boundaries of the unknown, 2024

    Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, 2024

    2024

  24. [32]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024

    2024

  25. [33]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  26. [34]

    Sutherland

    Yi Ren, Shangmin Guo, Wonho Bae, and Danica J. Sutherland. How to prepare your task head for finetuning, 2023

    2023

  27. [35]

    Sutherland

    Yi Ren and Danica J. Sutherland. Learning dynamics of llm finetuning, 2025

    2025

  28. [36]

    Beyond chinchilla- optimal: Accounting for inference in language model scaling laws

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla- optimal: Accounting for inference in language model scaling laws. InInternational Conference on Machine Learning (ICML), volume 235, pages 43445–43460, 2024

    2024

  29. [37]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017

    2017

  30. [39]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 12

    2024 arXiv

  31. [40]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  32. [41]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    2024

  33. [42]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  34. [43]

    Open Thoughts

    OpenThoughts Team. Open Thoughts. https://open-thoughts.ai, January 2025

    2025

  35. [44]

    Still-3-1.5b-preview: Enhancing slow thinking abilities of small models through reinforcement learning

    RUCAIBox STILL Team. Still-3-1.5b-preview: Enhancing slow thinking abilities of small models through reinforcement learning. 2025

    2025

  36. [45]

    AlphaZero-like tree-search can guide large language model decoding and train- ing

    Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus Mcaleer, Ying Wen, Weinan Zhang, and Jun Wang. AlphaZero-like tree-search can guide large language model decoding and train- ing. In International Conference on Machine Learning (ICML) , volume 235, pages 49890– 49920, 2024

    2024

  37. [46]

    Self-consistency improves chain of thought reasoning in lan- guage models, 2023

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in lan- guage models, 2023

    2023

  38. [47]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought rea- soning in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  39. [48]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  40. [49]

    Swe-rl: Advancing llm reason- ing via reinforcement learning on open software evolution

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing llm reason- ing via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449, 2025

    2025 arXiv

  41. [50]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024

    2024 arXiv

  42. [51]

    Self-evaluation guided beam search for reasoning

    Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems,...

    2023

  43. [52]

    Deepseek-prover: Advancing theorem proving in llms through large- scale synthetic data

    Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseek-prover: Advancing theorem proving in llms through large- scale synthetic data. arXiv preprint arXiv:2405.14333, 2024

    2024 arXiv

  44. [53]

    A minimalist approach to llm reasoning: from rejection sampling to reinforce, 2025

    Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, and Hanze Dong. A minimalist approach to llm reasoning: from rejection sampling to reinforce, 2025

    2025

  45. [54]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  46. [55]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

    2024 arXiv

  47. [56]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems (NeurIPS) , volume 36, pages 11809– 11822, 2023

    2023

  48. [57]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yux- uan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi ...

    2025

  49. [58]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    2025 arXiv

  50. [59]

    7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient

    Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https://hkust-nlp.notion.site/simplerl-reason, 2025. Notion Blog

    2025

  51. [60]

    The lessons of developing process reward models in mathematical reasoning

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025

    2025 arXiv

  52. [61]

    aha moments

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024

  53. [62]

    N −Ci,j kj N kj # = (1− ri,j)kj Since Ci,j follows a binomial distribution B(N, ri,j), we have: E

    identifies influential training examples to explain model predictions. Orthogonal to these works, we empirically investigate the dynamic fluctuations in answer correctness across diverse reasoning tasks, and harness the learning dynamics as a source of answer diversity to wide...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.