Pith. sign in

REVIEW 4 major objections 6 minor 44 references

ThinkTuning: Instilling Cognitive Reflections without Distillation

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ThinkTuning claims that adding a teacher model's feedback to a student's RL rollouts instills reflective reasoning that pure RL cannot, improving benchmarks by a few points.

desk verdict ThinkTuning is a plausible GRPO-plus-teacher-feedback recipe with modest, consistent gains, but the 'without distillation' claim is untested and the core AAS objective is not visible in the text. read the letter →

arxiv 2508.07616 v2 pith:2X4G4LAL submitted 2025-08-11 cs.AI cs.CLcs.LG

Aswin RRV , Jacob Dineen , Divij Handa , Md Nayem Uddin , Mihir Parmar , Chitta Baral , Ben Zhou This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords cognitivereflectionreinforcementlearningGRPOteacherfeedbackreasoningself-correctionexplorationtest-timescaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ThinkTuning is a training method for language models that aims to make them reason reflectively by having a teacher model comment on their answers during reinforcement learning. In each round, a fraction of the student's own sampled answers is sent to a teacher, whose appended feedback highlights a misstep and shows a corrected line of thought; the teacher can express one of four reflective behaviors—self-conflict, self-agreement, self-critique, or self-consultancy. The paper's central claim is that this implicit supervision instills reflective reasoning behaviors that plain reinforcement learning cannot, and that it improves reasoning accuracy on math, science, and commonsense benchmarks. If true, the method gives smaller models a way to learn thinking skills from a same-size teacher rather than needing a much larger model, a human-written chain-of-thought corpus, or pure trial and error.

What carries the argument

The central mechanism is the augmented rollout: for a fraction γ of the n rollouts sampled per question, the student's own attempt is passed to a teacher that returns an <opinion>, <reason>, and <phrase> showing a reflective cognitive behavior; this feedback is appended to the rollout, and the combined sequence enters the GRPO advantage computation. To keep the off-policy teacher tokens from destabilizing training, an Advantage-Aware Shaping weight adjusts the policy-gradient update for those tokens according to both the rollout's advantage and the student's current probability of producing the token. The four-behavior taxonomy (self-conflict, self-agreement, self-critique, self-consultancy)

What would settle it

Train a copy of the student with plain supervised fine-tuning on the same teacher-augmented rollouts used in ThinkTuning, matching data volume; if SFT reaches or exceeds ThinkTuning's benchmark gains, the improvement is better explained by imitation of the teacher's feedback than by the RL-style shaping, and the 'implicit supervision' mechanism is falsified.

Watch

Extended reading notes

Core claim

Worded as the authors do: ThinkTuning is a two-stage procedure. First, a small set of few-shot exemplars fixes the format of the teacher's feedback, which always contains an <opinion> (correct or incorrect), a <reason> that models the student's internal dialogue, and a <phrase> that demonstrates the reflective behavior by solving the problem. Then, during GRPO training on a training set of grade-school math problems, the algorithm samples n rollouts per question, selects a random subset γ to send to the same-size teacher, appends the teacher's feedback to those rollouts, and uses all n rollouts to estimate group advantages. An Advantage-Aware Shaping weight scales the loss on teacher-augment

Load-bearing premise

The load-bearing premise is that the teacher's feedback acts as a hint that redirects the student's own reasoning rather than as a template to be copied; if the student is simply learning to reproduce the teacher's worked solutions, the claim of instilling cognitive reflection without distillation fails.

Editorial extensions

If this is right

  • If the paper is right, weak reasoning models can acquire reflective behaviors through a same-size teacher's feedback during RL, reducing the need for a stronger, larger teacher or for hand-authored reasoning traces.
  • The guiding-exploration experiment implies that any behavior a teacher can verbalize—a stylistic tic, a verification habit, a formatting rule—can be injected into the student even if the student never samples it in its own rollouts.
  • Because rewards only check final answers, ThinkTuning shifts the burden of teaching intermediate reasoning from reward design to the teacher's feedback, making it easier to apply to tasks where process supervision is expensive.
  • The token-length analysis indicates ThinkTuning models spend more inference compute per question than GRPO, consistent with the idea that the learned behavior is genuinely longer reflective chains, which could matter for test-time scaling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own limitations section explicitly says the method depends on the teacher's ability to produce helpful guidance, uses only final-answer rewards, and was tested only on smaller models; these constraints bound the strength of the central claim.
  • The 'without distillation' framing is fragile because the teacher feedback in the appendix often contains fully worked solutions; a head-to-head against supervised fine-tuning on the same feedback would settle whether the student learns a transferable skill or copies an answer template.
  • The ethics statement's mention of proprietary LLMs used in the study suggests the 'same-size teacher' characterization should be read cautiously—at least some guidance may have come from models much larger than the 3B student.
  • The reported underperformance on GSM8K and StrategyQA suggests that reflective behaviors can turn into overthinking on simpler questions; a curriculum that lowers the guidance fraction γ on easy items is a natural testable follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ThinkTuning, a GRPO-based interactive training method in which a student model's rollouts are augmented with feedback from a teacher model. The teacher evaluates a student attempt, provides corrective feedback and a 'phrase' that demonstrates one of four cognitive behaviors (self-conflict, self-agreement, self-critique, self-consultancy). A subset of rollouts is randomly selected for teacher augmentation; the augmented and unaugmented rollouts are used together in the GRPO advantage computation, with an Advantage-Aware Shaping (AAS) weight intended to stabilize updates from off-policy teacher tokens. The method is evaluated by training Llama-3.2-3B-Instruct on GSM8K questions and testing on eight reasoning benchmarks, reporting gains over zero-shot baselines and over vanilla GRPO on several datasets. A separate experiment claims that ThinkTuning can instill a previously unknown behavior (ending responses with a Thalapathy Vijay quote) that pure RL cannot evoke. The conclusion is that ThinkTuning instills cognitive reflection 'without distillation.'

Significance. If the benchmark gains are robust, ThinkTuning offers a practical, low-overhead recipe for injecting reasoning behaviors into smaller models that do not already exhibit them. The method is simple, the code is released, and the token-length and error analyses are useful additions. The Thalapathy Vijay experiment is an interesting demonstration of exploration steering. However, the central 'without distillation' claim is not currently established. The teacher's feedback in Appendix A.1 contains fully worked solutions, so the augmented rollouts could simply act as supervised targets; the paper never compares against direct SFT or behavior cloning on the same trajectories. The reported gains are also small on several benchmarks and arrive without error bars or significance tests. These gaps are load-bearing for the paper's main claims, though they appear addressable with additional experiments.

major comments (4)
  1. [Appendix A.1, Section 3/4] The 'without distillation' claim is untested. The teacher's <phrase> in the Self-Conflict exemplar contains a complete solution ('20 chickens times 3 cups each gives 60 cups total...'), and these teacher-written tokens are appended to the student rollout and included in the GRPO/AAS update. A student could therefore improve simply by imitating the teacher's solution. The manuscript never compares against direct SFT or behavior cloning on the same teacher-feedback trajectories, so the observed gains may be supervised imitation rather than RL-instilled reflection. Adding this baseline is essential to support the title and the central claim.
  2. [Table 1, Abstract] The quantitative claims (e.g., 2.08%, 2.23%, and 3.99% over vanilla GRPO on MATH-500, AIME, and GPQA-Diamond) are reported without error bars, significance tests, or the number of independent runs. Given the small margins on several benchmarks, these numbers are not sufficient to establish that the improvements are systematic. Report mean and standard deviation over at least three seeds, and a paired significance test where appropriate.
  3. [Abstract vs. Ethics Statement] The abstract and introduction describe a teacher model 'of the same size,' but the Ethics Statement says GPT-4 and Gemini were used. If the teacher is a much larger proprietary model, the method is closer to distillation and the 'same-size' claim is false. The teacher model(s) and their sizes must be specified for each experiment. If the same-size claim is central, the teacher should be the same size as the student, or the claim should be revised.
  4. [Section 5, Figure 4] The Thalapathy Vijay experiment demonstrates exploration steering, but it is substantially forced by design: the reward explicitly pays for the target phrase and the teacher is prompted to emit it. This does not distinguish RL-instilled behavior from copying teacher-provided text. A control that trains on the same teacher-augmented rollouts with SFT, or with an advantage weight but without the RL update, would clarify whether the RL loop contributes anything beyond imitation.
minor comments (6)
  1. [Section 6] Typo: 'paves the the way' should be 'paves the way.'
  2. [Table 2] The error frequencies sum to well over 100%, which is presumably because multiple error types can co-occur in one instance. Please state this explicitly.
  3. [Throughout] The benchmark name is inconsistently written as 'GSM8k' and 'GSM8K.' Please standardize.
  4. [Section 5, reward definition] The reward equation for the Thalapathy Vijay experiment has garbled formatting with empty boxes. Please fix the equation so the reward components are legible.
  5. [Appendix A.1, Self-Consultancy example] The student response says '80% of the file' even though the question says '40% of the way.' This is presumably an intentional erroneous student attempt, but it may confuse readers; consider marking it as such.
  6. [Related Work] The paper cites Yan et al. (2025) on off-policy guidance but does not discuss it in the related-work narrative. Add a sentence positioning ThinkTuning relative to that work.

Circularity Check

1 steps flagged · score 2.0 of 10

Main benchmark derivation is self-contained and non-circular; the only forced-by-design element is the Thalapathy-Vijay 'instill unknown behavior' experiment, where the target phrase is part of both the reward and the teacher prompt.

  1. self definitional [Section 5, 'Can THINKTUNING instill unknown behaviors?' (Figure 4 experiment)]
    "We define our reward function r(o) as follows: r(o) = [1 if the final answer is correct, 0 otherwise] + [1 if "Thalapathy Vijay" appears in the response, 0 otherwise]. ... For THINKTUNING, we prompt the teacher model to provide guidance that quotes Thalapathy Vijay in a movie-like dialogue about approaching the problem. ... a model trained with THINKTUNING was able to generate responses ending with a quote from Thalapathy Vijay ... This experiment shows that THINKTUNING can successfully instill a behavior that was completely unknown to the student model beforehand."

    The target string 'Thalapathy Vijay' is an explicit input to both components of the training signal: the reward function pays for it and the teacher is prompted to emit it. The augmented rollout therefore already contains the exact behavior that the experiment later reports as 'instilled.' The observed output is a restatement of the experimental construction (target phrase in reward + target phrase in teacher prompt => target phrase in output), not an independently discovered behavior. The word 'unknown' applies only to the student before training, but the behavior is fully specified in the reward and teacher prompt, so the demonstration reduces to an input-repetition check rather than a test of whether ThinkTuning can discover novel behaviors. The main benchmark evaluations are unaffected

full rationale

The central derivation chain — GRPO rollouts augmented by teacher feedback, AAS-weighted updates, and the reported benchmark gains — is self-contained and not circular. The MATH-500/AIME/GPQA-Diamond improvements over vanilla GRPO are external evaluations on held-out benchmarks, not reconstructions of training inputs; no fitted constant is renamed as a prediction. The self-citations in the reference list (PlanTuning, PlanGen, ToW, TriplePO) are background and are not load-bearing for the main claim. The genuine interpretive weakness is that the paper never compares against SFT/behavior cloning on the same teacher-augmented trajectories, so the 'without distillation' framing is untested; however, a missing control is a validity gap, not a circular reduction. The only step that is forced by the paper's own construction is the Thalapathy-Vijay demonstration: the target string is written into both the reward and the teacher prompt, so the model's adoption of the quote is an input repetition rather than an independently discovered behavior. This is a minor, localized circularity; it does not affect the benchmark results.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce new physical or conceptual entities; it introduces a training procedure and a taxonomy of behaviors. The load-bearing assumptions are about teacher signal validity, off-policy correction, and the sufficiency of the four cognitive categories. The unstated AAS objective and the unstated teacher model identity are the main gaps in the ledger.

free parameters (4)
  • guidance fraction gamma = not reported
    Fraction of rollouts augmented with teacher feedback; controls how much off-policy signal enters GRPO and affects the central claim.
  • AAS weighting function parameters = not reported
    Advantage-aware shaping weights combine advantage and student token confidence; the exact form and any temperature or scale constants are not shown in the provided text.
  • number of rollouts n per question = not reported
    GRPO rollout count affects variance and exploration; not stated in the visible experimental section.
  • reward weights in behavior-steering experiment = binary reward for correctness plus binary reward for target phrase
    The toy experiment rewards the target quote explicitly, which makes the 'instill unknown behavior' result partly engineered.
assumptions (4)
  • domain assumption GRPO advantage estimation remains valid when off-policy teacher tokens are included and corrected by AAS.
    The paper states the teacher guidance violates importance-sampling assumptions and asserts AAS fixes this, but the derivation is not visible in the provided text.
  • domain assumption Teacher feedback teaches a generalizable cognitive skill rather than leaking the final answer.
    Appendix exemplars show complete worked solutions inside the feedback, so the student sees the answer through the teacher. The paper assumes this instills reflection rather than answer copying.
  • domain assumption The four chosen cognitive behaviors (Self-Conflict, Self-Agreement, Self-Critique, Self-Consultancy) are sufficient to represent reflective reasoning.
    The authors select four behaviors because they are well defined, but no evidence is given that these categories capture the mechanisms behind the observed gains.
  • domain assumption The Gandhi et al. 2025 result that RL only draws out existing behaviors is accepted as motivation.
    The paper's framing depends on this external claim but does not independently verify it on the base model used here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ThinkTuning: Instilling Cognitive Reflections without Distillation." pith.science (2026). https://pith.science/paper/2X4G4LAL

@misc{pith2026250807616,
  author       = {Pith},
  title        = {Pith review of: ThinkTuning: Instilling Cognitive Reflections without Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2X4G4LAL}},
  note         = {Machine review of arXiv:2508.07616}
}
read the original abstract

Recent advances in test-time scaling have led to the emergence of thinking LLMs that exhibit self-reflective behaviors and multi-step reasoning. While RL drives this self-improvement paradigm, a recent study (Gandhi et al., 2025) shows that RL alone does not truly instill these new reasoning abilities - it merely draws out behaviors already present in the base models. This raises a question: How can we train the models that don't exhibit such thinking behavior to develop it in the first place? To this end, we propose ThinkTuning, a GRPO-based interactive training approach where we augment the rollouts of a student model with the guidance from a teacher model. A simple idea from classroom practice inspires our method: a teacher poses a problem, lets the student try an answer, then gives corrective feedback -- enough to point the mind in the right direction and then show the solution. Each piece of feedback reshapes the student's thoughts, leading them to arrive at the correct solution. Similarly, we find that this type of implicit supervision through feedback from a teacher model of the same size improves the reasoning capabilities of the student model. In particular, on average, our method shows a 3.85% improvement over zero-shot baselines across benchmarks, and on MATH-500, AIME and GPQA-Diamond it shows 2.08%, 2.23% and 3.99% improvements over the vanilla-GRPO baseline. Source code is available at https://github.com/3rdAT/ThinkTuning.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 9 canonical work pages

  1. [1]

    B Camp and M Bash. 1978. Think aloud: Group manual (rev. ed.). Denver, CO: University of Colorado Medical School

    work page 1978

  2. [2]

    Anastasia Chouvalova, Anisha S Navlekar, Devin J Mills, Mikayla Adams, Sami Daye, Fatima De Anda, and Lisa B Limeri. 2024. Undergraduates’ reactions to errors mediates the association between growth mindset and study strategies. International Journal of STEM Education, 11(1):26

    work page 2024

  3. [3]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

  4. [4]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2110.14168, 9

    arXiv 2021

  5. [5]

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. 2025. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307

    arXiv 2025

  6. [6]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346--361

    2021

  7. [7]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    arXiv 2025

Show all 44 references
  1. [9]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  2. [10]

    Hubert JM Hermans. 2023. Dialogical self theory. In The Palgrave encyclopedia of the possible, pages 389--394. Springer

    2023

  3. [11]

    Hubert JM Hermans and Thorsten Gieser. 2011. Handbook of dialogical self theory. Cambridge University Press

    2011

  4. [12]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798

    2023 arXiv

  5. [13]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  6. [14]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  7. [15]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In NeurIPS

    2022

  8. [16]

    Training language models to self-correct via reinforcement learning, 2024

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, and 1 others. Training language models to self-correct via reinforcement learning, 2024. URL https://arxiv. org/abs/2409.12917

    2024 arXiv

  9. [17]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  10. [18]

    Majumder, Katherine M

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Bodhisattwa P. Majumder, Katherine M. Hermann, Sean Welleck, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. arXiv prep...

    2023 arXiv

  11. [19]

    Yeray Mera, Gabriel Rodr \' guez, and Eugenia Marin-Garcia. 2022. Unraveling the benefits of experiencing errors during learning: Definition, modulating factors, and explanatory theories. Psychonomic bulletin & review, 29(3):753--765

    2022

  12. [20]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393

    2025 arXiv

  13. [21]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katherine Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and R...

    2022 arXiv

  14. [22]

    Amy A Overman, Joseph DW Stephens, and Mary F Bernhardt. 2021. Enhanced memory for context associated with corrective feedback: evidence for episodic processes in errorful learning. Memory, 29(8):1017--1042

    2021

  15. [23]

    Mihir Parmar, Palash Goyal, Xin Liu, Yiwen Song, Mingyang Ling, Chitta Baral, Hamid Palangi, and Tomas Pfister. 2025 a . Plan-tuning: Post-training language models to learn step-by-step planning for complex problem solving. arXiv preprint arXiv:2507.07495

    2025 arXiv

  16. [24]

    Mihir Parmar, Xin Liu, Palash Goyal, Yanfei Chen, Long Le, Swaroop Mishra, Hossein Mobahi, Jindong Gu, Zifeng Wang, Hootan Nakhost, and 1 others. 2025 b . Plangen: A multi-agent framework for generating planning and reasoning trajectories for complex problem solving. arXiv pre...

    2025 arXiv

  17. [25]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290

    2023 arXiv

  18. [26]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  19. [27]

    Amir Saeidi, Shivanshu Verma, Aswin RRV, and Chitta Baral. 2024. Triple preference optimization: Achieving better alignment with less data in a single step optimization. arXiv preprint arXiv:2405.16681

    2024 arXiv

  20. [28]

    Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojciech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, and 1 others. 2018. Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.03835

    2018 arXiv

  21. [29]

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438

    2015 arXiv

  22. [30]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  23. [31]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Yankai Li, Yu Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  24. [32]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256

    2024 arXiv

  25. [33]

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. https://openreview.net/forum?id=4FWAwZtd2n Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning . In The Thirteenth International Conference on Learning Representations

    2025

  26. [34]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937

    2018 arXiv

  27. [35]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  28. [36]

    Hemish Veeraboina. 2023. https://www.kaggle.com/datasets/hemishveeraboina/aime-problem-set-1983-2024 Aime problem set 1983-2024

    2023

  29. [37]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain-of-thought reasoning in language models. In ICLR

    2023

  30. [38]

    Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, volume 35, pages 24824--24837

    2022

  31. [39]

    Zhikun Xu, Ming Shen, Jacob Dineen, Zhaonan Li, Xiao Ye, Shijie Lu, Aswin RRV, Chitta Baral, and Ben Zhou. 2024. Tow: Thoughts of words improve reasoning in large language models. arXiv preprint arXiv:2410.16235

    2024 arXiv

  32. [40]

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945

    2025 arXiv

  33. [41]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  34. [42]

    Yong Yao, Zhijie Xu, Eliana Reif, and Hang Zhao. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601

    2023 arXiv

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.