Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and Faster

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training a small language model on teacher rationales one chunk at a time, then teaching it which chunks to skip, makes the student reason more accurately and answer faster than standard chain-of-thought distillation.

desk verdict A promising chunk-wise distillation recipe whose main comparison is currently undermined by a duplicated baseline row in Table 2; worth refereeing but not in present form. read the letter →

arxiv 2505.18642 v1 pith:SYGQ5JPG submitted 2025-05-24 cs.CL

Xiao Chen , Sihang Zhou , Ke Liang , Xiaoyu Sun , Xinwang Liu This is my paper · ORCID
classification cs.CL
keywords chain-of-thoughtdistillationsmalllanguagemodelschunk-wisetrainingskip-thinkingtoken-levelbatchsizegradientover-smoothingreasoningaccelerationrationaleinternalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chain-of-thought distillation is supposed to pass a large teacher's reasoning ability to a small student, but students trained to imitate the full written rationale tend to parrot surface patterns and answer slowly. This paper argues the cause is a gradient-dilution effect: with the whole rationale in one training sequence, the student's gradients are averaged over so many token positions that the few tokens that actually determine the reasoning outcome are swamped by transitional and summary filler, driving the model to a sharp minimum. The proposed fix, chunk-wise training (CWT), feeds the student one loss-guided, semantically coherent chunk per iteration, concentrating the gradient signal on the reasoning steps; skip-thinking training (STT) then removes chunks the student can drop without changing the answer, teaching it to externalize only what matters. Across seven benchmarks and students from 60M to 3B parameters, the combined method reports higher accuracy than full-rationale distillation and faster inference. If the claim holds, better small-model reasoning does not require a bigger model or a better teacher, only a smarter way to present the teacher's reasoning.

What carries the argument

The mechanism that carries the argument is the token-level batch size identity of Section 3, where the training gradient is shown as an average over all $N = B \times (K - s)$ target tokens and then split into a core-reasoning set $S_1$ and a filler set $S_2$; the paper's whole case is that shrinking $N$ by chunking raises the relative gradient weight of $S_1$. Three concrete devices implement that: the chunk data generator, which turns one rationale into $M+1$ training sequences prefixed by stage markers $[m]$ and $[answer]$; search-based chunking (SBC), a greedy algorithm that moves chunk boundaries using the student's own cross-entropy loss as a coherence signal; and the skip data generator, which uses answer correctness after chunk deletion as the criterion for which chunks can be internalized, with a $[skip]$ prompt at test time letting the model jump to the essential reasoning.

What would settle it

Log per-token gradient norms of core reasoning tokens versus filler tokens during full-rationale and chunk-wise training. The mechanism predicts that CWT raises the share of total gradient norm carried by core tokens and that this share tracks test accuracy; if core-token gradients are already well-represented under full-rationale training, or if accuracy is unchanged when chunk boundaries are randomly shuffled, the over-smoothing story is not the operative cause.

Watch

Extended reading notes

Core claim

The paper's central claim is that the apparent reasoning ceiling of distilled small language models is largely an artifact of how the training signal is shaped, not a fixed capacity limit. Its formal core is a gradient identity: the per-iteration update averages over $N = B \times (K - s)$ target tokens, and when those tokens are split into core reasoning tokens $S_1$ and the remaining filler tokens $S_2$, the contribution of $S_1$ is smoothed away because $|S_2|$ dwarfs $|S_1|$, leaving the model to converge to a sharp minimum that reproduces the teacher's phrasing without its logic. Chunk-wise training shrinks the token-level batch by exposing the student to one chunk per iteration, so the core-token share of the gradient rises; search-based chunking (SBC) uses the student's own loss as a heuristic to place boundaries where the reasoning stays coherent. Skip-thinking training goes one step further: it deletes each chunk in turn, re-runs the student, and marks as skippable any chunk whose absence leaves the answer correct, so the trained model generates only the load-bearing chunks at inference time. The paper reports that this combination improves accuracy relative to full-rationale distillation on every student model and task set it tests, including GPT-2, T5, and Llama-3.2 students, while also shortening generation.

Load-bearing premise

The whole argument rests on the premise that a long training sequence dilutes token gradients and pushes the model into sharp minima in the same way that a large sample-level batch does, and the paper transfers that result to token-level training without directly testing the transfer.

Editorial extensions

If this is right

  • Students trained with CWT and STT should surpass full-rationale CoT distillation on reasoning accuracy across arithmetic, symbolic, commonsense, and logical tasks; the paper reports gains on every student size it tests, from 60M-parameter T5-small to Llama-3.2-3B.
  • Inference becomes faster because the student learns to omit transitional and summary chunks; reported speedups over full-thinking range from roughly 1.1x to 1.9x depending on the task, with the largest gains on commonsense and object-tracking tasks.
  • Because each training step processes only one chunk, the method uses less GPU memory than full-rationale distillation, making deeper distillation feasible on smaller hardware.
  • If the gradient-dilution account is right, token-level batch size, not just sample-level batch size, is a controllable hyperparameter in distillation, and students of a fixed size have more reasoning headroom than full-rationale training suggests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If gradient dilution is the operative cause, the chunking principle should transfer to other sequence-distillation settings with heterogeneous token importance, such as long code generation, structured reports, or multi-step tool use, none of which the paper's seven benchmarks cover.
  • The skip criterion could be widened from exact answer match to semantic equivalence or verifier scores, which would extend skip-thinking to open-ended generation tasks instead of only tasks with a single correct answer.
  • The paper's fixed chunk counts (M = 4 for arithmetic, M = 2 for tracking and commonsense) imply a testable scaling law: plotting accuracy against token-level batch size while holding everything else fixed would map the claimed mechanism more directly than the reported ablations.
  • The sharp-minima premise predicts a measurable signature, flatter loss landscapes for CWT-trained students, which could be checked directly with curvature estimates of the converged minimum.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes two training strategies for chain-of-thought (CoT) distillation into small language models: chunk-wise training (CWT), which divides a teacher rationale into a fixed number of chunks and trains the student on one chunk per iteration, and skip-thinking training (STT), which removes chunks whose absence does not change the answer and trains the student to externalize only essential reasoning chunks. The authors claim CWT mitigates gradient over-smoothing caused by large token-level batch sizes, thereby improving reasoning accuracy, and that STT accelerates inference while preserving accuracy. The paper reports experiments on GPT-2, T5, and Llama-3.2 models across seven reasoning benchmarks plus GSM8K, including ablations, speed-accuracy comparisons, case studies, and an analysis of core reasoning token confidence.

Significance. If the results hold, this is a practical and reasonably simple training recipe that improves both accuracy and speed of small-model reasoning, with evidence across multiple model families and datasets. The paper gives explicit credit for several design choices: a loss-guided search-based chunking algorithm, an answer-correctness criterion for deciding which chunks to skip, ablations distinguishing STT from skipping all chunks (SkipALL), measured speedup ratios in Table 4, and a discussion of limitations including the greedy search. The central claims, however, depend on the integrity of the CoT-distillation baseline, on matched-data/compute controls, and on direct evidence for the proposed gradient-over-smoothing mechanism; at present those dependencies are not fully supported.

major comments (4)
  1. [Table 2 / Table 1] The rows 'Standard finetune' and 'CoT-Finetuing' are numerically identical for every dataset and both student models (e.g., GPT2-base: 8.55, 10.08, 14.44, 10.66, 56.88, 21.33, 58.22), and Table 1's 'Base', described in Section 5.2 as the full-thinking CoT distillation baseline of Ho et al. (2023), repeats the same numbers. Exact equality with answer-only fine-tuning is implausible if teacher rationales actually contributed to training. Because the headline gains (e.g., TSO 56.88 to 100.00, MA 14.44 to 22.77, and the Figure 3 scatter) are measured against this row, the main empirical comparison is not established. The authors must correct the baseline or rerun the experiments, and should report multiple seeds with error bars or significance tests.
  2. [Appendix F] CWT multiplies the number of training sequences by about M+1, STT multiplies it further, and Appendix F states that the proposed method requires approximately twice the training time of other methods. No control is provided that trains a baseline on the same number of sequence chunks or on the same amount of compute/data. Without such a control, the observed accuracy gains could be attributed to seeing more or shorter training sequences rather than to reducing token-level batch size. The authors should add a matched-data or matched-compute baseline to isolate the effect of chunk-wise training.
  3. [Section 3, Equations (2)-(3)] The theoretical motivation borrows the large-batch sharp-minima result from Keskar et al. (2017) and Jastrzebski et al. (2018) and applies it to 'token-level batch size' N = B x (K-s). However, tokens within one rationale are not independent samples; averaging gradients over correlated token positions in a single long sequence is not equivalent to averaging over many independent training examples. Figure 5 shows accuracy versus batch size but does not measure flatness or sharpness of the loss landscape, so the claim that baseline failures are caused by sharp minima remains unsupported. A direct test, such as measuring loss-landscape curvature or comparing against an alternative that reduces sequence length without chunk-level training, is needed to validate the mechanism.
  4. [Tables 1-2, Figure 3] All accuracy numbers are reported as single point estimates with no variance, confidence intervals, or significance tests, despite several comparisons being small (e.g., SQA 58.22 to 60.55 and 56.04 to 59.97 in Table 1). Since the paper's central claims are empirical, the authors should report multiple runs or provide a statistical basis for the observed differences, especially for the smaller gains.
minor comments (6)
  1. [Throughout] There are frequent typos and inconsistent labels, including 'CoT-Finetuing', 'specail', 'SBA' (Appendix G.1) versus 'SBC', 'Based w. STT' in Tables 9 and 10, 'Datsets' and 'datails' in Appendix A, and 'exhibites' in Section 5.2.
  2. [Figures 3 and 4] Figure 3 lacks error bars and a description of how the average inference time and accuracy were computed, and Figure 4 has missing axis labels and unclear numeric values. Please make the figures self-contained.
  3. [Section 4.2] The relationship between the CWT-trained model used by the skip data generator and the final STT-trained model is unclear: the text says STT is initialized from pre-trained parameters, not from the CWT model, but also says CWT is incorporated into STT training. Please clarify whether the final model receives both chunk-wise and skip-training data in the same optimization run, and how the skip labels are generated without circularity.
  4. [Section 5.4 and Table 3] The confidence-score analysis uses the trained SLM's own generated rationales to identify core reasoning tokens, but the extraction procedure is described only informally. Please specify how 'core reasoning tokens' are automatically identified for each dataset and whether the reported confidence values are averaged over correct and incorrect generations.
  5. [Appendix F, Table 11] The training-cost comparison reports only one value per method without hardware details, and the text says the proposed method requires 'approximately twice' the training time while Table 11 shows 23 hours versus 11 hours (a factor of 2.09). Please provide standard deviations and specify the exact GPU model and early-stopping criteria.
  6. [Appendix G.1] The case studies are illustrative but are cherry-picked; please state how many cases were manually inspected and whether the pattern in Figure 6 is representative of the overall error distribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CWT/STT is an empirical training recipe with held-out task evaluation; the theoretical motivation is borrowed from external large-batch results, not derived from fitted constants.

full rationale

The paper's central claims are empirical: CWT and STT change the training data and are evaluated on held-out accuracy and speed (Tables 1, 2, 9, 10; Figure 3). Equations (2)-(3) only decompose the averaged gradient into S1/S2 token sets; the statement that |S2| >> |S1| over-smooths core-token gradients is an assumption borrowed from Keskar et al. (2017) and Jastrzebski et al. (2018), and it is tested by the batch-size experiment in Figure 5 rather than being the source of the reported gains. Algorithm 1 uses the SLM's own loss as a heuristic to regroup chunks, but the success of that grouping is validated by downstream accuracy on separate tasks. The skip data generator labels chunks as non-essential only when their removal leaves the answer correct, which is a data-construction step, not a prediction; the STT model is re-initialized from pretrained weights and evaluated on held-out benchmarks, so the speed-accuracy outcome is not forced by construction. The only self-citation (Chen et al. 2024a in Section 2.2) describes a related post-thinking mechanism and is not load-bearing. The skeptical observation that Table 2's 'Standard finetune' and 'CoT-Finetuing' rows are numerically identical, and Appendix F's report that CWT/STT roughly double training time, are empirical-validity and confound concerns, not instances of a derivation reducing to its inputs by construction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim depends on a handful of hand-chosen hyperparameters (chunk count M and merge threshold eta) and on three domain assumptions about gradient dynamics, loss-based chunk coherence, and skip-safety. No invented physical entities are introduced.

free parameters (2)
  • M (number of chunks) = M=4 for arithmetic tasks and Last Letter Concatenation; M=2 for Track Shuffled Objects and StrategyQA
    Chosen per dataset based on accuracy trends in Figure 4; no principled rule for selecting M.
  • eta (merge threshold in search-based chunking) = 0.1
    Set empirically to promote stable model training (Appendix A.4).
assumptions (3)
  • domain assumption Large token-level batch size causes gradient over-smoothing and convergence to sharp minima, by analogy with sample-level large-batch SGD.
    Invoked in Section 3 to motivate CWT; based on Keskar et al. 2017 and Jastrzebski et al. 2018, but not demonstrated for token-level sequence training in this paper.
  • domain assumption Lower model loss on a chunk indicates better comprehension of that chunk's content.
    Used in Algorithm 1 (search-based chunking) and justified by citing Wan et al. 2024.
  • domain assumption A chunk whose removal leaves the answer correct is non-essential and can be safely internalized.
    Used in the skip data generator (Section 4.2) to decide which chunks to skip; this is the core criterion for STT.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and Faster." pith.science (2026). https://pith.science/paper/SYGQ5JPG

@misc{pith2026250518642,
  author       = {Pith},
  title        = {Pith review of: Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and Faster},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SYGQ5JPG}},
  note         = {Machine review of arXiv:2505.18642}
}
read the original abstract

Chain-of-thought (CoT) distillation allows a large language model (LLM) to guide a small language model (SLM) in reasoning tasks. Existing methods train the SLM to learn the long rationale in one iteration, resulting in two issues: 1) Long rationales lead to a large token-level batch size during training, making gradients of core reasoning tokens (i.e., the token will directly affect the correctness of subsequent reasoning) over-smoothed as they contribute a tiny fraction of the rationale. As a result, the SLM converges to sharp minima where it fails to grasp the reasoning logic. 2) The response is slow, as the SLM must generate a long rationale before reaching the answer. Therefore, we propose chunk-wise training (CWT), which uses a heuristic search to divide the rationale into internal semantically coherent chunks and focuses SLM on learning from only one chunk per iteration. In this way, CWT naturally isolates non-reasoning chunks that do not involve the core reasoning token (e.g., summary and transitional chunks) from the SLM learning for reasoning chunks, making the fraction of the core reasoning token increase in the corresponding iteration. Based on CWT, skip-thinking training (STT) is proposed. STT makes the SLM automatically skip non-reasoning medium chunks to reach the answer, improving reasoning speed while maintaining accuracy. We validate our approach on a variety of SLMs and multiple reasoning tasks.

Figures

Figures reproduced from arXiv: 2505.18642 by the authors.

Figure 1
Figure 1. Illustration of CoT Distillation. The batch size is set to 1 as an illustrative example. The core reasoning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The illustration of the proposed methods. The flames indicate that the model is undergoing training, and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A comprehensive comparison of the average [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: SLM performance trend when the number of [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: GPT2-base’s performance trend when the batch size changes. Batch size is proportional to token￾level batch size. Chunk means using CWT with SBC. the confidence score of the core reasoning tokens and that of the common tokens is smaller after us￾ing CWT, which means tha…
Figure 6
Figure 6. Figure 6: The case for core reasoning tokens. Additionally, in the section 6, we discuss several approaches to mitigate SBC’s local optima issue, such as simulated annealing (SA). To intuitively demonstrate its effectiveness in mitigating SBC’s local optima, we integrate simulat…
Figure 7
Figure 7. Figure 7: The case for SBC. <·> represents a chunk. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: The case for Skip-thinking. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes

    cs.AI 2025-06 reject novelty 5.0 of 10

    AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.

Reference graph

Works this paper leans on

38 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Xiaoshu Chen, Sihang Zhou, Ke Liang, and Xinwang Liu. 2024 a . https://arxiv.org/abs/2404.09170 Distilling reasoning ability from large language models with adaptive thinking . Preprint, arXiv:2404.09170

    arXiv 2024

  2. [2]

    Xin Chen, Hanxian Huang, Yanjun Gao, Yi Wang, Jishen Zhao, and Ke Ding. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.409 Learning to maximize mutual information for chain-of-thought distillation . In Findings of the Association for Computational Linguistics: ACL 2024, pages 6857--6868, Bangkok, Thailand. Association for Computational Linguistics

    work page doi:10.18653/v1/2024.findings-acl.409 2024

  3. [3]

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2023. A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309.15402

    arXiv 2023

  4. [4]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

  5. [5]

    Yuntian Deng, Yejin Choi, and Stuart Shieber. 2024. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.14838

    arXiv 2024

  6. [6]

    Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. 2023. https://arxiv.org/abs/2311.01460 Implicit chain of thought reasoning via knowledge distillation . Preprint, arXiv:2311.01460

    arXiv 2023

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, and Archie Sravankumar et.al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    arXiv 2024

  8. [8]

    Fengli Gao and Huicai Zhong. 2020. Study on the large batch size training of neural networks based on the second order gradient. arXiv preprint arXiv:2012.08795

    work page Pith review arXiv 2020

Show all 38 references
  1. [9]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies . Transactions of the Association for Computational Linguistics (TACL)

    2021

  2. [10]

    Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. 2024. https://arxiv.org/abs/2310.02226 Think before you speak: Training language models with pause tokens . Preprint, arXiv:2310.02226

    2024 arXiv

  3. [11]

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2024. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769

    2024 arXiv

  4. [12]

    Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. https://doi.org/10.18653/v1/2023.acl-long.830 Large language models are reasoning teachers . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852--14882,...

    2023 doi

  5. [13]

    Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. https://doi.org/10.3115/v1/D14-1058 Learning to solve arithmetic word problems with verb categorization . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processi...

    2014 doi

  6. [14]

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. https://doi.org/10.18653/v1/2023.findings-acl.507 Distilling step-by-step! outperforming larger language models with less training d...

    2023 doi

  7. [15]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  8. [16]

    Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. 2018. https://openreview.net/forum?id=r1VF9dCUG Finding flatter minima with sgd

    2018

  9. [17]

    Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. 2017. On large-batch training for deep learning: Generalization gap and sharp minima. 5th International Conference on Learning Representations, ICLR 2017 ; Conference date: ...

    2017

  10. [18]

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf Large language models are zero-shot reasoners . In Advances in Neural Inf...

    2022

  11. [19]

    Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. https://doi.org/10.1162/tacl_a_00160 Parsing algebraic word problems into equations . Transactions of the Association for Computational Linguistics, 3:585--597

    2015 doi

  12. [20]

    Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Bin Sun, Xinglin Wang, Heda Wang, and Kan Li. 2023. Turning dust into gold: Distilling complex reasoning capabilities from llms by leveraging negative data. arXiv preprint arXiv:2312.12832

    2023 arXiv

  13. [21]

    Weize Liu, Guocong Li, Kai Zhang, Bang Du, Qiyuan Chen, Xuming Hu, Hongxia Xu, Jintai Chen, and Jian Wu. 2024. https://doi.org/10.18653/v1/2024.naacl-long.376 Mind`s mirror: Distilling self-evaluation capability and comprehensive thinking from large language models . In Procee...

    2024 doi

  14. [22]

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022. Teaching small language models to reason. arXiv preprint arXiv:2212.08410

    2022 arXiv

  15. [23]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  16. [24]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. https://doi.org/10.18653/v1/2021.naacl-main.168 Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling...

    2021 doi

  17. [25]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  18. [26]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1)

    2020

  19. [27]

    Siyu Ren and Kenny Zhu. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.169 Specializing pre-trained language models for better relational reasoning via network pruning . In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2195--2207, Seattle,...

    2022 doi

  20. [28]

    Subhro Roy and Dan Roth. 2015. https://doi.org/10.18653/v1/D15-1202 Solving general arithmetic word problems . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743--1752, Lisbon, Portugal. Association for Computational Linguistics

    2015 doi

  21. [29]

    Brown, Adam Santoro, and et al

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, and et al. Aditya Gupta. 2023. https://arxiv.org/abs/2206.04615 Beyond the imitation game: Quantifying and extrapolating the capabilities of language mo...

    2023 arXiv

  22. [30]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  23. [31]

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. https://openreview.net/pdf?id=jiDsk12qcz Knowledge fusion of large language models . In The Twelfth International Conference on Learning Representations

    2024

  24. [32]

    Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023. https://doi.org/10.18653/v1/2023.acl-long.304 SCOTT : Self-consistent chain-of-thought distillation . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V...

    2023 doi

  25. [33]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824--24837

    2022

  26. [34]

    Bohao Yang, Chen Tang, Kun Zhao, Chenghao Xiao, and Chenghua Lin. 2024. https://aclanthology.org/2024.lrec-main.492/ Effective distillation of table-based reasoning ability from LLM s . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Lan...

    2024

  27. [35]

    Yichun Zhao, Shuheng Zhou, and Huijia Zhu. 2024. https://aclanthology.org/2024.lrec-main.1140/ Probe then retrieve and reason: Distilling probing and reasoning capabilities into smaller language models . In Proceedings of the 2024 Joint International Conference on Computationa...

    2024

  28. [36]

    Xuekai Zhu, Biqing Qi, Kaiyan Zhang, Xingwei Long, and Bowen Zhou. 2023. Pad: Program-aided distillation specializes large models in reasoning. arXiv preprint arXiv:2305.13888

    2023 arXiv

  29. [37]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.