Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Replacing thinking with tool usage enables reasoning in small language models

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that reformulating a small model's extra inference tokens as a tool-usage trace, rather than as natural-language reasoning, lets 1B and 3B models improve code repair beyond direct answering while a chain-of-thought…

desk verdict Real recipe for small-model tool-usage reasoning, but the CoT-fails comparison rests on an unfair baseline. read the letter →

arxiv 2507.05065 v1 pith:GZBZ4ZYM submitted 2025-07-07 cs.LG cs.AI

Corrado Rainone , Tim Bakker , Roland Memisevic This is my paper
classification cs.LGcs.AI
keywords chain-of-editstoolusagesmalllanguagemodelstest-timescalingreinforcementlearningwithverifiablerewardscoderepairchain-of-thoughtLoRA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the extra tokens a model spends at inference time do not have to be natural-language reasoning: formatting them as a multi-turn tool-usage trace lets small language models put test-time compute to work. On a code-repair benchmark derived from the MBPP Python programming problems, a 1B model's pass@1 rises from 1.3% with direct answering to 7.82% after training on these Chains-of-Edits, and a 3B model rises from 6.9% to 13.8%; a natural-language chain-of-thought baseline trained on s1K instead collapses to 0.15% and 1.44% at those sizes. The same training does not help an 8B model, where direct answering is already stronger. If the claim holds, it offers a route to reasoning-like behavior in models too small to bootstrap their own free-form reasoning.

What carries the argument

The central object is the Chain-of-Edits (CoE), a trace of interactions between the language model and a stateful text editor. The model does not emit free-form reasoning; at each turn it issues one command from a small DSL (add line, delete line, replace line, replace word, or exit), the editor executes the command and runs the resulting code against unit tests, and the new state—line-numbered code plus execution feedback—is appended to the context. This mechanism carries the argument by shrinking the action space that RL must explore and by making the reward observable per turn, so a 1B model can learn to spend additional tokens productively without a large model's exploration prior.

What would settle it

Train the same 1B and 3B Llama checkpoints on a chain-of-thought dataset built directly from code-repair traces, with execution feedback included in the prompt, and evaluate on the same 1317-task test set: if that baseline reaches or exceeds the CoE pass@1 values of 7.82% and 13.8%, the claim that tool-usage traces uniquely unlock small-model inference-time reasoning is falsified.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that constraining 'thinking' tokens to be interactions with a stateful editor creates a learning problem small models can solve: the action space shrinks to a few DSL commands, the editor state plus unit-test feedback is appended after every turn, and the reward can be scored per turn rather than only at the end. With SFT on synthetic demonstrations followed by RLVR with LoRA, the CoE agents beat their direct-answer pass@1 (the fraction of test tasks solved on the first attempt) at 1B and 3B by a wide margin, while the s1K chain-of-thought baseline underperforms direct answering at those sizes. At 8B the ordering flips, which the paper reads as evidence that the constrained format helps where the model's exploration prior is weak, and can get in the way where the model is already strong.

Load-bearing premise

The central claim rests on the assumption that SFT on the math-focused s1K dataset, transferred to code repair through a generic prompt template, is a fair representative of text-based chain-of-thought training; if that baseline is unrepresentative, the CoE-beats-CoT result for 1B and 3B models is not established.

Editorial extensions

If this is right

  • If the CoE numbers are right, inference-time scaling does not require a large model: 1B and 3B Llama models can profit from spending extra tokens when those tokens are tool commands.
  • The 8B reversal indicates that the tool-trace format is not universally better; model scale determines whether constraining the action space helps or hurts.
  • Per-turn rewards from unit-test feedback make RLVR stable enough for small models trained with LoRA, so the pipeline runs on a single 16GB GPU for the 1B model.
  • The CoT baseline's degenerate outputs at small scale—repetition, simulated feedback, and delimiter failures—show a concrete failure mode that the CoE format avoids.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper only tests code repair on Llama models; if the action-space-reduction story is right, the same recipe should transfer to other verifiable tasks with compact stateful tools, such as SQL debugging or structured editing, but that is an extrapolation.
  • A CoT baseline trained on code-repair traces with the same reward would be the sharper comparison; without it, the CoE-vs-CoT gap may partly reflect the s1K transfer mismatch rather than the tool format itself.
  • Because the editor state is written into the context every turn, CoE relieves the model of maintaining hidden state, which suggests an untested benefit on longer tasks where hidden state drifts.
  • The reward design is not the only plausible one; ablating the dense per-turn reward against an end-of-trajectory reward would isolate whether reward density or action-space constriction drives the gain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Chain-of-Edits (CoE), an inference-time token parameterization in which a small language model interacts with a text editor through a restricted DSL, receiving execution feedback after each edit. The authors train 1B, 3B, and 8B Llama models with SFT on 35,223 synthetic CoE demonstrations and RLVR on a code-repair benchmark derived from MBPP. They compare CoE against SFT on the s1K dataset as a chain-of-thought (CoT) baseline and against a 3-shot direct-answer baseline. Table 3 reports that CoE improves pass@1 over direct answering at 1B (7.82% vs 1.3%) and 3B (13.8% vs 6.9%), while the s1K baseline is far lower (0.15% and 1.44%); at 8B the order reverses. The authors conclude that tool-usage traces, rather than natural-language reasoning, enable small models to profit from additional inference-time compute.

Significance. The core idea is original and well-motivated: constraining the action space with a DSL and providing dense, execution-based rewards addresses a known bottleneck for RLVR in small models. The synthetic demonstration generation from invertible corruptions is clean and avoids reliance on teacher-model outputs, and the paper experiments across three model sizes with LoRA-only training. The level of experimental detail, including the full hyperparameter tables and reward-function ablations in Appendix C, is commendable. However, the current evaluation does not fully support the central CoE-vs-CoT claim because the comparison is confounded by training signal and data volume, and the reported scores are selected on the test set without error bars. With a strengthened evaluation, the paper would be a useful contribution to test-time scaling and tool-augmented reasoning for small models.

major comments (4)
  1. [Section 5.2, Appendix D] The s1K baseline is not task-matched and does not isolate the token parameterization. CoE is trained on 35,223 task-specific CoE demonstrations (Section 4.1.1) plus RLVR on 9,760 code-repair tasks with execution feedback (Section 5.1), whereas the CoT baseline is SFT on 1,000 math-reasoning examples evaluated through a transfer prompt. The conclusion that the text-based CoT approach fails to induce the improvement (Section 1 and the abstract) therefore rests on a comparison that conflates token format, training data scale, and reward signal. A task-matched CoT baseline trained on the same code-repair demonstrations and, ideally, with RLVR on the same benchmark is needed before the CoE-beats-CoT conclusion is supported.
  2. [Section 5.3] Test-set checkpoint selection and lack of variance undermine the reported point estimates. The paper states that for runs that perform well on the evaluation dataset, the authors select at most two checkpoints to evaluate on the test set of 1317 problems and report the score of the best performing checkpoint, and for the s1K baseline they evaluate on the test set after 1 and 5 epochs and report the best performing checkpoint. Selecting the best result on the test set is an optimistic selection procedure and does not give an unbiased performance estimate. No error bars, multiple seeds, or bootstrap intervals are provided. The 1B gap may be robust, but the 3B gap (13.8% vs 6.9%) and the 8B reversal need variance information before they can be interpreted as genuine effects.
  3. [Table 6 and Appendix B.2] There is an inconsistency between Table 6 and Table 3. Table 6 is described as reporting 3-shot direct-answer results for the pre-trained models, but its Repair dataset (3-shot) column contains the s1K baseline values from Table 3 (e.g., 0.15%/0.53% for 1B, 1.44%/5.24% for 3B, 23.3%/46.2% for 8B), not the direct-answer values (1.3%/3.1%, 6.9%/12.0%, 33.4%/42.9%). This is either a mislabeling or a copy-paste error, and it must be corrected because it affects the interpretation of the difficulty of the repair benchmark and the baseline comparisons.
  4. [Section 5.3, Table 7] The 8B model is 4-bit quantized, while the 1B and 3B models are not. The paper interprets the 8B reversal (CoE below direct answer) as evidence that text-based reasoning becomes feasible at this size, but quantization could plausibly degrade the multi-turn DSL following and tool-use ability, contributing to the CoE drop. The size comparison is therefore confounded. At minimum, the paper should acknowledge this confound explicitly and, if possible, provide an 8B CoE run without quantization or otherwise control for the quantization effect.
minor comments (4)
  1. [Appendix B.1] The REPW reversal can produce a line different from the original (e.g., for i i range(10) from a typo on in), and the paper says such sequences are skipped. Reporting the fraction of skipped demonstrations would help assess the reliability of the synthetic data generation.
  2. [Section 5.3] The metric pass@4 is computed by sampling at temperature 0.2, but the paper does not state whether the reported number is the fraction of tasks with at least one successful sample among four or an unbiased pass@k estimator. Please clarify which convention is used.
  3. [Box 4 and Box 7] The example task in Box 4 is described as check if the triangle is scalene or not, but the function name and unit tests correspond to checking isosceles (any two sides equal). This mismatch appears in the prompt template and is confusing; if such mismatches occur in the benchmark data, they may affect the alignment between the natural-language description and the unit tests.
  4. [Section 6] The Limitations section does not mention the task-matched baseline gap or the test-set checkpoint selection issue; both should be added so that the stated limitations reflect the actual threats to the main claim.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: CoE results are measured on held-out repair tasks; the s1K baseline mismatch is a control-quality issue, not a circularity.

full rationale

The paper's central chain is training CoE on synthetic demonstrations built by inverting corruptions of ground-truth MBPP code, followed by RLVR on Llama-generated broken solutions, and evaluating on a held-out MBPP test split. None of these stages fits the test metric: CoE demonstrations are generated from the MBPP training split with separate validation and test sets, RLVR uses 4000 repair tasks from the training split, and test pass@1/pass@4 are computed on 1317 unseen repair tasks. The s1K baseline is an external dataset trained independently; its weakness as a representative CoT baseline is a fairness/control issue, not circularity. The one self-citation (Bakker et al., 2020) is used only to motivate a generic per-turn return baseline in RL and is not load-bearing for the paper's claims. No equation or definition equates a reported prediction with an input by construction; reward choices and hyperparameters are selected on validation data, which is standard practice. The overall derivation is self-contained against external benchmarks.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The Chain-of-Edits workflow and DSL are methodological components, not invented entities with independent falsifiable handles. The main assumptions are the reliability of unit-test feedback, the expressiveness of the DSL, and the fairness of the s1K baseline as a proxy for text-based reasoning.

free parameters (3)
  • format reward penalty = -0.5
    Hand-chosen penalty for malformed DSL actions; authors searched [-2.0, 0.0] and selected -0.5 (Appendix C.3).
  • EXIT-after-solved penalty = -0.5
    Penalty when the model does not emit EXIT after all tests pass (Section 4.2).
  • task reward function = OnlyWhenSolved
    Selected as best among three task rewards tried (OnlyWhenSolved, UnitTestFraction, UnitTestFraction+EditDistanceBonus); this choice affects the learning signal (Appendix C.3).
assumptions (3)
  • domain assumption Unit tests define correctness and provide reliable verifiable reward.
    All rewards and pass@k metrics depend on the unit tests being correct and execution feedback being accurate (Sections 3.1, 5.1).
  • domain assumption The DSL editing commands are expressive enough to repair the LLM-generated bugs in the benchmark.
    If some bugs required whole-function rewrites or structural changes beyond line/word edits, the action space would be insufficient; the paper does not analyze DSL coverage (Section 3.1).
  • ad hoc to paper SFT on the s1K dataset is a fair representative of text-based CoT training for small models.
    The comparison baseline is SFT on a math/reasoning dataset (s1K) rather than task-matched CoT training or CoT-RLVR; the paper's claim that text-based CoT fails relies on this baseline (Sections 5.2, 5.4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Replacing thinking with tool usage enables reasoning in small language models." pith.science (2026). https://pith.science/paper/GZBZ4ZYM

@misc{pith2026250705065,
  author       = {Pith},
  title        = {Pith review of: Replacing thinking with tool usage enables reasoning in small language models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GZBZ4ZYM}},
  note         = {Machine review of arXiv:2507.05065}
}
read the original abstract

Recent advances have established a new machine learning paradigm based on scaling up compute at inference time as well as at training time. In that line of work, a combination of Supervised Fine-Tuning (SFT) on synthetic demonstrations and Reinforcement Learning with Verifiable Rewards (RLVR) is used for training Large Language Models to expend extra compute during inference in the form of "thoughts" expressed in natural language. In this paper, we propose to instead format these tokens as a multi-turn interaction trace with a stateful tool. At each turn, the new state of the tool is appended to the context of the model, whose job is to generate the tokens necessary to control the tool via a custom DSL. We benchmark this approach on the problem of repairing malfunctioning Python code, and show that this constrained setup allows for faster sampling of experience and a denser reward signal, allowing even models of size up to 3B parameters to learn how to proficiently expend additional compute on the task.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SOD: Step-wise On-policy Distillation for Small Language Model Agents

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    SOD reweights on-policy distillation strength step-by-step using divergence to stabilize tool use in small language model agents, yielding up to 20.86% gains and 26.13% on AIME 2025 for a 0.6B model.

Reference graph

Works this paper leans on

14 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Austin, A

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,

    arXiv

  2. [4]

    We fine-tune for 5 epochs, following Muennighoff et al. [2025]. We report in table 10 all the hyperparameters used. When evaluating the finetuned models, we prompt them as shown in box 7, and simply grab and verify (i.e. run agains the unit tests) whatever the model generated between [BEGIN SOLUTION] and [END SOLUTION] delimiters. 5https://huggingface.co/...

    work page 2025

  3. [5]

    ", ":"); L 1 L 2 import re L 3 def fill_spaces(s): L 4 s = s.replace (

    Note that, although the final code snippet passes all unit tests and thus is considered correct, it 15 technically does not fully align with the MBPP task; it fails to replace commas or dots with a colon in any input string, but the unit tests fail to cover for this scenario. Box 5: Example (short) code repair trace generated by the trained 1B model. You ...

    work page 2020

  4. [7]

    Accessed: 2025-5-14. U. Piterbarg, L. Pinto, and R. Fergus. Training language models on synthetic edit sequences improves code synthesis. arXiv [cs.LG], Oct

    work page 2025

  5. [8]

    Schick, J

    T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Tool- former: Language models can teach themselves to use tools. Adv. Neural Inf. Process. Syst., abs/2302.04761, Feb

    arXiv

  6. [9]

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Fun- towicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the-art natural language processing. In Q. Liu and D. Schlangen, editors, Proceedings of the ...

    work page 2020

  7. [10]

    doi: 10.18653/v1/2020.emnlp-demos.6

    Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6/. G. Xinyu, L. L. Zhang, L. Yifei, S. Ning, S. Youran, Z. Yi, Y . Fan, and Y . Mao. RStar-math: Small LLMs can master math reasoning with self-evolved deep thinking. arXiv [cs.CL], Jan

    doi:10.18653/v1/2020.emnlp-demos.6 2020

  8. [11]

    M. Yang, D. Schuurmans, P. Abbeel, and O. Nachum. Chain of thought imitation with procedure cloning. Adv. Neural Inf. Process. Syst., abs/2205.10816, May

    work page Pith review arXiv

Show all 14 references
  1. [16]

    For the non-quantized models, we train separate adapters at each stage in our pipeline, always merging adapters of previous training stages before staring the next round of training (however, we keep these adapters saved separately, for flexibility and to save storage space). ...

    2024

  2. [2020]

    S. Hao, T. Liu, Z. Wang, and Z. Hu. ToolkenGPT: Augmenting frozen language models with massive tools via tool embeddings. Adv. Neural Inf. Process. Syst., abs/2305.11554, May

    arXiv

  3. [2022]

    R. Liu, J. Gao, J. Zhao, K. Zhang, X. Li, B. Qi, W. Ouyang, and B. Zhou. Can 1B LLM surpass 405B LLM? rethinking compute-optimal test-time scaling. arXiv [cs.CL], Feb. 2025a. Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin. Understanding R1-Zero-Like Train...

    arXiv

  4. [2023]

    URL https: //arxiv.org/abs/2312.03732. W. Kool, H. van Hoof, and M. Welling. Buy 4 REINFORCE Samples, Get a Baseline for Free!,

    arXiv

  5. [2024]

    URL https://arxiv.org/abs/2407.21783. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations,

    arXiv

  6. [2025]

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. Mankowitz, E. Sutherland Robson,...

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.