REVIEW 4 major objections 5 minor 24 references
Teaching LLMs to Refine with Tools
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper argues that a 7B-parameter language model can be trained to refine chain-of-thought answers by rewriting them as code, and that the refinement only becomes effective after a preference-optimization stage.
desk verdict A genuinely novel cross-format refinement method (CoT-to-PoT) with consistent gains, but the headline claim that DPO is the critical ingredient is confounded by the experimental design. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the multi-turn training instance (question, CoT attempt, critic verdict, PoT rewrite), built by best-of-N sampling from teacher models. A reference-based critic, trained on roughly 30K annotations from a large external model, labels each CoT and PoT attempt as correct or not; those labels select winning and losing pairs for SFT and DPO and also rank responses during inference. The training objective first learns to generate the winning PoT from the CoT context, then applies a DPO variant with an SFT regularizer to prefer the winning PoT over a losing one given the same context. Named components: CoT is step-by-step reasoning in natural language; PoT is solving the problem by writing and executing code.
What would settle it
Shuffle the critic's YES/NO labels during DPO training while keeping everything else fixed; if the refinement gain survives label noise, the critic is not the active mechanism, and if it collapses, the critic's judgment is doing the causal work.
Extended reading notes
Core claim
The paper claims that cross-format refinement is learnable: given a question, a prior chain-of-thought attempt, and a critic's verdict on that attempt, a trained model can produce a better program-of-thought solution. Using greedy decoding, CaP after SFT barely moves the backbone's own CoT accuracy (87.8% versus 86.8% for the initial attempt) and fails to refine a strong teacher's CoT answers; after DPO the gap widens to 89.5% on self-generated attempts and 90.5% on the teacher's attempts, so a 7B model effectively corrects answers from a model roughly ten times its size. The paper also reports that replacing PoT with CoT as the refinement format under identical training reproduces the known non-correcting behavior (86.8% to 86.8%), and that the gain transfers to three different backbone models. At inference time, reallocating half of a best-of-N budget to CoT sampling and half to PoT refinement (the BoNBoN strategy) outperforms spending the whole budget on PoT alone.
Load-bearing premise
The pipeline depends on a critic model, trained on about 30K annotations, to judge whether chain-of-thought and code answers are correct; if that critic is biased toward one format, the refinement gains could come from matching the critic's preference rather than from truly fixing errors.
Editorial extensions
If this is right
- A fixed 7B model can act as its own refiner, so post-training self-improvement no longer requires a stronger model to supply corrected answers.
- Off-policy CoT attempts are usable: refinement works on attempts from a weaker or larger model, not only on self-generated data, so training data can be collected cheaply from any teacher.
- Preference optimization should be considered an essential stage in any cross-format self-correction pipeline; SFT-only versions will mostly reproduce the first attempt.
- Within a fixed inference budget, sampling diversity matters across formats: splitting the budget between CoT and PoT (BoNBoN) beats using the entire budget for one format.
Reading between the lines
- The same paired data could be used the other way, training a model to translate code back into natural-language explanations, since the paper's data are solution-level parallel CoT/PoT pairs; the paper reports that adding the reverse task hurts, so the asymmetry itself is worth studying.
- Because code execution gives exact answer matching, the critic may be far more reliable for PoT than for CoT; a natural test is measuring how much of CaP's gain survives when the critic must judge a PoT sample whose code runs but whose logic is wrong.
- The BoNBoN result suggests that adaptive allocation, spending more budget on CoT for hard questions and more on PoT for easy ones, could outperform the fixed 50/50 split, though the paper only tests a balanced setting.
- If refinement is tied to a model's own problem-solving proficiency, as the paper observes for its weakest backbone, then raising the refiner's base reasoning ability would be a prerequisite for applying CaP to much stronger teachers.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CaP, a training pipeline that teaches a 7B- or 8B-parameter LLM to refine chain-of-thought (CoT) solutions by generating program-of-thought (PoT) solutions with code. The method first performs supervised fine-tuning on multi-turn data that pair a CoT attempt and a critic statement with a PoT solution, and then applies DPO with an additional SFT term on preference pairs built from positively and negatively judged PoT responses. Experiments on three Chinese mathematical benchmarks report that CaP_DPO outperforms CaP_SFT and several baselines, including when the CoT attempt comes from a 72B teacher, and that the proposed BoNBoN sampling strategy improves test-time compute efficiency. The central scientific claim is that preference optimization is critical for enabling effective refinement.
Significance. If the central claim is valid, CaP would be a practically useful contribution: it demonstrates that a relatively small model can refine the CoT output of a much larger model by switching to a code-based reasoning format, and it offers a concrete inference-time budget-allocation strategy (BoNBoN) with consistent gains. The paper also reports generalization across three backbone models and gives a clear, reproducible training-data construction recipe. However, the evidence for the headline claim is currently weakened by a confounded SFT-versus-DPO comparison and by the absence of any validation of the critic model that labels the data; both issues are fixable with additional experiments.
major comments (4)
- [Section 3.3, Eqs. (3) and (5)] The claim that preference optimization is critical is not cleanly supported because CaP_SFT and CaP_DPO differ in more than the preference signal. CaP_DPO adds negative PoT examples y-_pot that are absent from the SFT data, adds the explicit SFT term λ · log πθ(y+_pot | z) that re-trains on positive PoT responses, and consumes additional optimization steps on the preference dataset. The +2.1-point improvement on CoT72B in Table 2 (88.4→90.5) could therefore come from training longer on positive data or from the stabilizing SFT term rather than from the ranking between y+_pot and y-_pot. I ask for controlled ablations: for example, continue training CaP_SFT for the same number of additional steps on the positive pairs only, or remove the negative examples from the DPO loss while keeping all other factors fixed. Without such controls, the paper's main interpretation remains one of several plausible explanations.
- [Sections 2.1 and 3.1] The entire training-data construction and the Best-of-N inference selection depend on critic models whose accuracy and format-dependent behavior are not analyzed. A single reference-based critic, trained on about 30K GPT-4-0613 annotations, labels CoT and PoT responses for the SFT and DPO pairs, and both reference-based and reference-free critics rank responses during BoN inference. If the critic is more reliable at judging PoT outputs (which can be checked by code execution) than CoT texts, the resulting preference pairs may encode a bias toward PoT as a format rather than a genuine error-correction capability. I request a validation study: report the critic's agreement with ground-truth exact-answer or execution-based labels on a held-out sample, broken down by reasoning format and by whether the preceding CoT is correct. Also report how the main results change when the reference-free critic is used instead of the reference-based one at inference, or when the critic threshold is varied.
- [Tables 2 and 3] All experimental results are single runs with no error bars or significance tests, yet several headline differences are small. For example, in Table 2 the average gap between CaP_SFT and CaP_DPO on CoT7B is 1.5 points, and in Table 3 the BoNBoN advantage over standard BoN at the same budget is about 0.1–0.6 points on individual benchmarks. Without multiple seeds, bootstrap confidence intervals, or a paired test across the benchmark questions, it is difficult to determine whether the observed improvements are reliable. I ask for variance estimates on at least the main comparisons, or, failing that, a clear statement that the differences are not statistically assessed.
- [Section 3.3] The inference protocol 'consistently apply a positive critic to all CoT attempts' gives CaP an additional cue that the baselines do not receive. In the greedy-decoding comparisons of Table 2, the CoT7B, CoTself, and CoT72B rows are the model's raw CoT outputs, while CaP rows are conditioned on the phrase 'THE PROBLEM-SOLVING PROCESS MIGHT BE CORRECT' together with the CoT attempt. Telling the model that the attempt might be correct could act as a hint that biases it toward trusting the provided reasoning, independent of any refinement skill. I ask for a control condition in which the same multi-turn format is used with a neutral or uninformative cue (e.g., 'The following is a problem-solving process.'), or in which baselines are also provided with the positive critic, so the effect of the cue can be separated from the effect of the training objective.
minor comments (5)
- [Abstract and Introduction] The abstract and introduction state that the method teaches LLMs to 'refine with tools,' but the experiments only cover refinement from CoT to PoT; the paper should state this scope explicitly early on, including in the title or abstract, to avoid overgeneralization.
- [Section 3.1] The description of the sampling budgets is unclear: the text says N1 = 5 CoT samples and N2 = 3 PoT samples are used to construct training data, while Table 3 reports test-time budgets of 2, 8, and 32 samples. Please clarify how N1 and N2 relate to the BoN sampling sizes used at training time and at inference.
- [Table 3] The table uses inconsistent capitalization for the model names (CAPSFT and CAPDPO instead of CaP_SFT and CaP_DPO); please harmonize the notation throughout.
- [Section 2.2, Eq. (1)] The set notation D_CaP = {(q, y+_cot, c+, y+_pot)} ∪ {(q, y−_cot, c−, y+_pot)} is a bit terse; it would be helpful to state explicitly that c+ and c− are fixed strings defined in the text and that y+_cot and y−_cot are the critic-selected best positive and negative CoT responses, respectively.
- [Section 3.6] The discussion of robustness to the word 'please' is interesting but is presented without a table or numbers; please include the supporting measurements or reference a figure, so the claim can be verified.
Circularity Check
No significant circularity: CaP is an empirical training pipeline whose reported refinements are measured on external benchmark accuracy, not reconstructed from the critic or training equations.
full rationale
The paper does not derive any quantity from its own definition. Training data construction (Sections 2.1-2.3) uses a GPT-4-0613-annotated critic to label CoT/PoT responses and the authors' prior SIaM8B model as PoT teacher, but these are inputs to an SFT/DPO training pipeline, not predictions extracted from it. The central results in Tables 2 and 3 are accuracies on out-of-distribution Chinese math benchmarks (CM17K, APE, CMATH), which have their own reference answers; they are not scored by the same critic that selected the training pairs, so the reported refinement gains are not the critic's labels returned to the reader. Equation (3) and Equation (5) define losses over fixed paired data, and no fitted parameter is later renamed as a prediction. The only self-citations (SIaM, DOTS, critic-related work by the same lab) supply teacher models, baselines, or comparison implementations; they do not carry a load-bearing uniqueness or derivation claim. The paper's claim that preference optimization is critical rests on a CaP_SFT vs CaP_DPO comparison that also varies negative examples, an added SFT term (lambda in Eq. 5), and additional training steps, which is a legitimate experimental confound but not a circularity. No circular step can be exhibited by reducing an equation or result to its own input.
Assumptions & free parameters
free parameters (5)
- N1 (CoT samples per question) =
5
- N2 (PoT samples per question) =
3
- DPO regularization coefficient β
- SFT-loss weight λ in DPO
- BoNBoN budget split =
e.g., 4+4, 16+16, 8+32
assumptions (5)
- domain assumption Ground-truth reference answers are available for every training question.
- domain assumption The reference-based critic model accurately labels correctness of CoT and PoT responses.
- domain assumption PoT teacher SIaM8B produces sufficiently diverse and correct programs to serve as positive refinement targets.
- domain assumption Off-policy pairing (generating CoT from one model and PoT from another) does not introduce a distribution mismatch that breaks SFT/DPO.
- ad hoc to paper Always applying a positive critic to CoT attempts at inference is a fair and effective protocol.
Cite this review
Pith. "Pith review of Teaching LLMs to Refine with Tools." pith.science (2026). https://pith.science/paper/74ZB5UNA
@misc{pith2026241216871,
author = {Pith},
title = {Pith review of: Teaching LLMs to Refine with Tools},
year = {2026},
howpublished = {\url{https://pith.science/paper/74ZB5UNA}},
note = {Machine review of arXiv:2412.16871}
}
read the original abstract
Large language models (LLMs) can refine their responses based on feedback, enabling self-improvement through iterative training or test-time refinement. However, existing methods predominantly focus on refinement within the same reasoning format, which may lead to non-correcting behaviors. We propose CaP, a novel approach that uses external tools to refine chain-of-thought (CoT) responses generated by the same or other LLMs. CaP employs a two-stage training process: supervised fine-tuning followed by preference optimization with DPO variants. Our observations highlight the critical role of preference optimization in enabling effective refinement. Additionally, we compare several sampling strategies to leverage CoT and tools at inference time. Experimental results demonstrate CaP's potential for effective cross-reasoning refinement and efficient inference.
Figures
Reference graph
Works this paper leans on
-
[2]
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588,
-
[4]
URL https:// openreview.net/forum?id=Ep0TtjVoap. Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917,
-
[5]
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050,
-
[7]
Adaptive inference-time compute: Llms can predict if they can do better, even mid-generation
Rohin Manvi, Anikait Singh, and Stefano Ermon. Adaptive inference-time compute: Llms can predict if they can do better, even mid-generation. arXiv preprint arXiv:2410.02725,
-
[8]
Llm critics help catch llm bugs
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215,
-
[9]
Iterative reasoning preference optimization
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733,
-
[11]
Self-critiquing models for assisting human evaluators
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802,
-
[12]
Bond: Aligning llms with best-of-n distillation
Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning llms with best-of-n distillation. arXiv preprint arXiv:2407.14622,
Show all 24 references
-
[13]
Scaling llm test-time compute optimally can be more effective than scaling model parameters
9 Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314,
-
[14]
Toward self- improvement of llms via imagination, searching, and criticizing
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. Toward self- improvement of llms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253,
-
[15]
Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning
Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731, 2023a. Tianlu Wang, Ping Yu, Xiaoqin...
-
[16]
Cmath: can your language model pass chinese elementary school math test? arXiv preprint arXiv:2306.16636,
Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. Cmath: can your language model pass chinese elementary school math test? arXiv preprint arXiv:2306.16636,
-
[17]
Generating sequences by learning to self-correct
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053,
-
[18]
Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline
Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, et al. Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline. arXiv preprint arXiv:2404.02893,
-
[19]
Qwen2 technical report
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024a. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu,...
-
[20]
Siam: Self-improving code-assisted mathematical reasoning of large language models
Dian Yu, Baolin Peng, Ye Tian, Linfeng Song, Haitao Mi, and Dong Yu. Siam: Self-improving code-assisted mathematical reasoning of large language models. arXiv preprint arXiv:2408.15565,
-
[21]
Dots: Learning to reason dynamically in llms via optimal reasoning trajectories search
Murong Yue, Wenlin Yao, Haitao Mi, Dian Yu, Ziyu Yao, and Dong Yu. Dots: Learning to reason dynamically in llms via optimal reasoning trajectories search. arXiv preprint arXiv:2410.03864,
-
[22]
Generative verifiers: Reward modeling as next-token prediction
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240,
-
[23]
Ape210k: A large-scale and template-rich dataset of math word problems
Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. Ape210k: A large-scale and template-rich dataset of math word problems. arXiv preprint arXiv:2009.11506,
2009 arXiv
-
[24]
h " and the d ia me ter of its base be
Association for Computational Linguistics. URL http://arxiv.org/abs/2403.13372. 10 A Appendices Instruction Your goal is to evaluate whether the model’s candidate answer to the given math problem is correct. – First, carefully read the math problem. – Then, check the candidate...
-
[2021]
Recursive introspection: Teaching language model agents how to self-improve
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. arXiv preprint arXiv:2407.18219,
-
[2022]
Stepwise verification and remediation of student reasoning errors with large language model tutors
Nico Daheim, Jakub Macina, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. Stepwise verification and remediation of student reasoning errors with large language model tutors. arXiv preprint arXiv:2407.09136,
-
[2023]
Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436,
-
[2024]
Large language monkeys: Scaling inference compute with repeated sampling
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787,
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.