{"id":"022bea9a-d05e-492a-a75a-e9af08d6f5cd","arxiv_id":"2412.16871","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CaP trains LLMs to fix chain-of-thought math solutions by producing program-of-thought code, and shows that DPO preference optimization is essential for the refinement to actually improve accuracy.","lead":"This paper trains a 7B language model to refine chain-of-thought (CoT) math solutions by generating and running Python code, rather than refining in the same natural-language format. The method, CaP, shows that preference optimization is what makes the refinement effective, and that a small model can even improve solutions from a 72B model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that preference optimization is critical is confounded: CaP_DPO differs from CaP_SFT in negative examples, an added SFT loss, and additional training steps, with no control for data or compute.","rationale":"The reader's weakest_assumption focuses on critic bias, which is a valid data-quality concern but does not directly threaten the reported accuracy gains because the final evaluation uses exact reference matching, not the critic. The more load-bearing issue is the attribution of the empirical gains to the DPO objective, which is the paper's headline observation and is confounded with data and compute. The reader's CONDITIONAL verdict is appropriate; my concern also calls for the same condition (add ablations), so the verdict is unchanged. Additional secondary concerns (no error bars, unreleased hyperparameters, 'first time' overclaim) support the conditional assessment but are not the core attack.","tokens_in":12528,"tokens_out":6429,"duration_ms":55194,"concrete_test":"Run a controlled ablation: take the CaP_SFT checkpoint, then fine-tune it for the same number of steps and on the same preference-pair data as CaP_DPO, but with only the SFT term (cross-entropy on every y+_pot in D_CaP_pref, with the same λ weighting and no DPO ratio term). Evaluate on the Table 2 CoT72B setting. If this extra-SFT model reaches ≈90.5 average accuracy, the preference objective contributes nothing beyond additional data/repetition, and the paper's central claim should be weakened accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central observation (Section 3.3) is that SFT alone is insufficient and DPO is essential for refinement. However, the comparison between CaP_SFT (trained only on D_CaP and D_CoT, Eq. 3) and CaP_DPO (trained on D_CaP_pref with Eq. 5) varies at least three factors: (1) CaP_DPO sees pairs with negative PoT responses y-_pot (which never appear in the SFT stage's cross-entropy targets); (2) the DPO loss includes an explicit SFT term λ·log πθ(y+_pot | z) that trains on the chosen PoT responses, effectively re-training on positive data; and (3) DPO necessarily consumes additional optimization steps on new data. Thus the observed +2.1 points on the headline CoT72B refinement (88.4→90.5 in Table 2) could be due to simply training longer on positive examples or to the stabilizing SFT term, rather than to the preference ranking between y+_pot and y-_pot. Since the 'critical role of preference optimization' is the paper's main scientific claim, this confound is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CaP, a training pipeline that teaches a 7B- or 8B-parameter LLM to refine chain-of-thought (CoT) solutions by generating program-of-thought (PoT) solutions with code. The method first performs supervised fine-tuning on multi-turn data that pair a CoT attempt and a critic statement with a PoT solution, and then applies DPO with an additional SFT term on preference pairs built from positively and negatively judged PoT responses. Experiments on three Chinese mathematical benchmarks report that CaP_DPO outperforms CaP_SFT and several baselines, including when the CoT attempt comes from a 72B teacher, and that the proposed BoNBoN sampling strategy improves test-time compute efficiency. The central scientific claim is that preference optimization is critical for enabling effective refinement.","tokens_in":12794,"tokens_out":4621,"duration_ms":45706,"significance":"If the central claim is valid, CaP would be a practically useful contribution: it demonstrates that a relatively small model can refine the CoT output of a much larger model by switching to a code-based reasoning format, and it offers a concrete inference-time budget-allocation strategy (BoNBoN) with consistent gains. The paper also reports generalization across three backbone models and gives a clear, reproducible training-data construction recipe. However, the evidence for the headline claim is currently weakened by a confounded SFT-versus-DPO comparison and by the absence of any validation of the critic model that labels the data; both issues are fixable with additional experiments.","major_comments":[{"comment":"The claim that preference optimization is critical is not cleanly supported because CaP_SFT and CaP_DPO differ in more than the preference signal. CaP_DPO adds negative PoT examples y-_pot that are absent from the SFT data, adds the explicit SFT term λ · log πθ(y+_pot | z) that re-trains on positive PoT responses, and consumes additional optimization steps on the preference dataset. The +2.1-point improvement on CoT72B in Table 2 (88.4→90.5) could therefore come from training longer on positive data or from the stabilizing SFT term rather than from the ranking between y+_pot and y-_pot. I ask for controlled ablations: for example, continue training CaP_SFT for the same number of additional steps on the positive pairs only, or remove the negative examples from the DPO loss while keeping all other factors fixed. Without such controls, the paper's main interpretation remains one of several plausible explanations.","section":"Section 3.3, Eqs. (3) and (5)"},{"comment":"The entire training-data construction and the Best-of-N inference selection depend on critic models whose accuracy and format-dependent behavior are not analyzed. A single reference-based critic, trained on about 30K GPT-4-0613 annotations, labels CoT and PoT responses for the SFT and DPO pairs, and both reference-based and reference-free critics rank responses during BoN inference. If the critic is more reliable at judging PoT outputs (which can be checked by code execution) than CoT texts, the resulting preference pairs may encode a bias toward PoT as a format rather than a genuine error-correction capability. I request a validation study: report the critic's agreement with ground-truth exact-answer or execution-based labels on a held-out sample, broken down by reasoning format and by whether the preceding CoT is correct. Also report how the main results change when the reference-free critic is used instead of the reference-based one at inference, or when the critic threshold is varied.","section":"Sections 2.1 and 3.1"},{"comment":"All experimental results are single runs with no error bars or significance tests, yet several headline differences are small. For example, in Table 2 the average gap between CaP_SFT and CaP_DPO on CoT7B is 1.5 points, and in Table 3 the BoNBoN advantage over standard BoN at the same budget is about 0.1–0.6 points on individual benchmarks. Without multiple seeds, bootstrap confidence intervals, or a paired test across the benchmark questions, it is difficult to determine whether the observed improvements are reliable. I ask for variance estimates on at least the main comparisons, or, failing that, a clear statement that the differences are not statistically assessed.","section":"Tables 2 and 3"},{"comment":"The inference protocol 'consistently apply a positive critic to all CoT attempts' gives CaP an additional cue that the baselines do not receive. In the greedy-decoding comparisons of Table 2, the CoT7B, CoTself, and CoT72B rows are the model's raw CoT outputs, while CaP rows are conditioned on the phrase 'THE PROBLEM-SOLVING PROCESS MIGHT BE CORRECT' together with the CoT attempt. Telling the model that the attempt might be correct could act as a hint that biases it toward trusting the provided reasoning, independent of any refinement skill. I ask for a control condition in which the same multi-turn format is used with a neutral or uninformative cue (e.g., 'The following is a problem-solving process.'), or in which baselines are also provided with the positive critic, so the effect of the cue can be separated from the effect of the training objective.","section":"Section 3.3"}],"minor_comments":[{"comment":"The abstract and introduction state that the method teaches LLMs to 'refine with tools,' but the experiments only cover refinement from CoT to PoT; the paper should state this scope explicitly early on, including in the title or abstract, to avoid overgeneralization.","section":"Abstract and Introduction"},{"comment":"The description of the sampling budgets is unclear: the text says N1 = 5 CoT samples and N2 = 3 PoT samples are used to construct training data, while Table 3 reports test-time budgets of 2, 8, and 32 samples. Please clarify how N1 and N2 relate to the BoN sampling sizes used at training time and at inference.","section":"Section 3.1"},{"comment":"The table uses inconsistent capitalization for the model names (CAPSFT and CAPDPO instead of CaP_SFT and CaP_DPO); please harmonize the notation throughout.","section":"Table 3"},{"comment":"The set notation D_CaP = {(q, y+_cot, c+, y+_pot)} ∪ {(q, y−_cot, c−, y+_pot)} is a bit terse; it would be helpful to state explicitly that c+ and c− are fixed strings defined in the text and that y+_cot and y−_cot are the critic-selected best positive and negative CoT responses, respectively.","section":"Section 2.2, Eq. (1)"},{"comment":"The discussion of robustness to the word 'please' is interesting but is presented without a table or numbers; please include the supporting measurements or reference a figure, so the claim can be verified.","section":"Section 3.6"}],"recommendation":"major_revision","confidential_remarks":"The paper is labeled 'Work in progress' on the arXiv version, but the submission appears to be a full empirical study. The main concern is that the central claim about preference optimization is confounded and the critic model is unvalidated; both are addressable with additional experiments. If the authors can supply the controlled comparisons and critic validation requested above, the paper could become a solid contribution. I also note that some of the baselines (PairDPO, DOTS) are implemented by the authors; it would be useful to confirm with the original papers' code where possible, though I did not find evidence of misimplementation in the text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is new — training a small model to refine natural-language CoT responses by generating executable code (PoT), using an SFT-then-DPO recipe — and it shows consistent gains across three Chinese math benchmarks and three base models, even letting a 7B model improve a 72B model's CoT. That's worth attention. But the paper's central claim that preference optimization is essential is not actually isolated by the experiments, and the evaluation has some thin spots.\n\nWhat's genuinely good: cross-format refinement is a fresh direction; prior work stayed within one reasoning format or selected between CoT and PoT. The BoNBoN sampling strategy is a reasonable addition. The experiments are fairly thorough: multiple backbones, comparisons to CoT-only Pair-SFT/DPO and PoT-only SiAM, and an honest discussion of what doesn't work (CoT-to-CoT refinement fails, adding a PoT-to-CoT task hurts). The finding that a 7B model can refine 72B CoT output is practically interesting.\n\nSoft spots, in order of severity. First, the CaP_DPO vs. CaP_SFT comparison is confounded. The DPO stage introduces negative PoT examples, an explicit SFT loss on the chosen PoT responses, and more gradient steps. The observed +2.1 points on the headline CoT72B result could come from any of those factors, not necessarily from the preference ranking. The paper says 'preference optimization is critical' but provides no ablation that separates the ranking signal from additional training on positives. This is the load-bearing caveat for the main claim. Second, all results are single-run with no error bars or significance tests; 1–2 point differences could be noise. Third, the critic model used to construct pairs and for Best-of-N is a black box — no accuracy or bias analysis is provided. That's a real concern for training-data quality, though the final evaluation is against reference answers, so it doesn't directly invalidate the measured accuracy. Minor: the 'for the first time' phrasing is too strong, and the training data (1M Chinese QA pairs, 30K GPT-4 annotations) is proprietary and not released.\n\nVerdict: the paper deserves peer review — the direction is useful and the method seems to work — but the authors should be asked to add ablations (e.g., longer SFT on positives, DPO without the SFT term), multi-seed runs, and critic diagnostics before the strong claim is accepted. I'd cite the paper for the cross-format refinement idea; it's a good pointer for anyone working on self-correction and tool use.","headline":"A genuinely novel cross-format refinement method (CoT-to-PoT) with consistent gains, but the headline claim that DPO is the critical ingredient is confounded by the experimental design.","tokens_in":13359,"tokens_out":3043,"would_cite":true,"duration_ms":30213,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a 7B-parameter language model can be trained to refine chain-of-thought answers by rewriting them as code, and that the refinement only becomes effective after a preference-optimization stage.","keywords":["chain-of-thought refinement","program-of-thought","preference optimization","DPO","mathematical reasoning","test-time compute","self-improvement","tool-augmented reasoning"],"falsifier":"Shuffle the critic's YES/NO labels during DPO training while keeping everything else fixed; if the refinement gain survives label noise, the critic is not the active mechanism, and if it collapses, the critic's judgment is doing the causal work.","tokens_in":12309,"feed_emoji":"🛠️","tokens_out":8967,"duration_ms":75002,"temperature":0.7,"pith_summary":"CaP, the approach introduced here, trains a 7B-parameter model to refine chain-of-thought (step-by-step) answers by rewriting them as executable code, which the paper calls program-of-thought. Training proceeds in two stages: supervised fine-tuning on paired CoT attempts and correct PoT rewrites, followed by preference optimization with a DPO variant that prefers the PoT rewrite over a rejected PoT sample in the same context. The paper's central observation is that the preference stage is the active ingredient: after SFT alone the model barely improves its first attempt, while adding DPO lets it correct CoT answers from itself, from a weaker model, and from a model roughly ten times larger. If this holds, it gives a practical route to test-time self-improvement that needs no stronger teacher and no per-step human supervision, by converting natural-language reasoning into verifiable code.","feed_headline":"7B model corrects 10x bigger model's reasoning with code","feed_subtitle":"Only after preference optimization does a 7B model actually fix larger models' chain-of-thought mistakes.","key_machinery":"The load-bearing object is the multi-turn training instance (question, CoT attempt, critic verdict, PoT rewrite), built by best-of-N sampling from teacher models. A reference-based critic, trained on roughly 30K annotations from a large external model, labels each CoT and PoT attempt as correct or not; those labels select winning and losing pairs for SFT and DPO and also rank responses during inference. The training objective first learns to generate the winning PoT from the CoT context, then applies a DPO variant with an SFT regularizer to prefer the winning PoT over a losing one given the same context. Named components: CoT is step-by-step reasoning in natural language; PoT is solving the problem by writing and executing code.","core_discovery":"The paper claims that cross-format refinement is learnable: given a question, a prior chain-of-thought attempt, and a critic's verdict on that attempt, a trained model can produce a better program-of-thought solution. Using greedy decoding, CaP after SFT barely moves the backbone's own CoT accuracy (87.8% versus 86.8% for the initial attempt) and fails to refine a strong teacher's CoT answers; after DPO the gap widens to 89.5% on self-generated attempts and 90.5% on the teacher's attempts, so a 7B model effectively corrects answers from a model roughly ten times its size. The paper also reports that replacing PoT with CoT as the refinement format under identical training reproduces the known non-correcting behavior (86.8% to 86.8%), and that the gain transfers to three different backbone models. At inference time, reallocating half of a best-of-N budget to CoT sampling and half to PoT refinement (the BoNBoN strategy) outperforms spending the whole budget on PoT alone.","pith_inferences":["The same paired data could be used the other way, training a model to translate code back into natural-language explanations, since the paper's data are solution-level parallel CoT/PoT pairs; the paper reports that adding the reverse task hurts, so the asymmetry itself is worth studying.","Because code execution gives exact answer matching, the critic may be far more reliable for PoT than for CoT; a natural test is measuring how much of CaP's gain survives when the critic must judge a PoT sample whose code runs but whose logic is wrong.","The BoNBoN result suggests that adaptive allocation, spending more budget on CoT for hard questions and more on PoT for easy ones, could outperform the fixed 50/50 split, though the paper only tests a balanced setting.","If refinement is tied to a model's own problem-solving proficiency, as the paper observes for its weakest backbone, then raising the refiner's base reasoning ability would be a prerequisite for applying CaP to much stronger teachers."],"forward_implications":["A fixed 7B model can act as its own refiner, so post-training self-improvement no longer requires a stronger model to supply corrected answers.","Off-policy CoT attempts are usable: refinement works on attempts from a weaker or larger model, not only on self-generated data, so training data can be collected cheaply from any teacher.","Preference optimization should be considered an essential stage in any cross-format self-correction pipeline; SFT-only versions will mostly reproduce the first attempt.","Within a fixed inference budget, sampling diversity matters across formats: splitting the budget between CoT and PoT (BoNBoN) beats using the entire budget for one format."],"supporting_citations":[{"why":"Defines chain-of-thought reasoning, the format being refined.","marker":"Wei et al., 2022"},{"why":"Defines program-of-thought, the tool format used for refinement.","marker":"Chen et al., 2022"},{"why":"Supplies the idea of pairing correct and incorrect responses to simulate multi-turn self-correction.","marker":"Welleck et al., 2022"},{"why":"Used with Welleck as the inspiration for constructing CoT-CoT refinement pairs and for scaling test-time compute.","marker":"Snell et al., 2024"},{"why":"Shows the template of providing a prior failed attempt and reference answer to generate a successful attempt.","marker":"Zelikman et al., 2022"},{"why":"Motivates the claim that alignment is needed for self-correction and that same-format refinement shows non-correcting behavior.","marker":"Kumar et al., 2024"},{"why":"Provides the DPO objective used in the preference-optimization stage.","marker":"Rafailov et al., 2024"},{"why":"Motivates the critic/reward-modeling approach used to select training pairs.","marker":"Lightman et al., 2023"},{"why":"Supplies the DOTS baseline that dynamically selects CoT versus PoT, which the paper compares against.","marker":"Yue et al., 2024"},{"why":"Supplies the SIaM PoT-only baseline and the PoT teacher model used for data generation.","marker":"Yu et al., 2024"}],"fun_headline_variants":["Preference optimization unlocks tool-assisted cross-reasoning fixes","7B model corrects 10x bigger model's reasoning with code after DPO","Cross-format refinement from CoT to PoT, but only via preference learning","Tool use lets small model improve larger model's chain-of-thought answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline depends on a critic model, trained on about 30K annotations, to judge whether chain-of-thought and code answers are correct; if that critic is biased toward one format, the refinement gains could come from matching the critic's preference rather than from truly fixing errors.","fun_headline_variants_meta":{"raw":{"variants":["Preference optimization unlocks tool-assisted cross-reasoning fixes","7B model corrects 10x bigger model's reasoning with code after DPO","Cross-format refinement from CoT to PoT, but only via preference learning","Tool use lets small model improve larger model's chain-of-thought answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000724,"raw_usage":{"total_tokens":3219,"prompt_tokens":892,"completion_tokens":2327,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":2258}},"tokens_in":508,"tokens_out":2327,"duration_ms":15746,"temperature":1.0,"reasoning_tokens":2258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T06:01:31.841925+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the critic's YES/NO labels during DPO training while keeping everything else fixed; if the refinement gain survives label noise, the critic is not the active mechanism, and if it collapses, the critic's judgment is doing the causal work.","supporting_citations":[],"review_version":1}