{"id":"3a316aa7-38cd-4ad7-bf89-211e2b878408","arxiv_id":"2412.16964","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Enriched Instruction Tuning (EIT) uses GPT-4 to add planning and missing reasoning steps to human-annotated math solutions, and fine-tuning LLaMA-2 on this data yields 84.1% on GSM8K and 32.5% on MATH.","lead":"This paper proposes EIT, a way to enrich human-written math solutions with AI-generated plans and explanations, then fine-tune open-source LLMs on the enriched data. The method reports strong accuracy on GSM8K and MATH benchmarks without external tools, but missing controls leave the cause unclear.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal claim is confounded: no ablation trains on plain GPT-4-generated CoT matched for length, so ERP/ERS gains may reflect distillation or token-count effects rather than the enrichment design.","rationale":"I read the paper in good faith. The reported accuracies are plausible and the method is clearly described; the internal pipeline is consistent and the ablation table shows monotonic gains from adding ERP and ERS. However, the single load-bearing assumption behind the paper's causal narrative is that the enrichment structure itself, not the switch to GPT-4 as a data generator, explains the gains. The existing ablations compare enriched GPT-4-generated responses against human/standard-CoT baselines, but no condition isolates the effect of using GPT-4 while removing the ERP/ERS scaffolding. This is exactly the reader's weakest_assumption, and I agree with it. I also note two secondary limitations that do not change the main concern: Appendix B's 99.7% final-answer agreement is partly circular because the prompt instructs the model to preserve formulas and results, and Table 4's comparison to MetaMath uses different MetaMath variants (82.3 vs 80.4 on GSM8K) without making the matched configuration fully transparent. Neither is fatal, but both reinforce the need for a controlled baseline. The proposed concrete test would settle whether the ERP/ERS structure contributes beyond GPT-4 distillation and token-count effects. Since the reader already issued a CONDITIONAL verdict based on the same concern, my read does not move the verdict.","tokens_in":21430,"tokens_out":5601,"duration_ms":53632,"concrete_test":"Run an ablation with LLaMA-2-70B: construct a control set of 7.5k MATH + 7.5k GSM8K samples by prompting GPT-4-1106-preview with a simple 'solve step-by-step' instruction, no human answer, no ERP/ERS tags, and sample/select to match EITMath in average token count, final-answer correctness, and temperature. Fine-tune under identical hyperparameters (3 epochs, lr 2e-5, batch 32) and evaluate on MATH and GSM8K. If the control matches or exceeds the EIT rows in Table 3, the EIT-specific causal claim fails; if it is several points lower, the concern is resolved in the paper's favor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the ERP and ERS enrichment structure causes the gains in Table 3 (14.9→21.3 on MATH, 67.3→78.8 on GSM8K for LLaMA-2-70B). Every enriched condition is generated by GPT-4-1106-preview, while every baseline is human-annotated or standard CoT data. The design never includes the decisive control: a matched dataset where GPT-4 is prompted to write a detailed, self-contained chain of thought directly from the question, without being given the human answer and without the ERP/ERS scaffolding. Without this control, the improvements attributed to the enrichment structure are equally consistent with (a) distilling GPT-4's own reasoning ability and (b) the paper's own Fig. 3 (middle) observation that more tokens/lower perplexity alone improve accuracy. Appendix B's quality metric (99.7% final-answer agreement) is not independent evidence, because the prompt instructs GPT-4 not to change formulas or results, so high agreement is enforced by the prompt. The causal attribution to human-AI synergy rather than to the data generator is therefore unsupported by the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Enriched Instruction Tuning (EIT), a data-centric method that uses GPT-4 to enrich existing human-annotated math solutions with a high-level reasoning plan (ERP) and step-level elaborations (ERS), then fine-tunes open-source LLaMA-2 models on the resulting EITMath dataset. The authors report 84.1% on GSM8K and 32.5% on MATH for LLaMA-2-70B, surpassing several fine-tuning and prompting baselines and approaching tool-augmented systems. Ablations in Table 3 attribute gains to ERP and ERS separately and in combination, and additional experiments examine scaling, perplexity, and self-consistency.","tokens_in":21656,"tokens_out":5110,"duration_ms":46655,"significance":"If the causal attribution is supported, EIT is a practical and inexpensive route to improving mathematical reasoning in open-weight models without verifiers or search tools, and its claim that finer-grained reasoning trajectories are beneficial is a useful empirical insight. The paper has notable strengths: the prompts for ERP, ERS, and their ablations are given in full in Appendix H; the ablation isolates the contribution of each enrichment component; the benchmark comparisons cover multiple model sizes, MATH subtopics, and self-consistency settings. However, because the main experimental design lacks a control for GPT-4 as the data generator, the central claim that the enrichment structure itself causes the gains is not yet established.","major_comments":[{"comment":"The central causal claim is confounded. Every enriched condition in Table 3 is generated by GPT-4-1106-preview, while the baselines are either original human-annotated responses or standard CoT data; there is no control condition in which GPT-4 is prompted to write a plain, detailed chain of thought directly from the question without being given the human answer and without ERP/ERS scaffolding. Without such a matched control, the gains attributed to ERP and ERS could equally be due to distillation of GPT-4's own reasoning ability or to increased response length/token count, which the paper itself identifies as improving performance in Fig. 3 (middle). I recommend adding an ablation trained on plain GPT-4-generated CoT solutions matched for dataset size and response length.","section":"Section 3.1 and Table 3"},{"comment":"The reported 99.7% final-answer agreement is not independent evidence of data quality. The enrichment prompts explicitly instruct GPT-4 not to change the original pathways or modify formulas and results, so high final-answer agreement is enforced by the generation protocol rather than by the quality of the enrichment. To support the claim that EITMath is high-quality, the authors should measure step-level correctness or provide a human evaluation of the added reasoning steps, rather than relying on final-answer agreement with the source response.","section":"Appendix B and Examples H.1, H.2, H.4"},{"comment":"The claim that more fine-grained reasoning steps improve performance is conflated with response length and perplexity. The datasets compared in Fig. 3 (middle) differ in average token count, and the paper already shows that lower perplexity and more tokens correlate with better accuracy. Without a control that keeps the amount of new mathematical content fixed while varying granularity, or a control using plain verbose CoT of matched length, the observed gains do not specifically demonstrate that step-level granularity, rather than verbosity or token count, is the operative factor.","section":"Section 6.2 and Fig. 3 (middle)"},{"comment":"The scale-up from 7.5k to 70k examples is underspecified. The text says the larger dataset was constructed by sampling GPT-4's output distribution with different temperature coefficients, but it does not report the number of samples, the temperature values, whether the samples were filtered or deduplicated, or how the 70k examples are split between MATH and GSM8K. This matters because the scaling-law claim in Section 6.2 and the comparison to methods trained on much larger datasets depend on the construction of this 70k set.","section":"Section 4.2 and Table 2"}],"minor_comments":[{"comment":"The text states a 2.7% improvement over MetaMath-70B on MATH, but Table 4 lists two MetaMath-70B entries (26.6 and 29.8); the comparison should be identified explicitly, including which fine-tuning setup is used.","section":"Table 4 and Section 6.1"},{"comment":"The phrases '25.3% and 51.2% improvement' and '22.2% and 13.1% improvement' are ambiguous; the authors should state whether these are absolute percentage-point gains or relative improvements.","section":"Table 6 and Appendix C"},{"comment":"The perplexity is computed with an under-finetuned LLaMA-2-7B while the accuracy is reported for a fine-tuned LLaMA-2-70B; the evaluation protocol should be stated precisely, and the claim that lower perplexity 'leads to' better performance should be framed as a correlation.","section":"Fig. 3 (right)"},{"comment":"The sentence describing 'the work most closely related to ours is Auto-CoT' cites Jin et al. 2024, but that reference is titled 'The Impact of Reasoning Step Length on Large Language Models'; the citation appears mismatched and should be corrected.","section":"Appendix A"},{"comment":"The comparison of GPT-4's 99.7% accuracy with EIT prompts versus 71.5% without EIT prompts lacks details about the evaluation set, sampling temperature, and number of generations; these details should be reported.","section":"Appendix B and Section 6.4"}],"recommendation":"major_revision","confidential_remarks":"The reported results are plausible and the method is simple enough to be evaluated with additional controls. The decisive missing experiment is a matched plain-GPT-4-CoT fine-tuning condition; if the authors add that control and report the 70k construction details, the paper could be publishable. I also note that neither code nor the enriched dataset is released, which is a limitation for a data-centric contribution; the editor may wish to ask the authors to commit to releasing EITMath."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper describes a clean, practical recipe for upgrading human math solutions with GPT-4-generated plans and step elaborations, and the reported numbers are good. But the experiments never test the claim that the enrichment structure itself is what helps. Every enriched condition uses GPT-4, and all baselines are human or standard CoT data, so the gains could just be distillation or extra verbosity.\n\nWhat's actually new: the two-step prompting scheme (ERP then ERS) that takes existing human-annotated answers and asks GPT-4 to first write a high-level plan and then fill in missing reasoning steps, while explicitly forbidding changes to formulas/results. That's a sensible way to densify sparse reasoning traces, and the ablation shows both components add something. The paper also demonstrates that EIT combines well with MetaMath's question augmentation, and that scaling the enriched data helps rather than hurts—useful info.\n\nWhere it's soft: the missing control is the big one. To attribute the gains to ERP/ERS, you'd need a matched condition where GPT-4 is prompted to write a detailed, self-contained CoT directly from the question, without being handed the human answer or the planning scaffold. Without that, the improvement on MATH (14.9→21.3) and GSM8K (67.3→78.8) is equally explained by distillation from GPT-4 and by the paper's own observation that more tokens per response improve accuracy. The Appendix B quality check doesn't rescue it: 99.7% final-answer agreement is basically enforced by the prompt instruction, and the 71.5% comparison for GPT-4 without EIT is an apples-to-oranges comparison (solving from scratch vs. elaborating on a given answer). The 70k scale-up is also underspecified—what temperatures, how many samples, and does the improvement come from diversity or just more data?\n\nThat said, the core numbers are plausible and the method is easy to reproduce if they release the dataset. No code/data is provided, which makes the empirical claims harder to trust. The writing has occasional typos and some sloppy citations, but nothing that undermines the main idea.\n\nWho it's for: anyone working on synthetic data for math reasoning or on instruction tuning with model-generated explanations. It deserves a serious referee; the missing control is readily addressable and a revision with that experiment would be substantially stronger.","headline":"Plausible method for enriching math reasoning data, but the central causal claim is undermined by a missing control condition.","tokens_in":22198,"tokens_out":2457,"would_cite":false,"duration_ms":22083,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that enriching human-annotated math solutions with AI-generated reasoning plans and step-level elaborations enables open-source LLMs to match tool-augmented methods on GSM8K and MATH, achieving 84.1% and 32.5% accuracy…","keywords":["Enriched Instruction Tuning","system-2 reasoning","mathematical reasoning","fine-tuning","reasoning plan","reasoning step","GSM8K","MATH"],"falsifier":"Fine-tune LLaMA-2-70B on plain GPT-4-generated chain-of-thought completions of the same human-annotated answers, matched for length and detail but without the explicit 'planning' and 'solution' scaffolding; if this control achieves similar accuracy on GSM8K and MATH (within noise), the causal role of the ERP/ERS structure is falsified.","tokens_in":1795,"feed_emoji":"🧮","tokens_out":2684,"duration_ms":32541,"temperature":0.7,"pith_summary":"The paper argues that open-source large language models can be taught deliberate, step-by-step mathematical reasoning without external tools or symbolic verifiers, simply by fine-tuning them on instruction data that has been enriched with AI-generated reasoning plans and step-level elaborations. It introduces Enriched Instruction Tuning (EIT), which takes existing human-annotated question-answer pairs and uses GPT-4 to add a high-level plan and fill in missing reasoning contexts, producing smoother and more fine-grained reasoning trajectories. Training LLaMA-2-70B on this enriched data yields 84.1% accuracy on GSM8K and 32.5% on MATH, outperforming prior fine-tuning and prompting baselines and matching tool-augmented methods. The paper also finds that more fine-grained reasoning steps and more enriched data both improve performance, suggesting that data granularity as well as quantity matters.","feed_headline":"Richer reasoning data lifts math accuracy to 84.1%","feed_subtitle":"Fine-tuning LLaMA-2-70B on GPT-4-enriched solutions beats prompting baselines and matches tool-augmented models.","key_machinery":"The central mechanism is the two-stage prompting pipeline applied to existing human-annotated datasets: ERP generates a high-level plan that decomposes the instruction into simpler objectives, and ERS then expands the original response by filling in the missing contextual and logical steps under the guidance of that plan. The enriched responses preserve the original reasoning pathway and final answers (achieving 99.7% final-answer agreement with the original annotations) while being much more fine-grained, with lower perplexity than original or other augmented datasets. This enriched data is then used for standard supervised fine-tuning, improving the model's ability to produce system-2-like reasoning trajectories at inference time.","core_discovery":"The central claim is that the scarcity of deliberate multi-step reasoning data is a primary limiting factor for mathematical reasoning in LLMs, and that this scarcity can be addressed by enriching existing human-annotated datasets with AI-generated reasoning plans and step-level elaborations. The method, EIT, operates in two stages: Enriching with Reasoning Plan (ERP), where a privileged LLM decomposes the question into a sequence of lower-level objectives, and Enriching with Reasoning Step (ERS), where the same LLM fills in the reasoning contexts and implicit steps that human annotators omitted. The key insight is that human-annotated answers serve as 'meta-knowledge' that guides the LLM to generate more detailed and accurate reasoning trajectories, turning the task from open-ended generation into a fill-in-the-blank elaboration. Fine-tuning LLaMA-2-70B on this enriched data achieves 84.1% on GSM8K and 32.5% on MATH, surpassing state-of-the-art fine-tuning and prompting methods and matching tool-augmented approaches without using any external verifier.","pith_inferences":["The absence of a control condition where GPT-4 is prompted to write plain detailed chain-of-thought completions of the same original responses means the causal attribution of gains to the ERP/ERS scaffolding itself is not fully established; the gains could partly stem from GPT-4's superior generation quality rather than the plan-and-elaboration structure.","If the enrichment structure is what matters, then a similar approach could be applied to non-mathematical reasoning domains by replacing the math-specific prompts with domain-appropriate plan-and-elaboration templates, potentially improving commonsense or scientific reasoning as well.","The paper's findings suggest that the bottleneck for system-2 reasoning may be not just data quantity but the explicitness of intermediate reasoning steps; this could motivate interactive 'reasoning curriculum' where models are fine-tuned on progressively more detailed trajectories.","A direct testable extension would be to compare EIT against fine-tuning on plain GPT-4-generated expanded solutions of identical length, to isolate the effect of the plan-then-fill structure; if the plan structure adds no benefit, the result would collapse to a data-quality effect."],"forward_implications":["Fine-tuning on enriched reasoning trajectories can match or exceed the accuracy of tool-augmented methods (e.g., EIT-70B reaches 84.1% on GSM8K versus 83.9% for MathCoder and 84.3% for ToRA without self-consistency), suggesting that external tools may not be necessary for competitive mathematical reasoning.","More fine-grained reasoning steps in the training data improve test accuracy, indicating that the granularity of reasoning trajectories is as important as dataset size when training mathematical reasoners.","Combining EIT with question-augmentation methods such as MetaMath yields further gains (e.g., +3.6% on MATH and +6.9% on GSM8K over question augmentation alone), showing that response enrichment and question augmentation are complementary.","EIT improves performance even with small training sets (7.5k examples) and scales better than existing augmentation methods when more data is added, in contrast to the degradation observed with RFT-augmented data.","The lower perplexity of EITMath data correlates with higher downstream accuracy, supporting the claim that data simplicity and learnability are key drivers of reasoning performance."],"supporting_citations":[{"why":"Supplies the GSM8K benchmark, the human-annotated dataset whose responses EIT enriches.","marker":"Cobbe et al., 2021"},{"why":"Supplies the MATH benchmark and its original human-annotated solutions used for enrichment.","marker":"Hendrycks et al., 2021"},{"why":"Provides MetaMath, the primary question-augmentation baseline that EIT is compared against and combined with.","marker":"Yu et al., 2023"},{"why":"Defines chain-of-thought prompting, the standard prompting baseline and the contrast for EIT's fine-tuning approach.","marker":"Wei et al., 2022"},{"why":"Identifies GPT-4 as the privileged LLM used to generate enriched responses via ERP and ERS prompting.","marker":"Achiam et al., 2023"},{"why":"Provides MathCoder, a tool-augmented baseline that EIT is compared against.","marker":"Wang et al., 2023a"},{"why":"Provides ToRA, a tool-integrated reasoning baseline used in the comparisons and self-consistency analysis.","marker":"Gou et al., 2023"}],"fun_headline_variants":["AI-enriched step-by-step reasoning lifts LLM math to 84.1%","Human-guided enrichment powers 84.1% GSM8K accuracy","Fill-in reasoning steps boost math scores without verifier","Meta-knowledge from humans yields 84.1% math accuracy","Enriched instruction tuning beats baselines, hits 84.1%"],"cache_read_input_tokens":24320,"weakest_assumption_plain":"The load-bearing premise is that the observed accuracy gains come from the ERP and ERS enrichment structure itself, rather than from the simple fact that GPT-4 is a better generator of verbose solutions than the original annotators or standard CoT data; the paper does not include a control condition separating these factors.","fun_headline_variants_meta":{"raw":{"variants":["AI-enriched step-by-step reasoning lifts LLM math to 84.1%","Human-guided enrichment powers 84.1% GSM8K accuracy","Fill-in reasoning steps boost math scores without verifier","Meta-knowledge from humans yields 84.1% math accuracy","Enriched instruction tuning beats baselines, hits 84.1%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1594,"prompt_tokens":1036,"completion_tokens":558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":465}},"tokens_in":652,"tokens_out":558,"duration_ms":5430,"temperature":1.0,"reasoning_tokens":465,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:55:28.283376+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune LLaMA-2-70B on plain GPT-4-generated chain-of-thought completions of the same human-annotated answers, matched for length and detail but without the explicit 'planning' and 'solution' scaffolding; if this control achieves similar accuracy on GSM8K and MATH (within noise), the causal role of the ERP/ERS structure is falsified.","supporting_citations":[],"review_version":1}