{"id":"1560b76c-c8ea-493a-956e-a09aad602ac1","arxiv_id":"2507.23751","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CoT-Self-Instruct generates synthetic training instructions by prompting LLMs to reason about seed tasks and then filters them via answer-consistency or reward-model scores, outperforming existing datasets on reasoning and instruction-following benchmarks.","lead":"This paper introduces CoT-Self-Instruct, a way to have an AI model write its own training questions by first thinking step by step about example problems, then filtering the new questions for quality. The method improved reasoning and instruction-following benchmarks compared with existing human and synthetic datasets, which could lower the cost of building high-quality AI training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contamination check is missing: synthetic seeds come from s1k, so overlap with MATH500/AMC23/AIME24/GPQA-Diamond must be quantified before the 57.2% vs 44.6% headline is trusted.","rationale":"The reader's weakest_assumption is exactly the contamination/overlap risk: synthetic questions are generated from s1k seeds and evaluated on overlapping public benchmarks with no exclusion or overlap report. I agree with this diagnosis and with the CONDITIONAL verdict. The independent support in the paper (size-matched comparisons in Appendix Table 3, multiple generator/target ablations in Tables 4-10, and consistent trends across judges in Table 2) makes the method plausible, but it does not resolve the contamination question, because none of those ablations varies the evaluation set. The concrete test I propose directly targets the load-bearing assumption: it measures how much of the reported advantage disappears when overlapping training items are removed. If overlap is near zero, the headline should stand; if overlap is material, the comparison against s1k and OpenMathReasoning is compromised, and the paper would need either decontaminated results or a re-framed claim. I do not argue the paper is wrong; I argue the evidence as presented cannot distinguish a genuinely better data-generation method from a benchmark-overfitting artifact, and that distinction is exactly what the paper's central claim requires.","tokens_in":18574,"tokens_out":1866,"duration_ms":15442,"concrete_test":"Compute token-level and embedding-level overlap between the 2,926 Answer-Consistency-filtered CoT-Self-Instruct training examples and the 500+23+24+448 evaluation problems of MATH500, AMC23, AIME24, and GPQA-Diamond, using exact n-gram matching plus a thresholded embedding similarity (e.g., all-MiniLM-L6-v2 cosine > 0.9) followed by human inspection of the top matches. Report the percentage of training examples that duplicate or near-duplicate an evaluation question, then retrain with those examples removed. If the average accuracy on the four benchmarks drops by more than 2-3 points (roughly the gap between the 54.2% same-size result and the headline 57.2%, or between CoT-Self-Instruct and the next-best filter), the contamination concern lands and the headline claim needs qualification.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that CoT-Self-Instruct generates synthetic reasoning data that outperforms s1k and OpenMathReasoning when training Qwen3-4B-Base with GRPO (Table 1: 57.2% vs 44.6% and 47.5%). The method seeds generation with s1k instructions (Section 4.1) and evaluates on MATH500, AMC23, AIME24, and GPQA-Diamond (Section 5). The paper reports no check for overlap or near-duplicate contamination between the generated synthetic training set and these benchmarks. This is load-bearing because MATH500 and AIME24 are well-known public math benchmarks, and s1k itself was curated from public sources; a Qwen3-4B generator prompted with s1k seeds can reproduce benchmark-style problems or near-variants. If even a modest fraction of the 2,926 Answer-Consistency-filtered training problems overlap with evaluation items, the reported gains (e.g., AIME 24.6 vs 18.5, AMC 72.3 vs 51.3) are inflated and the comparison against s1k/OpenMathReasoning is no longer a fair test of data-generation quality. No deduplication, n-gram overlap analysis, embedding-similarity check, or manual audit is reported anywhere in the main text or appendix, and no code or data is released to allow readers to run such a check. The same gap applies to the instruction-following head of the paper: WildChat-derived seeds and AlpacaEval/Arena-Hard evaluation also receive no contamination analysis, although this risk is lower because those test sets are less likely to be exactly reproduced. The correct fix is not to assume contamination, but to measure it; the claim is conditional until that measurement exists.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoT-Self-Instruct, a two-stage synthetic data pipeline. In the first stage, an LLM is prompted with seed instructions and asked to reason about their common elements and complexity before generating a new instruction (and, for verifiable reasoning tasks, a target answer). In the second stage, generated data are filtered: Answer-Consistency for verifiable tasks, and RIP (Rejecting Instruction Preferences) for non-verifiable instruction-following tasks. The authors train Qwen3-4B-Base with GRPO on the generated reasoning data and Llama 3.1-8B-Instruct with DPO/online DPO on the instruction-following data. Their headline results are that CoT-Self-Instruct with Answer-Consistency filtering reaches 57.2% average accuracy on MATH500, AMC23, AIME24, and GPQA-Diamond, outperforming s1k (44.6%) and OpenMathReasoning (47.5%); and that CoT-Self-Instruct with RIP filtering reaches 54.7 on AlpacaEval 2 / Arena-Hard, outperforming human WildChat data (46.8).","tokens_in":19092,"tokens_out":5766,"duration_ms":51236,"significance":"If the central claims hold, the paper would be a useful empirical contribution: it provides extensive ablations, including size-matched comparisons (Tables 3 and 5), multiple filter variants, and different target-generation schemes (Tables 4-10), and it documents its prompts in the appendix. The use of external benchmarks and the same model family for generation and training is a standard self-training setup and is not circular by itself. However, two issues currently weaken the significance: the absence of any contamination analysis between the seed-driven synthetic data and the evaluation benchmarks, and the absence of variance or significance testing for the headline comparisons. The paper also contains no code or data release, which limits reproducibility and makes independent verification of the contamination question impossible.","major_comments":[{"comment":"The paper reports no contamination check between the generated synthetic training data and the evaluation benchmarks. The seed pool is s1k (Section 4.1), and s1k is curated from public reasoning benchmarks; evaluation is then run on MATH500, AMC23, AIME24, and GPQA-Diamond. Since the same model family is prompted with s1k seeds and asked to produce novel questions of comparable difficulty, exact or near-duplicate reproduction of benchmark questions is plausible. The headline gains (57.2 vs 44.6 for s1k, and 57.2 vs 47.5 for OpenMathReasoning) are only meaningful if the synthetic training questions are not duplicated in the test sets. Please add a contamination analysis: normalized exact-match removal, n-gram overlap statistics, embedding-similarity checks, and a manual audit of the nearest neighbors, and report the main results after excluding any overlapping examples. The same issue, though with lower a priori risk, applies to the WildChat-seeded instruction-following track evaluated on AlpacaEval 2 and Arena-Hard.","section":"Section 4.1 / Section 5.1, Table 1"},{"comment":"The headline comparisons are reported without any variance or significance testing. All numbers are point estimates of pass@1 averaged over 16 sampled rollouts, and no repeated GRPO training runs are reported. The abstract and Section 5.1 use the word \"significantly\" (e.g., \"significantly outperform\" and \"still significantly higher\"), but no standard errors, confidence intervals, or hypothesis tests support that wording. For example, the AIME difference between Answer-Consistency filtering (24.6) and Self-Consistency filtering (22.5) in Table 1 may be within sampling noise. Please provide standard errors or confidence intervals for the main comparisons, or per-problem matched-pair tests, so the reader can assess whether the ranking of methods is reliable.","section":"Section 5.1, Tables 1, 4, and 5"},{"comment":"The text states that 'in each case we see the same trend that CoT-Self-Instruct with Answer-Consistency is superior to Self-Instruct and other competing baselines' when targets are generated by other means (Appendix Tables 6, 7, and 8). This claim is not supported by those tables: Table 6 (majority-voted targets) and Table 7 (Best-of-K targets) contain no Answer-Consistency condition, and Table 8 (base-model targets) also contains no Answer-Consistency row. Either add the missing Answer-Consistency experiments for these target-generation settings or soften the claim to reflect only the conditions actually tested.","section":"Section 5.1, paragraph after Table 1"},{"comment":"The Answer-Consistency filter is defined as generating K responses and comparing the majority vote to the target answer, but the value of K is never reported in the main text or appendix. Similarly, the Self-Consistency threshold (SC >= 0.5) and RIP score threshold (RSc >= 0.5) are fixed without a sensitivity analysis. Without reporting K and without showing how results vary with the chosen thresholds, the method is not fully reproducible and the filter comparisons may depend on arbitrary parameter choices. Please state K explicitly and include a small sensitivity sweep over the filter thresholds.","section":"Section 3.2 / Section 4.1"}],"minor_comments":[{"comment":"The caption contains the typo 'AMIE24'; it should read 'AIME24'.","section":"Table 5 caption"},{"comment":"The word 'perfomance' should be 'performance'.","section":"Section 5.1, last paragraph"},{"comment":"The model name is written as 'LLama 3.3-70B-Instruct' and the training model as 'LLama 3.1-8B-Instruct'; please use consistent capitalization (Llama).","section":"Section 4.2, first paragraph"},{"comment":"The 10,000-example Answer-Consistency condition is listed in Table 1 but is not described in the experimental setup; please explain how this larger set was generated and filtered, and whether it is the same pipeline as the 5,000-example set.","section":"Table 1, Answer-Consistency (more data) row"},{"comment":"No code or data release is mentioned. Releasing the synthetic training sets and the filtering scripts would substantially help readers verify the contamination analysis and reproduce the results.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The missing contamination check is the key risk. If the authors can show low overlap between the s1k-seeded synthetic data and MATH500/AMC23/AIME24/GPQA-Diamond, the main reasoning claim may survive; if not, the 57.2% vs 44.6% comparison is invalid. The novelty of the method itself is modest (Self-Instruct with a CoT planning step plus existing filters), but the empirical scope and size-matched ablations give it value if the robustness issues are addressed. I would encourage the editor to ask for the contamination analysis and the significance testing before any acceptance decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful, well-run empirical study, and the headline claim is plausible but not fully earned until they measure overlap between their s1k-seeded synthetic data and the four benchmarks. That measurement is missing, and it is load-bearing.\n\nWhat's actually new: taking Self-Instruct and adding a CoT step that generates both the instruction and its target answer, then an Answer-Consistency filter that keeps samples where the majority-vote solution matches the generated target. The recipe is simple, and the paper does the right work to support it: ablations that isolate CoT versus no CoT, instruction-only versus instruction-plus-target, and three filtering methods; size-matched training sets; multiple seed models; and both math and instruction-following benchmarks. The gains over s1k and OpenMathReasoning (57.2 vs 44.6, 47.5) are large, and the component-wise ablations are consistent.\n\nThe soft spots. The missing contamination analysis is the one that matters. The synthetic instructions are seeded from s1k, which is itself a curated set of public benchmark-style problems, and the evaluation is on MATH500, AMC23, AIME24, GPQA-Diamond. There is no deduplication, no n-gram overlap, no embedding similarity check, no manual audit, and no release of data or code so a reviewer can do it themselves. This is not evidence of wrongdoing, but it is evidence of an unmeasured confound. The instruction-following head has the same gap, though the risk looks lower there because AlpacaEval and Arena-Hard prompts are not the kind of thing a few-shot generator seeds from WildChat would reproduce verbatim.\n\nSmaller weaknesses: no variance or significance across the reported numbers; the AlpacaEval/Arena-Hard runs use GPT-4-Turbo and GPT-4o judges rather than the standard judge versions, which makes cross-paper comparison approximate; and an odd raw chain-of-thought trace appears inside the manuscript text, which should be cleaned up.\n\nOverall, I believe the central claim holds up conditionally. The method is a straightforward extension of known components, the ablations are internally consistent, and the gains are big enough that even an inflated-but-positive result would still point to something useful. The paper is for people building synthetic post-training data for RLVR and instruction following. It deserves a serious referee. I'd send it out, and ask for the contamination measurement and for data/code release as revision conditions, not reject-and-resubmit conditions.","headline":"A practical synthetic-data recipe with strong ablations, but the s1k seeds make benchmark contamination a real open question.","tokens_in":19488,"tokens_out":2101,"would_cite":true,"duration_ms":19275,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that making an LLM reason and plan before writing new training prompts produces synthetic data that beats human-curated and existing synthetic reasoning datasets.","keywords":["synthetic data","chain-of-thought","self-instruct","data curation","mathematical reasoning","instruction following","reinforcement learning","answer consistency"],"falsifier":"An overlap audit would settle it: compute ROUGE-L or embedding similarity between every training example used in the experiments and every question in MATH500, AMC23, AIME24, GPQA-Diamond, AlpacaEval 2, and Arena-Hard, and if a substantial fraction of the solved training questions nearly duplicates benchmark items, the central claim fails. Re-running the same GRPO and DPO pipelines after deleting all examples above a similarity threshold would directly measure how much of the reported margin is contamination.","tokens_in":18412,"feed_emoji":"🧠","tokens_out":9995,"duration_ms":84672,"temperature":0.7,"pith_summary":"CoT-Self-Instruct claims that the quality of synthetic training data improves substantially when the generating model is made to reason and plan before it writes a new example, and then a filter removes the examples the model cannot reliably solve. The paper extends Self-Instruct, which prompts an LLM with seed tasks to produce new ones, by inserting a Chain-of-Thought stage that analyzes the seeds, brainstorms candidates, and self-evaluates them, generating both the question and its target answer in one reasoned pass. A curation step then rejects low-quality examples: for verifiable reasoning tasks it uses Answer-Consistency, where the target answer must match the majority vote of fresh samples, and for open-ended instruction following it uses a reward-model-based RIP filter. Trained with GRPO, Qwen3-4B-Base reaches 57.2% average accuracy on MATH500, AMC23, AIME24, and GPQA-Diamond, ahead of s1k's 44.6% and OpenMathReasoning's 47.5%, while Llama 3.1-8B-Instruct trained with DPO on the instruction-following data reaches 54.7 on AlpacaEval 2 and Arena-Hard, above human WildChat data at 46.8. The paper aims to show that reasoning-in-the-loop generation plus automatic filtering can replace ever-larger human-curated datasets.","feed_headline":"Reason-before-write prompts score 57.2% on hard reasoning benchmarks","feed_subtitle":"CoT-Self-Instruct outranks s1k, OpenMathReasoning, and human WildChat data on math and instruction-following tests","key_machinery":"The load-bearing object is the CoT-Self-Instruct prompt template: given two seed instructions, the model must list their common elements, write a plan for a new self-contained instruction of comparable quality and complexity, and produce the instruction in a strict bracketed format; for reasoning tasks it must also solve the new problem and emit a boxed final answer in the same pass. The second component is the curation filter. Answer-Consistency, the paper's new filter for verifiable tasks, samples K fresh solutions to each generated question, takes the majority vote, and deletes any example whose CoT-time target answer does not match that vote; the paper argues the CoT-time answer is more informative than a vote alone because it was produced together with the problem's construction. For non-verifiable tasks the paper adapts RIP, which judges instructions by the distribution of reward-model scores over sampled responses, modifying it to sample several instructions per few-shot prompt and keep the highest-scoring one, which avoids the topic shift a single global threshold causes.","core_discovery":"The central claim is that CoT-Self-Instruct produces training data good enough to beat both the seed instructions it starts from and the largest public synthetic reasoning datasets, in verifiable and non-verifiable settings alike. For reasoning, generating an instruction jointly with its answer through explicit step-by-step reasoning, then discarding any example whose target answer disagrees with the majority-vote answer from additional samples, yields 57.2% average accuracy on MATH500, AMC23, AIME24, and GPQA-Diamond when Qwen3-4B-Base is trained with GRPO, versus 44.6% for s1k and 47.5% for OpenMathReasoning; scaling the filtered set to 10,000 examples raises the average to 58.7%. For instruction following, generating prompts from domain-clustered seeds with a planning step, then keeping the single highest-reward-model-scored prompt per few-shot batch, gives a DPO-trained Llama 3.1-8B-Instruct 54.7 average on AlpacaEval 2 and Arena-Hard, beating human WildChat data at 46.8 and standard Self-Instruct at 47.4, and online-DPO training lifts the same data to 67.1 versus 63.1 for human data. The paper's stated conclusion is that reasoning-based creation plus curation creates high-quality synthetic instructions for RL training in both domains.","pith_inferences":["A direct test the paper does not run is a generation-evaluation overlap audit: measuring n-gram or embedding similarity between the synthetic training pool and the evaluation benchmarks would show how much of the reported margin is contamination-free.","A natural extension is to vary seed-pool composition and domain balance to test whether CoT-Self-Instruct amplifies seed quality or merely copies it, since the paper reports only the single s1k and WildChat seed configurations.","The Answer-Consistency filter treats the generator's majority vote as a correctness oracle; probing filter accuracy on seeds with known answers would reveal whether the filter can admit confidently wrong labels.","Because the same model family generates and trains, the gap over Self-Instruct may depend on the generator being only modestly stronger than the trainee; re-running the pipeline with a much stronger generator would test whether the advantage scales in a fully self-improving loop."],"forward_implications":["GRPO-trained models built on CoT-Self-Instruct data outperform models trained on the s1k seed set itself, meaning the pipeline extracts more training signal from 1,000 seed questions than the seeds provide directly.","Less filtered data beats more unfiltered data: filtering shrinks the training pool yet consistently improves accuracy, so data quality, not volume, is the dominant lever in this regime.","Answer-Consistency filtering outperforms Self-Consistency filtering, implying that the answer produced during question construction carries information beyond a fresh majority vote.","Adding more filtered synthetic data, from 5,000 to 10,000 examples, keeps improving results, suggesting the method scales where fixed public sets cannot.","Reasoning-based generation helps even non-reasoning tasks: longer CoT planning beats short CoT and no CoT for instruction-following, so the benefit is not specific to mathematics."],"supporting_citations":[{"why":"Supplies the Self-Instruct framework that CoT-Self-Instruct extends, and the baseline generation method it is compared against.","marker":"Wang et al., 2022a"},{"why":"Provides the s1k seed instructions for reasoning generation and the main training-data baseline it must beat.","marker":"Muennighoff et al., 2025"},{"why":"Provides OpenMathReasoning, the 10,000-question public dataset used as a training-data baseline.","marker":"Moshkov et al., 2025"},{"why":"Supplies the RIP reward-model-based filtering mechanism adapted for non-verifiable instruction-following tasks.","marker":"Yu et al., 2025"},{"why":"Provides the Self-Consistency filtering baseline that Answer-Consistency filtering is measured against for verifiable tasks.","marker":"Prasad et al., 2024"},{"why":"Supplies WildChat human instructions, the human-data baseline for instruction-following experiments.","marker":"Zhao et al., 2024"},{"why":"Provides GRPO, the reinforcement-learning algorithm used for all reasoning-task training runs.","marker":"Shao et al., 2024"},{"why":"Provides DPO, the training algorithm used for the instruction-following runs.","marker":"Rafailov et al., 2024"}],"fun_headline_variants":["CoT-Self-Instruct beats s1k and OpenMathReasoning on math benchmarks","Reason-first synthetic prompts outdo human and Self-Instruct baselines","57.2% on hard reasoning: CoT-Self-Instruct outranks s1k and OpenMath","Synthetic data from CoT planning beats human data on AlpacaEval","Plan-before-write synthetic training data wins on math and instruction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the generated training questions do not overlap with the evaluation questions, since the paper never filters or checks for contamination against MATH500, AMC23, AIME24, GPQA-Diamond, AlpacaEval, or Arena-Hard; if the seed-derived synthetic pool contains benchmark questions, the reported gains are inflated.","fun_headline_variants_meta":{"raw":{"variants":["CoT-Self-Instruct beats s1k and OpenMathReasoning on math benchmarks","Reason-first synthetic prompts outdo human and Self-Instruct baselines","57.2% on hard reasoning: CoT-Self-Instruct outranks s1k and OpenMath","Synthetic data from CoT planning beats human data on AlpacaEval","Plan-before-write synthetic training data wins on math and instruction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001245,"raw_usage":{"total_tokens":5132,"prompt_tokens":996,"completion_tokens":4136,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":4027}},"tokens_in":612,"tokens_out":4136,"duration_ms":28048,"temperature":1.0,"reasoning_tokens":4027,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:24:18.168762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An overlap audit would settle it: compute ROUGE-L or embedding similarity between every training example used in the experiments and every question in MATH500, AMC23, AIME24, GPQA-Diamond, AlpacaEval 2, and Arena-Hard, and if a substantial fraction of the solved training questions nearly duplicates benchmark items, the central claim fails. Re-running the same GRPO and DPO pipelines after deleting all examples above a similarity threshold would directly measure how much of the reported margin is contamination.","supporting_citations":[],"review_version":1}