{"id":"cb0bad4a-e1c9-407e-80ca-56e08a071475","arxiv_id":"2412.06136","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"AIDE generates task-relevant synthetic fine-tuning data from 10 seeds via attribute-guided multi-hop synthesis and residual connections, and reports gains over gold-data fine-tuning on several benchmarks.","lead":"AIDE is a data expansion method that turns 10 example questions into about 3,000 synthetic training examples for fine-tuning large language models, using attribute extraction, personas, and residual connections. The authors report that models fine-tuned on AIDE data beat models fine-tuned on human-curated data on several standard benchmarks, though the comparison has important caveats.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation of BIG-Bench and TruthfulQA uses test-set questions as seeds, so the claimed gains may reflect test-set leakage rather than synthesis quality.","rationale":"I agree with the reader that the central empirical claim is not yet fully supported, but I identify a more direct and more serious version of the contamination concern. The reader's weakest_assumption concerned Claude Sonnet 3.5 possibly memorizing public benchmark test sets. My concern is that for BIG-Bench and TruthfulQA—the tasks that drive the headline 'over 30% vs SOTA' claim—the 10 seeds themselves appear to be sampled from the test sets, because those benchmarks have no official training splits. This is a circular evaluation that does not require any assumption about a closed model's training data. If the seeds are test questions, then the synthetic data is generated from the evaluation distribution, and the fine-tuned model is tested on near-duplicates of its training data. This would inflate AIDE's accuracy on exactly the tasks used to support the abstract's claims. I do not allege any dishonesty; the paper simply does not state how seeds were selected for benchmarks without training sets, and Appendix E suggests they came from the Hugging Face datasets, which for these benchmarks are test sets. The proposed concrete test—rerunning with a disjoint held-out seed split and reporting overlap—would settle the issue. If the improvements survive, the method is validated; if they disappear, the main empirical contribution is an artifact. Given this, I keep the reader's CONDITIONAL verdict, but the condition should explicitly require disjoint seed/evaluation splits for all benchmarks.","tokens_in":18237,"tokens_out":7166,"duration_ms":68383,"concrete_test":"Re-run the BIG-Bench and TruthfulQA experiments in Tables 2 and 3 with seeds sampled from a held-out subset that is disjoint from the evaluation set: for each task, hold out 10 test questions as seeds, remove them from the evaluation set, generate synthetic data with AIDE, fine-tune, and report zero-shot accuracy on the remaining test questions. If AIDE's advantage over gold and Evol-Instruct persists under this disjoint split, the concern is resolved; if it collapses, the reported gains are partly an artifact of training on test-set seeds. Additionally, report exact and near-duplicate overlap between the generated synthetic data and the full test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's own setup makes the evaluation circular for two of the benchmark families. Appendix E says: 'Since BIG-Bench lacks a training set, we sample 10 task-related seed data points per task from Hugging Face datasets to generate synthetic data.' Appendix H confirms the five BIG-Bench tasks have only test sets (60, 153, 492, 7,688, and 1,000 points). Thus the 10 seeds used to synthesize data for Table 3 are drawn from the same test questions on which zero-shot accuracy is reported. Table 8 likewise lists TruthfulQA as 817 test questions with no training split, yet Table 2 reports AIDE results on TruthfulQA with 10 seeds and also reports a 'Gold training data' baseline whose provenance is unexplained. If seeds are test questions, the synthetic training data is generated from the evaluation distribution, and the fine-tuned model is evaluated on a test set whose near-duplicates it has seen. This inflates exactly the numbers that support the headline claims: the BIG-Bench comparison vs Evol-Instruct (>30% relative) and the TruthfulQA comparison vs gold (>15%). The concern does not depend on speculation about Claude's training data; it is visible in the paper's own described data splits. No overlap check between synthetic data and test questions is reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AIDE (Attribute-guided MultI-hop Data Expansion), a framework that synthesizes task-specific fine-tuning data from a small number of seed examples. AIDE extracts topic/attribute triplets from seeds, uses these to guide iterative multi-hop synthesis, injects personas for diversity, and applies residual connections to limit topical drift at larger hop depths. The empirical study fine-tunes Mistral-7B, Llama-3.1-8B, and Llama-3.2-3B on AIDE-generated data and compares against fine-tuning on human-curated gold data and against SOTA synthesis methods such as Evol-Instruct, Prompt2Model, and DataTune. The main claimed results are that AIDE beats gold-data fine-tuning by small margins on average, and beats SOTA synthesis methods by over 30% on BIG-Bench.","tokens_in":18495,"tokens_out":2987,"duration_ms":31513,"significance":"If the empirical claims are sound, AIDE would be a useful contribution: it is a clearly described, modular synthesis recipe with released code, and the residual-connection idea for preserving relevance at increasing hop depth is plausible and well-motivated. The paper also ships concrete prompts and an ablation study, which are valuable for reproducibility. However, the current experimental design does not yet establish the headline claims. The BIG-Bench and TruthfulQA evaluations appear to use test-set questions as synthesis seeds, the Evol-Instruct comparison is not a controlled same-budget comparison, and gold-data training sizes are not matched. These issues directly affect the main quantitative conclusions, so the paper needs substantial experimental revision before its central claims can be accepted.","major_comments":[{"comment":"The evaluation setup appears to use test-set questions as seeds for BIG-Bench and TruthfulQA, making the comparison circular for those benchmarks. Appendix E states that \"Since BIG-Bench lacks a training set, we sample 10 task-related seed data points per task from Hugging Face datasets to generate synthetic data,\" while Appendix H reports that the five selected BIG-Bench tasks consist only of test sets (60, 153, 492, 7,688, and 1,000 points). Appendix H similarly lists TruthfulQA as 817 test questions with no training split, yet Table 2 reports TruthfulQA results using 10 seeds and a \"Gold training data\" baseline of unexplained provenance. If the 10 seeds are drawn from the test sets, then the synthetic training data is generated from the evaluation distribution, and fine-tuning on near-duplicates of the test questions can inflate accuracy. This would directly inflate the BIG-Bench gains in Table 3 and the TruthfulQA gains in Table 2. The authors must either clarify the exact source of the seeds (including the specific Hugging Face dataset splits) or rerun the BIG-Bench and TruthfulQA evaluations with seeds drawn from a genuinely disjoint training source, and report an overlap check between synthetic data and test questions.","section":"Appendix E / Appendix H / Tables 2 and 3"},{"comment":"The comparison against Evol-Instruct is not a controlled experiment. Table 3 compares Mistral-7B fine-tuned with AIDE data against \"Mistral-7B with Evol-Instruct,\" which, according to Section 4.1 and footnote 1, is the external WizardLM-2-7B model fine-tuned on 250K Evol-Instruct examples. This differs from AIDE in three confounding ways: the amount of training data (250K versus roughly 3K), the fine-tuning recipe, and possibly the base model version. As a result, the claimed \"over 30%\" improvement over SOTA synthesis methods conflates data quantity, training budget, and synthesis quality. The authors should run Evol-Instruct on the same 10 seeds, the same data budget, and the same LoRA fine-tuning setup as AIDE, and report the variance across multiple runs.","section":"Table 3 / Section 4.1 / Appendix E"},{"comment":"The gold-data baselines in Table 2 are not matched in data quantity. For MedQA, Appendix H reports a 10,178-point training set, while AIDE synthesizes 2,173 points; for FinBen, the gold training set is 804 points versus 893 synthetic points; for MMLU, ARC-Challenge, and TruthfulQA, the paper does not report how many gold training examples were used per task. Only GSM8K is explicitly matched at 3.2K. The claim that AIDE \"surpasses models fine-tuned on human-curated data\" is therefore misleading if the gold models are trained on substantially larger or smaller datasets. The authors should either match the number of training examples across conditions or explicitly report the gold training sizes and interpret the comparison accordingly.","section":"Table 2 / Section 4.2"},{"comment":"The relevance and diversity analyses in Section 4.4 are partially circular because Claude Sonnet 3.5 both generates the synthetic data and serves as the grader for relevance and diversity. Section 4.4 and Appendix N report scores assigned by Claude to its own outputs, which does not provide an independent quality check. The self-reflection filter in Appendix E also uses the same model to grade and rewrite. The authors should add either human evaluation, a second independent LLM, or a quantitative overlap metric on a held-out set to support RQ3. This concern is secondary to the leakage issue but should still be addressed before the quality claims are accepted.","section":"Section 4.4 / Appendix N"}],"minor_comments":[{"comment":"The sentence \"We highlight the best performance and the base performance is in Table 3\" is incomplete and unclear; please state explicitly which row is the base model and which number should be compared.","section":"Table 4"},{"comment":"The text says the synthetic data clusters are \"largely disjoint\" from real data in the Code task but then says the synthetic data \"covers all real test data\" from two tasks; these statements are contradictory and need clarification.","section":"Figure 11 / Appendix G.2"},{"comment":"There is a typo in the x-axis label: \"Mistra-7B\" should be \"Mistral-7B\".","section":"Figure 4"},{"comment":"The residual-connection experiment on LegalBench is not listed in the main benchmark table or described in Appendix H; please add the dataset statistics and explain why this task is used only in the ablation.","section":"Section 4.3 / Figure 4"},{"comment":"The claim that AIDE achieves \"comparable complexity\" to Evol-Instruct is based on a visual histogram comparison; please provide numerical scores, sample sizes, and a statistical comparison.","section":"Appendix G.1"}],"recommendation":"major_revision","confidential_remarks":"The test-set leakage concern is the most serious issue and should be verified first. If the BIG-Bench and TruthfulQA seeds are indeed drawn from test sets, the headline percentages in the abstract and Section 4.2 would need to be substantially revised or withdrawn for those benchmarks. The Evol-Instruct comparison is also not a like-for-like evaluation. The paper's core synthesis framework is interesting, but the current empirical evidence is not yet sufficient to support the claimed margins."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper has a genuinely new data-synthesis recipe, but the main empirical claim is undermined by a data-split problem visible in the paper itself. On BIG-Bench and TruthfulQA, the 10 seeds are sampled from the test sets. Appendix E says BIG-Bench lacks a training set; Appendix H confirms those tasks only have test sets, and TruthfulQA is likewise just 817 test questions with no training split. So the synthetic training data is generated from the exact questions the model is later evaluated on. That inflates the headline \"surpasses gold\" and \"over 30% vs Evol-Instruct\" numbers, and no overlap check is reported. This is not speculation about Claude's training data; it is in the paper's own data splits.\n\nWhat is new and worth credit: AIDE combines attribute-triplet extraction, persona retrieval, and multi-hop recursive expansion with a residual connection that anchors deep hops to the original seed. That combination is not in Evol-Instruct, ProGen, ZeroGen, DataTune, or Self-Guide. The residual connection intuition is reasonable, and the ablations in Table 4 show each component contributes. The code is released, and the method is described clearly.\n\nThe soft spots beyond the split problem: the Evol-Instruct baseline is an external WizardLM-2-7B model fine-tuned on 250K examples, not a same-budget, same-seed run, so the >30% comparison is apples to oranges. Gold-data comparisons are not size-matched except for GSM8K. There are no error bars or repeated runs for the main tables; \"10 seeds\" refers to 10 seed data points per task, not 10 random trials. The quality analysis uses Claude both to generate and to grade the data, which is a minor circularity.\n\nBottom line: the framework is plausible and may well work, but the evidence as presented does not support the headline claims. A serious referee would need to see experiments redone with proper train/test separation, matched budgets, repeated seeds, and a contamination check. This deserves a reading-group slot because it is a clear example of a subtle leak that can pass a quick review. I would not cite it in its current form. Overall: send to review, but expect major revision.","headline":"A genuinely novel data-synthesis recipe is undercut by a test-set seeding leak for BIG-Bench and TruthfulQA; the framework deserves a redo, not a citation yet.","tokens_in":19033,"tokens_out":2774,"would_cite":false,"duration_ms":24337,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"From ten seed examples, AIDE synthesizes thousands of training points that fine-tune models to outperform human-curated gold data in the paper's experiments.","keywords":["data synthesis","multi-hop expansion","knowledge triplets","attribute-guided generation","task-specific fine-tuning","LLM fine-tuning","persona-based diversity","residual connections"],"falsifier":"Run AIDE with an open-source synthesizer whose pretraining corpus is documented to exclude MMLU, ARC-Challenge, TruthfulQA, MedQA, and BIG-Bench test sets, then compare fine-tuned accuracy against gold-data fine-tuning. If the reported average improvements (over 6% for Mistral-7B vs. gold, over 30% vs. evolution-based synthesis) shrink or vanish, the original result was partly an artifact of test-set leakage; if they persist, the method itself carries the gain.","tokens_in":18030,"feed_emoji":"🧩","tokens_out":8930,"duration_ms":78473,"temperature":0.7,"pith_summary":"The paper claims that task-specific fine-tuning data can be synthesized from as few as ten seed examples by repeatedly branching each example into new ones, guided by the topic and knowledge attributes of the seed and by a retrieved persona. The expansion runs for multiple hops, with each generated example becoming the seed for the next round, and a residual connection re-injects the original seed to keep deep hops on topic. In zero-shot evaluation, Mistral-7B fine-tuned on AIDE-generated data improves by over 6% relative to fine-tuning on human-curated gold data and by over 30% relative to existing data synthesis methods such as Evol-Instruct; Llama-3.1-8B and Llama-3.2-3B also match or beat gold-data fine-tuning. The paper concludes that with only a handful of labeled examples, practitioners can generate abundant, relevant, and diverse training data for fine-tuning. This matters because collecting task-specific data is often the main bottleneck in applying LLMs to new domains.","feed_headline":"Synthetic data from 10 seeds beats human-curated fine-tuning data","feed_subtitle":"AIDE's multi-hop expansion lifts Mistral-7B by over 30% versus leading synthesis baselines, the paper reports.","key_machinery":"The central object is the multi-hop synthesis tree with attribute-based control. For each data point $X_i^{(K)}$, an LLM extractor produces a representation $A_i^{(K)} = \\{\\langle t, r, a\\rangle\\}$ containing one topic and three related knowledge attributes. A synthesizer LLM then generates new data from a triplet plus a predefined operation (concretizing, adding constraints, adding reasoning), task demonstrations, and—for diversity—a persona retrieved by topic similarity from a persona hub. The process repeats for $K$ hops, and a residual connection feeds the original seed $X_i^{(0)}$ into the prompt for depths $d \\le L$ to keep generated data on task. A self-reflection grader scores each generated point for relevance and diversity, filtering or rewriting low-scoring data.","core_discovery":"The paper's central claim is that structured multi-hop expansion—not the volume of seed data—is what unlocks data synthesis for task-specific fine-tuning. AIDE represents each data point as knowledge triplets $\\langle t, r, a\\rangle$ (topic, relation, attribute) and uses those triplets, together with task demonstrations, predefined operations, and retrieved personas, to synthesize new questions. Each new question seeds the next hop, so ten seeds grow into roughly three thousand training points. A residual connection from the original seed into prompts up to depth $L$ prevents the topic drift that otherwise appears at deeper hops. Fine-tuning Mistral-7B, Llama-3.1-8B, and Llama-3.2-3B on AIDE data yields average relative improvements of about 6%, 0.7%, and 1.5% over gold-data fine-tuning across the evaluated benchmarks, and on BIG-Bench zero-shot tasks AIDE-trained Mistral-7B substantially outperforms Evol-Instruct, DataTune, and Prompt2Model.","pith_inferences":["If the result holds, the practical recipe for a new task becomes: gather roughly ten representative examples, let a capable synthesizer branch them under attribute and persona guidance, and fine-tune; specialist models could then be built without collecting large labeled datasets.","Because the synthesizer is a closed model trained on unknown web data and the seeds are public, a contamination check would settle how much of the advantage over gold data comes from the method rather than from test-set leakage; this is the natural next experiment.","The residual connection idea likely transfers to other recursive generation settings, such as iterative self-improvement or tree-of-thought data expansion, where maintaining a fixed anchor while branching is a general need.","The experiments use small test subsets for some tasks, so a larger-scale sweep with full benchmarks and multiple random seed sets would show whether the reported margins are stable."],"forward_implications":["Fine-tuning from ten seeds can match or beat fine-tuning on full human-curated training sets, which would remove the need for large labeled collections in new task domains.","Because Evol-Instruct used 250K synthetic points and AIDE used about 3K per task while still winning on BIG-Bench zero-shot tasks, task-relevance control can matter more than raw synthesis volume.","Residual connections make deeper multi-hop expansion usable: increasing hop depth $K$ adds data and can improve accuracy, while residual depth $L$ tunes how tightly the generated data stays anchored to the original seeds.","Persona-guided synthesis injects diversity without sacrificing task relevance, addressing the diversity–relevance trade-off that prior methods struggle with.","Embedding analysis suggests AIDE's synthetic data covers the target task's distribution even for benchmarks without official training sets, implying the method could be applied to tasks that currently have no training data at all."],"supporting_citations":[{"why":"Supplies the persona hub and personas used to diversify the synthesized data.","marker":"Chan et al., 2024"},{"why":"Evol-Instruct baseline and source of the complexity-increasing operations used during synthesis.","marker":"Xu et al., 2024"},{"why":"DataTune baseline, representative of synthesis methods that require large seed datasets and transform existing data.","marker":"Gandhi et al., 2024"},{"why":"Prompt2Model baseline, a task-specific data synthesis method from candidate datasets.","marker":"Viswanathan et al., 2023"},{"why":"Self-reflection technique used to grade and refine synthetic data.","marker":"Madaan et al., 2023"},{"why":"Provides empirical support that diverse data improves fine-tuning, motivating persona-guided synthesis.","marker":"Song et al., 2024"}],"fun_headline_variants":["From 10 seeds, AIDE synthesizes data that beats human-curated sets","Multi-hop expansion turns 10 seeds into top fine-tuning data","Attribute-guided synthesis lifts LLM fine-tuning beyond human data","10 seeds beat curated datasets via guided multi-hop expansion","Residual connections keep multi-hop synthesis on task, beating gold data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparisons to gold data assume the LLM used to generate synthetic data has not memorized the public test questions of MMLU, ARC, TruthfulQA, MedQA, or BIG-Bench, because the seeds come from public datasets and no contamination check is reported.","fun_headline_variants_meta":{"raw":{"variants":["From 10 seeds, AIDE synthesizes data that beats human-curated sets","Multi-hop expansion turns 10 seeds into top fine-tuning data","Attribute-guided synthesis lifts LLM fine-tuning beyond human data","10 seeds beat curated datasets via guided multi-hop expansion","Residual connections keep multi-hop synthesis on task, beating gold data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000917,"raw_usage":{"total_tokens":3957,"prompt_tokens":991,"completion_tokens":2966,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":2878}},"tokens_in":607,"tokens_out":2966,"duration_ms":21788,"temperature":1.0,"reasoning_tokens":2878,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:58:48.269633+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AIDE with an open-source synthesizer whose pretraining corpus is documented to exclude MMLU, ARC-Challenge, TruthfulQA, MedQA, and BIG-Bench test sets, then compare fine-tuned accuracy against gold-data fine-tuning. If the reported average improvements (over 6% for Mistral-7B vs. gold, over 30% vs. evolution-based synthesis) shrink or vanish, the original result was partly an artifact of test-set leakage; if they persist, the method itself carries the gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prompt2Model baseline, a task-specific data synthesis method from candidate datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Self-reflection technique used to grade and refine synthetic data."}],"review_version":1}