{"id":"33a53c72-ed8d-4f84-b7bf-63c753cf1d21","arxiv_id":"2504.15027","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Distilled Qwen2.5 models trained with multi-agent teacher data and top-10 logit fusion outperform their original checkpoints on instruction-following benchmarks.","lead":"This paper describes a practical recipe for making small Qwen2.5 language models that follow instructions better than the original checkpoints, using larger proprietary models as teachers. It is useful to a generalist because it shows a concrete way to get stronger small models for cheaper deployment, a central problem in running AI in production.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed superiority over original checkpoints is at risk from benchmark/training overlap and single-run scoring; the anomalous IFEval strict-prompt gain for the 1.5B model suggests format overfitting.","rationale":"The reader's weakest assumption is that the teacher-generated training data does not overlap with or bias AlpacaEval 2.0, MT-Bench, and IFEval, and that single-run scores are stable. I agree this is the load-bearing vulnerability. The paper provides no contamination analysis, no withheld benchmark, and no repeated runs; the released dataset link is absent from the manuscript, so the overlap cannot be checked from the text. The anomalous IFEval strict-prompt jump for the 1.5B model strengthens the concern beyond generic nervousness: it is exactly the signature of training on benchmark-style formatting constraints. The top-10-logit white-box approximation and the Figure 4/Table 1 inconsistencies are secondary; they do not threaten the central claim as directly as evaluation validity. Since the reader already conditioned acceptance on these issues, my read does not change the verdict.","tokens_in":11063,"tokens_out":6439,"duration_ms":61978,"concrete_test":"Audit the released processed dataset for overlap with AlpacaEval 2.0, MT-Bench, and IFEval prompts: after normalizing text, compute the maximum 8-gram and Rouge-L overlap between every benchmark instruction and every training/expanded/rewritten instruction. Then re-run the Table 1 evaluations with identical decoding settings for five independent runs and report per-model deltas with confidence intervals. If even one benchmark has a substantial fraction of prompts with near-duplicate training examples, or if the 95% CI for the Distil-vs-original delta includes zero for any model size, the central claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests entirely on Table 1, where DistilQwen2.5 models outperform their Qwen2.5 checkpoints. Those numbers come from public benchmarks whose prompts and formats are plausibly present in the training corpus. The corpus is built from OpenHermes 2.5 and Cleaned Alpaca plus teacher expansion/rewriting (Sec. 4.1, Sec. 3.1), sources known to contain instruction-following examples similar to AlpacaEval and MT-Bench; the rewriting agent can also reproduce IFEval-style format constraints. Moreover, AlpacaEval 2.0 and MT-Bench use GPT-4 as judge, and GPT-4 is one of the teacher models, so the evaluation rewards students that imitate teacher style. The 1.5B result is particularly suspicious: IFEval strict-prompt jumps from 40.11 to 73.02 while instruct-loose only moves 55.40 to 60.63, a pattern consistent with benchmark-specific overfitting rather than general instruction-following improvement. No deduplication against benchmark prompts, no held-out evaluation, and no repeated runs or error bars are reported. If training/evaluation overlap or run-to-run noise explains part of the gap, the paper's strongest claim is overstated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes the training and evaluation of DistilQwen2.5, a family of lightweight instruction-following models distilled from Qwen2.5. The pipeline has two stages: (i) black-box distillation via multi-agent data augmentation, where proprietary LLMs (Qwen-max, GPT-4/GPT-4o) act as agents to expand, rewrite, select, and verify instruction-response pairs; and (ii) efficient white-box distillation, where the teacher's logits are precomputed offline with top-10 truncation and the student minimizes a divergence over this truncated distribution after standard fine-tuning. The authors evaluate the 0.5B, 1.5B, 3B, and 7B models on AlpacaEval 2.0 (LC), MT-Bench, and IFEval, reporting that all distilled models outperform the original checkpoints, with the largest gains for the smallest models. Additional experiments analyze task-level MT-Bench scores, teacher-size scaling, and dataset-size scaling, and two industrial use cases (SQL completion and a cloud KD platform) are described.","tokens_in":11280,"tokens_out":7875,"duration_ms":60627,"significance":"The paper proposes a practical distillation pipeline combining black-box multi-agent data augmentation with efficient white-box top-10 logit fusion, and reports consistent improvements over the Qwen2.5 instruct checkpoints across four model sizes on three benchmarks. Strengths of the work include the release of the models and processed dataset, a clear ablation isolating the white-box fusion step (asterisk vs non-asterisk rows in Table 1), a measured 3-5x speedup for teacher logit generation, and a consistent trend of larger gains for smaller students. If the evaluation concerns are resolved, the industrial recipe is a useful contribution. However, the current evaluation lacks contamination control, variance estimation, and comparison with existing KD objectives, which limits the strength of the central claim.","major_comments":[{"comment":"The central claim of the paper (Abstract, Section 4.3) is that the distilled models are 'significantly stronger' than their original checkpoints, and the evidence is Table 1. However, the training corpus described in Section 4.1 includes OpenHermes 2.5 and Cleaned Alpaca, public instruction-tuning datasets whose prompts are likely to overlap with the evaluation benchmarks AlpacaEval 2.0 and MT-Bench. The paper reports no deduplication of training data against evaluation prompts, no contamination analysis, and no held-out evaluation set. Since the evaluators include GPT-4 (AlpacaEval 2.0 and MT-Bench use GPT-4 as judge) and one of the teacher models is GPT-4, a portion of the reported gain could reflect style imitation or format overfitting rather than general instruction-following improvement. The IFEval strict-prompt gain for the 1.5B model (40.11 to 73.02, while instruct-loose moves only 55.40 to 60.63) is a pattern consistent with benchmark-specific format overfitting. Please add a systematic overlap analysis (e.g., longest common substring or n-gram matching between training prompts and benchmark prompts) and, if possible, evaluate on a held-out set of instructions that are provably absent from the training data.","section":"Section 4.3 / Table 1"},{"comment":"Table 1 reports a single run per configuration. The differences between DistilQwen2.5 and the original checkpoints are often small on MT-Bench (e.g., 0.5B: 5.49 vs 5.71, a 0.22-point difference), and MT-Bench is known to be sensitive to decoding parameters and seed. Without repeated runs or bootstrap confidence intervals, the statistical significance of the improvements is unclear. Please report results over multiple seeds (or provide bootstrap intervals) for at least the central configurations (e.g., 1.5B and 7B), with the decoding hyperparameters fixed.","section":"Table 1"},{"comment":"Section 2 surveys existing white-box KD methods (MiniLLM, reverse-KL, FuseLLM), but the experimental section does not compare the proposed top-10 model fusion against any of these methods on the same backbone. Consequently, the contribution of the 'efficient model fusion' component over standard KD objectives is not demonstrated. Add a same-backbone comparison on at least one student size (e.g., 1.5B) between the proposed method and a standard KD baseline (e.g., MiniLLM's objective or a forward-KL baseline) under identical data and compute.","section":"Section 2 / Section 4.3"},{"comment":"Section 3.2: The assertion that 'the sum of the probabilities of the top-10 tokens is almost equal to 1' is a key justification for truncating teacher logits to K=10, but no empirical evidence is shown. Provide a plot of cumulative probability mass vs K for representative teacher outputs, and report the downstream performance of the student when trained with full-logit KD versus top-10 KD (the text mentions exploratory experiments but does not present them). This is needed to validate that the truncation is lossless for the distillation objective.","section":"Section 3.2"}],"minor_comments":[{"comment":"Section 4.4 is labeled 'Detailed Analysis' but is empty in the manuscript; please either remove the heading or supply the intended analysis.","section":"Section 4.4"},{"comment":"Figure 4 appears to list 'DistilQwen2.5-1.5B-Instruct' twice, with scores 13.69 and 20.93; the 20.93 entry is likely the 3B model (Table 1 reports 20.91 for the 3B).","section":"Figure 4"},{"comment":"Table 2: For the 1.5B model, the Writing score is unchanged (8.38) and STEM decreases (9.65 to 9.28); the text claims general improvement across tasks, so clarify or correct these entries.","section":"Table 2"},{"comment":"The MT-Bench citation should be to the original MT-Bench paper (Zheng et al., 2023), not to MT-Bench-101 (Bai et al., 2024), which is a different benchmark.","section":"References"},{"comment":"In Eq. (3), T denotes both the temperature and the teacher model; please use a distinct symbol such as tau for temperature to avoid ambiguity.","section":"Equation (3)"},{"comment":"The Limitations section would benefit from a sentence acknowledging that overlap between training data and evaluation benchmarks could affect the reported results.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"This manuscript reads like an industry technical report rather than a conventional research paper. The core engineering contribution (caching top-K teacher logits for efficient white-box KD) is potentially useful, but the evaluation is not yet robust enough for a research publication. The empty Section 4.4 and the incorrect MT-Bench citation are editorial problems that should be corrected in revision. I would be willing to review a revised version that addresses the contamination and variance issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short read: this is an industrial practice paper, not a new-principle paper. The genuinely useful pieces are the top-10 logit truncation for white-box KD (3-5x speedup in teacher logit generation, with no apparent quality loss in their tests) and the multi-agent black-box augmentation loop (expansion, CoT rewriting, selection, verification). The released DistilQwen2.5 checkpoints and processed dataset are a real artifact, though the manuscript omits HuggingFace URLs and training code.\n\nCredit where earned: Table 1 shows consistent gains across four model sizes and three benchmarks, with the largest relative gains on the smallest students. The method doesn't fit constants to the benchmarks, so the circularity burden is low. The limitations section is honest about teacher-bias propagation and domain generality.\n\nWhere it gets soft. All results are single-run, no error bars, no repeated trials. That alone would be okay for a systems paper, but the claim \"significantly stronger\" is exactly the kind that needs repeatability. More importantly, training and evaluation share surface area: OpenHermes 2.5 and Cleaned Alpaca resemble AlpacaEval and MT-Bench style prompts, and the rewriting agent can generate format-constrained examples like IFEval. Because AlpacaEval and MT-Bench are judged by GPT-4, one of the teachers, part of the gain may be style mimicry rather than general instruction-following. The 1.5B IFEval numbers are the tell: strict-prompt jumps from 40.11 to 73.02 while instruct-loose only moves 55.40 to 60.63. That pattern is consistent with format overfitting, not broad capability. I don't think this kills the paper—there is no claim of out-of-distribution generalization—but it should be probed with held-out benchmarks or a prompt-paraphrase test.\n\nTwo smaller warts. Figure 4 seems to list DistilQwen2.5-1.5B twice (20.93 and 13.69) and those don't match Table 1; likely a typo for the 3B, but confusing. Table 2 shows mixed fine-grained results: the 1.5B STEM score drops from 9.65 to 9.28, which the text glosses over. And there is no comparison against MiniLLM or similar KD methods on the same backbone, so the recipe's relative advantage over existing white-box KD is not established.\n\nWho this is for: practitioners deploying small instruct models and researchers working on cheap white-box KD. It deserves a serious referee. Send it out with a request for repeated runs, decontamination checks, and at least one held-out eval. I'd be skeptical of the headline numbers until then, but the engineering contribution is solid enough to engage with.","headline":"A practical industrial KD recipe with plausible gains and a useful top-10 logit trick, but the superiority-over-checkpoint claim needs robustness and decontamination checks before it can carry the paper.","tokens_in":11833,"tokens_out":4677,"would_cite":true,"duration_ms":37447,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that a two-stage distillation recipe—multi-agent black-box data augmentation followed by top-10-logit white-box fusion—makes smaller Qwen2.5 models beat their original instruct checkpoints across AlpacaEval 2.0…","keywords":["knowledge distillation","instruction following","large language models","model compression","multi-agent data augmentation","logit fusion","lightweight LLMs"],"falsifier":"Run an embedding- or n-gram-based overlap check between the augmented training set, including the in-house portions, and the prompts of AlpacaEval 2.0, MT-Bench, and IFEval; a high overlap rate would show the gain comes from memorization. Independently, repeat the evaluation of one distilled model, say the 1.5B version, five or more times with different seeds and sample orders; if the gap over the original checkpoint falls inside the run-to-run spread, the claimed improvement is not stable.","tokens_in":1824,"feed_emoji":"🤖","tokens_out":2474,"duration_ms":80236,"temperature":0.7,"pith_summary":"This paper claims a practical recipe for turning a family of large language models into smaller ones that follow instructions better than the checkpoints they were cut from. It argues that combining two kinds of knowledge distillation—multi-agent data augmentation with black-box teachers, then efficient white-box fusion using only the top ten token logits—produces students that outperform their original sizes on AlpacaEval 2.0, MT-Bench, and IFEval. The reported gains are largest for the smallest students, which matters because tiny models are the cheapest to deploy. If the recipe holds, distillation becomes a reliable upgrade for lightweight LLM deployment rather than a trade-off that accepts lower quality.","feed_headline":"Distilled Qwen2.5 models beat their originals at every size","feed_subtitle":"Two-stage teacher pipeline lifts instruction-following scores; gains are largest at 0.5B","key_machinery":"The load-bearing object is the top-10 logit fusion used in the white-box stage. The authors observe that the sum of the top-10 token probabilities in the teacher's distribution is almost 1, so they keep only the top ten logits, align student and teacher vocabulary indices, and minimize a divergence such as $\\mathrm{KL}$ over those ten entries after a temperature-scaled softmax. This reduces memory, storage, and reading time for teacher logits while preserving nearly all of the teacher's signal. The other half of the pipeline is the multi-agent controller: expansion, rewriting, selection, and verification agents that turn public instruction data into a larger, cleaner, CoT-rich set for black-box distillation. Together the two mechanisms let students inherit both task-level knowledge through data and token-level knowledge through logits.","core_discovery":"On its own terms, the paper's central discovery is that a distilled lightweight model can be stronger at instruction following than the original checkpoint it was derived from. Across 0.5B, 1.5B, 3B, and 7B sizes, the DistilQwen2.5 variants beat their Qwen2.5-Instruct counterparts on every reported metric, with the largest relative jumps at the smallest sizes (e.g., AlpacaEval 2.0 LC rises from 2.46 to 4.89 for the 0.5B model and from 6.69 to 13.69 for the 1.5B model). The two stages contribute separately: black-box KD supplies a higher-quality, CoT-reasoned training set, and white-box fusion adds further gains by matching top-10 token distributions. The authors also report that this white-box stage runs 3–5 times faster than the vanilla approach, and that a 3B student can nearly match a 7B teacher on SQL completion at 1.4× speed.","pith_inferences":["If top-10 concentration of token probability mass is common across large language models, top-K sparsification could also accelerate other logit-hungry applications such as ranking, reranking, and selective prediction, not just distillation.","The paper's observation that gains diminish when the teacher grows from 14B to 72B suggests a testable rule for budgeting: distilling from a medium-sized teacher may be the cost-effective choice for most student sizes, which would change how practitioners allocate teacher API calls.","The agent-based data augmentation is task-category-preserving, so the same pipeline could be reused as a data-engine service for domains beyond chat, such as summarization or tool-use instruction sets, by swapping the seed instructions."],"forward_implications":["Distillation can be used as a standard upgrade step: same-size students beat the original instruct checkpoints on all three reported benchmarks, so deployment teams do not have to sacrifice instruction-following quality when shrinking models.","The smallest students gain the most, which makes sub-1B models a viable target for edge and low-latency deployment rather than a last resort.","Top-10 logit fusion is efficient enough for industrial-scale white-box KD, since teacher logits can be precomputed offline with small storage and vocabulary mismatch handled by token alignment.","In a real SQL-completion task, a 3B student nearly matched a 7B teacher's adoption rate while cutting latency 1.4×, suggesting the pipeline transfers beyond generic instruction following.","White-box fusion adds measurable gains on top of black-box KD, so the two stages are complementary rather than redundant."],"supporting_citations":[{"why":"Establishes the foundational notion of distilling a teacher network's knowledge into a student, which motivates both the black-box data stage and the white-box logit stage.","marker":"Hinton et al., 2015"},{"why":"Shows that chain-of-thought and step-by-step distillation can make smaller models outperform larger ones, motivating the CoT rewriting used in the data augmentation pipeline.","marker":"Hsieh et al., 2023"},{"why":"Supplies the task-aware curriculum and task-balance selection heuristics that the selection agent uses to filter instruction-response pairs.","marker":"Yue et al., 2024b"},{"why":"Provides the token-alignment operation for merging models with mismatched vocabularies, which the white-box fusion stage relies on when teacher and student vocabularies differ.","marker":"Wan et al., 2024"},{"why":"Introduces the reverse-KLD white-box distillation objective that the paper's token-level divergence formulation builds on.","marker":"Gu et al., 2024"},{"why":"Defines AlpacaEval 2.0 length-controlled evaluation, the primary benchmark on which the distilled models are shown to beat the originals.","marker":"Dubois et al., 2024"},{"why":"Defines IFEval, the second benchmark used to argue for improved instruction-following ability.","marker":"Zhou et al., 2023"},{"why":"Defines MT-Bench, the third evaluation used in the performance comparison across task types and conversation settings.","marker":"Bai et al., 2024"}],"fun_headline_variants":["Distilled Qwen2.5 models beat originals at every size","Two-stage distillation lifts Qwen2.5 instruction following","Smaller Qwen2.5 distillations beat originals on every metric","Multi-teacher distillation: Qwen2.5 students surpass teachers","DistilQwen2.5: cheaper, faster, and stronger than the original"],"cache_read_input_tokens":13952,"weakest_assumption_plain":"The largest gain rests on the assumption that the three evaluation benchmarks are clean and that the teacher-generated training data does not overlap them; if contamination or run-to-run variance explains the gap, the superiority claim is overstated.","fun_headline_variants_meta":{"raw":{"variants":["Distilled Qwen2.5 models beat originals at every size","Two-stage distillation lifts Qwen2.5 instruction following","Smaller Qwen2.5 distillations beat originals on every metric","Multi-teacher distillation: Qwen2.5 students surpass teachers","DistilQwen2.5: cheaper, faster, and stronger than the original"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1313,"prompt_tokens":962,"completion_tokens":351,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":255}},"tokens_in":578,"tokens_out":351,"duration_ms":3554,"temperature":1.0,"reasoning_tokens":255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:34:38.122989+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an embedding- or n-gram-based overlap check between the augmented training set, including the in-house portions, and the prompts of AlpacaEval 2.0, MT-Bench, and IFEval; a high overlap rate would show the gain comes from memorization. Independently, repeat the evaluation of one distilled model, say the 1.5B version, five or more times with different seeds and sample orders; if the gap over the original checkpoint falls inside the run-to-run spread, the claimed improvement is not stable.","supporting_citations":[],"review_version":1}