{"id":"c10f3b80-bb47-4bed-b674-0a72c733cc0f","arxiv_id":"2411.19946","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"DELT achieves state-of-the-art dataset distillation accuracy by giving different synthetic images different optimization budgets and initializing them from teacher-ranked real patches.","lead":"This paper introduces a training trick called DELT for creating compact image training sets. It gives each synthetic image a different amount of refinement, making the set more varied, and reports accuracy gains of 2 to 5 points with up to 39 percent less generation time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fixed EarlyLate schedule does not transfer: on ImageNet-100 IPC10 DELT trails RDED by 7.8-11.5 points across all three backbones, yet the paper claims 13/15 SOTA and 2-5% average gains; the >5% diversity increase is also never numerically reported.","rationale":"The reader's weakest-assumption analysis already targeted the fixed-schedule transfer risk and the teacher-probability initialization. My stress-test confirms the transfer concern and sharpens it into an internal counterexample: ImageNet-100 IPC=10 is not an exotic setting, and the reported deficits are large and consistent across all three backbones. This is more decisive than a hypothetical transfer worry because it is present in the paper's own tables. The diversity claim is also unsupported numerically, but accuracy is the primary endpoint, so the schedule-transfer issue is the load-bearing one. I keep the verdict at CONDITIONAL rather than moving to REJECT because the ImageNet-1K results are strong, the code is released, and the failures may be fixable by per-dataset schedule selection. The proposed grid test is cheap and would settle whether the fixed schedule is a genuine property of EarlyLate or a dataset-specific tuning artifact.","tokens_in":16034,"tokens_out":6869,"duration_ms":64314,"concrete_test":"Run a fixed protocol on ImageNet-100 IPC=10: sweep MI in {2,000, 4,000} and RI in {250, 500, 1,000} using the paper's own recovery and validation settings, and evaluate ResNet-18 and ResNet-101 with the same post-training recipe. If no grid point reaches or exceeds RDED's 36.0/33.9, the single fixed schedule does not transfer and the reported average gains are dataset-specific. If some grid point does, report it and re-state the headline margins using that schedule; the current fixed-schedule numbers in Table 1 would then be an artifact of hyperparameter choice rather than a property of EarlyLate. In the same run, compute the Fig. 2 cosine-similarity diversity values in a table, including RDED, so the '>5% diversity' claim can be checked.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: (i) EarlyLate increases intra-class diversity, and (ii) this yields consistent 2-5 point accuracy gains. Part (i) is asserted without a number: the abstract says 'increasing diversity per class by more than 5%', but the only supporting evidence is Fig. 2's cosine-similarity curves, which omit RDED and report no numeric value. Part (ii) is contradicted by the paper's own Table 1. On ImageNet-100 with IPC=10, DELT scores 28.2 vs RDED 36.0 (ResNet-18), 22.4 vs 33.9 (ResNet-101), and 15.8 vs 23.6 (MobileNet-V2). These are not edge cases: they are three backbones on a benchmark the paper claims to cover, under the same MI=4,000/RI=500 schedule tuned on ImageNet-1K. The paper's '13 out of 15 configurations' tally appears to count only favorable rows, and its '2~5% on average' claim is not robust to these losses. The load-bearing assumption is that a single ImageNet-1K-tuned schedule creates useful complementary information everywhere; Table 1 already provides a counterexample on ImageNet-100, so the general claim is premature until this is explained or the schedule is shown to be per-dataset tunable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DELT, a simple diversity-driven EarlyLate training scheme for dataset distillation in the batch-to-global matching paradigm. The method initializes synthetic images from teacher-ranked real image patches and then optimizes different IPC subsets with progressively fewer iterations (MI = 4,000 for the earliest batch, RI = 500 for later batches), concatenating them into a single synthetic dataset. The authors claim this increases intra-class diversity by more than 5%, improves accuracy by 2 to 5 percentage points on average over prior state-of-the-art methods, and reduces synthesis time by up to 39.3%. Experiments are reported on CIFAR-10, Tiny-ImageNet, ImageNet-100, ImageNette, and ImageNet-1K across several backbones, with the headline result of 66.1% on ImageNet-1K IPC=50 with ResNet-101, outperforming RDED by 4.9%.","tokens_in":16420,"tokens_out":4349,"duration_ms":38038,"significance":"If the reported gains are robust, DELT is a useful contribution to large-scale dataset distillation: it is conceptually simple, orthogonal to existing diversity-enhancement methods, and it also cuts synthesis compute, which is a practical concern in the field. The paper's strengths include extensive comparisons with error bars across many datasets, IPC settings, and backbones; ablations that isolate the EarlyLate effect (Tables 4c, 4d, 9); a computational analysis with actual wall-clock times (Table 6); and a public code release. The main significance is the demonstration that a variable-iteration schedule can improve distilled-data diversity without additional models or matching signals. However, the significance is diminished by the paper's unsupported quantitative diversity claim and by clear counterexamples in its own Table 1 where DELT loses to RDED by large margins, so the central claim of consistent 2-5% gains is not established as stated.","major_comments":[{"comment":"The abstract's claim of 'increasing diversity per class by more than 5%' is not supported by any reported numerical diversity metric. The only evidence in Fig. 2 left is a cosine-similarity curve for SRe2L, CDA, and DELT, which omits the main competitor RDED and provides no numeric values, error bars, or protocol description. Please report the diversity metric (e.g., average pairwise intra-class cosine similarity or variance) with numbers and include RDED, or remove the quantitative '>5%' claim from the abstract.","section":"Abstract and Section 1, Fig. 2"},{"comment":"The paper claims in Section 4.1 to establish state-of-the-art in 13 out of 15 configurations, and the abstract claims average gains of 2-5%, but Table 1 shows DELT trailing RDED by 7.8 points on ResNet-18 (28.2 vs 36.0), 11.5 points on ResNet-101 (22.4 vs 33.9), and 7.8 points on MobileNet-V2 (15.8 vs 23.6) for ImageNet-100 with IPC=10. These are not edge cases; they are three backbone evaluations on a benchmark explicitly covered by the paper, using the same MI/RI schedule. The paper needs to explain this failure or qualify the headline claims so that they accurately reflect the observed variability.","section":"Table 1, ImageNet-100 IPC=10 rows"},{"comment":"The MI=4,000/RI=500 schedule is selected from a six-cell grid on ImageNet-1K (Table 4c) and then applied unchanged to all datasets and IPCs (Appendix Table 8b). The load-bearing assumption that this single schedule transfers is already contradicted by the ImageNet-100 IPC=10 results in Table 1. Please provide per-dataset sensitivity analysis for MI/RI, or explicitly treat these as dataset-specific hyperparameters and report how they were chosen for each benchmark.","section":"Section 4.3, Table 4c and Appendix Table 8b"},{"comment":"The 'medium probability' initialization hypothesis is supported by a difference of only 0.12 points over 'lowest probability' (57.67 vs 57.55) and 0.64 points over 'highest probability' (57.67 vs 57.03) in Table 4b. These gaps are within the typical run-to-run variation of distillation experiments, yet the text states this strategy 'achieves the best accuracy' and links it to the diversity mechanism. Please report error bars for these ablations and soften the claim unless the difference is statistically meaningful.","section":"Section 3, Selection Criteria and Table 4b"}],"minor_comments":[{"comment":"The title on page 1 reads 'Diversity-drivenEarlyLate' with a missing space, and the abstract's sentence 'Our approach is conceptually simple yet effective, it partitions...' would benefit from a semicolon or period instead of a comma.","section":"Title and Abstract"},{"comment":"The left subfigure's y-axis is unlabeled and the caption does not state which features were used for the cosine similarity computation; please clarify what is plotted and why lower values are better.","section":"Figure 2"},{"comment":"For CIFAR-10 with MobileNet-V2, SRe2L results are omitted without explanation; please indicate whether this is due to scalability, missing values, or a formatting choice.","section":"Table 1"},{"comment":"The statement that for ImageNet-100 the medium images are extracted 'based on the rankings of the original ImageNet-1K' rather than a teacher trained on ImageNet-100 is surprising and should be justified, since the ranking criterion is supposed to reflect the teacher's difficulty assessment of the target dataset.","section":"Appendix B, Ranking"},{"comment":"The formula for total iterations 'N × T − j(j−1)/2 RI' is stated without derivation; please define j and show how it relates to M and RI, and verify the arithmetic for the reported 39.3% saving with j=8 (or whichever value is used).","section":"Section 4.4, Computational Analysis"}],"recommendation":"major_revision","confidential_remarks":"The paper builds very directly on the authors' prior SRe2L and CDA frameworks, and the novelty lies mainly in the EarlyLate scheduling idea. That is acceptable for a strong empirical paper, but the authors should be pushed to make their claims match the data: the '>5% diversity increase' is unsupported, and the ImageNet-100 IPC=10 losses are a real counterexample that the current '13/15' tally papers over. I would be willing to see a revised version, but the current abstract and Section 4.1 overstate the consistency of the gains."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the EarlyLate schedule is a genuinely simple and effective trick—varying the number of optimization iterations per synthetic image saves compute and often improves accuracy. But the paper overclaims. Its own Table 1 shows DELT losing to RDED on ImageNet-100 IPC=10 by 7.8–11.5 points across all three backbones, and on ImageNette IPC=1 by 8–11 points, yet the abstract promises '2~5% on average' and the paper counts '13/15' SOTA. That count only works if you ignore the cells where DELT loses on every backbone. The >5% diversity improvement is asserted in the abstract but never quantified anywhere; Fig. 2 shows curves for SRe2L, CDA, and DELT but not RDED, and gives no number.\n\nWhat is actually new: giving early synthetic images many iterations (MI=4K) and later ones progressively fewer (down to RI=500) is a simple scheduling idea I haven't seen in the cited literature. The ablation in Table 4d isolates it cleanly: CDA+real-init vs DELT gives +2.6 on ImageNet-1K, +0.8 on Tiny-ImageNet, +3.6 on CIFAR-10. That's a real effect. The compute analysis is honest—actual hours in Table 6, ~39% savings on ImageNet-1K. Code is available. The appendix gives full hyperparameters, which is more than many papers in this area do.\n\nSoft spots, in proportion:\n1. The headline claim doesn't survive contact with the paper's own table. The losses on ImageNet-100 IPC=10 are not edge cases; they're three backbones on a benchmark the paper claims to cover with the same schedule it tuned on ImageNet-1K. That's a red flag for the 'fixed schedule transfers' assumption.\n2. The diversity claim is unmeasured. I'd want the cosine-similarity number for all methods, including RDED.\n3. The schedule hyperparameters (MI=4K, RI=500) are chosen via a six-cell grid on ImageNet-1K IPC=10 and then used everywhere. That's mild tuning-on-the-benchmark, and the paper should at least show the schedule is robust across datasets.\n4. Missing comparisons to D3, PDD, LPLD—all cited in the related work but absent from the tables. This matters because these methods also use subtask decomposition.\n\nBottom line: worth publishing as a solid empirical contribution, but the authors need to revise the claims, report the diversity metric, and explain the ImageNet-100 failure. I'd send it to review, with a request for major revision.","headline":"The EarlyLate schedule is a simple and effective compute-saving trick, but the paper's headline claims of consistent SOTA gains and >5% diversity improvement are not supported by its own tables or by any reported metric.","tokens_in":16939,"tokens_out":3712,"would_cite":true,"duration_ms":29202,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a simple staggered optimization schedule—early synthetic images get 4,000 iterations, later ones as few as 500—increases intra-class diversity in dataset distillation, raising ImageNet-1K top-1 accuracy with IPC 50…","keywords":["dataset distillation","synthetic data generation","intra-class diversity","batch-to-global matching","EarlyLate training schedule","image optimization","ImageNet-1K"],"falsifier":"A reader could settle the central claim in one experiment: with total compute matched, compare DELT against a baseline that gives every synthetic image the same MI=4,000 iterations but uses the same teacher-ranked medium-probability real patches as initialization, and measure both IPC-10 and IPC-50 top-1 accuracy on ImageNet-1K. If the equal-iteration baseline matches or beats DELT, the reported gains come from average iteration count or better initialization rather than from the stagger itself. A second check is the diversity metric: DELT predicts more than 5% lower intra-class cosine similarity to class centroids than SRe2L/CDA; on any dataset where that gap fails to appear, the paper's account predicts the accuracy gain should also disappear.","tokens_in":15822,"feed_emoji":"🖼️","tokens_out":8813,"duration_ms":71093,"temperature":0.7,"pith_summary":"Dataset distillation tries to compress a large training set into a few synthetic images per class that still train accurate models. In the batch-to-global matching variant, each synthetic image is optimized against global statistics of a frozen pretrained teacher, and a known failure mode is that images within the same class end up looking alike because they all receive the same supervision and the same number of optimization steps. This paper claims that a simple change—starting images at different times so that early images receive many more gradient steps (up to 4,000) than late ones (as few as 500)—breaks that uniformity. The resulting EarlyLate (DELT) images show lower intra-class cosine similarity, and across CIFAR-10, Tiny-ImageNet, ImageNet-100, ImageNette, and ImageNet-1K, the method reports average accuracy gains of 2 to 5 percentage points over prior methods while using roughly two-thirds of the optimization iterations. If the claim holds, dataset distillation becomes both more accurate and cheaper at large scale, with a method that requires no extra networks or matching signals beyond a frozen teacher and real image patches chosen at medium teacher confidence.","feed_headline":"Staggered training steps lift distilled ImageNet to 66.1%","feed_subtitle":"Early images get 4,000 optimization steps, later ones 500, boosting diversity and cutting synthesis time by 39.3%.","key_machinery":"The load-bearing object is the EarlyLate training schedule, defined by two numbers: maximum iteration MI=4,000 for the earliest batch and round iteration RI=500 as the gap between adjacent batches. Concretely, the procedure starts one gradient loop; after every RI iterations a new batch of synthetic images joins the optimization, so the first batch accumulates 4,000 steps, the second 3,500, and so on down to 500 for the last batch. All batches are concatenated in the objective, and a BatchNorm-distribution regularization term from the frozen pretrained teacher (the same term used in SRe2L) keeps the images realistic. The teacher also serves as a ranker to initialize each synthetic image from a real patch whose class probability sits near the class median, which the authors argue gives medium-difficulty starting points with room for the distillation gradients to add information. The schedule is what creates the claimed diversity: different images are optimized to different depths, so they settle into different local optima instead of converging to the same prototype.","core_discovery":"The central discovery, on the paper's own terms, is that in batch-to-global dataset distillation—where each synthetic image is optimized against global statistics of a frozen pretrained teacher—the uniformity of synthetic images within a class is not fixed by the matching objective itself; it is a by-product of giving every image the same full-length optimization trajectory. DELT partitions the images-per-class budget into consecutive batches and runs them through one shared gradient loop, with the first batch optimized for the full MI=4,000 iterations and each later batch entering later so it receives fewer iterations, down to RI=500. Because the later batches are concatenated with the already-optimized earlier images in the synthesis objective, each batch is distilled against a slightly different surrounding distribution, and each image ends at a different phase of optimization—some heavily compressed, some close to the original patch. The paper reports that this increases intra-class diversity by more than 5% as measured by cosine similarity to class centroids, improves top-1 accuracy by 2 to 5 percentage points on average across five datasets and multiple IPC budgets, and yields 66.1% on ImageNet-1K with IPC 50 under ResNet-101, 4.9% above RDED, with up to 39.3% less synthesis time.","pith_inferences":["Beyond the paper's reported benchmarks, the proposed mechanism predicts that an EarlyLate schedule should also improve other batch-to-global objectives, such as Wasserstein-metric matching or class-wise soft-label alignment, because the diversity mechanism lives in the schedule rather than in the BatchNorm-statistic loss; testing DELT on those frameworks would show whether the gain generalizes.","The schedule was fixed by a small grid search on ImageNet-1K (MI 1K/2K/4K by RI 500/1K), so a natural testable extension is an adaptive schedule that sets each batch's iteration count from per-image difficulty or dataset size; the paper gives no theory for why 4,000/500 is optimal, and an adaptive policy could do better or reveal the choice as dataset-dependent.","The medium-probability initialization is a second load-bearing choice. If teacher confidence is a noisy difficulty signal, an ensemble of rankers or a confidence-calibrated teacher would change the initialization pool; the paper does not test ranker sensitivity, so a reader should treat the initialization benefit as provisional."],"forward_implications":["Because DELT is a schedule rather than a new matching loss, it can be layered onto SRe2L- or CDA-style batch-to-global synthesis; the paper's ablations show it adds roughly 1 to 2 points over real-patch initialization alone and 2.4 points even without initialization.","Distilled datasets made with DELT retain cross-architecture transfer: gains appear on ResNet-18, EfficientNet-B0, MobileNet-V2, MnasNet, and RegNet-Y-8GF at IPC 10, so the diversity is not memorizing the teacher backbone.","The computation saving is structural: total optimization iterations drop to roughly two-thirds of SRe2L/CDA's N times T, with measured wall-clock reductions up to 39.3% on ImageNet-1K, 32% on Tiny-ImageNet, and about 1% on CIFAR-10 where data handling dominates.","Downstream, the more diverse distilled set improves data-free pruning (for example, +4.7 points over RDED at IPC 10) and 100-step class-incremental learning on ImageNet-1K by about 10% on average over G-VBSM."],"supporting_citations":[{"why":"SRe2L supplies the batch-to-global synthesis framework and the globally counted BatchNorm statistics used as the regularization term in DELT's objective, and is the main compute baseline whose N times T cost DELT reduces.","marker":"[40]"},{"why":"CDA is the batch-to-global curriculum synthesis method that DELT extends; the ablation labeled 'Early-only' is CDA with real-image initialization, and it is a time-comparison baseline.","marker":"[39]"},{"why":"RDED is the previous diversity method based on stitching teacher-ranked real patches; DELT's headline 4.9% ImageNet-1K gain is measured against it, and it motivates the real-patch initialization idea.","marker":"[32]"},{"why":"G-VBSM is the prior diversity-via-multiple-backbones baseline that DELT compares against on accuracy, synthesis time, and continual learning, and is the compute-heavy contrast.","marker":"[28]"},{"why":"D3 establishes the subtask-decomposition idea of distilling separate subsets and then combining them, which DELT reframes as staggered local optimizations inside one class.","marker":"[25]"},{"why":"Bai et al. supplies the early-stopping-with-different-training-depths motivation that DELT applies to image optimization rather than to network weights.","marker":"[1]"}],"fun_headline_variants":["Staggered steps in dataset distillation lift ImageNet to 66.1%","Less training for later images yields more diverse synthetic data","DELT cuts synthesis time 39% and beats prior distilled ImageNet","Diverse distillation via early-late iterations: +5% accuracy, less compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gain rests on the assumption that a single fixed stagger—4,000 steps for the first image down to 500 for the last, chosen by a small grid search on ImageNet-1K—creates complementary information on every dataset and IPC budget, rather than simply producing noisier late images, and that the frozen teacher's class probability reliably identifies which real patches make the best starting points.","fun_headline_variants_meta":{"raw":{"variants":["Staggered steps in dataset distillation lift ImageNet to 66.1%","Less training for later images yields more diverse synthetic data","DELT cuts synthesis time 39% and beats prior distilled ImageNet","Diverse distillation via early-late iterations: +5% accuracy, less compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000795,"raw_usage":{"total_tokens":3600,"prompt_tokens":1147,"completion_tokens":2453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":763,"completion_tokens_details":{"reasoning_tokens":2374}},"tokens_in":763,"tokens_out":2453,"duration_ms":17094,"temperature":1.0,"reasoning_tokens":2374,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:39:15.625133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the central claim in one experiment: with total compute matched, compare DELT against a baseline that gives every synthetic image the same MI=4,000 iterations but uses the same teacher-ranked medium-probability real patches as initialization, and measure both IPC-10 and IPC-50 top-1 accuracy on ImageNet-1K. If the equal-iteration baseline matches or beats DELT, the reported gains come from average iteration count or better initialization rather than from the stagger itself. A second check is the diversity metric: DELT predicts more than 5% lower intra-class cosine similarity to class centroids than SRe2L/CDA; on any dataset where that gap fails to appear, the paper's account predicts the accuracy gain should also disappear.","supporting_citations":[{"cited_title":"Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective","cited_arxiv_id":null,"evidence_quote":"SRe2L supplies the batch-to-global synthesis framework and the globally counted BatchNorm statistics used as the regularization term in DELT's objective, and is the main compute baseline whose N times T cost DELT reduces."},{"cited_title":"Dataset distillation via cur- riculum data synthesis in large data era","cited_arxiv_id":null,"evidence_quote":"CDA is the batch-to-global curriculum synthesis method that DELT extends; the ablation labeled 'Early-only' is CDA with real-image initialization, and it is a time-comparison baseline."},{"cited_title":"On the diversity and realism of distilled dataset: An efficient dataset distilla- tion paradigm","cited_arxiv_id":null,"evidence_quote":"RDED is the previous diversity method based on stitching teacher-ranked real patches; DELT's headline 4.9% ImageNet-1K gain is measured against it, and it motivates the real-patch initialization idea."},{"cited_title":"Generalized large-scale data condensa- tion via various backbone and statistical matching","cited_arxiv_id":null,"evidence_quote":"G-VBSM is the prior diversity-via-multiple-backbones baseline that DELT compares against on accuracy, synthesis time, and continual learning, and is the compute-heavy contrast."},{"cited_title":"Understand- ing and improving early stopping for learning with noisy la- bels","cited_arxiv_id":null,"evidence_quote":"Bai et al. supplies the early-stopping-with-different-training-depths motivation that DELT applies to image optimization rather than to network weights."}],"review_version":1}