{"id":"e35ad32c-1a7a-4825-9986-10b62c58e060","arxiv_id":"2411.11924","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Benchmarking DATM, DANCE, and RCIG on noisy labels shows symmetric noise is removed effectively while asymmetric noise is absorbed, and the paper's two bounds are tautological.","lead":"This paper tests whether dataset distillation, a technique that compresses a dataset into a few synthetic images, can remove label noise when training image classifiers. It finds that distillation filters random label noise effectively but can absorb structured asymmetric noise, offering a privacy-friendly alternative for noisy real-world data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Distilled-set label correctness is never checked, so the central denoising mechanism is inferred only from test accuracy; the paper also lacks standard noisy-label baselines and contains a reversed inequality in Corollary-II.","rationale":"The paper's core claim is a practical one: offline distillation yields a synthetic set that can replace the clean/noisy separation loop. The experiments support a weaker statement: training on distilled sets often beats CE on the full noisy set. That weaker statement can be true even if the synthetic set is not clean, because distillation to tiny IPC is a strong compression prior. The paper's own Insight-I invokes the common-pattern mechanism, so the lack of any direct check of synthetic-label correctness is a genuine gap. I would not reject the paper: the trend in Fig. 2 is interesting and the qualitative images in Fig. 3 are suggestive, and the released code is a plus. But the central claim should be conditional on either a label-level verification or an explicit comparison with existing noisy-label methods. Corollary-II's inequality is also reversed; it should be removed or corrected. I agree with the reader that label correctness is the weakest assumption, and further note that the missing robust baselines and the erroneous corollary make the practical 'good denoiser' wording too strong.","tokens_in":11500,"tokens_out":10480,"duration_ms":107239,"concrete_test":"Use the released code to reproduce DATM on CIFAR-10 with symmetric noise tau=0.8 at IPC=1. Then classify the ten distilled images with a clean-trained reference classifier (e.g., a standard ResNet trained on original clean CIFAR-10) and compare each predicted label with the synthetic label assigned by the distillation. If the labels match for at least 8/10 images, the common-pattern denoising mechanism is supported; if they do not, the observed test accuracy has a different explanation, and the central claim should be weakened. Repeat for one higher IPC (e.g., IPC=10) to confirm the check is not an artifact of a single image per class.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that distillation removes symmetric/natural label noise because it captures 'common patterns' while ignoring noisy outliers (Insight-I, Sec. 4). The only evidence is indirect: models trained on distilled sets outperform CE training on the full noisy set (Figs. 2, 5). This leaves the mechanism unverified. The accuracy gain could be caused by the strong compression/regularization that comes from distilling to a very small IPC, rather than by actual label denoising of the synthetic set. The paper does not check whether the distilled images carry the labels they are assigned (Fig. 3 is only illustrative). Corollary-I, Eq. (10), does not supply this missing support; it simply restates the expected number of clean samples under the symmetric-noise model. Moreover, no standard noisy-label learner (Co-teaching, DivideMix, GCE, etc.) is compared, so 'good denoiser' is asserted only against the weakest baseline. A separate internal problem is Corollary-II, Eq. (11): if clean samples per class must satisfy IPC <= |S|(1-tau)/C, the implied bound is tau <= 1 - C*IPC/|S|, not tau >= ..., so the stated probabilistic inference is algebraically reversed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an alternative to conventional noisy-label learning: instead of first estimating noise and then filtering/reweighting/relabeling, it suggests applying dataset distillation to a noisy dataset and training on the resulting synthetic set. The authors benchmark three dataset distillation methods (DATM, DANCE, RCIG) on CIFAR-10/100, Tiny-ImageNet, and CIFAR-N under symmetric, asymmetric, and natural label noise. Their central observations are that distillation outperforms cross-entropy training on the full noisy dataset under symmetric and natural noise at sufficiently high noise rates, but that structured asymmetric noise can be absorbed into the distilled samples. They also state two corollaries relating the number of distilled samples per class (IPC) to the noise rate. The paper is framed as an empirical study with a new perspective rather than a new algorithm.","tokens_in":11763,"tokens_out":5646,"duration_ms":53195,"significance":"If the central claim holds, the paper offers a practically appealing preprocessing strategy: a practitioner could distill a noisy dataset offline and train on the synthetic set, avoiding the iterative clean/noisy separation that characterizes most robust-learning methods. The benchmarking across three representative distillation families and several noise types is a useful service to the community, and the negative result for asymmetric noise is an honest and informative finding. The paper also releases source code. However, the significance is currently limited by the weakness of the empirical evidence: no error bars or multiple seeds, no comparison with state-of-the-art noisy-label methods, and no direct verification that distilled samples actually carry clean labels. The two corollaries, as stated, contain formal errors. The core observation is plausible and potentially interesting, but it is not yet convincingly established.","major_comments":[{"comment":"The mechanism behind the central claim is not verified. The paper asserts that distillation 'captures common patterns while ignoring outliers such as noise,' but it never checks whether the distilled synthetic samples actually correspond to the clean labels they are assigned; Figure 3 is only illustrative. The observed accuracy gain could arise from the strong regularization of training on very few synthetic images rather than from label denoising. To support the mechanism, please compare distillation against training on same-size random subsets of the noisy dataset, and inspect the label correctness of distilled samples (e.g., by evaluating a clean-data classifier on the distilled images or by human inspection of the synthetic set).","section":"Section 4, Experiment-I and Insight-I"},{"comment":"All reported results are single accuracy curves with no error bars or multiple seeds. Claims such as 'a single distilled sample per class surpasses the baseline' at τ=0.6/0.8 need variance estimates to be credible. Moreover, the only baseline is cross-entropy training on the full noisy dataset, which is known to be weak under high noise; standard noisy-label methods (e.g., Co-teaching, DivideMix, GCE) are not compared. Without such comparisons, the paper's claim that dataset distillation methods are 'good label denoisers' is not established relative to existing robust-learning techniques. Please report mean and standard deviation over at least three seeds and include at least one or two standard noisy-label baselines.","section":"Section 4, Figures 2, 4, 5 and Implementation Details"},{"comment":"The inequality in Corollary-II is algebraically reversed. From Corollary-I, IPC ≤ |S|(1−τ)/C, rearranging gives τ ≤ 1 − C·IPC/|S|, not τ ≥ 1 − C·IPC/|S| as stated in Eq. (11). Furthermore, no derivation of the probability bound is provided, and the statement 'with at least high probability 1−δ' introduces a parameter δ that never appears in any antecedent or proof. The probability expression is also not well-formed, as the event is not clearly defined over a random draw of the dataset. This corollary should be corrected or removed.","section":"Section 4, Corollary-II, Eq. (11)"},{"comment":"Corollary-I is essentially a restatement of the expected number of clean samples per class under the symmetric noise model, not a derived consequence of distillation behavior. The step from 'at most |S|(1−τ)/C clean samples per class' to 'the maximum IPC needed to perfectly preserve sample information is this quantity' is unjustified; distillation could in principle preserve information from noisy samples or from fewer clean samples. Please provide a proof or rephrase the statement as a heuristic motivation, and explicitly state the assumption of balanced class sizes.","section":"Section 4, Corollary-I, Eq. (10)"}],"minor_comments":[{"comment":"There are multiple typos and formatting issues: Section 2 header 'Realted Works' should be 'Related Works'; the Contributions list says 'bellow' instead of 'below'; Section 1 has 'Y ou need generalizability'; Section 3.1 has 'that isSymmetric' missing a space; and 'DA TM' is inconsistently spaced in Section 3.2.","section":"Throughout"},{"comment":"The probability notation in Eq. (11) is confusing because the same symbol P is used for both the probability over the dataset and the claimed bound; please use separate notation and make the event inside the probability explicit.","section":"Section 4, Eq. (11)"},{"comment":"The caption uses 'CIFAR10N' while the text uses 'CIFAR-10N'; please standardize the naming.","section":"Figure 5"},{"comment":"The two corollaries are presented without proofs and without a clear connection to the experimental protocol; consider renaming them 'Remarks' and providing formal statements or derivations.","section":"Section 4"},{"comment":"The phrase 'In the Wild' is broader than the experimental scope, which is limited to CIFAR-10/100, Tiny-ImageNet, and CIFAR-N; consider tempering the title or adding experiments on larger-scale noisy datasets such as Clothing1M or WebVision.","section":"Title and Abstract"}],"recommendation":"major_revision","confidential_remarks":"This is an empirical observation/benchmark paper rather than a new method. The core observation is plausible, but the current evidence is not strong enough for publication in a top-tier venue: the lack of error bars, the absence of any standard noisy-label baselines, and the unverified denoising mechanism are substantial gaps. The reversed inequality in Corollary-II is a concrete formal error that must be fixed. I would encourage the authors to strengthen the experimental protocol and to either prove or remove the corollaries."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: this is a plausible idea with a thin empirical base. The claim that dataset distillation removes symmetric and natural label noise while absorbing asymmetric noise is worth testing, but the evidence here does not establish it. The two corollaries are not contributions; one has a reversed inequality.\n\nWhat is genuinely new: the paper is the first to systematically benchmark three distillation methods (DATM, DANCE, RCIG) across symmetric, asymmetric, and realistic CIFAR-N noise. The observed pattern—distillation beats full CE training on symmetric noise at high rates, struggles with asymmetric noise—is consistent with the memorization literature and could be practically useful. Shipping code is a plus.\n\nThe soft spots are real. The central mechanism is inferred, not verified. The paper never checks whether the distilled images actually carry the assigned labels; the accuracy gain could come from the strong regularization of distilling to a few images per class rather than from denoising. No comparison to any standard noisy-label learner (Co-teaching, DivideMix, GCE) is included, so 'good denoiser' is only asserted against a weak baseline. The single accuracy curves without seeds or error bars also limit confidence.\n\nThe corollaries are the weakest part. Corollary-I is just the expected number of clean samples per class under the symmetric noise model—true by definition, not a result. Corollary-II has an algebra error: solving IPC ≤ |S|(1−τ)/C gives τ ≤ 1 − C·IPC/|S|, so the direction in the paper is backwards. That error undercuts the credibility of the theoretical framing.\n\nNone of this kills the empirical observation. The practical suggestion—distill your noisy dataset and train on the synthetic set as a privacy-preserving preprocessing step—has merit. But the paper needs stronger baselines, verification of the distilled labels, and removal or correction of the corollaries before the claim can be trusted.\n\nThis deserves a serious referee, but it should come back with major revisions. I would not cite it in its current form.","headline":"Plausible idea, thin evidence: interesting empirical trends, but missing baselines, no verification of the denoising mechanism, and a reversed inequality in Corollary-II mean this needs major revision.","tokens_in":12235,"tokens_out":3499,"would_cite":false,"duration_ms":30694,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dataset distillation, which compresses a training set into a few synthetic images per class, can act as a label denoiser under symmetric and natural label noise, but it can also absorb structured asymmetric noise into the distilled samples.","keywords":["dataset distillation","noisy label learning","symmetric label noise","asymmetric label noise","natural label noise","CIFAR-N","robust training","label denoising"],"falsifier":"Distill CIFAR-10 with known symmetric noise (say $\\tau=0.8$) at one image per class using any of the three methods, then have a clean-label oracle judge the distilled image for each class: if the distilled sample carries the majority noisy transition rather than the true clean class, the denoising interpretation fails. Equivalently, compute the effective noise rate of the distilled set against clean labels; the claim requires it to be substantially below $\\tau$.","tokens_in":11348,"feed_emoji":"🧹","tokens_out":5808,"duration_ms":50001,"temperature":0.7,"pith_summary":"This paper argues that dataset distillation—compressing a large training set into a small set of synthetic images—can replace the usual estimate-then-correct pipeline of noisy-label learning. By benchmarking three representative distillers on symmetric, asymmetric, and human-annotation noise, it finds that distilled synthetic sets trained with plain cross-entropy often beat training on the full noisy dataset once the noise rate passes a threshold. The paper's central insight is that distillation keeps the common patterns of a class while discarding outliers, so random label noise is washed out before training begins. The same mechanism fails when noise is structured: visually similar classes flip into each other under asymmetric noise, and the distilled set absorbs the noise pattern. The payoff is a denoising step that is offline, privacy-preserving, and free of the feedback loop between noise evaluation and relabeling.","feed_headline":"Distilled data can beat noisy-label training at high noise","feed_subtitle":"Symmetric and natural label noise wash out during synthesis; structured asymmetric noise gets baked into the distilled set.","key_machinery":"The load-bearing mechanism is the distillation objective itself, which forces a tiny synthetic set to reproduce the behavior of the full dataset. Each of the three methods compresses by a different route—DATM aligns student model weights with expert trajectories, DANCE matches feature and label distributions of real and synthetic sets, and RCIG solves a meta-learning loop with convexified implicit gradients—but all share the effect that a few synthetic images per class must stand for the majority pattern in that class. That majority voting is what averages out symmetric label noise and drops rare mislabeled outliers, and it is also why structured asymmetric noise survives: when the noise is a consistent transition between similar classes, it becomes part of the common pattern. The IPC bound and the noise-rate bound translate this qualitative mechanism into quantitative statements about how much clean information survives.","core_discovery":"The central claim is that for symmetric noise, existing dataset distillation methods serve as effective denoising tools, and for real-world natural noise they still generalize well at fixed noisy ratios. The paper supports this with experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet using DATM (trajectory matching), DANCE (distribution matching), and RCIG (meta-learning), showing that distillation beats the full-noisy-dataset baseline at noise rates 0.2–0.8, sometimes with a single image per class. For asymmetric noise, the claim is the opposite: structured label transitions among similar classes are carried into the synthetic samples, so distillation does not recover the clean distribution. The paper also states a bound, $IPC \\leq |\\tilde{S}|(1-\\tau)/C$, for how many distilled images per class are needed to preserve clean information under symmetric noise, and a probabilistic bound on the noise rate inferred from distillation accuracy. It further warns that clean but hard samples, such as tail-class images, can be lossily compressed.","pith_inferences":["If the denoising mechanism is majority-pattern averaging, then the denoising effect at a fixed noise rate should grow with the number of clean examples per class and shrink as the noise becomes class-correlated; a controlled experiment varying per-class noise correlation at constant $\\tau$ would test this directly.","The crossing point where distilled accuracy overtakes full-data accuracy could be inverted to estimate the noise rate of an unlabeled noisy dataset, an estimator the paper raises as an open problem but does not develop.","Because distillation is offline and operates on the data itself rather than on model confidence, it is a natural fit for federated or privacy-sensitive settings: the noisy data never leaves the owner, and no relabeling model sees individual samples.","Natural human annotation noise contains semantic confusions similar to asymmetric noise, so the CIFAR-N results likely sit between the symmetric and asymmetric cases; methods that first detect structured transitions and exclude the affected pairs before distillation would test whether the failure is removable."],"forward_implications":["At noise rates of 0.6 and 0.8, distilling even one image per class on CIFAR-10/100 and Tiny-ImageNet beats training on the full noisy dataset, so practitioners with heavily corrupted labels can distill first and train on the small synthetic set.","Under symmetric noise, the maximum useful distilled size per class is bounded by the expected number of clean samples per class, $|\\tilde{S}|(1-\\tau)/C$, and distilling beyond that point cannot preserve only-clean information.","On real natural noise (CIFAR-N), all three methods beat the full-data baseline under the Worst setting (about 40% noise) with fewer than 10 images per class, so an unknown fixed noise rate does not break the approach.","Asymmetric noise is a failure mode: methods generally underperform the full-noisy baseline unless the distilled set is large, so distillation should not be used when label corruption follows visually similar class pairs.","Clean but hard examples, like tail-class images, are at risk of lossy compression, meaning distillation-based denoising may trade overall accuracy for robustness in imbalanced settings."],"supporting_citations":[{"why":"Introduces dataset distillation as compressing a large dataset into a small synthetic set, the method being repurposed as a label denoiser.","marker":"[44]"},{"why":"Provides DATM, the trajectory-matching distiller used as Benchmark-I.","marker":"[13]"},{"why":"Provides DANCE, the distribution-matching distiller used as Benchmark-II.","marker":"[54]"},{"why":"Provides RCIG, the meta-learning distiller used as Benchmark-III.","marker":"[32]"},{"why":"Supplies the CIFAR-N human-annotation noisy datasets used for the natural noise experiments.","marker":"[46]"},{"why":"Supplies the memorization analysis that motivates the assumption that networks and distillers first capture common clean patterns.","marker":"[1]"}],"fun_headline_variants":["Distilled data washes out symmetric label noise","Label noise? Distill the dataset instead","Dataset distillation beats raw noisy labels","Symmetric noise? Distillation cleans it up","For high noise, distilled data wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the distilled synthetic images represent clean common patterns rather than noise, but it never verifies the labels of the distilled samples, so the denoising story stands on that unexamined assumption.","fun_headline_variants_meta":{"raw":{"variants":["Distilled data washes out symmetric label noise","Label noise? Distill the dataset instead","Dataset distillation beats raw noisy labels","Symmetric noise? Distillation cleans it up","For high noise, distilled data wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000521,"raw_usage":{"total_tokens":2541,"prompt_tokens":985,"completion_tokens":1556,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":1492}},"tokens_in":601,"tokens_out":1556,"duration_ms":11963,"temperature":1.0,"reasoning_tokens":1492,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:38:49.327316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Distill CIFAR-10 with known symmetric noise (say $\\tau=0.8$) at one image per class using any of the three methods, then have a clean-label oracle judge the distilled image for each class: if the distilled sample carries the majority noisy transition rather than the true clean class, the denoising interpretation fails. Equivalently, compute the effective noise rate of the distilled set against clean labels; the claim requires it to be substantially below $\\tau$.","supporting_citations":[{"cited_title":"Dance: Dual-view distri- bution alignment for dataset condensation, 2024","cited_arxiv_id":null,"evidence_quote":"Provides DANCE, the distribution-matching distiller used as Benchmark-II."},{"cited_title":"Dataset distillation with convexified implicit gradients, 2023","cited_arxiv_id":null,"evidence_quote":"Provides RCIG, the meta-learning distiller used as Benchmark-III."}],"review_version":1}