{"id":"734d8164-3caa-46a3-9f30-02785af87450","arxiv_id":"2505.06647","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"SLFD wraps GLaD's latent-space dataset distillation in a training-time stochastic loss that samples classifier logits from a low-rank multivariate normal, reporting improved cross-architecture accuracy on several benchmarks.","lead":"This paper adds a stochastic low-rank Gaussian model over classifier logits to GLaD, a latent-space dataset distillation method, and reports accuracy gains on ImageNet subsets, CIFAR-10, and MedMNIST. The proposed method, SLFD, aims to improve how well models trained on tiny synthetic datasets transfer to unseen architectures.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The stochastic loss in Eq. 6 is not the Monte Carlo negative log-likelihood claimed in Eq. 4: it omits the softmax denominator, has an incorrect averaging constant, and mismatches dimensions, leaving the core training objective ungrounded.","rationale":"The paper's central claim is state-of-the-art cross-architecture performance through a stochastic latent-feature model. That model is operationalized entirely by the stochastic loss in Sec. 3.3.1; if Eq. 6 is not a valid Monte Carlo approximation of Eq. 4, the method's training signal does not optimize the stated probabilistic objective. The reader's weakest_assumption identifies exactly this issue, and the concern is load-bearing because it attacks the core mechanism of the method, not a peripheral detail. The internal inconsistencies are concrete: missing softmax normalization, incorrect Monte Carlo constant, and a dimension mismatch between q_m and f(I_syn). These are not merely stylistic or presentational; they mean the described algorithm cannot be reproduced from the paper. The empirical tables provide only partial support: many DC-row gains over GLaD are within one or two accuracy points with overlapping standard deviations, and CIFAR-10 differences in Table 3 are similarly modest, so the results do not independently rescue the method from an invalid objective. The absence of code and the unfinished supplementary text further lower confidence that the implemented loss matches the equations. Given that the central theoretical grounding is broken at the level of the main loss equation, the manuscript cannot be accepted as written. The reader's REJECT verdict is appropriate, and my analysis does not change it.","tokens_in":12417,"tokens_out":6950,"duration_ms":67684,"concrete_test":"Symbolically simplify Eq. 6 with M=1. It reduces to -(1/Nw) Σ_w y_c·(q*_1)_c, an average per-w logit. Eq. 4 with M=1 is -log softmax(f_1)_y = -f_1,y + log Σ_c exp(f_1,c). The two differ by the missing softmax denominator; the +logM term also reveals the sign error when the intended 1/M factor is removed. Then, on one ImageNet subset (e.g., ImNet-A, ipc=1), implement both the published Eq. 6 and the corrected Monte Carlo NLL (using softmax probabilities and matched dimensions), and compare the resulting gradients or final cross-architecture accuracies; disagreement would show the reported results depend on an unstated or incorrect loss.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central contribution is SLFD, whose probabilistic training signal is defined by the stochastic loss in Sec. 3.3.1 and then fed into the gradient-matching objective. Eq. 4 states a Monte Carlo estimate of -log p(y|x) as -log(1/M Σ_m p(y|f_m)), where p(y|f_m) should be the softmax probability of the classifier output. The paper instead defines q_m ~ N(µ(ws),Σ(ws)) in R^{ipc×Nw×C}, forms q*_m = q_m - µ(ws) + f(I_syn) in Eq. 5, and uses Eq. 6. Three internal inconsistencies make Eq. 6 not the stated NLL. First, y_c·(q*_m)_c is a logit, not a log-probability; the class-normalizing denominator Σ_{c'} exp((q*_m)_{c'}) is absent. Second, the log-sum-exp expression contains +logM; a correct Monte Carlo average would require -logM, since log(1/M Σ exp(a_m)) = logsumexp(a_m) - logM. Third, f(I_syn) ∈ R^{ipc×C} is added to q_m ∈ R^{ipc×Nw×C}; broadcasting introduces an Nw-dependent tensor that has no counterpart in Eq. 4, and the outer sum over w is not a marginalization in the stated model. Because the stochastic loss directly determines the gradients used for distillation (Algorithm 1), the entire probabilistic machinery is not grounded in the likelihood that motivates the method. No code is supplied to show whether a corrected loss was actually implemented, and the supplementary text is visibly unfinished (it contains placeholder template instructions). These are internal mathematical inconsistencies, not a disagreement with community consensus, and they invalidate the central claim as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Stochastic Latent Feature Distillation (SLFD), a dataset distillation method that learns a low-rank multivariate normal distribution over the classifier outputs of synthetic images, draws samples from this distribution, and uses the resulting stochastic loss within a gradient-matching objective. The distilled latent codes are passed through a frozen StyleGAN-XL generator to produce synthetic images, and the method is evaluated on ImageNet subsets at 128x128 and 256x256, CIFAR-10, and MedMNIST by measuring cross-architecture test accuracy. The central claim is that SLFD achieves state-of-the-art cross-architecture performance, especially at high resolutions, while adding only three linear layers of computational overhead.","tokens_in":12861,"tokens_out":12367,"duration_ms":121209,"significance":"If the probabilistic formulation were sound, injecting structured, spatially correlated uncertainty into the distillation process would be a useful and broadly applicable contribution to dataset distillation, with potential relevance to medical imaging. The experimental scope is reasonably broad, covering multiple architectures, several ImageNet subsets, a low-resolution benchmark, and a histopathology dataset, and the paper includes ablations on the generative backbone and on the covariance rank. However, the central loss derivation in Sec. 3.3 is mathematically invalid as written, the object being modeled is classifier logits rather than latent features despite the title and abstract, and several empirical claims are not supported by the reported error bars. The contribution is therefore not currently established.","major_comments":[{"comment":"Equation (6) is not a valid Monte Carlo estimate of the negative log-likelihood in Eq. (4). A correct estimate would be -log(1/M \\sum_m softmax_y(q*_m)), including the softmax denominator over classes; Eq. (6) uses the raw logit y_c·(q*_m)_c without class normalization, so the argument of the logarithm is not a probability. In addition, the log M term has the wrong sign for a Monte Carlo average (the correct term is -log M), and the dimensions are incompatible: q_m is in R^{ipc x Nw x C} while f(I_syn) is in R^{ipc x C}, so the shift in Eq. (5) introduces an Nw-dependent broadcasting that has no counterpart in Eq. (4). Furthermore, the outer sum over m in Eq. (6) is redundant because the inner sum uses the same index, so the 1/L factor cancels and the expression does not average over the M samples as claimed. Since this loss directly determines the gradients used in Algorithm 1, the central training mechanism of SLFD is not grounded in the stated probabilistic model.","section":"Sec. 3.3.1, Eq. (6)"},{"comment":"The distribution is defined over classifier outputs f(I_syn), not over latent features. The abstract claims SLFD \"models the joint distribution of latent features\" and the method name repeats this, but Eq. (2) explicitly states f(I_syn) | G16(...) ~ N(mu(w_s), Sigma(w_s)), where mu(w_s) is in R^{B x Nw x C}. The paper should either rename the method and revise the claims or actually model the distribution of the style-code features. As written, the central conceptual claim does not match what is implemented.","section":"Sec. 3.2, Eqs. (1)-(2) and Abstract/Title"},{"comment":"The optimization of the stochastic module is unspecified. Algorithm 1 updates only the latent features w_s and f_n by backpropagation, but the mean, diagonal, and covariance factor are outputs of three fully connected layers that also have parameters. Without specifying how these FC layers are trained, the proposed module cannot learn the distribution it is supposed to model. This is a reproducibility-critical omission.","section":"Algorithm 1"},{"comment":"The text states that SLFD shows \"a consistent improvement\" over GLaD, but the table does not support this. At res=64, SLFD (36.67±0.6) is slightly below GLaD (36.88±1.1); at res=128, 36.27±1.7 versus 35.77±1.2 is within one standard deviation; only at res=256 is the gap (37.17±2.3 versus 35.64±1.7) favorable, and even there the error bars overlap substantially. The authors should report significance tests or additional seeds before claiming consistency.","section":"Table 4, MedMNIST"},{"comment":"The text claims that \"across both settings, our method consistently outperforms GLaD,\" but Table 3 lists no SLFD result for the Gr (randomly initialized generator) setting; only SLFDGt is shown. The comparison to GLaDGr is therefore missing, and the claim is not supported by the presented data.","section":"Table 3, CIFAR-10"}],"minor_comments":[{"comment":"Equation (4) writes p(y|f(I_syn)_m) as if f is stochastic, but earlier the classifier is described as deterministic. The notation should distinguish the deterministic classifier output from the sampled q_m.","section":"Sec. 3.3.1, Eq. (4)"},{"comment":"The prose reports gains \"up to 7%\" and \"up to 36%\" without specifying whether these are absolute or relative improvements. Please clarify the basis of these percentages.","section":"Sec. 4.3.1, Table 1"},{"comment":"The supplementary material contains only template placeholder text about compiling the supplementary with the main paper; it has no actual content. Either provide a real supplementary or remove the placeholder.","section":"Supplementary Material"},{"comment":"The runtime is reported as approximately 9 hours, but it is unclear whether this covers all five ImageNet subsets at both resolutions. Please specify the exact experimental configuration used for the runtime measurement.","section":"Sec. 4.2, Implementation details"},{"comment":"The qualitative figures would be easier to interpret if each panel was labeled with its class, since the reader cannot otherwise identify which synthetic images correspond to which class.","section":"Figures 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to have been submitted with a template-generated supplementary and a missing comparison row in Table 3, suggesting that the text was not carefully checked before submission. More importantly, the central loss in Eq. (6) is mathematically inconsistent with the stated probabilistic model, and no code is provided to verify which loss was actually implemented. If the authors can supply a corrected derivation and rerun the experiments, the contribution may be salvageable, but the current version is not publishable as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SLFD is a GLaD extension that adds a low-rank Gaussian over classifier logits and a mean-shift trick to the distillation loop. The experimental program is real, but the central loss in Eq. 6 is not the Monte Carlo negative log-likelihood the paper claims, and the supplementary is an unfinished template. I'd still send it to review, but the current version is not acceptable.\n\nWhat's genuinely new: the combination of a low-rank multivariate normal on logits with the shift q*_m = q_m - µ(ws) + f(Isyn) is not in GLaD or the cited prior work. The low-rank factorization keeps overhead modest, and the evaluation is more thorough than most papers in this area: five classifiers, multiple ImageNet subsets, CIFAR-10, MedMNIST, and a well-designed ablation swapping StyleGAN-XL for a diffusion backbone. The UViT result is honestly reported as worse, which is a point in the paper's favor.\n\nWhere it falls apart: Eq. 4 defines the objective as a Monte Carlo estimate of -log p(y|x), averaging M softmax probabilities. Eq. 6 is a different object. It lacks the softmax denominator, so y_c·(q*_m)_c is a logit, not a log-probability. It has +log M where a correct average needs -log M. And q*_m is in R^{ipc×Nw×C} while f(Isyn) is R^{ipc×C}, so the broadcast adds an Nw-dependent term with no counterpart in Eq. 4. The double sum over w and m is not the marginalization in the stated model. These are not cosmetic typos; the gradient signal that drives distillation is derived from this loss. Also, the paper says it models 'latent feature distributions' but the distribution in Eq. 1 is over classifier outputs, not latent features. Without code, I can't tell whether the implementation actually computes a corrected loss. Several Table 1 gains are within one standard deviation of GLaD; the DM rows look more substantial but also noisy. The supplementary text is clearly a leftover template.\n\nWho should read it: people working on dataset distillation will find the empirical comparison and the stochastic module idea worth a look, but only after the loss derivation is fixed.\n\nRecommendation: send to reviewers, not desk reject. The idea is salvageable and the experiments are substantial. But the authors should be required to derive the loss correctly, release code, and remove the placeholder text. If the loss is a typo and the implementation is correct, this is a decent incremental contribution. As written, it is not.","headline":"GLaD extension with a promising stochastic module but a central loss that doesn't match the stated likelihood; deserves review, not acceptance as is.","tokens_in":13316,"tokens_out":3614,"would_cite":false,"duration_ms":35908,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that dataset distillation works better when the synthetic latent features feeding a pretrained generator are treated as random draws from a learned distribution rather than fixed points; the result is a method called…","keywords":["dataset distillation","latent feature distribution","low-rank multivariate normal","cross-architecture generalization","StyleGAN-XL","gradient matching","distribution matching","medical image classification"],"falsifier":"Re-run distillation with the stochastic loss replaced by a direct, correctly dimensioned negative log-likelihood that includes the softmax denominator; if accuracy drops or training changes substantially, Eq. (6) is not equivalent to the model being optimized.","tokens_in":12275,"feed_emoji":"🎲","tokens_out":4286,"duration_ms":39096,"temperature":0.7,"pith_summary":"Dataset distillation compresses a large training set into a few synthetic images that can stand in for the original data when training classifiers. This paper argues that previous latent-space distillation methods are too deterministic: they ignore the spatial structure and uncertainty inherent in natural images. SLFD instead models the latent features that produce synthetic images as samples from a low-rank multivariate normal distribution, learned by a lightweight three-layer network. The paper reports that this stochastic approach consistently improves cross-architecture accuracy on ImageNet subsets, CIFAR-10, and a histopathology benchmark, with larger gains at higher resolutions. If correct, SLFD offers a low-overhead, generator-agnostic way to make distilled datasets generalize better to unseen model architectures.","feed_headline":"Stochastic latent features lift distilled dataset accuracy","feed_subtitle":"A low-rank probabilistic module makes tiny synthetic training sets generalize to unseen models, especially at high resolutions.","key_machinery":"The engine is a low-rank multivariate normal distribution over the generator's latent features, predicted by three linear layers that produce the mean μ(ws), the diagonal D, and the low-rank factor P, with covariance Σ = PPᵀ + D. This distribution makes the distillation stochastic and spatially correlated. The samples q_m are shifted by the classifier's deterministic output via q*_m = q_m − μ + f(I_syn), grounding sampled variability in semantic content, and a gradient-matching loss aligns synthetic and real training dynamics.","core_discovery":"The central claim is that explicitly modeling spatial correlation and uncertainty in the latent space of a pretrained generator produces synthetic datasets that train classifiers more effectively across architectures. The method learns a mean, a diagonal, and a low-rank covariance factor for the latent features, samples multiple latent codes, and blends the samples with the classifier's deterministic output before computing a stochastic cross-entropy loss; gradient matching between real and synthetic batches is retained. The paper reports consistent accuracy gains over GLaD on ten ImageNet subsets at 128×128 and five at 256×256, on CIFAR-10, and on MedMNIST PathMNIST at three resolutions, with no degradation as resolution increases. It also shows the low-rank covariance performs on par with a full-rank variant, and that the method transfers to a diffusion generator (UViT), albeit with weaker results than with StyleGAN-XL.","pith_inferences":["A testable alternative explanation is that most of the gain comes from the sample-shift operation in Eq. (5) acting as a form of latent-space data augmentation, rather than from the probabilistic model itself; evaluating SLFD with the shift but without sampling, or vice versa, would separate the two.","The reported stability at 256×256 suggests SLFD could extend to even higher-resolution distillation, where pixel-space methods typically degrade; this extrapolation is not tested in the paper.","For medical data sharing, the stochastic, non-photorealistic synthetic images may carry privacy benefits beyond compactness, but the paper does not measure re-identification risk."],"forward_implications":["Distilled datasets produced by SLFD transfer to unseen architectures (ResNet18, VGG11, ViT, AlexNet) better than GLaD baselines, with gains up to about 7% under gradient matching and about 36% under distribution matching on ImageNet subsets.","At 256×256 resolution, SLFD does not degrade and improves accuracy (e.g., 17.7% on ImNet-A), in contrast to earlier pixel-space and latent-space methods that often lose performance at higher resolution.","On histopathology (PathMNIST), SLFD improves with resolution while GLaD declines, suggesting the method is suitable for high-resolution medical image distillation and data sharing.","The low-rank covariance gives performance on par with the full-rank version, so the stochastic benefit does not require a full covariance matrix, keeping computational overhead low.","The method works with both GAN and diffusion generators, although StyleGAN-XL remains the stronger backbone in the reported experiments."],"supporting_citations":[{"why":"Sets the latent-space distillation pipeline with a pretrained generator and serves as the primary baseline throughout the paper.","marker":"[4]"},{"why":"Supplies the low-rank multivariate normal construction and the sample re-centering trick used in SLFD's stochastic loss.","marker":"[23]"},{"why":"Provides the frozen StyleGAN-XL generator whose latent space is modeled.","marker":"[26]"},{"why":"Defines the gradient-matching objective that SLFD combines with its stochastic loss.","marker":"[39]"},{"why":"Defines the distribution-matching objective, one of the two matching baselines outperformed.","marker":"[38]"},{"why":"Provides the MedMNIST/PathMNIST benchmark used for the medical-domain evaluation.","marker":"[32]"},{"why":"Provides the pixel-space matching-training-trajectories baseline included in comparisons.","marker":"[3]"},{"why":"Provides the UViT diffusion backbone tested in the generator ablation.","marker":"[2]"}],"fun_headline_variants":["Probabilistic latent features sharpen dataset distillation","Modeling latent correlations lifts distilled data accuracy","Low-rank latent covariance improves cross-architecture training","Sampling generator latents yields diverse, generalizable datasets","Uncertainty in latent space boosts distilled dataset generalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that Eq. (6) is a valid Monte Carlo approximation of the negative log-likelihood in Eq. (4); as written, Eq. (6) omits the class-normalizing softmax denominator and mixes tensors of different shapes, so the training signal may not actually match the stated probabilistic model.","fun_headline_variants_meta":{"raw":{"variants":["Probabilistic latent features sharpen dataset distillation","Modeling latent correlations lifts distilled data accuracy","Low-rank latent covariance improves cross-architecture training","Sampling generator latents yields diverse, generalizable datasets","Uncertainty in latent space boosts distilled dataset generalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1390,"prompt_tokens":912,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":406}},"tokens_in":528,"tokens_out":478,"duration_ms":5231,"temperature":1.0,"reasoning_tokens":406,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:37:05.398250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run distillation with the stochastic loss replaced by a direct, correctly dimensioned negative log-likelihood that includes the softmax denominator; if accuracy drops or training changes substantially, Eq. (6) is not equivalent to the model being optimized.","supporting_citations":[{"cited_title":"Generalizing dataset distillation via deep generative prior","cited_arxiv_id":null,"evidence_quote":"Sets the latent-space distillation pipeline with a pretrained generator and serves as the primary baseline throughout the paper."},{"cited_title":"Stochastic seg- mentation networks: Modelling spatially correlated aleatoric uncertainty","cited_arxiv_id":null,"evidence_quote":"Supplies the low-rank multivariate normal construction and the sample re-centering trick used in SLFD's stochastic loss."},{"cited_title":"Stylegan- xl: Scaling stylegan to large diverse datasets","cited_arxiv_id":null,"evidence_quote":"Provides the frozen StyleGAN-XL generator whose latent space is modeled."},{"cited_title":"Dataset condensation with gradient matching","cited_arxiv_id":null,"evidence_quote":"Defines the gradient-matching objective that SLFD combines with its stochastic loss."},{"cited_title":"Dataset condensation with distri- bution matching","cited_arxiv_id":null,"evidence_quote":"Defines the distribution-matching objective, one of the two matching baselines outperformed."},{"cited_title":"Medm- nist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification","cited_arxiv_id":null,"evidence_quote":"Provides the MedMNIST/PathMNIST benchmark used for the medical-domain evaluation."},{"cited_title":"Dataset distillation by matching training trajectories","cited_arxiv_id":null,"evidence_quote":"Provides the pixel-space matching-training-trajectories baseline included in comparisons."},{"cited_title":"All are worth words: A vit back- bone for diffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the UViT diffusion backbone tested in the generator ablation."}],"review_version":1}