{"id":"76f6386e-5f9f-4f04-9869-ed4d8a7be04d","arxiv_id":"2505.05732","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A diffusion model with a timestep-conditioned encoder learns embeddings that reach competitive linear probe accuracy on four of six datasets, with the optimal timestep varying by dataset.","lead":"This paper trains a timestep-dependent encoder alongside a diffusion transformer so the denoising task itself learns image embeddings, then compares their quality with linear probe accuracy on six datasets. It argues diffusion models can serve as representation learners, not only generators.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains hinge on test-set timestep selection plus missing variance; the semantic claim itself is plausible but not yet tightly evidenced.","rationale":"I read the paper in good faith: the core proposal is a coherent extension of diffusion training, where a timestep-dependent encoder epsilon_phi(x0,t) is trained via Eq. 4.6 and the resulting vectors are probed linearly. The abstract's claim is explicitly qualified as 'optimally learned embeddings,' and the paper is candid that optimal timesteps vary by dataset and that selecting t is hard (Section 6). The fundamental premise, that optimizing noise prediction with x0 conditioning yields semantically organized vectors, is supported by the LPA results and t-SNE visualizations; I do not find an internal inconsistency that would invalidate the method. However, the headline comparison is undermined by a protocol asymmetry: DiER's reported accuracy is the maximum over 11 test-evaluated timesteps, while each baseline is a single fixed representation. Table 2 shows a smooth, unimodal accuracy curve over t on every dataset, so the maximum is a favorable extreme of a family of representations; selecting it post hoc on test labels biases the comparison upward by an amount that is not quantified. The paper's own limitation section acknowledges that finding the optimal t is a challenge, but the experimental section does not provide a principled selection rule or a validation-based variant. Compounding this, Appendix A.2 states a single fixed seed and no error bars are reported anywhere, while several headline margins are under 2%, so statistical significance is unknown. I agree with the reader that the asserted semantics-from-denoising premise is not derived, but I see the evaluation-protocol issue as the more load-bearing concern because it directly affects the validity of the strongest claim; the semantic premise is at least probabilistically supported by the accuracy and clustering evidence. The reader identified the same general area, but their emphasis on the untested premise and their call for ablating the encoder objective is only partially aligned with my emphasis on test-set timestep selection and missing variance. A conditional verdict remains appropriate: the method is plausible and potentially useful, but the headline 'surpasses SOTA in most cases' should not be accepted as established until the selection protocol is made fair and uncertainty is quantified. My concrete test, a validation-based timestep selection plus multi-seed runs on the two closest datasets, directly settles whether the reported margins survive a fair protocol.","tokens_in":15391,"tokens_out":2085,"duration_ms":18910,"concrete_test":"Recompute Table 1 using a validation-based timestep selection: split each training set into train/val, train the linear probe on train for each of the 11 timesteps, pick the timestep with best validation accuracy, then report test accuracy at that timestep. Compare against the current test-selected maxima in Table 1. Additionally, run the full DiER pipeline with at least 3 random seeds on CIFAR10 and OCT2017 and report mean +/- std; if the validation-selected timestep accuracy falls below the best baseline by more than the seed std, the headline supremacy claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that optimally selected DiER embeddings outperform SOTA self-supervised methods on four of six datasets. The load-bearing weakness is the evaluation protocol, not the training objective itself. In Table 1 and Appendix A.2, the reported DiER number is the maximum LPA over 11 tested timesteps (0, 100, ..., 999) evaluated on the test set; the winning timestep is chosen after seeing test accuracy, while competing methods report a single fixed representation. This is a form of post hoc selection: any set of time-indexed features will have a best test timestep, and that best value is biased upward relative to the value obtained from a timestep chosen without test labels. The paper acknowledges the difficulty of selecting t (Section 6, Limitation) but does not provide a validation-based selection rule. A second issue is the absence of error bars or multiple seeds, despite Appendix A.2 stating that a single fixed random seed is used; with margins as small as 0.8% on CIFAR10 and 1.2% on OCT2017, seed noise could plausibly flip the comparison. The reader's identified weakest assumption, that Eq. 4.6 induces semantic rather than reconstruction shortcuts, is indeed asserted rather than derived, but the LPA results themselves provide positive evidence for that premise; the more fragile link is the fairness of the headline comparison. If the timestep were chosen on a validation split or by a fixed rule (e.g., the intermediate timestep or a low-noise timestep), the reported supremacy margins would likely shrink, and on CIFAR10/OCT2017 could disappear. This concern is addressable and does not require rejecting the method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DiER, a diffusion-based self-supervised representation learning method. A timestep-dependent encoder epsilon_phi maps a clean image x0 and a diffusion timestep t to an embedding vector v_t^s, which is injected into a Diffusion Transformer backbone through adaptive layer norm. The model is trained from scratch with the standard noise-prediction loss (Eq. 4.6), without using class labels. The authors evaluate linear probe accuracy (LPA) on six datasets (MNIST, CIFAR10, CIFAR100, Tiny-IN, BCCD, OCT2017) and compare against Diff-AE, MAE, CAE, DMJD, LMIM, and MAGE. At the timestep with the highest test LPA, DiER outperforms the best baseline on CIFAR10, OCT2017, CIFAR100, and Tiny-IN, while falling behind on MNIST and BCCD. The paper also reports per-timestep LPA curves, t-SNE visualizations, and qualitative reconstruction experiments.","tokens_in":15619,"tokens_out":5265,"duration_ms":51391,"significance":"If the empirical claims hold, this is a meaningful result: it would show that a simple modification of the standard diffusion objective suffices to obtain discriminative semantic embeddings, strengthening the connection between generative and discriminative self-supervised learning. The method is simple and the paper is unusually transparent about architectures, hyperparameters, and per-timestep accuracy, which aids reproducibility. However, the headline superiority claim is weakened by the test-set-based selection of the best timestep and by the absence of error bars or multiple seeds. Because the reported margins over state-of-the-art baselines are small on some datasets, the central claim should be regarded as promising but not yet fully established.","major_comments":[{"comment":"The reported DiER accuracy in Table 1 is the maximum over the 11 tested timesteps (0, 100, ..., 999), and the winning timestep is selected from test-set LPA. This is a post hoc selection: any time-indexed representation family will have a best test timestep, and that maximum is biased upward relative to a timestep chosen without access to test labels. The issue is load-bearing because the claimed wins are small on CIFAR10 (0.8%) and OCT2017 (1.2%). A validation-based timestep selection rule, or a fixed protocol (e.g., always use t=100), should be applied before comparisons are made, and the abstract's 'surpasses SOTA in most cases' should be conditioned on that rule.","section":"Section 5.2, Tables 1 and 2, Appendix A.2"},{"comment":"All experiments are run with a single fixed random seed and no error bars are reported. Given that the differences against the strongest baselines are between 0.8 and 6.5 percentage points on the four winning datasets, seed-level variation could plausibly change the ranking. The paper should report mean and standard deviation over at least three seeds for DiER and the relevant baselines, or otherwise justify that the differences are larger than run-to-run variability.","section":"Appendix A.2; Table 1"},{"comment":"The central premise that optimizing the diffusion noise-prediction loss induces semantically meaningful embeddings in epsilon_phi(x0,t) is stated as a natural expectation rather than derived or ablated. The LPA results support the premise, but the paper does not test whether the same embeddings could be obtained from a simpler denoising autoencoder at a single noise level, or whether removing timestep conditioning would degrade results. Adding such ablations would substantiate the claim that the diffusion denoising rule, rather than the autoencoding reconstruction, is responsible for the semantic content.","section":"Section 4, paragraph following Eq. (4.6)"}],"minor_comments":[{"comment":"The phrase 'as and employ a self-supervised model' appears to be missing words and is ungrammatical; please revise.","section":"Section 4, first paragraph"},{"comment":"'DiER consistently SOTA self-supervised methods' is missing a verb; it should read 'DiER consistently outperforms SOTA self-supervised methods' or similar.","section":"Section 5.2, first paragraph"},{"comment":"The subscript formatting is ambiguous; for example, the MNIST entry '89.9400' makes it difficult to distinguish the reported accuracy from the selected timestep. Please format subscripts clearly.","section":"Table 1"},{"comment":"The linear probing setup is described only as 'LPA testing spans 100 epochs'; please specify the classifier type (e.g., logistic regression, linear layer) and the feature normalization used.","section":"Appendix A.2"},{"comment":"The caption 'Embedding Representations with Small Margins' does not match the figure content described in the text; please clarify or rename the caption.","section":"Appendix B, Figure 7 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the core idea is interesting, but the evaluation protocol needs strengthening before the headline claim can be accepted. I found no evidence of integrity issues; the limitations are acknowledged, though the test-set timestep selection is downplayed. Providing code and per-seed results would substantially increase confidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2505.05732 before reading it. First, the method is a fairly direct extension of Diff-AE: add a timestep-conditioned encoder that maps (x0, t) to a vector, inject it via adaLN into a pixel-space DiT, and train with the standard noise-prediction loss. That is a reasonable thing to try, and the paper does it cleanly. Second, the headline claim—that the optimally learned embeddings surpass MAE, CAE, DMJD, LMIM, and MAGE on most datasets—is not supported by the experiments as reported, because the 'optimal' timestep is chosen after looking at test-set linear probe accuracy. The comparison is post hoc selection against single-fixed-representation baselines, with one seed and no error bars.\n\nWhat the paper does well: it systematically sweeps 11 timesteps per dataset and shows that LPA peaks at intermediate t, which is a useful observation. The training objective is standard diffusion, and labels are not used in pretraining, so the result is not circular in the obvious way. Six datasets, including two medical ones, give the evaluation some breadth. The t-SNE and reconstruction figures are a nice qualitative complement.\n\nWhere it falls short: the evaluation protocol is the load-bearing weakness. Table 2 makes the selection explicit—the best number across t is what gets reported in Table 1. On CIFAR10 the margin over CAE is 0.8%, and on OCT2017 it is 1.2% over DMJD. With a single random seed (stated in Appendix A.2), those margins could easily flip. The paper acknowledges t selection is hard in the limitations, but it offers no validation-based rule, so a reader cannot tell whether DiER would beat the baselines under an honest selection protocol. The semantic claim ('denoising rules summarize semantic knowledge') is asserted, not derived, but the LPA curves are positive evidence for it; I don't think that premise is the main problem. The main problem is fairness of the comparison.\n\nMy take: the core idea is worth taking seriously, the flaws are addressable, and the paper deserves a real referee. If I were handling it, I would ask for a validation-set or fixed-rule selection of t, multiple seeds with standard deviations, and code release. The method likely does learn discriminative representations at some noise levels; whether it actually beats the SOTA baselines is an open question that the current version does not settle. The paper is for researchers working on diffusion-based representation learning and for anyone who wants a concrete example of post hoc timestep selection in SSL evaluation.","headline":"Reasonable Diff-AE extension, but the SOTA claim rests on test-set t selection and a single seed.","tokens_in":16249,"tokens_out":4232,"would_cite":false,"duration_ms":40277,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion models trained only to denoise can learn image semantics that beat dedicated self-supervised learners on four of six benchmarks.","keywords":["diffusion models","self-supervised representation learning","denoising autoencoder","linear probe accuracy","timestep-dependent embedding","Diffusion Transformer","semantic embedding"],"falsifier":"Re-run the pipeline but choose the best timestep on a held-out validation split, fitting each linear probe on training data only, then report test accuracy at that timestep; if the margins over MAE on CIFAR100 and Tiny-IN and over DMJD on OCT2017 shrink to a point or reverse, the headline comparison is an artifact of test-set model selection. As a second check, freeze the encoder and retrain the denoiser with scrambled $v^s_t$ inputs; if linear probe accuracy barely drops, the embeddings themselves are not carrying the semantic signal.","tokens_in":15113,"feed_emoji":"🧠","tokens_out":13736,"duration_ms":123566,"temperature":0.7,"pith_summary":"The paper asks whether a denoising diffusion model — a generative model that learns by adding and removing noise — can double as a learner of visual semantics. To test this, it adds a timestep-conditioned encoder that compresses each clean image into a 1024-dimensional vector, then trains the model to predict noise exactly as in standard diffusion training. The claim, measured by linear probe accuracy, is that these learned vectors beat state-of-the-art self-supervised representation learners (MAE, CAE, DMJD, LMIM, MAGE) on four of six datasets once the best timestep is selected. If true, generation-oriented training alone would yield discriminative features good enough for downstream classification, with no labels, auxiliary losses, or contrastive pairs required.","feed_headline":"Diffusion embeddings beat self-supervised rivals on most benchmarks","feed_subtitle":"Timestep-conditioned diffusion embeddings top MAE, CAE, DMJD, and MAGE on four of six datasets.","key_machinery":"The load-bearing object is the encoder $\\varepsilon_\\varphi$ that maps a clean image $x_0$ together with a noise level $t$ to a vector $v^s_t$, injected into every transformer block of a DiT backbone through adaptive layer normalization (adaLN). The objective is the diffusion noise-prediction loss with this vector as conditioning, $L_{simple} = \\mathbb{E}_{x,\\epsilon,t}\\left[\\lVert \\epsilon - \\epsilon_\\theta(x_t, t, \\varepsilon_\\varphi(x_0,t))\\rVert_2^2\\right]$. This recasts the diffusion model as $T$ denoising autoencoders — networks that learn to undo corruption added to an image, one per noise level — so the only way the encoder can help the denoiser is to summarise, for the whole dataset, the rule that maps each corrupted version back to its source; the authors argue such denoising rules amount to knowledge about how images of particular classes are generated. No labels, data augmentation beyond horizontal flipping, or auxiliary losses are used, so any semantic content in $v^s_t$ must come from the denoising objective itself.","core_discovery":"DiER (Diffusion Embedding Representations) trains a timestep-conditioned encoder together with a Diffusion Transformer backbone using only the standard diffusion noise-prediction loss, with the encoder's output $v^s_t = \\varepsilon_\\varphi(x_0,t)$ fed into every block via adaptive layer normalization. The central claim is that these vectors are semantically meaningful image embeddings: at the best timestep, linear probe accuracy reaches 59.7 on CIFAR10, 98.1 on OCT2017, 37.7 (top-1) and 67.3 (top-5) on CIFAR100, and 36.7 (top-1) and 62.5 (top-5) on Tiny-IN, ahead of every compared self-supervised method on these four datasets. On MNIST and BCCD the same embeddings trail the strongest baselines, and the best timestep shifts with dataset content — $t=100$ for object-central natural images, $t=400$ to $500$ for datasets with small or sparse targets. The paper reads this as evidence that diffusion denoising, reframed as a multi-level denoising autoencoder, learns discriminative structure as a by-product of predicting noise.","pith_inferences":["Editorial inference: the best timestep is selected from test-set results, so the reported 'optimal' accuracies are likely optimistic; re-selecting the timestep on a validation split is the direct way to test how much of the advantage survives honest model selection.","Editorial inference: because different timesteps encode different granularities, concatenating or averaging $v^s_t$ across several timesteps could yield a richer representation than the single best one the paper reports.","Testable extension: the same recipe — a timestep-conditioned encoder trained by the denoising objective alone — could carry over to audio and text diffusion models, and the content-based optimal-timestep heuristic could be turned into a rule that avoids per-dataset probing entirely.","Editorial inference: the failures on MNIST and BCCD show the advantage is not universal; identifying what separates the four winning from the two losing datasets (target size, intra-class variance, background structure) would upgrade the claim into a predictive theory."],"forward_implications":["Diffusion pre-training can be dual-purpose: one noise-prediction run produces both a generator and embeddings whose class structure a linear probe can read, so representation learning needs no separate objective.","The learned representation is compact and scalable: a single 1024-dimensional vector per image reaches state-of-the-art linear probe accuracy, and the encoder can be replaced by higher-capacity architectures.","Timestep choice is the key transfer decision: downstream tasks must read the embedding at the dataset's optimal noise level, which the paper locates at intermediate timesteps ($t=100$, $400$, or $500$) rather than at the clean or fully noisy ends.","The optimal timestep correlates with image content: datasets whose classification target fills most of the frame prefer $t=100$, while datasets with small or sparse targets (digits, cells, lesions) prefer $t=400$–$500$, giving a content-based heuristic for where to probe."],"supporting_citations":[{"why":"Supplies the DDPM forward/backward Markov chains and the weighted noise-prediction loss that Eq. (4.6) rewrites with encoder conditioning.","marker":"[1]"},{"why":"Gives the denoising-autoencoder interpretation of diffusion models that motivates converting a DDM into multi-level DAEs.","marker":"[22]"},{"why":"The Diff-AE baseline used for comparison and the prior proposal of a semantic latent in diffusion models that DiER extends.","marker":"[28]"},{"why":"The MAE baseline that DiER beats on CIFAR100 and Tiny-IN, and the source of the linear probe accuracy evaluation protocol.","marker":"[30]"},{"why":"The DMJD baseline that DiER surpasses on OCT2017 but trails on BCCD.","marker":"[34]"},{"why":"The MAGE baseline that holds the best MNIST score DiER does not match.","marker":"[32]"},{"why":"The Diffusion Transformer backbone whose homogeneous blocks receive the embedding vectors through adaptive layer norm.","marker":"[45]"},{"why":"The adaptive conditioning mechanism (FiLM-style adaLN) through which $v^s_t$ is injected into the denoiser.","marker":"[48]"}],"fun_headline_variants":["Diffusion embeddings top self-supervised on 4 of 6 datasets","Timestep-conditioned diffusion embeddings beat MAE, CAE, DMJD, MAGE","DiER: diffusion learns embeddings that beat self-supervised rivals","Diffusion model's hidden embeddings surpass self-supervised leaders","Noise-prediction loss yields discriminative embeddings, best on 4 sets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result rests on the premise, stated as a natural expectation rather than proved, that optimizing the noise-prediction loss forces the encoder to learn class-level semantic knowledge instead of low-level reconstruction details, with the best reported accuracies selected from eleven timesteps evaluated on the test set.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion embeddings top self-supervised on 4 of 6 datasets","Timestep-conditioned diffusion embeddings beat MAE, CAE, DMJD, MAGE","DiER: diffusion learns embeddings that beat self-supervised rivals","Diffusion model's hidden embeddings surpass self-supervised leaders","Noise-prediction loss yields discriminative embeddings, best on 4 sets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000344,"raw_usage":{"total_tokens":1909,"prompt_tokens":984,"completion_tokens":925,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":827}},"tokens_in":600,"tokens_out":925,"duration_ms":7465,"temperature":1.0,"reasoning_tokens":827,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:57:35.298406+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline but choose the best timestep on a held-out validation split, fitting each linear probe on training data only, then report test accuracy at that timestep; if the margins over MAE on CIFAR100 and Tiny-IN and over DMJD on OCT2017 shrink to a point or reverse, the headline comparison is an artifact of test-set model selection. As a second check, freeze the encoder and retrain the denoiser with scrambled $v^s_t$ inputs; if linear probe accuracy barely drops, the embeddings themselves are not carrying the semantic signal.","supporting_citations":[{"cited_title":"Diffusion au- toencoders: Toward a meaningful and decodable repre- sentation","cited_arxiv_id":null,"evidence_quote":"The Diff-AE baseline used for comparison and the prior proposal of a semantic latent in diffusion models that DiER extends."},{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"The MAE baseline that DiER beats on CIFAR100 and Tiny-IN, and the source of the linear probe accuracy evaluation protocol."},{"cited_title":"Disjoint masking with joint distillation for efficient masked image modeling","cited_arxiv_id":null,"evidence_quote":"The DMJD baseline that DiER surpasses on OCT2017 but trails on BCCD."},{"cited_title":"Mage: Masked gener- ative encoder to unify representation learning and image synthesis","cited_arxiv_id":null,"evidence_quote":"The MAGE baseline that holds the best MNIST score DiER does not match."}],"review_version":1}