Pith. sign in

REVIEW 2 major objections 5 minor 12 cited by

Diffuse and Disperse: Image Generation with Representation Regularization

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a dispersion penalty to intermediate features improves diffusion image generation.

desk verdict A simple, well-tested regularizer that reliably improves diffusion/flow FID; the mechanism story is weaker than the empirical story. read the letter →

arxiv 2506.09027 v2 pith:VDU5H5YG submitted 2025-06-10 cs.CV cs.AIcs.LG

Runqian Wang , Kaiming He This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords dispersivelossdiffusionmodelsflowmatchingrepresentationregularizationcontrastivelearningimagegenerationone-stepFID
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Dispersive Loss, a small extra term added to the standard denoising loss of diffusion-based image generators. The term pushes the intermediate representations of different images apart in hidden space, playing the repulsive role of contrastive learning but without any positive pairs, because the denoising objective already provides the alignment. The authors report that this simple regularizer consistently lowers FID (a standard image-quality score, lower is better) for Transformer-based diffusion and flow models on ImageNet 256x256, and also improves one-step generation with MeanFlow. The practical payoff is that representation regularization can help generative models without pre-training, extra parameters, or external data. If the effect is general, it gives generative modeling a minimal bridge to self-supervised representation learning.

What carries the argument

The central object is the batch-level Dispersive Loss $$ \mathcal{L}_{\mathrm{Disp}}(X)=\log \mathbb{E}_{i,j}\big[\exp(-D(z_i,z_j)/\tau)\big], $$ where $z_i$ are the intermediate representations of the batch of noised images and $D$ is a dissimilarity such as squared $\ell^2$ distance or negative cosine similarity. This is the uniform/repulsive term of InfoNCE with the positive-pair term removed; the diffusion regression loss supplies the alignment that contrastive learning normally gets from positive pairs. Added with weight $\lambda=0.5$ and temperature $\tau=0.5$ at an intermediate Transformer block, the loss spreads representations apart, requires no extra parameters, and leaves the sampling process unchanged.

What would settle it

Retrain SiT-B/2 on ImageNet with Dispersive Loss applied at block 3 using $\lambda=0.5$, $\tau=0.5$ for five seeds, and compare the FID-50k distribution to the reported baseline of 36.49. If the mean FID is not clearly below the baseline with non-overlapping spreads, the claimed consistent improvement fails. A stronger test is to select $\lambda$ and $\tau$ on CIFAR-10 and apply them unchanged to ImageNet; if the gains disappear, the headline numbers reflect benchmark-specific tuning.

Watch

Extended reading notes

Core claim

The paper's central claim is that adding Dispersive Loss to the standard regression objective of diffusion models consistently improves generation quality across architectures and scales, with no pre-training, no extra parameters, and no external data. For example, SiT-B/2 on ImageNet 256x256 improves from 36.49 to 32.35 FID-50k at 80 epochs without classifier-free guidance, and SiT-XL/2 improves from 18.46 to 15.95 FID in the same setting; with the full SiT training protocol it reaches 1.97 FID, close to the external-alignment method while being fully self-contained. In one-step generation, MeanFlow-XL/2 improves from 3.43 to 3.21 FID, which the authors report as state of the art. The authors interpret the gains as regularization: the dispersion term prevents internal representation collapse, and larger or stronger baselines benefit more.

Load-bearing premise

The headline gains rest on the assumption that the chosen loss strength, temperature, and layer position would still work on a new benchmark, rather than being tuned to the specific ImageNet evaluation where they were tested.

Editorial extensions

If this is right

  • Across four model sizes (S/B/L/XL) and both DiT and SiT, adding Dispersive Loss lowers FID in every tested configuration, with larger relative gains on stronger baselines.
  • In one-step generation, MeanFlow-XL/2 with Dispersive Loss reaches 3.21 FID in a single step, the best result among the one-step diffusion/flow models compared in the paper.
  • All three dispersive variants (InfoNCE with $\ell^2$ or cosine dissimilarity, hinge, and covariance) beat the baseline, suggesting that the dispersion principle rather than a specific loss form drives the improvement.
  • The regularizer is plug-and-play: it requires no pretraining, no extra parameters, no external data, and does not change the diffusion sampling process.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the paper's logic is that the same positive-free dispersion regularizer should benefit other regression-trained generative models, such as autoregressive image models, where the prediction loss already provides the alignment term.
  • Because the authors report that Dispersive Loss increases representation norms even in blocks where it is not applied, one could test whether a simpler norm-growth penalty reproduces part of the FID gain; if it does, the mechanism is less about geometric spread and more about preventing collapsing scales.
  • The paper's hyperparameters were tuned on the same ImageNet benchmark used for the headline results; a fair out-of-benchmark test would be to fix $\lambda$ and $\tau$ on one dataset and apply them without re-tuning to another, which would separate a general regularization effect from benchmark-specific selection.
  • The uniformity perspective suggests a possible geometric consequence: by pushing intermediate features toward a high-entropy distribution, the loss may make the denoising trajectory more stable, which could explain gains in one-step generation; this could be tested by measuring trajectory straightness with and without the regularizer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Dispersive Loss, a batch-level regularizer applied to intermediate representations of diffusion and flow-matching transformers. The loss is constructed by removing the positive-pair term from contrastive objectives such as InfoNCE, hinge, and Barlow Twins, yielding a single-view, parameter-free regularizer that is added to the standard diffusion loss. The authors evaluate the method on ImageNet 256x256 with DiT and SiT models across several sizes, report consistent FID improvements over the baselines, ablate the loss variants, block position, and hyperparameters, and extend the method to one-step generation with MeanFlow, where they report 3.21 FID for MeanFlow-XL/2. The paper claims that the method requires no pretraining, no additional parameters, and no external data, and that it achieves state-of-the-art one-step diffusion-based generation while being a self-contained alternative to REPA.

Significance. If the empirical claims hold, this is a valuable and well-scoped contribution: a one-line regularizer that improves several strong diffusion/flow baselines across model scales and training budgets, with no extra parameters or external data, is likely to be widely adopted. The paper's strengths include concrete pseudocode (Algorithms 1 and 2), a broad set of experiments (Tables 2-7, Figure 4, Appendix B), and an honest accounting of compute and resources in the REPA comparison (Table 6). The controlled-seed CIFAR-10 experiment in Appendix B.2 also provides a useful check that the gain is not purely an ImageNet artifact. However, the central mechanism claim is weakened by the norm-inflation confound in the default L2 loss, and the headline "consistent improvements" claim is currently supported only by single-run ImageNet numbers with hyperparameters selected on the same benchmark.

major comments (2)
  1. [Section 3.2, Eq. (6), Algorithm 1, Figure 3] The default L2 Dispersive Loss is not scale-invariant. For fixed representational directions, every non-self term exp(-||z_i - z_j||^2 / tau) is driven toward zero as the overall scale of the z_i increases, so the objective in Eq. (6) can be substantially minimized by norm inflation rather than by spreading directions. Figure 3 confirms that the norm at Block 3 increases markedly when the loss is applied there, and the text in Section 4.2 uses this norm increase to "help explain the consistent gains." The cosine variant, which removes norm effects, improves SiT-B/2 FID from 36.49 to 34.33, whereas the default L2 variant reaches 32.35. Without a norm-controlled comparison, the additional 1.98 FID improvement cannot be attributed to dispersion. Please add an experiment that applies the same L2 loss to L2-normalized (or otherwise norm-controlled) representations and report both FID and representation-norm statistics. If the gain persists with normalized features, the dispersion interpretation is supported; if it does not, the mechanistic explanation must be revised.
  2. [Section 4.2, Tables 3 and 4, and Figure 4] The default hyperparameters (lambda = 0.5, tau = 0.5, and the single-block placement at the first quarter of the network) were selected after inspecting ablations on the same ImageNet benchmark, and all ImageNet FID numbers are from single training runs without repeated-seed statistics. This makes the "consistent improvements" claim vulnerable to selection on the evaluation set. Please report mean +/- standard deviation over at least three seeds for the central configurations (e.g., SiT-B/2 baseline versus the default Dispersive configuration, and at least one full row/column of Table 4), and state whether the default block position and hyperparameters were fixed before or after these ablations, or provide a held-out validation protocol.
minor comments (5)
  1. [Table 5, last row] The final row of Table 5 compares a 1200-epoch Dispersive Loss run with a 1400-epoch baseline taken from the SiT paper, so the comparison is not matched in training compute. The paper discloses this, but a matched-epoch baseline (or a 1200-epoch rerun) would make the improvement easier to interpret.
  2. [Table 3 and Section 4.2] Table 3 shows that applying the loss to all blocks gives the best FID (32.05), slightly better than any single block, yet all subsequent experiments use a single block at the first quarter. A sentence explaining the computational trade-off would avoid the appearance of inconsistency.
  3. [Section 4.2, paragraph after Table 2] There is a typo: "further complicates the problem.." has a double period. Also, "this is in contrary to" should be "this is in contrast to".
  4. [Figure 3 caption] The word "Evolvement" should be "Evolution" for standard usage.
  5. [Appendix B.2] The CIFAR-10 experiment reports that the reproduced baseline is 2.13 FID and the Dispersive model reaches 2.07 FID, matching the originally reported baseline of 2.07. Since these numbers are close, a brief clarifying note on the variance across runs would be helpful, even though Figure 6 already shows persistent gains under a controlled environment.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the claimed improvements are measured against external baselines, with only mild benchmark-tuning and self-citation flavor.

full rationale

The paper's central claim is empirical: adding Dispersive Loss improves generation quality, and this is evaluated by measuring FID/IS against DiT, SiT, and MeanFlow baselines. The derivation of the loss is explicit and definitional: Eq. (6) is obtained by removing the positive alignment term from InfoNCE, so no prediction is extracted from the loss by construction. The authors' prior work appears only as contextual citations (SimSiam for negative-free contrastive learning, MAE for masked modeling, MeanFlow as the one-step baseline whose codebase was shared), and none of these citations carries the load of the empirical improvement claim. The norm-inflation observation in Fig. 3 is an interpretation of the mechanism, and could be a confound for the stated dispersion story, but that is a correctness risk rather than circularity. Likewise, the choice of lambda=0.5 and tau=0.5 from ablations on the same ImageNet benchmark raises a selection-on-evaluation-set concern, but the reported gains are measured outcomes, not fitted parameters renamed as predictions. The circularity burden is therefore low; score 1 reflects only the mild self-referential flavor in the mechanism interpretation and benchmark tuning, not any reduction of the result to its inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method contributes a loss function but relies on several unproven modeling choices: removing positive pairs is justified by analogy to the regression target; the selected block, lambda, and tau are hand-chosen; and the hypothesis that repulsion on intermediate features acts as regularization is empirical. No new physical or architectural entities are introduced.

free parameters (5)
  • lambda (LDisp weight) = 0.5 default; 0.25 (MF-B/4), 1.0 (MF-B/2), 1.5 (MF-XL/2)
    Controls the strength of the regularizer. Chosen by hand after the sweep in Table 4, with different values used for MeanFlow models.
  • tau (InfoNCE temperature) = 0.5 default
    Temperature in Eq. (6). Selected from Table 4; the authors note that results are robust to tau.
  • regularization block index = block 3 for SiT-B/2, first quarter otherwise
    The block where Dispersive Loss is applied is chosen after the ablation in Table 3, then transferred to other models without per-model re-tuning.
  • Hinge margin epsilon = not specified
    Needed for the Hinge dispersive variant in Table 1 and Table 2, but its value is not reported in the text or appendix.
  • Covariance weight w = not specified
    Needed for the Covariance dispersive variant in Table 1 and Table 2, but its value is not reported in the text or appendix.
assumptions (4)
  • domain assumption Positive pairs are unnecessary because the diffusion regression term supplies an alignment objective.
    Section 3.1 states 'the regression terms provide predefined targets for training, making the use of positive pairs unnecessary'; this is the conceptual basis for dropping positive pairs.
  • domain assumption The intermediate block at the first quarter is a representative layer for regularization across models.
    Table 3 ablates blocks only on SiT-B/2; the chosen block 3 is then transferred to DiT sizes and MeanFlow without per-model re-tuning.
  • ad hoc to paper A repulsion-only loss on a single-view batch will not destabilize the per-sample denoising objective.
    Section 1 claims negligible overhead and no interference with sampling; this is asserted and tested empirically, not derived.
  • domain assumption FID differences are stable enough to interpret without repeated seeds.
    Appendix B.2 reports a reproduced baseline FID of 2.13 versus the original 2.07, showing environment sensitivity; the ImageNet tables report single runs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diffuse and Disperse: Image Generation with Representation Regularization." pith.science (2026). https://pith.science/paper/VDU5H5YG

@misc{pith2026250609027,
  author       = {Pith},
  title        = {Pith review of: Diffuse and Disperse: Image Generation with Representation Regularization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VDU5H5YG}},
  note         = {Machine review of arXiv:2506.09027}
}
read the original abstract

The development of diffusion-based generative models over the past decade has largely proceeded independently of progress in representation learning. These diffusion models typically rely on regression-based objectives and generally lack explicit regularization. In this work, we propose \textit{Dispersive Loss}, a simple plug-and-play regularizer that effectively improves diffusion-based generative models. Our loss function encourages internal representations to disperse in the hidden space, analogous to contrastive self-supervised learning, with the key distinction that it requires no positive sample pairs and therefore does not interfere with the sampling process used for regression. Compared to the recent method of representation alignment (REPA), our approach is self-contained and minimalist, requiring no pre-training, no additional parameters, and no external data. We evaluate Dispersive Loss on the ImageNet dataset across a range of models and report consistent improvements over widely used and strong baselines. We hope our work will help bridge the gap between generative modeling and representation learning.

Figures

Figures reproduced from arXiv: 2506.09027 by the authors.

Figure 1
Figure 1. Dispersive Loss for Generative Modeling. Left: A standard diffusion-based model (e.g., SiT [26] or DiT [31]) with a regression-driven diffusion loss, augmented by a Dispersive Loss applied to an intermediate block. Right: A zoom-in on the first few blocks with Dispersive Loss. Our loss encourages intermediate representations to disperse in the hidden space. It operates on the same input batch of noised images, share… view at source ↗
Figure 2
Figure 2. Effectiveness of Dispersive Loss. We show FID-50k on ImageNet 256×256 at different training epochs, comparing the standard SiT-XL/2 baseline [26] with its counterpart trained with Dispersive Loss. Classifier-free guidance is not used in this plot. More results and details are in Section 4. modeling deterministic trajectories. Despite these advancements, the regression-based training objective remains central to diff… view at source ↗
Figure 3
Figure 3. Evolvement of Representation Norm. Dispersive Loss is applied only at Block 3. Dispersive Loss significantly increases the representation norm compared to the baseline—even in layers where it is not directly applied. The model is SiT-B/2 on ImageNet. Among these variants, InfoNCE with the ℓ2 distance performs best: it improves the FID by a substantial margin of 4.14, or relative 11.35%. This is in contrast to common… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Dispersive Loss for Different Models. We evaluate DiT and SiT baselines across four model sizes, both with and without Dispersive Loss. Each subplot corresponds to a specific model size (XL is only for SiT due to limited computation). All models are trained on ImageNet…
Figure 5
Figure 5. Figure 5: Qualitative results. We present curated samples generated from SiT-XL/2 with Dispersive Loss. model epochs baseline dispersive ∆ MF-B/4 80 18.78 17.61 –6.23% MF-B/2 80 9.77 8.97 –8.18% MF-B/2 240 6.17 5.69 –7.77% MF-XL/2 240 3.43 3.21 –6.41% method params step NFE FID …
Figure 6
Figure 6. Figure 6: CIFAR-10 Result. We also evaluate our approach on non-transformer architectures in the CIFAR-10 dataset [21], where the commonly used network architecture is Unet [34]. The experiments are based on the publicly available code of Flow Matching [24].2 We use the same hyp…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SPARE: Structural Parameter-Free Affinity Regularization for Flow Matching

    cs.CV 2026-08 conditional novelty 7.0 of 10

    A parameter-free regularizer that aligns intermediate token affinities to clean VAE latent affinities, including cross-image pairs, lowers FID on ImageNet with SiT backbones at matched training budgets.

  2. Genuine pair density wave order on the kagome lattice

    cond-mat.supr-con 2026-04 unverdicted novelty 7.0 of 10

    A genuine primary pair-density-wave phase emerges as a competing ground state in a two-orbital kagome Hubbard model over a wide parameter range, driven by sublattice- and orbital-polarized Fermi pockets.

  3. Mitigating Compounding Error via Video Representation Regularization

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Compounding error in autoregressive video diffusion tracks effective-rank collapse of DiT hidden states, and representation regularization (SigReg/Unif) stabilizes long rollouts where data scaling does not.

  4. Self-transcendence: Is External Feature Guidance Indispensable for Accelerating Diffusion Transformer Training?

    cs.CV 2026-01 conditional novelty 6.0 of 10

    A self-supervised two-stage training method—VAE-latent feature alignment then feature-level classifier-free guidance—lets DiT models match or beat DINO-guided REPA training without any external feature extractor.

  5. MixFlow Training: Alleviating Exposure Bias with Slowed Interpolation Mixture

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Training diffusion models on a mixture of higher-noise interpolations (MixFlow) improves generation FID across SiT, REPA, RAE and SD3.5, reaching ImageNet 256 gFID 1.43 after post-training.

  6. Fitting Image Diffusion Models on Video Datasets

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A shared-noise temporal consistency regularizer for image diffusion training accelerates convergence and lowers FID on the HandCo video dataset.

  7. Amadeus: Autoregressive Model with Bidirectional Attribute Modelling for Symbolic Music

    cs.SD 2025-08 conditional novelty 6.0 of 10

    Amadeus generates symbolic music by autoregressively predicting note-level latents and decoding their attributes in parallel with a masked discrete diffusion model, yielding faster and more controllable generation tha...

  8. Contrastive Conditional-Unconditional Alignment for Long-tailed Diffusion Model

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A two-part training regularizer, an unconditional-only contrastive repulsion plus a large-timestep conditional-unconditional alignment, improves tail-class diversity and fidelity in diffusion models, cutting ImageNet-...

  9. CoGenCast: A Coupled Autoregressive-Flow Generative Framework for Time Series Forecasting

    cs.LG 2026-02 conditional novelty 5.0 of 10

    CoGenCast couples a Qwen-based encoder-decoder with flow matching and reports strong MSE/MAE on ten time-series benchmarks.

  10. Cross-Architecture Distillation Made Simple with Redundancy Suppression

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A simple redundancy-suppression loss, applied to penultimate-layer embeddings, outperforms the complex OFA method for cross-architecture knowledge distillation while adding far fewer parameters.

  11. A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Quality

    cs.CV 2025-07 conditional novelty 3.0 of 10

    A survey of 32 long-video generation papers, presenting a taxonomy and component recommendations for backbones, text encoders, objectives, and positional encodings.

  12. DiverseDiT++: Quantifying, Analyzing, and Promoting Representation Diversity in Diffusion Transformers

    cs.CV 2026-08

Reference graph

Works this paper leans on

45 extracted references · 33 canonical work pages · cited by 12 Pith papers

  1. [1]

    Building normalizing flows with stochastic interpolants

    Michael Samuel Albergo and Eric V anden-Eijnden. Building normalizing flows with stochastic interpolants. In International Conference on Learning Representations (ICLR), 2023. 2

    work page 2023

  2. [2]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu W ei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations. 3

    work page

  3. [3]

    Representation learning: A review and new perspectives

    Y oshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013. 1

    work page 2013

  4. [4]

    SARA: Structural and adversarial representation alignment for training-efficient diffusion models.arXiv preprint arXiv:2503.08253, 2025

    Hesen Chen, Junyan W ang, Zhiyu T an, and Hao Li. SARA: Structural and adversarial representation alignment for training-efficient diffusion models.arXiv preprint arXiv:2503.08253, 2025. 3

    arXiv 2025

  5. [5]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on machine learning, pages 1597–1607. PmLR,

  6. [6]

    Exploring simple siamese representation learning

    Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750–15758, 2021. 3

    work page 2021

  7. [7]

    Learning a similarity metric discriminatively, with application to face verification

    Sumit Chopra, Raia Hadsell, and Y ann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), pages 539–546. IEEE, 2005. 1, 3, 5

    work page 2005

  8. [8]

    ImageNet: A large-scale hierarchical image database

    Jia Deng, W ei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 6

    work page 2009

Show all 45 references
  1. [9]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-W ei Chang, Kenton Lee, and Kristina T outanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human lang...

    2019

  2. [10]

    Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021. 2

    2021

  3. [11]

    One step diffusion via shortcut models

    Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. In The Thirteenth International Conference on Learning Representations. 9

  4. [12]

    Mean flows for one-step generative modeling.arXiv preprint arXiv:2505.13447, 2025

    Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling.arXiv preprint arXiv:2505.13447, 2025. 2, 9, 13

    2025 arXiv

  5. [13]

    Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271–21284,

    Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin T allec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo A vila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural ...

  6. [14]

    Dimensionality reduction by learning an invariant mapping

    Raia Hadsell, Sumit Chopra, and Y ann LeCun. Dimensionality reduction by learning an invariant mapping. In IEEE Conference on Computer V ision and P attern Recognition (CVPR), pages 1735–1742. IEEE, 2006. 1

    2006

  7. [15]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Y uxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020. 1, 3, 4

    2020

  8. [16]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Y anghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. InIEEE Conference on Computer V ision and P attern Recognition (CVPR), 2022. 1, 3

    2022

  9. [17]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022. 6, 9

    2022 arXiv

  10. [18]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1, 2 10

    2020

  11. [19]

    Elucidating the design space of diffusion-based generative models.Advances in neural information processing systems, 35:26565–26577, 2022

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models.Advances in neural information processing systems, 35:26565–26577, 2022. 2

    2022

  12. [20]

    Supervised contrastive learning.Advances in neural information processing systems, 33:18661–18673, 2020

    Prannay Khosla, Piotr Teterwak, Chen W ang, Aaron Sarna, Y onglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning.Advances in neural information processing systems, 33:18661–18673, 2020. 3

    2020

  13. [21]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. 14

    2009

  14. [22]

    Aligning text to image in diffusion models is easier than you think

    Jaa-Y eon Lee, Byunghee Cha, Jeongsol Kim, and Jong Chul Y e. Aligning text to image in diffusion models is easier than you think. arXiv preprint arXiv:2503.08250, 2025. 3

    2025

  15. [23]

    Flow matching for generative modeling

    Y aron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. InThe Eleventh International Conference on Learning Representations. 2

  16. [24]

    Flow matching guide and code.arXiv preprint arXiv:2412.06264,

    Y aron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code.arXiv preprint arXiv:2412.06264,

    arXiv

  17. [25]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations. 2

  18. [26]

    SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric V anden-Eijnden, and Saining Xie. SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers. InEuropean Conference on Computer V ision, pages 23–40. Springer, 2024. 2, 3,...

    2024

  19. [27]

    Slip: Self-supervision meets language-image pre-training

    Norman Mu, Alexander Kirillov, David W agner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. InEuropean conference on computer vision, pages 529–544. Springer, 2022. 3

    2022

  20. [28]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (ICML). PMLR, 2021. 2

    2021

  21. [29]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Y azhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 1, 3, 4

    2018 arXiv

  22. [30]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, V asil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 1, 8, 9

    2023 arXiv

  23. [31]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023. 2, 6, 8

    2023

  24. [32]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong W ook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning (IC...

    2021

  25. [33]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 6

    2022

  26. [34]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InMedical image computing and computer-assisted intervention (MICCAI), 2015. 14

    2015

  27. [35]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric W eiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational conference on machine learning, pages 2256–2265. pmlr, 2015. 1, 2

    2015

  28. [36]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations. 2

  29. [37]

    Improved techniques for training consistency models

    Y ang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The T welfth International Conference on Learning Representations. 9

  30. [38]

    Generative modeling by estimating gradients of the data distribution.Neural Information Processing Systems (NeurIPS), 2019

    Y ang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.Neural Information Processing Systems (NeurIPS), 2019. 1

    2019

  31. [39]

    Understanding contrastive representation learning through alignment and unifor- mity on the hypersphere

    T ongzhou W ang and Phillip Isola. Understanding contrastive representation learning through alignment and unifor- mity on the hypersphere. InInternational conference on machine learning, pages 9929–9939. PMLR, 2020. 4, 5 11

    2020

  32. [40]

    Unsupervised feature learning via non-parametric instance discrimination

    Zhirong Wu, Y uanjun Xiong, Stella X Y u, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3733–3742, 2018. 4

    2018

  33. [41]

    Representation alignment for generation: Training diffusion transformers is easier than you think.arXiv preprint arXiv:2410.06940, 2024

    Sihyun Y u, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think.arXiv preprint arXiv:2410.06940, 2024. 1, 3, 8, 9

    arXiv

  34. [42]

    Barlow twins: Self-supervised learning via redundancy reduction

    Jure Zbontar, Li Jing, Ishan Misra, Y ann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. InInternational conference on machine learning, pages 12310–12320. PMLR, 2021. 5

    2021

  35. [43]

    How mask matters: T owards theoretical understandings of masked autoencoders

    Qi Zhang, Yifei W ang, and Yisen W ang. How mask matters: T owards theoretical understandings of masked autoencoders. Advances in Neural Information Processing Systems, 35:27127–27139, 2022. 3

    2022

  36. [44]

    Inductive moment matching.arXiv preprint arXiv:2503.07565,

    Linqi Zhou, Stefano Ermon, and Jiaming Song. Inductive moment matching.arXiv preprint arXiv:2503.07565,

    arXiv

  37. [2025]

    A Implementation SiT and DiT Experiments.W e faithfully follow the SiT/DiT codebase for ImageNet experiments

    9 12 model S/2 B/2 L/2 XL/2 model configurations params (M) 33 130 458 675 depth 12 12 24 28 hidden dim 384 768 1024 1152 patch size 2 2 2 2 heads 6 12 16 16 training configurations epochs 80 80 80 80 - 1200 batch size 256 optimizer AdamW optimizerβ1 0.9 optimizerβ2 0.95 weigh...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.