Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

REGLUE claims that feeding a diffusion model a compact, nonlinearly compressed copy of a vision model's patch-level semantics—not a linear projection or a global token alone—is what unlocks faster convergence and better FID.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 15:28 UTC pith:WOUZTXV5

load-bearing objection A solid empirical ablation paper on injecting compressed DINOv2 semantics into latent diffusion; the mechanism claims are plausible but not fully disentangled from the smoother-statistics alternative. the 3 major comments →

arxiv 2512.16636 v2 pith:WOUZTXV5 submitted 2025-12-18 cs.CV

REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion

classification cs.CV
keywords latent diffusiondiffusion transformersvision foundation modelsrepresentation alignmentsemantic compressionDINOv2ImageNet generationtraining convergence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Latent diffusion models learn high-level semantics slowly because their denoising objective supervises reconstruction, not meaning. REGLUE's claim is that this can be fixed by feeding the diffusion backbone a compact, spatially structured copy of a vision foundation model's patch-level features alongside the image latents—and that the way those features are compressed is decisive. A lightweight convolutional autoencoder, trained once offline on frozen DINOv2 features and then kept frozen, reduces the full multi-layer semantic map to 16 channels that are noised and added token-wise to the VAE latents; a global [CLS] token and an external alignment loss are added as smaller complementary signals. On ImageNet 256x256 this takes SiT-B/2 from 33.0 to 12.9 FID and SiT-XL/2 to 2.5 FID at 1M steps, reaching the 1M-step quality of two prior methods with a fraction of their iterations. If right, a cheap frozen preprocessing stage replaces a large share of diffusion training time.

Core claim

REGLUE rests on one central claim: patch-level VFM semantics, not global tokens or external alignment, are the primary engine of faster, better latent diffusion, provided they enter through a nonlinear compressor that preserves spatial structure. The ablation ladder: global [CLS] alone gives 25.7 FID; linear PCA patch features give 21.4; the same features through REGLUE's compressor give 14.3; adding multi-layer aggregation, the global token, and REPA-style alignment reaches 12.9 FID on SiT-B/2 at 400K. On SiT-XL/2, 200K steps reach 4.6 FID (REG: 5.0), 700K match REG's 1M-step 2.7, and 1M reach 2.5. Reading: spatial semantics are crucial, nonlinear compression unlocks them, global tokens and

What carries the argument

The load-bearing object is the frozen semantic compressor: a shallow convolutional autoencoder (input Conv 3072 to 256, one residual block, output Conv 256 to 16, symmetric decoder) trained offline with pure MSE to reconstruct channel-concatenated features of the last four DINOv2-B blocks. It produces a 16-channel, 16x16 spatially structured map that is resampled to the 4x32x32 VAE latent grid, noised under the same interpolant schedule, and added token-wise to the VAE latents inside the SiT backbone, so the transformer models latents and semantics as one sequence without extra sequence length. A global [CLS] token is concatenated as a separate token, and an auxiliary cosine alignment loss t

Load-bearing premise

The load-bearing premise is that a 16-channel representation trained only to reconstruct frozen DINOv2 features, with no coupling to the generative loss, stays semantically informative after being noised and added onto the image latents; the paper supports this indirectly with probing and segmentation accuracy, but the same pipeline gives 18.1 FID with CLIP-L features and 16.9 FID when early VFM layers are used.

What would settle it

Train SiT-B/2 with the same additive 16-channel semantic stream, but replace the frozen DINOv2-reconstruction compressor's output with channel-matched noise, or with features whose semantic content is destroyed by shuffling patch positions across images. If FID stays near 12.9 instead of rising toward the 21.4 of the linear-PCA baseline, semantic preservation is not what drives the gain.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Semantic guidance can be bought at preprocessing time: the compressor trains in under one hour on 8 A100s and is then frozen, so the gain costs almost nothing at diffusion training or inference time.
  • Training budgets for class-conditional ImageNet diffusion shrink materially: SiT-XL/2+REGLUE at 200K steps (4.6 FID) beats a 7M-step SiT-XL/2 baseline (8.3 FID) and matches or beats REG at equal or far fewer steps.
  • Nonlinear compression rather than dimensionality per se is the mechanism: at the same 8 channels, the learned compressor yields 14.3 FID and 67.1 mIoU on Cityscapes against 21.4 FID and 59.1 mIoU for PCA features.
  • The method transfers to data-limited regimes: at 20% of ImageNet, REGLUE outperforms REG by 5.5 FID at the same 80-epoch budget, so semantic entanglement helps most when training data is scarce.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the compressor is trained purely for reconstruction and then frozen, the framework implies a decoupling: any frozen dense visual encoder could be wrapped the same way. A direct test would be sweeping VFMs and layer sets to map which feature statistics transfer; the paper's CLIP-L result (18.1 FID vs 12.9 for DINOv2-B) suggests contrastive language-image features carry a different trade-off
  • The success of adding semantics onto latents token-wise suggests an interpretation of 'entanglement' as a learned prior that constrains the latent trajectory; if so, the same additive stream could be used as a controllable conditioning channel at sampling time, e.g., to steer generation toward a target patch-level semantic map.
  • The 16-channel sweet spot (performance degrades at 20 channels) points to a capacity-balance law between reconstruction latents and semantic latents; a principled rule for choosing that dimension per VFM and per VAE would generalize REGLUE beyond ImageNet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes REGLUE, a latent diffusion framework that jointly models VAE image latents, compact patch-level (local) VFM semantics, and a global [CLS] token inside a single SiT backbone. A lightweight convolutional autoencoder is trained offline, with an MSE objective, to compress multi-layer DINOv2 features into a 16-channel spatially structured representation; this representation is then noised with the same interpolant schedule as the VAE latents, added token-wise to the latent embedding sequence, and jointly denoised with a multimodal velocity objective. An optional REPA-style external alignment loss is added at an intermediate SiT block. On ImageNet 256x256, the method reports consistent FID improvements over SiT-B/2 and SiT-XL/2 baselines and over REPA, ReDi, and REG at matched training budgets, with the sharpest claim being that SiT-XL/2+REGLUE reaches the 1M-step performance of ReDi and REG using less than 30% and 80% of their iterations, respectively. The paper also includes ablations on compression channels, compressor capacity, VFM layer selection, loss variants, and VFM choice.

Significance. If the reported gains are robust, the paper makes a practically important contribution: a sub-hour-trained, frozen semantic compressor that is nearly parameter-free at diffusion inference time yields state-of-the-art-class FID on ImageNet 256x256 at a substantially reduced training budget. The experimental matrix in Table 1 is systematic and internally consistent, the arithmetic of the reported percentages checks out, and the code release supports reproducibility. The attentive-probing and Cityscapes mIoU analyses (Figs. 4 and 6) are a genuine effort to connect compression quality to generation quality, and the comparison between nonlinear compression and linear PCA is a clean, large-contrast experiment. The main weakness is that the causal link between 'semantic preservation' and 'generation usefulness' is asserted rather than tested: the paper does not rule out that the nonlinear compressor acts primarily as a learned smoother of feature statistics. This does not undermine the empirical recipe, but it does mean the mechanism claims (a) and (b) are stronger than the evidence supports.

major comments (3)
  1. [Sec. 3.3, Eq. (11); Sec. 4.2, Table 1; Sec. 4.4, Table 7] The central attribution of the 21.4 -> 14.3 FID gain to 'nonlinear compression unlocking semantics' assumes that an MSE-reconstructed VFM feature map, once frozen and noised, is useful to the diffusion model because it preserves semantics. The supporting evidence — attentive probing (Fig. 4), Cityscapes mIoU (Fig. 6), and the MSE+KL degradation (Table 7) — is indirect. It does not exclude the alternative that the compressor is a well-tuned nonlinear smoother of feature statistics and that the diffusion gain would persist (or even increase) with spatial/semantic structure destroyed. I ask for a control experiment: train the identical compressor architecture to reconstruct the same features with their spatial layout destroyed (e.g., channel-wise shuffled or randomly re-permuted patches), or use a fixed random nonlinear projection of the same channels, and run the same SiT-B/2 400K training
  2. [Sec. 3.2, Eqs. (3)-(5)] The shared noise schedule and the channel-wise additive fusion (z_tilde_t + s_tilde_t) are design choices that are load-bearing for the method but are not validated against alternatives. The paper states the two 'straightforward options' for combining latents and semantics but reports only the channel-wise merge, not the sequence-concatenation option (i). Since s* is a learned, unnormalized 16-channel representation, it is not a priori clear that the same (alpha_t, sigma_t) used for the SD-VAE latents is appropriate, nor that adding the projected semantics to the projected latents does not let the 16-channel semantics dominate the 4-channel VAE latents. Please add an ablation of the concatenation option (at least for SiT-B/2, 400K, matching Table 1 setting (l) or (m)) and/or per-modality schedule scaling, so the reader can see whether the reported gains are tied to the fusion mechanism r
  3. [Sec. 4.4, Table 6; Appendix A.3, Table 8] The VFM-dependence results are presented honestly, but the paper's wording sometimes overgeneralizes. Specifically, CLIP-L gives 18.1 FID versus 12.9 for DINOv2-B and 12.3 for DINOv3-B, and early-layer DINOv2 features (3,6,9,12) degrade to 16.9 FID versus 13.3 for last-four layers (9-12). These patterns are consistent with the 'semantics' story but are also consistent with feature statistics or scale differences. At minimum, the conclusion should explicitly bound the claim to self-supervised DINO-style features and note that the method does not improve over all VFMs. If the authors have a hypothesis for why CLIP-L underperforms, it would strengthen the paper to state and test it.
minor comments (6)
  1. [Table 1 caption] The checkmarks denote 'novel components proposed in our work' but do not distinguish, for example, the linear PCA baseline of ReDi from the nonlinear compressor of REGLUE. A clearer labeling (e.g., 'PCA' vs 'Ours') would make the table easier to parse.
  2. [Eq. (7)] 'where \tilde f_*^{(L)} denotes flattened spatial dimension to tokens' is grammatically unclear. Please rewrite, e.g., 'where \tilde f_*^{(L)} is the flattened patch feature sequence.'
  3. [Appendix A.2, Table 7] The MSE+KL and MSE+GAN variants are compared without mentioning whether they use the same random seed or multiple seeds. A single sentence on seed averaging would help assess the 17.2 vs 14.3 gap.
  4. [References] References [15] and [16] both cite the ImageNet paper (Deng et al.). Please merge or disambiguate.
  5. [Sec. 3.3] The notation 'D_s << \sum_l D_l' would be clearer if the sum were defined as the total input channel count, e.g., D_in = \sum_l D_l = 3072 in the default setting, before writing D_s = 16.
  6. [Figs. 4 and 6] The PCA point is labeled 'PCA 8 channels'; it would be helpful to state explicitly that this matches ReDi's default compression, and to mark the number of channels used in the Ours variants in the figure or caption.

Circularity Check

0 steps flagged

No significant circularity: central claims are benchmarked externally and the compressor is trained offline on frozen VFM features.

full rationale

REGLUE's quantitative claims are evaluated against external benchmarks (ImageNet FID/sFID/IS, attentive probing, Cityscapes mIoU) and against baselines that are reproduced in the paper's own tables. The semantic compressor is pretrained offline with a pure MSE reconstruction objective on frozen DINOv2 features (Eq. 11) and then frozen; at sampling, the VAE latents, compressed semantics, and global token are generated jointly from noise, so no fitted input is re-presented as a prediction. The ablations in Table 1 and the convergence comparisons in Tables 2–4 are empirical contrasts, not definitional identities. Self-citations (ReDi [33], attentive probing [52], DINO-foresight [28]) appear as baselines, related work, or measurement tools, but none is load-bearing for the central derivation: the FID improvements do not reduce to those citations. The paper's own limitations and ablations (e.g., MSE+KL degrading FID in Table 7, CLIP-L underperforming in Table 8) are validity/generalizability concerns, not evidence that an output is equivalent to an input by construction. The claim that reconstruction quality implies generation usefulness is an untested empirical assumption, which is a correctness risk, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

Everything the central claim rests on upstream: a frozen DINOv2-B whose features are semantically useful for generation, a frozen SD-VAE whose latents decode to 256x256 images, the SiT velocity-matching formalism, and two paper-specific modeling choices — a shared noise schedule across three modalities of different scale, and token-wise additive fusion of latents with compressed semantics. Six design choices (channels, hidden size, layer set, loss weights, alignment block, CFG settings) are fixed by FID ablations on the evaluation benchmark itself. No physical entities are postulated; the novel learned object is the 16-channel compressed semantic space, which does carry independent evidence via probing and segmentation.

free parameters (6)
  • compression channels = 16
    Ablated in Fig. 5; chosen for best ImageNet FID (4 too lossy, 20 degrades).
  • compressor hidden size = 256
    Ablated in Table 5; 128 degrades FID, 512 gives no gain at double compute.
  • VFM layers aggregated = blocks 9-12 of DINOv2-B
    Ablated in Table 6; blocks 3,6,9,12 degrade FID (16.9 vs 13.3).
  • loss weights lambda_s, lambda_cls, lambda_rep = 1, 0.03, 0.5
    Set in Sec. 4.1; no sensitivity ablation reported. lambda_cls=0.03 makes the global-token term nearly negligible in the total loss.
  • alignment block index k = 4 (SiT-B/2), 8 (SiT-XL/2)
    Taken from the REPA protocol [73]; not ablated.
  • CFG scale and interval = w=2.8, [0,0.9]
    From [35]; Table 10 shows w=2.7/2.8/2.9 all give 1.53 FID, so the choice is not sensitive.
axioms (5)
  • domain assumption DINOv2-B patch features from the last four layers carry semantic content useful for generating ImageNet images
    Core input to the whole method (Secs. 3.2-3.3); validated empirically via probing, but the method's value is VFM-dependent (CLIP-L gives only 18.1 FID, Table 8).
  • domain assumption Frozen SD-VAE-FT-EMA latents are a sufficient decode space for 256x256 ImageNet
    Standard in SiT/DiT pipelines (Sec. 4.1); the method inherits this from [44, 56].
  • standard math Velocity-matching objective for stochastic interpolants trains a generative model
    Adopted from SiT [44] (Eq. 2); the paper claims no new math here.
  • ad hoc to paper A single shared noise schedule (alpha_t, sigma_t) is appropriate for three modalities of different data scale
    Eq. 3 applies the same interpolant to VAE latents, compressed semantics, and CLS token; no normalization or scale alignment of s/cls vs z is discussed, so per-modal SNR may differ at a given t.
  • ad hoc to paper Channel-wise additive fusion z_t + s_t preserves both VAE and semantic information
    Eq. 5 replaces sequence concatenation with token-wise addition to avoid a 2x attention cost; no analysis of information loss or interference between modalities is provided.
invented entities (2)
  • 16-channel compressed semantic space (output of E_psi) independent evidence
    purpose: Compact spatial representation of multi-layer DINOv2 patch features, jointly denoised with VAE latents.
    Validated outside the diffusion objective by attentive probing accuracy (ImageNet, Fig. 4) and Cityscapes segmentation mIoU (Fig. 6); these are falsifiable external handles.
  • Entangled joint diffusion state (z_t, s_t, cls_t) no independent evidence
    purpose: Single diffusion state over latents plus semantics plus global token, denoised jointly.
    Internal modeling construct; no external observable distinguishes it from learning a joint prior over VFM features of the training distribution.

pith-pipeline@v1.3.0-alltime-deepseek · 20631 in / 23409 out tokens · 215811 ms · 2026-08-03T15:28:52.232708+00:00 · methodology

0 comments
read the original abstract

Latent diffusion models (LDMs) achieve state-of-the-art image synthesis, yet their reconstruction-style denoising objective provides only indirect semantic supervision: high-level semantics emerge slowly, requiring longer training and limiting sample quality. Recent works inject semantics from Vision Foundation Models (VFMs) either externally via representation alignment or internally by jointly modeling only a narrow slice of VFM features inside the diffusion process, under-utilizing the rich, nonlinear, multi-layer spatial semantics available. We introduce REGLUE (Representation Entanglement with Global-Local Unified Encoding), a unified latent diffusion framework that jointly models (i) VAE image latents, (ii) compact local (patch-level) VFM semantics, and (iii) a global (image-level) [CLS] token within a single SiT backbone. A lightweight convolutional semantic compressor nonlinearly aggregates multi-layer VFM features into a low-dimensional, spatially structured representation, which is entangled with the VAE latents in the diffusion process. An external alignment loss further regularizes internal representations toward frozen VFM targets. On ImageNet 256x256, REGLUE consistently improves FID and accelerates convergence over SiT-B/2 and SiT-XL/2 baselines, as well as over REPA, ReDi, and REG. Extensive experiments show that (a) spatial VFM semantics are crucial, (b) non-linear compression is key to unlocking their full benefit, and (c) global tokens and external alignment act as complementary, lightweight enhancements within our global-local-latent joint modeling framework. The code is available at https://github.com/giorgospets/reglue .

Figures

Figures reproduced from arXiv: 2512.16636 by Bill Psomas, Christos Sgouropoulos, Giorgos Petsangourakis, Giorgos Sfikas, Ioannis Kakogeorgiou, Theodoros Giannakopoulos.

Figure 1
Figure 1. Figure 1: Overview of REGLUE, Representation Entanglement with Global–Local Unified Encoding. The encoder of a vision foundation model (VFM) provides (i) local patch-level features and (ii) a global image-level feature (i.e. [CLS]). A lightweight semantic compressor (pre-trained offline) maps the patch features to compact spatial semantics. In parallel, a frozen VAE encoder produces image latents. We concatenate the… view at source ↗
Figure 2
Figure 2. Figure 2: REGLUE fast convergence. Qualitative evolution of SiT-B/2+REGLUE at 50K/100K/200K/400K training steps. All use identical noise, the same sampling schedule/step count, and no classifier-free guidance. REGLUE achieves high fidelity early. jointly model patch tokens, only aligning local VFM fea￾tures already provides a strong boost: the original REPA configuration (setting (b)) improves the default SiT-B/2 ba… view at source ↗
Figure 3
Figure 3. Figure 3: Semantic compressor architecture and training. The representations from the last four layers of the vision foundation model (VFM) encoder are concatenated and passed to the compression model, which projects them into a compact 16-channel semantic representation. In our default configuration (corresponding to the middle row of [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Attentive probing accuracy vs. generation quality on ImageNet for different DINOv2 patch-level compression vari￾ants. Each point shows top-1 attentive probing accuracy [52] and FID of the corresponding SiT model, with bubble area propor￾tional to the semantic feature dimensionality. Our non-linear se￾mantic compressors (8 and 16 channels) achieve substantially bet￾ter FID at higher probing accuracy than th… view at source ↗
Figure 5
Figure 5. Figure 5: Performance vs. compression channels. Ablation of the final compression channels, in DINOv2 last layer’s represen￾tation, using SiT-B/2 trained for 400K steps without REPA loss [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Semantic segmentation performance mIoU vs gen￾eration quality for different DINOv2 patch-level compres￾sion variants. Each point shows the segmentation mIoU on Cityscapes [13] using a DPT [55] head on frozen features follow￾ing implementation from [28, 68, 69] and the FID on ImageNet of the corresponding SiT model. Bubble area is proportional to feature dimensionality. Our non-linear semantic compressors (… view at source ↗
Figure 7
Figure 7. Figure 7: Dataset pruning on ImageNet. FID on ImageNet 256×256 for SiT-B/2 trained for 80 epochs on class-balanced subsets (20%, 50%, 100% of ImageNet). REGLUE consistently outperforms REG, with improvements of −5.5, −3.4, and −2.3 FID at 20%, 50%, and 100%, respectively. de-compressor mirrors this design (16→256→3072). The model is fully convolutional, preserves the spatial resolu￾tion, and is trained with an MSE r… view at source ↗
Figure 8
Figure 8. Figure 8: Compressor training. Training curve showing MSE loss over epochs. The compression model utilizes 4 last DINOv2- B layers. The Input layer is 256 and the compression is to 16 channels. A run finishes in less than one hour on 8\times A100 GPUs. B.2. SiT details Architecture settings. We adopt the official SiT config￾urations [44]. The base SiT-B/2 (132M params) uses 12 transformer blocks with embedding dimen… view at source ↗
Figure 9
Figure 9. Figure 9: Uncurated ImageNet 256 × 256 samples. Class-conditional generations from SiT-XL/2+REGLUE trained for 1M steps with CFG (w=4.0). Grids illustrate great fidelity and intra-class diversity. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Uncurated ImageNet 256 × 256 samples. Class-conditional generations from SiT-XL/2+REGLUE trained for 1M steps with CFG (w=4.0). Grids illustrate great fidelity and intra-class diversity. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MaskAlign: Token-Subset Representation Alignment for Efficient Diffusion Training

    cs.CV 2026-06 unverdicted novelty 7.0

    MaskAlign uses random token-subset alignment and pre-mask mixing to reduce diffusion models' reliance on complete clean-image token sets during representation alignment.

  2. Coevolving Representations in Joint Image-Feature Diffusion

    cs.CV 2026-04 unverdicted novelty 7.0

    CoReDi coevolves semantic representations with the diffusion model via a jointly learned linear projection stabilized by stop-gradient, normalization, and regularization, yielding faster convergence and higher sample ...

Reference graph

Works this paper leans on

82 extracted references · 16 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Build- ing normalizing flows with stochastic interpolants

    Michael Samuel Albergo and Eric Vanden-Eijnden. Build- ing normalizing flows with stochastic interpolants. InICLR, 2023

  2. [2]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619–15629, 2023

  3. [3]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 22669–22679, 2023

  4. [4]

    Beit: Bert pre-training of image transformers.arXiv preprint arXiv:2106.08254, 2021

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers.arXiv preprint arXiv:2106.08254, 2021

  5. [5]

    Springer, 2006

    Christopher M Bishop.Pattern recognition and machine learning. Springer, 2006

  6. [6]

    Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G Willcocks. Deep generative modelling: A comparative re- view of V AEs, GANs, normalizing flows, energy-based and autoregressive models.IEEE transactions on pattern analy- sis and machine intelligence, 44(11):7327–7347, 2021

  7. [7]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021

  8. [8]

    Videojam: Joint appearance-motion representations for en- hanced motion generation in video models.arXiv preprint arXiv:2502.02492, 2025

    Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance-motion representations for en- hanced motion generation in video models.arXiv preprint arXiv:2502.02492, 2025

  9. [9]

    Masked autoencoders are effective tokenizers for diffusion models

    Hao Chen, Yujin Han, Fangyi Chen, Xiang Li, Yidong Wang, Jindong Wang, Ze Wang, Zicheng Liu, Difan Zou, and Bhik- sha Raj. Masked autoencoders are effective tokenizers for diffusion models. InICML, 2025

  10. [10]

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017

  11. [11]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on ma- chine learning, pages 1597–1607. PmLR, 2020

  12. [12]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. InCVPR, 2022

  13. [13]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InCVPR, 2016

  14. [14]

    Vision transformers need registers.arXiv preprint arXiv:2309.16588, 2023

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Pi- otr Bojanowski. Vision transformers need registers.arXiv preprint arXiv:2309.16588, 2023

  15. [15]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR, pages 248–255, 2009

  16. [16]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

  17. [17]

    Diffusion models beat GANs on image synthesis

    Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. InNeurIPS, 2021

  18. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

  19. [19]

    Mdtv2: Masked diffusion transformer is a strong image synthesizer.arXiv preprint arXiv:2303.14389, 2023

    Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is a strong image synthesizer.arXiv preprint arXiv:2303.14389, 2023

  20. [20]

    Factor analysis, probabilistic principal component analysis, variational inference, and variational autoencoder: Tutorial and survey.arXiv preprint arXiv:2101.00734, 2021

    Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, and Mark Crowley. Factor analysis, probabilistic principal component analysis, variational inference, and variational autoencoder: Tutorial and survey.arXiv preprint arXiv:2101.00734, 2021

  21. [21]

    Un- supervised representation learning by predicting image rota- tions.arXiv preprint arXiv:1803.07728, 2018

    Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Un- supervised representation learning by predicting image rota- tions.arXiv preprint arXiv:1803.07728, 2018

  22. [22]

    Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271–21284, 2020

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271–21284, 2020

  23. [23]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

  24. [24]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000– 16009, 2022

  25. [25]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017

  26. [26]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

  27. [27]

    What to hide from your students: Attention-guided masked image modeling

    Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yan- nis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What to hide from your students: Attention-guided masked image modeling. InEuropean Conference on Computer Vision, pages 300–318. Springer, 2022

  28. [28]

    DINO-foresight: Looking into the future with DINO

    Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. DINO-foresight: Looking into the future with DINO. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 10

  29. [29]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017

  30. [30]

    Glow: Generative flow with invertible 1x1 convolutions.Advances in neural information processing systems, 31, 2018

    Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions.Advances in neural information processing systems, 31, 2018

  31. [31]

    Ilias: Instance-level image retrieval at scale

    Giorgos Kordopatis-Zilos, Vladan Stojni ´c, Anna Manko, Pavel Suma, Nikolaos-Antonios Ypsilantis, Nikos Efthymi- adis, Zakaria Laskar, Jiri Matas, Ondrej Chum, and Gior- gos Tolias. Ilias: Instance-level image retrieval at scale. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14777–14787, 2025

  32. [32]

    EQ-V AE: Equivariance regularized latent space for improved generative image modeling

    Theodoros Kouzelis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. EQ-V AE: Equivariance regularized latent space for improved generative image modeling. In Forty-second International Conference on Machine Learn- ing, 2025

  33. [33]

    Boosting generative image modeling via joint image-feature synthe- sis

    Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakoge- orgiou, Spyros Gidaris, and Nikos Komodakis. Boosting generative image modeling via joint image-feature synthe- sis. InNeurIPS, 2025

  34. [34]

    Improved precision and recall met- ric for assessing generative models.Advances in neural in- formation processing systems, 32, 2019

    Tuomas Kynk ¨a¨anniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall met- ric for assessing generative models.Advances in neural in- formation processing systems, 32, 2019

  35. [35]

    Applying guidance in a limited interval improves sample and distribution quality in diffusion models

    Tuomas Kynk ¨a¨anniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. InNeurIPS, 2024

  36. [36]

    A path towards autonomous machine intelli- gence version 0.9

    Yann LeCun. A path towards autonomous machine intelli- gence version 0.9. 2, 2022-06-27.Open Review, 62(1):1–62, 2022

  37. [37]

    REPA-E: Unlocking V AE for end-to-end tuning with latent diffusion transform- ers

    Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. REPA-E: Unlocking V AE for end-to-end tuning with latent diffusion transform- ers. InICCV, 2025

  38. [38]

    Autoregressive image generation without vec- tor quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization. InNeurIPS, 2024

  39. [39]

    Feature pyramid networks for object detection

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. InCVPR, 2017

  40. [40]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maxim- ilian Nickel, and Matthew Le. Flow matching for genera- tive modeling. InThe Eleventh International Conference on Learning Representations, 2023

  41. [41]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015

  42. [42]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019

  43. [43]

    Understanding diffusion models: A unified per- spective.arXiv preprint arXiv:2208.11970, 2022

    Calvin Luo. Understanding diffusion models: A unified per- spective.arXiv preprint arXiv:2208.11970, 2022

  44. [44]

    Sit: Exploring flow and diffusion-based generative models with scalable in- terpolant transformers

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable in- terpolant transformers. InECCV, page 23–40, 2024

  45. [45]

    FA V AE-effective frequency aware latent tok- enizer

    Tejaswini Medi, Hsien-Yi Wang, Arianna Rampini, and Mar- gret Keuper. FA V AE-effective frequency aware latent tok- enizer. InNeurIPS 2025 Workshop: Reliable ML from Unre- liable Data, 2025

  46. [46]

    Generating images with sparse representations

    Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.03841, 2021

  47. [47]

    Unsupervised learning of visual representations by solving jigsaw puzzles

    Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. InEuro- pean conference on computer vision, pages 69–84. Springer, 2016

  48. [48]

    Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

  49. [49]

    Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Je- gou, Julien Mairal, Patr...

  50. [50]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205, 2023

  51. [51]

    MIT press, 2023

    Simon JD Prince.Understanding deep learning. MIT press, 2023

  52. [52]

    Attention, please! revisiting attentive probing for masked image modeling.arXiv preprint arXiv:2506.10178, 2025

    Bill Psomas, Dionysis Christopoulos, Eirini Baltzi, Ioan- nis Kakogeorgiou, Tilemachos Aravanis, Nikos Komodakis, Konstantinos Karantzalos, Yannis Avrithis, and Giorgos To- lias. Attention, please! revisiting attentive probing for masked image modeling.arXiv preprint arXiv:2506.10178, 2025

  53. [53]

    Instance-level composed image retrieval

    Bill Psomas, George Retsinas, Nikos Efthymiadis, Panagio- tis Filntisis, Yannis Avrithis, Petros Maragos, Ondrej Chum, and Giorgos Tolias. Instance-level composed image retrieval. InThe Thirty-ninth Annual Conference on Neural Informa- tion Processing Systems, 2025

  54. [54]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, 2021

  55. [55]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. InCVPR, 2021

  56. [56]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. InCVPR, pages 10684– 10695, 2022

  57. [57]

    Improved techniques for training gans.Advances in neural information processing systems, 29, 2016

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans.Advances in neural information processing systems, 29, 2016

  58. [58]

    Laion-5b: An open large-scale dataset for train- ing next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo 11 Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Je- nia Jitsev. Laion-5b: An open large-scale dataset for train- ing next generation image-tex...

  59. [59]

    Latent diffusion model without variational autoen- coder.arXiv preprint arXiv:2510.15301, 2025

    Minglei Shi, Haolin Wang, Wenzhao Zheng, Ziyang Yuan, Xiaoshi Wu, Xintao Wang, Pengfei Wan, Jie Zhou, and Ji- wen Lu. Latent diffusion model without variational autoen- coder.arXiv preprint arXiv:2510.15301, 2025

  60. [60]

    Score-based generative modeling through stochastic differential equa- tions.arXiv preprint arXiv:2011.13456, 2020

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions.arXiv preprint arXiv:2011.13456, 2020

  61. [61]

    Lposs: Label propagation over patches and pixels for open-vocabulary semantic segmentation

    Vladan Stojni ´c, Yannis Kalantidis, Ji ˇr´ı Matas, and Giorgos Tolias. Lposs: Label propagation over patches and pixels for open-vocabulary semantic segmentation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 9794–9803, 2025

  62. [62]

    Any-to-any generation via composable diffu- sion.Advances in Neural Information Processing Systems, 36:16083–16099, 2023

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffu- sion.Advances in Neural Information Processing Systems, 36:16083–16099, 2023

  63. [63]

    Visual autoregressive modeling: Scalable im- age generation via next-scale prediction.arXiv preprint arXiv:2404.02905, 2024

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable im- age generation via next-scale prediction.arXiv preprint arXiv:2404.02905, 2024

  64. [64]

    Probabilistic principal component analysis.Journal of the Royal Statis- tical Society Series B: Statistical Methodology, 61(3):611– 622, 1999

    Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis.Journal of the Royal Statis- tical Society Series B: Statistical Methodology, 61(3):611– 622, 1999

  65. [65]

    Siglip 2: Multilingual vision- language encoders with improved semantic understand- ing, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier H ´enaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision- language encoders with improved semantic understand- ing, localization, and dense featu...

  66. [66]

    DDT: Decoupled diffusion transformer.arXiv preprint arXiv:2504.05741, 2025

    Shuai Wang, Zhi Tian, Weilin Huang, and Limin Wang. DDT: Decoupled diffusion transformer.arXiv preprint arXiv:2504.05741, 2025

  67. [67]

    Representation entanglement for genera- tion: Training diffusion transformers is much easier than you think.NeurIPS, 2025

    Ge Wu, Shen Zhang, Ruijing Shi, Shanghua Gao, Zhenyuan Chen, Lei Wang, Zhaowei Chen, Hongcheng Gao, Yao Tang, Jian Yang, et al. Representation entanglement for genera- tion: Training diffusion transformers is much easier than you think.NeurIPS, 2025

  68. [68]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. InCVPR, 2024

  69. [69]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. InNeurIPS, 2024

  70. [70]

    Fasterdit: Towards faster diffusion transformers train- ing without architecture modification

    Jingfeng Yao, Cheng Wang, Wenyu Liu, and Xinggang Wang. Fasterdit: Towards faster diffusion transformers train- ing without architecture modification. InNeurIPS, 2024

  71. [71]

    Reconstruc- tion vs

    Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruc- tion vs. generation: Taming optimization dilemma in latent diffusion models. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2025

  72. [72]

    Language model beats diffusion - tokenizer is key to visual generation

    Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang. Language model beats diffusion - tokenizer is key to visual generation. InThe Twelfth International Conference on Learning Representati...

  73. [73]

    Representation alignment for generation: Training diffusion transformers is easier than you think

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. InICLR, 2025

  74. [74]

    Dif- fusion models with deterministic normalizing flow priors

    Mohsen Zand, Ali Etemad, and Michael Greenspan. Dif- fusion models with deterministic normalizing flow priors. Transactions of Machine Learning Research, 2024

  75. [75]

    Language- guided image tokenization for generation

    Kaiwen Zha, Lijun Yu, Alireza Fathi, David A Ross, Cordelia Schmid, Dina Katabi, and Xiuye Gu. Language- guided image tokenization for generation. InCVPR, pages 15713–15722, 2025

  76. [76]

    Normalizing flows are capable generative models

    Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, and Josh Susskind. Normalizing flows are capable generative models. InICML, 2025

  77. [77]

    Sigmoid Loss for Language Image Pre- Training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre- Training . InICCV, pages 11941–11952, 2023

  78. [78]

    Diffusion normalizing flow

    Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. InNeurIPS, pages 16280–16291, 2021

  79. [79]

    Pyramid scene parsing network

    Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017

  80. [80]

    Fast training of diffusion models with masked transformers.arXiv preprint arXiv:2306.09305, 2023

    Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers.arXiv preprint arXiv:2306.09305, 2023

Showing first 80 references.