Pith. sign in

REVIEW 3 major objections 5 minor 121 references

Jodi: Unification of Visual Generation and Understanding via Joint Modeling

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a single diffusion transformer with a role switch mechanism learns the joint distribution over the image domain and multiple label domains, and that from this model it can perform joint generation, controllable…

desk verdict Strong generation framework, weak perception evidence: Jodi's role-switch and Joint-1.6M are real contributions, but the understanding claims are undercut by circular evaluation. read the letter →

arxiv 2505.19084 v1 pith:U4RVVOFJ submitted 2025-05-25 cs.CV cs.AIcs.LG

Yifeng Xu , Zhenliang He , Meina Kan , Shiguang Shan , Xilin Chen This is my paper
classification cs.CVcs.AIcs.LG
keywords diffusiontransformerjointmodelingvisualgenerationunderstandingimageperceptioncontrollableroleswitchmechanismmulti-domain
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that visual generation and visual understanding can be unified in a single diffusion model by learning the joint distribution over the image domain and seven label domains (depth, normal, albedo, edge, line art, segmentation, human skeleton). A role switch mechanism randomly assigns each domain as a generation target, a condition input, or an ignored domain during training, so one model learns three families of distributions: joint generation $p(x, y_1, y_2, \ldots)$, controllable generation $p(x \mid y_1, y_2, \ldots)$, and image perception $p(y_1, y_2, \ldots \mid x)$. Built on a linear diffusion transformer with masked linear attention and domain-invariant positional embeddings, the model handles eight domains at roughly 1024x1024 resolution with linear complexity in the number of domains. The paper also contributes Joint-1.6M, a dataset of 200,000 images with auto-generated labels in seven domains and LLM captions. If the approach is correct, a single architecture can both synthesize and perceive dense visual structure.

What carries the argument

The central mechanism is the role switch with masked linear attention and domain-invariant positional embeddings. At each training step every domain is randomly assigned one of three roles—[G] generation target, [C] condition input, [X] ignored—and the Switch module respectively adds noise, keeps values, or zeroes the tokens (Eq. 2). Training with rectified-flow velocity matching (Eq. 3) makes the model learn $p(\{y_m \mid \mathrm{role}_m=[G]\} \mid \{y_m \mid \mathrm{role}_m=[C]\})$ for all role assignments. Masked linear attention (Eq. 4) excludes [X] tokens from the computation and keeps complexity linear in the number of domains, while domain-invariant sinusoidal positional embeddings tie the same spatial location across domains so generated labels stay aligned with the image.

What would settle it

Train a Jodi variant on a perception task using only ground-truth labels (e.g., NYUv2 depth annotations) and compare its test error to the standard model trained on teacher-predicted labels from Joint-1.6M; if the ground-truth-trained variant does not match or beat the teacher-distilled model, the perception claim reduces to imitation of the teacher rather than learned understanding.

Watch

Extended reading notes

Core claim

Jodi learns $p(\{y_m \mid \mathrm{role}_m=[G]\} \mid \{y_m \mid \mathrm{role}_m=[C]\})$ for every random role assignment, which instantiates joint generation, controllable generation, and multi-label perception from one set of weights. The model uses a linear diffusion transformer with flow matching, where [G] tokens are noised and denoised, [C] tokens stay fixed, and [X] tokens are masked out of attention. Domain-invariant positional embeddings give an explicit cue for spatial alignment across domains. The claim is that a single architecture can both synthesize and perceive dense visual structure across eight domains, matching or surpassing specialist models on controllable generation and on depth, normal, albedo, and edge estimation, while enabling novel applications such as joint generation of images and labels and multi-conditional generation.

Load-bearing premise

The perception branch is trained mostly on pseudo-labels produced by specialist models (Depth Anything V2, Lotus, PiDiNet, OneFormer, OpenPose, RGB2X, Informative Drawings), and the same or equivalent models are used as evaluation baselines and condition extractors; if those pseudo-labels are biased or noisy, the reported understanding numbers measure distillation from the teachers rather than genuine visual understanding.

Editorial extensions

If this is right

  • Joint generation produces an image and all seven label maps in one sample, with spatial alignment enforced by domain-invariant positional embeddings.
  • Controllable generation accepts any subset of label domains as conditions; conditioning on multiple labels at once improves both fidelity and faithfulness (for example, FID drops from 13.6 to 10.2 when depth, normal, edge, and line art are combined).
  • Multi-label perception predicts all label domains simultaneously from one image, and ensembling five samples reaches or exceeds single-task accuracy on depth, normal, and segmentation.
  • The model extends to new visual domains by appending tokens and fine-tuning, demonstrated on doodle sketches, pixel, irradiance, and canny maps.
  • The unified model achieves these results with substantially less data (290K images) and compute (8 RTX A6000 for 130K steps) than comparable unified systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The role-switch recipe suggests that any discrete set of visual domains could be added by appending tokens and fine-tuning, so the same model could grow to dozens of label types or coordinate-based representations such as keypoints as numeric tokens rather than RGB maps.
  • Joint generation of image-plus-labels may serve as a synthetic data engine: the model's own outputs could provide paired supervision for other perception models or augment scarce ground-truth datasets.
  • The masked linear-attention trick is a generic complexity lever: if the number of domains grows to tens, only the ignored-domain masking needs to be maintained, so the linear-in-M complexity makes the 8-domain setting a proof of concept rather than an upper limit.
  • A caution the paper does not dwell on: because the same teacher models produce both training labels and evaluation comparisons, the perception numbers likely reflect how well the model distills those teachers; an independent ground-truth benchmark would clarify how much genuine cross-domain understanding the joint model has.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Jodi, a linear diffusion transformer that jointly models the image domain and seven label domains (depth, normal, albedo, edge, line art, segmentation, human skeleton) through a role-switch mechanism. At each training iteration, each domain is randomly assigned a generation, condition, or ignore role, and the model is trained with flow matching to predict the velocity of the generation-target tokens given the condition tokens. This yields three capabilities: joint generation p(x, y1, ...), controllable generation p(x | y1, ...), and image perception p(y1, ... | x). The authors introduce Joint-1.6M, a dataset of 200k images with auto-generated labels from specialist models plus 90k images with ground-truth labels, and report extensive experiments on generation fidelity, condition faithfulness, and perception benchmarks. The paper also demonstrates extensibility to new domains and analyzes the efficiency of linear attention versus full attention.

Significance. If the perception results are independently validated, Jodi would be a notable advance toward a single diffusion model that both synthesizes and understands dense visual structure, with a clean probabilistic framing and a practical linear-attention backbone. The controllable generation results in Table 2 are strong: Jodi achieves the best FID among all listed methods on depth, normal, edge, and lineart conditions, and the visual results in Figures 3 and 4 show coherent multi-domain generation. The paper also contributes a public dataset, a detailed efficiency comparison, and a useful role-switch formulation that others may build on. However, the perception branch is trained on pseudo-labels from the same models that later serve as evaluation baselines and condition extractors. Under this protocol, the perception and LPIPS condition-faithfulness results are compatible with teacher distillation rather than independent visual understanding. The paper's own limitation statements (Section 5, Table 12, Appendix D) acknowledge parts of this issue, but the central 'understanding' claim is not yet established by the current evaluation.

major comments (3)
  1. [Section 3.3 and Section 4.2, Tables 3–8] The perception evaluation is confounded by the teacher models used to auto-label Joint-1.6M. Depth Anything V2 and Lotus generate the depth training labels, and Lotus is a depth/normal baseline in Tables 3 and 4; PiDiNet generates edge labels and is an edge baseline in Table 6; RGB2X generates albedo labels and is an albedo baseline in Table 5; OneFormer generates segmentation labels and is a segmentation baseline in Table 8. In addition, Table 2's LPIPS condition-faithfulness metric re-extracts conditions from generated images, but the extractor models are not specified; if they are the same or equivalent to the label-generation teachers, then low LPIPS can be achieved by learning to invert the teacher's input-output mapping. Under this protocol, the perception results measure consistency with the teachers rather than an independent understanding of visual structure. To support the unification claim, the authors should specify the extractors, evaluate on held-out ground-truth labels (e.g., the real depth maps in NYUv2, ScanNet, and DIODE are already used but are not sufficient), include a direct-distillation baseline trained only to replicate teacher outputs, or vary the teacher set and show that results are not tied to a specific teacher.
  2. [Section 4.2, Tables 3 and 5] The text states that Jodi 'consistently achieves superior or comparable results to the other unified models and specialist models.' This is contradicted by the reported numbers. In Table 3, Jodi (ensemble) has AbsRel 8.3 on NYUv2 and 9.9 on ScanNet, versus Lotus-D's 5.1 and 5.5; only DIODE is comparable. In Table 5, Jodi's albedo PSNR is 15.5 versus RGB2X's 20.6, and LPIPS is 0.31 versus 0.18. These differences are substantial and indicate that Jodi is not competitive with specialist models on these perception tasks. The claim should be restricted to competitiveness among unified models, and the perception results should be reported with appropriate caveats about the pseudo-label training setup.
  3. [Appendix E, Table 8 and Section 5] The semantic segmentation comparison is not a fair test of the perception claim. Jodi is trained to predict 12 manually defined superclasses, while the baseline methods (Uniformer, OneFormer) are trained on the original 150 ADE20K classes; the baselines' predictions are then mapped to the 12 superclasses. The appendix acknowledges this is 'somewhat unfair' to the baselines, yet the table is still used to support the claim of competitive understanding. Because Jodi is solving a much easier task and still achieves mIoU 57.5 versus OneFormer's 83.9, this table provides negative evidence rather than support. A fair comparison would train or adapt the baselines to the same 12-class task, or evaluate Jodi on the original 150 classes (which the current RGB-space design cannot handle, as stated in Section 5). The segmentation results should be reported with this limitation clearly stated in the main text.
minor comments (5)
  1. [Table 6] The header 'IDS' should be 'OIS' (Optimal Image Scale) to match standard edge-detection terminology.
  2. [Section 3.2, Eq. (4)] There is a missing space in the text 'When mj = 0in Eq. (4)'; the equation reference should be formatted as '0 in Eq. (4)'.
  3. [Section 4.2, Table 2 caption] The condition re-extraction models used to compute LPIPS are not specified. Please state which models are used for depth, normal, edge, lineart, and openpose extraction, as this is necessary for reproducibility and for assessing the metric's validity.
  4. [Appendix C, Figure 12] The y-axis label '48VRAM (GiB)' in the first panel appears to be a formatting error; it should read 'VRAM (GiB)'.
  5. [Appendix E, Tables 9 and 10] The footnotes indicate that some baseline numbers are sourced from Lotus and some are evaluated 'by ourselves following the Lotus protocol,' but the protocol details (e.g., image preprocessing, metric implementation) are not provided. Please include these details for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Jodi's perception and generation claims are evaluated against independent ground-truth benchmarks and are not forced by construction.

full rationale

The paper's central derivation is the standard probabilistic identity that the joint distribution p(x, y) entails the marginal and conditional distributions used for generation and perception, and the role-switch objective in Eq. (1)-(3) is a direct training objective for a family of conditionals, not a renamed version of the evaluation metrics. The main circularity concern raised by a skeptical reader is that Joint-1.6M is auto-labeled by specialist teachers (Depth Anything V2, Lotus, PiDiNet, RGB2X, OneFormer, OpenPose, Informative Drawings) and that the same or equivalent models appear as baselines and condition extractors. That overlap would be problematic if the evaluation measured only agreement with those teachers, but the perception results are reported on independent ground-truth benchmarks: NYUv2, ScanNet, DIODE, iBims, Hypersim, BSDS500, and ADE20K. On those benchmarks Jodi is frequently worse than the teacher baselines (e.g., depth AbsRel 10.1 vs. Lotus-D 5.1 on NYUv2; albedo PSNR 15.5 vs. RGB2X 20.6 on Hypersim; segmentation mIoU 57.5 vs. OneFormer 83.9 on ADE20K), so the results do not reduce by construction to reproducing the teachers' outputs. The controllable-generation LPIPS/FID protocol re-extracts conditions using standard specialist models, but this is a standard faithfulness metric and does not constitute fitting the metric: the training loss in Eq. (3) is flow matching on latent tokens, with no teacher-extractor loss. The paper explicitly acknowledges pseudo-label limitations and the absence of ground-truth multi-label supervision in Appendix E's Table 12 discussion, which indicates transparency rather than a hidden circular derivation. No load-bearing self-citation, uniqueness theorem, or ansatz-smuggling citation is present; the cited Sana backbone and external specialists are used as standard building blocks and baselines. The mathematical claim connecting joint distributions to conditionals is an external probabilistic fact, and the model's ability to perform joint generation, controllable generation, and perception follows from the role-switch training distribution. Therefore, the paper's central claims are self-contained against external benchmarks and no circular step is exhibited.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central method rests on standard flow-matching mathematics plus domain assumptions about pseudo-label quality, RGB representation, and role sampling. No new physical entities are introduced.

free parameters (4)
  • Classifier-free guidance scale = 4.5
    Chosen for inference, not swept, and it affects all generation results.
  • Number of segmentation superclasses = 12
    Hand-defined grouping of ADE20K's 150 classes into 12 RGB superclasses, which limits segmentation granularity and affects mIoU comparisons.
  • Ensemble sampling count = 5
    Perception results are reported with five samples; ensembling improves numbers, so single-sample performance is the base method.
  • Role assignment distribution
    Not specified in the paper; random role sampling is central to training, and the exact distribution affects the balance of learned conditionals.
assumptions (5)
  • standard math Joint distribution identities: p(x,y) determines p(x), p(y), p(x|y), and p(y|x)
    Invoked in the introduction to motivate joint modeling; a standard probabilistic identity, not a contribution.
  • standard math Flow matching and rectified flow objectives learn the target conditional distributions
    Equation (3) assumes the velocity prediction loss trains the model to approximate the denoising conditionals.
  • domain assumption Pseudo-labels from specialist models are adequate supervision for perception tasks
    Section 3.3 builds Joint-1.6M with automatic labels; the perception claims depend on the accuracy of these labels.
  • domain assumption Text and labels are conditionally independent given the image
    Appendix B sets text to empty for perception tasks; if this assumption is false, perception training and inference are mismatched.
  • domain assumption Linear attention preserves the inter-domain correspondence needed for spatial alignment
    Section 3.2 selects linear attention for computational efficiency, but the consistency claims rely on it retaining cross-domain alignment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Jodi: Unification of Visual Generation and Understanding via Joint Modeling." pith.science (2026). https://pith.science/paper/U4RVVOFJ

@misc{pith2026250519084,
  author       = {Pith},
  title        = {Pith review of: Jodi: Unification of Visual Generation and Understanding via Joint Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U4RVVOFJ}},
  note         = {Machine review of arXiv:2505.19084}
}
read the original abstract

Visual generation and understanding are two deeply interconnected aspects of human intelligence, yet they have been traditionally treated as separate tasks in machine learning. In this paper, we propose Jodi, a diffusion framework that unifies visual generation and understanding by jointly modeling the image domain and multiple label domains. Specifically, Jodi is built upon a linear diffusion transformer along with a role switch mechanism, which enables it to perform three particular types of tasks: (1) joint generation, where the model simultaneously generates images and multiple labels; (2) controllable generation, where images are generated conditioned on any combination of labels; and (3) image perception, where multiple labels can be predicted at once from a given image. Furthermore, we present the Joint-1.6M dataset, which contains 200,000 high-quality images collected from public sources, automatic labels for 7 visual domains, and LLM-generated captions. Extensive experiments demonstrate that Jodi excels in both generation and understanding tasks and exhibits strong extensibility to a wider range of visual domains. Code is available at https://github.com/VIPL-GENUN/Jodi.

Figures

Figures reproduced from arXiv: 2505.19084 by the authors.

Figure 1
Figure 1. Our Jodi framework is capable of performing (a) joint generation, (b) controllable generation, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our Jodi framework. For the sake of clarity, only four domains are illustrated. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Joint generation of images and labels across a wide range of aspect ratios. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Visual comparison of controllable generation. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Visual comparison of image perception tasks on in-the-wild images. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Effect of positional embeddings. Generated labels are overlaid on images for a better view. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Visualization of attention map. A young woman with red hair and a red lipstick. Joint Generation Controllable Generation Image Perception [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Jodi shows consistency among joint generation, controllable generation, image perception. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Joint generation results of our model extended to new domains. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Detailed architecture of Jodi. For the sake of clarity, only four domains are illustrated. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: The probabilistic graph of text, image, and labels. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Comparison of actual computational cost among three types of attention. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Jodi shows consistency among joint generation, controllable generation, image perception. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Additional visual results of joint generation. [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Additional visual comparisons of controllable generation. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Additional visual results of controllable generation using depth, normal, or albedo as input. [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Additional visual results of controllable generation using edge, lineart, segmentation as input. [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: Additional visual comparisons of single-label perception on in-the-wild images. [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]
Figure 19
Figure 19. Figure 19: Additional visual results of multi-label perception on in-the-wild images. [PITH_FULL_IMAGE:figures/full_fig_p027_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

121 extracted references · 32 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Building normalizing flows with stochastic inter- polants

    Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic inter- polants. In The Eleventh International Conference on Learning Representations, 2023

    2023

  3. [3]

    SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation

    Ellie Arar, Yarden Frenkel, Daniel Cohen-Or, Ariel Shamir, and Yael Vinker. Swiftsketch: A diffusion model for image-to-vector sketch generation. arXiv preprint arXiv:2502.08642, 2025

    work page Pith review arXiv 2025

  4. [4]

    Contour detection and hierarchical image segmentation

    Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):898–916, 2010

    2010

  5. [5]

    ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022

    arXiv 2022

  6. [6]

    One transformer fits all distributions in multi-modal diffusion at scale

    Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. In International Conference on Machine Learning, pages 1692–1717. PMLR, 2023

    2023

  7. [7]

    BlackForestLab. Flux.1. https://blackforestlabs.io/flux-1/

  8. [8]

    Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y . A. Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

    2019

Show all 121 references
  1. [9]

    Intrinsic image decomposition via ordinal shading

    Chris Careaga and Ya˘gız Aksoy. Intrinsic image decomposition via ordinal shading. ACM Transactions on Graphics, 43(1):1–24, 2023

    2023

  2. [10]

    Colorful diffuse intrinsic image decomposition in the wild

    Chris Careaga and Ya ˘gız Aksoy. Colorful diffuse intrinsic image decomposition in the wild. ACM Transactions on Graphics (TOG), 43(6):1–12, 2024

    2024

  3. [11]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213–229, 2020

    2020

  4. [12]

    Artists as experts in visual cognition: An update

    Rebecca Chamberlain, Jennifer E Drake, Aaron Kozbelt, Rachel Hickman, Joseph Siev, and Johan Wagemans. Artists as experts in visual cognition: An update. Psychology of Aesthetics, Creativity, and the Arts, 13(1):58, 2019

    2019

  5. [13]

    Learning to generate line drawings that convey ge- ometry and semantics

    Caroline Chan, Frédo Durand, and Phillip Isola. Learning to generate line drawings that convey ge- ometry and semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7915–7925, 2022

    2022

  6. [14]

    Pixart- σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pages 74–91. Sprin...

    2024

  7. [15]

    Deep compression autoencoder for efficient high-resolution diffusion models

    Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  8. [16]

    Anydoor: Zero-shot object-level image customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6593–6602, 2024

    2024

  9. [17]

    Unireal: Universal image generation and editing via learning real-world dynamics

    Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao, Yilin Wang, et al. Unireal: Universal image generation and editing via learning real-world dynamics. arXiv preprint arXiv:2412.07774, 2024

    2024 arXiv

  10. [18]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025. 10

    2025 arXiv

  11. [19]

    Idadapter: Learning mixed features for tuning-free personalization of text-to-image models

    Siying Cui, Jia Guo, Xiang An, Jiankang Deng, Yongle Zhao, Xinyu Wei, and Ziyong Feng. Idadapter: Learning mixed features for tuning-free personalization of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 950–959, 2024

    2024

  12. [20]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017

    2017

  13. [21]

    Flashattention: Fast and memory- efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344– 16359, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344– 16359, 2022

    2022

  14. [22]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  15. [23]

    Nice: Non-linear independent components estimation

    Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014

    2014 arXiv

  16. [24]

    Evaluative and generative modes of thought during the creative process

    Melissa Ellamil, Charles Dobson, Mark Beeman, and Kalina Christoff. Evaluative and generative modes of thought during the creative process. Neuroimage, 59(2):1783–1794, 2012

    2012

  17. [25]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthes...

    2024

  18. [26]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  19. [27]

    The surprisingly powerful influence of drawing on memory

    Myra A Fernandes, Jeffrey D Wammes, and Melissa E Meade. The surprisingly powerful influence of drawing on memory. Current Directions in Psychological Science, 27(5):302–308, 2018

    2018

  20. [28]

    Univg: A generalist diffusion model for unified image generation and editing

    Tsu-Jui Fu, Yusu Qian, Chen Chen, Wenze Hu, Zhe Gan, and Yinfei Yang. Univg: A generalist diffusion model for unified image generation and editing. arXiv preprint arXiv:2503.12652, 2025

    2025 arXiv

  21. [29]

    Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image

    Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In European Conference on Computer Vision, pages 241–258. Springer, 2024

    2024

  22. [30]

    pexels-portrait

    gaunernst. pexels-portrait. https://huggingface.co/datasets/gaunernst/pexels-portrait

  23. [31]

    Rich feature hierarchies for accurate object detection and semantic segmentation

    Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014

    2014

  24. [32]

    Generative adversarial nets

    Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014

    2014

  25. [33]

    Pulid: Pure and lightning id customization via contrastive alignment

    Zinan Guo, Yanze Wu, Chen Zhuowei, Peng Zhang, Qian He, et al. Pulid: Pure and lightning id customization via contrastive alignment. Advances in neural information processing systems, 37:36777– 36804, 2024

    2024

  26. [34]

    Svdiff: Compact parameter space for diffusion fine-tuning

    Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323–7334, 2023

    2023

  27. [35]

    Transformer language models without positional encodings still learn positional information

    Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1382–1390, 2022

    2022

  28. [36]

    Lotus: Diffusion-based visual foundation model for high-quality dense prediction

    Jing He, Haodong LI, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, and Ying-Cong Chen. Lotus: Diffusion-based visual foundation model for high-quality dense prediction. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  29. [37]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 11

    2017

  30. [38]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  31. [39]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017

    2017

  32. [40]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  33. [41]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  34. [42]

    Oneformer: One transformer to rule universal image segmentation

    Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2989–2998, 2023

    2023

  35. [43]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019

    2019

  36. [44]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  37. [45]

    The impact of positional encoding on length generalization in transformers

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36:24892–24928, 2023

    2023

  38. [46]

    Repurposing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9492–9502, 2024

    2024

  39. [47]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  40. [48]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  41. [49]

    Evaluation of cnn-based single- image depth estimation methods

    Tobias Koch, Lukas Liebel, Friedrich Fraundorfer, and Marco Korner. Evaluation of cnn-based single- image depth estimation methods. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018

    2018

  42. [50]

    Intrinsic image diffusion for indoor single- view material estimation

    Peter Kocsis, Vincent Sitzmann, and Matthias Nießner. Intrinsic image diffusion for indoor single- view material estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5198–5208, 2024

    2024

  43. [51]

    Artists as experts in visual cognition

    Aaron Kozbelt. Artists as experts in visual cognition. Visual cognition, 8(6):705–723, 2001

    2001

  44. [52]

    Imagenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012

    2012

  45. [53]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1931–1941, 2023

    1931

  46. [54]

    One diffusion to generate them all

    Duong H Le, Tuan Pham, Sangho Lee, Christopher Clark, Aniruddha Kembhavi, Stephan Mandt, Ranjay Krishna, and Jiasen Lu. One diffusion to generate them all. arXiv preprint arXiv:2411.16318, 2024

    2024 arXiv

  47. [55]

    Gradient-based learning applied to document recognition

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  48. [56]

    Exploiting diffusion prior for generalizable dense prediction

    Hsin-Ying Lee, Hung-Yu Tseng, and Ming-Hsuan Yang. Exploiting diffusion prior for generalizable dense prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7861–7871, 2024. 12

    2024

  49. [57]

    Playground v2

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245, 2024

    2024 arXiv

  50. [58]

    Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  51. [59]

    Uniformer: Unified transformer for efficient spatial-temporal representation learning

    Kunchang Li, Yali Wang, Gao Peng, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatial-temporal representation learning. In International Conference on Learning Representations, 2022

    2022

  52. [60]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023

    2023

  53. [61]

    Photomaker: Customizing realistic human photos via stacked id embedding

    Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked id embedding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8640–8650, 2024

    2024

  54. [62]

    Dual diffusion for unified image generation and understanding

    Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. Dual diffusion for unified image generation and understanding. arXiv preprint arXiv:2501.00289, 2024

    2024 arXiv

  55. [63]

    Pixwizard: Versatile image-to-image visual assistant with open-language instructions

    Weifeng Lin, Xinyu Wei, Renrui Zhang, Le Zhuo, Shitian Zhao, Siyuan Huang, Junlin Xie, Peng Gao, and Hongsheng Li. Pixwizard: Versatile image-to-image visual assistant with open-language instructions. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  56. [64]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023

    2023

  57. [65]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023

    2023

  58. [66]

    Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022

    2022 arXiv

  59. [67]

    Came: Confidence- guided adaptive memory efficient optimization

    Yang Luo, Xiaozhe Ren, Zangwei Zheng, Zhuo Jiang, Xin Jiang, and Yang You. Came: Confidence- guided adaptive memory efficient optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume1: Long Papers), pages 4442–4453, 2023

    2023

  60. [68]

    T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4...

    2024

  61. [69]

    Novelai improvements on stable diffusion, 2022

    NovelAI. Novelai improvements on stable diffusion, 2022

    2022

  62. [70]

    pexels-photos-janpf

    opendiffusionai. pexels-photos-janpf. https://huggingface.co/datasets/opendiffusionai/ pexels-photos-janpf

  63. [71]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  64. [72]

    Ld-znet: A latent diffusion approach for text-based image segmentation

    Koutilya Pnvr, Bharat Singh, Pallabi Ghosh, Behjat Siddiquie, and David Jacobs. Ld-znet: A latent diffusion approach for text-based image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4157–4168, 2023

    2023

  65. [73]

    Unicontrol: A unified diffusion model for controllable visual generation in the wild

    Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual generation in the wild. Advances in Neural Information Processing Systems, 36, 2024

    2024

  66. [74]

    Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference on ...

    2021

  67. [75]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 13

    2022

  68. [76]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part ...

    2015

  69. [77]

    Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 225...

    2023

  70. [78]

    Photorealistic text-to- image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in neural informatio...

    2022

  71. [79]

    Ziplora: Any subject in any style by effectively merging loras

    Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. In European Conference on Computer Vision, pages 422–438. Springer, 2024

    2024

  72. [80]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746–760. Spr...

    2012

  73. [81]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  74. [82]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015

    2015

  75. [83]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019

    2019

  76. [84]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021

    2021

  77. [85]

    Pixel difference networks for efficient edge detection

    Zhuo Su, Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian, Matti Pietikäinen, and Li Liu. Pixel difference networks for efficient edge detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5117–5127, 2021

    2021

  78. [86]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015

    2015

  79. [87]

    Ominicontrol: Minimal and universal control for diffusion transformer

    Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Minimal and universal control for diffusion transformer. arXiv preprint arXiv:2411.15098, 2024

    2024 arXiv

  80. [88]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  81. [89]

    Pixel recurrent neural networks

    Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pages 1747–1756. PMLR, 2016

    2016

  82. [90]

    Diode: A dense indoor and outdoor depth dataset

    Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R Walter, et al. Diode: A dense indoor and outdoor depth dataset. arXiv preprint arXiv:1908.00463, 2019

    1908 arXiv

  83. [91]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  84. [92]

    Mmgen: Unified multi-modal image generation and understanding in one go

    Jiepeng Wang, Zhaoqing Wang, Hao Pan, Yuan Liu, Dongdong Yu, Changhu Wang, and Wenping Wang. Mmgen: Unified multi-modal image generation and understanding in one go. arXiv preprint arXiv:2503.20644, 2025

    2025 arXiv

  85. [93]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  86. [94]

    Instantid: Zero-shot identity- preserving generation in seconds

    Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. arXiv preprint arXiv:2401.07519, 2024. 14

    2024 arXiv

  87. [95]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  88. [96]

    VILA-u: a unified foundation model integrating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, and Yao Lu. VILA-u: a unified foundation model integrating visual understanding and generation. In The Thirteenth International Conference on Learn...

    2025

  89. [97]

    Infinite-id: Identity-preserved person- alization via id-semantics decoupling paradigm

    Yi Wu, Ziqiang Li, Heliang Zheng, Chaoyue Wang, and Bin Li. Infinite-id: Identity-preserved person- alization via id-semantics decoupling paradigm. In European Conference on Computer Vision, pages 279–296. Springer, 2024

    2024

  90. [98]

    Omnigen: Unified image generation

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv preprint arXiv:2409.11340, 2024

    2024 arXiv

  91. [99]

    SANA: Efficient high-resolution text-to-image synthesis with linear diffusion transformers

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. SANA: Efficient high-resolution text-to-image synthesis with linear diffusion transformers. In The Thirteenth International Conference on Learning R...

    2025

  92. [100]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. In The Thirteenth International Conference on ...

    2025

  93. [101]

    Holistically-nested edge detection

    Saining Xie and Zhuowen Tu. Holistically-nested edge detection. InProceedings of the IEEE international conference on computer vision, pages 1395–1403, 2015

    2015

  94. [102]

    What matters when repurposing diffusion models for general dense perception tasks? In The Thirteenth International Conference on Learning Representations, 2025

    Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. What matters when repurposing diffusion models for general dense perception tasks? In The Thirteenth International Conference on Learning Representations, 2025

    2025

  95. [103]

    Open- vocabulary panoptic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955–2966, 2023

    2023

  96. [104]

    CtrloRA: An extensible and efficient framework for controllable image generation

    Yifeng Xu, Zhenliang He, Shiguang Shan, and Xilin Chen. CtrloRA: An extensible and efficient framework for controllable image generation. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  97. [105]

    Depth anything v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875–21911, 2024

    2024

  98. [106]

    Stablenormal: Reducing diffusion variance for stable and sharp normal

    Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphics (TOG), 43(6):1–18, 2024

    2024

  99. [107]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721, 2023

    2023 arXiv

  100. [108]

    Diffusionedge: Diffusion proba- bilistic model for crisp edge detection

    Yunfan Ye, Kai Xu, Yuhang Huang, Renjiao Yi, and Zhiping Cai. Diffusionedge: Diffusion proba- bilistic model for crisp edge detection. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 6675–6683, 2024

    2024

  101. [109]

    Rgbx: Image decomposition and synthesis using material- and lighting-aware diffusion models

    Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling- Qi Yan, and Miloš Hašan. Rgbx: Image decomposition and synthesis using material- and lighting-aware diffusion models. In ACM SIGGRAPH 2024 Conference Papers, SIGGRAPH ’24, New...

    2024

  102. [110]

    Diffusion-4k: Ultra-high-resolution image synthesis with latent diffusion models

    Jinjin Zhang, Qiuyu Huang, Junjie Liu, Xiefan Guo, and Di Huang. Diffusion-4k: Ultra-high-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  103. [111]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836– 3847, 2023. 15

    2023

  104. [112]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018

    2018

  105. [113]

    Easycontrol: Adding efficient and flexible control for diffusion transformer

    Yuxuan Zhang, Yirui Yuan, Yiren Song, Haofan Wang, and Jiaming Liu. Easycontrol: Adding efficient and flexible control for diffusion transformer. arXiv preprint arXiv:2503.07027, 2025

    2025 arXiv

  106. [114]

    Diception: A generalist diffusion model for visual perceptual tasks

    Canyu Zhao, Mingyu Liu, Huanyi Zheng, Muzhi Zhu, Zhiyue Zhao, Hao Chen, Tong He, and Chunhua Shen. Diception: A generalist diffusion model for visual perceptual tasks. arXiv preprint arXiv:2502.17157, 2025

    2025

  107. [115]

    Uni-controlnet: All-in-one control to text-to-image diffusion models

    Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  108. [116]

    Unleashing text-to-image diffusion models for visual perception

    Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5729–5739, 2023

    2023

  109. [117]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017

    2017

  110. [118]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Chunting Zhou, LILI YU, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. In The Thirteenth International Conference on Learn...

    2025

  111. [119]

    Unleashing the potential of the diffusion model in few-shot semantic segmentation

    Muzhi Zhu, Yang Liu, Zekai Luo, Chenchen Jing, Hao Chen, Guangkai Xu, Xinlong Wang, and Chunhua Shen. Unleashing the potential of the diffusion model in few-shot semantic segmentation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  112. [120]

    𝐲!# 𝐲"# 𝐲

    Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Xiangyang Zhu, Fu-Yun Wang, Zhanyu Ma, Xu Luo, Zehan Wang, Kaipeng Zhang, Lirui Zhao, Si Liu, Xiangyu Yue, Wanli Ouyang, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-next : Making lumina-t2x stro...

    2024

  113. [2024]

    Association for Computing Machinery

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.