Pith. sign in

REVIEW 4 major objections 4 minor 74 references

UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understanding

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that a single consistency model, shared across image and text denoising trajectories, can accelerate both text-to-image generation and image-to-text understanding while matching or beating specialized baselines.

desk verdict A coherent, trained unified consistency model for both T2I and I2T, with real speedups and honest trade-offs; the text-side fixed-point assumption is the main soft spot, not a fatal flaw. read the letter →

arxiv 2502.05415 v2 pith:ATFJD7V4 submitted 2025-02-08 cs.CV cs.AI

Chenkai Xu , Xu Wang , Zhenyi Liao , Yishun Li , Tianqi Hou , Zhijie Deng This is my paper · ORCID
classification cs.CVcs.AI
keywords consistencymodelsmultimodalgenerationandunderstandingtext-to-imageimage-to-textdiscretedenoisingtrajectorydistillationparalleldecodingaccelerationmaskeddiffusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that one consistency model can serve both text-to-image generation and image-to-text understanding. The authors argue that the two tasks share a single discrete denoising trajectory: image tokens follow masked diffusion, while text tokens follow the parallel decoding trace of an autoregressive language model. On that foundation they distill a unified model, UniCMs, from Show-o with one objective that maps every point on either trajectory to the shared endpoint. If the argument is right, a 1.3B model can draw images in 2--8 steps without classifier-free guidance, beating SD3 on GenEval, ImageReward, and CLIP Score at about one-eighth the sampling time, and can read images back faster than Show-o while improving MMMU accuracy. The practical stake is that fast multimodal behavior does not require separate specialist models.

What carries the argument

The central object is the unified consistency mapping defined over two discrete denoising trajectories: image tokens follow masked diffusion, and text tokens follow the parallel-decoding trace of the autoregressive teacher. That mapping is what lets a single transformer predict the endpoint of either trajectory from any intermediate point, so few-step sampling means following the trajectory to its endpoint. The training uses the total loss of Eq. (6), with trajectory segmentation to keep each consistency step local and logits-based regularization to prevent the model from collapsing to trivial outputs.

What would settle it

Decode a fixed set of MMMU questions with UniCMs in 1-step and 8-step mode and compare the resulting text to Show-o's greedy autoregressive output; if the 1-step outputs frequently contain tokens that the parallel trace never visited, or the number of iterations needed to reach the fixed point on held-out prompts exceeds 16 tokens per block, the text-trajectory premise fails.

Watch

Extended reading notes

Core claim

UniCMs establishes a unified consistency mapping over multimodal discrete trajectories. The image side uses the standard mask-diffusion path that starts from a fully masked token sequence and progressively reveals image tokens; the text side replaces discrete diffusion with the greedy parallel-decoding trajectory of an autoregressive model, arguing that each Jacobi-style iteration fixes at least one token and thus behaves like a denoising step. The model is trained with the objective $L = \mathcal{L}_u^c + \alpha \mathcal{L}_v^c + \beta \mathcal{L}_u^{REG} + \gamma \mathcal{L}_v^{REG}$, where the consistency losses push any intermediate point toward the segment endpoint and the regularizations anchor the model to the teacher's endpoint tokens and logits. On the text-to-image side this yields GenEval 0.638, ImageReward 0.963, and CLIP Score 0.318 in eight steps without CFG, compared with SD3's 0.620, 0.787, and 0.308 in 24 steps; on the understanding side UniCMs reaches MMMU 26.3 at 61.1 tokens per second, against Show-o's 24.6 at 40.3 tokens per second.

Load-bearing premise

The argument depends on the assumption that greedy parallel decoding of an autoregressive multimodal model follows a smooth denoising trajectory that reaches the teacher's fixed point in a small number of rounds, so that a consistency model can faithfully map intermediate points to that endpoint.

Editorial extensions

If this is right

  • Text-to-image generation with UniCMs reaches GenEval 0.638, ImageReward 0.963, and CLIP Score 0.318 in eight steps without classifier-free guidance, while SD3 needs 24 steps to score 0.620, 0.787, and 0.308.
  • Image-to-text generation reaches MMMU 26.3 at 61.1 tokens per second, beating Show-o's 24.6 at 40.3 tokens per second, so long captions and visual reasoning become cheaper.
  • The same unified consistency objective can be applied to any unified model that combines autoregressive text decoding with masked-diffusion image tokens, because both modalities are represented as discrete token trajectories.
  • Trajectory segmentation and two-stage distillation, where the second stage collects trajectories from the model trained in the first stage, improve convergence and enable longer-range consistency at inference.
  • The acceleration comes with a measured trade-off: on NoCaps and Flickr30K captioning, UniCMs scores slightly below Show-o, which the paper attributes to the distillation cost and suggests could be remedied by more advanced trajectories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: applying the same unified objective to a larger base model than the 1.3B Show-o initialization would test whether the speed and quality gains scale with capacity.
  • Extension: because the text-side trajectory is tied to greedy parallel decoding, a natural stress test is to distill from a teacher that samples non-greedily, which would reveal whether a deterministic trace is required for the consistency mapping to hold.
  • Extension: the logits-based image regularization stores the teacher's full prediction distribution along the trajectory, so the method could be adapted to expose confidence calibration in few-step generation, enabling risk-aware sampling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes UniCMs, a unified consistency model that applies consistency distillation to a single multimodal model for both text-to-image generation and image-to-text understanding. The key idea is to represent both modalities as discrete tokens and to construct a unified denoising trajectory: image tokens follow masked-diffusion trajectories, while text tokens follow the parallel-decoding trace of an autoregressive model rather than a discrete diffusion trajectory. The model is initialized from Show-o and fine-tuned with a combined consistency loss, regularizations, and a trajectory-segmentation strategy. Experiments report that UniCMs outperforms SD3 on GenEval, ImageReward, and CLIP Score at about 1/8 of SD3's sampling time, and that it outperforms Show-o on MMMU while decoding text about 1.5x faster.

Significance. The conceptual goal is timely: extending consistency models across modalities in a single architecture could make unified multimodal models substantially more efficient. The paper ships with ablations for segmentation, regularization, and top-k sampling, and it reports results on external benchmarks with an explicit acknowledgment of the T2I/MMU trade-off (e.g., POPE drops while MMMU rises). If validated, the work would be a strong demonstration that consistency distillation can span image and text modalities with a shared objective. The main reservations are that the text-side acceleration mechanism is not directly verified on the distilled student, and several headline differences are small and reported without variance information.

major comments (4)
  1. [Section 3.2-3.3 and Footnote 3] The text-acceleration claim rests on the assumption that the student's parallel-decoding fixed point agrees with Show-o's AR output, but the paper only proves the K≤n+1 bound for the teacher's triangular argmax map. The consistency loss in Eq. (5) aligns the student's distribution at noisy states to its own endpoint distribution, where the endpoint is the teacher's greedy tokens; nothing in the objective forces the student's own fixed point to coincide with Show-o's AR outputs at inference. The paper should report the per-block distribution of iterations-to-fixed-point, the agreement rate between the student's greedy parallel decoding and Show-o's AR decoding, and per-benchmark variances, especially since Table 9 shows POPE dropping from 83.2 to 78.4 and NoCaps/Flickr30K falling. Without such evidence, the 1.5x speedup in understanding tasks may reflect a drifted fixed point rather than a consistency-preserving acceleration.
  2. [Section 3.3 and Appendix D] The training objective is not fully specified because Eq. (4)-(6) define consistency to the full-trajectory endpoints uK and vK, while the described training procedure enforces consistency within segments and to segment endpoints. Appendix D explains the segmentation verbally, but the main text does not give the segmented objective with segment-index notation, nor does it state how the regularization logits targets are combined with segmented consistency losses. Since the segmentation strategy is claimed to be important for convergence, the actual loss used in training should be written out explicitly.
  3. [Tables 1 and 7] The headline claim that UniCMs outperform SD3 on GenEval relies on a difference of 0.018 (0.638 vs 0.620) with no error bars, multiple seeds, or significance tests. The equal-step comparison against Show-o in Table 7 is more convincing, and the ImageReward and CLIP Score differences are larger, but the GenEval claim in particular should be supported by variance estimates. The same holds for the MMMU improvement over Show-o in Table 2 (26.3 vs 24.6), which is a single-point difference on a noisy benchmark.
  4. [Section 4.1 and Appendix E] The claim that the second training stage improves acceleration is plausible, but the stage-2 trajectories are collected from UniCMs* after fine-tuning, and the consistency objective then distills the student's own trajectories. This self-distillation loop is a key part of the method, yet no convergence statistics or fixed-point agreement rates are reported for the stage-2 model. A simple diagnostic, such as the fraction of blocks where the student reaches the teacher's AR output within a bounded number of iterations, would directly test whether the acceleration is consistency-based.
minor comments (4)
  1. [Eq. (6) and Appendix E] Appendix E mentions setting δ=2 following Show-o, but δ does not appear in the total loss in Eq. (6); all loss coefficients should be defined consistently in one place.
  2. [Section 4.2 and References] ImageReward is cited as [63] in the benchmarks paragraph, but the introduction cites Image Reward as [25], which is a different paper (Eagle-2); the citation should be corrected.
  3. [Figure 3] The figure's decoding examples contain repeated 'Show-o Turbo:' labels that appear to be leftover text; this should be cleaned up because it is confusing in a manuscript about UniCMs.
  4. [Section 3.2] The text trajectory starts from 'randomly initialized text tokens' (v0), but the initialization distribution is not specified; please state whether tokens are sampled uniformly, from the model's prior, or from another distribution, since this affects both trajectory collection and reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core claims are benchmarked externally and the self-cited components are not used as definitions of the reported results.

full rationale

UniCMs' central derivation is a consistency-distillation objective over collected multimodal trajectories, and its reported performance is measured against external benchmarks (GenEval, ImageReward, CLIP Score, MMMU, POPE, NoCaps). The consistency loss in Eq. (4)-(6) is a standard distillation objective: it maps points on teacher-collected trajectories to the teacher endpoint, and the final evaluations are not obtained by re-reading those fitted endpoints. The paper does rely on the authors' prior CLLMs work [21] to justify using the parallel-decoding trace as the text denoising trajectory, and on MLCM [62] for trajectory segmentation; however, these are design choices and training techniques rather than definitions of the measured quantities. The Footnote 3 bound "K ≤ n + 1 because there is at least one token being correctly predicted in each iteration" is a mathematical property of greedy parallel decoding for a triangular argmax map, not a fitted statement about the student, and the paper's text-acceleration claim is supported by measured iteration counts and benchmark scores rather than by this bound alone. No equation in the paper defines a predicted benchmark result in terms of the same benchmark result, and no fitted parameter is renamed as a prediction. The acknowledged trade-offs (e.g., POPE dropping from 83.2 to 78.4 while MMMU rises) are reported as empirical observations. Thus, the derivation chain is not circular.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim depends on standard mask-diffusion trajectory assumptions, on CLLMs-style text trajectories, and on several hand-set constants (loss coefficients, CFG scales, segment counts, top-k). No new physical or architectural entities are postulated beyond the trained model itself.

free parameters (5)
  • Loss weights alpha, beta, gamma = alpha=10, beta=40, gamma=200 at 512; beta=20, gamma=100 at 256
    Selected via ablation in Table 5 on the same evaluation benchmarks; they control the balance between consistency and regularizations.
  • CFG scale for trajectory collection = stage 1: 15; stage 2: 1.75 (512) and 10 / 1.5 (256)
    Hand-chosen per training stage; affects teacher trajectories. The student runs without CFG at inference.
  • Trajectory length and segment count = K=32 with 8 segments, then K=16 with 4 segments (512)
    Ablated in Table 4; fewer or more segments degrade quality or acceleration.
  • Top-k sampling size at inference = 200 for 4 steps and 10 for 2 steps at 256 (main 512 values not specified)
    Chosen empirically in Table 3; important for 2-4 step image sampling.
  • Parallel decoding block size = 16 text tokens per block
    Set following CLLMs [21] to get acceleration while preserving quality.
assumptions (5)
  • domain assumption Discrete tokenization for both image and text best preserves language modeling capacity.
    Abstract and Section 3.2 use this to justify discrete image tokens over continuous vectors; it is a modeling choice inherited from VQ-based unified models, not proven here.
  • domain assumption The parallel decoding trace of an autoregressive language model is a valid discrete denoising trajectory for text.
    Section 3.2, Eq. (3); this is the text-side foundation of the unified consistency objective and is imported from CLLMs [21] without a multimodal-specific derivation.
  • domain assumption Masked diffusion steps define a denoising trajectory for image tokens whose endpoint is the clean image.
    Section 3.2, Eq. (2); standard in mask diffusion models (Muse, MaskGIT, Show-o) and used to define image consistency targets.
  • ad hoc to paper Consistency loss plus teacher-logit regularization prevents trivial collapse and preserves generation quality.
    Section 3.3 and Eq. (6); Table 5 shows collapse with zero regularization, so this assumption is empirically load-bearing and specific to the proposed training recipe.
  • standard math Each parallel decoding iteration corrects at least one token, so the trajectory terminates in at most n+1 iterations.
    Footnote 3, Section 3.2; plausible for token-level greedy updates with causal masking, but only sketched, and block-wise decoding is used in training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understanding." pith.science (2026). https://pith.science/paper/ATFJD7V4

@misc{pith2026250205415,
  author       = {Pith},
  title        = {Pith review of: UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ATFJD7V4}},
  note         = {Machine review of arXiv:2502.05415}
}
abstract

Consistency models (CMs) have shown promise in the efficient generation of both image and text. This raises the natural question of whether we can learn a unified CM for efficient multimodal generation (e.g., text-to-image) and understanding (e.g., image-to-text). Intuitively, such a model could be acquired by applying the consistency distillation (CD) to existing unified multimodal models. However, the key challenge is establishing a unified denoising perspective for both image and text generation, which is essential for establishing the consistency mapping. To tackle this, at the representation level, we advocate for discrete tokens for both modalities to best preserve language modeling capabilities. Critically, instead of defining the text denoising trajectory via recent discrete diffusion language modeling principles, we specify it using the parallel decoding trace of an autoregressive language model, benefiting from the latter's superior performance in general text generation tasks. The denoising trajectory of image tokens adheres to standard discrete diffusion. We train our unified consistency models (UniCMs) on these combined multimodal trajectories simultaneously with a unified objective. We introduce a trajectory segmentation strategy to further improve the training convergence. Empirically, in text-to-image generation, UniCMs outperform SD3 on GenEval, Image Reward, and CLIP Score metrics, while requiring only approximately ${1}/{8}$ of the sampling time. Meanwhile, in image-to-text generation, UniCMs surpass Show-o on the MMMU benchmark while being $1.5 \times$ faster at long-sequence generating speed. The code is available at https://github.com/zhijie-group/UniCMs.

Figures

Figures reproduced from arXiv: 2502.05415 by the authors.

Figure 1
Figure 1. 512 × 512 images generated by UniCMs. All images are generated by UniCMs in 4 sampling steps without reliance on classifier-free guidance [19]. This paper aims to address this. We first advocate for discrete tokenization for both modalities at the data representation level, which avoids degraded language modeling abilities. Thus, the core problem boils down to constructing a unified discrete denoising trajectory for… view at source ↗
Figure 2
Figure 2. Illustration of the unified denoising perspective of text and image generation. As shown, the trajectories both display a denoising pattern. The black line denotes the unified abstraction of the multimodal trajectory, and the red lines illustrate the objective of UniCMs—to map an arbitrary point on the sampling trajectory to the same endpoint for both text and image generation. Note that we omit the trajectory segme… view at source ↗
Figure 3
Figure 3. The text sampling trajectory of UniCMs in MMU cases. UniCMs realize acceleration by predicting multiple successive tokens in one iteration and correctly guessing the later tokens. 4.2 Main Results Benchmarks. We evaluate the performance of UniCMs in the T2I task on Human Preference Dataset v2 (HPD) [60], using metrics including Human Preference Score v2 (HPS) [60], ImageReward (IR) [63], and CLIP Score (CS) [18]. In… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Comparison between UniCMs, Show-o, and SD3 in T2I generation at the resolution of 512 × 512. Show-o is shown at 16 steps (using CFG), while UniCMs demonstrates performance at 8, 4, and 2 steps. SD3 results are included for comparison with UniCMs. computational efficien…
Figure 5
Figure 5. Figure 5: shows that UniCMs can efficiently fill in missing parts of an image with high quality in just 2 to 4 steps, based on the given prompt. Meanwhile, [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Visualization of image extrapolation by UniCMs on 256 resolution. From top to bottom are the 2, 4, and 8 steps sampling. B Settings of CFG As shown in [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Visualization of regularization label for image trajectory distillation. For each iteration, we only record the logits of the region converted from the mask to the image token, and finally concatenate them into the regularization logits label. We abuse the θ to denote …
Figure 8
Figure 8. Figure 8: 512 × 512 images generated by UniCMs. From left to right, the images are generated by UniCMs in 2, 4, 8 and 16 sampling steps without CFG. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: 256 × 256 images generated by UniCMs. From left to right, the images are generated by UniCMs in 2, 4, 8 and 16 sampling steps without CFG. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 3 canonical work pages

  1. [1]

    Nocaps: Novel object captioning at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948–8957, 2019

    2019

  2. [2]

    Meissonic: Revitalizing masked generative transformers for efficient high- resolution text-to-image synthesis

    Jinbin Bai, Tian Ye, Wei Chow, Enxin Song, Qing-Guo Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high- resolution text-to-image synthesis. In The Thirteenth International Conference on Learning Representations, 2024

    work page 2024

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  4. [4]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005

    2005

  5. [5]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325, 2022

    2022

  6. [6]

    Muse: Text-to-image generation via masked generative transformers

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023

    arXiv 2023

  7. [7]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023

    arXiv

  8. [8]

    Pixart- δ: Fast and controllable image generation with latent consistency models

    Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart- δ: Fast and controllable image generation with latent consistency models. arXiv preprint arXiv:2401.05252, 2024

    arXiv 2024

Show all 74 references
  1. [9]

    Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation

    Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation. arXiv preprint arXiv:2407.06135, 2024

    2024 arXiv

  2. [10]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    2023

  3. [11]

    Dreamllm: Synergistic multimodal comprehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jian- jian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499, 2023

    2023 arXiv

  4. [12]

    Scaling rectified flow transform- ers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In Forty-first International Conference on Machin...

    2024

  5. [13]

    Video-r1: Reinforcing video reasoning in mllms

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776, 2025

    2025 arXiv

  6. [14]

    Geneval: An object-focused framework for evaluating text-to-image alignment, 2023

    Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment, 2023. URL https://arxiv.org/abs/2310.11513

    2023 arXiv

  7. [15]

    Scaling diffusion language models via adaptation from autoregressive models

    Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.17891, 2024. 10

    arXiv

  8. [16]

    Distil- lation of discrete diffusion through dimensional correlations

    Satoshi Hayakawa, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, and Yuki Mitsufuji. Distil- lation of discrete diffusion through dimensional correlations. arXiv preprint arXiv:2410.08709, 2024

    2024 arXiv

  9. [17]

    Multistep consistency models

    Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv preprint arXiv:2403.06807, 2024

    2024 arXiv

  10. [18]

    Clipscore: A reference-free evaluation metric for image captioning, 2022

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, 2022. URL https://arxiv.org/abs/ 2104.08718

    2022 arXiv

  11. [19]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021

    2021

  12. [20]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  13. [21]

    Cllms: Consistency large language models

    Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: Consistency large language models. arXiv preprint arXiv:2403.00835, 2024

    2024 arXiv

  14. [22]

    Orthus: Autoregressive interleaved image-text generation with modality-specific heads

    Siqi Kou, Jiachun Jin, Zhihong Liu, Chang Liu, Ye Ma, Jian Jia, Quan Chen, Peng Jiang, and Zhijie Deng. Orthus: Autoregressive interleaved image-text generation with modality-specific heads. arXiv preprint arXiv:2412.00127, 2024

    2024 arXiv

  15. [23]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  16. [24]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  17. [25]

    Eagle-2: Faster inference of language models with dynamic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858, 2024

    2024 arXiv

  18. [26]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Ying- fang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405.08748, 2024

    2024 arXiv

  19. [27]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024

    2024 arXiv

  20. [28]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings...

    2014

  21. [29]

    World model on million-length video and language with blockwise ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.08268, 2024

    2024 arXiv

  22. [30]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  23. [31]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  24. [32]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  25. [33]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024. 11

    2024

  26. [34]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507–2...

    2022

  27. [35]

    Latent consistency models: Synthesizing high-resolution images with few-step inference

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023

    2023 arXiv

  28. [36]

    Star: Scale-wise text-to-image generation via auto-regressive representations

    Xiaoxiao Ma, Mohan Zhou, Tao Liang, Yalong Bai, Tiejun Zhao, Huaian Chen, and Yi Jin. Star: Scale-wise text-to-image generation via auto-regressive representations. arXiv preprint arXiv:2406.10797, 2024

    2024 arXiv

  29. [37]

    Scaling up masked diffusion models on text

    Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024

    2024 arXiv

  30. [38]

    Large language diffusion models

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025

    2025 arXiv

  31. [39]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv pr...

    2023 arXiv

  32. [40]

    Plummer, Liwei Wang, Christopher M

    Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. IJCV, 123(1):74–93, 2017

    2017

  33. [41]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations

  34. [42]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  35. [43]

    Hyper-sd: Trajectory segmented consistency model for efficient image synthesis

    Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. arXiv preprint arXiv:2404.13686, 2024

    2024 arXiv

  36. [44]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  37. [45]

    Adversarial diffusion distillation

    Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision, pages 87–103. Springer, 2024

    2024

  38. [46]

    Vlm-r1: A stable and generalizable r1-style large vision-language model

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  39. [47]

    Improved techniques for training consistency models

    Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14189, 2023

    2023 arXiv

  40. [48]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  41. [49]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023. 12

    2023 arXiv

  42. [50]

    Score-based continuous-time discrete diffusion models

    Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. arXiv preprint arXiv:2211.16750, 2022

    2022 arXiv

  43. [51]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024

    2024 arXiv

  44. [53]

    Chameleon: Mixed-modal early-fusion foundation models, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024. URL https://arxiv. org/abs/2405.09818, 9, 2024

    2024 arXiv

  45. [54]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  46. [55]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017

    2017

  47. [56]

    Phased consistency model

    Fu-Yun Wang, Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, et al. Phased consistency model. arXiv preprint arXiv:2405.18407, 2024

    2024 arXiv

  48. [57]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  49. [58]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848, 2024

    arXiv

  50. [59]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023

    2023 arXiv

  51. [60]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023

    2023 arXiv

  52. [61]

    Show-o: One single trans- former to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  53. [62]

    Mlcm: Multistep consistency distillation of latent diffusion model

    Qingsong Xie, Zhenyi Liao, Zhijie Deng, Shixiang Tang, Haonan Lu, et al. Mlcm: Multistep consistency distillation of latent diffusion model. arXiv preprint arXiv:2406.05768, 2024

    2024 arXiv

  54. [63]

    Imagereward: Learning and evaluating human preferences for text-to-image generation,

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation,

  55. [64]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  56. [65]

    Dream 7b, 2025

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025. URL https://hkunlp.github.io/blog/2025/dream

    2025

  57. [66]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2024

  58. [67]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.TACL, 2:67–78, 2014

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.TACL, 2:67–78, 2014

    2014

  59. [68]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2023

  60. [69]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  61. [70]

    Monoformer: One transformer for both diffusion and autoregression

    Chuyang Zhao, Yuxing Song, Wenhao Wang, Haocheng Feng, Errui Ding, Yifan Sun, Xinyan Xiao, and Jingdong Wang. Monoformer: One transformer for both diffusion and autoregression. arXiv preprint arXiv:2409.16280, 2024

    2024 arXiv

  62. [71]

    Trajectory consistency distillation

    Jianbin Zheng, Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding, Dacheng Tao, and Tat-Jen Cham. Trajectory consistency distillation. arXiv preprint arXiv:2402.19159, 2024

    2024 arXiv

  63. [72]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  64. [73]

    Minigpt-4: En- hancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  65. [74]

    Llava-phi: Efficient multi-modal assistant with small language model

    Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-modal assistant with small language model. arXiv preprint arXiv:2401.02330, 2024. 14 A Inpainting and Extrapolation Figure 5 shows that UniCMs can efficiently fill in missing p...

    2024 arXiv

  66. [2023]

    URL https://arxiv.org/abs/2304.05977

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.