Pith. sign in

REVIEW 3 major objections 6 minor 6 cited by

OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OpenUni claims that a frozen multimodal LLM and a frozen diffusion model, joined by one small trainable bridge, can match or beat much larger unified models on text-to-image benchmarks after a short fine-tune on 60,000 teacher images.

desk verdict Useful open baseline; headline GenEval gains likely owe more to distilled training data than to the connector. read the letter →

arxiv 2505.23661 v3 pith:5ZIQXABM submitted 2025-05-29 cs.CV

Size Wu , Zhonghua Wu , Zerui Gong , Qingyi Tao , Sheng Jin , Qinyue Li , Wei Li , Chen Change Loy This is my paper
classification cs.CV
keywords unifiedmultimodalmodeltext-to-imagegenerationlearnablequeriesfrozenLLMdiffusiontransformerGenEvalinstructionalignmentopen-sourcebaseline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OpenUni is a recipe for unifying image understanding and image generation without training a giant model from scratch: keep a multimodal large language model and a diffusion model frozen, and train only 256 learnable queries plus a six-layer transformer connector to translate one into the other. The paper claims this is enough to reach 0.84 on GenEval with only 1.1B activated parameters, matching an 8B unified model, and 0.86 with 3.1B, the best open-source unified-model score on that benchmark in the comparison. The same architecture retains the frozen language model's full understanding performance, so both tasks live in one system. The fragile part is that the GenEval headline depends on a final fine-tune over 60,000 images generated by stronger models; without that data, OpenUni-B-512 scores 0.62 on GenEval.

What carries the argument

The load-bearing mechanism is the learnable-query bridge. During generation, the user prompt is fed to the frozen multimodal LLM; N=256 learnable queries are prepended, and their final hidden states carry the conditioning signal. A six-layer transformer connector, adapted from a vision encoder design, converts those states into tokens that the diffusion transformer consumes via cross-attention. The queries and connector are the only trainable components in Stage 1, and the connector plus diffusion model are unlocked in Stage 2. This is what transfers the LLM's language and world knowledge into the diffusion model without retraining either large module.

What would settle it

Replace the 60,000 example set with an equal-size set whose captions avoid the 'a photo of X and Y' phrasing used by GenEval, retrain the second stage, and re-run the benchmark. If the score falls back close to 0.62, the reported 0.84 was mostly style-specific distillation; if it stays near 0.84, the gain is general instruction-following.

Watch

Extended reading notes

Core claim

The central discovery claimed by the paper is that the gap between a frozen multimodal LLM and a frozen text-to-image diffusion model can be closed by a small trainable bridge: N learnable queries read hidden states of the LLM, a lightweight transformer connector reshapes them into conditioning tokens, and the diffusion model attends to those tokens through its cross-attention. After Stage 1 alignment on 23M recaptioned image-text pairs with both large models frozen, the system can already generate images, and after Stage 2 fine-tuning of the diffusion model on 60,000 teacher-generated images it follows complex prompts well enough to score 0.84 and 0.86 on GenEval, exceed prior unified models on DPG-Bench, and match much larger systems on WISE. The author would phrase the claim as: a simple, parameter-efficient bridge plus high-quality distillation data is a strong baseline for unified multimodal modeling, and most of the expensive parts can be borrowed off the shelf.

Load-bearing premise

The headline generation scores come from a short second training stage on 60,000 example images produced by stronger models, and the key premise is that those examples teach general instruction-following rather than the specific prompt pattern of the GenEval benchmark; if that premise fails, the parameter-efficiency claim is not supported.

Editorial extensions

If this is right

  • If the recipe holds, a strong unified multimodal model can be assembled from off-the-shelf components in roughly 110,000 training steps, with most of the benefit coming from data rather than architecture.
  • The understanding side of the model is inherited unchanged from the frozen multimodal LLM, so unified models can track progress in multimodal understanding by swapping the base model and re-running the cheap alignment.
  • The GenEval result specifically implies that a small model can compete with an 8B model on prompt-following when the training signal is a high-quality teacher dataset, not parameter count.
  • Because OpenUni reports results with and without the distillation set, the paper provides a direct measurement of how much of the benchmark gain is attributable to teacher data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the sharp 0.62-to-0.84 jump on GenEval suggests the benchmark's fixed 'a photo of A and B' template makes it unusually susceptible to distillation; on DPG-Bench the same data buys only about 1.25 points, so gains on free-form prompts are smaller.
  • Editorial inference: if this pattern generalizes, benchmark-specific teacher data is becoming part of the modeling recipe, and reporting both 'with teacher' and 'without teacher' scores, as OpenUni does, should become standard practice for unified generation models.
  • Editorial inference: the architecture's dependence on cross-attention conditioning implies that improvements in the connector or in query design may transfer directly to generation quality, making the bridge itself the natural locus for future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper describes OpenUni, a unified multimodal model formed by a frozen InternVL3 MLLM and a frozen SANA diffusion model connected by 256 learnable queries and a six-layer transformer connector. Training is two-stage: Stage I trains only the connector on 23M LLM-captioned public image-text pairs; Stage II unfreezes the diffusion model and fine-tunes on the 60K-image BLIP3-o distillation set. The authors report three variants (B-512, L-512, L-1024) and evaluate generation on GenEval, DPG-Bench, and WISE and understanding on standard MLLM benchmarks, claiming state-of-the-art GenEval at low activated parameter counts.

Significance. If the GenEval results are robust after leakage controls, OpenUni would be a valuable open baseline: it releases weights, code, and a 23M-pair dataset; the connector is simple and cheap to train; and the authors are transparent about which results depend on distillation data. The paper also includes a useful ablation quantifying the distillation set's contribution. The unresolved question is whether the headline gains measure the proposed architecture or the teacher signal in BLIP3-o-60K; that question is central to the paper's main parameter-efficiency claim.

major comments (3)
  1. [Section 4.1, Table 3] The headline GenEval scores (0.84, 0.85, 0.86) are all marked as obtained after Stage 2 fine-tuning on BLIP3-o-60K, and the paper's own ablation in Table 3 shows OpenUni-B-512 falls from 0.84 to 0.62 without this set. Since BLIP3-o-60K was created by prompting GPT-4o and was already used by BLIP3-o to report GenEval, and since GenEval prompts follow the rigid template 'a photo of A and B', the +0.22 gain cannot be attributed to the learnable-query connector without evidence that the training captions do not overlap or closely paraphrase GenEval prompts. The discussion in Section 4.1 mentions community leakage concerns but provides no such evidence; please add a prompt-overlap analysis, a fine-tune on a non-distilled high-quality dataset of comparable size, and GenEval results with rewritten prompts.
  2. [Section 4.1, Tables 3 and 4] The differential effect of the distillation data on GenEval (0.62 to 0.84) versus DPG-Bench (79.04 to 80.29) suggests that the benefit is concentrated in the narrow, template-like style of GenEval rather than in general instruction following. To test this, report per-category GenEval scores for the model trained without BLIP3-o-60K and with it, and show whether the biggest jumps occur in the categories most sensitive to the fixed template (e.g., position and color attributes). Without this breakdown, the abstract's claim of 'exceptional performance' with 1.1B activated parameters remains supported only by a borrowed teacher signal.
  3. [Related Work, Table 1] The related-work comparison states that OpenUni 'achieves significantly higher performance with fewer learnable parameters' than MetaQuery, but Table 1 lists the connector parameter counts for MetaQuery-L and MetaQuery-XL as 'Unknown'. Please either obtain these numbers from the MetaQuery release or restrict the fewer-parameters claim to the variants for which counts are available.
minor comments (6)
  1. [Abstract and Introduction] In both places, 'OpenUni-L-1024 archives the best performance' should read 'achieves'.
  2. [Section 3.2, Stage 1] State which LLM was used for re-captioning the 23M pairs and provide the captioning template; this is needed to reproduce the released dataset.
  3. [Table 3] Add an explicit caption note that the second OpenUni-B-512 row (0.62) omits BLIP3-o-60K, rather than leaving this to be inferred from the main text.
  4. [Table 4] Clarify in the caption whether the 80.29 OpenUni-B-512 row used the distillation data, since the ¶ footnote is attached only to the 79.04 row.
  5. [Table 6] Use consistent formatting for MME-P: OpenUni-B is listed as 1491.22 while all other entries are integers.
  6. [Section 5] The Limitations list does not mention the dependence of headline GenEval results on distillation data; Section 4.1 discusses it well, but the abstract should also qualify 'exceptional performance'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: OpenUni's GenEval results are empirical benchmark scores with transparent disclosure of the distillation-data ablation, not predictions derived from their own inputs.

full rationale

The paper's central claims are empirical: a frozen InternVL3 MLLM, 256 learnable queries, a six-layer SigLIP-style connector, and a SANA diffusion model are trained in two stages and evaluated on GenEval, DPG-Bench, and WISE. No load-bearing step reduces by definition to its inputs. The architecture is explicitly adopted from MetaQuery rather than derived, and the understanding scores are explicitly attributed to the frozen InternVL3 backbone. The GenEval results are reported with a dagger indicating training on BLIP3-o-60K, and Section 4.1 openly reports that removing this distillation data drops OpenUni-B-512 from 0.84 to 0.62 and that the community has raised data-leakage concerns. This is a transparent benchmark-contamination and attribution risk, but it is not circularity: the paper does not fit a parameter to GenEval, does not define a quantity in terms of the benchmark, and does not present the score as a derivation. The fixed 'a photo of A and B' prompt template is discussed as a gameability concern, not hidden. Self-citations to Harmon and F-LMM appear only in related-work context and are not load-bearing. Therefore no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on design choices (query count, connector depth, training data) and on the assumption that the frozen components can be bridged by a small connector. No new physical or theoretical entities are introduced.

free parameters (4)
  • Number of learnable queries = 256
    Hand-chosen in Section 3.1 without ablation; controls how much conditioning information is extracted from the LLM.
  • Connector depth = 6 transformer layers
    Adapted from SigLIP's visual encoder in Section 3.1; no ablation isolates the effect of depth on generation quality.
  • Classifier-free guidance dropout = 10% empty captions
    Set in Section 3.1 to enable CFG at inference; standard value, but its interaction with the distilled finetune data is not studied.
  • Training hyperparameters = lr 1e-4/1e-5, batch 512/256, steps 100k/10k
    Table 2 lists chosen values without sensitivity analysis; the two-stage recipe depends on them.
assumptions (4)
  • domain assumption Frozen InternVL3 features, elicited by 256 learnable queries during the LLM forward pass, contain enough information to condition SANA for high-quality image generation.
    Section 3.1: the connector translates LLM output features into diffusion conditioning; the paper provides no analysis of information loss or why this bridge is sufficient.
  • domain assumption The SANA diffusion transformer can accept and benefit from connector output through cross-attention.
    Section 3.1: this is the core interface of the architecture, inherited from MetaQuery and SANA without verification in this paper.
  • domain assumption The 23M LLM-recaptioned public images plus the BLIP3-o-60K distilled set are appropriate training data for aligning understanding and generation.
    Section 3.2: dataset composition is listed but quality, deduplication, and leakage against GenEval are not measured.
  • domain assumption Standard benchmarks (GenEval, DPG-Bench, WISE, MMBench, MME-P, MMMU, etc.) truthfully measure the claimed capabilities.
    Section 4: scores are accepted as objective; single-run reporting without error bars limits confidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation." pith.science (2026). https://pith.science/paper/5ZIQXABM

@misc{pith2026250523661,
  author       = {Pith},
  title        = {Pith review of: OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ZIQXABM}},
  note         = {Machine review of arXiv:2505.23661}
}
read the original abstract

In this report, we present OpenUni, a simple, lightweight, and fully open-source baseline for unifying multimodal understanding and generation. Inspired by prevailing practices in unified model learning, we adopt an efficient training strategy that minimizes the training complexity and overhead by bridging the off-the-shelf multimodal large language models (LLMs) and diffusion models through a set of learnable queries and a light-weight transformer-based connector. With a minimalist choice of architecture, we demonstrate that OpenUni can: 1) generate high-quality and instruction-aligned images, and 2) achieve exceptional performance on standard benchmarks such as GenEval, DPG- Bench, and WISE, with only 1.1B and 3.1B activated parameters. To support open research and community advancement, we release all model weights, training code, and our curated training datasets (including 23M image-text pairs) at https://github.com/wusize/OpenUni.

Figures

Figures reproduced from arXiv: 2505.23661 by the authors.

Figure 1
Figure 1. OpenUni delivers strong performance with efficient parameter usage across both [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. OpenUni is able to generate high-quality images and precisely follow user instructions. The above results are [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. OpenUni has inherited InternVL3 [37]’s excellent multimodal understanding ability to recognize visual patterns and comprehend general world knowledge. The results are obtained by OpenUni-L (based on InternVL3-2B). 9 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Test-Time Curriculum for Open-Set AIGC Detection

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A curriculum-based test-time adaptation method, using balanced confident pseudo-labels and multi-scale refinement, improves AIGC detector accuracy on unseen generators by 11 to 29 points over its starting detector.

  2. IB-Flow: Information Bottleneck-Guided CFG Distillation for Few-Step Text-to-Image Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Information-Bottleneck closed forms for adaptive CFG supervisor timestep and strength yield SOTA 2-NFE text-to-image fidelity across FLUX, OpenUni, and Qwen-Image.

  3. Learning Preference-Based Objectives from Clinical Narratives for Dynamic Sepsis Treatment

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    CN-PR learns reward functions from LLM-derived preferences over clinical trajectories to improve RL policies for sequential treatment decisions, showing correlation with quality scores and better recovery outcomes.

  4. Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation

    cs.CV 2026-01 conditional novelty 6.0 of 10

    Adding depth- and segmentation-generation objectives to UMM post-training improved spatial understanding and reduced hallucinations on Harmon and OpenUni while preserving generation quality.

  5. Reconstruction Alignment Improves Unified Multimodal Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    RECA, a self-supervised post-training objective that conditions unified multimodal models on their own visual understanding embeddings to reconstruct input images, improves text-to-image and editing benchmarks across ...

  6. Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A 1.5B unified autoregressive model with separate encoders for generation and understanding reports strong text-to-image and editing scores while running on commodity hardware.

Reference graph

Works this paper leans on

73 extracted references · 7 canonical work pages · cited by 6 Pith papers

  1. [1]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023

    2023

  2. [2]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  3. [3]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  4. [4]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    arXiv 2024

  5. [5]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    2023

  6. [6]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  7. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024

    2024

  8. [8]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    arXiv 2024

Show all 73 references
  1. [9]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InCVPR, 2022

    2022

  2. [11]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. InICLR, 2024

    2024

  3. [13]

    PixArt-Sigma: Weak-to-strong training of diffusion transformer for 4K text-to-image generation.arXiv preprint arXiv:2403.04692, 2024

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. PixArt-Sigma: Weak-to-strong training of diffusion transformer for 4K text-to-image generation.arXiv preprint arXiv:2403.04692, 2024

    2024 arXiv

  4. [14]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InInternational conference on machine learning, pages 8821–8831. Pmlr, 2021

    2021

  5. [15]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  6. [16]

    Improving image generation with better captions.Computer Science

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions.Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    2023

  7. [17]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  8. [18]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  9. [19]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  10. [20]

    Vila-u: a unified foundation model integrating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024. 10

    2024 arXiv

  11. [21]

    Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  12. [22]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    arXiv

  13. [23]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025

    2025

  14. [24]

    Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding.arXiv preprint arXiv:2412.09604, 2024

    Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiaogang Wang, Hongsheng Li, Lewei Lu, et al. Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding.arXiv preprint arXiv:2412.09604, 2024

    2024 arXiv

  15. [25]

    Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    2025 arXiv

  16. [26]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2024

  17. [27]

    Illume: Illuminating your llms to see, draw, and self-enhance.arXiv preprint arXiv:2412.06673, 2024

    Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illuminating your llms to see, draw, and self-enhance.arXiv preprint arXiv:2412.06673, 2024

    2024 arXiv

  18. [28]

    Metamorph: Multimodal understanding and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    2024 arXiv

  19. [29]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

    2025

  20. [30]

    Harmonizing visual representations for unified multimodal understanding and generation.arXiv preprint arXiv:2503.21979, 2025

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Zhonghua Wu, Qingyi Tao, Wentao Liu, Wei Li, and Chen Change Loy. Harmonizing visual representations for unified multimodal understanding and generation.arXiv preprint arXiv:2503.21979, 2025

    2025 arXiv

  21. [31]

    Transfer between modalities with metaqueries, 2025

    Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Ji Hou, and Saining Xie. Transfer between modalities with metaqueries, 2025

    2025

  22. [32]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    2023

  23. [33]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  24. [34]

    Sigmoid loss for language image pre- training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023

    2023

  25. [35]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  26. [36]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  27. [37]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li,...

    2025

  28. [38]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  29. [39]

    Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814, 2024

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814, 2024

    2024 arXiv

  30. [40]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  31. [41]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  32. [42]

    Hunyuan-DiT: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding.arXiv preprint arXiv:2405.08748, 2024

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-DiT: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding.arXiv preprint arXiv:2405.08748, 2024

    2024 arXiv

  33. [43]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  34. [44]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InMedical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part I...

    2015

  35. [45]

    Scaling rectified flow transformers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transf...

    2024

  36. [46]

    Sana: Efficient high-resolution image synthesis with linear diffusion transformers, 2024

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. Sana: Efficient high-resolution image synthesis with linear diffusion transformers, 2024

    2024

  37. [47]

    Flux.https://github.com/black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/black-forest-labs/flux, 2024

    2024

  38. [48]

    Sit: Exploring flow and diffusion-based generative models wfith scalable interpolant transformers

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models wfith scalable interpolant transformers. InEuropean Conference on Computer Vision, pages 23–40. Springer, 2024

    2024

  39. [49]

    Lumina-next: Making lumina-t2x stronger and faster with next-dit.arXiv preprint arXiv:2406.18583, 2024

    Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Lirui Zhao, Fu-Yun Wang, Zhanyu Ma, et al. Lumina-next: Making lumina-t2x stronger and faster with next-dit.arXiv preprint arXiv:2406.18583, 2024

    2024 arXiv

  40. [50]

    Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer.arXiv preprint arXiv:2501.18427, 2025

    Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, et al. Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer.arXiv preprint arXiv:2501.18427, 2025

    2025 arXiv

  41. [51]

    Deep compression autoencoder for efficient high-resolution diffusion models.arXiv preprint arXiv:2410.10733, 2024

    Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models.arXiv preprint arXiv:2410.10733, 2024

    2024 arXiv

  42. [52]

    Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction

    Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 17302–17313, 2023

    2023

  43. [53]

    F-lmm: Grounding frozen large multimodal models.arXiv preprint arXiv:2406.05821, 2024

    Size Wu, Sheng Jin, Wenwei Zhang, Lumin Xu, Wentao Liu, Wei Li, and Chen Change Loy. F-lmm: Grounding frozen large multimodal models.arXiv preprint arXiv:2406.05821, 2024

    2024 arXiv

  44. [54]

    Llamafusion: Adapting pretrained language models for multimodal generation.arXiv preprint arXiv:2412.15188, 2024

    Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Llamafusion: Adapting pretrained language models for multimodal generation.arXiv preprint arXiv:2412.15188, 2024

    2024 arXiv

  45. [55]

    Scaling laws for native multimodal models.arXiv preprint arXiv:2504.07951, 2025

    Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, and Alaaeldin El-Nouby. Scaling laws for native multimodal models.arXiv preprint arXiv:2504.07951, 2025

    2025 arXiv

  46. [56]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  47. [57]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  48. [58]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 12

    2017 arXiv

  49. [59]

    Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    2024 arXiv

  50. [60]

    High-resolution image synthesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022

    2022

  51. [61]

    Pixart- alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart- alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023

    arXiv

  52. [62]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  53. [63]

    Flow-grpo: Training flow matching models via online rl, 2025

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl, 2025

    2025

  54. [64]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    2024 arXiv

  55. [65]

    World model on million-length video and language with ringattention.arXiv preprint arXiv:2402.08268, 2024

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention.arXiv preprint arXiv:2402.08268, 2024

    2024 arXiv

  56. [66]

    Simplear: Push- ing the frontier of autoregressive visual generation through pretraining, sft, and rl.arXiv preprint arXiv:2504.11455, 2025

    Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. Simplear: Push- ing the frontier of autoregressive visual generation through pretraining, sft, and rl.arXiv preprint arXiv:2504.11455, 2025

    2025 arXiv

  57. [67]

    text-to-image-2M: A high-quality, diverse text–image training dataset

    Jacky He and contributors. text-to-image-2M: A high-quality, diverse text–image training dataset. https: //huggingface.co/datasets/jackyhate/text-to-image-2M, 2024

    2024

  58. [68]

    Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in neural information processing systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in neural informat...

    2022

  59. [69]

    Megalith-10M: A dataset of 10 million public-domain photographs

    Ollin Matsubara and Draw Things AI Team. Megalith-10M: A dataset of 10 million public-domain photographs. https://huggingface.co/datasets/madebyollin/megalith-10m, 2024. CC0/Flickr-Commons images; Florence-2 captions available in the *megalith-10m-florence2* variant

    2024

  60. [70]

    RedCaps: Web-curated image–text data created by the people, for the people

    Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. RedCaps: Web-curated image–text data created by the people, for the people. InNeurIPS, Datasets and Benchmarks Track, 2021

    2021

  61. [71]

    Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation.arXiv preprint arXiv:2402.17245, 2024

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation.arXiv preprint arXiv:2402.17245, 2024

    2024 arXiv

  62. [72]

    Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36, 2024

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36, 2024

    2024

  63. [73]

    Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    2024 arXiv

  64. [74]

    Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025

    Yuwei Niu, Munan Ning, Mengren Zheng, Bin Lin, Peng Jin, Jiaqi Liao, Kunpeng Ning, Bin Zhu, and Li Yuan. Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025

    2025 arXiv

  65. [75]

    Tokenflow: Unified image tokenizer for multimodal understanding and generation.arXiv preprint arXiv:2412.03069, 2024

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation.arXiv preprint arXiv:2412.03069, 2024. 13

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.