UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation

Hao Li; Jun Zhang; Lirui Zhao; Quanfeng Lu; Teng Li; Wenqi Shao; Xizhou Zhu; Yu Qiao

REVIEW 4 major objections 6 minor 5 cited by

UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Y-shaped Transformer that shares early layers but splits late layers into understanding and generation branches outperforms both fully shared and task-specific unified image models.

desk verdict UniFork earns its keep with a reproducible alignment-curve diagnostic and a clean Y-shaped recipe, but the claim that divergent alignment causes the shared-backbone compromise is under-tested, not parameter-matched. read the letter →

arxiv 2506.17202 v1 pith:UP6GKG5P submitted 2025-06-20 cs.CV

Teng Li , Quanfeng Lu , Lirui Zhao , Hao Li , Xizhou Zhu , Yu Qiao , Jun Zhang , Wenqi Shao This is my paper · ORCID

classification cs.CV

keywords modalityalignmentunifiedmultimodalmodelimageunderstandinggenerationnext-tokenpredictionY-shapedarchitecturemutualk-nearestneighborstransformerbackbone

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Unified image models try to make one Transformer both see and draw, but the two tasks pull representations in opposite directions. Measuring how strongly image and text features align at every layer, this paper finds that understanding wants alignment to keep rising with depth, while generation wants it to rise early and then fall so deep layers can rebuild spatial detail. A fully shared backbone collapses both curves into a single compromise. The paper proposes UniFork, a Y-shaped architecture that shares the early layers and forks the later layers into task-specific branches, and shows it beats the shared backbone and matches expert models on both tasks.

What carries the argument

The central diagnostic is the mutual-kNN modality alignment score, the fraction of mutual nearest neighbors between averaged vision features and text prompt features, traced layer by layer to see how tightly the two modalities couple at each depth. The central architectural mechanism is the Y-shaped, shared-then-split Transformer: of $M+N$ total layers, the first $M$ are shared across tasks, while the last $N$ fork into two structurally identical but independently parameterized branches, one refining semantic representations for understanding and one recovering spatial detail for generation. The design interpolates between two known extremes: $N=0$ reproduces the fully shared Emu3 architecture, and $M=0$ resembles the Mixture-of-Transformers design of BAGEL.

What would settle it

Train a parameter-matched fully shared model using UniFork's exact Stage III alternating task fine-tuning and sweep the fork point from fully shared to fully split at fixed active-parameter counts: if the shared variant matches UniFork, or if moving the fork changes performance without changing whether each branch's alignment curve matches its expert profile, the alignment-conflict explanation fails. A second check is an intervention that pushes the generation branch's deep-layer alignment toward the understanding profile without changing parameters, which should measurably degrade GenEval if alignment is causal. The paper's own stated limitation, that its tokenizer is trained at 256 resolution while the model generates at 384, is a spatial-mismatch confound worth controlling before attributing generation gains to the architecture.

Watch

Extended reading notes

Core claim

Using a per-layer mutual-kNN score between averaged image features and text prompt features, the paper establishes that image understanding and image generation have opposite alignment needs: understanding benefits from monotonically increasing text-image alignment across depth, while generation requires strong early alignment followed by decreasing alignment in deep layers so fine spatial detail can be recovered. In Emu3-base, a fully shared next-token-prediction model, the two task curves nearly overlap in a rise-then-fall hybrid, which the paper interprets as a representational compromise; task-specific fine-tunes of the same base recover the distinct expert trends. UniFork is the architectural response: the first half of the Transformer is shared, the second half is duplicated into an understanding branch and a generation branch, letting each task follow its preferred alignment trajectory. In controlled ablations at matching active-parameter counts, UniFork exceeds the fully shared LLM on every reported benchmark and matches or beats task-specific experts, and at larger scale reaches 46 percent on GenEval and a 10.6 FID on MJHQ-30K.

Load-bearing premise

The load-bearing premise is that the alignment curves are causal, not just correlated, with performance: divergent alignment profiles are why fully shared models compromise, and matching expert curves with separate branches is what produces UniFork's gains, rather than the gains coming from extra parameters or the fine-tuning procedure.

Editorial extensions

If this is right

Fully shared Transformer backbones under next-token prediction are not a neutral container for unified image tasks; the paper's finding implies such models will systematically compromise whichever task's alignment needs diverge from the shared curve.
The final fine-tuning stage updates only each branch's parameters, so understanding and generation can be optimized on their own datasets without any data-ratio balancing, which the paper presents as a practical advantage over fully shared fine-tuning.
Scaling UniFork from 0.57B to 0.76B active generation parameters with no architectural change raises GenEval from 33 to 46 percent and cuts MJHQ-30K FID from 16.3 to 10.6, and the paper expects further gains from better tokenizers, more parameters, and higher-quality data.
On understanding benchmarks, a 0.5B-active UniFork matches or beats much larger models (SEEDv1 55.2 versus IDEFICS-9B's 45.0; POPE 85.8 versus MobileVLM-2.7B's 84.9), supporting the claim that removing task interference, not raw scale, drives the improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same mutual-kNN diagnostic generalizes into a design rule: for any pair of tasks sharing one backbone, measure whether their alignment profiles diverge in direction, and if they do, try a shared-then-split layout. This extends naturally to audio, video, or 3D unification, which the paper only lists as future work.
The paper fixes the split at half the layers, but the alignment curves suggest a principled alternative: split where the two tasks' alignment trajectories begin to diverge. A testable prediction is that the optimal $M$ tracks the layer at which the expert alignment curves separate.
Because the ablation's fully shared baseline receives the same per-task fine-tuning, the reported gains isolate the architecture only if the Stage III procedure interacts identically with both layouts; a parameter-matched shared model that alternates task updates the same way would settle whether the fork's benefit (0.33 versus 0.28 GenEval) comes from the split or from training dynamics.
If alignment is causal rather than merely correlated, an intervention that shifts the generation branch's deep-layer alignment toward the understanding profile, for example through alignment-shaped regularization on deep layers only, should degrade generation quality; the paper does not run this experiment, but it is the cleanest test of its central mechanism.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The thing to remember here is the modality-alignment diagnostic. The authors measure mutual-kNN between image and text features at each layer, and show that understanding wants alignment to keep increasing with depth, while generation wants a rise-then-fall curve. The pattern holds across expert models and across two datasets (GenEval and MJHQ-30K). That's a crisp, reproducible observation and a genuinely useful design lens for unified NTP models.

UniFork itself is a clean instantiation: share the early transformer layers, split the later layers into two task-specific branches, and initialize from Qwen2.5-0.5B. The main-model results are strong—MJHQ FID 10.6 with 0.76B active parameters is a real data point, and the understanding numbers are respectable at 0.5B. The ablation shows UniFork beating a fully shared LLM on both tasks and roughly matching single-task experts. The code is public.

Where the paper wobbles is the causal bridge. The leap from 'experts show these curves' to 'shared backbones compromise and matching the curves is what works' is asserted, not tested. The ablation is cleaner than a first read suggests: it trains all variants on the same Stage I+II subset, so the Stage III fine-tuning advantage isn't a confound in that table. But the generation comparison still isn't parameter-matched: UniFork has 0.76B active generation parameters versus 0.5B for the fully shared baseline, so part of the GenEval gain could be capacity, not the Y-shape. And the split point M:N is never swept. A parameter-matched control with two full branches (same total active params) would settle whether the alignment-matching story or simple capacity is driving the gains. The paper itself defers the split-ratio question to future work, which is honest but leaves the core mechanism underdetermined.

That said, the cited prior work is handled fairly (BAGEL's M=0 equivalence is acknowledged), the limitations are stated plainly, and the alignment analysis is independently useful regardless of whether the causal story survives. This isn't a desk reject; it's a revise where the reviewers should ask for a split-point sweep and a capacity-matched control.

Take it to a reading group if you work on unified multimodal models. I'd cite the alignment analysis. Send it to referees.

Best,

Referee Report

4 major / 6 minor

Summary. This paper studies the architectural design of unified multimodal models that perform both image understanding and generation via next-token prediction. Using a layer-wise mutual-kNN alignment metric, the authors report that expert generation models exhibit a rise-then-fall alignment curve across layers while understanding models show monotonically increasing alignment, and that a fully shared backbone trained on both tasks collapses these curves. They propose UniFork, a Y-shaped Transformer that shares early layers and uses two task-specific branches in later layers, trained with a three-stage pipeline. Experiments on understanding benchmarks (MME-P, POPE, SEED-I, VQAv2, GQA) and generation benchmarks (GenEval, MJHQ-30K) show that UniFork outperforms a fully shared baseline and is competitive with or better than larger expert and unified models.

Significance. If the central claim is established, the paper offers a simple and practical architectural recipe for unified multimodal models that improves both tasks without full duplication. The layer-wise alignment diagnostic is a useful analysis tool, and the paper provides a fairly extensive set of ablations and public benchmark comparisons, with code released. However, the empirical evidence as presented does not yet uniquely support the causal alignment-conflict narrative: the main ablation conflates architecture with parameter count for generation, and the alignment analysis is correlational. The paper would be significant for the multimodal foundation-model community if the requested controls confirm the effect.

major comments (4)

[Sec. 4.2, Table 2] The stated 'fair comparison' is not parameter-matched. The UniFork generation path activates 0.76B backbone parameters (Sec. 4.1), whereas the Fully Shared LLM activates only 0.5B plus a 0.07B vision head, so UniFork's GenEval gain (0.33 vs 0.28) and MJHQ gain (16.3 vs 17.2) may reflect additional capacity rather than the Y-shaped design. Please add a capacity-matched control, e.g., a fully shared model with a comparable number of active parameters (by widening the backbone or using a larger LLM) or a UniFork variant with matched active parameters, and report the comparison under identical total parameter budgets.
[Sec. 3.1, Sec. 4.5] The causal interpretation is asserted but not tested. The overlapping alignment curves of Emu3-base and the distinct curves of fine-tuned variants are correlational evidence; the paper even concedes in Future Work that the optimal M:N split ratio is underexplored. To support the claim that divergent alignment patterns create a conflict the Y-shape resolves, please (i) sweep the split point (e.g., M=6/12/18 with N adjusted) with matched active parameters and show that performance tracks the alignment curves, and/or (ii) intervene on alignment directly (e.g., an auxiliary loss that sculpts the curves) without changing the architecture. Figure 8 alone is not independent evidence, since UniFork was designed to exhibit those curves.
[Tables 2-5] No seed variance, error bars, or significance tests are reported. Several differences are small (e.g., VQAv2 69.6 vs 69.9, SEED-I 53.9 vs 55.1 in Table 2), so the claim that UniFork 'consistently outperforms' fully shared models is not statistically supported. Please report means and standard deviations over at least three independent runs for the ablation, and preferably for the main results as well.
[Sec. 3.1] The alignment protocol for understanding is underspecified: 'we feed the generated images into the model' does not state which model generated them. If the images are generated by LlamaGen or another expert generator, the understanding alignment curves are measured on out-of-distribution inputs, which could bias the comparison across models. Please specify the image source and repeat the analysis with images generated by the model under study (or its training-domain reconstructions) to confirm the trends.

minor comments (6)

[Table 2 caption] The word 'dente' should be 'denote'.
[Sec. 3.1, Sec. 3.2] The Emu3-base reference is given as (Sun et al., 2023b) in Sec. 3.1 and Figure 2, but Emu3 is correctly cited as (Wang et al., 2024) in Sec. 3.2; the citation is inconsistent.
[Throughout] The paper uses inconsistent capitalization: 'Unifork' vs 'UniFork' and 'Geneval' vs 'GenEval' appear in different places.
[Sec. 3.1] The mutual-kNN metric is not fully defined; please specify the value of k and how ties are handled.
[Sec. 3.3, Sec. 4.2] Please state explicitly that the ablation in Table 2 excludes Stage III; the claimed training flexibility of Stage III is not directly ablated, so the contribution of Stage III to the final model remains unquantified.
[Figure 8] Consider overlaying the fully shared model's alignment curves with UniFork's in Figure 8 to directly show the architectural effect.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: UniFork's performance claims rest on benchmark ablations independent of its alignment motivation.

full rationale

The paper's derivation chain is not circular. The motivation in Section 3.1—that expert models show increasing alignment for understanding and rise-then-fall alignment for generation, while a fully shared NTP backbone shows converging curves—is an empirical observation made on external models (LlamaGen, LLaVA-1.5, and Emu3 variants), not an equation derived from UniFork's own design. The proposed Y-shape is tested in Section 4.2, Table 2, against a Fully Shared LLM on MME-P, VQAv2, SEED-I, GenEval, and MJHQ-30K; those benchmark numbers are external and are not reconstructed from the mutual-kNN curves in Figures 2, 5, or 8. Section 4.5 verifies that UniFork displays the alignment trends intended by the architecture, but that is a design check, not the evidence for the headline performance claims. The training objective in Eq. (1) is standard next-token cross-entropy; no parameter is fitted to the alignment measurement and then reported as a prediction. The causal language in Section 3.1 ('This suggests that the understanding task may have compromised the generation objective') is an inference that the ablation does not uniquely establish, because the Fully Shared LLM and UniFork differ in total parameter count and in the Stage III task-specific fine-tuning protocol; this is a confound for the causal narrative, but a confound is a correctness risk, not circularity. Citations to prior work, including Emu3, supply empirical baselines and are not used as a self-referential 'uniqueness' or 'must be this way' argument. The appended Limitations section identifies tokenizer-resolution and data-quality issues, which are orthogonal to circularity. Accordingly, no step reduces to its own input, and the score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central design rests on an unproven causal interpretation of alignment curves, a fixed split ratio, and a chosen data mixture; no new physical entities are introduced.

free parameters (3)

Split ratio M:N = 15 shared, 15+15 task-specific layers
Section 4.1 duplicates the latter half of 30 Qwen layers; no sweep is reported, and the future work section says the optimal ratio is underexplored.
Stage II data ratio = 32.5M generation vs 16.5M understanding samples
Section 3.3 Stage II; the mixture is chosen, not optimized, and the paper's flexibility claim about data balancing applies only after Stage III.
Classifier-free guidance scale = 4.0
Section 4.1: set to balance fidelity and diversity; a standard inference hyperparameter.

assumptions (3)

domain assumption Mutual-kNN alignment on 500 prompts is a valid probe of task-relevant image-text alignment.
Section 3.1 and Section 4.2; the paper checks trends on a second dataset but reports no variance or significance.
domain assumption Divergent alignment trends cause the performance compromise in fully shared backbones.
Section 3.1 says the understanding task 'may have compromised' the generation objective; this causal step is never tested directly.
domain assumption Qwen2.5-0.5B and the VILA-U tokenizer provide suitable starting points.
Section 3.2 and Section 4.1; inherited from prior work, not re-derived here.

how reviews work

0 comments

Cite this review

Pith. "Pith review of UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation." pith.science (2026). https://pith.science/paper/UP6GKG5P

@misc{pith2026250617202,
  author       = {Pith},
  title        = {Pith review of: UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UP6GKG5P}},
  note         = {Machine review of arXiv:2506.17202}
}

read the original abstract

Unified image understanding and generation has emerged as a promising paradigm in multimodal artificial intelligence. Despite recent progress, the optimal architectural design for such unified models remains an open challenge. In this work, we start by analyzing the modality alignment behaviors of task-specific expert models for understanding and generation, as well as current unified models. Our analysis reveals a crucial observation: understanding tasks benefit from a progressively increasing modality alignment across network depth, which helps build up semantic information for better comprehension; In contrast, generation tasks follow a different trend: modality alignment increases in the early layers but decreases in the deep layers to recover spatial details. These divergent alignment patterns create a fundamental conflict in fully shared Transformer backbones, where a uniform representational flow often leads to performance compromises across two tasks. Motivated by this finding, we introduce UniFork, a novel Y-shaped architecture that shares the shallow layers for cross-task representation learning, while employing task-specific branches in deeper layers to avoid task interference. This design effectively balances shared learning and task specialization. Through extensive ablation experiments, we demonstrate that Unifork consistently outperforms conventional fully shared Transformer architectures, and achieves performance on par with or better than task-specific models.

Figures

Figures reproduced from arXiv: 2506.17202 by the authors.

**Figure 2.** Modality alignment analysis. We visualize how text-image feature alignment evolves across Transformer layers for both image understanding and generation tasks: (a) Image generation exhibits a rise-then-fall alignment trend across layers. (b) Image understanding shows an increasing alignment pattern. (c) When using a fully shared Transformer for both tasks under the nexttoken prediction objective, the alignment cur… view at source ↗

**Figure 3.** Overall framework of UniFork. UniFork adopts a Y-shaped Transformer backbone. The early layers are shared across both image generation and understanding tasks to facilitate joint semantic representation learning, while the later layers are split into task-specific branches to learn specialized representations. Und.: understanding. Gen.: generation. Proj.: projection. et al., 2024). Instead, we leverage the tokenizer… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Three-stage training pipeline for UniFork. The first stage focuses on aligning visual and textual modalities. The second stage performs joint training to enhance both image understanding and generation capabilities. In the third stage, task-specific parameters are alte…

**Figure 5.** Figure 5: Modality alignment analysis on MJHQ-30K. The observed alignment patterns on this dataset are consistent with those reported in Section 3.1. achieves a more effective trade-off between shared semantic learning and task-specific representation. By decoupling the later l…

**Figure 6.** Figure 6: Qualitative results on the text-to-image generation task. We compare image samples generated by SDv1.5, LlamaGen, and UniFork, with respective resolutions of 512×512, 512×512, and 384 × 384. What are the things I should be cautious about when I visit here? UniFork (0.5…

**Figure 7.** Figure 7: Qualitative results on the image understanding task. The key points in the answers are highlighted in red. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Modality alignment analysis for UniFork. 5 CONCLUSION In this paper, we analyzed modality alignment patterns in expert models and NTP-based unified models for image generation and understanding. We found that fully sharing a Transformer backbone may lead to task inter…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering
cs.CV 2026-07 conditional novelty 7.0 of 10

Steering vectors from the understanding branch can control image generation, but vectors from the generation branch cannot control understanding, showing UMMs are architecturally unified but representationally asymmetric.
Demystifying Video Reasoning
cs.CV 2026-03 conditional novelty 7.0 of 10

Video diffusion models reason along the denoising trajectory (Chain-of-Steps), not primarily across frames, and this mechanism can be nudged by ensembling latent trajectories.
Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction
cs.CV 2026-08 conditional novelty 6.0 of 10

A training framework that adds a removable image-generation branch to multimodal LLMs improves visual understanding benchmarks with zero inference-time cost.
Transferability Between Understanding and Generation in Unified Multimodal Models
cs.CV 2026-07 conditional novelty 6.0 of 10

Cross-task capability transfer in UMMs is architecture-dependent and can be exploited by training understanding to improve generation while avoiding distribution shift.
Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation
cs.CV 2026-07 conditional novelty 5.0 of 10

A compact unified model that reuses a frozen VLM encoder and hybrid continuous/discrete tokens reaches competitive image understanding and generation with 15.6M training images and about $2,000 in compute.

Reference graph

Works this paper leans on

21 extracted references · 1 canonical work pages · cited by 5 Pith papers

[5]

Seed-x: Multimodal models with unified multi-granularity comprehension and generation

Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396, 2024a. Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models ...

arXiv
[8]

Unified language-vision pretraining in llm with dynamic discrete visual tokenization.arXiv preprint arXiv:2309.04669,

Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin Chen, Chenyi Lei, An Liu, Chengru Song, et al. Unified language-vision pretraining in llm with dynamic discrete visual tokenization.arXiv preprint arXiv:2309.04669,

arXiv
[9]

Accessed: 2025-01-30. H. Laurenc ¸on, D. van Strien, S. Bekman, L. Tronchon, L. Saulnier, T. Wang, S. Karamcheti, A. Singh, G. Pistilli, Y . Jernite, et al. Introducing IDEFICS: An open reproduction of state- of-the-art visual language model.https://huggingface.co/blog/idefics,

work page 2025
[10]

Seed-bench: Bench- marking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023a

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023a. Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image gener...

arXiv
[11]

Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding

Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiaogang Wang, Hongsheng Li, Lewei Lu, et al. Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding. InProceedings of the Computer Vision and Pattern Recognition Conference, pp. 29767–29779, 2025a. Yifan Li, Yifan Du, Kun ...

arXiv
[12]

Journeydb: A benchmark for generative image understanding

Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances in neural information processing systems, 36:49659–49678, 2023a. Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregre...

arXiv
[13]

Emu: Generative pretraining in multimodality

Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023b. Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,

arXiv
[14]

Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

arXiv

Show all 21 references

[16]

Mint: Multi-modal chain of thought in unified generative models for enhanced image generation.arXiv preprint arXiv:2503.01298, 2025b

Yi Wang, Mushui Liu, Wanggui He, Longxiang Zhang, Ziwei Huang, Guanghao Zhang, Fangxun Shu, Zhong Tao, Dong She, Zhelun Yu, et al. Mint: Multi-modal chain of thought in unified generative models for enhanced image generation.arXiv preprint arXiv:2503.01298, 2025b. Yuqing Wang,...
[17]

Next-gpt: Any-to-any multi- modal llm

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- modal llm. InForty-first International Conference on Machine Learning, 2024a. Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu ...

arXiv
[18]

Representation alignment for generation: Training diffusion transformers is easier than you think.arXiv preprint arXiv:2410.06940,

Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think.arXiv preprint arXiv:2410.06940,

arXiv
[19]

Unified multimodal understanding and generation models: Advances, challenges, and opportunities.arXiv preprint arXiv:2505.02567,

Xinjie Zhang, Jintao Guo, Shanshan Zhao, Minghao Fu, Lunhao Duan, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. Unified multimodal understanding and generation models: Advances, challenges, and opportunities.arXiv preprint arXiv:2505.02567,
[20]

Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039,

Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039,

arXiv
[21]

Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479,

Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479,

arXiv
[2017]

Neighboring autoregressive modeling for efficient visual generation.arXiv preprint arXiv:2503.10696,

Yefei He, Yuanyu He, Shaoxuan He, Feng Chen, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Neighboring autoregressive modeling for efficient visual generation.arXiv preprint arXiv:2503.10696,

arXiv
[2019]

The platonic representation hypothesis.arXiv preprint arXiv:2405.07987,

Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis.arXiv preprint arXiv:2405.07987,

arXiv
[2020]

Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang

Accessed: 2020-03. Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504.11455, 2025a. Xinlong Wang, Xiaosong Zhang...

2020 arXiv
[2022]

Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025a

11 Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025a. Junsong ...

arXiv
[2023]

Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,

Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,

arXiv
[2024]

Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394,

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394,

arXiv
[2025]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923,

arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [5]

Seed-x: Multimodal models with unified multi-granularity comprehension and generation

Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396, 2024a. Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models ...

arXiv

[2] [8]

Unified language-vision pretraining in llm with dynamic discrete visual tokenization.arXiv preprint arXiv:2309.04669,

Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin Chen, Chenyi Lei, An Liu, Chengru Song, et al. Unified language-vision pretraining in llm with dynamic discrete visual tokenization.arXiv preprint arXiv:2309.04669,

arXiv

[3] [9]

Accessed: 2025-01-30. H. Laurenc ¸on, D. van Strien, S. Bekman, L. Tronchon, L. Saulnier, T. Wang, S. Karamcheti, A. Singh, G. Pistilli, Y . Jernite, et al. Introducing IDEFICS: An open reproduction of state- of-the-art visual language model.https://huggingface.co/blog/idefics,

work page 2025

[4] [10]

Seed-bench: Bench- marking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023a

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023a. Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image gener...

arXiv

[5] [11]

Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding

Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiaogang Wang, Hongsheng Li, Lewei Lu, et al. Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding. InProceedings of the Computer Vision and Pattern Recognition Conference, pp. 29767–29779, 2025a. Yifan Li, Yifan Du, Kun ...

arXiv

[6] [12]

Journeydb: A benchmark for generative image understanding

Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances in neural information processing systems, 36:49659–49678, 2023a. Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregre...

arXiv

[7] [13]

Emu: Generative pretraining in multimodality

Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023b. Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,

arXiv

[8] [14]

Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

arXiv