Pith. sign in

REVIEW 4 major objections 6 minor 83 references

X-Fusion: Introducing New Modality to Frozen Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A frozen pretrained LLM gains image understanding and generation from a parallel trainable vision tower, without losing its language knowledge.

desk verdict A useful, well-ablated architecture paper whose central claim is overstated in the abstract but whose data-centric findings hold up on inspection; worth refereeing, not desk-rejecting. read the letter →

arxiv 2504.20996 v1 pith:WFBFMNVW submitted 2025-04-29 cs.CV

Sicheng Mo , Thao Nguyen , Xun Huang , Siddharth Srinivasan Iyer , Yijun Li , Yuchen Liu , Abhishek Tandon , Eli Shechtman
show 4 more authors
Krishna Kumar Singh Yong Jae Lee Bolei Zhou Yuheng Li
This is my paper · ORCID
classification cs.CV
keywords frozenlargelanguagemodelsmultimodalunderstandingandgenerationdual-towerarchitectureimage-to-texttext-to-imageper-layertokenroutingdiffusionlosscatastrophicforgetting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a pretrained, frozen large language model can learn both to understand and to generate images while keeping its language abilities intact, by adding a parallel trainable vision tower. The key move is per-layer routing: at each transformer layer, text tokens go through the frozen text block and image tokens through a trainable vision block, then the streams are recombined before the next frozen layer. On LLaMA-3.2-1B, the authors report that this dual-tower design beats single-tower, gated, and dual-projection alternatives on image quality (FID 14.20 vs 19.10 for single tower) and captioning (BLIP 31.3 vs 30.2), while preserving MMLU at 32.2 instead of dropping to chance-level 25.0. They further claim that training with clean images for understanding tasks and a 2:1 generation-to-understanding data ratio improves both tasks, and that aligning with pretrained vision features helps small models but not large ones.

What carries the argument

The central object is the dual-tower transformer block. At each layer, the same input sequence runs through a frozen text block and a trainable vision block initialized by copying the language layer; the output sequence takes text-block outputs at text positions and vision-block outputs at image positions, so vision information is injected into the frozen language stream at every intermediate layer rather than only at the input. An optional X-Fuse operation merges features from both towers with learnable scalars, trading extra FLOPs for improved performance.

What would settle it

Train the dual-tower model with the vision tower initialized randomly instead of copied from the language layers, holding all data and training budget fixed; if the FID and BLIP gains over the single-tower baseline shrink or MMLU preservation breaks, the reported advantage depends on the initialization choice rather than on per-token routing.

Watch

Extended reading notes

Core claim

X-Fusion's central claim is that modality-specific weights inside each block - a frozen language transformer block and a copied-then-trained vision transformer block - let a frozen LLM process interleaved image and text tokens in both directions. For each token, the output is taken from the block that matches its modality, so image tokens are computed by the vision tower and text tokens by the language tower; the next frozen language layer then sees text features produced by frozen weights and image features produced by trainable ones. The paper argues this beats the natural alternatives: single tower fine-tunes the LLM and sacrifices language knowledge, gated tower is too weak because it only adds a gated residual, and dual projection, which duplicates attention QKV and MLP but computes joint attention, cannot generate fresh text key/value representations for the image tokens. The reported consequence is that a frozen LLM gains image generation and understanding at the same FLOPs as these alternatives while its MMLU score stays exactly at the pretrained level.

Load-bearing premise

That the frozen text layers can safely condition on features produced by the trainable vision tower, so the two towers never drift into incompatible feature distributions; the paper tests this on in-house data with InternVL captions but does not report how sensitive the result is to the vision tower's initialization or choice of image encoder.

Editorial extensions

If this is right

  • Frozen LLMs can adopt a new output modality without degrading MMLU, so language knowledge can be preserved while adding image generation.
  • Understanding-focused data improves generation quality, so a skewed 2:1 generation-to-understanding data ratio should be preferred over a balanced mix.
  • Clean images in image-to-text samples improve both understanding and generation, challenging the practice of applying diffusion noise to understanding samples.
  • Feature alignment with pretrained vision representations accelerates convergence for small models but can slightly hurt large models, so its value shrinks with scale.
  • The dual-tower design reaches the same attention FLOPs as single-tower, gated, and dual-projection alternatives while giving the vision tower its own attention and MLP weights.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the routing mechanism would be to freeze the vision tower after initialization; if generation quality stays high, the per-layer routing is doing less work than claimed.
  • If the dual-tower recipe transfers, a similar frozen-model design could add audio or video tokens by placing a third tower alongside the text tower and applying the same per-token selection rule.
  • The reported asymmetry - understanding data helps generation but not vice versa - suggests an ordering heuristic for training schedules: spend early budget on clean understanding samples, then add generation pairs.
  • The paper's 8B comparison against other unified models uses different training budgets, so a matched-budget head-to-head with the closest transformer-level baselines would sharpen the claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes X-Fusion, a framework that extends a frozen pretrained LLM with a trainable parallel vision tower for both image understanding and image generation. In each transformer layer, text tokens are processed by the frozen text block and image tokens by a trainable vision block, with outputs routed by token modality; an optional X-Fuse operation merges features across towers. The paper compares Dual Tower against Single Tower, Gated Tower, and Dual Projection variants on a 1B LLaMA-3.2 model (Table 1), ablates the effect of image noise in image-to-text samples (Fig. 6), the data ratio between text-to-image and image-to-text tasks (Fig. 5), and CLIP feature alignment at 1B/3B/8B scale (Fig. 7). It also demonstrates extensions: X-Fuse, initialization from a pretrained DiT, and fine-tuning for editing/VQA/localization. The central claims are that X-Fusion preserves language capability (MMLU at 32.2) while outperforming alternative architectures on both image-to-text and text-to-image tasks, and that its data ablations yield transferable training insights.

Significance. If the empirical claims are robust, X-Fusion is a practically useful recipe for adding modalities to frozen LLMs: the per-layer dual-tower design with token routing is clean, the idea of keeping the language tower untouched is appealing, and the ablations on clean images for understanding data and on asymmetric task synergy are potentially valuable for the multimodal training community. The paper is honest in places: it includes a supplementary comparison with other unified models (Table A) and a limitations section (Sec. D) acknowledging hallucinations, parameter overhead, and room for improvement in image quality. However, the paper's headline claim of outperforming alternative architectures is considerably broader than the evidence: all results come from single runs with no variance estimates, the understanding-task margin in Table 1 is thin, and the paper's own Table A shows X-Fusion trailing several state-of-the-art models on FID. The contribution is therefore best characterized as a promising architecture plus a useful ablation study, not as an established state-of-the-art result.

major comments (4)
  1. [Abstract and Table 1] The abstract's claim that 'X-Fusion consistently outperforms alternative architectures on both image-to-text and text-to-image tasks' is not supported at the level claimed. Every number in Table 1 comes from a single training run with no multiple seeds, no standard deviations, and no significance test. On the understanding axis the margin is particularly thin: Dual Tower reaches BLIP 31.3 versus 30.9 for Dual Projection and 30.2 for Single Tower, so the 0.4-point advantage is not demonstrable without variance estimates. The FID gap (14.20 vs 19.10/20.22) is large, but it is still a single-run observation. Please report results over at least three seeds with mean and standard deviation, or, if that is impractical, explicitly qualify all comparative statements as single-run observations.
  2. [Table A (Supplementary) vs Abstract] The scope of 'alternative architectures' is ambiguous. If it means only the three ablation blocks in Table 1, the claim should say so explicitly. If it means leading unified multimodal models, the paper's own supplementary Table A contradicts the unqualified wording: X-Fusion-8B reaches COCO FID 11.5, whereas Transfusion reports 6.7, Janus 8.5, and Show-O 9.2, and BLIP 80.0 is below InternVL's 81.1. The main text should define which comparison class the headline claim refers to, and the abstract should be revised to avoid the impression that X-Fusion outperforms state-of-the-art unified models, since the supplementary table itself does not support that reading.
  3. [Sec. 4, Data and Task paragraphs] The validity of all visual-task metrics depends on a clean separation between the in-house licensed training data and the MS-COCO evaluation set, but the paper never states whether the training set overlaps with MS-COCO. Since the in-house data is described only as 'licensed' and COCO is a standard public dataset, the possibility of training/evaluation contamination must be addressed; if any COCO images or captions were used during training, the reported FID and BLIP numbers are uninterpretable. Please state explicitly whether the training data is disjoint from the evaluation set, and also report how many images are used for the 30K-image COCO evaluation and how the BLIP2-ITM score is averaged over such a sample without confidence intervals.
  4. [Sec. 4, Dual Tower and Sec. 5] The paper frames the preservation of MMLU at 32.2 as a key advantage, but this preservation is guaranteed by construction: the text tower is frozen and pure text inputs never pass through the vision tower, so the text-only forward pass is identical to the original LLM. The nontrivial burden rests on showing that the frozen later text layers can interpret the vision tower's outputs in the interleaved setting, yet the paper does not analyze this distributional assumption. The only supporting evidence is in-house captioning quality and the linear-probe experiment in Fig. 8, which probes the vision tower features, not the frozen text tower's ability to condition on them. Please either add an analysis of the feature distribution seen by frozen text layers (e.g., linear probing of the text tower on visual input, or a check of attention statistics on mixed sequences) or soften the claims framed around 'preserving language capabilities,' since the architecture's real risk is not language forgetting but visual-feature interpretability.
minor comments (6)
  1. [Sec. 5 and Table 1] The text says 'Dual Tower and Dual Projection share a common insight: modality-specific operations,' but the comparison would be easier to interpret if the table also reported the number of trainable parameters for each variant; the current text mentions the same 'number of training parameters' for Single Tower and Dual Tower only in passing.
  2. [Sec. 8.2] The reference to 'Figure 12 shows that this operation further enhances the model's capability' appears to be a figure-numbering error: Figure 12 is the X-Fuse ablation, whereas the pretrained-DiT transfer results are shown in Figure 9. Please correct the cross-reference.
  3. [Sec. 6.1 and Fig. 5] The Fig. 5 caption states '2.4 times acceleration,' but the definition of this acceleration is not given anywhere. Specify how the acceleration factor is computed and which curves it compares.
  4. [Sec. 4, Training paragraph] The default loss weights are given as lambda_AR = 0.2 and lambda_DM = 1, but Sec. 7 states 'We use lambda_AR = 0.5 in this ablation study.' Please clarify which configuration is used for the main experiments and whether different lambda values across the REPA ablation affect the comparability of the 1B/3B/8B curves in Fig. 7.
  5. [Throughout] Several typos remain: 'effiecient' in the conclusion, 'pretraind' in the Sec. 8.2 title, 'texture features' should presumably be 'text features' in Sec. 4, and stray '/edt' tokens appear in Sec. 6.1 and Sec. 6.2. These do not affect the technical content but should be cleaned up.
  6. [References] References [44] and [62] are the same paper (Rombach et al., High-resolution image synthesis with latent diffusion models) cited twice with different venues; please merge them.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's central claims rest on external benchmarks and controlled ablations, not on equations that reduce to their inputs.

full rationale

X-Fusion is an empirical architecture and training-strategy paper. Its central comparisons are measured against external benchmarks (MMLU, COCO FID, CLIP, and BLIP2-ITM) and against ablation variants that differ only in the transformer block design while sharing tokenizers, encoders, and decoders. No prediction is obtained by fitting a parameter to the same data it later claims to predict; the ablations of data ratio, noise level, and feature alignment are controlled observations rather than quantities defined by the inputs. The preservation of MMLU at 32.2 is expected by construction because the text tower is frozen, but the paper presents this as a sanity check rather than as a derived prediction, and the nontrivial generation and understanding claims are independently evaluated. The paper's self-citations (e.g., prior LLaVA work by overlapping authors) appear in related-work context and are not load-bearing for the architecture choice or the experimental conclusions. The weaker head-to-head numbers in Table A relative to Transfusion, Janus, and Show-O are a concern about the strength of the stated claim, not a circularity concern.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method relies on the standard hybrid training losses from Transfusion and SD3 and on a small set of hand-picked hyperparameters. The main unstated premise is feature compatibility between the trainable vision tower and the frozen text tower, plus reliance on InternVL captions and the BLIP metric. No new physical entities or external structures are introduced.

free parameters (6)
  • loss weights lambda_AR and lambda_DM = 0.2, 1.0
    Chosen in Sec 4 Training without a reported sweep; balances autoregressive language loss and diffusion image loss.
  • CFG guidance scale = 5.5
    Selected empirically for inference in Supplementary A.1.
  • T2I:I2T data ratio = 66/33 (2:1)
    Recommended from the ablation in Sec 6.2; used for all final model training.
  • I2T max noise = 0% (clean images)
    Set from the Fig 6 ablation; the paper argues clean understanding images improve both tasks.
  • REPA alignment layer and target = layer 8, CLIP penultimate feature
    Adopted from REPA in Sec 7; tested at 1B, 3B, and 8B scales.
  • learning rate, batch size, steps = 1e-4, 0.8M tokens, 100k steps
    Training hyperparameters in Sec 4; not ablated.
assumptions (5)
  • standard math Autoregressive next-token loss for text and flow-matching diffusion loss for images are compatible in a single model.
    Taken from Transfusion and SD3 as the training framework; no new derivation is provided.
  • domain assumption The frozen text tower can process a mixed sequence where image positions contain features produced by the trainable vision tower.
    Core architectural premise in Sec 4 Dual Tower; if false, understanding or language preservation would fail.
  • domain assumption InternVL-2.0 generated captions are accurate enough to supervise both understanding and generation.
    Used as the sole training signal for all image-caption pairs in Sec 4 Data.
  • domain assumption BLIP2-ITM is a valid measure of caption quality for long captions.
    The paper argues CIDEr, BertScore, and CLIP scores are unsuitable in Supplementary A.2, but BLIP reliability is not independently validated.
  • domain assumption The in-house licensed dataset is representative of natural images for COCO evaluation.
    Training distribution is not public; evaluation is on MS-COCO.

how reviews work

0 comments
Cite this review

Pith. "Pith review of X-Fusion: Introducing New Modality to Frozen Large Language Models." pith.science (2026). https://pith.science/paper/WFBFMNVW

@misc{pith2026250420996,
  author       = {Pith},
  title        = {Pith review of: X-Fusion: Introducing New Modality to Frozen Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WFBFMNVW}},
  note         = {Machine review of arXiv:2504.20996}
}
read the original abstract

We propose X-Fusion, a framework that extends pretrained Large Language Models (LLMs) for multimodal tasks while preserving their language capabilities. X-Fusion employs a dual-tower design with modality-specific weights, keeping the LLM's parameters frozen while integrating vision-specific information for both understanding and generation. Our experiments demonstrate that X-Fusion consistently outperforms alternative architectures on both image-to-text and text-to-image tasks. We find that incorporating understanding-focused data improves generation quality, reducing image data noise enhances overall performance, and feature alignment accelerates convergence for smaller models but has minimal impact on larger ones. Our findings provide valuable insights into building efficient unified multimodal models.

Figures

Figures reproduced from arXiv: 2504.20996 by the authors.

Figure 1
Figure 1. We introduce X-Fusion - a novel framework that adapts pretrained LLMs (e.g., LLaMA) to new modalities (e.g., vision) while retaining their language capabilities and world knowledge. guage with diffusion loss for images. This hybrid architec￾ture has significantly advanced performance, demonstrat￾ing greater promise than purely autoregressive approaches like Chameleon [26]. However, training such models from scratch … view at source ↗
Figure 2
Figure 2. Captions generated by X-Fusion demonstrate high details and strong visual alignment with the image inputs. unified, this approach creates several limitations: limited cross-modal reasoning, restricted in-context learning, and increased error accumulation [45]. Most critically, these ap￾proaches typically require fine-tuning the LLM backbone, degrading inherited text generation ability [37, 38]. This raises fundament… view at source ↗
Figure 3
Figure 3. Images generated by X-Fusion demonstrate high visual quality and strong text alignment with the input prompts. Training LMMs using both diffusion and next-token pre￾diction losses (e.g., Transfusion [28]). Following the third approach, which has achieved state-of-the-art results across modalities, we instead propose initializing from frozen LLMs rather than training from scratch, significantly reduc￾ing computationa… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Conceptual comparison of four model architecture baselines. Here, we illustrate how each layer processes the sequential multi-modal feature. (a) Single Tower: Directly fine-tuning pre-trained LLM. (b) Gated Layer: Duplicate Each LLM layer as the gated vision layer, (c)…
Figure 5
Figure 5. Figure 5: Performance of image generation and understanding at various data ratios. Increasing visual understanding data improves visual generation performance. 20 40 60 80 100 15 20 25 30 35 40 45 Image Quality - FID 20 40 60 80 100 19 20 21 22 23 Image-Text Alignment - CLIP Ma…
Figure 6
Figure 6. Figure 6: Performance of image generation and understanding at various noise limits in the image-to-text samples. Providing clear images for image-to-text samples enhances visual generation and understanding simultaneously. L Text processing flexibility affects image generation …
Figure 7
Figure 7. Figure 7: Performance comparison of models of different sizes (1B, 3B, and 8B) with and without additional feature alignment loss. The effectiveness of alignment diminishes as model size increases. 4 6 8 10 12 14 16 Layer Depth 30 40 50 60 Val Accuracy (%) Image Understanding 4 …
Figure 8
Figure 8. Figure 8: Linear Probe Results. We use the trained model as a feature extractor and train an additional linear layer for image classification on ImageNet [74]. Models trained with our training strategy constantly obtain higher feature quality. L Using clean images for visual und…
Figure 9
Figure 9. Figure 9: Qualitative comparison between pretrained DiT model, X-Fusion(DiT) and vanilla X-Fusion on image generation and un￾derstanding task. By initializing the vision tower from pretrained text-to-image diffusion model, X-Fusion(DiT) achieves stronger image generation capabil…
Figure 10
Figure 10. Figure 10: Interactive Generation. Our X-Fusion model can fol￾low user instructions to understand, generate, or edit images. 8.2. Transfer from Pretrained Diffusion Model While X-Fusion successfully kept its language generation capability, its image generation capability still n…
Figure 11
Figure 11. Figure 11: Qualitative results of fine-tuned X-Fusion model on downstream tasks including: visual question answering (VQA), image editing, localization, and in/out-painting tasks. 3 6 9 12 15 18 Total Tokens: Billion 20 30 40 50 60 Image Quality FID 3 6 9 12 15 18 Total Tokens: …
Figure 12
Figure 12. Figure 12: Ablation: X-Fuse layer. Our X-Fusion model with the X-Fuse layer outperforms the baseline X-Fusion model on image generation and understanding tasks. to-image DiT model as its vision tower, notated as X￾Fusion(Pretrained DiT), using the same training recipe in the pre…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

83 extracted references · 45 canonical work pages

  1. [1]

    Improving language understanding by generative pre-training

    Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018. 1, 2, 3

    2018

  2. [2]

    Language models are unsuper- vised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsuper- vised multitask learners. 2019

    2019

  3. [3]

    Brown and et al

    Tom B. Brown and et al. Language models are few-shot learners, 2020

    2020

  4. [4]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learn- ing Research, 2020

    work page 2020

  5. [5]

    Rae and et al

    Jack W. Rae and et al. Scaling language models: Methods, analysis & insights from training gopher, 2022

    work page 2022

  6. [6]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y . Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V . Le. Finetuned language models are zero-shot learn- ers, 2022

    work page 2022

  7. [7]

    Training compute-optimal large language models, 2022

    Jordan Hoffmann and et al. Training compute-optimal large language models, 2022

    work page 2022

  8. [8]

    Palm: Scaling language modeling with path- ways, 2022

    PaLM team. Palm: Scaling language modeling with path- ways, 2022

    work page 2022

Show all 83 references
  1. [9]

    Palm 2 technical report, 2023

    PaLM 2 team. Palm 2 technical report, 2023

    2023

  2. [10]

    Jiang and et al

    Albert Q. Jiang and et al. Mixtral of experts, 2024

    2024

  3. [11]

    Jiang and et al

    Albert Q. Jiang and et al. Mistral 7b, 2023

    2023

  4. [12]

    Deepseek llm: Scaling open-source language models with longtermism, 2024

    DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism, 2024

    2024

  5. [13]

    Gemma: Open models based on gemini re- search and technology, 2024

    Gemma Team. Gemma: Open models based on gemini re- search and technology, 2024. 1, 2, 3

    2024

  6. [14]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 1

    2023

  7. [15]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Llama 2 team. Llama 2: Open foundation and fine-tuned chat models, 2023

    2023

  8. [16]

    Phi-3 technical report: A highly capable lan- guage model locally on your phone, 2024

    Phi-3 team. Phi-3 technical report: A highly capable lan- guage model locally on your phone, 2024

    2024

  9. [17]

    The llama 3 herd of models

    Abhimanyu Dubey and et al. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024. 5, 6

    2024 arXiv

  10. [18]

    Qwen technical report

    Qwen team. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 1

    2023 arXiv

  11. [19]

    Internlm2 technical report, 2024

    InternLM2 team. Internlm2 technical report, 2024

    2024

  12. [20]

    Deepseek-v2: A strong, economical, and ef- ficient mixture-of-experts language model, 2024

    DeepSeek-AI. Deepseek-v2: A strong, economical, and ef- ficient mixture-of-experts language model, 2024. 1

    2024

  13. [21]

    Evaluating large language models trained on code, 2021

    Mark Chen and et al. Evaluating large language models trained on code, 2021. 1, 3

    2021

  14. [22]

    Codegen: An open large language model for code with multi-turn program synthesis, 2023

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis, 2023

    2023

  15. [23]

    Starcoder: may the source be with you!, 2023

    Raymond Li and et al. Starcoder: may the source be with you!, 2023. 1, 3

    2023

  16. [24]

    Scaling autoregressive multi-modal models: Pretraining and instruction tuning, 2023

    Lili Yu and et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning, 2023. 1

    2023

  17. [25]

    Gemini: A family of highly capable multi- modal models, 2024

    Gemini Team. Gemini: A family of highly capable multi- modal models, 2024

    2024

  18. [26]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 1, 2

    2024 arXiv

  19. [27]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 1, 3

    2024 arXiv

  20. [28]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model. 2024. 1, 3, 4, 6, 7, 16

    2024

  21. [29]

    Dreamllm: Synergistic multimodal compre- hension and creation, 2024

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Dreamllm: Synergistic multimodal compre- hension and creation, 2024. 1, 2, 3

    2024

  22. [30]

    Gen- erating images with multimodal language models

    Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Gen- erating images with multimodal language models. NeurIPS,

  23. [31]

    Metamorph: Multimodal understanding and generation via instruction tuning

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412.14164, 2024. 1, 3

    2024 arXiv

  24. [32]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1, 2, 3

    2023

  25. [33]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  26. [34]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representa- tions, 2024. 3

    2024

  27. [35]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer...

    2024

  28. [36]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. arXiv preprint arXiv:2103.00020, 2021. 1, 2, 3,...

    2021 arXiv

  29. [37]

    Investigating the catas- trophic forgetting in multimodal large language model fine- tuning

    Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catas- trophic forgetting in multimodal large language model fine- tuning. In Conference on Parsimony and Learning (Proceed- ings Track), 2023. 1, 2

    2023

  30. [38]

    Model tailor: mitigating catastrophic forgetting in multi-modal large lan- guage models

    Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Chao Wu, and Kun Kuang. Model tailor: mitigating catastrophic forgetting in multi-modal large lan- guage models. In Proceedings of the 41st International Con- 11 ference on Machine Learning , ICML’24. JMLR.org, 2...

    2024

  31. [39]

    Generative multi- modal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multi- modal models are in-context learners. 2023. 1

    2023

  32. [40]

    Seed-x: Mul- timodal models with unified multi-granularity comprehen- sion and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Mul- timodal models with unified multi-granularity comprehen- sion and generation. arXiv preprint arXiv:2404.14396, 2024. 3

    2024 arXiv

  33. [41]

    Making llama see and draw with seed tokenizer.arXiv preprint arXiv:2310.01218, 2023

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer.arXiv preprint arXiv:2310.01218, 2023

    2023 arXiv

  34. [42]

    Codi-2: In-context, in- terleaved, and interactive any-to-any generation

    Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. Codi-2: In-context, in- terleaved, and interactive any-to-any generation. 2023

    2023

  35. [43]

    Janus- pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus- pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 ,

    arXiv

  36. [44]

    High-resolution image syn- thesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2022. 1, 2, 14

    2022

  37. [45]

    Multimodal machine learning: A survey and tax- onomy, 2017

    Tadas Baltru ˇsaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and tax- onomy, 2017. 2

    2017

  38. [46]

    You only look once: Unified, real-time object de- tection, 2016

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object de- tection, 2016. 2

    2016

  39. [47]

    A style-based generator architecture for generative adversarial networks,

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks,

  40. [48]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. 3

    2024 arXiv

  41. [49]

    Llamafu- sion: Adapting pretrained language models for multimodal generation

    Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Llamafu- sion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:2412.15188, 2024. 3, 6

    2024 arXiv

  42. [50]

    Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V

    Noam M. Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V . Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ArXiv, abs/1701.06538, 2017. 3

    2017 arXiv

  43. [51]

    Shazeer, and Z

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, De- hao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam M. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic shard- ing. ArXiv, abs/2006.16668, 2020

    2006 arXiv

  44. [52]

    William Fedus, Barret Zoph, and Noam M. Shazeer. Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity. ArXiv, abs/2101.03961, 2021

    2021 arXiv

  45. [53]

    Glam: Efficient scaling of language mod- els with mixture-of-experts

    Nan Du and et al. Glam: Efficient scaling of language mod- els with mixture-of-experts. 2021. 3

    2021

  46. [54]

    Scaling vision with sparse mix- ture of experts

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr ´e Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mix- ture of experts. In Neural Information Processing Systems,

  47. [55]

    Collaging class-specific gans for semantic image synthesis

    Yuheng Li, Yijun Li, Jingwan Lu, Eli Shechtman, Yong Jae Lee, and Krishna Kumar Singh. Collaging class-specific gans for semantic image synthesis. 2021 IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 14398–14407, 2021

    2021

  48. [56]

    Exploring sparse moe in gans for text-conditioned image synthesis

    Jiapeng Zhu, Ceyuan Yang, Kecheng Zheng, Yinghao Xu, Zifan Shi, and Yujun Shen. Exploring sparse moe in gans for text-conditioned image synthesis. ArXiv, abs/2309.03904,

    arXiv

  49. [57]

    Imagebind one embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind one embedding space to bind them all. 2023 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 15180–15190, 2023. 3

    2023

  50. [58]

    Image as a foreign language: Beit pretraining for all vision and vision-language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. ArXiv, abs/2208.10442, 2022. 3

    2022 arXiv

  51. [59]

    Vlmo: Unified vision-language pre-training with mixture-of- modality-experts

    Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of- modality-experts. ArXiv, abs/2111.02358, 2021

    2021 arXiv

  52. [60]

    Scaling vision-language mod- els with sparse mixture of experts

    Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language mod- els with sparse mixture of experts. ArXiv, abs/2303.07226,

    arXiv

  53. [61]

    Playground v3: Improving text-to- image alignment with deep-fusion large language models

    Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to- image alignment with deep-fusion large language models. ArXiv, abs/2409.10695, 2024. 4

    2024 arXiv

  54. [62]

    Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer

    Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022. 4, 16

    2022

  55. [63]

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy Q. V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernan- dez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russ Howes, Po-Yao (Bernie) Huang, Shang-Wen Li, Ishan Misr...

    arXiv

  56. [64]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first international conference on mach...

  57. [65]

    Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parameters. 12 In Proceedings of the 26th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, ...

    2020

  58. [66]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  59. [67]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 5, 14

    2017

  60. [68]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 5

    2023

  61. [69]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion, 2015. 5, 14

    2015

  62. [70]

    Wein- berger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Wein- berger, and Yoav Artzi. Bertscore: Evaluating text genera- tion with bert, 2020. 5, 14

    2020

  63. [71]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:237...

  64. [72]

    Mea- suring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding, 2021. 6

    2021

  65. [73]

    Representation alignment for generation: Training dif- fusion transformers is easier than you think

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training dif- fusion transformers is easier than you think. ArXiv, abs/2410.06940, 2024. 7, 8, 14

    arXiv

  66. [74]

    Imagenet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In Computer Vision and Pattern Recognition,

  67. [75]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 4195–4205,

  68. [76]

    Direct inversion: Boosting diffusion-based edit- ing with 3 lines of code

    Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Direct inversion: Boosting diffusion-based edit- ing with 3 lines of code. arXiv preprint arXiv:2310.01506,

    arXiv

  69. [77]

    Smartedit: Exploring complex instruction-based image editing with multimodal large lan- guage models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-based image editing with multimodal large lan- guage models. In Proceedings of the IEEE/CVF Conferen...

    2024

  70. [78]

    De- constructing denoising diffusion models for self-supervised learning

    Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. De- constructing denoising diffusion models for self-supervised learning. ArXiv, abs/2401.14404, 2024. 14 13 X-Fusion: Introducing New Modality to Frozen Large Language Models (Supplementary Material) In the supplementary m...

    2024 arXiv

  71. [80]

    This is the view of a kitchen stove, kitchen sink, counter, and cabinets

  72. [81]

    A wooden kitchen center island with a rug in front of it

  73. [82]

    The sink is on the island of a large kitchen

  74. [83]

    A kitchen with a sink, stove, flower vase and wine rack

    A kitchen with a stove a sink and a counter 5 . A kitchen with a sink, stove, flower vase and wine rack. Sample 2: This image is a photo of kitchen. CLIPBLIPBertScore 91.2 10.4 30.90.86 0.89 30.9 Figure A. Comparison of Different Evaluation Metrics. The BLIP score effectively ...

  75. [2009]

    IEEE Conference on , pages 248–255

    CVPR 2009. IEEE Conference on , pages 248–255. IEEE, 2009. 7, 8

    2009

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.