Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

Are Unified Vision-Language Models Necessary: Generalization Across Understanding and Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training one model on both understanding and generation outperforms separate specialists, across several architectures.

desk verdict Useful controlled study that undercuts its own headline claim: the mutual-benefit result is confounded with training-data quantity, so the 'necessity of unified VLMs' is not established as-is. read the letter →

arxiv 2505.23043 v1 pith:EN5VB2LH submitted 2025-05-29 cs.CV cs.AI

Jihai Zhang , Tianle Li , Linjie Li , Zhengyuan Yang , Yu Cheng This is my paper
classification cs.CVcs.AI
keywords unifiedvision-languagemodelsmutualbenefitcross-taskgeneralizationvisioninput-outputalignmentknowledgetransfertext-to-imagegenerationvisualquestionansweringsyntheticdatasetevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a single vision-language model trained on both understanding tasks (visual question answering and captioning) and generation tasks (text-to-image) is genuinely better than two separate specialists. The answer given is yes: across four different unified architectures, mixed training improves both VQA accuracy and image quality, and the improvement grows as more data is added. The paper also claims that aligning the vision input and output spaces is key to this cross-task benefit, and that knowledge learned during generation can transfer into understanding inside the base language model itself. If correct, this means unified models are not merely a packaging convenience but a real performance advantage, with practical implications for how vision-language training data and architectures are chosen.

What carries the argument

The load-bearing object is the Smart Watch UI dataset, a synthetic image-text world in which every image is generated from six attributes: time, weather, weather position, battery level, battery position, and watch face color, with rule-based VQA questions, captions, and text-to-image instructions. Around this dataset the paper builds four unified VLM configurations by crossing SigLIP and VQ-VAE vision encoders with VQ-token and SigLIP-embedding generation outputs, so that some variants (SigLIP-SigLIP, VQ-VQ) share the same latent space for input and output while others (SigLIP-VQ, VQ-SigLIP) do not. Two controlled perturbations supply the causal evidence: a reversible affine transformation inserted after the understanding vision adapter to deliberately misalign input and output spaces, and biased datasets that suppress one attribute in understanding data but keep it in generation data. The evaluation measures understanding with per-attribute VQA accuracy and generation with FID, and the biased-attribute experiments are supported by t-SNE visualizations and linear probing of the vision tokens.

What would settle it

A decisive test would train a unified VLM and an understanding-only VLM on natural images in which one attribute, say object color, is nearly absent from VQA and caption data but present in text-to-image data, then compare VQA accuracy on that attribute. If the unified model does not clearly beat the understanding-only model, the claimed generation-to-understanding transfer does not generalize beyond the synthetic setting.

Watch

Extended reading notes

Core claim

The central discovery is that unified VLMs trained with a mixture of understanding and generation data exhibit mutual benefits across all tested architectures: they outperform understanding-only and generation-only counterparts on both VQA accuracy and FID, a lower-is-better measure of generated-image quality. This mutual benefit scales with data, so increasing generation data alone improves understanding performance and increasing understanding data alone improves generation performance. The paper further claims that better alignment between the vision input and output spaces causes better generalization, since inserting a reversible affine distortion between the two spaces selectively degrades unified models while leaving understanding-only models intact. Finally, when an attribute is made scarce in understanding data but remains present in generation data, unified models still learn it with near-perfect accuracy, while understanding-only models fail; the authors show this transfer survives even when the two vision adapters are separate, and locate the mechanism in the base language model's implicit alignment of input and output vision representations.

Load-bearing premise

The central case rests on the assumption that a synthetic watch-face dataset with six rule-controlled attributes is a faithful proxy for real-world visual understanding and generation, since only one real-data test is offered.

Editorial extensions

If this is right

  • Under the paper's findings, vision-language models should be trained on both understanding and generation data rather than one task alone, since the mixed model beats both the understanding-only and generation-only counterparts.
  • Mutual benefits scale with data: increasing generation data improves understanding accuracy even when understanding data is fixed, and increasing understanding data improves generation quality when generation data is fixed.
  • Alignment between vision input and output spaces is a design lever: shared or closely aligned vision spaces improve cross-task generalization, while a reversible distortion of the input space selectively reduces the unified model's advantage.
  • Generation data can fill gaps in understanding data: when an attribute is nearly absent from VQA and captioning but present in text-to-image instructions, unified models still learn it, and the learned relationship lives in the base language model, not just in the vision adapters.
  • The real-data experiment extends the conclusion to a standard open VLM: adding generation training to the LLaVA-1.5-7B recipe improves several benchmarks without sacrificing others.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper does not run is whether diffusion-based unified VLMs, which are excluded from the study, show the same cross-task transfer; if they do not, the mutual-benefit claim would be limited to autoregressive token and embedding architectures.
  • Because the paper locates the transfer mechanism in the base language model, a natural extension is to scale the LLM base and measure whether the transfer effect amplifies; the paper uses only a 7B base, so the claim is currently tied to that scale.
  • The synthetic-dataset transfer could be validated on natural images by repeating the biased-attribute experiment with real VQA and text-to-image data, which would separate dataset simplicity from a general principle of cross-task generalization.
  • A practical consequence the authors leave implicit is that when understanding data for a rare visual concept is expensive to annotate, generation training may substitute for some of that data, changing how data budgets for VLMs are allocated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates whether unified vision-language models (VLMs) trained jointly on visual understanding and generation outperform task-specific models. Using a synthetic Smart Watch UI dataset with six discrete visual attributes, it compares four unified architectures (SigLIP-VQ, VQ-VQ, SigLIP-SigLIP, VQ-SigLIP) against understanding-only and generation-only baselines, and reports mutual benefits, scaling with data, alignment effects from artificial affine distortions, and generation-to-understanding knowledge transfer. A real-world experiment based on LLaVA-1.5 is included as an additional check. The central conclusion is that unified modeling is necessary for vision-language tasks.

Significance. If the central claims held, the paper would provide a strong, controlled argument for unified VLMs and useful architectural guidance, especially the role of shared vision input and output spaces. The strengths are the controlled synthetic dataset, the coverage of multiple unified architectures, the attribute-biased transfer experiments, and the t-SNE/linear-probing analysis of where knowledge transfer occurs. However, the headline comparisons are confounded by unequal training data quantities, the scaling evidence is non-monotonic and lacks error bars, and the real-world validation relies on an external baseline rather than a matched run. These issues mean the main claims are not yet established.

major comments (4)
  1. [Sec. 4, Fig. 3, Table A2] The headline comparison is confounded by total training data. SigLIP-VQ and VQ-VQ are trained on 120K understanding plus 60K generation samples, while SigLIP_u is trained on 120K understanding samples and VQ_g on 60K generation samples; no 180K understanding-only or 180K generation-only baseline is reported. The reported superiority of unified models could therefore reflect exposure to 1.5 to 3 times more training data rather than cross-task generalization, so the paper's central claim is not established by these comparisons.
  2. [Sec. 4.2, Fig. 5, Table A3] The scaling claim is not supported by the reported numbers. In Table A3, VQ-VQ total VQA accuracy goes 89.3 (120K understanding, 60K generation), 90.2 (180K/60K), 94.0 (240K/60K), then drops to 91.8 (300K/60K); SigLIP-VQ goes 85.6, 95.9, 90.6, 99.3 across the same understanding-data increases, and in the generation-data direction VQ-VQ total accuracy is 89.3, 88.3, 84.5, 89.3. These non-monotonic trends, without multiple seeds or error bars, do not demonstrate that mutual benefits scale monotonically with data, and the claim that adding generation data can be more efficient than adding understanding data also lacks a matched-data comparison.
  3. [Sec. 4.4, Table 1] The real-world experiment compares the unified LLaVA variant, trained on the original data plus 350K generation samples, against external official LLaVA-1.5 numbers, not against an understanding-only LLaVA trained on the same total data. Any improvement could be a data-volume effect. Additionally, only one base architecture is tested, and several benchmark differences are within one or two points, so the real-world evidence for the central claim is weak.
  4. [Sec. 3.3, Fig. 3, Table A2] The SigLIP-SigLIP configuration is presented as a unified VLM, but no generation-side FID evaluation is reported for it, and the paper does not explain how generated SigLIP embeddings are decoded into images for FID computation. Consequently, the claim that unified models improve generation performance is only demonstrated for models with VQ-based generation heads, not for all four architectures described in Section 3.4.
minor comments (5)
  1. [Throughout] There are several grammar and typographical issues, including 'mutual benefits can scale up' in the abstract, a duplicated 'Specifically, Specifically,' in Section 1, and subject-verb agreement errors such as 'Vision-Language Models(VLMs) has emerged'; these should be corrected.
  2. [Sec. 3.2] The time accuracy formula 1 - error_h/6 - error_m/30 - error_s/30 can produce negative scores, and the weights 6, 30, 30 are unexplained; the battery accuracy formula 1 - error_b/100 is similarly ad hoc. Please justify these metrics or normalize them so that they behave as accuracies.
  3. [Table A3] The total_acc column in Table A3 is not the arithmetic mean of the four listed attribute accuracies; for example, SigLIP-VQ at 180K/60K has attribute accuracies 80.4, 100, 100, 100, whose mean is 95.1, but the table reports 95.9. Please clarify how total_acc is computed.
  4. [Fig. 4] The axis label rendered as 'w\o-Affine' should be typeset as 'w/o Affine', and the legend text in Figures 3 and 4 is difficult to read at the current resolution.
  5. [Table A3] Rows with 0 understanding data list '-' for all accuracy columns but report an FID score; please clarify that generation-only models are not evaluated on VQA, and consider reporting FID for the generation-only baselines at larger data quantities for completeness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's conclusions are empirical comparisons on independently generated data, not derivations that reduce to their own inputs.

full rationale

This is an empirical study rather than a formal derivation, so the circularity patterns do not apply. The central claim—that mixed understanding and generation training yields mutual benefits—is tested by comparing unified models against understanding-only and generation-only baselines on independently generated synthetic test sets and on external real-world benchmarks. The compared quantities are not identical by construction, and no fitted parameter is subsequently renamed as a prediction. The alignment claim is operationalized through adapter sharing versus separation and through an affine distortion intervention, both of which are experimentally tested rather than assumed. The knowledge-transfer claim is supported by biased-attribute experiments and linear probing, again independently measured. Citations to prior unified-VLM papers and to the LMMs-Eval benchmark provide context and external evaluation numbers, but they are not the load-bearing evidence for the paper's conclusions, and no self-citation chain forbids alternative explanations. The noted training-data imbalance between unified and task-specific models is a possible validity confound, not a circularity: it concerns whether the experiments isolate cross-task generalization from data quantity, not whether an output reduces by definition to an input. Under the rule that a non-finding is appropriate when no step exhibits a specific equation-level or fitted-parameter reduction, the correct circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no theoretical parameters; the free parameters listed are hand-chosen metric and loss weights that could affect quantitative scores but not the qualitative direction of the findings. The axioms are domain assumptions about representativeness of the synthetic dataset and metrics.

free parameters (3)
  • time accuracy error weights (divide by 6, 30, 30)
    Hand-chosen metric weights in Section 3.2; the soft accuracy formula could under- or over-state model capability, but conclusions rely on relative comparisons.
  • battery accuracy denominator (100)
    Part of the battery accuracy metric (1 - error/100), chosen by hand; relative comparisons are robust.
  • generation loss weight for SigLIP-VQ = 0.2
    Set to 0.2 in the SigLIP-VQ model to balance cross-entropy loss scales (Appendix A); affects the understanding-generation trade-off in that model.
assumptions (3)
  • domain assumption The synthetic Smart Watch UI dataset approximates real-world vision-language tasks.
    Section 3.1 describes the six-attribute rule-based dataset; the paper's main experiments all use it.
  • domain assumption VQA accuracy and FID are valid measures of understanding and generation quality.
    Section 3.2 defines metrics; these are standard but the custom soft accuracy formulas are not externally validated.
  • domain assumption The four architecture configurations are representative unified VLMs.
    Section 3.4; diffusion-based unified models (Emu3, Transfusion) are excluded, acknowledged in Limitations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Unified Vision-Language Models Necessary: Generalization Across Understanding and Generation." pith.science (2026). https://pith.science/paper/EN5VB2LH

@misc{pith2026250523043,
  author       = {Pith},
  title        = {Pith review of: Are Unified Vision-Language Models Necessary: Generalization Across Understanding and Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EN5VB2LH}},
  note         = {Machine review of arXiv:2505.23043}
}
read the original abstract

Recent advancements in unified vision-language models (VLMs), which integrate both visual understanding and generation capabilities, have attracted significant attention. The underlying hypothesis is that a unified architecture with mixed training on both understanding and generation tasks can enable mutual enhancement between understanding and generation. However, this hypothesis remains underexplored in prior works on unified VLMs. To address this gap, this paper systematically investigates the generalization across understanding and generation tasks in unified VLMs. Specifically, we design a dataset closely aligned with real-world scenarios to facilitate extensive experiments and quantitative evaluations. We evaluate multiple unified VLM architectures to validate our findings. Our key findings are as follows. First, unified VLMs trained with mixed data exhibit mutual benefits in understanding and generation tasks across various architectures, and this mutual benefits can scale up with increased data. Second, better alignment between multimodal input and output spaces will lead to better generalization. Third, the knowledge acquired during generation tasks can transfer to understanding tasks, and this cross-task generalization occurs within the base language model, beyond modality adapters. Our findings underscore the critical necessity of unifying understanding and generation in VLMs, offering valuable insights for the design and optimization of unified VLMs.

Figures

Figures reproduced from arXiv: 2505.23043 by the authors.

Figure 1
Figure 1. Unified VLMs surpass understanding￾only and generation-only models. Alignment of vision input-output spaces further boosts perfor￾mance. Results from Section4. Despite the growing interest in unified VLMs, most existing works have predominantly fo￾cused on architectural innovations or training strategies. However, a critical question remains largely unexplored: Are unified vision-language models (VLMs) truly necessa… view at source ↗
Figure 2
Figure 2. Samples from the Smart Watch UI Dataset with different ground truth attributes. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Image understanding and generation performance of VLMs during training. “_g” denotes [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison between VLMs with and without vision input space distortion. Vision [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Performance of SigLIP-VQ and VQ-VQ under varying data scales. Only increase the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison between unified VLMs and understanding-only VLMs, trained on [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Performance comparison between unified VLMs and understanding-only VLMs, trained on [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: t-SNE visualization of input vision tokens corresponding to the ViT patch representing the [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Multimodal pretraining transfers asymmetrically: language boosts vision, understanding boosts generation, generation is mostly neutral, and early unified training prevents vision laziness.

  2. Unified Audio Intelligence Without Regressing on Text Intelligence

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified 30B MoE audio-text LLM achieves state-of-the-art audio understanding, generation, and speech tasks while preserving text reasoning comparable to its text-only backbone.

  3. Transferability Between Understanding and Generation in Unified Multimodal Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Cross-task capability transfer in UMMs is architecture-dependent and can be exploited by training understanding to improve generation while avoiding distribution shift.

  4. IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Interleaving motion generation with text-motion assessment and refinement improves alignment between generated human motion and goal text.

  5. Skywork-R1V3 Technical Report

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A 38B open-source VLM reaches 76.0% on MMMU using RL post-training and connector-only tuning, with a critical-token entropy metric for checkpoint selection.

Reference graph

Works this paper leans on

20 extracted references · 2 canonical work pages · cited by 5 Pith papers

  1. [1]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024a

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024a. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning....

    arXiv

  2. [3]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952,

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952,

    arXiv

  3. [4]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction.Advances in neural information processing systems, 37:84839–84865, 2024a

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction.Advances in neural information processing systems, 37:84839–84865, 2024a. Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal unde...

    arXiv

  4. [7]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024b. Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,

    arXiv

  5. [8]

    Metamorph: Multimodal understanding and generation via instruction tuning.arXiv preprint arXiv:2412.14164,

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning.arXiv preprint arXiv:2412.14164,

    arXiv

  6. [9]

    Liquid: Language models are scalable and unified multi-modal generators.arXiv preprint arXiv:2412.04332, 2024a

    J Wu, Y Jiang, C Ma, Y Liu, H Zhao, Z Yuan, S Bai, and X Bai. Liquid: Language models are scalable and unified multi-modal generators.arXiv preprint arXiv:2412.04332, 2024a. Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual un...

    arXiv

  7. [11]

    Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer.arXiv preprint arXiv:2401.10208, 2024b

    10 Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, et al. Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer.arXiv preprint arXiv:2401.10208, 2024b. Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wan...

    arXiv

  8. [12]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222,

    arXiv

Show all 20 references
  1. [14]

    Instruction tuning with gpt-4.arXiv preprint arXiv:2304.03277,

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4.arXiv preprint arXiv:2304.03277,

    arXiv

  2. [15]

    Sharegpt4v: Improving large multi-modal models with better captions.arXiv preprint arXiv:2311.12793,

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions.arXiv preprint arXiv:2311.12793,

    arXiv

  3. [16]

    Lmms- eval: accelerating the development of large multimoal models (2024).URL: https://github

    B Li, P Zhang, K Zhang, F Pu, X Du, Y Dong, H Liu, Y Zhang, G Zhang, C Li, et al. Lmms- eval: accelerating the development of large multimoal models (2024).URL: https://github. com/EvolvingLMMs-Lab/lmms-eval. Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real...

    2024

  4. [17]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394,

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394,

    arXiv

  5. [18]

    Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024b

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024b. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfen...

    arXiv

  6. [19]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355,

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355,

    arXiv

  7. [393]

    _u” refers to understanding-only; “_g

    To optimize GPU memory usage, we employed Low-Rank Adaptation (LoRA) [Hu et al., 2022], setting lora_r to 128 and lora_alpha to 256, with a learning rate of2e−4 following the configuration in LLaV A-1.5. The learning rates for the understanding vision adapter, generation visio...

    2022

  8. [2017]

    Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525,

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525,

    arXiv

  9. [2022]

    Dreamllm: Synergistic multimodal comprehension and creation.arXiv preprint arXiv:2309.11499,

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation.arXiv preprint arXiv:2309.11499,

    arXiv

  10. [2023]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024a

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024a. James Betker, Gabriel Goh,...

    arXiv

  11. [2024]

    Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039,

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039,

    arXiv

  12. [2025]

    Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528,

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.