Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read UniRL claims a unified multimodal model can improve both generation and understanding by training on its own generated images, with no external image data.

desk verdict The self-improvement loop is ungrounded: the reward comes from the prompt, not the generated image, so the claimed mutual enhancement is likely memorization — but the setup is novel and worth a conditional revision. read the letter →

arxiv 2505.23380 v1 pith:ZAKNLGD2 submitted 2025-05-29 cs.CV

Weijia Mao , Zhenheng Yang , Mike Zheng Shou This is my paper
classification cs.CV
keywords unifiedmultimodalmodelsself-improvingpost-trainingGRPOsupervisedfine-tuningtext-to-imagegenerationunderstandingtaskimbalance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes UniRL, a post-training method in which a unified multimodal model (one that both draws and understands images) improves itself without any external image data. In each round, the model generates images from text prompts, answers questions about those same images, and is then optimized so that both tasks improve together. The authors show that this loop works with supervised fine-tuning and with Group Relative Policy Optimization, and report that it raises Show-o's GenEval score from 0.60 to 0.77 while also making generation and understanding more consistent. The broader point being argued is that the two abilities can reinforce each other, rather than being trained in isolation.

What carries the argument

The load-bearing mechanism is the self-training loop itself, powered by the Straight-Through Gumbel-Softmax (ST-GS) estimator, which makes the discrete image tokens produced by the generation module differentiable so that the answer-prediction loss or reward can back-propagate into image generation. In models whose two tasks share one image representation (Show-o), this allows true end-to-end joint optimization; in models with separate representations (Janus), UniRL switches to a non-end-to-end variant where the two modules are optimized by separate GRPO losses that still link through review of the same generated images. The other named machinery is Group Relative Policy Optimization (GRPO), used here without chain-of-thought: a group of images is generated from one prompt, each image is answered, rewards are normalized within the group, and the policy is updated toward higher-reward answers.

What would settle it

Run an ablation that replaces the Straight-Through Gumbel-Softmax path with hard, non-differentiable sampling (or that stops gradients at the image tokens) while keeping everything else fixed. If the GenEval score and the understanding accuracy do not drop substantially, the claimed end-to-end gradient flow is not the source of the improvement; if training collapses or freezes without the estimator, the claim is supported.

Watch

Extended reading notes

Core claim

UniRL's central claim is that a unified multimodal model can be post-trained to generate images and to understand images better by using its own generations as the training signal, in a closed loop: a prompt produces a batch of images; the images, with a question derived from the prompt, are fed back into the model to predict answers; and the correctness of those answers (via SFT loss or GRPO rewards) is used to update both the understanding path and, through a Straight-Through Gumbel-Softmax estimator, the image-generation path. Because the same generated image is used for both sides, the supervision is shared, which the paper argues reduces the imbalance between generation and understanding that it measures with new bidirectional conditional accuracies. On Show-o the method reaches 0.77 on GenEval; on Janus, which uses separate image representations, a non-end-to-end variant of the same idea reaches 0.65.

Load-bearing premise

The Straight-Through Gumbel-Softmax estimator in Equation (3) passes useful, meaningful gradients through the discrete image tokens, so that answer-correctness signals genuinely improve the image generation module rather than just the understanding module.

Editorial extensions

If this is right

  • Post-training can improve both generation and understanding on a unified model without any external image data.
  • Using the same self-generated image pool for both tasks reduces the imbalance between text-to-image generation and multimodal understanding.
  • GRPO, applied without chain-of-thought, is a more robust post-training strategy than SFT across both shared-representation (Show-o) and separate-representation (Janus) architectures.
  • A small number of additional training steps (on the order of thousands of iterations on 8 GPUs) suffices to produce these gains.
  • The proposed bidirectional metric exposes task imbalance in existing unified models and can quantify progress on it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Straight-Through Gumbel-Softmax gradient path is what carries the improvement, the same self-training loop could be applied to other unified models with discrete image tokens, such as Emu3 or Chameleon.
  • The loop's reliance on hand-designed prompts with known ground-truth answers limits it to synthetic, well-specified visual attributes; extending it to open-ended captions or reasoning would require a reward signal that is not as easy to define, such as a learned verifier.
  • Because the model's own generations are the only training data, the ceiling of the method may be tied to the base model's generation coverage; a model that cannot produce certain visual configurations may never receive gradient pressure to improve them, which could be tested by checking whether GenEval categories with low base scores (e.g., position) see bounded gains.
  • The non-end-to-end variant for Janus suggests that even without gradient sharing, reward-based cross-task coupling can align the two modules; a follow-up could measure whether this alignment transfers to models trained from scratch at lower cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes UniRL, a post-training method for unified multimodal models (Show-o and Janus) that uses the model's own generated images as training data. For each constructed prompt, the model generates images, answers a question about the image, and is optimized with either SFT or a GRPO-style loss against a prompt-derived ground-truth answer. The authors claim that this loop requires no external image data, improves both generation and understanding, and reduces the imbalance between the two tasks. They introduce a bidirectional conditional-accuracy metric for generation-understanding balance and report GenEval, DSG-1K, CLIP-Score, and custom MMU results.

Significance. If the central claims hold, the paper would be a useful contribution: self-improving post-training without external image data is appealing, the imbalance metric is a sensible diagnostic, and the reported GenEval gains are substantial and falsifiable. The paper also promises code and models. However, the significance is currently undercut by two load-bearing issues: the implemented loss is not the GRPO objective defined in the paper, and the training reward is a deterministic function of the prompt rather than of the generated image, which calls into question the claimed mutual enhancement between generation and understanding. The straight-through Gumbel-Softmax gradient path that is supposed to connect the two tasks is also unvalidated. These issues are fixable in principle, but they require additional experiments and a reformulation of the training objective.

major comments (3)
  1. [Section 3.4, Eq. (5) vs. Eq. (2)] The loss in Eq. (5) is not the GRPO objective defined in Eq. (2). Eq. (2) contains importance ratios s_j, a clipped surrogate min(s A, clip(s, 1-delta, 1+delta) A), and an average over the group; Eq. (5) is a softmax-reward-weighted maximum-likelihood term with no importance ratio and no clipping, plus a separate KL penalty. The paper's comparison of 'SFT vs GRPO' is therefore a comparison of SFT against a reward-weighted MLE, not against GRPO as claimed. Please either implement the actual GRPO surrogate or rename the method and justify reward-weighted MLE as the optimization objective.
  2. [Section 3.3-3.4, Tables 1 and 5] The training reward is not grounded in the generated image. The ground-truth answer a* is determined by the prompt (e.g., 'three vases' from 'a photo of three vases'), so for any generated image that does not contain three vases, Eq. (4) and Eq. (5) optimize the model to produce a false label for that image. The end-to-end pipeline does not filter or re-label self-generated images; Appendix A.7 only discusses error accumulation for non-end-to-end SFT. The collapse of UniRL(SFT) on the single-object category (0.99 to 0.08 in Table 5) is exactly the expected consequence of this misaligned supervision. To support the claimed mutual enhancement, the paper must (i) filter or re-label generated images so that rewards reflect image content, and (ii) include a question-only baseline to show that the MMU improvements are not achieved by ignoring the image and exploiting the prompt-derived answer.
  3. [Section 3.3, Eq. (3), and Appendix A.2] The Straight-Through Gumbel-Softmax path through discrete image tokens is central to the end-to-end training claim, but it is not validated. There is no ablation comparing end-to-end training with a detached image-token baseline, no analysis of whether answer-prediction gradients change the generation module's output, and no report of training instability or gradient statistics. Please add such an ablation (for example, a detached generation baseline or a non-end-to-end control on Show-o) to verify that the cross-task gradient path is actually informative rather than noise.
minor comments (6)
  1. [Abstract and Section 4.2, Table 4] The reported 'GenEval score of 0.77 for Show-o' corresponds to the SFT variant, while the GRPO variant achieves 0.71; please state this explicitly in the abstract to avoid ambiguity.
  2. [Section 3.4, Eq. (5)] The notation mixes pi_theta and p_theta without defining their relationship, and the KL term in Eq. (5) is written differently from the KL term in Eq. (2); please unify the notation.
  3. [Section 3.6, Eq. (6)] The definitions of N_I, N_A, and N_A_intersection_I should specify the evaluation set and sample size; as written, the reader cannot tell how many prompts per category were used or how the conditional accuracies were averaged.
  4. [Appendix A.7] The discussion of error accumulation in non-end-to-end SFT should also address the same risk in end-to-end SFT and GRPO, since the ungrounded-label problem identified in the major comments is present in both settings.
  5. [Section 3.5 and Appendix A.3] The claim that non-end-to-end GRPO enables 'cross-task interaction' is not supported by the loss formulas in A.3, which are two independent GRPO losses; please clarify the mechanism by which reward signals from one module influence the other.
  6. [Figure 2] The SFT panel shows a single generated image, but the method is described as iterative self-improvement; please indicate how iterations are performed, how many passes over the prompt set are used, and whether generated images from previous iterations are reused.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: UniRL's self-training loop is an empirical iterative procedure whose central claims are evaluated against external benchmarks, not derived from its own definitions.

full rationale

The paper's central claims are empirical rather than definitional. The self-improving loop (Sec. 3.3-3.4) generates images from prompts, feeds them back with constructed questions, and optimizes answer prediction against ground-truth answers derived from the prompt (Table 1, Eqs. 4-5). This creates a self-referential training signal and a potential reward-misalignment risk if generated images do not match the prompt, but that is a correctness and robustness concern, not a circular derivation: the loss and reward are functions of the model's own predictions and the prompt-derived target, and the claimed outcomes are measured on held-out external benchmarks (GenEval, DSG-1K, CLIP-Score, POPE, MMMU) and on out-of-distribution images from other models. The ST-GS estimator (Sec. A.2) is a standard technique described in the appendix, not an imported theorem, and its effectiveness is an empirical assumption rather than a circular premise. Self-citations to Show-o and related work are contextual and not load-bearing as justification of the method's predictions. The paper even acknowledges the instability of unfiltered self-supervision in non-end-to-end settings (Sec. A.7), which further confirms that the authors treat the loop as an empirical design choice, not as a logically forced result. No equation or training objective reduces by construction to the evaluation metric it is claimed to predict, so no circularity is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced. The balance metric is a new evaluation quantity, not an invented entity. The central claims rely on the listed self-training, gradient estimator, and reward-validation assumptions.

free parameters (6)
  • Reward score weights per category = 1 (single), 2 (two/counting), 3 (position), 4 (attribute)
    Hand-chosen keyword counts in Table 2 determine reward magnitudes and thus optimization balance across categories.
  • KL coefficient beta = 0.2
    Chosen by hand for the GRPO KL penalty in both Show-o and Janus experiments.
  • Reward temperature alpha = not reported
    Controls the sharpness of reward weights in Eq. (5); the value is not stated in the paper.
  • Group size K = 3 for Show-o GRPO
    Number of images sampled per prompt in GRPO, chosen without stated criteria.
  • ST-GS temperature tau = not reported
    Temperature in the Gumbel-Softmax estimator; not specified, affecting gradient approximation quality.
  • Training steps and learning rate = Show-o: 5000 SFT + 3000 GRPO; Janus: 1000; LR 1e-5
    Stopping points and learning rate are chosen by hand without a stated selection procedure.
assumptions (5)
  • domain assumption The model's own generated images form a useful training distribution that improves rather than degrades the model.
    Invoked in Sec. 3.3; no experiments isolate whether self-generated data helps versus hurts relative to external data.
  • ad hoc to paper ST-GS gradients through discrete image tokens are informative for the generation module.
    Invoked in Sec. 3.3 and Eq. (3); no ablation validates this in the masked-diffusion setting of Show-o.
  • domain assumption Keyword matching in the reward function is a valid measure of answer correctness.
    Table 2 defines rewards by keyword presence; a model could satisfy keywords without true visual understanding.
  • domain assumption External LLM judges (Qwen/GPT) reliably decide whether predicted answers match ground-truth answers.
    Sec. A.1 uses LLM judges for the new metric; no human validation or agreement rates are reported.
  • domain assumption GenEval's six-category taxonomy captures the fundamental visual features needed for generation-understanding alignment.
    The paper adopts GenEval categories for training and evaluation, limiting scope to basic visual attributes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning." pith.science (2026). https://pith.science/paper/ZAKNLGD2

@misc{pith2026250523380,
  author       = {Pith},
  title        = {Pith review of: UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZAKNLGD2}},
  note         = {Machine review of arXiv:2505.23380}
}
read the original abstract

Unified multimodal large language models such as Show-o and Janus have achieved strong performance across both generation and understanding tasks. However, these models typically rely on large-scale datasets and require substantial computation during the pretraining stage. In addition, several post-training methods have been proposed, but they often depend on external data or are limited to task-specific customization. In this work, we introduce UniRL, a self-improving post-training approach. Our approach enables the model to generate images from prompts and use them as training data in each iteration, without relying on any external image data. Moreover, it enables the two tasks to enhance each other: the generated images are used for understanding, and the understanding results are used to supervise generation. We explore supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) to optimize the models. UniRL offers three key advantages: (1) it requires no external image data, as all training samples are generated by the model itself during training; (2) it not only improves individual task performance, but also reduces the imbalance between generation and understanding; and (3) it requires only several additional training steps during the post-training stage. We evaluate UniRL on top of Show-o and Janus, achieving a GenEval score of 0.77 for Show-o and 0.65 for Janus. Code and models will be released in https://github.com/showlab/UniRL.

Figures

Figures reproduced from arXiv: 2505.23380 by the authors.

Figure 1
Figure 1. The imbalance between Text-to-Image Generation (T2I) and Multi-modal Understanding [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Training pipeline of UniRL (a) SFT Optimization: A prompt is used to generate a single [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison of our method with the original models on both text-to-image [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization results of our method. A.7 More analysis of SFT and GRPO End-to-end training. For end-to-end training, supervision for T2I is inherently indirect, as the optimization signal comes from downstream tasks such as answer prediction rather than direct supervis…

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Model Guides You How to Draw: Adaptive Visual Gating for Unified Multimodal Reasoning

    cs.CV 2026-07 conditional novelty 6.5 of 10

    Early Generation Intent and Visual Fidelity signals let AdaViG abort low-utility intermediate image generations in UMM math CoT, improving accuracy up to 5.7% and cutting visual FLOPs 25–91%.

  2. STBridge: Shared-Target Alignment for Bridging Understanding and Generation in UMMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Treating image-editing as a shared target between understanding and generation, with sequential reinforcement learning, improves a unified multimodal model's caption-image consistency and several benchmark scores.

  3. Reconstruction Alignment Improves Unified Multimodal Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    RECA, a self-supervised post-training objective that conditions unified multimodal models on their own visual understanding embeddings to reconstruct input images, improves text-to-image and editing benchmarks across ...

  4. SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models

    cs.CV 2025-10 conditional novelty 5.0 of 10

    A unified multimodal model can improve its own text-to-image generation by using its understanding module as a rewarder in a global-plus-local reward-weighted training loop.

Reference graph

Works this paper leans on

51 extracted references · 7 canonical work pages · cited by 4 Pith papers

  1. [1]

    Phi-3 technical re- port: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 1, 2023

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 1, 2023

    arXiv 2023

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities.CoRR, abs/2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.CoRR, abs/2308.12966, 2023

    arXiv 2023

  4. [4]

    Sft or rl? an early investigation into training r1-like reasoning large vision-language models.arXiv preprint arXiv:2504.11468, 2025

    Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training r1-like reasoning large vision-language models.arXiv preprint arXiv:2504.11468, 2025

    arXiv 2025

  5. [5]

    Mmdetection: Open mmlab detection toolbox and benchmark.CoRR, abs/1906.07155, 2019

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Mmdetection: Open mmlab detection toolbox and b...

    arXiv 1906

  6. [6]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.CoRR, abs/2501.17811, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.CoRR, abs/2501.17811, 2025

    arXiv 2025

  7. [7]

    Schwing, Alexander Kirillov, and Rohit Girdhar

    Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. InCVPR, pages 1280–

    work page

  8. [8]

    Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tuset, and Su Wang

    Jaemin Cho, Yushi Hu, Jason M. Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. InICLR. OpenReview.net, 2024

    work page 2024

Show all 51 references
  1. [9]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  2. [10]

    DreamLLM: Synergistic multimodal comprehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. DreamLLM: Synergistic multimodal comprehension and creation. InICLR, 2024

    2024

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  4. [12]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InCVPR, pages 12873–12883, 2021

    2021

  5. [13]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    2024 arXiv

  6. [14]

    Geneval: An object-focused framework for evaluating text-to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. InNeurIPS, 2023

    2023

  7. [15]

    Can we generate images with cot? let’s verify and reinforce image generation step by step.CoRR, abs/2501.13926, 2025

    Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. Can we generate images with cot? let’s verify and reinforce image generation step by step.CoRR, abs/2501.13926, 2025

    2025 arXiv

  8. [16]

    Openai o1 system card.CoRR, abs/2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya K...

    2024 arXiv

  9. [17]

    Orthus: Autoregressive interleaved image-text generation with modality-specific heads.CoRR, abs/2412.00127, 2024

    Siqi Kou, Jiachun Jin, Chang Liu, Ye Ma, Jian Jia, Quan Chen, Peng Jiang, and Zhijie Deng. Orthus: Autoregressive interleaved image-text generation with modality-specific heads.CoRR, abs/2412.00127, 2024

    2024 arXiv

  10. [18]

    Imagine while reasoning in space: Multimodal visualization-of-thought.CoRR, abs/2501.07542, 2025

    Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulic, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-of-thought.CoRR, abs/2501.07542, 2025

    2025 arXiv

  11. [19]

    Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding.CoRR, abs/2412.09604, 2024

    Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiao- gang Wang, Hongsheng Li, Lewei Lu, and Jifeng Dai. Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding.CoRR, abs/2412.09604, 2024

    2024 arXiv

  12. [20]

    Omniflow: Any-to-any generation with multi-modal rectified flows

    Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Zichun Liao, Yusuke Kato, Kazuki Kozuka, and Aditya Grover. Omniflow: Any-to-any generation with multi-modal rectified flows. CoRR, abs/2412.01169, 2024

    2024 arXiv

  13. [21]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InEMNLP, pages 292–305. Association for Computational Linguistics, 2023

    2023

  14. [22]

    Textbooks are all you need ii: phi-1.5 technical report.arXiv preprint arXiv:2309.05463, 2023

    Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report.arXiv preprint arXiv:2309.05463, 2023

    2023 arXiv

  15. [23]

    Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining.CoRR, abs/2408.02657, 2024

    Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining.CoRR, abs/2408.02657, 2024. 11

    2024 arXiv

  16. [24]

    World model on million-length video and language with ringattention.arXiv preprint, 2024

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention.arXiv preprint, 2024

    2024

  17. [25]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  18. [26]

    Visual instruction tuning.NeurIPS, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.NeurIPS, 36, 2024

    2024

  19. [27]

    Janusflow: Harmonizing autoregres- sion and rectified flow for unified multimodal understanding and generation.arXiv preprint arXiv:2411.07975, 2024

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregres- sion and rectified flow for unified multimodal understanding and generation.arXiv preprint arXiv:2411.07975, 2024

    2024 arXiv

  20. [28]

    Unimod: Efficient unified multimodal transformers with mixture-of-depths.CoRR, abs/2502.06474, 2025

    Weijia Mao, Zhenheng Yang, and Mike Zheng Shou. Unimod: Efficient unified multimodal transformers with mixture-of-depths.CoRR, abs/2502.06474, 2025

    2025 arXiv

  21. [29]

    Du, Zehuan Yuan, and Xinglong Wu

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K. Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation.CoRR, abs/2412.03069, 2024

    2024 arXiv

  22. [30]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InICML, pages 8748...

    2021

  23. [31]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InNeurIPS, 2023

    2023

  24. [32]

    Proximal policy optimization algorithms.CoRR, abs/1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.CoRR, abs/1707.06347, 2017

    2017 arXiv

  25. [33]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.CoRR, abs/2402.03300, 2024

    2024 arXiv

  26. [34]

    Llamafusion: Adapting pretrained language models for multimodal generation

    Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Llamafusion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:2412.15188, 2024

    2024 arXiv

  27. [35]

    Journeydb: A benchmark for generative image understanding

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li. Journeydb: A benchmark for generative image understanding. InNeurIPS, 2023

    2023

  28. [36]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. InICLR, 2023

    2023

  29. [37]

    Any-to-any generation via composable diffusion.NeurIPS, 36, 2024

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion.NeurIPS, 36, 2024

    2024

  30. [38]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  31. [39]

    Metamorph: Multimodal under- standing and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal under- standing and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    2024 arXiv

  32. [40]

    Llama: Open and efficient foundation language models.CoRR, abs/2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023 arXiv

  33. [41]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  34. [42]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.CoRR, abs/2410.13848, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation.CoRR, abs/2410.13848, 2024

    arXiv

  35. [43]

    Liquid: Language models are scalable multi-modal generators.CoRR, abs/2412.04332, 2024

    Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable multi-modal generators.CoRR, abs/2412.04332, 2024

    2024 arXiv

  36. [44]

    Next-gpt: Any-to-any multimodal llm.arXiv preprint arXiv:2309.05519, 2023

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm.arXiv preprint arXiv:2309.05519, 2023

    2023 arXiv

  37. [45]

    VILA-U: a unified foundation model integrating visual understanding and generation.CoRR, abs/2409.04429, 2024

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, and Yao Lu. VILA-U: a unified foundation model integrating visual understanding and generation.CoRR, abs/2409.04429, 2024

    2024 arXiv

  38. [46]

    Show-o: One single trans- former to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  39. [47]

    Qwen2.5-1m technical report.CoRR, abs/2501.15383, 2025

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang R...

    2025 arXiv

  40. [48]

    Hermesflow: Seamlessly closing the gap in multimodal understanding and generation

    Ling Yang, Xinchen Zhang, Ye Tian, Chenming Shang, Minghao Xu, Wentao Zhang, and Bin Cui. Hermesflow: Seamlessly closing the gap in multimodal understanding and generation. CoRR, abs/2502.12148, 2025

    2025

  41. [49]

    MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI

    Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A m...

    2024

  42. [50]

    Doracycle: Domain-oriented adaptation of unified generative model in multimodal cycles.CoRR, abs/2503.03651, 2025

    Rui Zhao, Weijia Mao, and Mike Zheng Shou. Doracycle: Domain-oriented adaptation of unified generative model in multimodal cycles.CoRR, abs/2503.03651, 2025

    2025 arXiv

  43. [51]

    Transfusion: Predict the next token and diffuse images with one multi-modal model.CoRR, abs/2408.11039, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model.CoRR, abs/2408.11039, 2024. 13 A Technical appendice...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.