Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MindOmni claims that reinforcement learning with a CLIP consistency reward lets a unified vision-language model generate explicit chains of thought before images, reaching 0.71 on WISE and 0.83 on GenEval.

desk verdict Solid RL-for-generation pipeline with honest ablations, but the headline reasoning-generation gains are partly a data-boosted variant and the CLIP consistency reward likely overlaps with WISE's scoring, so the central claim needs a careful referee. read the letter →

arxiv 2505.13031 v2 pith:R5UDKTPT submitted 2025-05-19 cs.AI

Yicheng Xiao , Lin Song , Yukang Chen , Yingmin Luo , Yuxin Chen , Yukang Gan , Wei Huang , Xiu Li
show 2 more authors
Xiaojuan Qi Ying Shan
This is my paper · ORCID
classification cs.AI
keywords reasoninggenerationvisionlanguagemodelreinforcementlearningchain-of-thoughttext-to-imageunifiedmultimodalCLIPconsistencyrewardRGPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MindOmni is a unified multimodal model that both understands and generates images, and the paper argues that a three-stage pipeline can make it produce explicit chains of thought before rendering. The stages are: pretraining a lightweight connector between a vision-language backbone and a decoder-only diffusion module, supervised fine-tuning on coarse-to-fine instruction data that includes reasoning, and a new reinforcement-learning step called Reasoning Generation Policy Optimization (RGPO). The central claim is that RGPO's multimodal consistency reward, CLIP similarity between the generated image and the ground-truth prompt, is what turns ordinary text-to-image generation into reasoning generation for prompts that need arithmetic, world knowledge, or spatial and temporal inference. If the claim holds, it offers a recipe for teaching generative models to reason aloud before drawing without sacrificing their basic generation or understanding abilities.

What carries the argument

The central object is RGPO, a group-relative policy optimization algorithm modified for multimodal rollouts. During training, the policy emits a chain of thought in text and then a diffusion-decoded image; the reward combines a format check for the CoT tags with a consistency reward computed as CLIP cosine similarity between the generated image and the ground-truth prompt. Separate KL regularizers for text and image distributions stabilize training and prevent the model from forgetting basic generation and understanding during the RL stage.

What would settle it

Score MindOmni's WISE outputs with a human panel or with detection-based checks, such as counting objects or verifying colors and spatial relations, instead of embedding similarity; if the 0.71 overall score falls toward the 0.43 no-thinking baseline while CLIP-based scores stay high, the consistency reward is optimizing the evaluator rather than the reasoning itself.

Watch

Extended reading notes

Core claim

The paper's central claim is that reinforcement learning can be applied to the text-and-image generation policy of a unified model with a reward that looks at the rendered image, not just the text tokens. RGPO samples several chain-of-thought-and-image rollouts per prompt, scores them with a format reward, whether the CoT is wrapped in <think> and <answer> tags, and a consistency reward, CLIP cosine similarity between the generated image and the reference prompt, normalizes these scores into an advantage, and updates the policy with a clipped GRPO-style objective. Two KL regularizers, one on text tokens and one on visual features, keep the policy near its reference model. The paper reports that this raises the model's overall WISE reasoning-generation score to 0.60, or 0.71 with higher-quality data and thinking mode, keeps GenEval at 0.81/0.83, and preserves MMMU understanding at 51.6, close to the original backbone.

Load-bearing premise

The consistency reward in RGPO is CLIP cosine similarity between the generated image and the ground-truth prompt, and the paper does not specify how WISE computes its scores; if WISE also relies on the same embedding-based semantic matching, the RL signal is directly optimizing the benchmark, and the reported gains could reflect reward overfitting rather than general reasoning.

Editorial extensions

If this is right

  • Explicit reasoning-before-generation can be trained with reinforcement learning, not just hand-written CoT templates.
  • The consistency reward ties language quality to image correctness; as RL proceeds, both consistency reward and output length increase.
  • Without the CoT SFT stage, applying RGPO alone yields weaker results (WISE 0.49 vs 0.60), so the SFT stage is load-bearing for the RL gain.
  • Separate KL regularizers let the model improve reasoning without sacrificing basic generation: GenEval stays at 0.83 and understanding benchmarks remain near the backbone.
  • The method generalizes across reasoning categories such as culture, time, space, biology, physics, and chemistry, with the largest margins over prior models in time and physics reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the consistency reward uses CLIP similarity and the paper does not state WISE's evaluation metric, a reader should test whether WISE also scores images by embedding similarity; if it does, part of the measured reasoning gain could be reward overfitting rather than general reasoning.
  • A testable extension is to replace the CLIP consistency reward with object-detection-based or question-answering-based rewards; RGPO only requires a differentiable reward, so the same pipeline could optimize for more concrete visual properties.
  • The same RL recipe could be applied to vision editing by defining a consistency reward on the edited region, such as VAE feature similarity to the reference, which the paper shows but does not train with RGPO.
  • The observed weakness that longer completions do not linearly improve WISE suggests a future direction of allocating reasoning length by problem difficulty, which the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MindOmni, a unified vision-language model that performs both image understanding and generation, and introduces a three-stage training pipeline. The first stage trains a connector between a VLM (Qwen2.5-VL) and a decoder-only diffusion module (OmniGen); the second stage performs supervised fine-tuning on chain-of-thought (CoT) instruction data; the third stage applies the proposed Reasoning Generation Policy Optimization (RGPO), an RL algorithm with a format reward, a CLIP-based consistency reward, and separate text and image KL regularizers. The authors report state-of-the-art scores on WISE (0.71), GenEval (0.83), and DPG-Bench (83.0), while preserving understanding performance (MMMU 51.6, MMBench 83.2). The central claim is that RL with multimodal feedback, especially the CLIP consistency reward, improves the model's ability to generate images from prompts that require explicit reasoning.

Significance. If the results hold, the paper would make a useful contribution to reasoning-driven image generation by demonstrating that reinforcement learning can be applied to a unified multimodal model with diffusion-based generation. The three-stage pipeline is clearly described, and the ablations (Tables 5, 7, 8, 9) provide useful information about stage contributions, KL coefficients, group sizes, and reward functions. The code release promise and the relatively honest reporting of some negative controls (e.g., 'w/o thinking') are positive aspects. However, the headline WISE result is confounded by the use of additional external data in the starred variant, and the paper never specifies WISE's evaluation metric or analyzes its overlap with the CLIP-based consistency reward. Additionally, the ablation evidence attributes only a small, possibly non-significant gain to the consistency reward itself, which weakens the claim that multimodal feedback is the key driver of RGPO. These issues need to be resolved before the paper's central claims can be fully accepted.

major comments (3)
  1. [Abstract; Table 3; Sec. 4.2] The headline WISE score of 0.71 is reported for MindOmni*, the variant trained with additional higher-quality external data (references [25, 2]) and evaluated with thinking mode. The abstract and introduction state this as the method's result without qualification, while the unstarred MindOmni achieves 0.60. Because the gain from 0.60 to 0.71 is confounded with the change in training data, it cannot be attributed to the proposed RGPO algorithm. The paper should either report the unstarred result in the abstract or explicitly separate the contributions of data quality and RGPO, for example by training the starred variant without Stage 3.
  2. [Sec. 3.4; Table 3; Appendix A.1] The consistency reward in RGPO is defined as the CLIP cosine similarity between the generated image and the ground-truth prompt, but the manuscript never specifies how WISE (reference [23]) computes its score. If WISE relies on CLIP or a similar embedding-based semantic matching, then RGPO is directly optimizing a proxy of the benchmark's scoring function, and the reported WISE gains from RL would reflect reward overfitting rather than a general improvement in reasoning generation. The paper must state WISE's evaluation metric and provide an analysis of the overlap, for example by correlating the CLIP consistency reward with WISE scores or by ablating the reward with an alternative semantic-alignment measure (e.g., a VQA-based reward) to show that the effect is not driven by metric overlap.
  3. [Sec. 4.4; Table 9] The ablation in Table 9 shows that the format reward improves WISE from 56.1 to 59.4, while adding the consistency reward only increases it to 59.8, a gain of 0.4 points. The text states that 'Consistency reward plays a key role in steering reasoning generation,' but this is not supported by the numbers. Moreover, the format reward is a text-only reward that checks the presence of <think> and <answer> tags; therefore the ablation does not demonstrate that multimodal feedback (the claimed novelty of RGPO over GRPO) is the main driver of the observed WISE improvement. The authors should present the individual reward contributions more fairly and, if they wish to claim that the consistency reward is important, provide additional evidence such as an evaluation on a held-out reasoning-generation benchmark.
minor comments (5)
  1. [Sec. 4.2; Table 4] The text says the model's understanding performance after RL is 'nearly identical to the original VLM backbone, with only a 0.1% average difference,' but Table 4 shows differences of +0.5 (MMMU), -0.2 (MMB), and -0.8 (RWQA), which average -0.17 in signed difference and 0.5 in absolute difference; please reconcile this statement with the table.
  2. [Table 3] 'MindOmni (w/o thinking)' scores 0.43 on WISE, which is below the OmniGen baseline of 0.44; the paper does not comment on this regression. It would be helpful to explain why the base model without CoT thinking performs worse than the diffusion-only baseline.
  3. [Eq. (5)] The text KL divergence term is written as π_ref/π_θ - log(π_ref/π_θ) - 1, which is a form of KL(π_ref||π_θ), but the notation D^T_KL(π_θ||π_ref) suggests the opposite direction; please clarify the definition, the expectation, and whether the term is per-token or sequence-level.
  4. [Appendix A.3] The limitation section mentions that CoT content is in plain text, but it does not mention the potential reward-evaluation overlap on WISE or the data confound for the starred variant; these are directly relevant to the paper's main claims.
  5. [Throughout] There are several typos and minor errors, including 'Ldiffuison' in Eq. (2), 'responss' in the Fig. 10 caption, and 'Qualitatvie' in Appendix A.4; the manuscript should be proofread.

Circularity Check

0 steps flagged · score 0.0 of 10

No demonstrated circularity: the CLIP-based RGPO reward is a training objective and the paper's headline results are measured on external benchmarks; any reward-evaluation overlap with WISE is not specified and therefore not a by-construction reduction.

full rationale

MindOmni's derivation chain is self-contained with respect to external benchmarks. The consistency reward in Sec. 3.4 is defined as CLIP cosine similarity between the generated image and the reference ground-truth prompt, but this is a training-time reward used in Eq. (4) and Eq. (7); it is not itself presented as a prediction. The paper's reasoning-generation claims are supported by the external WISE benchmark [23], GenEval [12], DPG-Bench [16], and understanding benchmarks, plus stage-wise ablations (Tables 5, 7, 8, 9). The paper does not state that WISE's overall score is computed with the same CLIP similarity, so the specific reduction that would make the RGPO reward equal to the evaluation metric is not exhibited; asserting that equivalence would require speculation about the external benchmark's internals. The ablations attribute WISE gains to SFT and RGPO empirically rather than by construction, and the RL objective is a standard GRPO-style policy loss with added KL regularizers. Self-citations ([48], [49]) appear only in implementation details (cosine scheduler and computer-vision data mentions) and are not load-bearing. The lack of detail on WISE's metric is a legitimate correctness or reward-overfitting risk, but it is not an established circular step under the requirement to quote a specific reduction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the validity of the CLIP-based consistency reward, the independence of the WISE benchmark from that reward, the quality of model-generated CoT data, and the preservation of the backbone capabilities. The paper tunes several hyperparameters (KL coefficients, group size) that affect the reported results.

free parameters (4)
  • KL coefficient beta_T (text) = 0.01
    Tuned hyperparameter; default value used in RGPO (Sec 4.1).
  • KL coefficient beta_I (image) = 0.06
    Tuned hyperparameter; default value used in RGPO (Sec 4.1).
  • Group number G = 4
    Number of rollouts per prompt in RGPO; ablation shows G=4 best (Table 8).
  • Learning rates and resolutions per stage = 1e-4/5e-5, 256/512
    Training schedule details in Sec A.1; these affect results but are standard.
assumptions (4)
  • domain assumption CLIP cosine similarity between generated image and ground-truth prompt is a valid reward for reasoning generation.
    Used in consistency reward (Sec 3.4); assumes CLIP captures the semantic alignment that the task requires.
  • domain assumption The WISE benchmark is an independent and valid measure of reasoning generation.
    Used for evaluation (Table 3); the paper does not detail WISE's metric, so the overlap with the CLIP reward is unknown.
  • domain assumption Qwen2.5-VL and OmniGen are suitable backbones and retain their capabilities after integration.
    The model builds on these pretrained models (Sec 3.2) and assumes the connector preserves their abilities.
  • domain assumption The CoT instruction data generated by Qwen2.5-VL and Qwen3 is high quality and unbiased.
    Stage 2 and 3 use model-generated reasoning texts (Sec A.1); systematic errors in these generators could be amplified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO." pith.science (2026). https://pith.science/paper/R5UDKTPT

@misc{pith2026250513031,
  author       = {Pith},
  title        = {Pith review of: MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R5UDKTPT}},
  note         = {Machine review of arXiv:2505.13031}
}
read the original abstract

Recent text-to-image systems face limitations in handling multimodal inputs and complex reasoning tasks. We introduce MindOmni, a unified multimodal large language model that addresses these challenges by incorporating reasoning generation through reinforcement learning. MindOmni leverages a three-phase training strategy: i) design of a unified vision language model with a decoder-only diffusion module, ii) supervised fine-tuning with Chain-of-Thought (CoT) instruction data, and iii) our proposed Reasoning Generation Policy Optimization (RGPO) algorithm, utilizing multimodal feedback to effectively guide policy updates. Experimental results demonstrate that MindOmni outperforms existing models, achieving impressive performance on both understanding and generation benchmarks, meanwhile showcasing advanced fine-grained reasoning generation capabilities, especially with mathematical reasoning instruction. All codes will be made public at https://github.com/TencentARC/MindOmni

Figures

Figures reproduced from arXiv: 2505.13031 by the authors.

Figure 1
Figure 1. Core capabilities of our proposed unified model, MindOmni. i) Multimodal Reasoning Generation: MindOmni incorporates our proposed RGPO reinforcement learning algorithm to improve Chain-of-Thought generation, resulting in more interpretable and responsible reasoning outputs, thereby advancing reasoning generation. ii) Beyond style transfer, MindOmni preserves low-level details, such as texture, posture, and spatial l… view at source ↗
Figure 2
Figure 2. Overview of our inference framework. MindOmni accomplishes vision understanding, multimodal reasoning generation, and vision editing tasks in a unified large model. independent structure limits its flexibility and adaptability. Meanwhile, SimpleAR [42] demonstrates the effectiveness of reinforcement learning algorithms in discrete autoregressive generative models. However, previous approaches remain focused on stren… view at source ↗
Figure 3
Figure 3. Overview of Training Pipeline. We propose a three-stage training framework comprising pretraining, instruction-based supervised fine-tuning, and reinforcement learning with RGPO. is a diffusion decoder that offers a promising X2Image generation performance with a streamlined, LLM-style network design. We utilize a connector comprising two standard LLM decoder layers to bridge two models. Additionally, by leveraging … view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Curves of different Metric in RGPO. “Completion Length” indicates the output length of the policy model during rollout process. 100 200 300 400 500 600 57.0 57.5 58.0 58.5 59.0 59.5 60.0 60.5 Overall Score (%) 58.0 59.2 58.8 59.8 57.8 59.6 Training step [PITH_FULL_IMA…
Figure 5
Figure 5. Figure 5: Scatter plot showing the performance on WISE Benchmark. The Y and X axes represent the overall score and training step during RL, respec￾tively. The size of each circle reflects the completion length of our model in a certain step. Model Size MMMU MMB RWQA Und. only LL…
Figure 6
Figure 6. Figure 6: Qualitative comparison among leading models on reasoning-aware image generation. Stage 1 Stage 2 Stage 3 GenEval WISE ✓ 0.73 0.42 ✓ ✓ 0.81 0.54 ✓ ✓ 0.72 0.49 ✓ ✓ ✓ 0.81 0.60 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison among leading models on reasoning-aware image generation with multimodal user input. β T β I GenEval WISE 0 0.01 0.79 58.8 0.01 0.06 0.81 59.8 0.004 0.01 0.80 58.9 0.01 0 0.78 58.2 [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: More qualitative results across different resolutions. A.1 Implementary Details In the first stage of training, we utilize the Laion COCO [32] and JourneyDB [35] image-caption pairs as foundational text2image data. Additionally, we incorporate the X2I dataset [47], whi…
Figure 9
Figure 9. Figure 9: Detailed response of our MindOmni. A.3 Limitation In this work, our Chain-of-Thought (CoT) content is presented in plain text, while multimodal CoT processes will be addressed in future research. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Detailed responss of our MindOmni. A.4 More Qualitatvie Results We provide more qualitative results about text2image generation of our MindOmni as shown in [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Detailed response of our MindOmni. User: <IMG1> Generate an image of the animal growing up. the growth of the lungs for breathing air, and the transformation of the tail into hind legs. The skin also thickens, becoming smoother and drier. Given the instruction to gene…
Figure 12
Figure 12. Figure 12: Detailed response of our MindOmni. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Demystifying Video Reasoning

    cs.CV 2026-03 conditional novelty 7.0 of 10

    Video diffusion models reason along the denoising trajectory (Chain-of-Steps), not primarily across frames, and this mechanism can be nudged by ensembling latent trajectories.

  2. LoRA-Gen: Specializing Large Language Model via Online LoRA Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LoRA-Gen generates task-specific LoRA weights with a cloud-side LLM and reparameterizes them into a smaller edge model, enabling training-free specialization with compressed context.

  3. HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training recipe that initializes different depth segments of one transformer from pretrained ViT, LLM, and DiT models, then jointly tunes them to do multimodal understanding and generation.

  4. TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    TokLIP semanticizes VQ image tokens with a causal CLIP-style encoder, improving multimodal comprehension while preserving autoregressive image generation.

Reference graph

Works this paper leans on

57 extracted references · 10 canonical work pages · cited by 4 Pith papers

  1. [23]

    arXiv preprint arXiv:2503.07265 (2025)

    Niu, Y ., Ning, M., Zheng, M., Lin, B., Jin, P., Liao, J., Ning, K., Zhu, B., Yuan, L.: Wise: A world knowledge-informed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.07265 (2025)

    arXiv 2025

  2. [1]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    arXiv 2025

  3. [2]

    arXiv preprint arXiv:2505.09568 (2025)

    Chen, J., Xu, Z., Pan, X., Hu, Y ., Qin, C., Goldstein, T., Huang, L., Zhou, T., Xie, S., Savarese, S., et al.: Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint arXiv:2505.09568 (2025)

    arXiv 2025

  4. [3]

    arXiv preprint arXiv:2310.00426 (2023)

    Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y ., Wang, Z., Kwok, J., Luo, P., Lu, H., et al.: Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426 (2023)

    arXiv 2023

  5. [4]

    r1-v: Reinforcing super generalization ability in vision-language models with less than 3

    Chen, L., Li, L., Zhao, H., Song, Y .: Vinci. r1-v: Reinforcing super generalization ability in vision-language models with less than 3

    work page

  6. [5]

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C.: Janus-pro: Unified multimodal understanding and generation with data and model scaling (2025)

    2025

  7. [6]

    Science China Information Sciences67(12), 220101 (2024) 10

    Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al.: How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences67(12), 220101 (2024) 10

    work page 2024

  8. [7]

    arXiv preprint arXiv:2505.14683 (2025)

    Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., Shi, G., Fan, H.: Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683 (2025)

    arXiv 2025

Show all 57 references
  1. [8]

    arXiv preprint arXiv:2309.11499 (2023)

    Dong, R., Han, C., Peng, Y ., Qi, Z., Ge, Z., Yang, J., Zhao, L., Sun, J., Zhou, H., Wei, H., et al.: Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499 (2023)

    2023 arXiv

  2. [9]

    arXiv preprint arXiv:2503.10639 (2025)

    Fang, R., Duan, C., Wang, K., Huang, L., Li, H., Yan, S., Tian, H., Zeng, X., Zhao, R., Dai, J., et al.: Got: Unleashing reasoning capability of multimodal large language model for visual generation and editing. arXiv preprint arXiv:2503.10639 (2025)

    2025 arXiv

  3. [10]

    arXiv preprint arXiv:2309.17179 (2023)

    Feng, X., Wan, Z., Wen, M., McAleer, S.M., Wen, Y ., Zhang, W., Wang, J.: Alphazero-like tree- search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179 (2023)

    2023 arXiv

  4. [11]

    arXiv preprint arXiv:2404.14396 (2024)

    Ge, Y ., Zhao, S., Zhu, J., Ge, Y ., Yi, K., Song, L., Li, C., Ding, X., Shan, Y .: Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396 (2024)

    2024 arXiv

  5. [12]

    Advances in Neural Information Processing Systems pp

    Ghosh, D., Hajishirzi, H., Schmidt, L.: Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems pp. 52132–52152 (2023)

    2023

  6. [13]

    Google: Gemini 2.5: Our most intelligent ai model (2025), https://blog.google/ technology/google-deepmind/

    2025

  7. [14]

    arXiv preprint arXiv:2501.12948 (2025)

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al.: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  8. [15]

    arXiv preprint arXiv:2501.13926 (2025)

    Guo, Z., Zhang, R., Tong, C., Zhao, Z., Gao, P., Li, H., Heng, P.A.: Can we generate images with cot? let’s verify and reinforce image generation step by step. arXiv preprint arXiv:2501.13926 (2025)

    2025 arXiv

  9. [16]

    arXiv preprint arXiv:2403.05135 (2024)

    Hu, X., Wang, R., Fang, Y ., Fu, B., Cheng, P., Yu, G.: Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135 (2024)

    2024 arXiv

  10. [17]

    arXiv preprint arXiv:2504.01934 (2025)

    Huang, R., Wang, C., Yang, J., Lu, G., Yuan, Y ., Han, J., Hou, L., Zhang, W., Hong, L., Zhao, H., et al.: Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement. arXiv preprint arXiv:2504.01934 (2025)

    2025 arXiv

  11. [18]

    arXiv preprint arXiv:2402.01694 (2024)

    Khanov, M., Burapacheep, J., Li, Y .: Args: Alignment as reward-guided search. arXiv preprint arXiv:2402.01694 (2024)

    2024 arXiv

  12. [19]

    arXiv preprint arXiv:2409.12917 (2024)

    Kumar, A., Zhuang, V ., Agarwal, R., Su, Y ., Co-Reyes, J.D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., et al.: Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917 (2024)

    2024 arXiv

  13. [20]

    Labs, B.F.: Flux.https://github.com/black-forest-labs/flux(2023)

    2023

  14. [21]

    arXiv:2408.03326 (2024)

    Li, B., Zhang, Y ., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y ., Liu, Z., Li, C.: Llava-onevision: Easy visual task transfer. arXiv:2408.03326 (2024)

    2024 arXiv

  15. [22]

    arXiv preprint arXiv:2503.06520 (2025)

    Liu, Y ., Peng, B., Zhong, Z., Yue, Z., Lu, F., Yu, B., Jia, J.: Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520 (2025)

    2025 arXiv

  16. [24]

    https://openai.com/o1 (2024)

    OpenAI: Openai o1. https://openai.com/o1 (2024)

    2024

  17. [25]

    OpenAI: Introducing 4o image generation (2025),https://openai.com 11

    2025

  18. [26]

    arXiv preprint arXiv:2504.06256 (2025)

    Pan, X., Shukla, S.N., Singh, A., Zhao, Z., Mishra, S.K., Wang, J., Xu, Z., Chen, J., Li, K., Juefei- Xu, F., et al.: Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.06256 (2025)

    2025 arXiv

  19. [27]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4195–4205 (2023)

    2023

  20. [28]

    arXiv preprint arXiv:2307.01952 (2023)

    Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023)

    2023 arXiv

  21. [29]

    arXiv preprint arXiv:2406.14544 (2024)

    Qiao, Y ., Duan, H., Fang, X., Yang, J., Chen, L., Zhang, S., Wang, J., Lin, D., Chen, K.: Prism: A framework for decoupling and assessing the capabilities of vlms. arXiv preprint arXiv:2406.14544 (2024)

    2024 arXiv

  22. [30]

    arXiv preprint arXiv:2412.03069 (2024)

    Qu, L., Zhang, H., Liu, Y ., Wang, X., Jiang, Y ., Gao, Y ., Ye, H., Du, D.K., Yuan, Z., Wu, X.: Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069 (2024)

    2024 arXiv

  23. [31]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)

    2022

  24. [32]

    URL https://laion

    Schuhmann, C., Köpf, A., Vencu, R., Coombes, T., Beaumont, R.: Laion coco: 600m synthetic captions from laion2b-en. URL https://laion. ai/blog/laion-coco5(2022)

    2022

  25. [33]

    arXiv preprint arXiv:2402.03300 (2024)

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

    2024 arXiv

  26. [34]

    stability.ai: Introducing stable diffusion 3.5 (2024), https://stability.ai/news/ introducing-stable-diffusion-3-5

    2024

  27. [35]

    Advances in Neural Information Processing Systems36(2024)

    Sun, K., Pan, J., Ge, Y ., Li, H., Duan, H., Wu, X., Zhang, R., Zhou, A., Qin, Z., Wang, Y ., et al.: Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems36(2024)

    2024

  28. [36]

    arXiv preprint arXiv:2406.06525 (2024)

    Sun, P., Jiang, Y ., Chen, S., Zhang, S., Peng, B., Luo, P., Yuan, Z.: Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525 (2024)

    2024 arXiv

  29. [37]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sun, Q., Cui, Y ., Zhang, X., Zhang, F., Yu, Q., Wang, Y ., Rao, Y ., Liu, J., Huang, T., Wang, X.: Generative multimodal models are in-context learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14398–14409 (2024)

    2024

  30. [38]

    arXiv preprint arXiv:2307.05222 (2023)

    Sun, Q., Yu, Q., Cui, Y ., Zhang, F., Zhang, X., Wang, Y ., Gao, H., Liu, J., Huang, T., Wang, X.: Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222 (2023)

    2023 arXiv

  31. [39]

    arXiv preprint arXiv:2405.09818 (2024)

    Team, C.: Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 (2024)

    2024 arXiv

  32. [40]

    https://qwenlm.github.io/blog/qwen3/ (2025)

    Team, Q.: Qwen3: Think deeper, act faster. https://qwenlm.github.io/blog/qwen3/ (2025)

    2025

  33. [41]

    arXiv preprint arXiv:2412.14164 (2024)

    Tong, S., Fan, D., Zhu, J., Xiong, Y ., Chen, X., Sinha, K., Rabbat, M., LeCun, Y ., Xie, S., Liu, Z.: Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412.14164 (2024)

    2024 arXiv

  34. [42]

    arXiv preprint arXiv:2504.11455 (2025)

    Wang, J., Tian, Z., Wang, X., Zhang, X., Huang, W., Wu, Z., Jiang, Y .G.: Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504.11455 (2025)

    2025 arXiv

  35. [43]

    arXiv preprint arXiv:2409.18869 (2024) 12

    Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y ., Wang, J., Zhang, F., Wang, Y ., Li, Z., Yu, Q., et al.: Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869 (2024) 12

    2024 arXiv

  36. [44]

    arXiv preprint arXiv:2203.11171 (2022)

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

    2022 arXiv

  37. [45]

    Advances in neural information processing systems35, 24824–24837 (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V ., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems35, 24824–24837 (2022)

    2022

  38. [46]

    arXiv preprint arXiv:2410.13848 (2024)

    Wu, C., Chen, X., Wu, Z., Ma, Y ., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., et al.: Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848 (2024)

    2024 arXiv

  39. [47]

    arXiv preprint arXiv:2409.11340 (2024)

    Xiao, S., Wang, Y ., Zhou, J., Yuan, H., Xing, X., Yan, R., Wang, S., Huang, T., Liu, Z.: Omnigen: Unified image generation. arXiv preprint arXiv:2409.11340 (2024)

    2024 arXiv

  40. [48]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Xiao, Y ., Luo, Z., Liu, Y ., Ma, Y ., Bian, H., Ji, Y ., Yang, Y ., Li, X.: Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18709...

    2024

  41. [49]

    Advances in Neural Information Processing Systems37, 75329–75354 (2024)

    Xiao, Y ., Song, L., Wang, J., Song, S., Ge, Y ., Li, X., Shan, Y ., et al.: Mambatree: Tree topology is all you need in state space model. Advances in Neural Information Processing Systems37, 75329–75354 (2024)

    2024

  42. [50]

    arXiv preprint arXiv:2501.18427 (2025)

    Xie, E., Chen, J., Zhao, Y ., Yu, J., Zhu, L., Wu, C., Lin, Y ., Zhang, Z., Li, M., Chen, J., et al.: Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer. arXiv preprint arXiv:2501.18427 (2025)

    2025 arXiv

  43. [51]

    arXiv preprint arXiv:2408.12528 (2024)

    Xie, J., Mao, W., Bai, Z., Zhang, D.J., Wang, W., Lin, K.Q., Gu, Y ., Chen, Z., Yang, Z., Shou, M.Z.: Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528 (2024)

    2024 arXiv

  44. [52]

    arXiv preprint arXiv:2411.10440 (2024)

    Xu, G., Jin, P., Hao, L., Song, Y ., Sun, L., Yuan, L.: Llava-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440 (2024)

    2024 arXiv

  45. [53]

    In: Forty-first International Conference on Machine Learning (2024)

    Yang, L., Yu, Z., Meng, C., Xu, M., Ermon, S., Cui, B.: Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms. In: Forty-first International Conference on Machine Learning (2024)

    2024

  46. [54]

    Yuan, L., Haodong, D., Yuanhan, Z., Bo, L., Songyang, Z., etc.: Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281 (2023)

    2023 arXiv

  47. [55]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yue, X., Ni, Y ., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y ., et al.: Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  48. [56]

    arXiv preprint arXiv:2408.11039 (2024)

    Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., Levy, O.: Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039 (2024)

    2024 arXiv

  49. [57]

    aha moment

    Zhou, H., Li, X., Wang, R., Cheng, M., Zhou, T., Hsieh, C.J.: R1-zero’s" aha moment" in visual reasoning on a 2b non-sft model. arXiv preprint arXiv:2503.05132 (2025) 13 A Appendix Figure 8:More qualitative results across different resolutions. A.1 Implementary Details In the ...

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.