Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read UniGen claims that a 1.5B unified multimodal model reaches state-of-the-art text-to-image results by verifying its own generated images step-by-step, scoring 0.78 on GenEval and 85.19 on DPG-Bench.

desk verdict CoT-V is a solid and transferable test-time scaling idea, but the SOTA claims rest on a Best-of-20 vs single-sample comparison and a partially trained GenEval metric. read the letter →

arxiv 2505.14682 v1 pith:ULW6Q5O4 submitted 2025-05-20 cs.CV

Rui Tian , Mingfei Gao , Mingze Xu , Jiaming Hu , Jiasen Lu , Zuxuan Wu , Yinfei Yang , Afshin Dehghan This is my paper
classification cs.CV
keywords unifiedmultimodallargelanguagemodeltext-to-imagegenerationimageunderstandingchain-of-thoughtverificationtest-timescalingBest-of-Nselectiondirectpreferenceoptimizationmaskedtokenprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniGen is a 1.5B unified multimodal language model that performs both image understanding and text-to-image generation, trained entirely on open-source data across two pre-training stages, supervised fine-tuning, and direct preference optimization. The paper claims this fully open-data recipe reaches state-of-the-art generation scores of 0.78 on GenEval and 85.19 on DPG-Bench while staying competitive on understanding benchmarks. The central new mechanism is Chain-of-Thought Verification (CoT-V): at test time the model generates 20 candidate images per prompt, then acts as its own verifier by decomposing the prompt into atomic visual questions, answering each yes/no in a chain-of-thought format, and keeping the highest-scoring candidates. Ablations show each training stage adds a measurable gain and that CoT-V adds a further margin without sacrificing understanding capability. If correct, the work demonstrates that generation quality can be improved by test-time self-verification in a single model, using compute rather than parameter count or proprietary data.

What carries the argument

The central object is CoT-V (Chain-of-Thought Verification), a test-time scaling strategy in which UniGen acts simultaneously as image generator and image verifier. Given a text prompt, the model first produces N candidate images (N=20 in the main experiments), then for each candidate it is prompted to break the prompt into atomic visual questions, answer each question with yes or no inside a chain-of-thought block, and the final quality score is the fraction of 'yes' answers; the top K=4 candidates by score are selected. The machinery also includes the training recipe that makes this possible: masked token prediction with a cosine masking schedule for generation, decoupled SigLIP and MAGVIT-v2 encoders, a DPO stage for preference alignment, and the 500-step CoT-V post-training that teaches the model to emit the verification format using pseudo-labeled question-answer triplets.

What would settle it

Run GenEval and DPG-Bench under a matched protocol: give Janus-Pro, Show-o, and Janus the same 20-sample Best-of-N budget (using their own confidence scores or a shared verifier) and also report UniGen with a single sample per prompt; if UniGen's margin over baselines disappears, the state-of-the-art claim rests on extra inference compute rather than on CoT-V. A second direct check is to compare CoT-V's top-4 selection with random selection among the same 20 samples on DPG-Bench; if random selection scores equally well, the verifier itself is not the cause of the gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single autoregressive LLM with decoupled visual encoders—a continuous encoder for understanding and a discrete tokenizer for generation—can, with the right open-data training recipe, surpass larger and more specialized text-to-image models on GenEval and DPG-Bench while matching strong understanding-focused MLLMs. The key discovery is that letting the same model verify its own outputs, in a step-by-step chain-of-thought manner, produces reliable enough quality scores to make Best-of-N selection work: UniGen goes from 0.74 to 0.78 on GenEval and 85.02 to 85.19 on DPG-Bench when CoT-V is applied, whereas a single yes/no outcome question gives no improvement. The paper also establishes that this verification ability can be instilled with only 500 steps of lightweight post-training and that the resulting CoT-V post-trained model preserves the model's general understanding performance almost unchanged.

Load-bearing premise

The reported state-of-the-art comparison assumes that a test-time-scaled UniGen, which sees 20 generated candidates per prompt and keeps the top 4 by its own verifier's score, can be directly compared with published baseline scores obtained without that same selection budget.

Editorial extensions

If this is right

  • A model can improve its own generation quality by spending more test-time compute in a verifier loop, so unified understanding capability becomes a direct asset for generation rather than a separate head.
  • Open-source training data plus careful stage-wise recipes can bring a 1.5B unified model to benchmark levels previously reached by much larger or proprietary-data models.
  • CoT-V transfers: fine-tuning Show-o with the same DPO and CoT-V data raises its GenEval from 0.56 to 0.64 and DPG-Bench from 71.70 to 76.32.
  • The per-stage gains (pre-training, SFT, DPO, CoT-V) are additive on both GenEval and DPG-Bench, so each pipeline component earns its keep.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because CoT-V's score is an average over atomic yes/no answers, selection quality is bounded by the question decomposition; using a stronger decomposer or human-filtered CoT labels should raise the ceiling, and the paper's own failure cases on long prompts are consistent with this.
  • The headline margins over Janus-Pro and Show-o are reported with UniGen using 20 samples per prompt and top-4 selection, while baselines are quoted without that budget; until matched-budget comparisons exist, the margin should be read as including extra test-time compute.
  • The same CoT-V scorer could be recycled as a reward model for another round of preference optimization or reinforcement learning, since it scores arbitrary image-prompt pairs; the paper does not explore this closed loop.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents UniGen, a 1.5B unified multimodal LLM that handles both image understanding and text-to-image generation. The authors describe a three-stage training pipeline (pre-training, SFT, DPO) built from open-source datasets, and propose Chain-of-Thought Verification (CoT-V), a test-time scaling method in which the model generates N candidate images, verifies each in a step-by-step manner, and selects top-K. On GenEval and DPG-Bench they report scores of 0.78 and 85.19, claiming state-of-the-art among unified models, and they provide extensive ablations of each training stage and design choice.

Significance. If the results withstand scrutiny, the paper makes a useful contribution: it demonstrates a clean open-data recipe for unified MLLMs, provides careful ablations of each training stage, and shows that a lightweight CoT-V post-training (500 steps) can turn the model into an effective self-verifier, with transfer to Show-o. The internal ablations in Tables 3 through 6 are consistent and well-controlled. However, the headline SOTA claim on generation is currently not supported because the evaluation protocol is asymmetric and partially circular. The CoT-V mechanism itself remains promising but needs re-evaluation under a fair protocol.

major comments (2)
  1. [Sec. 4.1-4.2, Tables 2-3] The reported GenEval score of 0.78 and DPG-Bench score of 85.19 are obtained with N=20 candidate images per prompt and top-K=4 selection by the CoT-V verifier (Sec. 4.1), while all baseline numbers in Table 2 (Janus-Pro, Show-o, etc.) are single-sample published results. This asymmetry is load-bearing: Table 3 shows UniGen-DPO without CoT-V already reaches 0.73 on GenEval, identical to Janus-Pro's 0.73 in Table 2, so the headline +0.05 margin on GenEval is entirely produced by the test-time selection budget, not by model quality. The paper neither reports UniGen's single-sample GenEval/DPG-Bench scores in Table 2 nor runs baselines under the same Best-of-20 protocol. Moreover, it is not specified how the K selected images are aggregated into the official metric (best-of-K vs. averaging); if each selected image is submitted and the maximum is taken, this further inflates the score. The abstract's 'state-of-the-art' claim therefore conflates inference budget with model quality. Please report single-sample numbers, specify the aggregation rule, and, ideally, evaluate baselines under the same selection budget.
  2. [Sec. 3.4 and Appendix E.0.5] The GenEval evaluation is partially circular with the training data. For short prompts, DPO preference pairs are selected using GenEval metrics (Sec. 3.4), and the CoT-V post-training labels for PARM prompts are derived from GenEval's templated sub-questions and rules (Appendix E.0.5). The same GenEval benchmark is then used for the main evaluation in Table 2. This means the model is in part trained to reproduce GenEval's scoring criteria, so the GenEval improvements (0.73 to 0.78) cannot be taken as evidence of general text-to-image alignment. The authors should demonstrate that the CoT-V gain persists on a benchmark not used in any training or data-labeling stage, or at least analyze per-category results after excluding categories whose rules were used for labeling.
minor comments (5)
  1. [Sec. 3.1] The symbol X_U is used both for the input image and its encoded feature sequence; please use distinct notation (e.g., X_U and Z_U) to avoid confusion.
  2. [Fig. 4] The scores in panels (b) and (c) are not clearly mapped to the formula in Eq. (2); please add a caption explaining how 0.66 is computed.
  3. [Sec. 4.1] The phrase 'select top-K (K=4) out of it, sending for evaluation' is ambiguous; state explicitly whether the reported GenEval and DPG-Bench numbers are the average over the K selected images or the best of them.
  4. [Sec. 5 (Limitations)] The acknowledged failure of CoT-V on free-form complex prompts (also shown in Fig. 9) is in tension with the DPG-Bench claim; consider tempering the SOTA wording or reporting a DPG-Bench breakdown for prompts of varying complexity.
  5. [Table 2] The 'Color Attri.' column for UniGen (0.54) is lower than Janus-Pro's 0.56, yet the text claims an 'overwhelming advantage' on Counting; please make the per-category discussion balanced.

Circularity Check

2 steps flagged · score 6.0 of 10

GenEval headline is partially trained into the pipeline: DPO preferences are scored using GenEval metrics and CoT-V post-training labels are generated by GenEval rules, and the same benchmark is then used for evaluation.

  1. fitted input called prediction [Section 3.4 (Direct Preference Optimization), 'Preference Dataset', short-prompts paragraph]
    "For short prompts, we use the prompts from PARM [18], which generally depict the objects, and their attributes and relationship in the scene. Following [18], we use the GENEVAL metrics to evaluate the generation quality of the images. We select the highest-scored example as preferred and the lowest as rejected."

    The DPO stage constructs its preference labels using the GenEval metric itself, and Section 4.2/Table 2 then reports GenEval as headline evidence of generation quality. The reported GenEval gains from DPO (0.63 to 0.73 in Table 3, and the final 0.78 with CoT-V) are thus not independent measurements of the generation policy: the reward signal used to fit the model is the same benchmark used for evaluation. The generator is not forced to satisfy GenEval by construction, but a high GenEval score is statistically expected after optimizing toward GenEval-rule-based preferences.

  2. fitted input called prediction [Appendix E.0.5 (CoT-V Post-Training Data)]
    "For PARM, we separate each prompt into fine-grained sub-questions according to the templates originally used for generating the prompt. Rules of GENEVAL are used to label each sub-question corresponding to the image with yes or no."

    The CoT-V verifier is trained on labels produced by GenEval's own rules, and then during test-time scaling it is used to score and select the top-K candidates that are sent to the GenEval benchmark (Sections 3.5 and 4.1). Table 4's 0.74-to-0.78 GenEval improvement from CoT-V is therefore the verifier applying a learned approximation of GenEval's scoring rule to choose among candidates; the selection mechanism and the evaluation metric share the same ground-truth labels. This makes the GenEval component of the CoT-V claim partially self-fulfilling, although the image generator itself still produces unconstrained candidates.

full rationale

The clearest circularity-like issue is that GenEval is used both as a training signal and as the headline evaluation benchmark. In Sec. 3.4, DPO preference pairs for short prompts are selected by scoring images with GenEval metrics; in Appendix E.0.5, the CoT-V verifier is trained on sub-question labels produced by GenEval rules. The final GenEval score of 0.78 is then reported as evidence that CoT-V boosts generation quality. This is not a fully forced equivalence, because image candidates are still sampled from a generator whose training objective is masked token prediction, and the verifier can make mistakes. However, the CoT-V gain on GenEval is substantially a fit to the evaluation metric: the verifier is literally trained to reproduce GenEval's rule-based judgments and then used to select which samples are scored by that same benchmark. The DPG-Bench result is less affected, because the CoT-V labels for T2I-Comp prompts come from Qwen2.5-VL pseudo-labels rather than DPG-Bench rules. The comparison of Best-of-20 top-4 UniGen scores against single-sample published baselines is a real evaluation-protocol concern, but it is not a circularity issue under the requested criteria. No load-bearing self-citation chain was found: the cited PARM, Show-o, and Janus-Pro works are independent prior work, and the understanding-benchmark results are not entangled with the benchmark-labeled training loop. Overall, the central method has independent content, but the headline GenEval number is partially trained into the pipeline, giving a score of 6.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The load-bearing choices are hyperparameters, pseudo-labeled data, and evaluation-protocol decisions. Because no artifacts are released, several training choices such as the DPO beta value remain unreported.

free parameters (6)
  • DPO regularization beta = not reported
    Eq. 1 defines the DPO loss with beta controlling deviation from the SFT policy, but the value is not given in the main text or in Table 13.
  • Classifier-free guidance scale = 5.0
    Inference hyperparameter set in Sec 4.1; it affects all generated images and therefore all benchmark scores.
  • Best-of-N sample count N = 20
    Number of image candidates generated per text prompt at test time (Sec 4.1); larger N tends to inflate best-case scores.
  • Top-K selected images K = 4
    Number of candidates selected per prompt for final evaluation (Sec 4.1); this choice directly shapes the reported GenEval and DPG-Bench results.
  • Masked generation inference steps T = 50
    Number of cosine-masking iterations used at inference (Sec 3.1 and Sec 4.1).
  • CoT-V post-training steps = 500
    Lightweight fine-tuning budget chosen in Sec 3.5.1; it controls how strongly the verifier adapts and is an empirical choice.
assumptions (4)
  • domain assumption Pseudo-labels from Qwen2.5-7B and Qwen2.5-VL-7B are reliable enough to serve as preference and CoT-V supervision.
    DPO pairs in Sec 3.4 and CoT-V labels in Sec E.0.5 are generated by these models. The Limitations section admits that the pseudo-labeler produces noisy CoT data for complex prompts.
  • domain assumption GenEval and DPG-Bench scores are valid proxies for text-to-image alignment, and optimizing toward them improves real-world generation.
    Both benchmarks are the paper's headline metrics, and DPO and CoT-V are partly optimized toward GenEval-derived rules.
  • domain assumption Frozen encoders (SigLIP and MAGVITv2) and the pretrained Qwen2.5-1.5B provide sufficient base capabilities.
    The paper builds on these components without retraining them, so their quality bounds what UniGen can express.
  • domain assumption Test-time Best-of-N selection with a learned verifier is comparable to the published baseline evaluation protocols.
    Tables 1 and 2 compare UniGen with test-time scaling against baselines that, per their original papers, are not given the same 20-sample selection budget. If the protocols differ, the comparison is invalid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation." pith.science (2026). https://pith.science/paper/ULW6Q5O4

@misc{pith2026250514682,
  author       = {Pith},
  title        = {Pith review of: UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULW6Q5O4}},
  note         = {Machine review of arXiv:2505.14682}
}
read the original abstract

We introduce UniGen, a unified multimodal large language model (MLLM) capable of image understanding and generation. We study the full training pipeline of UniGen from a data-centric perspective, including multi-stage pre-training, supervised fine-tuning, and direct preference optimization. More importantly, we propose a new Chain-of-Thought Verification (CoT-V) strategy for test-time scaling, which significantly boosts UniGen's image generation quality using a simple Best-of-N test-time strategy. Specifically, CoT-V enables UniGen to act as both image generator and verifier at test time, assessing the semantic alignment between a text prompt and its generated image in a step-by-step CoT manner. Trained entirely on open-source datasets across all stages, UniGen achieves state-of-the-art performance on a range of image understanding and generation benchmarks, with a final score of 0.78 on GenEval and 85.19 on DPG-Bench. Through extensive ablation studies, our work provides actionable insights and addresses key challenges in the full life cycle of building unified MLLMs, contributing meaningful directions to the future research.

Figures

Figures reproduced from arXiv: 2505.14682 by the authors.

Figure 1
Figure 1. Comparison against state-of-the-art unified MLLMs. UniGen-1.5B outperforms Show￾o-1.3B, Janus-1.3B and Janus-Pro-1.5B across understanding and generation benchmarks. 1 Introduction Unifying understanding and generation within a single framework represents a key step toward general-purpose artificial intelligence models [48]. Pioneering work [59, 14, 85, 76, 74, 8] has made Preprint. arXiv:2505.14682v1 [cs.CV] 20 May… view at source ↗
Figure 2
Figure 2. The architecture of UniGen, which is based on an autoregressive LLM and decoupled vision en￾coders for image understanding and generation tasks. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The workflow of UniGen using test-time scaling and CoT-V. Left: Illustration of Best-of￾N selection with CoT-V. UniGen first generates 6 image candidates and then selects the two images with the highest score after self-verification using CoT-V. Right: Visualization of the step-by-step reasoning process in CoT-V for computing the final quality score. dataset, while using the same image-to-text and text-only ones. We… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: An example of using different image verification methods: (a) Outcome Verification, (b) Rule-based Verification and (c) Chain-of-Thought Verification. • For long prompts, we leverage 6K randomly sampled prompts from our re-annotated SA1B that contains high-quality imag…
Figure 5
Figure 5. Figure 5: Visual examples of UniGen’s results using CoT-V. The first three rows show examples for counting, position, and color attribute, respectively, and the last row shows images generated by free-form prompts. The first column contains images selected by UniGen as the test-…
Figure 6
Figure 6. Figure 6: Visual examples of our generated preference data for DPO training. atomic question-answer pairs corresponding to visual facts presented in the image. For PARM, we separate each prompt into fine-grained sub-questions according to the templates originally used for genera…
Figure 7
Figure 7. Figure 7: Successful examples and CoT-V verification on GENEVAL. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Successful examples and CoT-V verification on DPG-BENCH [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Failure cases on GENEVAL and DPG-BENCH. The top half of the image shows failed examples of CoT-V on short prompts. The bottom half of the image shows additional cases with bad or missing questions when CoT-V parses the complicated and long prompts. We present qualitati…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reconstruction Alignment Improves Unified Multimodal Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    RECA, a self-supervised post-training objective that conditions unified multimodal models on their own visual understanding embeddings to reconstruct input images, improves text-to-image and editing benchmarks across ...

Reference graph

Works this paper leans on

95 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv:2404.14219, 2024

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv:2404.14219, 2024. 16

    arXiv 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning.NeurIPS, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.NeurIPS, 2022

    2022

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science., 2023

    2023

  5. [5]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. InCVPR, 2022

    2022

  6. [6]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InCVPR, 2021

    2021

  7. [7]

    Sets: Leveraging self-verification and self-correction for improved test-time scaling

    Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, and Sercan Ö Arık. Sets: Leveraging self-verification and self-correction for improved test-time scaling. arXiv:2501.19306, 2025

    arXiv 2025

  8. [8]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv:2501.17811, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv:2501.17811, 2025

    arXiv 2025

Show all 95 references
  1. [9]

    Deepseek-v3 technical report.arXiv:2412.19437, 2024

    DeepSeek-AI. Deepseek-v3 technical report.arXiv:2412.19437, 2024

    2024 arXiv

  2. [10]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv:2409.17146, 2024

    2024 arXiv

  3. [11]

    Insight-v: Exploring long-chain visual reasoning with multimodal large language models.arXiv:2411.14432, 2024

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models.arXiv:2411.14432, 2024

    2024 arXiv

  4. [12]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InCVPR, 2021

    2021

  5. [13]

    Got: Unleashing reasoning capability of multimodal large language model for visual generation and editing.arXiv:2503.10639, 2025

    Rongyao Fang, Chengqi Duan, Kun Wang, Linjiang Huang, Hao Li, Shilin Yan, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, et al. Got: Unleashing reasoning capability of multimodal large language model for visual generation and editing.arXiv:2503.10639, 2025

    2025 arXiv

  6. [14]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv:2404.14396, 2024

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv:2404.14396, 2024

    2024 arXiv

  7. [15]

    Geneval: An object-focused frame- work for evaluating text-to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused frame- work for evaluating text-to-image alignment. InAdvances in Neural Information Processing Systems, 2023

    2023

  8. [16]

    https://x.ai/news/grok-1.5v, 2024

    Grok-1.5. https://x.ai/news/grok-1.5v, 2024

    2024

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv:2501.12948, 2025

    2025 arXiv

  10. [18]

    Can we generate images with cot? let’s verify and reinforce image generation step by step.arXiv:2501.13926, 2025

    Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. Can we generate images with cot? let’s verify and reinforce image generation step by step.arXiv:2501.13926, 2025

    2025 arXiv

  11. [19]

    Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis

    Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv:2412.04431, 2024

    2024 arXiv

  12. [20]

    Classifier-free diffusion guidance.arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv:2207.12598, 2022. 17

    2022 arXiv

  13. [21]

    Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv:2403.05135, 2024

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv:2403.05135, 2024

    2024 arXiv

  14. [22]

    Efficient test-time scaling via self-calibration.arXiv:2503.00031, 2025

    Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. Efficient test-time scaling via self-calibration.arXiv:2503.00031, 2025

    2025 arXiv

  15. [23]

    T2i-compbench: A com- prehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A com- prehensive benchmark for open-world compositional text-to-image generation. InNeurIPS, 2023

    2023

  16. [24]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InCVPR, 2019

    2019

  17. [25]

    Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback.NeurIPS, 2024

    Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hanna Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback.NeurIPS, 2024

    2024

  18. [26]

    text-to-image-2m: A high-quality, diverse text-to-image training dataset

    jackyhate. text-to-image-2m: A high-quality, diverse text-to-image training dataset. https: //huggingface.co/datasets/jackyhate/text-to-image-2M, 2024

    2024

  19. [27]

    Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding.arXiv:2504.04423, 2025

    Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding.arXiv:2504.04423, 2025

    2025 arXiv

  20. [28]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InECCV, 2016

    2016

  21. [29]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023

    2023

  22. [30]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv:2408.03326, 2024

    2024 arXiv

  23. [31]

    Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv:2307.16125, 2023

    2023 arXiv

  24. [32]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InICML, 2023

    2023

  25. [33]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InEMNLP, 2023

    2023

  26. [34]

    Hunyuan-dit: A powerful multi- resolution diffusion transformer with fine-grained chinese understanding.arXiv:2405.08748, 2024

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi- resolution diffusion transformer with fine-grained chinese understanding.arXiv:2405.08748, 2024

    2024 arXiv

  27. [35]

    Imagegen-cot: Enhancing text-to-image in-context learning with chain-of-thought reasoning

    Jiaqi Liao, Zhengyuan Yang, Linjie Li, Dianqi Li, Kevin Lin, Yu Cheng, and Lijuan Wang. Imagegen-cot: Enhancing text-to-image in-context learning with chain-of-thought reasoning. arXiv:2503.19312, 2025

    2025 arXiv

  28. [36]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR, 2024

    2024

  29. [37]

    Visual instruction tuning.NeurIPS, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.NeurIPS, 2023

    2023

  30. [38]

    Nvila: Efficient frontier visual language models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv:2412.04468, 2024

    2024 arXiv

  31. [39]

    Oryx MLLM: On-demand spatial-temporal understanding at arbitrary resolution.ICLR, 2025

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx MLLM: On-demand spatial-temporal understanding at arbitrary resolution.ICLR, 2025. 18

    2025

  32. [40]

    Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action.arXiv:2312.17172, 2023

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action.arXiv:2312.17172, 2023

    2023 arXiv

  33. [41]

    Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv:2206.08916, 2022

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv:2206.08916, 2022

    2022 arXiv

  34. [42]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InICLR, 2024

    2024

  35. [43]

    Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation.arXiv:2411.07975, 2024

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation.arXiv:2411.07975, 2024

    2024 arXiv

  36. [44]

    Mm1: methods, analysis and insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al. Mm1: methods, analysis and insights from multimodal llm pre-training. InECCV, 2024

    2024

  37. [45]

    Finite scalar quantization: Vq-vae made simple.arXiv:2309.15505, 2023

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple.arXiv:2309.15505, 2023

    2023 arXiv

  38. [46]

    4M: Massively multimodal masked modeling

    David Mizrahi, Roman Bachmann, O ˘guzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4M: Massively multimodal masked modeling. InNeurIPS, 2023

    2023

  39. [48]

    Gpt-4o, 2024

    OpenAI. Gpt-4o, 2024

    2024

  40. [49]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv:23...

    2023 arXiv

  41. [50]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. InICLR, 2024

    2024

  42. [51]

    Tokenflow: Unified image tokenizer for multimodal understanding and generation.arXiv:2412.03069, 2024

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation.arXiv:2412.03069, 2024

    2024 arXiv

  43. [52]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, 2021

    2021

  44. [53]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InNeurIPS, 2023

    2023

  45. [54]

    Hierarchical text-conditional image generation with clip latents.arXiv:2204.06125, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv:2204.06125, 2022

    2022 arXiv

  46. [55]

    Imagenet-21k pretraining for the masses.arXiv:2104.10972, 2021

    Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses.arXiv:2104.10972, 2021

    2021 arXiv

  47. [56]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. InACL, 2018

    2018

  48. [57]

    Journeydb: A benchmark for generative image understanding

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. InNeurIPS, 2023. 19

    2023

  49. [58]

    Aligning large multimodal models with factually augmented rlhf.arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf.arXiv:2309.14525, 2023

    2023 arXiv

  50. [59]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv:2405.09818, 2024

    2024 arXiv

  51. [60]

    Kimi k1.5: Scaling reinforcement learning with llms.arXiv:2501.12599, 2025

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with llms.arXiv:2501.12599, 2025

    2025 arXiv

  52. [61]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. InNeurIPS, 2024

    2024

  53. [62]

    Metamorph: Multimodal under- standing and generation via instruction tuning.arXiv:2412.14164, 2024

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal under- standing and generation via instruction tuning.arXiv:2412.14164, 2024

    2024 arXiv

  54. [63]

    Llama: Open and efficient foundation language models.arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv:2302.13971, 2023

    2023 arXiv

  55. [64]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, an...

    2025 arXiv

  56. [65]

    Neural discrete representation learning.NeurIPS, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.NeurIPS, 2017

    2017

  57. [66]

    Illume: Illuminating your llms to see, draw, and self-enhance.arXiv:2412.06673, 2024

    Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illuminating your llms to see, draw, and self-enhance.arXiv:2412.06673, 2024

    2024 arXiv

  58. [67]

    Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl.arXiv:2504.11455, 2025

    Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl.arXiv:2504.11455, 2025

    2025 arXiv

  59. [68]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv:2409.12191, 2024

    2024 arXiv

  60. [69]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization.arXiv:2411.10442, 2024

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization.arXiv:2411.10442, 2024

    2024 arXiv

  61. [70]

    Visualprm: An effective process reward model for multimodal reasoning.arXiv:2503.10291, 2025

    Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning.arXiv:2503.10291, 2025

    2025 arXiv

  62. [71]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv:2409.18869, 2024

    2024 arXiv

  63. [72]

    Mint: Multi-modal chain of thought in unified generative models for enhanced image generation.arXiv:2503.01298, 2025

    Yi Wang, Mushui Liu, Wanggui He, Longxiang Zhang, Ziwei Huang, Guanghao Zhang, Fangxun Shu, Zhong Tao, Dong She, Zhelun Yu, et al. Mint: Multi-modal chain of thought in unified generative models for enhanced image generation.arXiv:2503.01298, 2025

    2025

  64. [73]

    Large language models are better reasoners with self-verification

    Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. InEMNLP, 2023

    2023

  65. [74]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv:2410.13848, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv:2410.13848, 2024. 20

    arXiv

  66. [75]

    Vila-u: a unified foundation model integrating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. InICLR, 2025

    2025

  67. [76]

    Show-o: One single transformer to unify multimodal understanding and generation.arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv:2408.12528, 2024

    2024 arXiv

  68. [77]

    Llava-critic: Learning to evaluate multimodal models.arXiv:2410.02712, 2024

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models.arXiv:2410.02712, 2024

    2024 arXiv

  69. [78]

    Slowfast-llava: A strong training-free baseline for video large language models.arXiv:2407.15841, 2024

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free baseline for video large language models.arXiv:2407.15841, 2024

    2024 arXiv

  70. [79]

    Slowfast-llava-1.5: A family of token-efficient video large language models for long-form video understanding.arXiv:2503.18943, 2025

    Mingze Xu, Mingfei Gao, Shiyu Li, Jiasen Lu, Zhe Gan, Zhengfeng Lai, Meng Cao, Kai Kang, Yinfei Yang, and Afshin Dehghan. Slowfast-llava-1.5: A family of token-efficient video large language models for long-form video understanding.arXiv:2503.18943, 2025

    2025 arXiv

  71. [81]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  72. [82]

    Mmar: Towards lossless multi-modal auto-regressive probabilistic modeling.arXiv:2410.10798, 2024

    Jian Yang, Dacheng Yin, Yizhou Zhou, Fengyun Rao, Wei Zhai, Yang Cao, and Zheng-Jun Zha. Mmar: Towards lossless multi-modal auto-regressive probabilistic modeling.arXiv:2410.10798, 2024

    2024 arXiv

  73. [83]

    Hermesflow: Seamlessly closing the gap in multimodal understanding and generation

    Ling Yang, Xinchen Zhang, Ye Tian, Chenming Shang, Minghao Xu, Wentao Zhang, and Bin Cui. Hermesflow: Seamlessly closing the gap in multimodal understanding and generation. arXiv:2502.12148, 2025

    2025

  74. [84]

    Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search.arXiv:2412.18319, 2024

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search.arXiv:2412.18319, 2024

    2024 arXiv

  75. [85]

    X-vila: Cross-modality alignment for large language model

    Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al. X-vila: Cross-modality alignment for large language model. arXiv:2405.19335, 2024

    2024 arXiv

  76. [86]

    Language model beats diffusion-tokenizer is key to visual generation

    Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokenizer is key to visual generation. InICLR, 2024

    2024

  77. [87]

    Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G

    Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion – tokenizer is key to visual ...

    arXiv

  78. [88]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  79. [89]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InICCV, 2023

    2023

  80. [90]

    VideoLLaMA 3: Frontier multimodal foundation models for image and video understanding.arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. VideoLLaMA 3: Frontier multimodal foundation models for image and video understanding.arXiv:2501.13106, 2025

    2025 arXiv

  81. [91]

    Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning.arXiv:2409.20566, 2024

    2024 arXiv

  82. [92]

    Video instruction tuning with synthetic data.arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv:2410.02713, 2024

    2024 arXiv

  83. [93]

    Image and video tokenization with binary spherical quantization.arXiv:2406.07548, 2024

    Yue Zhao, Yuanjun Xiong, and Philipp Krähenbühl. Image and video tokenization with binary spherical quantization.arXiv:2406.07548, 2024

    2024 arXiv

  84. [94]

    Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv:2408.11039, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv:2408.11039, 2024

    2024 arXiv

  85. [95]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv:2504.10479, 2025

    2025 arXiv

  86. [96]

    Vargpt-v1

    Xianwei Zhuang, Yuxin Xie, Yufan Deng, Dongchao Yang, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt-v1. 1: Improve visual autoregressive large unified model via iterative instruction tuning and reinforcement learning.arXiv:2504.02949, 2025

    2025 arXiv

  87. [97]

    Apollo: An exploration of video understanding in large multimodal models.arXiv:2412.10360, 2024

    Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models.arXiv:2412.10360, 2024. 22

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.