Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FUDOKI demonstrates that a purely discrete flow matching model can match autoregressive MLLMs on both visual understanding and image generation.

desk verdict FUDOKI credibly demonstrates a non-autoregressive unified MLLM via discrete flow matching, but the parity claim rests on undisclosed in-house training data; the decisive fixes are a contamination audit and a matched mask-path ablation. read the letter →

arxiv 2505.20147 v3 pith:G4P7ADHR submitted 2025-05-26 cs.CV

Jin Wang , Yao Lai , Aoxue Li , Shifeng Zhang , Jiacheng Sun , Ning Kang , Chengyue Wu , Zhenguo Li
show 1 more author
Ping Luo
This is my paper
classification cs.CV
keywords discreteflowmatchingunifiedmultimodalmodelvisualunderstandingtext-to-imagegenerationkinetic-optimalvelocitynon-autoregressiveself-correctingreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that discrete flow matching can replace autoregressive next-token prediction as the backbone of a unified multimodal model. It introduces FUDOKI, a 1.5B model that handles both visual understanding and text-to-image generation in a single discrete flow framework, initialized from a pretrained autoregressive MLLM and converted to bidirectional flow sampling. The authors report performance comparable to state-of-the-art autoregressive MLLMs: GenEval 0.77 overall, MM-Vet 38.0, MMB 73.9, close to Janus-Pro-1B's 0.73, 39.8, and 75.5. The significance, if the results hold, is that non-autoregressive discrete flow models are a viable foundation for next-generation unified multimodal systems, with built-in self-correction and bidirectional context rather than sequential left-to-right generation.

What carries the argument

The load-bearing mechanism is the kinetic-optimal velocity of discrete flow matching on metric-induced paths. Given a distance $d(\cdot,\cdot)$ on token embeddings and a monotone schedule $\beta_t$, the conditional path is $p_t(x^i | x^1) = \mathrm{softmax}(-\beta_t d(x^i, x^1))$ and the optimal velocity is $u_t^i(x^i, z | x^1) = p_t(x^i | x^1)\dot{\beta}_t [d(z^i, x^1) - d(x^i, x^1)]_+$. This velocity increases the probability of moving a token to a closer embedding while forbidding jumps away from the target, which is what gives the sampler its monotone, self-correcting character. Because the path is defined on the full sequence rather than on masked positions, every token stays updatable throughout generation. The paper also uses AR initialization and omits time embeddings so that the model infers noise level from the corrupted input itself, which the ablation shows speeds convergence.

What would settle it

Compare the 8.76 million in-house training prompts and images against GenEval, DPG-Bench, and the understanding benchmark sets; finding substantial exact or near-duplicate overlap would falsify the claim that discrete flow matching alone explains the reported parity.

Watch

Extended reading notes

Core claim

FUDOKI's central claim is that a unified multimodal model can be built purely on discrete flow matching, without any autoregressive component, and still match autoregressive MLLMs on both halves of the task. The model treats image tokens and text tokens alike as discrete sequences, corrupts them along metric-induced probability paths $p_t(x^i | x^1) = \mathrm{softmax}(-\beta_t \, d(x^i, x^1))$, and learns to predict the clean target from any corrupted state. At inference, an Euler solver on the continuous-time Markov chain uses the kinetic-optimal velocity $u_t^i(x^i, z | x^1) = p_t(x^i | x^1) \dot{\beta}_t [d(z^i, x^1) - d(x^i, x^1)]_+$, so each step moves tokens closer to the model's prediction and, critically, can rewrite tokens that were already generated. The paper reports GenEval 0.77 (Janus-Pro-1B: 0.73), MM-Vet 38.0 (Janus-Pro-1B: 39.8), and DPG-Bench 83.63, and shows that judge-based test-time scaling raises GenEval to 0.88 and MM-Vet to 55.5. In the paper's telling, this is direct evidence that discrete flow matching, not next-token prediction, can serve as the foundation of a unified multimodal model.

Load-bearing premise

The load-bearing premise is that the benchmark scores faithfully measure capability, because 8.76 million in-house generation samples used in training are not disclosed and could overlap with the evaluation benchmarks, which would make the claimed parity with autoregressive models an artifact.

Editorial extensions

If this is right

  • Discrete flow matching becomes a candidate backbone for unified multimodal models, removing the causal-mask and raster-scan constraints of AR generation.
  • Errors made early in generation no longer have to be final: FUDOKI can rewrite tokens during sampling, which the paper connects to coherent reasoning and accurate attribute binding.
  • The AR-to-flow transfer recipe (initialize from pretrained weights, drop time embeddings, fine-tune on about 13M samples) offers a cheap route to non-autoregressive multimodal models.
  • Test-time scaling with a judge model improves FUDOKI's GenEval from 0.77 to 0.88 and MM-Vet from 38.0 to 55.5, pointing to reinforcement learning as a further lever on the same architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the parity survives contamination controls, a corollary the paper does not spell out is that bidirectional flow models may be better suited than AR models to tasks requiring revision of an initial interpretation, since they can revisit earlier tokens.
  • A natural extension is to generalize the fixed-length sampling to variable-length sequences by making the scheduling or distance function position-dependent; the paper's fixed sequence length is a stated limitation.
  • The maze-navigation results, though qualitative, suggest discrete flow matching could be tested quantitatively on planning and embodied benchmarks, where self-correction and bidirectional context are expected to matter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents FUDOKI, a unified multimodal model for visual understanding and text-to-image generation that replaces the autoregressive next-token objective with discrete flow matching on metric-induced probability paths, using the kinetic-optimal velocities of Shaul et al. [34]. The model is initialized from Janus-1.5B and fine-tuned with a cross-entropy denoising loss over text and image token sequences. The authors report GenEval 0.77 (0.88 with inference scaling), DPG-Bench 83.63, and understanding scores on POPE/MME/MMB/MM-Vet that are close to Janus-Pro-1B, concluding that discrete flow matching is a viable non-autoregressive foundation. The paper also includes ablations on AR initialization and time embeddings, qualitative self-correction examples, and a maze-navigation proof of concept.

Significance. If the benchmark results are taken at face value, the paper provides the first evidence at the 1.5B scale that a fully flow-based unified MLLM can match AR-based systems on both understanding and generation, with additional gains from test-time scaling. The formulation is clearly grounded in the existing discrete flow matching literature, and the paper is transparent about its training recipe and limitations (Appendix F). However, the significance is conditional on two unresolved issues: the undeclared 8.76M in-house generation prompts create an unquantified contamination risk for the central parity claim, and the AR initialization confound prevents attribution of the results to the discrete flow matching design. The paper also ships no code, data, or weights, and the qualitative figures contain garbled text that undermines the presented evidence.

major comments (3)
  1. [§4.1 and Appendix D, Tables 1 and 3] The generation subset is described only as "in-house generation data for text-to-image generation", with no prompt list, collection protocol, or filtering criteria. Since GenEval has 553 prompts and DPG-Bench has 1,065 prompts, and the training set contains 8.76M generation prompts, even a small fraction of duplicated or templated prompts could materially inflate the reported overall scores (0.77 on GenEval, 83.63 on DPG-Bench) that underlie the parity claim. The authors should either release the prompt list, report exact or approximate overlap with the evaluation benchmarks (e.g., n-gram or embedding distance), or remove possibly overlapping samples and re-evaluate. Without this audit, the central claim is not currently supported by the evidence in the manuscript.
  2. [§3.3 and Figure 5] The model is initialized from Janus-1.5B, an AR-based MLLM, and the paper's only ablation of this choice reports training loss curves (Fig. 5 left) rather than final benchmark performance. A reader therefore cannot tell whether the competitive numbers in Tables 1–2 are due to the discrete flow matching training, the kinetic-optimal path, or simply the transferred AR knowledge. The authors should provide a matched comparison of the same architecture and data under (a) AR initialization + DFM objective, (b) random initialization + DFM objective, and, ideally, (c) AR initialization + a mask-based discrete diffusion objective to isolate the effect of the metric-induced path. This is necessary to support the claim that "the discrete flow matching framework of FUDOKI" is what drives the results.
  3. [§4.2, Table 1] The "Inference Scaling" results select the top 4 images from 32 candidates per GenEval prompt via VILA-Judge, a model whose training data is not specified. If VILA-Judge was trained on GenEval or on the same public image-caption corpora from which the in-house prompts may have been drawn, the 0.88 score could reflect benchmark-specific selection. Additionally, using the test set prompts to select candidates is a form of test-time tuning. The authors should specify VILA-Judge's training data and report the performance of random selection and self-consistency baselines to quantify the selection gain.
minor comments (6)
  1. [§4.1 vs Appendix D] There is an inconsistency in the reported size of the in-house generation data: Section 4.1 states "9M in-house generation data" while Appendix D states "8.76M Generation samples"; the numbers should be reconciled.
  2. [Figures 4 and 7] The qualitative evidence for self-correction is obscured by garbled text in the figures, for example 'Inthe last equation, Pred44 + ت =39...' in the Fig. 4 caption and the corrupted intermediate outputs in Fig. 7; these should be regenerated with clean rendering.
  3. [§3.2] The "shifting operation" imported from [45] is not described in detail; please specify how it interacts with the first token and with masked positions during discrete flow matching training.
  4. [Appendix C] The project page URL is inconsistent: the header gives 'https://fudoki-hku.github.io/' while Appendix C gives 'fudoki-dfm.github.io/fudoki/'; please unify the URLs.
  5. [Abstract and conclusion] The wording "purely based on discrete flow matching" is at odds with the AR-based initialization described in §3.3; consider clarifying that the inference-time generative process is purely flow-based, since the pretrained weights come from an AR model.
  6. [Table 2] The exclusion of UniDisc is only noted in a footnote; the caption should state that UniDisc does not support visual question answering, so readers understand why it is absent from the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: equations are imported from external prior work, the loss is standard cross-entropy on a prescribed path, and the central parity claim is tested on external benchmarks.

full rationale

The derivation chain is not circular. The metric-induced probability path (Eq. 4), the kinetic-optimal velocity (Eq. 5), and the continuity equation are imported by citation from the external discrete flow matching literature [33,34]; FUDOKI applies these formulas as given, and the only training loss (Eq. 6) is standard cross-entropy between a corrupt sample x_t drawn from the prescribed path and the ground-truth x_1. There is no fitted parameter that is later renamed as a prediction: the text and image embedding distances are fixed inputs from external models (Janus-Pro-7B and LlamaGen), the beta schedule is a fixed curve, and the benchmark scores (GenEval, DPG-Bench, POPE, MM-Vet, etc.) are computed by external metrics on the model's outputs. The paper does initialize from the same-group Janus-1.5B and borrows embeddings and test-time-scaling tricks from prior work, but these are inputs to the recipe rather than consequences of the target claim. The disclosed training-data composition (8.76M in-house generation samples) raises a possible contamination question, but that is an evidence/completeness concern for the empirical parity claim, not a circularity of the derivation.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper contributes no new mathematical objects. Its central claim rests on four domain assumptions: the validity of discrete flow matching, the kinetic-optimal velocity formula from [34], the sufficiency of implicit time-step inference, and the semantic meaningfulness of external-embedding metrics. The only hand-chosen hyperparameter in the path construction is the beta schedule (c=3, a=0.9). No invented particles or entities appear.

free parameters (1)
  • beta_t schedule constants = c=3, a=0.9
    The probability-path schedule beta_t = c (t/(1-t))^a is chosen following [34] and affects the corruption dynamics; it is not fitted to FUDOKI's data but is a hand-selected constant.
assumptions (4)
  • domain assumption Discrete flow matching framework, including the continuity equation and the rate condition (Eqs 2-3, Definition 1), is valid and applicable to multimodal token sequences.
    Imported without re-derivation from [33,34]; the paper builds its training and inference on it.
  • domain assumption Kinetic-optimal velocity formula (Eq 5) is a valid generation velocity for the metric-induced path (Eq 4).
    Taken from [34]; the paper does not rederive the optimality, only states it in Appendix E.
  • domain assumption The model can infer the time step from the corrupted input, so time embeddings are unnecessary.
    Stated in Section 3.2 and supported only by a training-loss comparison in Fig 5 (middle), not by analysis.
  • domain assumption L2 distances on external embeddings (Janus-Pro-7B text, LlamaGen image codebook) define a semantically meaningful metric for the probability path.
    Section 4.1 states this choice; if the external embedding geometry does not match the model's learned space, the claimed advantage of metric paths over masks is unsupported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities." pith.science (2026). https://pith.science/paper/G4P7ADHR

@misc{pith2026250520147,
  author       = {Pith},
  title        = {Pith review of: FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G4P7ADHR}},
  note         = {Machine review of arXiv:2505.20147}
}
read the original abstract

The rapid progress of large language models (LLMs) has catalyzed the emergence of multimodal large language models (MLLMs) that unify visual understanding and image generation within a single framework. However, most existing MLLMs rely on autoregressive (AR) architectures, which impose inherent limitations on future development, such as the raster-scan order in image generation and restricted reasoning abilities in causal context modeling. In this work, we challenge the dominance of AR-based approaches by introducing FUDOKI, a unified multimodal model purely based on discrete flow matching, as an alternative to conventional AR paradigms. By leveraging metric-induced probability paths with kinetic optimal velocities, our framework goes beyond the previous masking-based corruption process, enabling iterative refinement with self-correction capability and richer bidirectional context integration during generation. To mitigate the high cost of training from scratch, we initialize FUDOKI from pre-trained AR-based MLLMs and adaptively transition to the discrete flow matching paradigm. Experimental results show that FUDOKI achieves performance comparable to state-of-the-art AR-based MLLMs across both visual understanding and image generation tasks, highlighting its potential as a foundation for next-generation unified multimodal models. Furthermore, we show that applying test-time scaling techniques to FUDOKI yields significant performance gains, further underscoring its promise for future enhancement through reinforcement learning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A compact unified model that reuses a frozen VLM encoder and hybrid continuous/discrete tokens reaches competitive image understanding and generation with 15.6M training images and about $2,000 in compute.

  2. A Survey on Diffusion Language Models

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    A comprehensive survey of diffusion language models covering taxonomy, training and inference techniques, and comparisons with autoregressive models.

Reference graph

Works this paper leans on

178 extracted references · 1 canonical work pages · cited by 2 Pith papers

  1. [34]

    Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky T. Q. Chen. Flow matching with general discrete paths: A kinetic-optimal perspective. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  2. [1]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  3. [2]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  4. [3]

    Hartshorn, Aobo Yang, et al

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, et al. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024

    arXiv 2024

  5. [4]

    Internlm2 technical report, 2024

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, et al. Internlm2 technical report, 2024

    2024

  6. [5]

    OpenAI. Chatgpt. https://chat.openai.com/, 2023

    2023

  7. [6]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  8. [7]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    arXiv 2023

Show all 178 references
  1. [8]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    2023

  2. [9]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525, 2024

    2024 arXiv

  3. [10]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    2024 arXiv

  4. [11]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021

    2021

  5. [12]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Om- mer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  6. [13]

    Pixart-𝑎𝑙𝑝ℎ𝑎 : Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-𝑎𝑙𝑝ℎ𝑎 : Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023

    arXiv

  7. [14]

    Patrick Esser, Sumith Kulal, A. Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformer...

    2024 arXiv

  8. [15]

    Black Forest Labs. Flux. https://github.com/black-forest-labs/flux, 2024

    2024

  9. [16]

    Planting a seed of vision in large language model

    Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.08041, 2023

    2023 arXiv

  10. [17]

    Making llama see and draw with seed tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218, 2023

    2023 arXiv

  11. [18]

    Emu3: Next-token prediction is all you need, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need, 2024

    2024

  12. [19]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  13. [20]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation. ArXiv, abs/2410.13848, 2024

    arXiv

  14. [21]

    Illume: Illuminating your llms to see, draw, and self-enhance, 2024

    Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illuminating your llms to see, draw, and self-enhance, 2024

    2024

  15. [22]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. ArXiv, abs/2501.17811, 2025

    2025 arXiv

  16. [23]

    Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement, 2025

    Runhui Huang, Chunwei Wang, Junwei Yang, Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, and Hang Xu. Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement, 2025

    2025

  17. [24]

    Sparks of artificial general intelligence: Early experiments with gpt-4, 2023

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with g...

    2023

  18. [25]

    Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transforme...

    2023

  19. [26]

    The pitfalls of next-token prediction

    Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. ArXiv, abs/2403.06963, 2024

    2024 arXiv

  20. [27]

    Beyond autoregression: Discrete diffusion for complex reasoning and planning

    Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. ArXiv, abs/2410.14157, 2024

    2024 arXiv

  21. [28]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. ArXiv, abs/2310.01798, 2023. 25

    2023 arXiv

  22. [29]

    Structured denoising diffusion models in discrete state-spaces

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural informa- tion processing systems, 34:17981–17993, 2021

    2021

  23. [30]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning, pages 32819–32848, 2024

    2024

  24. [31]

    Simplified and generalized masked diffusion for discrete data

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37:103131–103167, 2024

    2024

  25. [32]

    Simple and effective masked diffusion language models

    Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136–130184, 2024

    2024

  26. [33]

    Discrete flow matching

    Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37:133345–133385, 2024

    2024

  27. [35]

    Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design

    Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In International Conference on Machine Learning, pages 5453–5512. PMLR, 2024

    2024

  28. [36]

    URL https://www.inceptionlabs.ai/news

    Mercury coder, 2025. URL https://www.inceptionlabs.ai/news

    2025

  29. [37]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022

    2022 arXiv

  30. [38]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  31. [39]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023

    2023

  32. [40]

    Large language diffusion models

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025

    2025 arXiv

  33. [41]

    Dream 7b, 2025

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025. URL https://hkunlp.github.io/blog/2025/dre am

    2025

  34. [42]

    Dual diffusion for unified image generation and understanding

    Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. Dual diffusion for unified image generation and understanding. arXiv preprint arXiv:2501.00289, 2024. 26

    2024 arXiv

  35. [43]

    Unified discrete diffusion for simulta- neous vision-language generation

    Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang, Dacheng Tao, and Ponnuthurai N Suganthan. Unified discrete diffusion for simulta- neous vision-language generation. arXiv preprint arXiv:2211.14842, 2022

    2022 arXiv

  36. [44]

    Unified multimodal discrete diffusion

    Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragkiadaki. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503.20853, 2025

    2025 arXiv

  37. [45]

    Scaling diffusion language models via adaptation from autoregressive models

    Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.17891, 2024

    arXiv

  38. [46]

    Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer.arXiv preprint arXiv:2501.18427, 2025

    Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, et al. Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer.arXiv preprint arXiv:2501.18427, 2025

    2025 arXiv

  39. [47]

    Dancegrpo: Unleashing grpo on visual generation, 2025

    Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo. Dancegrpo: Unleashing grpo on visual generation, 2025

    2025

  40. [48]

    Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement

    Runhui Huang, Chunwei Wang, Junwei Yang, Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, et al. Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement. arXiv preprint arXiv:2504.01934, 2025

    2025 arXiv

  41. [49]

    VILA-u: a unified foundation model integrating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, and Yao Lu. VILA-u: a unified foundation model integrating visual understanding and generation. In The Thirteenth International Conference on Learn...

    2025

  42. [50]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  43. [51]

    Unified language-vision pretraining in LLM with dynamic discrete visual tokenization

    Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin CHEN, Chengru Song, dai meng, Di ZHANG, Wenwu Ou, Kun Gai, and Yadong MU. Unified language-vision pretraining in LLM with dynamic discrete visual tokenization. In The Twelfth International Conferen...

    2024

  44. [52]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  45. [53]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  46. [54]

    Diffusionbert: Improving generative masked language models with diffusion models

    Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuan-Jing Huang, and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

    2023

  47. [55]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 27

    2023

  48. [56]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024

    2024 arXiv

  49. [57]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023

    2023 arXiv

  50. [58]

    wendlerc/renderedtext, 2023

    Chris Wendler. wendlerc/renderedtext, 2023

    2023

  51. [59]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, 2021

    2021

  52. [60]

    Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model, 2020

    Jason Obeid and Enamul Hoque. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model, 2020

    2020

  53. [61]

    Visualmrc: Machine reading compre- hension on document images

    Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmrc: Machine reading compre- hension on document images. In AAAI, 2021

    2021

  54. [62]

    G-llava: Solving geomet- ric problem with multi-modal large language model, 2023

    Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-llava: Solving geomet- ric problem with multi-modal large language model, 2023

    2023

  55. [63]

    Xing, and Liang Lin

    Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P . Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning, 2022

    2022

  56. [64]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. ArXiv, abs/2406.08464, 2024

    2024 arXiv

  57. [65]

    World model on million-length video and language with blockwise ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  58. [66]

    Hierarchical text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  59. [67]

    Sdxl: Improving latent diffusion models for high- resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high- resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  60. [68]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    2023

  61. [69]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396, 2024

    2024 arXiv

  62. [70]

    Tokenflow: Unified image tokenizer for multimodal understanding and generation

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069, 2024. 28

    2024 arXiv

  63. [71]

    Geneval: An object-focused framework for evaluating text-to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Process- ing Systems, 36, 2024

    2024

  64. [72]

    Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023

    2023 arXiv

  65. [73]

    Mobilevlm v2: Faster and stronger baseline for vision language model

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024

    2024 arXiv

  66. [74]

    Llava-phi: Efficient multi-modal assistant with small language model.arXiv preprint arXiv:2401.02330, 2024

    Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-modal assistant with small language model.arXiv preprint arXiv:2401.02330, 2024

    2024 arXiv

  67. [75]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  68. [76]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  69. [77]

    Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023

    Hugo Laurençon, Daniel van Strien, Stas Bekman, Leo Tronchon, Lucile Saulnier, Thomas Wang, Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, and et al. Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023

    2023

  70. [78]

    Unified language-vision pretraining with dynamic discrete visual tokenization

    Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al. Unified language-vision pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.04669, 2023

    2023

  71. [79]

    Metamorph: Multimodal understanding and generation via instruction tuning

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412.14164, 2024

    2024 arXiv

  72. [80]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  73. [81]

    Vila-u: a unified foundation model integrating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024

    2024 arXiv

  74. [82]

    Evaluat- ing object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluat- ing object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  75. [83]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 29

    2023 arXiv

  76. [84]

    Seed- bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  77. [85]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  78. [86]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  79. [87]

    Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi. In Proceed- ings of the IEEE/CVF Conference o...

    2024

  80. [88]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  81. [89]

    Nvila: Efficient frontier visual language models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04468, 2024

    2024 arXiv

  82. [90]

    Flow-grpo: Training flow matching models via online rl

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05470, 2025

    2025 arXiv

  83. [91]

    Unified-io: A unified model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kem- bhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022

    2022 arXiv

  84. [92]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision an...

    2024

  85. [93]

    Anygpt: Unified multimodal llm with discrete sequence modeling

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal llm with discrete sequence modeling. arXiv preprint arXiv:2402.12226, 2024

    2024 arXiv

  86. [94]

    Generative multimodal pretraining with discrete diffusion timestep tokens

    Kaihang Pan, Wang Lin, Zhongqi Yue, Tenglong Ao, Liyu Jia, Wei Zhao, Juncheng Li, Siliang Tang, and Hanwang Zhang. Generative multimodal pretraining with discrete diffusion timestep tokens. arXiv preprint arXiv:2504.14666, 2025

    2025 arXiv

  87. [95]

    Dreamllm: Synergistic multimodal comprehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. In ICLR, 2024

    2024

  88. [96]

    Minigpt-5: Interleaved vision-and-language generation via generative vokens

    Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and-language generation via generative vokens. arXiv preprint arXiv:2310.02239, 2023. 30

    2023

  89. [97]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. In Forty-first International Conference on Machine Learning, 2024

    2024

  90. [98]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

    2025

  91. [99]

    Janusflow: Harmonizing autore- gression and rectified flow for unified multimodal understanding and generation

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autore- gression and rectified flow for unified multimodal understanding and generation. arXiv preprint arXiv:2411.07975, 2024

    2024 arXiv

  92. [100]

    Building normalizing flows with stochastic interpolants

    Michael Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In ICLR 2023 Conference, 2023

    2023

  93. [101]

    Efficient video prediction via sparsely conditioned flow matching

    Aram Davtyan, Sepehr Sameni, and Paolo Favaro. Efficient video prediction via sparsely conditioned flow matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23263–23274, 2023

    2023

  94. [102]

    Wan: Open and advanced large-scale video generative models, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pande...

    2025

  95. [103]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024

    2024

  96. [104]

    Hunyuanvideo: A systematic framework for large video generative models, 2025

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, J...

    2025

  97. [105]

    Generative pre-training for speech with flow matching

    Alexander H Liu, Matt Le, Apoorv Vyas, Bowen Shi, Andros Tjandra, and Wei-Ning Hsu. Generative pre-training for speech with flow matching. arXiv preprint arXiv:2310.16338, 2023

    2023 arXiv

  98. [106]

    Voicebox: Text-guided multilingual universal speech generation at scale

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36:14005–14...

    2023

  99. [107]

    Audiobox: Unified audio generation with natural language prompts

    Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821, 2023

    2023 arXiv

  100. [108]

    Fast protein backbone generation with se (3) flow matching

    Jason Yim, Andrew Campbell, Andrew YK Foong, Michael Gastegger, José Jiménez- Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan S Veeling, Regina Barzilay, Tommi Jaakkola, et al. Fast protein backbone generation with se (3) flow matching. arXiv preprint arXiv:2310.05297, 2023

    arXiv

  101. [109]

    Alphafold meets flow matching for generating protein ensembles

    Bowen Jing, Bonnie Berger, and Tommi Jaakkola. Alphafold meets flow matching for generating protein ensembles. In International Conference on Machine Learning , pages 22277–22303. PMLR, 2024

    2024

  102. [110]

    Se (3)-stochastic flow matching for protein backbone generation

    Avishek Joey Bose, Tara Akhound-Sadegh, Guillaume Huguet, Kilian Fatras, Jarrid Rector- Brooks, Cheng-Hao Liu, Andrei Cristian Nica, Maksym Korablyov, Michael Bronstein, and Alexander Tong. Se (3)-stochastic flow matching for protein backbone generation. arXiv preprint arXiv:2...

    arXiv

  103. [111]

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Jam...

    2024

  104. [112]

    Improved denoising diffusion proba- bilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion proba- bilistic models. In International conference on machine learning, pages 8162–8171. PMLR, 2021

    2021

  105. [113]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021

    2021

  106. [114]

    Diffusion-lm improves controllable text generation

    Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. Advances in neural information process- ing systems, 35:4328–4343, 2022

    2022

  107. [115]

    Diffuseq: Sequence to sequence text generation with diffusion models

    Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations, 2023

    2023

  108. [116]

    Likelihood-based diffusion language models

    Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36:16693–16715, 2023

    2023

  109. [117]

    Argmax flows and multinomial diffusion: Learning categorical distributions

    Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 34:12454–12465, 2021

    2021

  110. [118]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315–11325, 2022

    2022

  111. [119]

    Film: Fill-in language models for any-order generation

    Tianxiao Shen, Hao Peng, Ruoqi Shen, Yao Fu, Zaid Harchaoui, and Yejin Choi. Film: Fill-in language models for any-order generation. arXiv preprint arXiv:2310.09930, 2023. 32

    2023 arXiv

  112. [120]

    A reparameterized discrete diffusion model for text generation

    Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. In First Conference on Language Modeling, 2024

    2024

  113. [121]

    Score-based continuous- time discrete diffusion models

    Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous- time discrete diffusion models. In The Eleventh International Conference on Learning Repre- sentations, 2023

    2023

  114. [122]

    A continuous time framework for discrete denoising models

    Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligianni- dis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266–28279, 2022

    2022

  115. [123]

    Scaling up masked diffusion models on text

    Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024

    2024 arXiv

  116. [124]

    Diffusion of thought: Chain-of-thought reasoning in diffusion language models

    Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Xin Jiang, Zhenguo Li, Wei Bi, et al. Diffusion of thought: Chain-of-thought reasoning in diffusion language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  117. [125]

    Implicit search via discrete diffusion: A study on chess

    Jiacheng Ye, Zhenyu Wu, Jiahui Gao, Zhiyong Wu, Xin Jiang, Zhenguo Li, and Ling- peng Kong. Implicit search via discrete diffusion: A study on chess. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  118. [126]

    Beyond autoregression: Discrete diffusion for complex reasoning and planning

    Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  119. [127]

    Ella: Equip diffusion models with llm for enhanced semantic alignment

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024

    2024 arXiv

  120. [128]

    Lumina-Next: Making Lumina-T2X stronger and faster with Next-DiT

    Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Lirui Zhao, Fu-Yun Wang, Zhanyu Ma, et al. Lumina-Next: Making Lumina-T2X stronger and faster with Next-DiT. arXiv preprint arXiv:2406.18583, 2024

    2024 arXiv

  121. [129]

    Playground v2

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245, 2024

    2024 arXiv

  122. [130]

    Hunyuan-DiT: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-DiT: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405.08748, 2024

    2024 arXiv

  123. [131]

    Pixart- 𝜎: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- 𝜎: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pages 74–91. Sprin...

    2024

  124. [132]

    Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2023

    Shanghai AI Laboratory. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2023

    2023

  125. [133]

    Visual spatial reasoning

    Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023. 33

    2023

  126. [134]

    Allava: Harnessing gpt4v-synthesized data for lite vision-language models, 2024

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for lite vision-language models, 2024

    2024

  127. [135]

    Iconqa: A new benchmark for abstract diagram understand- ing and visual language reasoning

    Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understand- ing and visual language reasoning. In NeurIPS, 2021

    2021

  128. [136]

    To see is to believe: Prompting gpt-4v for better visual instruction tuning

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023

    2023 arXiv

  129. [137]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023

    2023 arXiv

  130. [138]

    ViQuAE, a dataset for knowledge-based visual question answering about named entities

    Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, Jose G Moreno, and Jesús Lovón Melgarejo. ViQuAE, a dataset for knowledge-based visual question answering about named entities. In Proceedings of The 45th International ACM SIGIR Conference on ...

    2022

  131. [139]

    Raven: A dataset for relational and analogical visual reasoning

    Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In CVPR, 2019

    2019

  132. [140]

    Visual7W: Grounded Question Answering in Images

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7W: Grounded Question Answering in Images. In IEEE Conference on Computer Vision and Pattern Recognition, 2016

    2016

  133. [141]

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V . Jawahar. Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, 2019. doi: 10.1109/...

    2019

  134. [142]

    Funsd: A dataset for form understanding in noisy scanned documents

    Jean-Philippe Thiran Guillaume Jaume, Hazim Kemal Ekenel. Funsd: A dataset for form understanding in noisy scanned documents. In Accepted to ICDAR-OST, 2019

    2019

  135. [143]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019

    2019

  136. [144]

    URL https://ai.100tal.com/icdar

    Mlhme-38k, 2025. URL https://ai.100tal.com/icdar

    2025

  137. [145]

    Mishra, K

    A. Mishra, K. Alahari, and C. V . Jawahar. Scene text recognition using higher order language priors. In BMVC, 2012

    2012

  138. [146]

    Syntax-aware network for handwritten mathematical expression recognition

    Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax-aware network for handwritten mathematical expression recognition. arXiv preprint arXiv:2203.01601, 2022

    2022 arXiv

  139. [147]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision (ECCV), 2022. 34

    2022

  140. [148]

    Visual information extraction in the wild: practical dataset and end-to-end solution

    Jianfeng Kuang, Wei Hua, Dingkang Liang, Mingkun Yang, Deqiang Jiang, Bo Ren, and Xiang Bai. Visual information extraction in the wild: practical dataset and end-to-end solution. In International Conference on Document Analysis and Recognition , pages 36–53. Springer, 2023

    2023

  141. [149]

    The iam-database: an english sentence database for offline handwriting recognition

    U-V Marti and Horst Bunke. The iam-database: an english sentence database for offline handwriting recognition. International journal on document analysis and recognition, 5:39–46, 2002

    2002

  142. [150]

    Textcaps: a dataset for image captioning with reading comprehension, 2020

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension, 2020

    2020

  143. [151]

    Coco-text: Dataset and benchmark for text detection and recognition in natural images

    Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140, 2016

    2016 arXiv

  144. [152]

    Saavedra, David Contreras, Juan Manuel Barrios, and Luiz S

    Markus Diem, Stefan Fiel, Florian Kleber, Robert Sablatnig, Jose M. Saavedra, David Contreras, Juan Manuel Barrios, and Luiz S. Oliveira. Proceedings of ieee international conference on frontiers in handwriting recognition. In 2014 14th International Conference on Frontiers in...

    2014 doi

  145. [153]

    URL https://wandb.ai/stacey/deepform_v1/reports/DeepFo rm-Understand-Structured-Documents-at-Scale--VmlldzoyODQ3Njg

    Deepform, 2025. URL https://wandb.ai/stacey/deepform_v1/reports/DeepFo rm-Understand-Structured-Documents-at-Scale--VmlldzoyODQ3Njg

    2025

  146. [154]

    Kleister: key information extraction datasets involving long documents with complex layouts

    Tomasz Stanisławek, Filip Grali ´ nski, Anna Wróblewska, Dawid Lipi ´ nski, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemysław Biecek. Kleister: key information extraction datasets involving long documents with complex layouts. In International Conference on...

    2021

  147. [155]

    Towards complex document understanding by discrete reasoning

    Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. Towards complex document understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4857–4866, 2022

    2022

  148. [156]

    Compositional semantic parsing on semi-structured tables

    Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305, 2015

    2015 arXiv

  149. [157]

    Dynamic prompt learning via policy gradient for semi- structured mathematical reasoning

    Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi- structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023

    2023

  150. [158]

    Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations

    Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations. arXiv preprint arXiv:2306.14321, 2023

    2023 arXiv

  151. [159]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL, 2022

    2022

  152. [160]

    Plotqa: Reason- ing over scientific plots

    Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reason- ing over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1527–1536, 2020. 35

    2020

  153. [161]

    Dvqa: Understanding data visualizations via question answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In CVPR, 2018

    2018

  154. [162]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pages 1697–1706, 2022

    2022

  155. [163]

    Tang, Angie Boggust, and Arvind Satyanarayan

    Benny J. Tang, Angie Boggust, and Arvind Satyanarayan. Vistext: A benchmark for semantically rich chart captioning, 2023

    2023

  156. [164]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  157. [165]

    Aligning large multi-modal model with robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023

    2023 arXiv

  158. [166]

    Websrc: a dataset for web-based structural reading comprehension

    Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. Websrc: a dataset for web-based structural reading comprehension. arXiv preprint arXiv:2101.09465, 2021

    2021 arXiv

  159. [167]

    Mavis: Mathematical visual instruction tuning, 2024

    Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, Peng Gao, and Hongsheng Li. Mavis: Mathematical visual instruction tuning, 2024

    2024

  160. [168]

    Geomverse: A systematic evaluation of large models for geometric reasoning

    Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241, 2023

    2023 arXiv

  161. [169]

    Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning, 2021

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning, 2021

    2021

  162. [170]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169, 2024

    2024

  163. [171]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024

    2024 arXiv

  164. [172]

    Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension

    Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern rec...

    2017

  165. [173]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  166. [174]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. ...

    2016

  167. [175]

    Mammoth: Building math generalist models through hybrid instruction tuning

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023

    2023 arXiv

  168. [176]

    Open- bezoar: Small, cost-effective and open models trained on mixes of instruction data

    Chandeepa Dissanayake, Lahiru Lowe, Sachith Gunasekara, and Yasiru Ratnayake. Open- bezoar: Small, cost-effective and open models trained on mixes of instruction data. arXiv preprint arXiv:2404.12195, 2024

    2024 arXiv

  169. [177]

    Mammoth2: Scaling instructions from the web

    Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 37:90629–90660, 2024

    2024

  170. [178]

    Emova: Empowering language models to see, hear and speak with vivid emotions

    Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, et al. Emova: Empowering language models to see, hear and speak with vivid emotions. arXiv preprint arXiv:2409.18042, 2024. 37

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.