Pith. sign in

REVIEW 4 major objections 5 minor 159 references

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language and visual understanding act as strong priors that drive image generation, so joint training from scratch on a language-heavy data mix reaches competitive generation at five percent of the usual generative-token budget.

desk verdict Strong small-scale study with a genuinely novel CLEVR protocol, but the scale validation of early unification is arithmetically impossible as written and the small-scale timing sweep is confounded; the central claims need fixing before they can be trusted. read the letter →

arxiv 2608.05000 v1 pith:W57O62RG submitted 2026-08-05 cs.CV cs.LGcs.MM

Junlin Han , Shengbang Tong , David Fan , Minghao Chen , Philip Torr , Filippos Kokkinos , Mike Lewis This is my paper
classification cs.CVcs.LGcs.MM
keywords multimodalpretrainingunifiedmodelsasymmetricknowledgetransfermodalitysynergyearlyunificationvisionlazinessflowmatchingmixture-of-experts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to replace heuristics in unified multimodal pretraining with controlled evidence about how language, visual understanding, and visual generation interact when trained inside one model. It claims that knowledge flow between the three abilities is asymmetric and concept-dependent: language boosts every visual capability, visual understanding transfers strongly to generation, and generation gives little back. It further claims that unification must be early and simultaneous, because delaying vision lets the language trunk harden and the vision pathway go 'lazy,' making the model lean on language priors instead of looking at the image. Architecture matters too: shared attention and normalization with modality-specific feed-forward networks turn potential competition into synergy, across four different visual tokenizer designs. If these claims hold, a single recipe emerges — roughly 70% language, 25% visual understanding, and 5% visual generation tokens, trained jointly from scratch in a mixture-of-experts model — which the authors validate on 13.5B-parameter models trained on 2T tokens, reaching strong image generation at five percent of the usual generative-token budget.

What carries the argument

Four mechanisms carry the argument. The first is the Transfusion-style training objective, in which one transformer predicts discrete text tokens by cross-entropy and denoises continuous image latents by flow matching, so that language, understanding, and generation share a single forward pass and every comparison is controlled within it. The second is the split-FFN transformer block — shared attention and shared normalization, with modality-specific feed-forward networks — which localizes the interaction: shared attention and normalization act as the bridge for cross-modal synergy, while decoupled FFNs absorb capacity competition; the scaled validation uses a 13.5B mixture-of-experts model with 256 experts routing to the top 16, two of them fixed to be modality-specific. The third is the CLEVR concept-removal protocol, a procedurally generated scene-graph dataset in which specific concepts (color, shape, relation, size, count) are ablated from one modality stream at a time, turning correlational real-data findings into causal statements about which concepts transfer and which leave only latent priors. The fourth is the vision-laziness probe battery, four activation- and attention-based measurements taken across identical checkpoints that quantify how much the vision pathway commits itself as a function of the language warm-up length.

What would settle it

Retrain the unification-timing sweep while holding the vision token count constant across conditions — for example, keep 500B vision tokens in every run by extending the unified stage or upsampling vision data as the language prefix grows; if visual metrics still fall monotonically with a longer language warm-up, early unification is doing the work, and if they flatten or invert, the claimed necessity collapses into a data-allocation effect. A cheaper second check is to repeat the 0B versus 800B comparison across at least three seeds and require the roughly 2-4 point VQA and generation gaps to persist outside noise, since the paper currently reports a single seed.

Watch

Extended reading notes

Core claim

On the paper's own terms, its contribution is an empirical account of the 'physics' of unified multimodal pretraining, with four load-bearing findings. First, knowledge flow is asymmetric: language is a universal booster for all visual tasks, visual understanding is a strong prior that drives visual generation, and visual generation is essentially neutral for language and understanding. Using a synthetic CLEVR testbed in which individual concepts are ablated from one modality stream at a time, the paper shows the flow is also concept-dependent: low-level attributes such as color and shape fail to transfer zero-shot in either direction, while structural concepts such as spatial relation, size, and count transfer from understanding to generation, and generation leaves latent priors that speed up later understanding of low-level concepts. Second, whether modalities synergize or compete is governed by data complexity — simple tasks in one modality improve the other, while complex real-world distributions compete for capacity — and by architecture, where shared attention and normalization with modality-specific feed-forward networks is the sweet spot that generalizes across four visual tokenization designs. Third, unification must come early and simultaneously: delaying vision systematically degrades visual understanding and generation, and four mechanistic probes (training-time image-FFN activations, wrapper-token embedding norms, inference-time vision activations, and the fraction of attention placed on image tokens) show the vision pathway progressively disengages as the language warm-up lengthens, a 'vision laziness' the paper links to the elementary visual errors of late-fused multimodal models. Fourth, these principles compose into a recipe: a data mix of roughly 70% language, 25% understanding, and 5% generation tokens, trained jointly from scratch in an MoE architecture, which at 13.5B parameters on 2T tokens improves on balanced and late-fusion baselines while cutting generative tokens fivefold.

Load-bearing premise

The timing experiment that anchors the early-unification claim sweeps the pure-language prefix from 0B to 800B tokens inside a fixed 1T-token budget, which also cuts vision tokens from 500B to 100B, so the visual decline may reflect data allocation rather than late unification, and every run uses seed 0, leaving small differences unchecked against seed variance.

Editorial extensions

If this is right

  • A unified model can spend the bulk of its tokens on language and understanding and still match balanced-recipe image quality: at 13.5B scale the 70/25/5 mix holds FID at 5.234 versus 5.131 for a balanced recipe while improving language accuracy and the visual-understanding average.
  • Sequential training in any modality order, with or without a replay buffer, underperforms simultaneous joint training on nearly every visual metric, so curriculum-style staging is a poor substitute for co-evolution.
  • Late-fused models built on a pretrained language trunk are predicted to show the vision-laziness signature — quiet image-branch activations, shrunk image-wrapper embeddings, and low attention on image tokens — which the paper connects to elementary visual mistakes in existing multimodal LLMs.
  • The asymmetry and the split-FFN advantage hold across RAE latents, raw pixels, CLIP-plus-VAE latents, and autoregressive UniTok codes, so the findings describe the learning objectives themselves rather than one visual tokenizer.
  • Low-complexity data in one modality improves the other, so simplified auxiliary streams act as cross-modal regularizers rather than distractions during pretraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the asymmetry extends to video and audio, the recipe generalizes predictably: each new generative modality should enter the mix with a small token share and lean on understanding and language priors, with joint training starting from the beginning of pretraining rather than after a unimodal warm-up.
  • Vision laziness suggests a cheap intervention worth testing: during late-fusion fine-tuning, a regularizer that keeps image-branch activations above a floor or enforces a minimum attention-on-image fraction could recover part of the visual gap without full re-pretraining.
  • The complexity result hints that data mixing could be adaptive: instead of a fixed 70/25/5 split, a curriculum that routes low-complexity streams to one modality while the other carries hard data would harvest the synergy effect continuously.
  • The paper's own limitations section concedes that beyond roughly 1T parameters generative modeling might start acting as an internal world simulator, in which case bidirectional knowledge flow could emerge and the 5% generation allocation would need to be re-tuned upward.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents a systematic empirical study of unified multimodal pretraining, using a 1.5B/2.3B Transfusion-style decoder-only model with split FFNs as the default testbed. It investigates knowledge flow among language, visual understanding, and visual generation on both real web-scale data and a controlled synthetic CLEVR benchmark; it identifies task-complexity and parameter-sharing conditions for cross-modal synergy; it argues that early joint unification is superior to late alignment and introduces a "vision laziness" mechanism; and it derives an asymmetric data recipe (L70/U25/G5) that is then validated on 13.5B MoE models trained on 2T tokens. The central claims are that knowledge flow is asymmetric and concept-dependent, that early unification is necessary to avoid vision laziness, and that strong generation can be bootstrapped with a small fraction of generative tokens.

Significance. If the claims hold, the paper would provide practically useful evidence on how to allocate compute and share parameters in unified multimodal pretraining. The synthetic CLEVR concept-removal protocol is a well-designed causal probe, the four independent mechanistic readouts for vision laziness are creative and falsifiable, and the robustness checks across four visual tokenizers strengthen the transfer results. The paper also evaluates on public benchmarks and includes held-out large-scale comparisons. However, the significance is currently conditional: the early-unification conclusion rests on a confounded small-scale timing sweep and an arithmetically impossible large-scale baseline, and the absence of seed variance makes fine-grained recipe rankings unreliable. These issues are identifiable and fixable, but they are load-bearing for the headline claims.

major comments (4)
  1. [§5.1, Figure 11] The timing sweep does not isolate unification timing because vision token volume is confounded with the length of the language-only prefix. With a fixed 1T total budget and a 50/50 unified stage, the runs with 0, 200, 400, 600, and 800B language-prefix tokens receive 500, 400, 300, 200, and 100B vision tokens, respectively. The observed decline in visual metrics across the sweep is therefore exactly what would be expected from simply reducing the visual training data, independent of when unification occurs. Please provide a matched-token-volume control (e.g., by holding vision tokens constant and extending the total budget, or by adding vision-only tokens during the prefix phase) or otherwise disentangle the two effects.
  2. [§6.2, Table 2] The Late-Fusion baseline as described is arithmetically inconsistent with the stated 2T-token budget. The Balanced Recipe is L50/U25/G25, i.e., 1T language and 1T vision tokens; the Late-Fusion baseline "begins with language-only data" for 60% of progress and then packs "the total volume of vision tokens ... identical" into the remaining 40%. Sixty percent of 2T is 1.2T language-only tokens, leaving only 0.8T, which cannot contain the promised 1T vision tokens plus any language or understanding tokens. This makes the Early-vs-Late comparison in Table 2 and the corresponding scale validation of early unification not a well-defined controlled experiment. Please supply the exact per-phase token schedule, or re-run the baseline with a feasible matched-vision-volume budget.
  3. [§2.1, Table 1, Figures 1–13] All experiments use a single fixed seed (seed 0) and no error bars, yet several central claims are statements about small monotonic differences. In Table 1, for example, the recommended 70/25/5 cell differs from neighboring 70/20/10 and 70/15/15 cells by 0.2–0.4 points on several VQA axes and by 0.02–0.05 on DPG/GenEval; Figure 3's "minor fluctuations" and Figure 8's ΔPPL of -0.008 to -0.015 are of similar magnitude. Without multiple seeds or a statistical treatment, the monotonicity claims in §3.1 and the optimal-ratio claim in §6.1 are not established. Please provide repeated-seed results (at least 3 seeds for the main comparisons) or a justified variance model, and report effect sizes.
  4. [§6.1, Table 1] The recipe ratio 70/25/5 is selected through a grid search on the same evaluation suite that is later used to argue for its optimality. The 13.5B/2T Full run provides some independent validation, so this is not fatal, but the selection rule is not stated and adjacent grid cells are close in score. Please state the selection criterion (e.g., best average, best generation, or Pareto rule) and show that the ranking is stable under seed variation or on a held-out subset of benchmarks.
minor comments (5)
  1. [Table 1] The 60/20/20 row contains the entry "44.633.0", which appears to be a missing-space formatting error and should read "44.6 33.0".
  2. [§2.1] The parameter count is confusing: the text says "a 1.5B Llama-3-like model" that "totaling 2.3B parameters" due to split FFNs; please clarify whether 1.5B is the active parameter count and 2.3B the total, and define both for the MoE models as well.
  3. [Abstract] The abstract's phrase "using only 5% of the compute budget" is misleading because the 70/25/5 recipe allocates 5% of tokens to generation while total compute includes language and understanding tokens; please rephrase as "5% of the token budget allocated to generation" or equivalent.
  4. [§5.3] "Vision laziness" is an apt descriptive label, but the four mechanistic readouts are correlational; the text should avoid claiming a single causal mechanism without a formal intervention or mediation analysis.
  5. [Appendix A] The limitations section is candid, and I appreciate the acknowledgment that video/audio and extreme-scale behavior are open; these statements should be reflected in the abstract's currently universal claims, which should be qualified as holding for the studied text/image regime.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central findings are empirical, benchmark-tested, and self-contained; the recipe grid-search-to-scale path and the broken Late-Fusion baseline create selection and control concerns but no definitional reduction.

full rationale

The paper's central claims are not circular in the definitional sense required by the rubric. Knowledge flow, synergy, early unification, and the recipes are established through controlled experiments with fixed budgets, public held-out benchmarks (GenEval, DPG, DCLM perplexity), synthetic CLEVR leave-one-concept-out ablations, and separate 2T-token scaling runs. The knowledge-flow findings in Section 3 are not defined in terms of the recipe ratios in Section 6; they are measured first, and the grid search then tests the asymmetric-mixture hypothesis. The 70/25/5 recipe is selected in Table 1 and later scaled in Table 2, which is a selected-then-validated procedure rather than a fitted parameter renamed as a prediction, so it does not reduce to its own inputs by construction. Self-citations (Cambrian, Beyond Language Modeling, Transfusion) supply protocols and architecture choices, but the load-bearing evidence is generated within this paper against external benchmarks, so no self-citation chain forces the conclusions. The Late-Fusion baseline in Section 6.2 is arithmetically inconsistent as written (60% of a 2T budget leaves only 0.8T tokens, which cannot contain the promised 1T vision tokens plus language), and the Section 5.1 timing sweep confounds unification timing with the number of vision tokens; these are correctness and experimental-design flaws, not circularity. Accordingly, the circularity score is low despite these validity concerns.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claims rest on several domain assumptions: the training framework is representative, the CLEVR and VLM metrics measure transfer, single-seed runs are reliable, and small-scale findings extrapolate to 13.5B. The only numeric quantity fitted to achieve the headline recipe is the L70/U25/G5 mix; the 3.0 diffusion loss weight is a chosen hyperparameter that also shapes all results.

free parameters (2)
  • Data mixing ratio (language/understanding/generation) = 70/25/5
    Selected from grid search in Table 1 as the best configuration and used in the scaled Full recipe.
  • Diffusion loss weight = 3.0x
    Chosen to balance cross-modal learning; not swept, but affects all comparisons and the cross-modal loss balance.
assumptions (5)
  • domain assumption Transfusion-style flow matching with RAE tokens is a representative setting for unified multimodal pretraining.
    All conclusions are drawn from this framework; results may not transfer to other unification designs.
  • domain assumption Synthetic CLEVR concept ablations and the Qwen3-VL-8B judge accurately measure concept transfer.
    Regex filters and VLM judging assume no leakage and reliable scoring; used for zero-shot transfer claims in section 3.2.
  • ad hoc to paper Single-seed runs with fixed seed 0 and no error bars are treated as reliable for monotonicity claims.
    Required to interpret small between-configuration differences in Figures 1 to 3 and Table 1 as real.
  • domain assumption Findings from 1.5B and 100B-token models generalize to 13.5B and 2T models.
    Stated in section 6.2; only one model per condition supports this extrapolation.
  • domain assumption Evaluation metrics such as perplexity, diffusion loss, and VQA suites capture the relevant capabilities.
    The headline claims about synergy and competition are read off these metrics, which may not capture all relevant model behavior.
invented entities (1)
  • Vision laziness
    purpose: Explains why late-aligned models underperform on vision; measured via activation norms, embedding norms, and attention fractions.
    Postulated mechanism with in-paper probes only; no external benchmark prediction that would falsify it outside this study.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes." pith.science (2026). https://pith.science/paper/W57O62RG

@misc{pith2026260805000,
  author       = {Pith},
  title        = {Pith review of: Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W57O62RG}},
  note         = {Machine review of arXiv:2608.05000}
}
read the original abstract

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

159 extracted references · 10 canonical work pages

  1. [1]

    Cm3: A causal masked multimodal model of the internet.arXiv preprint arXiv:2201.07520, 2022

    Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. Cm3: A causal masked multimodal model of the internet.arXiv preprint arXiv:2201.07520, 2022

    arXiv 2022

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. InNeurIPS, 2022

    2022

  3. [3]

    ICML 2024 Tutorial: Physics of Language Models, July 2024

    Allen-Zhu, Z. ICML 2024 Tutorial: Physics of Language Models, July 2024. Project page:https://physics.allen-zhu. com/

    2024

  4. [4]

    Vision foundation models as effective visual tokenizers for autoregressive generation.Advances in Neural Information Processing Systems, 38: 62656–62675, 2026

    Anlin, Z., Wen, X., Zhang, X., Ma, C., Wang, T., Yu, G., Zhang, X., and Qi, X. Vision foundation models as effective visual tokenizers for autoregressive generation.Advances in Neural Information Processing Systems, 38: 62656–62675, 2026

    2026

  5. [5]

    Toddler-inspired visual object learning.Advances in neural information processing systems, 31, 2018

    Bambach, S., Crandall, D., Smith, L., and Yu, C. Toddler-inspired visual object learning.Advances in neural information processing systems, 31, 2018

    2018

  6. [6]

    G., Shwartz-Ziv, R., and Lipson, H

    Ben-Levi, D., Goldfeder, J., Zhao, W., Lapid, R., LeVi, A., Roush, A. G., Shwartz-Ziv, R., and Lipson, H. Mirage probes: How vision models fake visual understanding.arXiv preprint arXiv:2606.13870, 2026

    arXiv 2026

  7. [7]

    Piqa: Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432–7439, 2020

    2020

  8. [8]

    Omni3d: A large benchmark and model for 3d object detection in the wild

    Brazil, G., Kumar, A., Straub, J., Ravi, N., Johnson, J., and Gkioxari, G. Omni3d: A large benchmark and model for 3d object detection in the wild. InCVPR, 2023

    2023

Show all 159 references
  1. [9]

    train on the test set

    Brown, E., Yang, J., Yang, S., Fergus, R., and Xie, S. Benchmark designers should" train on the test set" to expose exploitable non-visual shortcuts.arXiv preprint arXiv:2511.04655, 2025

    2025

  2. [10]

    Blip3o-next: Next frontier of native image generation.arXiv preprint arXiv:2510.15857, 2025a

    Chen, J., Xue, L., Xu, Z., Pan, X., Yang, S., Qin, C., Yan, A., Zhou, H., Chen, Z., Huang, L., et al. Blip3o-next: Next frontier of native image generation.arXiv preprint arXiv:2510.15857, 2025a

  3. [11]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025b

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., and Ruan, C. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025b

    arXiv

  4. [12]

    From pixels to feelings: Aligning mllms with human cognitive perception of images.arXiv preprint arXiv:2511.22805, 2025c

    Chen, Y., Han, J., Bai, T., Tong, S., Kokkinos, F., and Torr, P. From pixels to feelings: Aligning mllms with human cognitive perception of images.arXiv preprint arXiv:2511.22805, 2025c

  5. [13]

    Ideal: In-depth alignment makes a discrete representation autoencoder.arXiv preprint arXiv:2606.11096, 2026

    Chen, Y., Diao, Z., Wang, J., Kong, L., Ren, Y., He, B., Jiang, Y.-G., and Wu, Z. Ideal: In-depth alignment makes a discrete representation autoencoder.arXiv preprint arXiv:2606.11096, 2026

    2026 arXiv

  6. [14]

    Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044, 2019

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  7. [15]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  8. [16]

    Emu: Enhancing image generation models using photogenic needles in a haystack.arXiv preprint arXiv:2309.15807, 2023

    Dai, X., Hou, J., Ma, C.-Y., Tsai, S., Wang, J., Wang, R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., et al. Emu: Enhancing image generation models using photogenic needles in a haystack.arXiv preprint arXiv:2309.15807, 2023

    2023 arXiv

  9. [17]

    Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., Shi, G., and Fan, H. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    2025 arXiv

  10. [18]

    Sensenova-u1: Unifying multimodal understanding and generation with neo-unify architecture, 2026

    Diao, H., Wu, P., Deng, H., Wang, J., Bai, S., Wu, S., Fan, W., Ye, W., Tong, W., Fan, X., Li, Y., Wang, Y., Cao, Z., Lin, Z., Yang, Z., Cai, Z., Niu, Y., Zhu, Y., Liu, B., Lv, C., Yu, H., Xie, H., Wang, H., Fan, J., Li, J., Lu, J., Ni, J., Xu, J., Liang, K., Shi, L., Dai, L.,...

    2026 arXiv

  11. [19]

    Dreamllm: Synergistic multimodal comprehension and creation

    Dong, R., Han, C., Peng, Y., Qi, Z., Ge, Z., Yang, J., Zhao, L., Sun, J., Zhou, H., Wei, H., et al. Dreamllm: Synergistic multimodal comprehension and creation. InICLR, 2024

    2024

  12. [20]

    Scaling language-free visual representation learning

    Fan, D., Tong, S., Zhu, J., Sinha, K., Liu, Z., Chen, X., Rabbat, M., Ballas, N., LeCun, Y., Bar, A., et al. Scaling language-free visual representation learning. InICCV, 2025. 34

    2025

  13. [21]

    MME: A comprehensive evaluation benchmark for multimodal large language models

    Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., Ji, R., Shan, C., and He, R. MME: A comprehensive evaluation benchmark for multimodal large language models. InNeurIPS D&B Track, 2025

    2025

  14. [22]

    Lance: Unified multimodal modeling by multi-task synergy, 2026

    Fu, F., Huang, M., Wu, S., Jiang, Y., Huo, Y., Li, H., Song, Y., Ding, F., Guo, J., He, Q., Fu, Z., Mao, Z., and Zhang, Y. Lance: Unified multimodal modeling by multi-task synergy, 2026. URLhttps://arxiv.org/abs/2605.18678

    2026 arXiv

  15. [23]

    Planting a seed of vision in large language model.arXiv preprint arXiv:2307.08041, 2023

    Ge, Y., Ge, Y., Zeng, Z., Wang, X., and Shan, Y. Planting a seed of vision in large language model.arXiv preprint arXiv:2307.08041, 2023

    2023 arXiv

  16. [24]

    SEED-X: multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    Ge, Y., Zhao, S., Zhu, J., Ge, Y., Yi, K., Song, L., Li, C., Ding, X., and Shan, Y. SEED-X: multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    2024 arXiv

  17. [25]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gemini. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    2025 arXiv

  18. [26]

    X- omni: Reinforcement learning makes discrete autoregressive image generative models great again.arXiv preprint arXiv:2507.22058, 2025

    Geng, Z., Wang, Y., Ma, Y., Li, C., Rao, Y., Gu, S., Zhong, Z., Lu, Q., Hu, H., Zhang, X., et al. X- omni: Reinforcement learning makes discrete autoregressive image generative models great again.arXiv preprint arXiv:2507.22058, 2025

    2025 arXiv

  19. [27]

    Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    Ghosh, D., Hajishirzi, H., and Schmidt, L. Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    2023

  20. [29]

    A., and Ommer, B

    Gui, M., Schusterbauer, J., Phan, T., Krause, F., Susskind, J., Bautista, M. A., and Ommer, B. Adapting self-supervised representations as a latent space for efficient generation.arXiv preprint arXiv:2510.14630, 2025

    2025 arXiv

  21. [30]

    Vision as a dialect: Unifying visual understanding and generation via text-aligned representations.Advances in Neural Information Processing Systems, 38:158430–158459, 2026a

    Han, J., Chen, H., Zhao, Y., Wang, H., Zhao, Q., Yang, Z., He, H., Yue, X., and Jiang, L. Vision as a dialect: Unifying visual understanding and generation via text-aligned representations.Advances in Neural Information Processing Systems, 38:158430–158459, 2026a

  22. [31]

    Learning to see before seeing: Demystifying llm visual priors from language pre-training

    Han, J., Tong, S., Fan, D., Ren, Y., Sinha, K., Torr, P., and Kokkinos, F. Learning to see before seeing: Demystifying llm visual priors from language pre-training. InICLR, 2026b

  23. [32]

    Tv2tv: A unified framework for interleaved language and video generation.arXiv preprint arXiv:2512.05103, 2025

    Han, X., Emad, Y., Hall, M., Nguyen, J., Padthe, K., Robbins, L., Bar, A., Chen, D., Drozdzal, M., Elbayad, M., et al. Tv2tv: A unified framework for interleaved language and video generation.arXiv preprint arXiv:2512.05103, 2025

    2025

  24. [33]

    Clipscore: A reference-free evaluation metric for image captioning

    Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. Clipscore: A reference-free evaluation metric for image captioning. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514–7528, 2021

    2021

  25. [34]

    GANs trained by a two time-scale update rule converge to a local nash equilibrium

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by a two time-scale update rule converge to a local nash equilibrium. InNeurIPS, 2017

    2017

  26. [35]

    Hiippala, T., Alikhani, M., Haverinen, J., Kalliokoski, T., Logacheva, E., Orekhova, S., Tuomainen, A., Stone, M., and Bateman, J. A. Ai2d-rst: A multimodal corpus of 1000 primary school science diagrams.Language Resources and Evaluation, 55:661–688, 2021

    2021

  27. [36]

    Omni-view: Unlocking how generation facilitates understanding in unified 3d model based on multiview images.arXiv preprint arXiv:2511.07222, 2025

    Hu, J., Zhao, S., Chen, Q.-G., Qiu, X., Liu, J., Xu, Z., Luo, W., Zhang, K., and Lu, Y. Omni-view: Unlocking how generation facilitates understanding in unified 3d model based on multiview images.arXiv preprint arXiv:2511.07222, 2025

    2025

  28. [37]

    Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    Hu, X., Wang, R., Fang, Y., Fu, B., Cheng, P., and Yu, G. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    2024 arXiv

  29. [38]

    Bagelvla: Enhancing long-horizon manipulation via interleaved vision-language-action generation.arXiv preprint arXiv:2602.09849, 2026

    Hu, Y., Zhang, J., Luo, Y., Guo, Y., Chen, X., Sun, X., Feng, K., Lu, Q., Chen, S., Zhang, Y., et al. Bagelvla: Enhancing long-horizon manipulation via interleaved vision-language-action generation.arXiv preprint arXiv:2602.09849, 2026

    2026

  30. [39]

    Cross-model transferability among large language models on the platonic representations of concepts

    Huang, Y., Huang, C., Feng, D., Lei, W., and Lv, J. Cross-model transferability among large language models on the platonic representations of concepts. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3686–...

    2025

  31. [40]

    Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InCVPR, 2019

    2019

  32. [41]

    The platonic representation hypothesis

    Huh, M., Cheung, B., Wang, T., and Isola, P. The platonic representation hypothesis. InICML, 2024. 35

    2024

  33. [42]

    Pi0.7: a steerable generalist robotic foundation model with emergent capabilities.arXiv preprint arXiv:2604.15483, 2026

    Intelligence, P., Ai, B., Amin, A., Aniceto, R., Balakrishna, A., Balke, G., Black, K., Bokinsky, G., Cao, S., Charbonnier, T., et al. Pi0.7: a steerable generalist robotic foundation model with emergent capabilities.arXiv preprint arXiv:2604.15483, 2026

    2026 arXiv

  34. [43]

    Dino-tok: Adapting dino for visual tokenizers.arXiv preprint arXiv:2511.20565, 2025

    Jia, M., Li, M., Shu, Z., Zheng, A., Fan, L., Guo, J., Shi, T., Lu, D., Li, Z., Guo, X., et al. Dino-tok: Adapting dino for visual tokenizers.arXiv preprint arXiv:2511.20565, 2025

    2025

  35. [44]

    Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding

    Jiao, Y., Qiu, H., Jie, Z., Chen, S., Chen, J., Ma, L., and Jiang, Y.-G. Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding. InCVPR, 2025

    2025

  36. [45]

    Latentum: Unleashing the potential of interleaved cross-modal reasoning via a latent-space unified model.arXiv preprint arXiv:2604.02097, 2026

    Jin, J., Zhou, Z., Yang, X., Zhang, H., Liu, P., Zhu, J., and Deng, Z. Latentum: Unleashing the potential of interleaved cross-modal reasoning via a latent-space unified model.arXiv preprint arXiv:2604.02097, 2026

    2026

  37. [46]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. InCVPR, 2017

    2017

  38. [47]

    S., and Zettlemoyer, L

    Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611, 2017

    2017

  39. [48]

    Kimi k2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    K2.5. Kimi k2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    2026 arXiv

  40. [49]

    C., and Han, K

    Kim, H., Kang, S., Choi, Y., Ji, S., Woo, J., Chung, H., Han, S. C., and Han, K. Physics-based phenomenological characterization of cross-modal bias in multimodal models.arXiv preprint arXiv:2602.20624, 2026

    2026

  41. [50]

    Natural questions: a benchmark for question answering research.Transactions of the Association for Computational Linguistics, 7:453–466, 2019

    Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: a benchmark for question answering research.Transactions of the Association for Computational Linguistics, 7:453–466, 2019

    2019

  42. [51]

    Autoregressive image generation using residual quantization

    Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.-S. Autoregressive image generation using residual quantization. InCVPR, 2022

    2022

  43. [52]

    Ueval: A benchmark for unified multimodal generation.arXiv preprint arXiv:2601.22155, 2026a

    Li, B., Yin, Y., Chai, W., Fu, X., and Liu, Z. Ueval: A benchmark for unified multimodal generation.arXiv preprint arXiv:2601.22155, 2026a

  44. [53]

    Aria: An open multimodal native mixture-of-experts model.arXiv preprint arXiv:2410.05993, 2024a

    Li, D., Liu, Y., Wu, H., Wang, Y., Shen, Z., Qu, B., Niu, X., Zhou, F., Huang, C., Li, Y., et al. Aria: An open multimodal native mixture-of-experts model.arXiv preprint arXiv:2410.05993, 2024a

    arXiv

  45. [54]

    Onecat: Decoder-only auto-regressive model for unified understanding and generation.arXiv preprint arXiv:2509.03498, 2025a

    Li, H., Peng, X., Wang, Y., Peng, Z., Chen, X., Weng, R., Wang, J., Cai, X., Dai, W., and Xiong, H. Onecat: Decoder-only auto-regressive model for unified understanding and generation.arXiv preprint arXiv:2509.03498, 2025a

  46. [55]

    Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding

    Li, H., Tian, C., Shao, J., Zhu, X., Wang, Z., Zhu, J., Dou, W., Wang, X., Li, H., Lu, L., et al. Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding. InCVPR, 2025b

  47. [56]

    Spar: Semantic-pixel self-alignment and adaptive routing for unified multimodal models, 2026b

    Li, H., Chen, H., Zhu, C., Huang, X., Cai, J., Jiang, X., Hu, Y., and Chen, L. Spar: Semantic-pixel self-alignment and adaptive routing for unified multimodal models, 2026b. URLhttps://arxiv.org/abs/2606.23041

    arXiv

  48. [57]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InICML, 2023

    2023

  49. [58]

    W., Jitsev, J., Kollar, T., Dimakis, A

    Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., Keh, S., Arora, K., Garg, S., Xin, R., Muennighoff, N., Heckel, R., Mercat, J., Chen, M., Gururangan, S., Wortsman, M., Albalak, A., Bitton, Y., Nezhurina, M., Abbas, A., Hsieh, C.-Y., Ghosh,...

  50. [59]

    and He, K

    Li, T. and He, K. Back to basics: Let denoising generative models denoise.arXiv preprint arXiv:2511.13720, 2025

    2025 arXiv

  51. [60]

    Manzano: A simple and scalable unified multimodal model with a hybrid vision tokenizer, 2025c

    Li, Y., Qian, R., Pan, B., Zhang, H., Huang, H., Zhang, B., Tong, J., You, H., Du, X., Gan, Z., Kim, H., Jia, C., Wang, Z., Yang, Y., Gao, M., Dou, Z.-Y., Hu, W., Gao, C., Li, D., Dufter, P., Wang, Z., Yin, G., Zhang, Z., Chen, C., Zhao, Y., Pang, R., and Chen, Z. Manzano: A s...

  52. [61]

    Uniworld-v2: Reinforce image editing with diffusion negative-aware finetuning and mllm implicit feedback.arXiv preprint arXiv:2510.16888, 2025d

    Li, Z., Liu, Z., Zhang, Q., Lin, B., Yuan, S., Yan, Z., Ye, Y., Yu, W., Niu, Y., and Yuan, L. Uniworld-v2: Reinforce image editing with diffusion negative-aware finetuning and mllm implicit feedback.arXiv preprint arXiv:2510.16888, 2025d

    arXiv

  53. [62]

    Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.arXiv preprint arXiv:2411.04996, 2024

    Liang, W., Yu, L., Luo, L., Iyer, S., Dong, N., Zhou, C., Ghosh, G., Lewis, M., Yih, W.-t., Zettlemoyer, L., et al. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.arXiv preprint arXiv:2411.04996, 2024

    2024 arXiv

  54. [63]

    Mogao: An omni foundation model for interleaved multi-modal generation, 2025

    Liao, C., Liu, L., Wang, X., Luo, Z., Zhang, X., Zhao, W., Wu, J., Li, L., Tian, Z., and Huang, W. Mogao: An omni foundation model for interleaved multi-modal generation, 2025

    2025

  55. [64]

    Gear: Guided end-to-end autoregression for image synthesis, 2026

    Lin, B., Liu, Z., Lin, C., Chen, S., Ge, Y., Lin, Y., Zhang, J., Yang, M., Zhong, Z., Bo, L., and Yuan, L. Gear: Guided end-to-end autoregression for image synthesis, 2026. URLhttps://arxiv.org/abs/2606.32039

    2026 arXiv

  56. [65]

    Exploring mllm-diffusion information transfer with metacanvas.arXiv preprint arXiv:2512.11464, 2025

    Lin, H., Pan, X., Huang, Z., Hou, J., Wang, J., Chen, W., He, Z., Juefei-Xu, F., Sun, J., Fan, Z., et al. Exploring mllm-diffusion information transfer with metacanvas.arXiv preprint arXiv:2512.11464, 2025

    2025

  57. [66]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pp. 740–7...

    2014

  58. [67]

    V., Shrivastava, A., Luo, L., Iyer, S., Lewis, M., Ghosh, G., Zettlemoyer, L., and Aghajanyan, A

    Lin, X. V., Shrivastava, A., Luo, L., Iyer, S., Lewis, M., Ghosh, G., Zettlemoyer, L., and Aghajanyan, A. Moma: Efficient early-fusion pre-training with mixture of modality-aware experts.arXiv preprint arXiv:2407.21770, 2024

    2024 arXiv

  59. [68]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. InNeurIPS, 2023a

  60. [69]

    On the hidden mystery of ocr in large multimodal models.arXiv preprint arXiv:2305.07895, 2023b

    Liu, Y., Li, Z., Li, H., Yu, W., Huang, M., Peng, D., Liu, M., Chen, M., Li, C., Jin, L., et al. On the hidden mystery of ocr in large multimodal models.arXiv preprint arXiv:2305.07895, 2023b

    arXiv

  61. [70]

    Mmbench: Is your multi-modal model an all-around player? InECCV, 2024

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? InECCV, 2024

    2024

  62. [71]

    Tuna: Taming unified visual representations for native unified multimodal models.arXiv preprint arXiv:2512.02014, 2025

    Liu, Z., Ren, W., Liu, H., Zhou, Z., Chen, S., Qiu, H., Huang, X., An, Z., Yang, F., Patel, A., et al. Tuna: Taming unified visual representations for native unified multimodal models.arXiv preprint arXiv:2512.02014, 2025

    2025

  63. [72]

    Tuna-2: Pixel embeddings beat vision encoders for multimodal understanding and generation.arXiv preprint arXiv:2604.24763, 2026

    Liu, Z., Ren, W., Huang, X., Chen, S., Li, T., Chen, M., Ji, Y., He, S., Schult, J., Zeng, B., et al. Tuna-2: Pixel embeddings beat vision encoders for multimodal understanding and generation.arXiv preprint arXiv:2604.24763, 2026

    2026 arXiv

  64. [73]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Llama3. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  65. [74]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025

    Llama4. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025

    2025

  66. [75]

    Unified-io: A unified model for vision, language, and multi-modal tasks

    Lu, J., Clark, C., Zellers, R., Mottaghi, R., and Kembhavi, A. Unified-io: A unified model for vision, language, and multi-modal tasks. InICLR, 2022a

  67. [76]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. InCVPR, 2024

    2024

  68. [77]

    Atoken: A unified tokenizer for vision, 2025

    Lu, J., Song, L., Xu, M., Ahn, B., Wang, Y., Chen, C., Dehghan, A., and Yang, Y. Atoken: A unified tokenizer for vision, 2025. URLhttps://arxiv.org/abs/2509.14476

    2025

  69. [78]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. InNeurIPS, 2022b

  70. [79]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InICLR, 2023

    2023

  71. [80]

    Probing visual language priors in vlms.arXiv preprint arXiv:2501.00569, 2024

    Luo, T., Cao, A., Lee, G., Johnson, J., and Lee, H. Probing visual language priors in vlms.arXiv preprint arXiv:2501.00569, 2024

    2024 arXiv

  72. [81]

    Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation

    Ma, Y., Liu, X., Chen, X., Liu, W., Wu, C., Wu, Z., Pan, Z., Xie, Z., Zhang, H., Yu, X., et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. InCVPR, 2025

    2025

  73. [82]

    X., Tan, J

    Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InACL, 2022

    2022

  74. [83]

    Docvqa: A dataset for vqa on document images

    Mathew, M., Karatzas, D., and Jawahar, C. Docvqa: A dataset for vqa on document images. InWACV, 2021. 37

    2021

  75. [84]

    Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  76. [85]

    Nguyen, J., Havasi, M., Berrada, T., Zettlemoyer, L., and Chen, R. T. Oneflow: Concurrent mixed-modal and interleaved generation with edit flows.arXiv preprint arXiv:2510.03506, 2025

    2025

  77. [86]

    Does understanding inform generation in unified multimodal models? from analysis to path forward.arXiv preprint arXiv:2511.20561, 2025a

    Niu, Y., Jin, W., Liao, J., Feng, C., Jin, P., Lin, B., Li, Z., Zhu, B., Yu, W., and Yuan, L. Does understanding inform generation in unified multimodal models? from analysis to path forward.arXiv preprint arXiv:2511.20561, 2025a

  78. [87]

    Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025b

    Niu, Y., Ning, M., Zheng, M., Jin, W., Lin, B., Jin, P., Liao, J., Feng, C., Ning, K., Zhu, B., et al. Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025b

    arXiv

  79. [88]

    Oord, A. v. d., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning. InNeurIPS, 2017

    2017

  80. [89]

    Orhan, A. E. and Lake, B. M. Learning high-level visual representations from a child’s perspective without strong inductive biases.Nature Machine Intelligence, 6(3):271–283, 2024

    2024

  81. [90]

    N., Singh, A., Zhao, Z., Mishra, S

    Pan, X., Shukla, S. N., Singh, A., Zhao, Z., Mishra, S. K., Wang, J., Xu, Z., Chen, J., Li, K., Juefei-Xu, F., et al. Transfer between modalities with metaqueries.arXiv preprint arXiv:2504.06256, 2025

    2025 arXiv

  82. [91]

    N., Fan, X., Mishra, S

    Pan, X., Singh, A., Shukla, S. N., Fan, X., Mishra, S. K., and Xie, S. Repfusion: Leveraging multimodal priors for denoising in representation space, 2026. URLhttps://arxiv.org/abs/2606.14700

    2026

  83. [92]

    Unified multimodal autoregressive modeling with shared context — visual tokenizer is key to unification

    Peng, W., Meng, L., Cai, Y., Zhuang, X., Yang, Y., Fang, R., Wu, C., Lin, J., Wu, Z., and Bai, S. Unified multimodal autoregressive modeling with shared context — visual tokenizer is key to unification. InICML, 2026

    2026

  84. [93]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Qwen3-vl. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    2025 arXiv

  85. [94]

    Qwen3.5-omni technical report.arXiv preprint arXiv:2604.15804, 2026

    Qwen3.5-omni. Qwen3.5-omni technical report.arXiv preprint arXiv:2604.15804, 2026

    2026 arXiv

  86. [95]

    Generating diverse high-fidelity images with vq-vae-2

    Razavi, A., van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2. InNeurIPS, 2019

    2019

  87. [96]

    Reddy, S., Chen, D., and Manning, C. D. Coqa: A conversational question answering challenge.Transactions of the Association for Computational Linguistics, 7:249–266, 2019

    2019

  88. [97]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    2021

  89. [98]

    Social iqa: Commonsense reasoning about social interactions

    Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social iqa: Commonsense reasoning about social interactions. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language proces...

    2019

  90. [99]

    Fuselip: Multimodal embeddings via early fusion of discrete tokens.arXiv preprint arXiv:2506.03096, 2025

    Schlarmann, C., Croce, F., Flammarion, N., and Hein, M. Fuselip: Multimodal embeddings via early fusion of discrete tokens.arXiv preprint arXiv:2506.03096, 2025

    2025 arXiv

  91. [100]

    V., Zettlemoyer, L., and Yu, L

    Shi, W., Han, X., Zhou, C., Liang, W., Lin, X. V., Zettlemoyer, L., and Yu, L. Lmfusion: Adapting pretrained language models for multimodal generation.arXiv preprint arXiv:2412.15188, 2024

    2024 arXiv

  92. [101]

    Realunify: Do unified models truly benefit from unification? a comprehensive benchmark

    Shi, Y., Dong, Y., Ding, Y., Wang, Y., Zhu, X., Zhou, S., Liu, W., Tian, H., Wang, R., Wang, H., et al. Realunify: Do unified models truly benefit from unification? a comprehensive benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2026

  93. [102]

    Shukor, M., Fini, E., da Costa, V. G. T., Cord, M., Susskind, J., and El-Nouby, A. Scaling laws for native multimodal models. InICCV, 2025

    2025

  94. [103]

    Towards vqa models that can read

    Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. InCVPR, 2019

    2019

  95. [104]

    Improved baselines with representation autoencoders.arXiv preprint arXiv:2605.18324, 2026

    Singh, J., Zheng, B., Wu, Z., Zhang, R., Shechtman, E., and Xie, S. Improved baselines with representation autoencoders.arXiv preprint arXiv:2605.18324, 2026

    2026 arXiv

  96. [105]

    and Gasser, M

    Smith, L. and Gasser, M. The development of embodied cognition: Six lessons from babies.Artificial life, 11 (1-2):13–29, 2005

    2005

  97. [106]

    Steinberg, D. D. and Steinberg, M. T. Reading before speaking.Visible Language, 9(3), 1975

    1975

  98. [107]

    Generation enhances understanding in unified multimodal models via multi-representation generation.arXiv preprint arXiv:2601.21406, 2026

    Su, Z., Wei, H., Cen, K., Wang, Y., Chen, G., Yuan, C., and Chu, X. Generation enhances understanding in unified multimodal models via multi-representation generation.arXiv preprint arXiv:2601.21406, 2026. 38

    2026 arXiv

  99. [108]

    Generative multimodal models are in-context learners

    Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Luo, Z., Wang, Y., Rao, Y., Liu, J., Huang, T., and Wang, X. Generative multimodal models are in-context learners. InCVPR, 2024

    2024

  100. [109]

    Cosmos 3: Omnimodal world models for physical ai, 2026a

    Team, C. . Cosmos 3: Omnimodal world models for physical ai, 2026a. URLhttps://arxiv.org/abs/2606.02800

    arXiv

  101. [110]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Team, C. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  102. [111]

    Team, M. L. Longcat-next: Lexicalizing modalities as discrete tokens, 2026b. URLhttps://arxiv.org/abs/2603. 27538

  103. [112]

    Interaction models: A scalable approach to human-ai collaboration.Thinking Machines Lab, May 2026

    TML. Interaction models: A scalable approach to human-ai collaboration.Thinking Machines Lab, May 2026. doi: 10.64434/tml.20260511. https://thinkingmachines.ai/blog/interaction-models/

    2026 doi

  104. [113]

    C., Yang, J., Yang, S., Iyer, A., Pan, X., et al

    Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. InNeurIPS, 2024a

  105. [114]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InCVPR, 2024b

  106. [115]

    Metamorph: Multimodal understanding and generation via instruction tuning

    Tong, S., Fan, D., Zhu, J., Xiong, Y., Chen, X., Sinha, K., Rabbat, M., LeCun, Y., Xie, S., and Liu, Z. Metamorph: Multimodal understanding and generation via instruction tuning. InICCV, 2025

    2025

  107. [116]

    Beyond language modeling: An exploration of multimodal pretraining.ICML, 2026a

    Tong, S., Fan, D., Nguyen, J., Brown, E., Zhou, G., Qian, S., Zheng, B., Vallaeys, T., Han, J., Fergus, R., Murray, N., Ghazvininejad, M., Lewis, M., Ballas, N., Bar, A., Rabbat, M., Verbeek, J., Zettlemoyer, L., Sinha, K., LeCun, Y., and Xie, S. Beyond language modeling: An e...

  108. [117]

    Scaling text-to-image diffusion transformers with representation autoencoders.arXiv preprint, 2026b

    Tong, S., Zheng, B., Wang, Z., Tang, B., Ma, N., Brown, E., Yang, J., Fergus, R., LeCun, Y., and Xie, S. Scaling text-to-image diffusion transformers with representation autoencoders.arXiv preprint, 2026b

  109. [118]

    Reversing the flow: Generation-to-understanding synergy in large multimodal models

    Tong, Y., Chang, D., Yin, Z., Liu, X., Fang, Y., and Ma, Z. Reversing the flow: Generation-to-understanding synergy in large multimodal models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6976–6986, 2026c

  110. [119]

    LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  111. [120]

    F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al

    Tschannen, M., Gritsenko, A., Wang, X., Naeem, M. F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint ...

    2025 arXiv

  112. [121]

    K., Wang, W., Orhan, A

    Vong, W. K., Wang, W., Orhan, A. E., and Lake, B. M. Grounded language acquisition through the eyes and ears of a single child.Science, 383(6682):504–511, 2024

    2024

  113. [122]

    Quantifying the gap between understanding and generation within unified multimodal models.arXiv preprint arXiv:2602.02140, 2026a

    Wang, C., Chen, Y., Hu, Z., Chen, D., Chen, W., Wiegreffe, S., and Zhou, T. Quantifying the gap between understanding and generation within unified multimodal models.arXiv preprint arXiv:2602.02140, 2026a

  114. [123]

    Ovis-u1 technical report.arXiv preprint arXiv:2506.23044, 2025a

    Wang, G.-H., Zhao, S., Zhang, X., Cao, L., Zhan, P., Duan, L., Lu, S., Fu, M., Chen, X., Zhao, J., et al. Ovis-u1 technical report.arXiv preprint arXiv:2506.23044, 2025a

    arXiv

  115. [124]

    Ernie 5.0 technical report.arXiv preprint arXiv:2602.04705, 2026b

    Wang, H., Wu, H., Wu, T., Sun, Y., Liu, J., Yu, D., Ma, Y., He, J., He, Z., Hong, D., et al. Ernie 5.0 technical report.arXiv preprint arXiv:2602.04705, 2026b

  116. [125]

    Arm: An autoregressive large multimodal model with unified discrete representations.arXiv preprint arXiv:2606.11188, 2026c

    Wang, J., Wang, X., Pan, J., Hu, X., Li, F., Sun, J., Deng, C., Chen, Z., Chen, Y., Tian, K., et al. Arm: An autoregressive large multimodal model with unified discrete representations.arXiv preprint arXiv:2606.11188, 2026c

    arXiv

  117. [126]

    Image understanding makes for a good tokenizer for image generation.Advances in Neural Information Processing Systems, 37:31015–31035, 2024a

    Wang, L., Zhao, Y., Zhang, Z., Feng, J., Liu, S., and Kang, B. Image understanding makes for a good tokenizer for image generation.Advances in Neural Information Processing Systems, 37:31015–31035, 2024a

  118. [127]

    Uniddt: Unifying multimodal understanding and generation with decoupled diffusion transformer, 2026d

    Wang, S., Li, L., Chen, Y., Gao, R., Teng, Y., and Wang, L. Uniddt: Unifying multimodal understanding and generation with decoupled diffusion transformer, 2026d. URLhttps://arxiv.org/abs/2606.16255

  119. [128]

    L., Isola, P., and Cheung, B

    Wang, S. L., Isola, P., and Cheung, B. Words that make language models perceive.arXiv preprint arXiv:2510.02425, 2025b

  120. [129]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024b

    Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q., et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024b. 39

    arXiv

  121. [131]

    Lightbagel: A light-weighted, double fusion framework for unified multimodal understanding and generation.arXiv preprint arXiv:2510.22946, 2025c

    Wang, Z., Chen, Z., Gou, C., Li, F., Deng, C., Zhu, D., Li, K., Yu, W., Tu, H., Fan, H., et al. Lightbagel: A light-weighted, double fusion framework for unified multimodal understanding and generation.arXiv preprint arXiv:2510.22946, 2025c

  122. [132]

    M., et al

    Wang, Z., Zhang, J., Ge, J., Lian, L., Fu, L., Dunlap, L., Goldberg, K., Wang, X., Stoica, I., Chan, D. M., et al. Visgym: Diverse, customizable, scalable environments for multimodal agents.arXiv preprint arXiv:2601.16973, 2026f

  123. [133]

    Univideo: Unified understanding, generation, and editing for videos.arXiv preprint arXiv:2510.08377, 2025

    Wei, C., Liu, Q., Ye, Z., Wang, Q., Wang, X., Wan, P., Gai, K., and Chen, W. Univideo: Unified understanding, generation, and editing for videos.arXiv preprint arXiv:2510.08377, 2025

    2025 arXiv

  124. [134]

    Unig2u-bench: Do unified models advance multimodal understanding?arXiv preprint arXiv:2603.03241, 2026

    Wen, Z., Li, B., Zhang, W., Lei, J., Chen, X., Fan, Y., Zhang, Q., Wang, Y., Qiu, L., Li, B., et al. Unig2u-bench: Do unified models advance multimodal understanding?arXiv preprint arXiv:2603.03241, 2026

    2026

  125. [135]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. InCVPR, 2025a

  126. [136]

    Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025b

    Wu, C., Li, J., Zhou, J., Lin, J., Gao, K., Yan, K., Yin, S.-m., Bai, S., Xu, X., Chen, Y., et al. Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025b

    arXiv

  127. [137]

    Scaling native multimodal pre-training from scratch, 2026

    Wu, H., Wu, A., Wang, H., Wu, J., Ou, J., and Yu, B. Scaling native multimodal pre-training from scratch, 2026. URLhttps://arxiv.org/abs/2607.22043

    2026 arXiv

  128. [138]

    Liquid: Language models are scalable and unified multi-modal generators.arXiv preprint arXiv:2412.04332, 2024a

    Wu, J., Jiang, Y., Ma, C., Liu, Y., Zhao, H., Yuan, Z., Bai, S., and Bai, X. Liquid: Language models are scalable and unified multi-modal generators.arXiv preprint arXiv:2412.04332, 2024a

    arXiv

  129. [139]

    A., Yao, L., et al

    Wu, J., Xiong, Y., Li, X., Xia, Y., Wang, R., Wang, Y., Yu, T., Kim, S., Rossi, R. A., Yao, L., et al. Mitigating visual knowledge forgetting in mllm instruction-tuning via modality-decoupled gradient descent.arXiv preprint arXiv:2502.11740, 8, 2025c

  130. [140]

    Vila-u: a unified foundation model integrating visual understanding and generation.arXiv preprint arXiv:2409.04429, 2024b

    Wu, Y., Zhang, Z., Chen, J., Tang, H., Li, D., Fang, Y., Zhu, L., Xie, E., Yin, H., Yi, L., et al. Vila-u: a unified foundation model integrating visual understanding and generation.arXiv preprint arXiv:2409.04429, 2024b

    arXiv

  131. [142]

    Mindomni: Unleashing reasoning generation in vision language models with rgpo

    Xiao, Y., Song, L., Chen, Y., Luo, Y., Chen, Y., Gan, Y., Huang, W., Li, X., Qi, X., and Shan, Y. Mindomni: Unleashing reasoning generation in vision language models with rgpo. InNeurIPS, 2025

    2025

  132. [143]

    J., Wang, W., Lin, K

    Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. InICLR, 2025

    2025

  133. [144]

    Lumina-dimoo: An omni diffusion large language model for multi-modal generation and understanding.arXiv preprint arXiv:2510.06308, 2025

    Xin, Y., Qin, Q., Luo, S., Zhu, K., Yan, J., Tai, Y., Lei, J., Cao, Y., Wang, K., Wang, Y., et al. Lumina-dimoo: An omni diffusion large language model for multi-modal generation and understanding.arXiv preprint arXiv:2510.06308, 2025

  134. [145]

    Tbac-uniimage: Unified understanding and generation by ladder-side diffusion tuning.arXiv preprint arXiv:2508.08098, 2025

    Xu, J., Yin, Y., and Chen, X. Tbac-uniimage: Unified understanding and generation by ladder-side diffusion tuning.arXiv preprint arXiv:2508.08098, 2025

    2025 arXiv

  135. [146]

    Context unrolling in omni models.arXiv preprint arXiv:2604.21921, 2026a

    Yang, C., Lin, Z., Zhao, Y., Xiao, F., He, H., Zhao, Q., Deng, C., Li, K., Ding, Z., Guo, Y., et al. Context unrolling in omni models.arXiv preprint arXiv:2604.21921, 2026a

    arXiv

  136. [147]

    Y., Liu, Y., Ma, X., and Berg-Kirkpatrick, T

    Yang, C., Shi, C., Shui, B., Wu, Y., Tao, M., Wang, H., Lee, I. Y., Liu, Y., Ma, X., and Berg-Kirkpatrick, T. From reasoning to pixels: Benchmarking the alignment gap in unified multimodal models.arXiv preprint arXiv:2602.08336, 2026b

    arXiv

  137. [148]

    Rae-ar: Taming autoregressive models with representation autoencoders.arXiv preprint arXiv:2604.01545, 2026

    Yu, H., Xu, H., Huang, J., Xue, Z., Huang, H., Duan, N., and Zhao, F. Rae-ar: Taming autoregressive models with representation autoencoders.arXiv preprint arXiv:2604.01545, 2026

    2026

  138. [149]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InCVPR, 2024

    2024

  139. [150]

    Uniflow: A unified pixel flow tokenizer for visual understanding and generation

    Yue, Z., Zhang, H., Zeng, X., Chen, B., Wang, C., Zhuang, S., Dong, L., Du, K., Wang, Y., Wang, L., and Wang, Y. Uniflow: A unified pixel flow tokenizer for visual understanding and generation. InICLR, 2026. 40

    2026

  140. [151]

    Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  141. [152]

    Sigmoid loss for language image pre-training

    Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international conference on computer vision, pp. 11975–11986, 2023

    2023

  142. [153]

    J., and Ma, Y

    Zhai, Y., Tong, S., Li, X., Cai, M., Qu, Q., Lee, Y. J., and Ma, Y. Investigating the catastrophic forgetting in multimodal large language models. InCPAL, 2024

    2024

  143. [154]

    Hydra-x: Native unified multimodal models with holistic visual tokenizers.arXiv preprint arXiv:2606.13289, 2026a

    Zhang, G., Qiu, X., Cui, Y., Song, T., Li, C., Li, J., Huang, T., Zhang, X., Li, Y., Wu, J., et al. Hydra-x: Native unified multimodal models with holistic visual tokenizers.arXiv preprint arXiv:2606.13289, 2026a

    arXiv

  144. [155]

    Nextflow: Unified sequential modeling activates multimodal understanding and generation.arXiv preprint arXiv:2601.02204, 2026b

    Zhang, H., Qu, L., Liu, Y., Chen, H., Song, Y., Dong, Y., Sun, S., Li, X., Wang, X., Jiang, Y., et al. Nextflow: Unified sequential modeling activates multimodal understanding and generation.arXiv preprint arXiv:2601.02204, 2026b

  145. [156]

    Are unified vision-language models necessary: Generalization across understanding and generation.arXiv preprint arXiv:2505.23043, 2025

    Zhang, J., Li, T., Li, L., Yang, Z., and Cheng, Y. Are unified vision-language models necessary: Generalization across understanding and generation.arXiv preprint arXiv:2505.23043, 2025

    2025 arXiv

  146. [157]

    Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation.arXiv preprint arXiv:2502.05178, 2025

    Zhao, Y., Xue, F., Reed, S., Fan, L., Zhu, Y., Kautz, J., Yu, Z., Krähenbühl, P., and Huang, D.-A. Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation.arXiv preprint arXiv:2502.05178, 2025

    2025 arXiv

  147. [158]

    Hita: Holistic tokenizer for autoregressive image generation.arXiv preprint arXiv:2507.02358, 2025

    Zheng, A., Wang, H., Zhao, Y., Deng, W., Wang, T., Zhang, X., and Qi, X. Hita: Holistic tokenizer for autoregressive image generation.arXiv preprint arXiv:2507.02358, 2025

    2025 arXiv

  148. [159]

    Diffusion transformers with representation autoencoders

    Zheng, B., Ma, N., Tong, S., and Xie, S. Diffusion transformers with representation autoencoders. InICLR, 2026

    2026

  149. [160]

    Semantic understanding of scenes through the ade20k dataset.IJCV, 2019

    Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., and Torralba, A. Semantic understanding of scenes through the ade20k dataset.IJCV, 2019

    2019

  150. [161]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and diffuse images with one multi-modal model. InICLR, 2025

    2025

  151. [162]

    and Chuang, I

    Ziyin, L. and Chuang, I. Proof of a perfect platonic representation hypothesis.arXiv preprint arXiv:2507.01098, 2025. 41

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.