Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read UniMoD claims that token redundancy in unified multimodal transformers is task- and layer-specific, so pruning should use a separate router per task.

desk verdict Useful task-aware MoD idea, but the ARank metric as written is degenerate and the layer-selection instructions contradict themselves; the paper needs a substantive revision before its efficiency claims are supportable. read the letter →

arxiv 2502.06474 v1 pith:YJOELYFK submitted 2025-02-10 cs.CV

Weijia Mao , Zhenheng Yang , Mike Zheng Shou This is my paper · ORCID
classification cs.CV
keywords Mixture-of-Depthstokenpruningunifiedmultimodaltransformerstask-awareroutingARanktrainingefficiencydiffusionandautoregressivemodelsShow-oEmu3
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniMoD claims that token redundancy in unified multimodal transformers is task- and layer-dependent, so a single pruning router must trade off between generation and understanding tasks and loses quality. The paper proposes giving each task its own router that decides which tokens skip a transformer layer, and uses a rank-based redundancy statistic called ARank to select which layers become prunable. On Show-o and Emu3 this cuts training FLOPs by about 15% and 40% respectively while keeping or improving most benchmark scores. If correct, efficient training of unified models should be planned per task, not globally.

What carries the argument

The carrying mechanism is the Layer Switch Module, which uses the ARank statistic --- the mean rank of the attention matrices across heads, $\tau(x_i, D_i)=\frac{1}{n_h}\sum_{h=1}^{n_h}\mathrm{rank}(A_h)$ --- to decide which dense transformer layers become prunable Mixture-of-Depths (MoD) layers; high ARank means low token redundancy. UniMoD then replaces selected layers with three kinds of MoD blocks: one that prunes only generation (T2I) tokens, one that prunes only understanding (MMU) tokens, and one shared block that prunes both, each with its own router, capacity, and threshold. The routers use a Gumbel-style binary assignment during the task-competition analysis and a capacity loss to enforce token budgets.

What would settle it

Train UniMoD with the same per-task routers and the same pruning budget, but pick layers with the lowest ARank instead of the highest; if benchmark scores end up about equal, ARank is not what makes the method work. Then rerun with a single shared router at the same layers and FLOPs; if it matches UniMoD, task-splitting is not the reason.

Watch

Extended reading notes

Core claim

UniMoD's central claim is that a unified multimodal transformer should not prune tokens with one global router. The paper shows empirically that attention patterns, ARank values, and token importance all differ between the generation task and the understanding task, and that both also vary across layers, then builds a training method with a separate router per task and a Layer Switch Module that converts selected dense layers into Mixture-of-Depths blocks. On Show-o and Emu3 the method reduces training FLOPs by about 15% and 40% respectively while holding benchmark scores level or improving them. The claim is that task-aware routing, not token pruning per se, is what achieves this efficiency-performance trade-off.

Load-bearing premise

The load-bearing premise is that ARank, measured on just 50 samples per task, identifies the layers where tokens can be pruned without loss — a premise the experiments do not fully test, since Show-o converts its last 12 layers and Emu3 its last 16 rather than the top-ARank layers the method section prescribes.

Editorial extensions

If this is right

  • Show-o training FLOPs drop about 15% and Emu3 training FLOPs drop about 40% while benchmark scores stay level or improve, so unified transformers do not need full-token computation in every layer.
  • At the same pruning budget, a single shared router and interleaved-layer pruning both score worse than UniMoD, so the per-task split is what preserves quality.
  • The saving is larger when images are represented by more tokens: Emu3's 4096-token images enable a roughly 40% FLOP reduction, while Show-o's 1024-token images give about 15%.
  • The same ARank-based pruning transfers to pure diffusion generation models DiT and PixArt, so the mechanism is not limited to unified architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's reported implementations convert the last 12 layers of Show-o and the last 16 of Emu3 rather than the top-ARank layers the method section selects, so the experiments mainly validate per-task routing; ARank-based layer selection is a design choice that still awaits a direct test.
  • The Emu3 comparison is against a re-trained baseline using LLaVA-v1.5-mix-665K for understanding, because Emu3's original training data and code are not public, so the 40% saving is measured for this finetuning setup rather than for reproducing Emu3's own published training run.
  • A testable extension is to let the task assignment be inferred per sequence instead of given, so mixed or unknown task mixtures could be routed without explicit labels.
  • If the per-task redundancy pattern is stable across training, the same routers could be applied at inference time, a speedup the paper does not measure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes UniMoD, a task-aware token-pruning method for unified multimodal transformers. The authors first analyze attention weight patterns, layer importance via layer-skipping experiments, and token redundancy using an ARank metric, concluding that token redundancy differs by task and by layer. They then convert selected dense transformer layers into task-specific Mixture-of-Depths (MoD) blocks with separate routers for text-to-image generation and multimodal understanding, and report roughly 15% training FLOP reduction for Show-o and 40% for Emu3 while maintaining or slightly improving several benchmark scores. Extensions to pure diffusion models DiT and PixArt are also reported.

Significance. If the efficiency results are robust, the paper addresses a real cost bottleneck in training unified multimodal transformers and proposes a sensible design space: per-task routing instead of a single global router. The experiments cover two representative architectures and include ablations against basic MoD, interleaved-layer pruning, and single-router variants, together with FLOP, memory, and wall-clock measurements. The main weakness is that the ARank metric that motivates the method is degenerate as written, and the layer-selection protocol is stated inconsistently in different sections; these issues currently prevent validation of the paper's central design rationale. The paper promises code release, which would help reproducibility once the protocol is clarified.

major comments (4)
  1. [Sec. 3.3, Eq. (5)] As written, ARank is degenerate: for a single token x_i, A_h = (W_Q^h x_i)(W_K^h x_i)^T is an outer product of two vectors, so its rank is at most 1 and is 1 for essentially every token, layer, and task. The text also states A_h ∈ R^{l×l}, but with x_i ∈ R^d and W_Q^h, W_K^h ∈ R^{d×d_h}, the matrix is in R^{d_h×d_h}. Consequently, Fig. 3's layer/task variation and Observations 3–4 cannot be obtained from the stated formula, and the Layer Switch Module in Sec. 4.1 is built on an unvalidated quantity. The authors need to redefine ARank, presumably using the full attention map A_h = (X W_Q^h)(X W_K^h)^T or an equivalent sequence-level quantity, and re-run the redundancy analysis.
  2. [Sec. 4.1 vs. Sec. 5.1 and App. A.3] The layer-selection protocol is stated in three incompatible ways. Sec. 4.1 says to select the top 12 layers with the highest ARank values for each task; Sec. 5.1 says the last 12 layers are converted for Show-o and the last 16 for Emu3; App. A.3 says the 14 layers with the lowest ARank values are selected for DiT and PixArt. Since the paper defines higher ARank as lower redundancy, selecting the highest-ARank layers is also in tension with the stated goal of pruning highly redundant layers. The implementation must state one unambiguous, reproducible rule for layer selection.
  3. [Sec. 5.1 and Table 5] The relationship between the ARank-based pruning-ratio selection and the implemented capacities is not specified. Sec. 4.1 says the token pruning ratio is determined proportionally from ARank values, but the experiments use a capacity schedule from 1 down to 0.2 for Show-o MMU, a fixed 20% T2I pruning ratio for Show-o, and 80% token pruning in Emu3, with no formula connecting these numbers to ARank. Because Table 5 fixes the pruning rate across all ablations, the specific contribution of the ARank-derived per-task ratios is not isolated; a sensitivity analysis over capacities and task-specific thresholds δ_t is needed.
  4. [Sec. 3.4 and Fig. 5] Observation 5 is supported only by an underspecified competition experiment: the text gives no dataset size, aggregation details, or quantitative table, and Fig. 5 reports 'Number of Tokens Weight=1' without explaining how it is computed or averaged. Since this observation directly motivates the per-task router design, the evidence should be reported in a reproducible form.
minor comments (4)
  1. [References] The bibliography contains clearly placeholder entries such as 'Alpher, F. Frobnication' and 'LastName, F. The frobnicatable foo filter'; these must be removed or replaced before submission.
  2. [Sec. 5.1 and Table 4] The paper alternates between 'UniMoD' and 'UniMod', and Table 3 reports Show-o UniMoD as 43.3 TFLOPS while Table 4 gives 45.9 TFLOPS for T2I and 40.8 TFLOPS for MMU; the relationship between these numbers should be stated explicitly.
  3. [Eq. (6) and Sec. 4.1] In Eq. (6), D_t(x_i) is described as a 'task-specific router function,' but in Eq. (4) D_i denotes the i-th transformer layer; the notation for the layer transformation and the routing function should be separated to avoid confusion.
  4. [Fig. 1 and Sec. 3.4] There are typographical issues in the figure and caption ('KeyObservation2' and a duplicated caption line), and Fig. 5 lacks axis details describing the dataset and aggregation used for 'Number of Tokens Weight=1'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the efficiency gains are externally benchmarked, and the degenerate ARank definition is a correctness risk, not a circular derivation.

full rationale

The paper's main quantitative claims—roughly 15% training-FLOP reduction for Show-o and 40% for Emu3 with maintained or improved benchmark scores—are measured directly against full-computation baselines on external benchmarks (GQA, POPE, MME, MMMU, VQAv2, GenEval) and are not obtained by plugging fitted values back into the definition of the quantity being reported. The task-aware router design is motivated by the empirical observations in Section 3, but the benchmark results then validate the design externally; this is a design loop, not a logical circularity. Citations to Show-o and VideoLLM-MoD include overlapping authors, but they are used as base models and prior methods with public checkpoints and external evaluations, not as unverified authorities, so they are not load-bearing circularity. Two correctness concerns should be flagged without inflating the circularity score. First, Eq. (5) as written defines A_h as a per-token outer product (W_Q^h x_i)(W_K^h x_i)^T, whose rank is at most 1, and the stated dimensions A_h in R^{l x l} with W_Q^h in R^{d x d_h} are inconsistent; if this is the metric actually used, Observations 3 and 4 and the ARank-based layer selection in Section 4.1 are unsupported, and Section 5.1 instead converts the last 12 layers rather than the top-12 ARank layers. Second, this is a validation and technical-error issue for the analysis section, not a case of a prediction reducing to its input by construction. The FLOPs and accuracy results stand or fall on the external comparisons, so the circularity score remains low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or postulated entities. Its free parameters are the per-task pruning capacities and layer-selection counts, which are hand-chosen or ARank-derived. The key assumptions are that ARank is a valid redundancy signal, that later layers are safer to prune, that the ST-Gumbel routers train stably, and that the modified training pipeline is an appropriate baseline. These assumptions are plausible but not all independently verified.

free parameters (5)
  • MMU token capacity schedule (Show-o) = scaled from 1.0 down to 0.2 across layers
    Chosen by hand to set how many MMU tokens are retained per layer; no sensitivity analysis is reported.
  • T2I token pruning ratio (Show-o) = 20% of tokens pruned in later layers
    Chosen by hand based on ARank observations; no ablation over the ratio is provided.
  • Emu3 token pruning ratio = 80% of tokens pruned in each of the last 16 layers
    Chosen by hand; no sensitivity analysis or per-layer breakdown is given.
  • Number of converted MoD layers = 12 for Show-o, 16 for Emu3, 14 for DiT/PixArt
    Selected via ARank, but the main method text (top 12 highest ARank) contradicts the implementation (last 12 layers), and the diffusion experiments use the lowest-ARank layers.
  • Task-specific router threshold delta_t = not reported
    Equation (6) introduces task-specific thresholds delta_t, but their values and tuning procedure are never stated.
assumptions (4)
  • domain assumption ARank (mean rank of attention matrices) is a valid proxy for token redundancy and prunability in unified transformers.
    The Layer Switch Module (Sec 4.1) uses ARank from Eq. (5) to select layers and set pruning ratios, but the paper does not validate that low-ARank tokens are safely prunable without performance loss in these specific models.
  • domain assumption Pruning later layers is less harmful than pruning early layers.
    Justified by Table 1 (skipping early layers hurts GQA more) and ARank trends, leading to the choice of the last 12 layers for Show-o; this is treated as a general property without a formal account.
  • domain assumption Task-specific routers trained with straight-through Gumbel softmax converge and remain stable during training.
    The router uses ST-Gumbel (Appendix A.4) with an auxiliary budget loss, but no convergence analysis or training-curve diagnostics are provided.
  • domain assumption The revised two-stage training pipeline with Cambrian and internal high-quality data is a fair baseline for Show-o.
    Full-computation Show-o numbers in Table 3 use the authors' pipeline, not the original Show-o training recipe, so 'maintaining performance' is relative to a modified baseline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths." pith.science (2026). https://pith.science/paper/YJOELYFK

@misc{pith2026250206474,
  author       = {Pith},
  title        = {Pith review of: UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YJOELYFK}},
  note         = {Machine review of arXiv:2502.06474}
}
read the original abstract

Unified multimodal transformers, which handle both generation and understanding tasks within a shared parameter space, have received increasing attention in recent research. Although various unified transformers have been proposed, training these models is costly due to redundant tokens and heavy attention computation. In the past, studies on large language models have demonstrated that token pruning methods, such as Mixture of Depths (MoD), can significantly improve computational efficiency. MoD employs a router to select the most important ones for processing within a transformer layer. However, directly applying MoD-based token pruning to unified transformers will result in suboptimal performance because different tasks exhibit varying levels of token redundancy. In our work, we analyze the unified transformers by (1) examining attention weight patterns, (2) evaluating the layer importance and token redundancy, and (3) analyzing task interactions. Our findings reveal that token redundancy is primarily influenced by different tasks and layers. Building on these findings, we introduce UniMoD, a task-aware token pruning method that employs a separate router for each task to determine which tokens should be pruned. We apply our method to Show-o and Emu3, reducing training FLOPs by approximately 15% in Show-o and 40% in Emu3, while maintaining or improving performance on several benchmarks. Code will be released at https://github.com/showlab/UniMoD.

Figures

Figures reproduced from arXiv: 2502.06474 by the authors.

Figure 1
Figure 1. (a) Pipeline and challenges of applying Mixture of Depths (MoD) to unified transformers. A single router prunes [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Attention weight for text and image tokens across different transformer layers for two tasks: Multi-Modal [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. ARank variations across different layers for four unified transformers: Show-o, JanusFlow, Emu3, and Lumina [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Pipeline of UniMoD. The Layer Switch Module transforms dense transformer layers into three specialized types: [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Token weight assignment using Gumbel Softmax. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A self-improving post-training method that uses a model's own generated images as training data, with SFT and GRPO, improves generation and understanding and reduces task imbalance.

Reference graph

Works this paper leans on

45 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [1]

    A., Awan, A

    Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219,

    arXiv

  2. [6]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y ., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023a. Chen, J., Lv, Z., Wu, S., Lin, K. Q., Song, C., Gao, D., Liu, J., Gao, Z., Mao, D., and Shou, M. Z. Videollm-online: Online vi...

    arXiv

  3. [9]

    Layer skip: Enabling early exit inference and self-speculative decoding

    Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., et al. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710, 2024a. Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S....

    arXiv

  4. [11]

    Model tells you what to discard: Adaptive KV cache compression for llms

    Ge, S., Zhang, Y ., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive KV cache compression for llms. In ICLR. OpenReview.net, 2024a. 10 Submission and Formatting Instructions for ICML 2025 Ge, Y ., Zhao, S., Zhu, J., Ge, Y ., Yi, K., Song, L., Li, C., Ding, X., and Shan, Y . Seed-x: Multimodal models with unified multi-gra...

    arXiv 2025

  5. [14]

    Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,

    arXiv

  6. [15]

    Videopoet: A large language model for zero- shot video generation

    Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang, J., Hornung, R., Adam, H., Akbari, H., Alon, Y ., Birodkar, V ., et al. Videopoet: A large language model for zero- shot video generation. arXiv preprint arXiv:2312.14125,

    arXiv

  7. [16]

    Orthus: Autoregressive interleaved image- text generation with modality-specific heads

    Kou, S., Jin, J., Liu, C., Ma, Y ., Jia, J., Chen, Q., Jiang, P., and Deng, Z. Orthus: Autoregressive interleaved image- text generation with modality-specific heads. CoRR, abs/2412.00127,

    arXiv

  8. [18]

    Janusflow: Har- monizing autoregression and rectified flow for unified multimodal understanding and generation

    Ma, Y ., Liu, X., Chen, X., Liu, W., Wu, C., Wu, Z., Pan, Z., Xie, Z., Zhang, H., Zhao, L., et al. Janusflow: Har- monizing autoregression and rectified flow for unified multimodal understanding and generation. arXiv preprint arXiv:2411.07975,

    arXiv

Show all 45 references
  1. [19]

    Mm1: Methods, analysis & insights from multimodal llm pre-training

    McKinzie, B., Gan, Z., Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P., Shah, D., Du, X., Peng, F., Weers, F., et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611,

    arXiv

  2. [20]

    github.io/book2

    URL http://probml. github.io/book2. Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741,

    arXiv

  3. [21]

    Kosmos-2: Grounding multimodal large language models to the world

    Peng, Z., Wang, W., Dong, L., Hao, Y ., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. CoRR, abs/2306.14824,

    arXiv

  4. [22]

    Sdxl: Im- proving latent diffusion models for high-resolution image synthesis

    Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., M ¨uller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952,

    arXiv

  5. [23]

    K., Yuan, Z., and Wu, X

    Qu, L., Zhang, H., Liu, Y ., Wang, X., Jiang, Y ., Gao, Y ., Ye, H., Du, D. K., Yuan, Z., and Wu, X. Tokenflow: Unified image tokenizer for multimodal understanding and generation. CoRR, abs/2412.03069,

    arXiv

  6. [24]

    12 Submission and Formatting Instructions for ICML 2025 Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y ., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning resea...

    2025

  7. [25]

    Hierarchical text-conditional image generation with CLIP latents

    Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/2204.06125, 2022a. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip laten...

    arXiv

  8. [26]

    Lazydit: Lazy learning for the acceleration of diffusion transform- ers

    Shen, X., Song, Z., Zhou, Y ., Chen, B., Li, Y ., Gong, Y ., Zhang, K., Tan, H., Kuen, J., Ding, H., et al. Lazydit: Lazy learning for the acceleration of diffusion transform- ers. arXiv preprint arXiv:2412.12444,

    arXiv

  9. [27]

    V ., Zettle- moyer, L., and Yu, L

    Shi, W., Han, X., Zhou, C., Liang, W., Lin, X. V ., Zettle- moyer, L., and Yu, L. Llamafusion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:2412.15188,

    arXiv

  10. [28]

    Journeydb: A benchmark for generative image understanding

    Sun, K., Pan, J., Ge, Y ., Li, H., Duan, H., Wu, X., Zhang, R., Zhou, A., Qin, Z., Wang, Y ., Dai, J., Qiao, Y ., Wang, L., and Li, H. Journeydb: A benchmark for generative image understanding. In NeurIPS, 2023a. Sun, P., Jiang, Y ., Chen, S., Zhang, S., Peng, B., Luo, P., and...

    arXiv

  11. [29]

    Generative multimodal models are in-context learners

    Sun, Q., Cui, Y ., Zhang, X., Zhang, F., Yu, Q., Luo, Z., Wang, Y ., Rao, Y ., Liu, J., Huang, T., and Wang, X. Generative multimodal models are in-context learners. CoRR, abs/2312.13286, 2023b. Sun, Q., Yu, Q., Cui, Y ., Zhang, F., Zhang, X., Wang, Y ., Gao, H., Liu, J., Huan...

    arXiv

  12. [30]

    Chameleon: Mixed-modal early-fusion foundation models

    Team, C. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818,

    arXiv

  13. [31]

    C., Yang, J., Yang, S., Iyer, A., Pan, X., Wang, A., Fergus, R., LeCun, Y ., and Xie, S

    Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., Wang, A., Fergus, R., LeCun, Y ., and Xie, S. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. CoRR, abs/2406.16860, 2024a. Tong, S., Brown, E., Wu...

    arXiv

  14. [33]

    Emu3: Next-token prediction is all you need

    Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y ., Wang, J., Zhang, F., Wang, Y ., Li, Z., Yu, Q., et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869,

    arXiv

  15. [34]

    J., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J

    Wortsman, M., Liu, P. J., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322,

    arXiv

  16. [35]

    Liquid: Language models are scalable multi-modal generators

    Wu, J., Jiang, Y ., Ma, C., Liu, Y ., Zhao, H., Yuan, Z., Bai, S., and Bai, X. Liquid: Language models are scalable multi-modal generators. CoRR, abs/2412.04332, 2024a. Wu, J. Z., Ge, Y ., Wang, X., Lei, S. W., Gu, Y ., Shi, Y ., Hsu, W., Shan, Y ., Qie, X., and Shou, M. Z. Tu...

    arXiv

  17. [36]

    J., Wang, W., Lin, K

    Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang, W., Lin, K. Q., Gu, Y ., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528,

    arXiv

  18. [37]

    Openmoe: An early effort on open mixture- of-experts language models

    Xue, F., Zheng, Z., Fu, Y ., Ni, J., Zheng, Z., Zhou, W., and You, Y . Openmoe: An early effort on open mixture- of-experts language models. In ICML. OpenReview.net, 2024a. Xue, Z., Song, G., Guo, Q., Liu, B., Zong, Z., Liu, Y ., and Luo, P. Raphael: Text-to-image generation v...

    arXiv

  19. [38]

    X- vila: Cross-modality alignment for large language model

    Ye, H., Huang, D.-A., Lu, Y ., Yu, Z., Ping, W., Tao, A., Kautz, J., Han, S., Xu, D., Molchanov, P., et al. X- vila: Cross-modality alignment for large language model. arXiv preprint arXiv:2405.19335, 2024a. Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J....

    arXiv

  20. [40]

    Y ., Luong, T., Baid, G., Wang, Z., Va- sudevan, V ., Ku, A., Yang, Y ., Ayan, B

    Yu, J., Xu, Y ., Koh, J. Y ., Luong, T., Baid, G., Wang, Z., Va- sudevan, V ., Ku, A., Yang, Y ., Ayan, B. K., et al. Scaling autoregressive models for content-rich text-to-image gen- eration. arXiv preprint arXiv:2206.10789, 2(3):5,

    arXiv

  21. [41]

    B., Versari, L., Sohn, K., Minnen, D., Cheng, Y ., Gupta, A., Gu, X., Haupt- mann, A

    Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y ., Gupta, A., Gu, X., Haupt- mann, A. G., et al. Language model beats diffusion– tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737,

    arXiv

  22. [42]

    Learning to skip for language modeling

    Zeng, D., Du, N., Wang, T., Xu, Y ., Lei, T., Chen, Z., and Cui, C. Learning to skip for language modeling. CoRR, abs/2311.15436,

    arXiv

  23. [43]

    p-mod: Building mixture-of-depths mllms via progres- sive ratio decay

    Zhang, J., Meng, D., Qi, J., Huang, Z., Wu, T., and Wang, L. p-mod: Building mixture-of-depths mllms via progres- sive ratio decay. CoRR, abs/2412.04449, 2024a. 14 Submission and Formatting Instructions for ICML 2025 Zhang, L., Xiong, Y ., Yang, Z., Casas, S., Hu, R., and Urta...

    2025 arXiv

  24. [44]

    Minigpt-4: Enhancing vision-language understand- ing with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understand- ing with advanced large language models. CoRR, abs/2304.10592, 2023a. Zhu, J., Ding, X., Ge, Y ., Ge, Y ., Zhao, S., Zhao, H., Wang, X., and Shan, Y . VL-GPT: A generative p...

    2025 arXiv

  25. [45]

    dataset and large-scale text-image paired data to achieve effective text-image alignment. The third stage leverages high-quality data to develop generation capabilities, while the final two stages utilize the LLaV A dataset (Liu et al., 2024d) to enhance understanding capabili...

    2025

  26. [324]

    LastName, F

    Supplied as supplemental material fg324.pdf. LastName, F. Frobnication tutorial, 2014b. Supplied as supplemental material tr.pdf. Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L., Gao, J., et al. Multimodal foundation models: From specialists to general-purpose assistants...

    2025 arXiv

  27. [2004]

    M., Hauth, A., Millican, K., et al

    Anil, R., Borgeaud, S., Wu, Y ., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 1,

    arXiv

  28. [2006]

    and Salimans, T

    Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598,

    arXiv

  29. [2009]

    Dreamllm: Syn- ergistic multimodal comprehension and creation

    Dong, R., Han, C., Peng, Y ., Qi, Z., Ge, Z., Yang, J., Zhao, L., Sun, J., Zhou, H., Wei, H., et al. Dreamllm: Syn- ergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499,

    arXiv

  30. [2014]

    Y ., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y ., Baldridge, J., and Wu, Y

    Yu, J., Li, X., Koh, J. Y ., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y ., Baldridge, J., and Wu, Y . Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627,

    arXiv

  31. [2017]

    N., Kaiser,Ł., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser,Ł., and Polosukhin, I. Attention is all you need. NeurIPS, 30, 2017a. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser,Ł., and Polosukhin, I. Attention is a...

    2025 arXiv

  32. [2019]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de Las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M., Stock, P., Sub- ramanian, S., Yang, S., Antoniak, S., Scao,...

    arXiv

  33. [2020]

    A survey on mixture of experts

    Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, J. A survey on mixture of experts. CoRR, abs/2407.06204,

    arXiv

  34. [2021]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308.12966,

    arXiv

  35. [2022]

    T., Rubinstein, M., et al

    Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. Muse: Text-to-image genera- tion via masked generative transformers. arXiv preprint arXiv:2301.00704,

    arXiv

  36. [2023]

    Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference

    Del Corro, L., Del Giorno, A., Agarwal, S., Yu, B., Awadal- lah, A., and Mukherjee, S. Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference. arXiv preprint arXiv:2307.02628,

    arXiv

  37. [2024]

    MME: A comprehensive evaluation benchmark for mul- timodal large language models

    Fu, C., Chen, P., Shen, Y ., Qin, Y ., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., Li, K., Sun, X., and Ji, R. MME: A comprehensive evaluation benchmark for mul- timodal large language models. CoRR, abs/2306.13394,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.