Pith. sign in

REVIEW 51 references

VLM accuracy can be predicted from a scalar capability score derived from LLM text benchmarks plus multimodal data volume via a fitted transfer-absorption scaling law.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:45 UTC pith:DAZQVSOR

arxiv 2608.00013 v1 pith:DAZQVSOR submitted 2026-06-24 cs.CL cs.AIcs.CV

What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs

Ziran Li , Qiang Wang , Zhengyu Chen , Shanglin Lei , Borun Chen , Jingang Wang , Xunliang Cai This is my paper
classification cs.CL cs.AIcs.CV
keywords modelfamiliesframeworkscalingtextualbenchmarkscapabilitymultimodal
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The authors trained over 150 vision-language models by combining 34 different open-source text-only language models with the same vision encoder, projector, and training data. They then measured each language model on dozens of text benchmarks and used principal component analysis to compress those scores into a single "capability score" S. Their proposed scaling law is P = A*S + (B0 - Bm*S)*ln(D) + P0, where P is average VLM accuracy, D is multimodal training tokens, and A, B0, Bm, P0 are constants fitted to the training runs. The formula says a stronger text backbone helps at the start, but stronger backbones absorb less benefit from additional multimodal data.

The paper validates the formula by predicting the performance of Qwen2.5-72B, a backbone much larger than any used in fitting, and by leaving entire model families out of the fit. It also reports that some text benchmarks seem to hurt VLM performance ("transfer tax") and that base models are more data-efficient VLM backbones than instruction-tuned ones.

The empirical effort is large and the cross-family results are suggestive. But the law is a fitted linear model rather than a derivation, the out-of-sample evidence is thin (two 72B models; a few families), and some claims in the abstract do not match the tables. The GitHub link is provided but no commit hash or raw trajectories are in the paper.

Core claim

Eq. 7 is the load-bearing assertion: P = A_hat*S + (B0 - Bm*S)*ln(D_mm) + P0, with fitted constants, predicts VLM benchmark accuracy from a scalar capability score S extracted from LLM textual benchmarks, and generalizes to 72B-scale backbones and held-out model families. If true, practitioners can predict VLM performance and optimal hyperparameters before training.

Load-bearing premise

That a single scalar S = w^T (top-K PCA components of the text-benchmark matrix) is a sufficient cross-family proxy for the LLM properties that determine multimodal transfer, so that e^S can stand in for parameter count N in Eq. 6 and P is linear in S and ln(D_mm). This enters at Eqs. 3-5 and is not derived; if the PCA basis estimated on training families does not capture the capability axes of held-out families or 72B models, the framework's extrapolations fail.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The central claim depends on fitted constants (w, A_hat, B0, Bm, P0) and on the assumption that PCA of text benchmarks captures the LLM properties relevant to multimodal transfer. No new physical entities are introduced.

free parameters (6)
  • A_hat (transfer rate) = not reported for global fit; type/family fits 0.212-0.361 (Tables 2, 3)
    Linear coefficient of S in Eq. 7; fitted jointly with w via Algorithm 1.
  • B0 (baseline absorption rate) = Base 1.60e-2, Chat 1.49e-2 (Table 2); family 0.28e-2 to 2.46e-2 (Table 3)
    Intercept of the ln(D_mm) coefficient in Eq. 8; fitted.
  • Bm (absorption decay rate) = Base 0.78e-2, Chat 1.04e-2 (Table 2); family up to 4.55e-2 (Table 3)
    Interaction coefficient on S * ln(D_mm); fitted.
  • P0 (benchmark baseline) = not reported
    Additive offset in Eq. 7; fitted.
  • w (capability aggregation weights) = not reported
    Projection of K PCA components onto scalar S; jointly optimized in Algorithm 1 and used to define transfer coefficients lambda_i.
  • K (number of PCA components) = K=3 for loss fit; not stated for Eq. 7
    Number of components chosen to explain >=95% variance in X; a modeling choice.
axioms (5)
  • domain assumption Benchmark-model matrix X has low-rank structure (Eq. 3)
    Imported from Ruan et al. 2024; needed for PCA to yield a meaningful capability vector.
  • ad hoc to paper S ∝ log C_text (Eq. 5)
    Follows from Eq. 3 only if the linear projection does not cancel the log-compute relation; used to justify substituting e^S for N in Eq. 6.
  • ad hoc to paper P is linear in S and ln(D_mm) with interaction (Eq. 7)
    Phenomenological ansatz, not derived from Eq. 6 or from a learning-theoretic model.
  • domain assumption Power-law data term B/D^β_mm holds for multimodal training
    Borrowed from Kaplan/Hoffmann text scaling laws; assumed without re-derivation for VLM training.
  • domain assumption Unified LLaVA-OneVision recipe with frozen SigLIP is a representative base case
    The Limitations section explicitly restricts the framework to late-fusion, frozen-encoder settings.
invented entities (1)
  • Capability score S (latent vector S_m) independent evidence
    purpose: Compress many text benchmark scores into a scalar predictor of VLM accuracy
    S_new is computable from a new LLM's text benchmarks and yields falsifiable VLM accuracy predictions, but the projection w and PCA basis are fit on the training models, so it is not independently measured.

pith-pipeline@v1.3.0-alltime-deepseek · 24610 in / 17267 out tokens · 148874 ms · 2026-08-04T01:45:05.121111+00:00 · methodology

0 comments
read the original abstract

Choosing the right large language model (LLM) backbone is the most consequential decision when building a vision-language model (VLM), yet it remains fundamentally unprincipled: compute-based scaling laws fail to generalize across model families, and no framework exists for directly predicting VLM performance before training begins. We propose the Capability-Driven Multimodal Scaling Law, the first cross-family framework that predicts VLM benchmark accuracy from directly observable textual capability. Given a low-dimensional capability score $S$ extracted from LLM textual benchmarks via PCA, we model VLM performance as a function of $S$, with a per-backbone transfer rate and an absorption rate that quantifies data-scaling efficiency. To fit and validate the framework, we train over 150 VLMs on 34 LLMs spanning 7 model families under a strictly controlled recipe. Evaluations on more than 200 textual and 50 multimodal benchmarks show that the law accurately extrapolates transfer rate from models up to 8B parameters to 72B-scale backbones, predicts full VLM training trajectories with high fidelity, and generalizes to entirely held-out model families. Beyond the scaling law, our analysis surfaces actionable insights: certain textual benchmarks negatively correlate with multimodal performance, exposing latent benchmark-gaming behavior; base LLMs outperform instruction-tuned counterparts as VLM backbones due to higher absorption rates and lower data-scaling decay; and different model families occupy distinct positions in the transfer--absorption space. The framework turns backbone selection from costly empirical sweeps into a principled, quantitative decision. Code and data are available at https://github.com/wangq-dev/CDMScaling.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 32 linked inside Pith

  1. [1]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  2. [2]

    B Related Works Vision Language Models.Driven by rapid advancements in LLMs [Grattafiori et al., 2024, Yang et al., 2025, Team et al., 2025], VLMs [Li et al., 2026, Wang et al., 2025a, Wu et al., 2024, Guo et al., 2025] have achieved remarkable progress in recent years. Built upon pretrained LLMs, mainstream VLM architectures such as Qwen3-VL [Li et al., ...

    2024

  3. [4]

    BB: BIG-Bench [Srivastava et al., 2022]; BBH: BIG-Bench Hard [Suzgun et al., 2022]; MMLU: Massive Multitask Language Understand- ing [Hendrycks et al., 2021b]

    19 Capability-Driven Multimodal Scaling Law Table 5 | Textual benchmarks used in this study, grouped by capability dimension. BB: BIG-Bench [Srivastava et al., 2022]; BBH: BIG-Bench Hard [Suzgun et al., 2022]; MMLU: Massive Multitask Language Understand- ing [Hendrycks et al., 2021b]. Category Benchmark Source Information Extraction CrossNER [Liu et al., ...

    2022

  4. [6]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al

    URLhttps://arxiv.org/abs/2412.15115. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  5. [7]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  6. [8]

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al

    URLhttps://arxiv.org/abs/2408.00118. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954,

    Pith/arXiv arXiv

  7. [10]

    Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data.arXiv preprint arXiv:2410.18558,

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data.arXiv preprint arXiv:2410.18558,

    Pith/arXiv arXiv

  8. [11]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290, 2024a

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290, 2024a. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty,...

    Pith/arXiv arXiv

  9. [12]

    Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261,

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261,

    Pith/arXiv arXiv

  10. [14]

    URLhttps://arxiv.org/abs/2311. 07911. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024b. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Z...

    Pith/arXiv arXiv

  11. [16]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Pith/arXiv arXiv

  12. [17]

    Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking.arXiv preprint arXiv:2601.04720,

    Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, et al. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking.arXiv preprint arXiv:2601.04720,

    Pith/arXiv arXiv

  13. [18]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302,

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302,

    Pith/arXiv arXiv

  14. [19]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023a

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023a. DeyaoZhu, JunChen, XiaoqianShen, XiangLi, andMohamedElhoseiny. Minigpt-4: Enhancingvision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592,

    Pith/arXiv arXiv

  15. [20]

    Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4.arXiv preprint arXiv:2308.12067,

    Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4.arXiv preprint arXiv:2308.12067,

    Pith/arXiv arXiv

  16. [21]

    Language models scale reliably with over-training and on downstream tasks.arXiv preprint arXiv:2403.08540,

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks.arXiv preprint arXiv:2403.08540,

    Pith/arXiv arXiv

  17. [22]

    Scaling laws for downstream task performance of large language models

    Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. InICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models,

    2024

  18. [23]

    Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.arXiv preprint arXiv:2401.00448,

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.arXiv preprint arXiv:2401.00448,

    Pith/arXiv arXiv

  19. [24]

    Scaling laws for native multimodal models.arXiv preprint arXiv:2504.07951,

    Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, and Alaaeldin El-Nouby. Scaling laws for native multimodal models.arXiv preprint arXiv:2504.07951,

    Pith/arXiv arXiv

  20. [25]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.arXiv preprint arXiv:2206.04615,

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.arXiv preprint arXiv:2206.04615,

    Pith/arXiv arXiv

  21. [27]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.arXiv preprint arXiv:2007.08124,

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.arXiv preprint arXiv:2007.08124,

    Pith/arXiv arXiv 2007

  22. [28]

    Proofwriter: Generating implications, proofs, and abductive statements over natural language

    Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. InFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3621–3634,

    2021

  23. [29]

    Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023a

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023a. Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench-2: Benchmarking multimodal large language models.arXiv preprint arXiv:2311.17...

    Pith/arXiv arXiv

  24. [30]

    Task me anything.arXiv preprint arXiv:2406.11775, 2024a

    Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything.arXiv preprint arXiv:2406.11775, 2024a. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world know...

    Pith/arXiv arXiv

  25. [31]

    Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models

    14 Capability-Driven Multimodal Scaling Law Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7907–79...

    2025

  26. [32]

    V*: Guided visual search as a core mechanism in multimodal llms.arXiv preprint arXiv:2312.14135,

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms.arXiv preprint arXiv:2312.14135,

    Pith/arXiv arXiv

  27. [33]

    Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehension

    Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, et al. Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehension. InAI for Accelerated Materials Design-Vienna 2024, 2024d. Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynam...

    2024

  28. [34]

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan

    URL https://arxiv.org/abs/2411.00836. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. InThe 36th Conference on Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv

  29. [35]

    Tablevqa-bench: A visual question answering benchmark on multiple table domains.arXiv preprint arXiv:2404.19205,

    Yoonsik Kim, Moonbin Yim, and Ka Yeon Song. Tablevqa-bench: A visual question answering benchmark on multiple table domains.arXiv preprint arXiv:2404.19205,

    Pith/arXiv arXiv

  30. [37]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023b. YifanLi, YifanDu, KunZhou, JinpengWang, WayneXinZhao, andJi-RongWen. Evaluatingobjecthallucination in large vision-language models. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023c. URLhttps://openreview.net/forum?id=xozJw0...

    Pith/arXiv arXiv 2023

  31. [38]

    An llm-free multi-dimensional benchmark for mllms hallucination evaluation.arXiv preprint arXiv:2311.07397,

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free multi-dimensional benchmark for mllms hallucination evaluation.arXiv preprint arXiv:2311.07397,

    Pith/arXiv arXiv

  32. [39]

    Gaussian error linear units (gelus).arXiv preprint arXiv:1606.08415,

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus).arXiv preprint arXiv:1606.08415,

    Pith/arXiv arXiv

  33. [40]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024d. Bo Li, Hao Zhang, Kaichen Zhang, Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. Llava-next: What else influence...

    2024

  34. [44]

    The fitting procedure is detailed in Appendix D. 0.0 0.2 0.4 0.6 0.8 1.0 Number of Tokens 1e11 0.70 0.75 0.80 0.85 0.90 0.95 1.00 1.05 1.10 Training Loss Falcon3-1B-Base (Measured) Falcon3-3B-Base (Measured) Falcon3-7B-Base (Measured) Falcon3-10B-Base (Measured) Falcon3-1B-Base (Fitted) Falcon3-3B-Base (Fitted) Falcon3-7B-Base (Fitted) Falcon3-10B-Base (F...

    2022

  35. [46]

    study over-training effects on task-specific metrics, while others investigate scaling laws for translation and agentic benchmarks [Isik et al., 2024, Ruan et al., 2024]. Beyond training, inference- time scaling [Sardana et al., 2023] reveals critical trade-offs: for high inference demand, models should be over-trained beyond Chinchilla-optimal points usi...

    2024

  36. [47]

    In parallel, recent work on native multimodal models has focused on early-fusion architectures [Shukor et al., 2025]

    examined multimodal models that tokenize text, speech, and image modalities into discrete tokens for unified generation. In parallel, recent work on native multimodal models has focused on early-fusion architectures [Shukor et al., 2025]. In contrast, we systematically investigate late-fusion models that process image-text inputs and generate text outputs...

    2025

  37. [49]

    20 Capability-Driven Multimodal Scaling Law Table 6 | Multimodal benchmarks used in this study, grouped by capability dimension. Category Benchmark General VQA MMBench_DEV_EN_V11 [Liu et al., 2024b] MMStar [Chen et al., 2024] SEEDBench_IMG [Li et al., 2023a] SEEDBench2 [Li et al., 2023b] SEEDBench2_Plus [Li et al., 2024c] MME [Fu et al., 2026] TaskMeAnyth...

    2024

  38. [50]

    Specifically, the vision tower employs SigLIP [Zhai et al., 2023] (∼400M parameters) to extract visual features from input images

    Model Architecture.All models are built upon the LLaVA-OneVision architecture [Li et al., 2024b], comprising a vision tower, a projector, and a language tower. Specifically, the vision tower employs SigLIP [Zhai et al., 2023] (∼400M parameters) to extract visual features from input images. A two-layer MLP with GELU activation [Hendrycks and Gimpel, 2016, ...

    2023

  39. [51]

    The objective is to jointly learn the capability aggregation weightsw and the scaling coefficients { ˆA, B0, Bm, P0}fromMLLM–VLM training trajectories. Algorithm Walkthrough Step 1 — Latent Capability Extraction.To prevent overfitting on high-dimensional text benchmarks [Ruan et al., 2024], we apply PCA to the benchmark-model matrixX. We extract a low-dim...

    2024

  40. [1964]

    doi:10.1214/aoms/1177703732

    ISSN 0003-4851, 2168-8990. doi:10.1214/aoms/1177703732. URL https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-35/issue-1/ Robust-Estimation-of-a-Location-Parameter/10.1214/aoms/1177703732.full. Jorge Nocedal. Updating Quasi-Newton Matrices with Limited Storage.Mathematics of Computation, 35(151): 773–782,

    arXiv

  41. [1980]

    doi:10.2307/2006193

    ISSN 0025-5718. doi:10.2307/2006193. URLhttps://www.jstor.org/stable/2006193. Rainer Storn. Differrential evolution-a simple and efficient adaptive scheme for global optimization over continuous spaces.Technical report, International Computer Science Institute, 11,

    arXiv

  42. [1995]

    To empirically validate this, we collect and analyze the evaluation results of 17 representative VLMs and their corresponding LLMs from the OpenCompass leaderboard

    A Preliminary Observations A.1 Correlation Between Textual and Multimodal Capabilities Intuitively, a stronger LLM backbone should yield a more capable VLM. To empirically validate this, we collect and analyze the evaluation results of 17 representative VLMs and their corresponding LLMs from the OpenCompass leaderboard. Figure 6 illustrates the correlatio...

    2023

  43. [2016]

    Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12), December 2024c

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12), December 2024c. ISSN 1869-1919. doi:10.1007/s11432-024-4235-6. URLhttp://dx.doi.org/10.1007/s11432-024-4235-6. Weihao...

    doi:10.1007/s11432-024-4235-6 1919

  44. [2019]

    Mia-bench: Towards better instruction following evaluation of multimodal llms

    Yusu Qian, Hanrong Ye, Jean-Philippe Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. Mia-bench: Towards better instruction following evaluation of multimodal llms. InInternational Conference on Learning Representations, volume 2025, pages 35145–35165,

    2025

  45. [2020]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Pith/arXiv arXiv

  46. [2021]

    Wic: the word-in-context dataset for evaluating context-sensitive meaning representations

    Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267–1273,

    2019

  47. [2022]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a

    10 Capability-Driven Multimodal Scaling Law Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URLhttps://llava-vl.github.io/blog/ 2024-01-30-llava-next/. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, ...

    Pith/arXiv arXiv 2024

  48. [2023]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al

    URLhttps://arxiv.org/abs/2310.06825. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024b. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- trainin...

    Pith/arXiv arXiv

  49. [2024]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report.arXiv preprint arXiv:2505.07062,

    Pith/arXiv arXiv

  50. [2025]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025a. Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei L...

    Pith/arXiv arXiv 2024

  51. [2026]

    Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica

    URL https://arxiv.org/abs/2605.00674. Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InInternational Conference on Learning Representations, volume 2025, pages 58791–58831,

    Pith/arXiv arXiv 2025