Pith. sign in

REVIEW 3 major objections 4 minor 54 references

Training-Free Reasoning and Reflection in MLLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Merging task vectors layer by layer gives off-the-shelf multimodal models reasoning and reflection with no training.

desk verdict The per-layer merging direction is worth taking seriously, but the paper's closed-form derivation does not survive its own assumptions, so the method stands as an empirically motivated heuristic rather than a theoretically optimal one. read the letter →

arxiv 2505.16151 v1 pith:P4IPE6DT submitted 2025-05-22 cs.CV

Hongchen Wei , Zhenzhong Chen This is my paper
classification cs.CV
keywords training-freemodelmergingmultimodalreasoningtaskarithmeticlayer-wisefusionclosed-formweightsR1-stylereflectionmodalitypriorsMMMUbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FRANK proposes that reasoning and reflection can be injected into an existing multimodal LLM by merging it, layer by layer, with a language-only reasoning model that shares the same base weights—no gradient updates, no RL data, no extra supervision. The enabling observation is a functional split inside the decoder: shallow layers attend mostly to visual tokens and deep layers to text, so the vision and reasoning weight shifts occupy nearly orthogonal subspaces and can be superposed. The paper derives closed-form, data-free fusion weights from a Taylor expansion of the layer loss and an attention-fit exponential prior, then reports that the resulting 38B model reaches 69.2 on MMMU, beating its vision-only branch by 5.3 points and a proprietary frontier model. If correct, the result means time-consuming multimodal reasoning training can be replaced by a simple, interpretable weight operation.

What carries the argument

The machinery is the layer-wise task vector $\tau^{(l)}_t = \theta^{(l)}_t - \theta^{(l)}_0$, split into a vision branch and a reasoning branch, together with the closed-form fusion rule of Equation (13). The rule is derived by writing the fused layer as $\theta^{(l)}_0 + \lambda_V \tau^{(l)}_V + \lambda_R \tau^{(l)}_R$, expanding the loss gap through a second-order Taylor series, and then using NTK linearization (output changes linearly in small weight shifts), zero-gradient near-convergence, an isotropic Hessian approximation, and orthogonality between the two task vectors so all cross terms disappear. The final weights depend only on squared task-vector norms; an attention-guided exponential decay prior, $w^{(l)}_V = e^{-\alpha l} / \sum_j e^{-\alpha j}$ and $w^{(l)}_R = 1 - w^{(l)}_V$, is fit by linear regression on per-layer visual attention, enforcing the shallow-vision/deep-reasoning split. This combination turns model merging from a heuristic into a per-layer, data-free optimization.

What would settle it

Run the same merge on a pair of homologous models whose per-layer cosine similarity between vision and reasoning shifts is clearly above zero; if the closed-form weights (which are derived by dropping the cross terms) then fail to predict measured loss changes or benchmark accuracy, the orthogonality premise is falsified. A more direct check is to compare the predicted Taylor loss bound against the actual loss increase after merging.

Watch

Extended reading notes

Core claim

The central discovery is that reasoning and reflection can be grafted onto an existing multimodal LLM by superposing layer-wise task vectors, provided the vision-tuned and reasoning-tuned branches are homologous. The paper shows that shallow decoder layers of the MLLM put most of their attention on visual tokens, while deep layers attend to text, so the model already separates perception from cognition; this separation lets the fusion strengthen deep layers with reasoning while keeping shallow layers visually grounded. Under Neural Tangent Kernel linearization, a second-order Taylor expansion of the layer loss, and measured near-orthogonality of the two task vectors, the per-layer loss increase is bounded by the norms of the task vectors, and minimizing it yields $\lambda_t^{(l)} = \|\tau^{(l)}_t\|^2 / (\|\tau^{(l)}_V\|^2 + \|\tau^{(l)}_R\|^2)$. The authors then reweight these closed-form weights with attention-derived exponential priors. On five multimodal reasoning benchmarks, the resulting FRANK variants improve over their vision branches at 8B, 15B, and 38B, and the largest variant reaches 69.2 MMMU accuracy, above its 38B vision baseline and a proprietary frontier model.

Load-bearing premise

The entire fusion formula relies on the assumption that the two weight shifts are small, non-interfering additions: the loss surface is locally flat and uniform in every direction, and the vision and reasoning shifts are nearly perpendicular; the paper directly verifies only the perpendicular part, while borrowing the flat-and-uniform part from a single prior experiment on a different model.

Editorial extensions

If this is right

  • The same layer-wise recipe should apply to any pair of homologous vision- and reasoning-tuned models, so the method transfers across architectures without new datasets.
  • Larger models absorb the merged weights better: the 38B variant gains consistently on all five benchmarks, while the 8B and 15B variants lose ground on MathVista, showing the benefit is scale-dependent.
  • Perceptual accuracy is largely preserved: the attention-guided prior keeps MME scores close to the vision-only upper bound, and the no-prior ablation drops further, so the schedule is doing real work.
  • Reflection is observable in the merged model's outputs: explicit reflection tokens appear thousands of times where the vision-only branch emits none, and output length grows with task difficulty.
  • Because all weights are frozen after merging, the cost is inference-only, avoiding retraining and verifiable-reasoning-data bottlenecks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Pith inference: the norm-proportional fusion rule suggests a cheap screening test—if a candidate reasoning LLM's layer-wise task-vector norms vanish in the deep layers, the merge should not help; measuring the norm profile before merging could save experimentation.
  • Pith inference: the paper's reflection-token evidence is correlational; a stronger test would ablate or mask the reflection tokens and check whether final-answer accuracy changes, separating style from function.
  • Pith inference: the modality prior is fit on MSCOCO images, so an untested but natural extension is to fit the decay on the target benchmark's own images or on reasoning-heavy multimodal inputs; a shift in the fitted $\alpha$ would indicate the prior is data-dependent.
  • Pith inference: if the MMMU gains transfer to held-out domains, the training-free merging route could sidestep the scarcity of verifiable multimodal reasoning data, but the MathVista regressions at smaller scales suggest the transfer is not uniform across tasks and sizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes FRANK, a training-free method that equips off-the-shelf Multimodal LLMs (MLLMs) with reasoning and reflection by merging a vision-finetuned MLLM with a reasoning-specialized LLM. The method exploits an observed layer-wise functional specialization: shallow decoder layers attend more to visual tokens, while deeper layers attend to textual semantics. For each decoder layer, the paper defines a Layer-wise Average Loss Difference (LALD) objective, derives what it calls a Taylor-based closed-form fusion weight, and modulates the fusion with an attention-guided exponential decay prior. Three variants (8B, 15B, 38B) are evaluated on MMMU, MMMU-Pro, MathVista, MathVision, WeMath, and MME, with FRANK-38B reported to reach 69.2 on MMMU, surpassing InternVL2.5-38B by 5.3 points and GPT-4o by 0.1 points. The paper also reports reflection-token statistics, output-length analysis, and qualitative case studies. No code or per-layer fused weights are released.

Significance. If the empirical results were reliable, the idea of injecting R1-style reasoning into existing MLLMs through layer-wise task-vector merging would be practically valuable and would extend model-merging literature to the multimodal setting. The paper has several genuine strengths: it identifies and measures a layer-wise visual-attention hierarchy in multiple MLLMs, it includes ablations of the modality prior, it reports orthogonality measurements of task vectors, and it evaluates across three model scales. These elements are useful even if the theoretical derivation is revised. However, the central closed-form derivation is internally inconsistent, the reported perceptual results are partly misdescribed, and the headline empirical claims lack variance estimates. The significance is therefore conditional on substantial revision and on verification of the actual fusion schedule used.

major comments (3)
  1. [Section 3.2.2, Eqs. (8)-(15) and Appendix A.8] Eq. (13) is not the minimizer of the LALD bound stated in Eq. (12). From the paper's own Eq. (8) with the constraint lambda_V + lambda_R = 1, one obtains h_V = lambda_R (tau_R - tau_V) and h_R = lambda_V (tau_V - tau_R); under the stated orthogonality and equal-curvature assumptions, Eq. (12) reduces to LALD = (delta/2)(||tau_V||^2 + ||tau_R||^2)((1-lambda_V)^2 + lambda_V^2), whose unique minimizer is lambda_V = 1/2, independent of the task-vector norms. The norm-proportional formula in Eq. (13) therefore solves a different, unstated norm-weighted objective, not the LALD objective. Eq. (15) is likewise not the solution of Eq. (14): with the same assumptions and the constraint lambda_V + lambda_R = 1, the minimizer of the weighted objective is lambda_V = w_V/(w_V + w_R), and as written Eq. (15) does not even force lambda_V + lambda_R = 1, so the merged model need not lie on the task-arithmetic interpolation path. The 'Taylor-derived closed-form' support for the layer-wise schedule thus collapses; the reported MMMU gains are not tied to the stated derivation.
  2. [Tables 1-2 and Section 4.3] The empirical presentation overstates the visual-preservation claim and the consistency of the gains. On MME (Table 2), FRANK-15B drops from 86.7 to 60.0 on Position and from 67.5 to 60.0 on Text Translation relative to NVIL-15B; describing this as 'very slight drops' is contradicted by the table and weakens the central claim that the modality prior preserves visual grounding. On MathVista (Table 1), FRANK-8B and FRANK-15B are 7.7 and 12.2 points below their respective vision branches, which is acknowledged in the per-dataset analysis but inconsistent with the conclusion that the method 'consistently outperforms' baselines at all scales. In addition, no error bars, confidence intervals, or repeated runs are reported for any benchmark; with a single run, the headline MMMU gap of 69.2 vs 69.1 over GPT-4o is not a meaningful superiority claim.
  3. [Section 3.2.2 and Appendices A.5-A.6] The theoretical derivation also rests on assumptions that are not validated for the models used. NTK linearization is inherited from a single cited experiment on LLaMA-2-7B (MetaGPT, [29]); the isotropic-Hessian and uniform-curvature assumptions (delta_V approximately delta_R) are asserted rather than measured for Qwen2.5-14B/32B or LLaMA-3.1-8B. Only task-vector orthogonality is directly measured (Figures 3, 9, 11). Since the algebra in Eq. (13) fails even when all these assumptions are granted, the issue is not merely about plausible idealizations. Moreover, no code or per-layer fusion weights are released, so it is impossible to verify which schedule was actually executed in Tables 1-2; given the discrepancy between Eqs. (13)/(15) and the stated objective, this is a concrete reproducibility gap.
minor comments (4)
  1. [Tables 1-2] Table 1 contains the typo 'Samll' (should be 'Small'), and Table 2's 'Poseter' should be 'Poster'.
  2. [Eq. (12)] The inner sum in Eq. (12) is written as sum over k,t without specifying k != t; the intended exclusion should be made explicit, since this ambiguity contributes to the incorrect optimization leading to Eq. (13).
  3. [Section 3.2.2 and Section 3.2.3] The fusion weights are described as 'data-agnostic' while the modality prior is fitted on 1000 MSCOCO images; the text should qualify this as task-label-free rather than data-free.
  4. [Figure 5 and reference [53]] The spelling 'Idecifics3-8B' in Figure 5 should be standardized to 'Idefics3-8B', and the MME benchmark should be cited to its original dataset paper rather than to survey reference [53].

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: fusion weights and attention prior are not fit to evaluation labels; the flawed closed-form derivation is a correctness concern, not a self-referential reduction.

full rationale

FRANK's evaluation scores on MMMU, MathVista, etc. are external to the method's inputs: the layer-wise fusion weights come from task-vector norms (Eq. 13) and an attention-derived exponential prior (Eq. 16), neither tuned to benchmark labels. The modality prior is fit to 1000 unlabeled MSCOCO images (Sec. 4.1, Appendix A.9), so the benchmark gains are not forced by construction. The NTK linearization assumption is inherited from the external MetaGPT citation [29] with an empirical check on LLaMA-2-7B, and task-vector orthogonality is directly measured in Figures 3, 9, and 11; no load-bearing self-citation chain exists. The theoretical derivation of Eq. 13 from Eq. 12 is internally problematic: under the paper's own equal-curvature and orthogonality assumptions, the LALD bound is minimized at lambda = 1/2, not at the norm-proportional ratio, and the stationarity equation in Appendix A.8 (Eq. 36) is not satisfied by the proposed Eq. 37. This is a mathematical correctness gap, not circularity, because Eq. 13 is not made equivalent to the objective by definition; it simply fails the stated minimization. Hence the circularity score is 0.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The closed-form fusion rests on six assumptions about parameter geometry and loss curvature; only task-vector orthogonality is directly measured (for three variants), while NTK linearity, near-convergence, isotropy, and uniform curvature are inherited from theory or asserted. Two additional parameters (alpha, C) are fit to attention data from 1000 images.

free parameters (2)
  • alpha (visual attention decay rate) = not reported
    Fitted by least squares on log per-layer visual attention ratios from 1000 MSCOCO images (Section 4.1, A.9); sets the layer-wise prior w_V(l).
  • C (attention normalization constant) = not reported
    Fitted jointly with alpha in the model a_l approx C exp(-alpha l); normalizes the attention profile.
assumptions (6)
  • domain assumption The fused models are homologous: the MLLM and the reasoning LLM are task-fine-tuned variants of the same base model, so their task vectors can be added.
    Required for Eq. (2) and (3); the paper selects models sharing a base (Table 5) but this limits generality.
  • domain assumption NTK linearization: small weight perturbations change outputs approximately linearly during fine-tuning, so the loss landscape is locally quadratic.
    Invoked in Section 3.2.2 and A.5; supported only by a citation to MetaGPT's LLaMA-2-7B validation, not by experiments on the 8B/14B/32B backbones used here.
  • domain assumption Task-vector orthogonality: the vision and reasoning task vectors at each layer lie in nearly orthogonal subspaces.
    Verified by cosine similarity for three variants (Figures 3, 9, 11), but treated as a general property in the derivation.
  • domain assumption Near-convergence: gradients of the task loss vanish at the fine-tuned parameters.
    Used to drop the first-order Taylor term in Eq. (9); not demonstrated for the fine-tuned models used.
  • ad hoc to paper The per-layer Hessian is isotropic: it is proportional to the identity matrix.
    Introduced in A.5 to turn the quadratic form into a scalar times a norm; no direct evidence supports isotropy for these transformers.
  • ad hoc to paper The curvature scalar is uniform across branches: delta_V approx delta_R.
    Introduced in A.8 (Eq. 38) to eliminate delta from the final closed form; this is what makes lambda proportional to the squared task-vector norm, but it is not empirically checked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training-Free Reasoning and Reflection in MLLMs." pith.science (2026). https://pith.science/paper/P4IPE6DT

@misc{pith2026250516151,
  author       = {Pith},
  title        = {Pith review of: Training-Free Reasoning and Reflection in MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P4IPE6DT}},
  note         = {Machine review of arXiv:2505.16151}
}
read the original abstract

Recent advances in Reasoning LLMs (e.g., DeepSeek-R1 and OpenAI-o1) have showcased impressive reasoning capabilities via reinforcement learning. However, extending these capabilities to Multimodal LLMs (MLLMs) is hampered by the prohibitive costs of retraining and the scarcity of high-quality, verifiable multimodal reasoning datasets. This paper introduces FRANK Model, a training-FRee ANd r1-liKe MLLM that imbues off-the-shelf MLLMs with reasoning and reflection abilities, without any gradient updates or extra supervision. Our key insight is to decouple perception and reasoning across MLLM decoder layers. Specifically, we observe that compared to the deeper decoder layers, the shallow decoder layers allocate more attention to visual tokens, while the deeper decoder layers concentrate on textual semantics. This observation motivates a hierarchical weight merging approach that combines a visual-pretrained MLLM with a reasoning-specialized LLM. To this end, we propose a layer-wise, Taylor-derived closed-form fusion mechanism that integrates reasoning capacity into deep decoder layers while preserving visual grounding in shallow decoder layers. Extensive experiments on challenging multimodal reasoning benchmarks demonstrate the effectiveness of our approach. On the MMMU benchmark, our model FRANK-38B achieves an accuracy of 69.2, outperforming the strongest baseline InternVL2.5-38B by +5.3, and even surpasses the proprietary GPT-4o model. Our project homepage is at: http://iip.whu.edu.cn/frank/index.html

Figures

Figures reproduced from arXiv: 2505.16151 by the authors.

Figure 1
Figure 1. Non-reasoning MLLMs lack reasoning and reflection abilities, while reasoning LLMs are unable to perceive visual [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Layer-wise visual attention of NVIL-15B. Each curve [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Average output length of the FRANK on the MMMU [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Output examples from FRANK-8B and the non-reasoning baseline model Idecifics3-8B. Here, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Output examples from FRANK-15B and the non-reasoning baseline model NVIL-15B. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Output examples from FRANK-38B and the non-reasoning baseline model InternVL2.5-38B. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 10
Figure 10. Figure 10: Layer-wise visual attention of InternVL2.5-38B. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 9
Figure 9. Figure 9: Cosine similarity between task vectors of [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 33 canonical work pages

  1. [29]

    Metagpt: Merging large language models using model exclusive task arithmetic

    Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. Metagpt: Merging large language models using model exclusive task arithmetic. In EMNLP, pages 1711– 1724, Miami, FL, USA, 2024

    work page 2024

  2. [1]

    DeepSeek-R1: Incentivizing reason- ing capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reason- ing capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  3. [2]

    Kimi K1.5: Scaling reinforce- ment learning with llms

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi K1.5: Scaling reinforce- ment learning with llms. arXiv preprint arXiv:2501.12599, 2025

    arXiv 2025

  4. [3]

    S 2 R: Teaching llms to self-verify and self-correct via reinforcement learning

    Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S 2 R: Teaching llms to self-verify and self-correct via reinforcement learning. arXiv preprint arXiv:2502.12853, 2025

    arXiv 2025

  5. [4]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schul- man, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In ICLR, 2024

    work page 2024

  6. [5]

    OpenAI o1

    OpenAI. OpenAI o1. https: //openai.com/o1/, 2024

    work page 2024

  7. [6]

    LLaV A-o1: Let vision language models reason step-by-step

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. LLaV A-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440 , 2024.7 Question: Janet Ludlow's firm requires all its analysts to use a two-stage dividend discount model (DDM) and the capital asset pricing model (CAPM) to value stocks. Using the CAPM and DDM, L...

    arXiv 2024

  8. [7]

    Insight-V: Exploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-V: Exploring long-chain visual reasoning with multimodal large language models. In CVPR, 2025

    work page 2025

Show all 54 references
  1. [8]

    Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319, 2024

    2024 arXiv

  2. [9]

    Visual- RFT: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual- RFT: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  3. [10]

    Vision- R1: Incentivizing reasoning capability in multimodal large language models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision- R1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  4. [11]

    LMM-R1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. LMM-R1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025

    2025 arXiv

  5. [12]

    R1-VL: Learning to reason with multimodal large language models via step-wise group relative policy optimization

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-VL: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937, 2025

    2025 arXiv

  6. [13]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy with...

    2022

  7. [14]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023

    2023

  8. [15]

    Composing parameter-efficient modules with arithmetic operation

    Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operation. In NeurIPS, New Orleans, LA, USA, 2023

    2023

  9. [16]

    Gradual progression from sensory to task-related processing in cerebral cortex

    Scott L Brincat, Markus Siegel, Constantin von Nicolai, and Earl K Miller. Gradual progression from sensory to task-related processing in cerebral cortex. Proceedings of the National Academy of Sciences, 115(30):E7202–E7211, 2018

    2018

  10. [17]

    Hierarchical processing of visual and language information in the brain

    Haruka Kawasaki, Satoshi Nishida, and Ichiro Kobayashi. Hierarchical processing of visual and language information in the brain. In AACL-IJCNLP, pages 405–410, 2022

    2022

  11. [18]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In NeurIPS, 2023

    2023

  12. [19]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  13. [20]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, New Orleans, LA, USA, 2023

    2023

  14. [21]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In ICLR, Vienna, Austria, 2024

    2024

  15. [22]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reason- ing in large language models. In NeurIPS, New Orleans, LA, USA, 2022

    2022

  16. [23]

    Improve vision lan- guage model chain-of-thought reasoning

    Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruom- ing Pang, and Yiming Yang. Improve vision lan- guage model chain-of-thought reasoning. arXiv preprint arXiv:2410.16198, 2024

    2024 arXiv

  17. [24]

    Compositional chain-of-thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. In CVPR, pages 14420– 14431, Seattle, W A, USA, 2024

    2024

  18. [25]

    TextCoT: Zoom in for en- hanced multimodal text-rich image understanding

    Bozhi Luan, Hao Feng, Hong Chen, Yonghui Wang, Wen- gang Zhou, and Houqiang Li. TextCoT: Zoom in for en- hanced multimodal text-rich image understanding. arXiv preprint arXiv:2404.09797, 2024

    2024 arXiv

  19. [26]

    MM-Eureka: Exploring the frontiers of multimodal reasoning with rule-based re- inforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. MM-Eureka: Exploring the frontiers of multimodal reasoning with rule-based re- inforcement learning. arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  20. [27]

    Merging models with fisher-weighted averaging

    Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. In NeurIPS, New Orleans, LA, USA, 2022

    2022

  21. [28]

    Raffel, and Mohit Bansal

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving inter- ference when merging models. In NeurIPS, New Orleans, LA, USA, 2023

    2023

  22. [30]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxi- ang Cheng. Dataless knowledge fusion by merging weights of language models. In ICLR, Kigali, Rwanda, 2023

    2023

  23. [31]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InICML, Vienna, Austria, 2024

    2024

  24. [32]

    Bring reason to vision: Understanding perception and reasoning through model merging

    Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, and Junxian He. Bring reason to vision: Understanding perception and reasoning through model merging. arXiv preprint arXiv:2505.05464, 2025

    2025 arXiv

  25. [33]

    Neu- ral Tangent Kernel: Convergence and generalization in10 neural networks

    Arthur Jacot, Clément Hongler, and Franck Gabriel. Neu- ral Tangent Kernel: Convergence and generalization in10 neural networks. In NeurIPS, pages 8580–8589, Montreal, Canada, 2018

    2018

  26. [34]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  27. [35]

    AGIEval: A human-centric benchmark for evaluating foundation models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In NAACL, pages 2299– 2314, Mexico City, Mexico, 2024

    2024

  28. [36]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26286–26296, Seattle, W A, USA, 2024

    2024

  29. [37]

    Llava- next

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuan- han Zhang, Sheng Shen, and Yong Jae Lee. Llava- next. https://llava-vl.github.io/blog/2024-01-30-llava-next/, 2024

    2024

  30. [38]

    VILA: on pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. VILA: on pre-training for visual language models. In CVPR, pages 26679–26689, Seattle, W A, USA, 2024

    2024

  31. [39]

    Building and better understanding vision- language models: insights and future directions

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision- language models: insights and future directions. In Work- shop on RBFM, 2024

    2024

  32. [40]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, volume 15075, pages 370–387, Milan, Italy, 2024

    2024

  33. [41]

    Nvila: E fficient frontier visual language models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yux- ian Gu, Dacheng Li, et al. Nvila: E fficient frontier visual language models. arXiv preprint arXiv:2412.04468, 2024

    2024 arXiv

  34. [42]

    LLaV A-OneVision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaV A-OneVision: Easy visual task transfer. TMLR, 2024

    2024

  35. [43]

    Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  36. [44]

    OpenAI. GPT-4o. https://openai.com/index/hello-gpt-4o/, 2024

    2024

  37. [45]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  38. [46]

    MMMU: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert AGI

    Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A m...

    2024

  39. [47]

    MMMU-Pro: A more ro- bust multi-discipline multimodal understanding bench- mark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. MMMU-Pro: A more ro- bust multi-discipline multimodal understanding bench- mark. arXiv preprint arXiv:2409.02813, 2024

    2024 arXiv

  40. [48]

    MathVista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun- yuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, Vienna, Austria, 2024

    2024

  41. [49]

    Measuring multimodal mathematical reasoning with math- vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houx- ing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math- vision dataset. In NeurIPS, Vancouver, BC, Canada, 2024

    2024

  42. [50]

    We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024

    Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024

    2024 arXiv

  43. [51]

    Qwen. Qwq. https: //qwenlm.github.io/blog/qwq-32b- preview/, 2024

    2024

  44. [52]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, volume 8693, pages 740–755, Zurich, Switzerland, 2014

    2014

  45. [53]

    Non-Reasoning MLLM

    Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In CAICE, pages 405–409, 2024. 11 A A ppendix A.1 Implementation Details As shown in Table 5, we summarize the precise fusion config- urations used ...

    2024

  46. [54]

    the Hessian∇2 f (l) shrinks asO(1/ √ width) (a direct conse- quence of NTK’s linearization effect). Under NTK conditions: ∇2L(l) t (θ(l))≈∇ f (l)(xt;θ(l) 0 )∇ f (l)(xt;θ(l) 0 )⊤ (29) Taking the isotropic approximation for layers: ∇2L(l) t ≈δ(l) t Idl, δ (l) t = 1 dl tr ∇ f (l)...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.