Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fusing math and code MLLMs with dual gates beats task arithmetic.

desk verdict A plausible new parameter-fusion recipe, but the learnable gate is never specified and the evidence is thin; deserves a serious referee but needs major revision. read the letter →

arxiv 2506.23940 v2 pith:XJFF3DD5 submitted 2025-06-30 cs.CL

Yang Dai , Jianxiang An , Tianwei Lin , Hongyang He , Hongzhe Huang , Wenqiao Zhang , Zheqi Lv , Siliang Tang
show 1 more author
Yueting Zhuang
This is my paper
classification cs.CL
keywords modelmergingmultimodallargelanguagemodelsparameterfusionLoRAadaptersdual-gategatingentropyweightingdomaincompatibilityactivationanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that separately fine-tuned multimodal experts can be combined into one model by splicing parameters selectively rather than retraining. The proposed Graft method gives every output channel of a layer a learned local weight based on how much two experts' parameters differ, then modulates those weights with a global scalar derived from the entropy difference of the two parameter distributions. On a Qwen2-VL-2B backbone, fusing a math expert with a code expert lifts MathVista accuracy to 52.2 percent and keeps HumanEval pass@1 at 15.9 percent, outperforming task arithmetic, TIES, and DARE. The paper also introduces an activation-based compatibility score that predicts, before fusion, which expert pairs will benefit from merging. If correct, Graft offers a modular route to domain-adaptive MLLMs without additional fine-tuning.

What carries the argument

The load-bearing object is the dual-gate fusion mechanism. The local gate is a learnable network $\phi$ that takes the per-channel absolute-difference vector $d_i=\sum_j|W_b[i,j]-W_g[i,j]|$ and produces channel weights through a sigmoid. The global gate is a scalar derived from the entropy difference $H(W_b)-H(W_g)$ using an arctan shaping function, so it expresses which module appears more information-rich overall. The two gating signals are combined through exponential transforms and normalized by softmax, and the final fused weight matrix is a channel-wise weighted sum of the two modules. A separate activation-based compatibility score, computed as $\rho_i = \mu_i(1-s_i)\sqrt{v_i}$ from mean magnitude, sparsity, and variance of activations on a small dataset sample, is used to decide which expert pairs are worth fusing.

What would settle it

Replace the learnable local gate with a constant 0.5 and re-run the Math+Code fusion; if MathVista stays at 52.2, the learned channel network is not carrying the result. Separately, compute compatibility scores for 20 random expert pairs; if the Spearman correlation with actual gains does not reproduce the reported 0.86, the activation metric does not predict fusion utility.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a dual-gate fusion rule, combining per-channel gating with a global entropy gate, can reconcile parameters from domain-specialized models better than global linear interpolation. For each layer, the absolute difference between base and graft weights is mapped through a learnable network $\phi$ and a sigmoid into local weights $w_{\text{local}}$; the entropy gap $H(W_b)-H(W_g)$ is mapped through an arctan into a global scalar $w_{\text{global}}$; the two are combined nonlinearly and normalized by softmax before the weighted sum $W_f = w_b \odot W_b + w_g \odot W_g$. The paper reports that this mechanism, applied to LoRA adapters, yields the best balanced scores among the tested fusion methods, that a small-sample activation compatibility score correlates with downstream gains (Spearman rank correlation 0.86), and that multi-domain fusion of three or four experts keeps improving MathVista to 53.0 without collapsing coding performance.

Load-bearing premise

The load-bearing premise is that the learnable channel gating network $\phi$ can be trained, initialized, and regularized so that it generalizes without contaminating the test benchmarks, because the paper never specifies that training procedure.

Editorial extensions

If this is right

  • Domain experts tuned as LoRA adapters can be composed into one model without retraining, preserving strong coding ability while improving math reasoning.
  • The compatibility score gives a cheap, data-light screening step: with a small sample of activations, a practitioner can rank candidate expert pairs before committing to fusion.
  • The framework extends beyond two experts: reported gains on MathVista are monotonic from two to four domains, reaching 53.0, while HumanEval stays roughly steady, so multi-expert composition is feasible.
  • Fusing all attention and MLP projections together performs better than fusing either family alone, so full-layer grafting is the recommended granularity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the training procedure for $\phi$ unspecified; a natural extension is to test whether the local gate can be learned once and reused across different domain pairs, which would make Graft a genuinely plug-and-play merger.
  • Because the compatibility score is computed from activations on a small sample of the target dataset, it could be turned into an online router that selects among experts at inference time rather than only at merge time.
  • The entropy gate assumes parameter entropy tracks information content; if a domain expert has high entropy merely from noise in rarely used channels, the global scalar could dilute a high-quality graft, a failure mode the experiments do not probe.
  • A testable extension is to apply the dual gate to adapters of different ranks or architectures; the paper only demonstrates fusion of homogeneous adapters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Graft, a parameter-fusion framework for multimodal large language models that composes two or more domain-specialized models by combining a channel-level local gate (Section 3.1), an entropy-based global gate (Section 3.2), and an activation-based compatibility analysis (Section 3.4). Two variants are presented: GraftModel for fully fine-tuned parameters and GraftLoRA for LoRA adapters. Experiments on Qwen2-VL-2B fuse Math, Code, Finance, and Medical experts and evaluate on MathVista, HumanEval, MMMU, and MME, comparing against Task Arithmetic, TIES-Merging, and DARE. The paper reports that Graft achieves the most balanced improvements across benchmarks, that the compatibility score correlates with downstream gains (Spearman ρ = 0.86), and that multi-domain fusion scales up to four experts without degrading the primary tasks.

Significance. If the central claim is validated, Graft would be a practically useful method for modular composition of domain-specialized MLLMs, especially for LoRA adapters, with negligible inference overhead. The paper has notable strengths: it covers both full fine-tuning and LoRA settings, evaluates on multiple public benchmarks, includes ablations of the local and global gates, provides a layer-wise fusion analysis, and adds a human preference study. However, the current evidence is not sufficient to support the central claim because the learnable gating network φ is never specified, the compatibility correlation rests on a very small number of fusion pairs, and the fusion formulas and hyperparameters are introduced heuristically without sensitivity analysis. These issues are load-bearing for reproducibility and for the validity of the reported gains.

major comments (4)
  1. [Section 3.1, Eq. (2), Algorithm 1] The local gate is defined as w_local = σ(φ(d)), with φ called "learnable," but the paper never specifies the training objective, the data or labels used to fit φ, its initialization, optimizer, learning rate, or regularization. Algorithm 1 simply takes φ as an input. Because the local gate is half of the dual-gate mechanism and the ablation in Table 5 shows that Local-Gate alone (52.0 MathVista) substantially outperforms Global-Gate alone (51.7 MathVista), every reported improvement in Table 1 depends on an unspecified fitting step. This must be fully specified, including a clear separation between any data used to fit φ and the evaluation benchmarks, before the central claim can be assessed.
  2. [Section 3.4, Table 2] The compatibility score and the claimed Spearman correlation ρ = 0.86 are computed over a very small number of fusion pairs (at most five in Table 2), with no confidence intervals, no number of pairs reported in the text, and no held-out validation. Moreover, Section 3.4 does not state whether the K samples used for the activation statistics are the same samples used to fit φ or to tune a, c, and n; if they are, the correlation is partly circular. Please report K, the threshold rule, the exact number of pairs, and a bootstrap or out-of-sample evaluation of the correlation.
  3. [Section 4.1, Eq. (4), Algorithm 1] The hyperparameters a = 0.4, c = 500, and n = 10 are selected once on the evaluation benchmarks, with no sensitivity analysis or separate validation set, which weakens the claim that Graft is generally applicable. In addition, Eq. (4) defines w_global = a/c * arctan(c[H(W_b) - H(W_g)]) + 1/2, while Algorithm 1 line 5 uses a/π * arctan(c[H_b - H_g]) + 1/2. With the stated a = 0.4 and c = 500, the a/c version would make the global gate nearly constant, so the discrepancy is material and must be resolved.
  4. [Tables 1 and 2] All benchmark numbers are reported without standard deviations or multiple seeds. Several comparisons are within 0.1-1.6 points (for example, MathVista LoRA: Graft 52.2 vs. TIES 52.1; MMMU: Graft 37.6 vs. Code 37.8), so without variance estimates the headline "most balanced improvements" is not statistically established. Reporting mean ± std over at least three fine-tuning and fusion runs, or otherwise providing significance tests, would be needed.
minor comments (5)
  1. [Eq. (3)] The entropy formula has a missing parenthesis: "H(W = -Σ p_k log p_k" should read "H(W) = -Σ p_k log p_k". Please correct the notation.
  2. [Table 1] The row labeled "Average" in Table 1 is not introduced as a baseline in Section 4.2, and the caption says four baselines are compared although the table appears to include five non-backbone rows. Please clarify what "Average" represents and align the caption with the table.
  3. [Table 2] The column layout of Table 2 is difficult to parse: the domain-composition check marks, the compatibility scores, and the benchmark scores run together, and some rows appear to have misaligned entries. Please reformat the table so each fusion pair is unambiguous.
  4. [Section 4.2, Figure 4] The human evaluation is described only briefly; please report the number of queries, the annotation protocol, whether the evaluators were blinded to method identity, and inter-annotator agreement.
  5. [General] The paper does not provide code or trained checkpoints; given the unspecified φ, this is particularly limiting for reproducibility. Releasing code would substantially address the concerns raised above.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Graft's fusion rule is an explicit arithmetic function of the two expert parameter sets, and its reported gains are benchmark measurements rather than quantities derived from fitted inputs.

full rationale

The central fusion rule (Eqs. 1-8) is defined directly on the base and graft parameters Wb and Wg: d is the absolute parameter difference, wlocal is a sigmoid of a learnable gate on d, wglobal is an arctan of the entropy difference, and the fused output is a softmax-weighted sum. None of these quantities is defined in terms of the evaluation scores they later predict; the benchmark numbers in Tables 1-5 are external measurements. The learnable gate phi is underspecified (no training objective, data split, or initialization described), which is a reproducibility gap, not a circular reduction, because the paper does not claim a derivation from phi's training data. The compatibility score (Eqs. 9-14) is a fixed product of activation statistics; it is computed on K target-dataset samples and later correlated with gains on the same benchmarks, which raises potential leakage or post-hoc-analysis concerns, but the score is not fitted to the gains and no equation equates it to downstream performance. The hyperparameters a=0.4, c=500, and n=10 are stated as global settings rather than fitted per benchmark. Self-citations appear in the introduction and related work but are not used to justify the fusion mechanism or to suppress alternatives. Accordingly, no load-bearing step reduces by construction to its own inputs, and the paper is not circular in the sense assessed here.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several hand-set hyperparameters (a, c, n) and unspecified quantities (K, compatibility threshold). The method introduces two ad hoc modeling assumptions: weight entropy as an information signal and the activation composite as a compatibility predictor. No new physical or ontological entities are introduced.

free parameters (5)
  • a (global gating scaling) = 0.4
    Hand-set in Section 4.1; controls the range of the arctan-based global weight; no sensitivity analysis.
  • c (global gating slope) = 500
    Hand-set in Section 4.1; steepness of the arctan mapping from entropy difference; no sensitivity analysis.
  • n (entropy bins) = 10
    Hand-set in Section 4.1; discretization granularity for weight-entropy computation.
  • K (samples for compatibility analysis) = not specified
    Section 3.4 says K is small but no value is given; the compatibility metric depends on K.
  • compatibility threshold = not specified
    Section 3.4 mentions a threshold for accepting modules but never defines it.
assumptions (4)
  • domain assumption Weight-space differences between pre- and post-fine-tuning parameters encode task-specific knowledge.
    Section 2, drawing on Ilharco et al. (Task Arithmetic); foundational to all merging methods, reasonable but not proven within this paper.
  • ad hoc to paper The Shannon entropy of discretized weight values measures the information content of a module, so higher entropy implies richer information.
    Section 3.2, Eqs. (3)-(4); this mapping from weight histogram entropy to fusion preference is introduced without justification.
  • ad hoc to paper The composite activation statistic rho = mu * (1 - s) * sqrt(v) measures a module's sensitivity to a dataset and predicts fusion compatibility.
    Section 3.4, Eqs. (9)-(14); the product form and normalization are arbitrary and untested against alternative statistics.
  • standard math Standard mathematical tools (entropy, arctan, sigmoid, softmax) and their properties.
    Used throughout Section 3; assumed valid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs." pith.science (2026). https://pith.science/paper/XJFF3DD5

@misc{pith2026250623940,
  author       = {Pith},
  title        = {Pith review of: Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XJFF3DD5}},
  note         = {Machine review of arXiv:2506.23940}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have achieved success across various domains. However, their applicability tends to degrade when confronted with different types of data inputs, especially for MLLMs that have been fine-tuned for specific tasks. Despite its importance, the study of knowledge sharing among domain-specific MLLMs--such as those trained for mathematics or code--remains largely underexplored. To address the fragmentation of knowledge across domain-specialized MLLMs, we propose a unified parameter integration framework that enables modular composition of expert capabilities. Our method is grounded in a novel Compatibility-Aware Parameter Splicing (CAPS) strategy, which leverages both local functional attribution and global information-theoretic signals to guide selective parameter fusion. By extending this mechanism to the low-rank adaptation layer granularity, we ensure efficient integration with minimal inference overhead. Furthermore, we introduce a domain compatibility scoring mechanism that quantifies inter-expert alignment at the activation level and correlates with downstream task utility. This principled fusion protocol allows the final model to synergize heterogeneous expertise while preserving structural modularity. Extensive evaluations across diverse multimodal benchmarks validate the effectiveness of our framework, offering a scalable path toward compositional, domain-adaptive MLLMs.

Figures

Figures reproduced from arXiv: 2506.23940 by the authors.

Figure 1
Figure 1. Performance comparison between Graft and other methods on fully fine-tuned and LoRA [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed Graft method, illustrating how base and graft model or LoRA [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance of different projection layer fu [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.

Reference graph

Works this paper leans on

31 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [7]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186,

    arXiv

  2. [8]

    Think twice, click once: Enhancing gui grounding via fast and slow systems

    Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song, Weiming Lu, and Yueting Zhuang. Think twice, click once: Enhancing gui grounding via fast and slow systems. arXiv preprint arXiv:2503.06470,

    arXiv

  3. [9]

    Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666,

    arXiv

  4. [10]

    Deep model fusion: A survey

    Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698, 2023d. Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. arXiv preprint arXiv:2001.02312,

    arXiv 2001

  5. [11]

    Optimize Incompatible Parameters through Compatibility-aware Knowledge Integration

    Zheqi Lv, Keming Ye, Zishu Wei, Qi Tian, Shengyu Zhang, Wenqiao Zhang, Wenjie Wang, Kun Kuang, Tat-Seng Chua, and Fei Wu. Optimize incompatible parameters through compatibility- aware knowledge integration. arXiv preprint arXiv:2501.07596,

    work page Pith review arXiv

  6. [12]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089,

    arXiv

  7. [13]

    Duet: A tuning-free device-cloud collaborative parameters generation framework for efficient device model generalization

    Zheqi Lv, Wenqiao Zhang, Shengyu Zhang, Kun Kuang, Feng Wang, Yongwei Wang, Zhengyu Chen, Tao Shen, Hongxia Yang, Beng Chin Ooi, et al. Duet: A tuning-free device-cloud collaborative parameters generation framework for efficient device model generalization. In Proceedings of the ACM Web Conference 2023, pages 3077–3085,

    work page 2023

  8. [14]

    Intelligent model update strategy for sequential recommendation

    Zheqi Lv, Wenqiao Zhang, Zhengyu Chen, Shengyu Zhang, and Kun Kuang. Intelligent model update strategy for sequential recommendation. In Proceedings of the ACM Web Conference 2024, pages 3117–3128,

    work page 2024

Show all 31 references
  1. [15]

    Bridging local details and global context in text-attributed graphs

    Yaoke Wang, Yun Zhu, Wenqiao Zhang, Yueting Zhuang, Yunfei Li, and Siliang Tang. Bridging local details and global context in text-attributed graphs. arXiv preprint arXiv:2406.12608, 2024b. Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, ...

    arXiv

  2. [16]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langu...

    2019

  3. [17]

    Improving language understanding by generative pre-training.(2018),

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training.(2018),

    2018

  4. [18]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691,

    arXiv

  5. [19]

    P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602,

    arXiv

  6. [20]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849,

    arXiv

  7. [21]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2310.02575, 2023a. Wei Ji, Li Li, Zheqi Lv, Wenqiao Zhang, Mengze Li, Zhen Wan, Wenqiang Lei, and Roger Z...

    arXiv

  8. [22]

    Mart: Learning hierarchical music audio representations with part-whole transformer

    Dong Yao, Jieming Zhu, Jiahao Xun, Shengyu Zhang, Zhou Zhao, Liqun Deng, Wenqiao Zhang, Zhenhua Dong, and Xin Jiang. Mart: Learning hierarchical music audio representations with part-whole transformer. In Companion Proceedings of the ACM Web Conference 2024, pages 967–970,

    2024

  9. [23]

    Eoc-bench: Can mllms identify, recall, and forecast objects in an egocentric world? arXiv preprint arXiv:2506.05287, 2025a

    Yuqian Yuan, Ronghao Dang, Long Li, Wentong Li, Dian Jiao, Xin Li, Deli Zhao, Fan Wang, Wenqiao Zhang, Jun Xiao, et al. Eoc-bench: Can mllms identify, recall, and forecast objects in an egocentric world? arXiv preprint arXiv:2506.05287, 2025a. Ziqi Jiang, Shengyu Zhang, Siyuan...

    arXiv

  10. [24]

    Makima: Tuning-free multi-attribute open- domain video editing via mask-guided attention modulation

    Haoyu Zheng, Wenqiao Zhang, Zheqi Lv, Yu Zhong, Yang Dai, Jianxiang An, Yongliang Shen, Juncheng Li, Dongping Zhang, Siliang Tang, et al. Makima: Tuning-free multi-attribute open- domain video editing via mask-guided attention modulation. arXiv preprint arXiv:2412.19978,

    arXiv

  11. [25]

    Boosting private domain understanding of efficient mllms: A tuning-free, adaptive, universal prompt optimization framework

    Jiang Liu, Bolin Li, Haoyuan Li, Tianwei Lin, Wenqiao Zhang, Tao Zhong, Zhelun Yu, Jinghao Wei, Hao Cheng, Wanggui He, et al. Boosting private domain understanding of efficient mllms: A tuning-free, adaptive, universal prompt optimization framework. arXiv preprint arXiv:2412.19684,

    arXiv

  12. [26]

    Photo stream question answer

    Wenqiao Zhang, Siliang Tang, Yanpeng Cao, Jun Xiao, Shiliang Pu, Fei Wu, and Yueting Zhuang. Photo stream question answer. In Proceedings of the 28th ACM International Conference on Multimedia, pages 3966–3975, 2020b. Dong Yao, Shengyu Zhang, Zhou Zhao, Jieming Zhu, Wenqiao Zh...

    arXiv

  13. [27]

    Multimodal large language models: A survey

    Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S Yu. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247–2256. IEEE,

    2023

  14. [28]

    Mm-llms: Recent advances in multimodal large language models

    Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601, 2024b. 13 Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Z...

    arXiv

  15. [29]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1):1, 2023b. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wa...

    arXiv

  16. [30]

    Math-llava: Bootstrapping mathematical reasoning for multimodal large language models

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294,

    arXiv

  17. [32]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024d

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    arXiv

  18. [2019]

    Hero: Hierarchical spatio-temporal reasoning with contrastive action correspondence for end-to-end video object grounding

    Mengze Li, Tianbao Wang, Haoyu Zhang, Shengyu Zhang, Zhou Zhao, Wenqiao Zhang, Jiaxu Miao, Shiliang Pu, and Fei Wu. Hero: Hierarchical spatio-temporal reasoning with contrastive action correspondence for end-to-end video object grounding. In Proceedings of the 30th ACM Interna...

    arXiv

  19. [2020]

    Sujet AI

    URL https://arxiv.org/abs/2003.10286. Sujet AI. Sujet-Finance-QA-Vision-100k. https://huggingface.co/datasets/sujet-ai/ Sujet-Finance-QA-Vision-100k ,

    2003 arXiv

  20. [2021]

    Relational graph learning for grounded video description generation

    Wenqiao Zhang, Xin Eric Wang, Siliang Tang, Haizhou Shi, Haochen Shi, Jun Xiao, Yueting Zhuang, and William Yang Wang. Relational graph learning for grounded video description generation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 3807–3828, 2...

    arXiv

  21. [2022]

    Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation

    Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, et al. Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation. arXiv preprint ar...

    arXiv

  22. [2024]

    Minerva: Solving quantitative reasoning problems with language models

    Ethan Dyer and Guy Gur-Ari. Minerva: Solving quantitative reasoning problems with language models. June, 30:2022,

    2022

  23. [2025]

    Fin-r1: A large language model for financial reasoning through reinforcement learning

    Zhaowei Liu, Xin Guo, Fangqi Lou, Lingfeng Zeng, Jinyi Niu, Zixuan Wang, Jiajie Xu, Weige Cai, Ziwei Yang, Xueqian Zhao, et al. Fin-r1: A large language model for financial reasoning through reinforcement learning. arXiv preprint arXiv:2503.16252,

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.