Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes MMER, a training-free method that merges the LLM parameters of several existing MLLMs and decouples them with binary masks, claiming multimodal expansion with about 99% retention of each original model's performance and…

desk verdict Training-free mask-based decoupling for MLLM merging is a clean and useful idea, but the 99% retention and 'reuse existing MLLMs' claims are only demonstrated on a self-fine-tuned same-base family with tuned thresholds. read the letter →

arxiv 2505.17110 v1 pith:P4JW2FQA submitted 2025-05-21 cs.CL

Junlin Li , Guodong DU , Jing Li , Sim Kuan Goh , Wenya Wang , Yequan Wang , Fangming Liu , Ho-Kin Tang
show 3 more authors
Saleh Alharbi Daojing He Min Zhang
This is my paper · ORCID
classification cs.CL
keywords modelmergingtaskvectorsmultimodallargelanguagemodelsparameterdecouplingbinarymaskscatastrophicforgettingtraining-freeexpansionmodality-specificrouting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MMER is a training-free way to give one LLM multiple modality abilities by combining models that already have them. Starting from several multimodal LLMs that share the same pretrained backbone, the method merges the vectors of their fine-tuning updates (task vectors) into one vector, then builds a binary mask per modality that selects which entries of the merged vector belong to that modality. The claim is that these masks decouple the merged model well enough that each original model can be reconstructed nearly exactly (about 99% retention on the paper's benchmarks), so the same merged model can handle image, audio, video, and point-cloud inputs without retraining and without the usual performance drop of model merging. The paper also claims the same mechanism absorbs newly fine-tuned models and thereby mitigates catastrophic forgetting while adapting to new tasks.

What carries the argument

The load-bearing objects are task vectors and modality masks. A task vector $\tau_i$ is the difference between a fine-tuned MLLM's LLM weights and the shared pretrained weights $\theta_{pre}$, so it represents what that modality's training added. TIES merging sparsifies these vectors and combines sign-consistent entries into $\tau^*$. The modality mask $m_i$ is a binary vector that keeps an entry of $\tau^*$ only when original and merged signs agree and the original entry is dominant (Eq. 3). During inference, each modality's tokens are processed through $\theta_{pre} + m_i \circ \tau^*$ for attention projections, while text tokens use the average mask, so every modality gets its own effective weights without any change to the encoder stack.

What would settle it

Compute the per-parameter relative reconstruction error $|m_i(p)\tau^*(p) - \tau_i(p)| / |\tau_i(p)|$ for each modality of the paper's four MLLMs. If the audio modality, whose task-vector magnitudes are about six times smaller than vision and point, shows large error yet still retains performance, or if applying MMER to a different pair of same-backbone MLLMs with denser task vectors drops original-task retention well below 99%, the decoupling approximation would be shown to be setup-dependent rather than a general property of merged task vectors.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that a single merged task vector can carry multiple modality-specific models inside it, and that a simple comparison of the original and merged task vectors is enough to find them. For each modality $i$, MMER keeps a parameter of the merged task vector $\tau^*$ in the reconstruction exactly when that parameter has the same sign in $\tau_i$ and $\tau^*$ (Directional Congruence) and when $\tau_i$'s magnitude is at least $\lambda\cdot 50\%$ of $\tau^*$'s magnitude (Dominant Significance), with $\lambda$ controlling the number of selected parameters. The resulting mask $m_i$ yields $\theta_{pre} + m_i \circ \tau^*$ as a stand-in for the original model $\theta_i$. The same recipe gives multimodal expansion (each input type is processed through its own decoupled weights), multimodal retention (reconstructed models score about 99% of original performance), and catastrophic-forgetting mitigation (a newly fine-tuned MLLM is treated as one more vector to merge and decouple).

Load-bearing premise

The decoupling assumption that selected entries of the merged task vector can stand in for the corresponding entries of each original task vector without any magnitude rescaling is load-bearing; when $|\tau_i|$ is between 50% and 100% of $|\tau^*|$, the reconstructed weight uses $\tau^*$'s magnitude, which can be up to twice $\tau_i$, and the paper gives no error bound for this approximation.

Editorial extensions

If this is right

  • A single merged MLLM can accept image, audio, video, and point-cloud inputs, including combinations, with no additional training or extra inference parameters.
  • Original MLLMs can be reconstructed from the shared merged task vector plus per-modality masks, retaining roughly 99% of trimmed-average performance while using less storage than keeping all original models.
  • Adding a newly fine-tuned MLLM to the merge set preserves previous-task scores and recovers nearly all of the new task's gain, mitigating catastrophic forgetting in both single-task and cross-modal multi-task scenarios.
  • Because modality masks are sparse (the audio mask selects only 2.2% of parameters), the success of decoupling depends on task vectors being highly redundant, consistent with earlier findings that supervised fine-tuning adds sparse delta parameters.
  • Storage grows roughly linearly with the number of merged MLLMs (base LLM, merged task vector, masks, and per-modality encoders), but inference cost remains that of one model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mask-based decoupling is as general as the experiments suggest, the approach could be applied to any collection of same-backbone models that differ only by fine-tuning, not just modality-specific MLLMs; the main constraint is that base-model checkpoints must match exactly.
  • A testable extension is to treat the mask selection threshold $\lambda\cdot 50\%$ as a per-modality calibration knob and measure whether retention degrades smoothly enough to make the method robust without validation sets.
  • The method's ability to absorb newly fine-tuned models implies that catastrophic forgetting can be reframed as a merging problem: instead of updating one model, keep the old vector and merge in the new one, which scales naturally to many tasks only if mask overlap stays low.
  • The reported gains on classification tasks for audio and point cloud suggest that merging can unlock instruction-following behavior that the original fine-tuned models lacked; whether this unlocks or merely rescues capabilities is not settled by the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MMER, a training-free method for multimodal expansion of LLMs by merging the LLM parameters of several MLLMs into a single model while retaining each original MLLM's performance. MMER uses TIES-style task-vector merging to obtain a merged task vector τ*, then constructs per-modality binary masks by comparing each original task vector τ_i with τ* under a directional-consistency and dominant-magnitude criterion (Eq. 3). The masked merged vector is added to the pre-trained LLM to approximate each original LLM (Eq. 1), allowing modality-specific inputs to be processed with modality-specific parameters. The paper reports experiments on four self-fine-tuned Vicuna-7B MLLMs covering vision, audio, video, and point cloud: multimodal expansion across 10 input combinations (Table 1), retention of 99% of original performance averaged over 14 dual-modal tasks (Table 2), and mitigation of catastrophic forgetting in single-task and cross-modal multi-task scenarios (Table 3). Ablation and hyperparameter analyses show the importance of directional congruence, dominant significance, and the scaling factor λ.

Significance. If the results hold beyond the current setup, MMER is a useful contribution: it extends task-vector merging to multimodal models with a simple, train-free decoupling mechanism, and it demonstrates a practical application of model merging to catastrophic forgetting in MLLMs. The paper provides a broad experimental surface (10 input combinations, 14 retention tasks, two forgetting scenarios) and gives an exact per-coordinate derivation of the L1-optimal binary mask for λ=1 (Eqs. 2–3). The main risks are that all experiments use self-fine-tuned MLLMs with per-modality hyperparameters calibrated on the target tasks, and that the central comparison with DAMC is not run. These issues directly affect the strength of the headline '99% retention' and 'existing MLLMs' claims, but they are addressable in revision.

major comments (4)
  1. [§4.1, §5, Table 2, §B.2] The headline 'retains 99% of original performance' is an in-sample, calibrated result. Section B.2 reports that λ_i is calibrated per modality, and Section 3.1 says λ_i follows the same principle as α, which is calibrated on validation sets from target tasks; moreover, all four MLLMs are self-fine-tuned from the same base (Vicuna-7B, §4.1), and the retention tasks in Table 2 are the same types of tasks used to fine-tune those models and to build the masks. The abstract and introduction say MMER 'integrates existing MLLMs' and targets a training-free expansion for off-the-shelf models, but no experiment uses an existing pre-trained MLLM. Please add an experiment with off-the-shelf same-architecture MLLMs (or an explicitly labeled surrogate) and, failing that, narrow the claims in the abstract and Section 1 to self-fine-tuned MLLMs. This is load-bearing because the 99% number is the paper's central empirical promise.
  2. [§3.1, Eqs. (2)–(3), §D.1] The mask formula is not the claimed L1 minimizer unless λ_i=1. The per-coordinate solution to Eq. (2) is m_i^{(p)}=1 iff |τ_i^{(p)}| ≥ 0.5|τ_*^{(p)}| and the signs agree. Equation (3) inserts λ_i, and the authors tune λ_i; Figure 6 shows that varying λ_i changes MCUB accuracy, and the ablation in Table 4 shows that λ contributes to the result. When λ_i≠1, Eq. (3) selects a mask that is suboptimal under Eq. (2), so the 'minimizing Manhattan distance' derivation in Section D.1 does not justify the actual mask. Please report the tuned λ_i values, state explicitly that Eq. (3) is a heuristic relaxation, or provide a bound relating the relaxation to the L1 objective.
  3. [§1, §2, Appendix A, Tables 1–2] DAMC is a central comparison point but is never evaluated. The Introduction frames the contribution as bypassing the NaiveMC/DAMC trade-off, and Appendix A claims MMER 'achieves similar results' to DAMC. DAMC is absent from Tables 1 and 2 and from the forgetting experiments, so the reader cannot verify the trade-off claim. If DAMC can be run under the same protocol, please include it in at least the expansion and retention experiments; otherwise, remove or soften the claim that MMER matches DAMC.
  4. [§3.1, Eq. (1)] The decoupling assumption that selected entries of τ* can stand in for the corresponding entries of each τ_i has no error bound or failure-mode analysis. When |τ_i^{(p)}| lies between 50% and 100% of |τ_*^{(p)}|, the reconstructed parameter uses τ_*^{(p)} unmodified, so the per-coordinate error can be as large as |τ_i^{(p)}|; when the ratio is below 50%, the coordinate is dropped entirely. The paper shows empirically that this works for four self-fine-tuned Vicuna-7B models, but it does not analyze the regime where several task vectors align in sign and accumulate magnitude in τ*. I would like a discussion, or a targeted experiment, of when the no-rescaling reconstruction is expected to hold beyond this model family.
minor comments (5)
  1. [Table 1 and throughout] The benchmark name is spelled 'MUSIC-A VQA' in the text and references, but appears as 'MUSCI-A VQA' in Table 1 and its caption; please unify the spelling.
  2. [§3.1 vs. §B.2] The role of α is confusing: Section 3.1 defines the merged parameter as θ* = θ_pre + α·τ* with α calibrated on validation sets, while Section B.2 states that α is not set because the TIES-merged model is not used in MMER. Please clarify whether α appears anywhere in MMER and, if not, remove the description or explain why the merged vector needs no scaling.
  3. [§5, Tables 1–3] The paper uses the word 'significantly' (e.g., 'MMER significantly outperforms') but reports no error bars, multiple seeds, or significance tests. The consistent improvement across tasks is visible in the tables, but the statistical language should be softened or supported.
  4. [Eq. (2)] The notation in Eq. (2) repeats 'arg min' and uses |·| ambiguously; the expression |mi ◦ τ* − τ_i| should be defined explicitly as the ℓ1 norm over the P coordinates, with the double sum notation cleaned up.
  5. [Figure 6] The left panel of Figure 6 mixes bar plots and line plots with several y-axes; the caption states that the lines show performance retention, while the bars show mask percentages. Please use clearly separated panels and a legend that matches the caption so the reader can map each element to its axis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MMER's mask construction is transparently an L1 reconstruction heuristic, and the 99% retention is an empirical result, not a consequence of the construction alone.

full rationale

The paper's core operation is not a hidden reduction. Eq. (2) openly defines the mask as minimizing the L1 distance between the masked merged task vector and the original task vector, so Eq. (1) is a stated construction objective, not an independently derived prediction; the paper then tests functional retention empirically on standard benchmarks, which is an independent check. The thresholds lambda_i and TopK are tuned per modality or task family (Sec. 3.1, Sec. B.2), which weakens the headline as an out-of-the-box claim but is ordinary hyperparameter selection, not a fitted parameter renamed as a prediction, and no equation forces the reported 99% retention. The Limitations section explicitly concedes that the experiments use four self-fine-tuned Vicuna-7B MLLMs and that 'finding multiple existing MLLMs with the same architecture across modalities is currently challenging,' so external generalizability is limited; this is a scope caveat, not circular reasoning. Self-citations (e.g., PCB-Merging and knowledge-fusion work) appear only as baselines or related-work motivation and are not load-bearing. No circular step meets the evidentiary bar set by the review criteria.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central method depends on four domain assumptions and two fitted hyperparameters (λ_i and TopK%). No invented physical entities are introduced. The binary masks are algorithmic constructs, not independent entities.

free parameters (2)
  • λ_i (per-modality mask threshold scale) = calibrated per modality, exact values not reported
    Controls the threshold |τ_i| >= λ_i * 50% * |τ*| for mask selection. Calibrated on validation sets or general per-modality performance (Section 3.1, B.2).
  • TopK% (TIES sparsity) = 80%
    Fraction of largest task-vector entries retained before merging. Set manually and varied in ablation (Section 4.1, Figure 6 right).
assumptions (4)
  • domain assumption Task vectors from MLLMs fine-tuned from the same pre-trained LLM are additively composable.
    The paper relies on TA and TIES results (Ilharco et al., 2023; Yadav et al., 2023) that arithmetic on task vectors yields functional merging (Section 3.1).
  • domain assumption The merged task vector τ* retains sufficient information from each original task vector.
    Cites Panigrahi et al. (2023) and Wang et al. (2024) for skill localization; without this, mask decoupling cannot recover original behavior (Section 3.1).
  • ad hoc to paper Binary masking without magnitude rescaling can approximate the original task vectors.
    The reconstruction θ̂_i = θ_pre + m_i ∘ τ* uses selected entries of τ* unmodified, even when |τ_i| differs from |τ*|; the paper does not provide a bound or rescaling (Eq. 1).
  • domain assumption All merged MLLMs share the same base LLM θ_pre.
    The method requires aligned architectures; the paper fine-tunes all four MLLMs from Vicuna-7B-v1.5 in the same environment (Section 4.1, Limitations).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling." pith.science (2026). https://pith.science/paper/P4JW2FQA

@misc{pith2026250517110,
  author       = {Pith},
  title        = {Pith review of: Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P4JW2FQA}},
  note         = {Machine review of arXiv:2505.17110}
}
read the original abstract

Fine-tuning Large Language Models (LLMs) with multimodal encoders on modality-specific data expands the modalities that LLMs can handle, leading to the formation of Multimodal LLMs (MLLMs). However, this paradigm heavily relies on resource-intensive and inflexible fine-tuning from scratch with new multimodal data. In this paper, we propose MMER (Multi-modality Expansion and Retention), a training-free approach that integrates existing MLLMs for effective multimodal expansion while retaining their original performance. Specifically, MMER reuses MLLMs' multimodal encoders while merging their LLM parameters. By comparing original and merged LLM parameters, MMER generates binary masks to approximately separate LLM parameters for each modality. These decoupled parameters can independently process modality-specific inputs, reducing parameter conflicts and preserving original MLLMs' fidelity. MMER can also mitigate catastrophic forgetting by applying a similar process to MLLMs fine-tuned on new tasks. Extensive experiments show significant improvements over baselines, proving that MMER effectively expands LLMs' multimodal capabilities while retaining 99% of the original performance, and also markedly mitigates catastrophic forgetting.

Figures

Figures reproduced from arXiv: 2505.17110 by the authors.

Figure 1
Figure 1. The key ideas of MMER. Multi-Modality Ex￾pansion creates a versatile model from existing MLLMs via a training-free, extensible process. Multi-Modality Retention reconstructs original or new task MLLMs to retain performance and mitigate catastrophic forgetting. processing multimodal inputs. This paradigm has led to the successful creation of numerous MLLMs across various modalities (Wu et al., 2024; Jiang et al., 202… view at source ↗
Figure 2
Figure 2. The overview of MMER, considering only the Vision and Point Cloud modalities for clarity. Each block corresponds to the same weight matrix, with empty blocks denoting zero value. “≈” signifies similar performance. Model Merging and Model Composition. Model merging (Yang et al., 2024; Du et al., 2024b) can improve single-task performance (Gupta et al., 2020), out-of-distribution generalization (Arpit et al., 2022), o… view at source ↗
Figure 3
Figure 3. Details of MMER’s dynamic processing. and represent the Hadamard product and addition. we refer to as Dominant Significance. We further introduce a scaling factor λi to refine this selection process, accommodating the varying numbers and modalities of original MLLMs, where a smaller λi selects more parameters. The selection of λi fol￾lows the same principle as α, enabling the modality￾specific inputs to be processed… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance retention vs. MLLMs quantity. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: (Left). The bar plots illustrate the percentage of parameters selected by masks, while the lines show performance retention of NaiveMC and MMER across various dual-modal tasks. (Right). The lines depict the variations in MCUB average accuracy across different sparsity …
Figure 7
Figure 7. Figure 7: Storage cost vs. Number of MLLMs. Additionally, we provide the calculation of stor￾age costs for MMER approach and the relevant meth￾ods mentioned above. Let N, P, P ′ , and P ∗ repre￾sent the number of original MLLMs, the total pa￾rameters of the LLMs, the number of t…
Figure 8
Figure 8. Figure 8: Qualitative results [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-objective Large Language Model Alignment with Hierarchical Experts

    cs.CL 2025-05 conditional novelty 6.0 of 10

    HoE claims to align a single LLM to any preference vector over multiple objectives using training-free LoRA experts, lightweight trained routers, and nearest-neighbor preference routing.

Reference graph

Works this paper leans on

76 extracted references · 59 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)

    work page 2022

  5. [5]

    Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. 2022. Ensemble of averages: Improving model selection and boosting performance in domain generalization. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)

    work page 2022

  6. [6]

    Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, and Yang Liu. 2024 a . Model composition for multimodal large language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

    work page 2024

  7. [7]

    Chen and William B

    David L. Chen and William B. Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 190--200

    work page 2011

  8. [8]

    Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. 2023 a . X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.04160

    arXiv 2023

Show all 76 references
  1. [9]

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. 2023 b . Beats: Audio pre-training with acoustic tokenizers. In Proceedings of the International Conference on Machine Learning (ICML), volume 202, pages 5178--5193

    2023

  2. [10]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024 b . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Com...

    2024

  3. [11]

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919

    2023 arXiv

  4. [12]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Proceedings of the Advances in Neural Informati...

    2023

  5. [13]

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. 2023. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2023

  6. [14]

    Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. 2023. Pengi: An audio language model for audio tasks. In Proceedings of the Advances in neural information processing systems (NeurIPS)

    2023

  7. [15]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognit...

    2021

  8. [16]

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. 2020. Clotho: an audio captioning dataset. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , pages 736--740

    2020

  9. [17]

    Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, and Min Zhang. 2024 a . Parameter competition balancing for model merging. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)

    2024

  10. [18]

    Guodong Du, Jing Li, Hanting Liu, Runhua Jiang, Shuyang Yu, Yifei Guo, Sim Kuan Goh, and Ho-Kin Tang. 2024 b . Knowledge fusion by evolving weights of language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

    2024

  11. [19]

    Rohit Girdhar, Alaaeldin El - Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Imagebind one embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15180--15190

    2023

  12. [20]

    Liu, Leonid Karlinsky, and James R

    Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass. 2024. Listen, think, and understand. In Proceedings of the International Conference on Learning Representations (ICLR)

    2024

  13. [21]

    Yuan Gong, Jin Yu, and James R. Glass. 2022. Vocalsound: A dataset for improving human vocal sounds recognition. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , pages 151--155

    2022

  14. [22]

    Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211

    2013 arXiv

  15. [23]

    Yash Goyal, Tejas Khot, Douglas Summers - Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), ...

    2017

  16. [24]

    Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. 2020. Stochastic weight averaging in parallel: Large-batch training that generalizes well. In Proceedings of the International Conference on Learning Representations (ICLR)

    2020

  17. [25]

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVP...

    2018

  18. [26]

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. 2024. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p...

    2024

  19. [27]

    Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. 2023. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03905

    2023 arXiv

  20. [28]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700--6709

    2019

  21. [29]

    Gabriel Ilharco, Marco T \' u lio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In Proceedings of the International Conference on Learning Representations (ICLR)

    2023

  22. [30]

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. 2023. Motiongpt: Human motion as a foreign language. In Proceedings of the Advances in neural information processing systems (NeurIPS)

    2023

  23. [31]

    Xisen Jin, Xiang Ren, Daniel Preotiuc - Pietro, and Pengxiang Cheng. 2023. Dataless knowledge fusion by merging weights of language models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2023

  24. [32]

    Junlin Lee, Yequan Wang, Jing Li, and Min Zhang. 2024. Multimodal reasoning with multimodal knowledge graph. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

    2024

  25. [33]

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji - Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19086--19096

    2022

  26. [34]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023 a . BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning (ICML), pages 19730--19742

    2023

  27. [35]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji - Rong Wen. 2023 b . Evaluating object hallucination in large vision-language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 292--305

    2023

  28. [36]

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122

    2023 arXiv

  29. [37]

    Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. 2022. Clotho-aqa: A crowdsourced dataset for audio question answering. In Proceedings of the European Signal Processing Conference (EUSIPCO), pages 1140--1144

    2022

  30. [38]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296--26306

    2024

  31. [39]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)

    2023

  32. [40]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Proceedings of the Advances in Neural Information Pr...

    2022

  33. [41]

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. 2023. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207

    2023 arXiv

  34. [42]

    Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. 2023. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.09093

    2023 arXiv

  35. [43]

    Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, and Fahad Khan. 2024. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

    2024

  36. [44]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3195--3204

    2019

  37. [45]

    Michael Matena and Colin Raffel. 2022. Merging models with fisher-weighted averaging. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)

    2022

  38. [46]

    Plumbley, Yuexian Zou, and Wenwu Wang

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. 2024. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE ACM Transactions on Audio, Speech, a...

    2024

  39. [47]

    Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen. 2017. DCASE2017 challenge setup: Tasks, datasets and baseline system. In Proceedings of the Workshop on Detection and Classification of Acoust...

    2017

  40. [48]

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. 2023. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)

    2023

  41. [49]

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. 2024. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning. In Pr...

    2024

  42. [50]

    Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. 2023. Task-specific skill localization in fine-tuned language models. In Proceedings of the International Conference on Machine Learning (ICML), pages 27011--27033

    2023

  43. [51]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21:140:1--140:67

    2020

  44. [52]

    Feiliang Ren, Longhui Zhang, Shujuan Yin, Xiaofeng Zhao, Shilei Liu, Bochao Li, and Yaduo Liu. 2021. A novel global feature-oriented relational triple extraction model based on table filling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing ...

    2021

  45. [53]

    Feiliang Ren, Longhui Zhang, Xiaofeng Zhao, Shujuan Yin, Shilei Liu, and Bochao Li. 2022. A simple but effective bidirectional framework for relational triple extraction. In Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM), page 824–832

    2022

  46. [54]

    Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zal \'a n Borsos, F \'e lix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. 2023. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925

    2023 arXiv

  47. [55]

    Zesheng Shi and Yucheng Zhou. 2023. Topic-selective graph network for topic-focused summarization. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), pages 247--259

    2023

  48. [56]

    Mustafa Shukor, Corentin Dancette, Alexandre Ram \' e , and Matthieu Cord. 2023. Unival: Unified model for image, video, audio and language tasks. Transactions on Machine Learning Research (TMLR), 2023

    2023

  49. [57]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8317--8326

    2019

  50. [58]

    Sundar, Chao - Han Huck Yang, David M

    Anirudh S. Sundar, Chao - Han Huck Yang, David M. Chan, Shalini Ghosh, Venkatesh Ravichandran, and Phani Sankar Nidadavolu. 2024. Multimodal attention merging for improved speech recognition and audio event classification. In Proceedings of the Workshop on the IEEE Internation...

    2024

  51. [59]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  52. [60]

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2024

  53. [61]

    Feng Wang, Zesheng Shi, Bo Wang, Nan Wang, and Han Xiao. 2025. Readerlm-v2: Small language model for HTML to markdown and JSON . CoRR

    2025

  54. [62]

    Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz - Jim \' e nez, Fran c ois Fleuret, and Pascal Frossard. 2024. Localizing task information for improved model merging and compression. In Proceedings of the International Conference on Machine Learning (ICML)

    2024

  55. [63]

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat - Seng Chua. 2024. Next-gpt: Any-to-any multimodal LLM . In Proceedings of the International Conference on Machine Learning (ICML)

    2024

  56. [64]

    Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1912--1920

    2015

  57. [65]

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5288--5296

    2016

  58. [66]

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. 2024. Pointllm: Empowering large language models to understand point clouds. In Proceedings of the European Conference on Computer Vision (ECCV)

    2024

  59. [67]

    Raffel, and Mohit Bansal

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. 2023. Ties-merging: Resolving interference when merging models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)

    2023

  60. [68]

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666

    2024 arXiv

  61. [69]

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics (TACL), 2:67--78

    2014

  62. [70]

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Proceedings of the International Conference on Machine Learning (ICML)

    2024

  63. [71]

    Longhui Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, and Min Zhang. 2024. A two-stage adaptation of large language models for text ranking. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

    2024

  64. [72]

    Lei Zhao, Junlin Li, Lianli Gao, Yunbo Rao, Jingkuan Song, and Heng Tao Shen. 2022. Heterogeneous knowledge network for visual dialog. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 33(2):861--871

    2022

  65. [73]

    Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. 2023. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv preprint arXiv:2305.16103

    2023 arXiv

  66. [74]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei - Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the Advances in...

    2023

  67. [75]

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Hongfa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Caiwan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. 2024 a . Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. I...

    2024

  68. [76]

    Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Chao Wu, and Kun Kuang. 2024 b . Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. In Proceedings of the International Conference on Machine Learning (ICML)

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.