REVIEW 4 major objections 6 minor 39 references
A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Wander, a low-rank sequence multimodal adapter, achieves token-level fusion across any number of modalities while training far fewer parameters than existing adapters, matching or beating full fine-tuning on benchmarks with two, three…
desk verdict A clean extension of LMF to sequence-level fusion, but the empirical evidence is too thin to support the 'consistent SOTA' claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central identity is Equation 13, which factorizes the sequence-level interaction tensor through CP (CANDECOMP/PARAFAC) decomposition. Instead of materializing the full M-modal outer product tensor, the output is expressed as an element-wise (Hadamard) product across modalities of sums of rank-one terms, each term being a per-modality low-rank matrix multiplied with the sequence features. This turns an exponential parameter and memory cost into one that is linear in the number of modalities, and it is the mechanism by which Wander achieves token-level interactions among any number of modalities with few trainable parameters.
What would settle it
On a three-modality benchmark, compute the exact sequence-interaction tensor of Equation 9 with reduced dimensions and measure the relative Frobenius-norm error of its best rank-8 CP approximation; if the error is large while downstream accuracy stays high, the method would be relying on something other than faithful low-rank reconstruction of the interaction tensor, and if the error is large and accuracy drops sharply at higher ranks, the low-rank assumption itself would fail.
Extended reading notes
Core claim
The paper claims that Wander performs fine-grained, token-level fusion of multimodal sequences in a parameter-efficient way, and that this generalizes to more than two modalities. Starting from the outer-product fusion of unimodal representations, Wander factorizes both the feature-projection weight and the sequence-interaction weight via CP decomposition into sums of rank-one tensors. The key result is Equation 13, which rewrites the fused output as an element-wise (Hadamard) product across modalities of sums of low-rank terms, eliminating the need to materialize the exponential interaction tensor. Empirically, the paper reports that Wander outperforms state-of-the-art efficient transfer learning methods on UPMC-Food 101 (2 modalities), CMU-MOSI and IEMOCAP (3 modalities), and MSRVTT (7 modalities), with fewer trainable parameters than competing adapters and, on CMU-MOSI and MSRVTT, better accuracy than full fine-tuning.
Load-bearing premise
The method assumes the information needed for downstream tasks lives in a low-rank subspace of the full inter-modality token-interaction tensor, so a rank-8 CP factorization preserves it.
Editorial extensions
If this is right
- Wander can be added to any frozen Transformer-based multimodal backbone and trained with only the task head, regardless of the number of input modalities.
- On the tested benchmarks, Wander matches or beats full fine-tuning while using roughly one to two orders of magnitude fewer trainable parameters, for example 0.9 million versus 80 million on CMU-MOSI.
- Because the interaction tensor is never materialized, the method avoids the exponential memory blow-up of explicit outer-product fusion, making it feasible for many modalities.
- The CP rank can be kept small (the default is 8) with little performance change, so the parameter savings persist as the number of modalities grows.
Reading between the lines
- The same CP-factorized Hadamard-product identity could be applied to other high-order interaction layers in Transformers, such as factorizing attention across multiple input sequences, not just multimodal fusion.
- A natural next step is a formal bound on the approximation error of the rank-R factorization as a function of R and the modality dimensions; the paper only provides empirical rank-sensitivity results.
- Because the fusion treats all modalities symmetrically, it may suit settings with missing modalities or dynamically changing input types, which the paper does not test.
- The reported gains on the seven-modality MSRVTT benchmark suggest the efficiency advantage grows with modality count, so a systematic scaling study would quantify the savings beyond seven modalities.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes Wander, a parameter-efficient adapter for fine-tuning multimodal models with an arbitrary number of modalities. The authors first observe that existing multimodal adapters are limited to vision-language pairs and to vector-level fusion. They then propose a sequence-level fusion that applies an outer product to token sequences and factorizes the resulting high-order interaction tensor with a low-rank decomposition. The core derivation in Section 3.3, Eq. (13), shows that the materialization of the full outer product can be avoided, yielding a parameter count that grows linearly in the number of modalities. Experiments on UPMC-Food 101, CMU-MOSI, IEMOCAP, and MSRVTT report that Wander outperforms several efficient transfer learning baselines and, in some cases, matches or exceeds full fine-tuning.
Significance. If the results are reproducible, the paper makes a useful contribution: it generalizes multimodal efficient transfer learning beyond two modalities, introduces a token-level interaction mechanism that is more fine-grained than vector-level fusion, and provides a factorized form (Eq. 13) that avoids explicitly constructing the high-order tensor. The algebraic reduction is a genuine strength, and the paper evaluates on datasets with 2, 3, and 7 modalities, which is broader than many adapter papers. The parameter-efficiency argument is plausible for large numbers of modalities. However, the empirical support for the headline claim is currently weakened by the lack of repeated runs and by inconsistencies in the reported parameter counts, and the terminology 'CP decomposition' needs to be aligned with the actual factorization used.
major comments (4)
- [Section 3.3, Eqs. (5)-(6)] The method is described as using CP decomposition, but the factors w^r_{h,m} in R^{dh x dm} are matrices, not vectors, and the equality W_h = sum_r tensorprod_m w^r_{h,m} does not correspond to a standard CP decomposition of the (M+1)-way tensor W_h. The construction is a per-output-slice rank-R factorization, with parameter count R * dh * sum_m d_m rather than the CP count R * (dh + sum_m d_m). Please correct the terminology and complexity statements, and clarify the meaning of Rh and Rt in Figure 5 in light of this distinction.
- [Section 4.2, Table 3, and Section 4.3, Table 5] The reported tunable-parameter counts for the same Wander configuration are inconsistent. Table 3 lists Wander(d=16) with 0.3M parameters on CMU-MOSI, whereas Table 5 lists Wander(d=16) with 0.1M; Table 5 also lists Wander(d=32) with 0.1M, equal to d=16, and the '-nonlinearity' and '-residual' rows with 0.2M, which is larger than the complete model. Please unify the counting convention (adapter-only vs. including the prediction head) and correct the numbers, since the parameter-efficiency claim depends on them.
- [Section 4.2, Tables 1-4] The claim that Wander 'outperforms state-of-the-art efficient transfer learning methods consistently' rests on single point estimates. Several decisive margins are small: on UPMC-Food 101, Wander(d=64) is 91.1 vs. UniAdapter 90.8; on IEMOCAP, Wander(d=64) is 74.7 vs. full fine-tuning 74.8; on MSRVTT, Wander(d=16) ties full fine-tuning on Text-to-Video R@5 at 57.2. For datasets of these sizes, 0.1-1.0 point differences are typically within seed-to-seed noise. Please report means and standard deviations over multiple seeds, or confidence intervals, for at least the main tables, and temper the consistency claim accordingly.
- [Section 4.5, Figure 5] The rank sensitivity analysis is the only evidence for the default choice Rh=Rt=8, but it is based on single runs and appears to show non-negligible variation between ranks 4 and 12, particularly for Rt. The statement that 'the value of the rank will not affect the performance significantly' is not supported by the data as presented. Please provide repeated runs for the rank sweep and, ideally, a measure of the approximation error of the factorized interaction tensor, to justify the default rank.
minor comments (6)
- [Section 3.3, Eq. (13)] The tensor contraction and the 'transpose' of a multi-index expression are not defined; please specify the index order for the tensor product and for the transposition to make the derivation unambiguous.
- [Section 4.3 and Table 5] SF-VF is described as 'the vector fusion form of sequence fusion in Equation 8', but Equation 8 is the per-token vector fusion; please clarify what sequence-level aggregation is used for the SF-VF baseline.
- [Figure 4] The x-axis labels appear garbled and the values are not legible; please replace the figure with one having clear axis labels and units.
- [Section 4.2] The modifications to UniAdapter, MaPLe, and PMF for the non-vision-language backbones and the more-than-two-modality settings are described only briefly; please provide details on how each baseline was adapted to ensure a fair comparison.
- [Section 4.4 and Table 6] It is unclear whether the GPU time and memory figures are for the reduced-dimension settings described in Section 4.3, and the FLOPs unit is not stated; please clarify the experimental conditions for the cost comparison.
- [General] The paper does not mention code release; given the parameter-count inconsistencies and the small performance margins, releasing code and training configurations would substantially aid reproducibility.
Circularity Check
No circularity found: Wander's low-rank sequence fusion is a mathematical reparameterization of the outer-product contraction (Eq. 13), and the reported gains rest on external empirical comparisons rather than on the model's own definitions.
full rationale
The paper's central derivation, Equations 8-13, is an algebraic reparameterization: the high-dimensional outer-product tensors in Equations 9-10 are rewritten, via standard CP decomposition, as sums of rank-one factors so that the full outer product is never materialized. No target quantity is defined in terms of the model's output, and no fitted parameter is later relabeled as a prediction. The choice of rank R_h=R_t=8 and down-projection dimension d is empirical hyperparameter selection; Figure 5 provides sensitivity analysis, and the headline 'outperforms state-of-the-art' is an empirical claim backed by comparisons on four external datasets, not a consequence entailed by the factorization. The only overlapping-author citations (Guo, Jin, and Zhao 2024; Yan et al. 2024) appear in the related-work survey of prompt-based methods and are not load-bearing for the proposed architecture or its results. The method is also not a pure rename of the cited low-rank multimodal fusion (Liu et al. 2018), since it extends vector-level fusion to token-level sequence fusion with an additional factorized sequence tensor W_t. Concerns about single-run point estimates, missing confidence intervals, and dataset reuse are evaluation-quality issues, not circularity under the definition used here. The derivation chain is self-contained, so no circular step is identified.
Assumptions & free parameters
free parameters (3)
- CP rank R (Rh and Rt) =
8 (default)
- Down-projection dimension d =
16, 32, 64, 128
- Optimizer hyperparameters =
learning rates 1e-3 to 3e-3, step sizes, decay rates
assumptions (2)
- domain assumption The multimodal interaction tensor can be well-approximated by a rank-R CP decomposition with small R.
- domain assumption Frozen pre-trained backbones provide transferable features for all modalities.
Cite this review
Pith. "Pith review of A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter." pith.science (2026). https://pith.science/paper/BBH63TXU
@misc{pith2026241208979,
author = {Pith},
title = {Pith review of: A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter},
year = {2026},
howpublished = {\url{https://pith.science/paper/BBH63TXU}},
note = {Machine review of arXiv:2412.08979}
}
read the original abstract
Efficient transfer learning methods such as adapter-based methods have shown great success in unimodal models and vision-language models. However, existing methods have two main challenges in fine-tuning multimodal models. Firstly, they are designed for vision-language tasks and fail to extend to situations where there are more than two modalities. Secondly, they exhibit limited exploitation of interactions between modalities and lack efficiency. To address these issues, in this paper, we propose the loW-rank sequence multimodal adapter (Wander). We first use the outer product to fuse the information from different modalities in an element-wise way effectively. For efficiency, we use CP decomposition to factorize tensors into rank-one components and achieve substantial parameter reduction. Furthermore, we implement a token-level low-rank decomposition to extract more fine-grained features and sequence relationships between modalities. With these designs, Wander enables token-level interactions between sequences of different modalities in a parameter-efficient way. We conduct extensive experiments on datasets with different numbers of modalities, where Wander outperforms state-of-the-art efficient transfer learning methods consistently. The results fully demonstrate the effectiveness, efficiency and universality of Wander.
Figures
Reference graph
Works this paper leans on
-
[1]
Busso, C.; Bulut, M.; Lee, C.-C.; Kazemzadeh, E. A.; Provost, E. M.; Kim, S.; Chang, J. N.; Lee, S.; and Narayanan, S. S. 2008. IEMOCAP: interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42: 335--359
work page 2008
-
[2]
Chen, Y.; Qian, S.; Tang, H.; Lai, X.; Liu, Z.; Han, S.; and Jia, J. 2024. LongLo RA : Efficient Fine-tuning of Long-Context Large Language Models. In The Twelfth International Conference on Learning Representations
work page 2024
-
[3]
Dettmers, T.; Pagnoni, A.; Holtzman, A.; and Zettlemoyer, L. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36
2024
-
[4]
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
arXiv 2018
-
[5]
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929
arXiv 2020
-
[6]
Gabeur, V.; Sun, C.; Alahari, K.; and Schmid, C. 2020. Multi-modal Transformer for Video Retrieval . In European Conference on Computer Vision (ECCV)
work page 2020
-
[7]
Gao, P.; Geng, S.; Zhang, R.; Ma, T.; Fang, R.; Zhang, Y.; Li, H.; and Qiao, Y. 2024. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2): 581--595
2024
-
[8]
Guo, Z.; Jin, T.; and Zhao, Z. 2024. Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1726--1736
work page 2024
Show all 39 references
-
[9]
Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019. Parameter-efficient transfer learning for NLP. In International conference on machine learning, 2790--2799. PMLR
2019
-
[10]
J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W
Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685
2021 arXiv
-
[11]
Joze, H. R. V.; Shaban, A.; Iuzzolino, M. L.; and Koishida, K. 2020. MMTM: Multimodal transfer module for CNN fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 13289--13299
2020
-
[12]
U.; Rasheed, H.; Maaz, M.; Khan, S.; and Khan, F
Khattak, M. U.; Rasheed, H.; Maaz, M.; Khan, S.; and Khan, F. S. 2023. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19113--19122
2023
-
[13]
Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023 a . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 19730--19742. PMLR
2023
-
[14]
Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, 12888--12900. PMLR
2022
-
[15]
L.; and Liang, P
Li, X. L.; and Liang, P. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190
2021 arXiv
-
[16]
Li, Y.; Quan, R.; Zhu, L.; and Yang, Y. 2023 b . Efficient multimodal fusion via interactive prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2604--2613
2023
-
[17]
Li, Y.; Yu, Y.; Liang, C.; Karampatziakis, N.; He, P.; Chen, W.; and Zhao, T. 2024. LoftQ: Lo RA -Fine-Tuning-aware Quantization for Large Language Models. In The Twelfth International Conference on Learning Representations
2024
-
[18]
Liang, T.; Xie, H.; Yu, K.; Xia, Z.; Lin, Z.; Wang, Y.; Tang, T.; Wang, B.; and Tang, Z. 2022. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 35: 10421--10434
2022
-
[19]
Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2024. Visual instruction tuning. Advances in neural information processing systems, 36
2024
-
[20]
Liu, X.; Zheng, Y.; Du, Z.; Ding, M.; Qian, Y.; Yang, Z.; and Tang, J. 2023 a . GPT understands, too. AI Open
2023
-
[21]
B.; Liang, P
Liu, Z.; Shen, Y.; Lakshminarasimhan, V. B.; Liang, P. P.; Zadeh, A.; and Morency, L.-P. 2018. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064
2018 arXiv
-
[22]
L.; and Han, S
Liu, Z.; Tang, H.; Amini, A.; Yang, X.; Mao, H.; Rus, D. L.; and Han, S. 2023 b . Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In 2023 IEEE international conference on robotics and automation (ICRA), 2774--2781. IEEE
2023
-
[23]
Lu, H.; Huo, Y.; Yang, G.; Lu, Z.; Zhan, W.; Tomizuka, M.; and Ding, M. 2024. UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling. In The Twelfth International Conference on Learning Representations
2024
-
[24]
Miech, A.; Zhukov, D.; Alayrac, J.-B.; Tapaswi, M.; Laptev, I.; and Sivic, J. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, 2630--2640
2019
-
[25]
P \'e rez-R \'u a, J.-M.; Vielzeuf, V.; Pateux, S.; Baccouche, M.; and Jurie, F. 2019. Mfas: Multimodal fusion architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6966--6975
2019
-
[26]
Sung, Y.-L.; Cho, J.; and Bansal, M. 2022. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5227--5237
2022
-
[27]
H.; Bai, S.; Liang, P
Tsai, Y.-H. H.; Bai, S.; Liang, P. P.; Kolter, J. Z.; Morency, L.-P.; and Salakhutdinov, R. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, 6558. NIH Pu...
2019
-
[28]
Vu, T.; Lester, B.; Constant, N.; Al-Rfou, R.; and Cer, D. 2021. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint arXiv:2110.07904
2021 arXiv
-
[29]
Wang, H.; Yang, X.; Chang, J.; Jin, D.; Sun, J.; Zhang, S.; Luo, X.; and Tian, Q. 2024. Parameter-efficient tuning of large-scale multimodal foundation model. volume 36
2024
-
[30]
Wang, X.; Kumar, D.; Thome, N.; Cord, M.; and Precioso, F. 2015. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia Expo Workshops (ICMEW), 1--6
2015
-
[31]
Xing, Y.; Wu, Q.; Cheng, D.; Zhang, S.; Liang, G.; Wang, P.; and Zhang, Y. 2023. Dual modality prompt tuning for vision-language pre-trained model. IEEE Transactions on Multimedia
2023
-
[32]
Xu, J.; Mei, T.; Yao, T.; and Rui, Y. 2016. MSR-VTT: A Large Video Description Dataset for Bridging Video and Language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 5288--5296
2016
-
[33]
Yan, W.; Wang, Y.; Lin, W.; Guo, Z.; Zhao, Z.; and Jin, T. 2024. Low-rank Prompt Interaction for Continual Vision-Language Retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, 8257--8266
2024
-
[34]
Zadeh, A.; Chen, M.; Poria, S.; Cambria, E.; and Morency, L.-P. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250
2017 arXiv
-
[35]
Zadeh, A.; Zellers, R.; Pincus, E.; and Morency, L.-P. 2016. Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages. IEEE Intelligent Systems, 31: 82--88
2016
-
[36]
Zang, Y.; Li, W.; Zhou, K.; Huang, C.; and Loy, C. C. 2022. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225
2022 arXiv
-
[37]
Zhang, R.; Fang, R.; Zhang, W.; Gao, P.; Li, K.; Dai, J.; Qiao, Y.; and Li, H. 2021. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930
2021 arXiv
-
[38]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[39]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.