Pith. sign in

REVIEW 5 major objections 4 minor 2 cited by

PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PUMA claims that layer pruning a 7B multimodal retriever to 3B, with self-distillation from the removed layers, keeps M-BEIR recall within 1.8 points while doubling throughput.

desk verdict Solid efficiency contribution for MLLM-based unified retrieval, but the headline 1.8-point gap is unauditable until per-dataset numbers, seeds, and one consistent FLOPs figure are provided; still worth a serious referee. read the letter →

arxiv 2507.08064 v4 pith:LJBX42KQ submitted 2025-07-10 cs.MM cs.CV

Yibo Lyu , Rui Shao , Gongwei Chen , Yijie Zhu , Weili Guan , Liqiang Nie This is my paper · ORCID
classification cs.MMcs.CV
keywords unifiedmultimodalretrievallayerpruningself-distillationcontrastivelearningmodality-adaptiveM-BEIRbenchmarkMLLMefficiencyembedding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a 7B-class multimodal large language model can be made practical for unified multimodal retrieval by cutting it down to its first 12 of 28 transformer layers, then using self-distillation from the removed layers to repair whatever the cut damages. The resulting 3B model keeps an average gap of only 1.8 Recall points to the full 7B model on the M-BEIR benchmark, while cutting FLOPs by roughly half, using about 4x less GPU memory, and raising inference throughput by 95.8%. The paper also claims a modality-adaptive contrastive loss that separates intra- and inter-modality negatives and assigns different temperatures recovers additional accuracy, especially when batch size is small. A sympathetic reader should take away that efficient MLLM-based retrieval is possible without abandoning the representational quality that motivates using an LLM for retrieval in the first place.

What carries the argument

The load-bearing object is the hidden state of the special [RET] token in the retrieval prompt. Layer-pruned self-distillation uses the original model's last-layer [RET] state as teacher $\mathbf{h}_t$ and the pruned model's layer-$k$ state as student $\mathbf{h}_s$, minimizing $\|\mathbf{h}_t-\mathbf{h}_s\|_2^2$ jointly with a contrastive loss. MAC-Loss is the second mechanism: it rewrites the InfoNCE denominator so that negatives with the same modality as the query's target candidate form an intra-modality group scored at temperature $\tau_{\mathrm{hard}} = \tau_0 e^{-\lambda t}$, while inter-modality negatives stay at $\tau_0$, steering the model toward the harder same-modality negatives without extra sampling cost. Together they carry the argument that pruning cost can be repaid by distillation signal and by a loss that makes limited batches harder.

What would settle it

Train the $k=12$ pruned student on M-BEIR with the same contrastive and self-distillation losses, then ablate only the distillation term; if Recall@k collapses on a specific task such as WebQA or OVEN relative to the 7B teacher, that task's required semantics live in the dropped layers and the shallow-layer transfer claim would be falsified for it.

Watch

Extended reading notes

Core claim

The central claim is that the retrieval-relevant information in an MLLM is concentrated early: after contrastive fine-tuning of Qwen2-VL, the [RET] token's attention to image and text tokens is dense and informative in the first dozen layers, so the deep layers can be discarded. PUMA's layer-pruned self-distillation then makes the shallow student reproduce the teacher's final-layer [RET] hidden state, letting a pruned 3B model inherit the teacher's representation while jointly training with InfoNCE. On M-BEIR, the pruned model beats CLIP-based supervised retrievers by 3.6 points and the sub-4B LamRA-Ret baseline by 2.3 points, and it stays within 1.8 average points of the full 7B model while using half the FLOPs. The second claimed component, MAC-Loss, partitions in-batch negatives by whether they share the query's target candidate modality and applies a decaying temperature to the harder intra-modality group, which the ablations credit with most of the remaining gain under constrained batch sizes.

Load-bearing premise

The load-bearing premise is that the first 12 layers of Qwen2-VL contain essentially all retrieval-relevant multimodal information, so the removed deep layers contribute little to the [RET] embedding unless some M-BEIR task relies on semantics that only deep layers encode.

Editorial extensions

If this is right

  • MLLM-based unified retrieval becomes deployable at roughly 3B scale: half the FLOPs, a quarter of the GPU memory, and twice the inference throughput for a 1.8-point average Recall cost on M-BEIR.
  • The pruning recipe transfers across architectures, since the supplementary LLaVA-v1.5 experiment shows a 57.8% FLOP reduction with most capability retained.
  • MAC-Loss strengthens contrastive training even when batch size is small, which reduces the GPU requirement for instruction tuning of retrieval models.
  • Layer pruning accelerates text-only retrieval as well as image retrieval, unlike token-compression methods that only shorten visual token sequences.
  • The combined techniques recover 1.3 average points over the pruned base, so both components are needed to close most of the gap to the 7B teacher.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would expect the 1.8-point average gap to be unevenly distributed: perceptual duplicate tasks such as NIGHTS should lose almost nothing, while knowledge-heavy tasks such as WebQA, OVEN, and InfoSeek are the likeliest places where deep-layer semantics are actually needed.
  • Because MAC-Loss needs no extra candidates, it should transfer as a drop-in replacement for InfoNCE in any small-batch multimodal contrastive training, even when no layer pruning is used; the paper does not test that setting.
  • Layer pruning and image-token compression attack different bottlenecks, so stacking them on image-heavy tasks should compound the speedup; the paper only compares them rather than combining them.
  • The paper's observation of a sharp similarity drop for geography-style OVEN queries suggests unlabeled in-batch positives could be mined from near-duplicate candidates and used as extra supervision for retrieval-augmented generation, an application the authors gesture at but do not develop.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper proposes PUMA, a method for unified multimodal retrieval that combines layer-pruned self-distillation with a modality-adaptive contrastive loss (MAC-Loss). The authors prune Qwen2-VL 7B to its first 12 layers (yielding a 3B model), use the hidden state of the [RET] token from the original full model as a teacher signal during pre-training, and during instruction tuning separate in-batch negatives into intra- and inter-modality groups with different temperatures. Evaluated on M-BEIR, PUMA is reported to outperform the <4B LamRA-Ret baseline and to come within roughly 1.8 average Recall points of the 7B LamRA-Ret model while using about half the FLOPs and nearly double the inference throughput. The paper also reports ablations of the distillation and MAC-Loss components, a comparison with FastV token compression, and experiments on LLaVA in the appendix.

Significance. If the reported trade-off were fully documented, the paper would make a practical contribution: it shows a simple layer-pruning recipe plus feature-level self-distillation can turn a 7B MLLM retriever into a 3B model with modest Recall loss, and MAC-Loss is a cheap in-batch hard-negative weighting scheme that requires no additional sampling or larger batches. The paper ships pseudocode and a code link, evaluates against external baselines on M-BEIR, and does not claim parameter-free derivations; the free hyperparameters (k, alpha1/alpha2, lambda, tau0, LoRA rank) are explicitly used. The main risk is that the central efficiency/accuracy numbers are not currently auditable because of missing per-dataset 7B results, missing variance estimates, and an internal FLOPs inconsistency.

major comments (5)
  1. [§5.2, Table 2] The paper reports a 57.3% FLOPs reduction in the main text, but Table 2 shows FLOPs of 3.48 for PUMA versus 7.36 for LamRA-Ret, which is a 52.7% reduction. Since the FLOPs figure is central to the efficiency claim, the discrepancy must be resolved and the calculation (including whether the teacher forward pass is counted) stated explicitly.
  2. [§5.2, Table 2] The claim that PUMA maintains 7B performance 'within 1.8 points across all datasets' is not auditable: Table 2 reports only three grouped averages for the 7B baseline and for PUMA, and it does not give per-dataset Recall values for the 7B model or any measure of variance. Because the stated gap is comparable to seed-to-seed variation in contrastive retrieval training on a 5.6M-candidate benchmark, please provide per-dataset results for the 7B baseline and multiple seeds (or error bars) for both models, and state the exact aggregation rule used to obtain the 1.8-point figure.
  3. [Tables 1 and 2] The grouped averages in Table 2 do not reproduce the per-dataset values in Table 1 under the group definitions in Section 5.1: for LamRA-Ret the per-dataset Single/Mixed/Multi averages are approximately 54.3/44.0/54.6, while Table 2 reports 53.6/55.2/69.8; for PUMA the corresponding averages are approximately 56.5/46.2/56.8, while Table 2 reports 51.3/52.6/69.6. Please clarify the relationship between the <4B LamRA-Ret in Table 1 and the 7B LamRA-Ret in Table 2, and specify how the grouped scores are computed.
  4. [§4.2, §5.1] The self-distillation setup is underspecified in a way that affects the efficiency interpretation: the paper does not state whether the teacher (full-model) forward pass is run for every training batch, whether the teacher is frozen, or where LoRA adapters are placed relative to the pruned layers. If the teacher forward is performed at each step, the training FLOPs are not reduced by the reported factor even if inference FLOPs are; please specify the exact implementation and report training cost accordingly.
  5. [§4.1, Figure 3] The justification for pruning to the first 12 layers rests on a qualitative attention visualization and on prior findings from VQA, but no direct evidence is given that retrieval-relevant information is absent from deeper layers of Qwen2-VL after UMR training. Because the efficiency/accuracy trade-off depends on this assumption, please add a layer-wise study of the pruned model without self-distillation (or a probing experiment) to show where retrieval-relevant information is actually lost.
minor comments (4)
  1. [§5.4] The text refers to 'Table 5 shows the results' when the comparison with FastV is in Table 6; please correct the cross-reference.
  2. [§4.3, Algorithm 1] Equation (8) defines the temperature decay as a function of the current iteration t, while Algorithm 1 decays the temperature on the ratio current_epoch/total_epochs; these two descriptions should be aligned.
  3. [§5.4] The sentence stating that PUMA requires 'fewer than 0.62 FLOPs' should be reworded to '0.62x the FLOPs' or 'a 38% reduction in FLOPs' to avoid an incorrect unit.
  4. [§5.3, Table 3] The text says the combined techniques give an average performance increase of 1.3 points, but from Table 3 the average increase is approximately 1.4 points (49.2 to 50.6); please state the aggregation used.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the paper's claims are empirical and benchmarked externally.

full rationale

The paper does not claim a parameter-free derivation; its contributions are an architecture (layer pruning with self-distillation) and a loss (MAC-Loss), and its evidence is external retrieval scores on M-BEIR, Flickr30k, and COCO. The load-bearing premise that shallow layers carry retrieval-relevant information is supported by external interpretability studies [5,13,51,68,69] and by the paper's own attention visualization (Figure 3); it is a design assumption rather than a result derived from the method's equations, so it is not circular. The self-distillation loss (Eq. 2) aligns the shallow [RET] hidden state with the original model's last-layer hidden state, but retrieval performance is measured independently on held-out benchmark tasks, so the teacher signal does not by construction force the reported Recall numbers. MAC-Loss (Eq. 7) is a modified InfoNCE objective with modality-dependent temperatures; its benefit is established by ablations on external metrics, not by construction. The only self-referential elements are related-work citations to the authors' own token-compression paper [66] and the observation-based selection of k=12 on the same benchmark; neither is load-bearing or a fitted input renamed as a prediction. The conclusion's admission that single-modal tasks do not clearly beat CLIP is a limitation, not a circular step. The reviewer's concern about missing per-dataset 7B numbers is an auditability or statistical issue, not circularity. Score 1 reflects minor non-load-bearing self-citations.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. The method relies on several domain assumptions, most notably the sufficiency of shallow layers for retrieval and the hardness ordering of intra- versus inter-modality negatives. The free parameters are standard training hyperparameters, with k, alpha, and lambda directly affecting the reported results.

free parameters (5)
  • k (number of retained shallow layers) = 12
    Chosen by balancing performance and efficiency; paper states 'pruning k=12 layers yields saturated performance' (Section 5.1, Figure 5).
  • alpha1 (contrastive weight) / alpha2 (self-distill weight) = 0.9 / 0.1
    Set in Section 5.1; Table 4 explores dynamic schedules but the main model uses the fixed ratio.
  • lambda (MAC-Loss decay sparsity) = 0.5
    Hyperparameter for temperature decay in Eq. 8; no ablation over lambda is reported.
  • tau0 (normal contrastive temperature) = not specified
    Base temperature in Eq. 8; value not given in the paper, assumed inherited from prior work.
  • LoRA rank/alpha = r=128, alpha=256
    Model capacity hyperparameters from Section 5.1.
assumptions (4)
  • domain assumption Shallow layers of MLLMs capture retrieval-relevant multimodal fusion; deep layers mainly serve next-token prediction.
    Adopted in Section 4.1 from VQA-focused studies [5,13,63,69] and extended to UMR based on qualitative attention maps (Figure 3).
  • domain assumption In-batch negatives whose modality equals the positive candidate's modality are harder than inter-modality negatives.
    Basis of MAC-Loss (Section 4.3, Eq. 6-7); supported by a t-SNE plot (Figure 4) but not by a quantitative hardness measure.
  • domain assumption The last-layer [RET] hidden state of the full teacher model is a useful supervision target for the shallow student.
    Assumed in Eq. 2 (self-distillation); the paper does not report the teacher's own retrieval accuracy or feature analysis.
  • standard math InfoNCE contrastive loss with cosine similarity is an appropriate objective for UMR.
    Standard approach, consistent with LamRA and MMEmbed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning." pith.science (2026). https://pith.science/paper/LJBX42KQ

@misc{pith2026250708064,
  author       = {Pith},
  title        = {Pith review of: PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LJBX42KQ}},
  note         = {Machine review of arXiv:2507.08064}
}
read the original abstract

As multimedia content expands, the demand for unified multimodal retrieval (UMR) in real-world applications increases. Recent work leverages multimodal large language models (MLLMs) to tackle this task. However, their large parameter size results in high training costs and low inference efficiency. To address this, we propose PUMA: a Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning. Our approach improves UMR from both structural and learning perspectives. (1) Structurally, we propose Layer-Pruned Self-Distillation, which prunes MLLMs by keeping only shallow layers while distilling features from dropped deep layers as teacher signals. This reduces parameters and preserves representation capability. (2) On the learning side, we introduce Modality-Adaptive Contrastive Learning Loss (MAC-Loss), which separates in-batch negatives into harder intra-modality and easier inter-modality groups based on the target modality, assigning different temperature strategies to enhance learning efficiency. Experiments show our method significantly reduces resource usage while maintaining strong performance.

Figures

Figures reproduced from arXiv: 2507.08064 by the authors.

Figure 1
Figure 1. The pipeline of PUMA. We propose an efficient [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the PUMA framework. Our method comprises two key components from both the model architecture [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the attention map of retrieval em [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Visualization of data distribution. We use t-SNE for [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Performance visualization across layers. We display [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Some qualitative results. We present visualizations of representative cases from different retrieval tasks. The gray box [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Visualization of the attention weights from the [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Visualization of Quantitative Results. We present visualizations of some representative examples to qualitatively [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval

    cs.IR 2026-06 unverdicted novelty 6.0 of 10

    ELVA uses rule-based RL rewards to rank negatives by similarity, reducing grain blindness in universal multimodal retrieval and reporting a 13.1% gain on a new multi-grain benchmark.

  2. FreeRet: MLLMs as Training-Free Retrievers

    cs.CV 2025-09 unverdicted novelty 6.0 of 10

    FreeRet enables pretrained MLLMs to act as training-free retrievers via semantically grounded embeddings and reasoning-based reranking, outperforming models trained on millions of pairs on MMEB benchmarks.

Reference graph

Works this paper leans on

75 extracted references · 35 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. 2024. Lion: Empowering multimodal large language model with dual-level visual knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26540–26550

    work page 2024

  2. [2]

    Gongwei Chen, Xurui Zhou, Rui Shao, Yibo Lyu, Kaiwen Zhou, Shuai Wang, Wentao Li, Yinchuan Li, Zhongang Qi, and Liqiang Nie. 2025. Less is More: Empowering GUI Agent with Context-Aware Simplification. InProceedings of the IEEE/CVF International Conference on Computer Vision

    work page 2025

  3. [3]

    Junya Chen, Zhe Gan, Xuan Li, Qing Guo, Liqun Chen, Shuyang Gao, Tagyoung Chung, Yi Xu, Belinda Zeng, Wenlian Lu, et al. 2021. Simpler, faster, stronger: Breaking the log-k curse on contrastive learners with flatnce. arXiv preprint arXiv:2107.01152 (2021)

    arXiv 2021

  4. [4]

    Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu

  5. [5]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision. Springer, 19–35

    2024

  6. [6]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Inter- national conference on machine learning . PmLR, 1597–1607

    2020

  7. [7]

    Xinlei Chen, Saining Xie, and Kaiming He. 2021. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision . 9640–9649

    2021

  8. [8]

    Abrar Fahim, Alex Murphy, and Alona Fyshe. 2024. It’s Not a Modality Gap: Char- acterizing and Addressing the Contrastive Gap. arXiv preprint arXiv:2405.18570 (2024)

    arXiv 2024

Show all 75 references
  1. [9]

    Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. 2024. Not all layers of llms are necessary during inference. arXiv preprint arXiv:2403.02181 (2024)

    2024 arXiv

  2. [10]

    Tim Fischer, Chris Biemann, et al. 2024. Large language models are overparame- terized text encoders. arXiv preprint arXiv:2410.14578 (2024)

    2024 arXiv

  3. [11]

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . 6894–6910

    2021

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, et al . 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  5. [13]

    Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. [n. d.]. The unreasonable ineffectiveness of the deeper layers,

  6. [14]

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9729–9738

    2020

  7. [15]

    org/abs/2403.17887 ([n

    URL https://arxiv. org/abs/2403.17887 ([n. d.])

    arXiv

  8. [16]

    Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. 2023. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Com...

    2023

  9. [17]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3

    2022

  10. [18]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision- language representation learning with noisy text supervision. In International conference on machine learning . PMLR, 4904–4916

    2021

  11. [19]

    Lang Huang, Qiyu Wu, Zhongtao Miao, and Toshihiko Yamasaki. 2025. Joint Fusion and Encoding: Advancing Multimodal Retrieval from the Ground Up. arXiv preprint arXiv:2502.20008 (2025)

    2025 arXiv

  12. [20]

    Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. 2024. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580 (2024)

    2024 arXiv

  13. [21]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

    2024 arXiv

  14. [22]

    Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. 2020. Hard negative mixing for contrastive learning. Advances in neural information processing systems 33 (2020), 21798–21809

    2020

  15. [23]

    Omri Kaduri, Shai Bagon, and Tali Dekel. 2024. What’s in the Image? A Deep- Dive into the Vision of Vision Language Models. arXiv preprint arXiv:2411.17491 (2024)

    2024 arXiv

  16. [24]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

    2024 arXiv

  17. [25]

    Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV) . 201–216

    2018

  18. [26]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742

    2023

  19. [27]

    Hao Li, Qi Lv, Rui Shao, Xiang Deng, Yinchuan Li, Jianye HAO, and Liqiang Nie. [n. d.]. STAR: Learning Diverse Robot Skill Abstractions through Rotation- Augmented Vector Quantization. In Forty-second International Conference on Machine Learning

  20. [28]

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language repre- sentation learning with momentum distillation. Advances in neural information processing systems 34 (2021), 9694–9705

    2021

  21. [29]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning. PMLR, 12888–12900

    2022

  22. [30]

    Yinchuan Li, Xinyu Shao, Jianping Zhang, Haozhi Wang, Leo Maxime Brunswic, Kaiwen Zhou, Jiqian Dong, Kaiyang Guo, Xiu Li, Zhitang Chen, et al . 2025. Generative models in decision making: A survey. arXiv preprint arXiv:2502.17100 (2025)

    2025

  23. [31]

    Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nie. 2025. Lion-fs: Fast & slow video-language thinker as online video assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference . 3240–3251

    2025

  24. [32]

    Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. 2025. Optimus-2: Multimodal minecraft agent with goal-observation-action conditioned policy. In Proceedings of the Computer Vision and Pattern Recognition Conference. 9039–9049

    2025

  25. [33]

    Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. 2024. Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks. In Advances in Neural Information Processing Systems , Vol. 37. 49881–49913

    2024

  26. [34]

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. 2022. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems 35 (2022), 17612–17625

    2022

  27. [35]

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281 (2023)

    2023 arXiv

  28. [36]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proce...

    2014

  29. [37]

    Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catan- zaro, and Wei Ping. 2025. MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS. InThe Thirteenth International Conference on Learn- ing Representations. https://openreview.net/forum?id=i45NQb2iKO

    2025

  30. [38]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  31. [39]

    Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang, Zecheng Wang, Fangying Rong, Qingbin Liu, Yanchao Hao, Bo Li, Xi Chen, et al. 2024. Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging. InProceedings of the 2024 Conference on Empirical Methods in Na...

    2024

  32. [40]

    Yikun Liu, Yajie Zhang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yan- feng Wang, and Weidi Xie. 2025. Lamra: Large multimodal model as your advanced retrieval assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference. 4015–4025

    2025

  33. [41]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning. Advances in neural information processing systems 36 (2023), 34892–34916

    2023

  34. [42]

    Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive Text Embedding Benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics . 2014–2037

    2023

  35. [43]

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing 508 (2022), 293–304

    2022

  36. [44]

    Renjing Pei, Jianzhuang Liu, Weimian Li, Bin Shao, Songcen Xu, Peng Dai, Juwei Lu, and Youliang Yan. 2023. Clipping: Distilling clip-based models with a student base for video-language retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2023

  37. [45]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

    2018 arXiv

  38. [46]

    Zhanyue Qin, Haochuan Wang, Deyuan Liu, Ziyang Song, Cunhang Fan, Zhao Lv, Jinlin Wu, Zhen Lei, Zhiying Tu, Dianhui Chu, et al. 2024. UNO Arena for Evaluating Sequential Decision-Making Capability of Large Language Models. In Proceedings of the 2024 Conference on Empirical Met...

    2024

  39. [47]

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hock- enmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to- phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on comput...

    2015

  40. [48]

    Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021. CON- TRASTIVE LEARNING WITH HARD NEGATIVE SAMPLES. In International Conference on Learning Representations (ICLR)

    2021

  41. [49]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al

  42. [50]

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2014. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014)

    2014 arXiv

  43. [51]

    Mason Sawtell, Tula Masterman, Sandi Besen, and Jim Brown. 2024. Light- weight safety classification using pruned language models. arXiv preprint arXiv:2412.13435 (2024)

    2024 arXiv

  44. [52]

    François Role, Sébastien Meyer, and Victor Amblard. 2025. Fill the Gap: Quanti- fying and Reducing the Modality Gap in Image-Text Representation Learning. arXiv preprint arXiv:2505.03703 (2025)

    2025 arXiv

  45. [53]

    Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C Yuen. 2019. Multi-adversarial discriminative deep domain generalization for face presentation attack detec- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10023–10031

    2019

  46. [54]

    Rui Shao, Tianxing Wu, and Ziwei Liu. 2023. Detecting and grounding multi- modal media manipulation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition . 6904–6913

    2023

  47. [55]

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. Llava- prumerge: Adaptive token reduction for efficient large multimodal models.arXiv preprint arXiv:2403.15388 (2024)

    2024

  48. [56]

    Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. 2024. MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Lan- guage Models. In Advances in Neural Information Processing Systems , Vol. 37. 42048–42070

    2024

  49. [57]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  50. [58]

    Rui Shao, Tianxing Wu, Jianlong Wu, Liqiang Nie, and Ziwei Liu. 2024. Detecting and grounding multi-modal media manipulation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  51. [59]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191 (2024)

    2024 arXiv

  52. [60]

    Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. 2024. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision . Springer, 387–404

    2024

  53. [61]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Improving Text Embeddings with Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 11897–11916

    2024

  54. [62]

    Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, and Liqiang Nie. 2025. GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent. InAnnual Meeting of the Association for Computational Linguistics (ACL)

    2025

  55. [63]

    Yifei Yang, Zouying Cao, and Hai Zhao. 2024. LaCo: Large Language Model Pruning via Layer Collapse. In Findings of the Association for Computational Linguistics: EMNLP 2024. 6401–6417

    2024

  56. [64]

    Yuxin Wen, Qingqing Cao, Qichen Fu, Sachin Mehta, and Mahyar Najibi. 2024. Efficient vision-language models by summarizing visual tokens into compact registers. arXiv preprint arXiv:2410.14072 (2024)

    2024 arXiv

  57. [65]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision . 11975–11986

    2023

  58. [66]

    Renshan Zhang, Yibo Lyu, Rui Shao, Gongwei Chen, Weili Guan, and Liqiang Nie. 2024. Token-level correlation-guided compression for efficient multimodal document understanding. arXiv preprint arXiv:2407.14439 (2024)

    2024 arXiv

  59. [67]

    Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, and Yansong Tang

  60. [68]

    arXiv preprint arXiv:2406.12275 (2024)

    Voco-llama: Towards vision compression with large language models. arXiv preprint arXiv:2406.12275 (2024)

    2024 arXiv

  61. [69]

    Xiaofeng Zhang, Yihao Quan, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. 2025. From Redun- dancy to Relevance: Enhancing Explainability in Multimodal Large Language Models. Annual Conference of the Nations of the Ameri...

    2025

  62. [70]

    Yang Zhang, Yawei Li, Xinpeng Wang, Qianli Shen, Barbara Plank, Bernd Bischl, Mina Rezaei, and Kenji Kawaguchi. [n. d.]. FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models. In Workshop on Machine Learning and Compression, NeurIPS 2024

    2024

  63. [71]

    Renshan Zhang, Rui Shao, Gongwei Chen, Miao Zhang, Kaiwen Zhou, Weili Guan, and Liqiang Nie. 2025. FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registers. In Proceedings of the IEEE/CVF International Confe...

    2025

  64. [72]

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. 2025. LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token. arXiv preprint arXiv:2501.03895 (2025)

    2025 arXiv

  65. [75]

    Ying Zhang and Huchuan Lu. 2018. Deep cross-modal projection learning for image-text matching. InProceedings of the European conference on computer vision (ECCV). 686–701. PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learn...

    2018

  66. [2021]

    In International conference on machine learning

    Learning transferable visual models from natural language supervision. In International conference on machine learning . PmLR, 8748–8763

  67. [2024]

    In Findings of the As- sociation for Computational Linguistics: ACL 2024 , Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.)

    M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. In Findings of the As- sociation for Computational Linguistics: ACL 2024 , Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computat...

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.