Pith. sign in

REVIEW 3 major objections 5 minor 11 cited by

Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that pruning visual tokens by maximizing instruction-conditioned diversity—rather than attention or raw similarity—preserves 94–99% of MLLM accuracy while removing 78–94% of visual tokens.

desk verdict Solid training-free token pruning via instruction-conditioned DPP; the core idea holds, but the abstract's efficiency numbers and the embedding-space notation need cleanup before this is citable at face value. read the letter →

arxiv 2506.10967 v2 pith:FN6NJHTG submitted 2025-06-12 cs.CV cs.AI

Qizhe Zhang , Mengzhen Liu , Lichen Li , Ming Lu , Yuan Zhang , Junwen Pan , Qi She , Shanghang Zhang This is my paper
classification cs.CVcs.AI
keywords visualtokenpruningmultimodallargelanguagemodelsdeterminantalpointprocessconditionaldiversityinstructionrelevancetraining-freeinferenceaccelerationvision-languagebenchmarksreduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CDPruner, a training-free visual-token-pruning method, argues that the right criterion for dropping visual tokens in multimodal LLMs is neither attention importance nor raw visual similarity but conditional diversity: keep the subset that is jointly diverse and relevant to the user's question. The paper encodes this as a determinantal point process over visual tokens whose kernel multiplies pairwise cosine similarity by per-token instruction relevance, and selects the subset with maximum determinant. On LLaVA-1.5-7B, it reports 99.0% of original average accuracy with 77.8% of tokens pruned, and 94.3% when only 32 of 576 tokens remain. Similar patterns appear on LLaVA-NeXT, LLaVA-Video, Qwen2.5-VL, and InternVL3, with about a tenfold FLOP reduction. If this holds, visual-token compression for open-source MLLMs can be done at inference time, without training, and without sacrificing much accuracy even at extreme compression ratios.

What carries the argument

The central object is the conditional DPP kernel $\tilde L = \mathrm{diag}(\tilde r)\, L\, \mathrm{diag}(\tilde r)$, where $L_{ij}$ is the cosine similarity between visual tokens $i$ and $j$, and $\tilde r_i$ is the min-max normalized cosine similarity between visual token $i$ and the text embedding of the instruction. Under a DPP, the probability of selecting subset $S$ is proportional to $\det(\tilde L_S)$, so maximizing the determinant is the same as maximizing $\sum_{i\in S}\log \tilde r_i^2 + \log\det(L_S)$: prefer tokens relevant to the question while penalizing mutual overlap. The paper selects the subset by fast greedy MAP inference with Cholesky updates, which runs in $O(nm^2)$ and adds less than 10 ms per image, and the kernel requires only the visual and text embeddings the MLLM already produces.

What would settle it

A concrete check: on a fixed image set and token budget, run CDPruner with the true question and again with an unrelated fixed question, and compare the retained token subsets and the downstream accuracy. If the subsets have near-perfect overlap and accuracy is unchanged, then the instruction-conditioning term is not doing the causal work claimed and the reported gains come from diversity alone. A companion check: shuffle the order of the question's text tokens before computing Eq. (5); if pruned accuracy does not fall, the relevance scores are not carrying the conditioning signal.

Watch

Extended reading notes

Core claim

The paper's central claim is that the most useful visual tokens to keep are the ones that maximize $\det(\mathrm{diag}(\tilde r)\, L\, \mathrm{diag}(\tilde r))_S$, where $L_{ij}$ is the cosine similarity between visual tokens $i$ and $j$, and $\tilde r_i$ is the normalized cosine relevance of token $i$ to the instruction text. That determinant balances two forces: tokens should individually match what the question asks about, and as a set they should carry as little duplicated information as possible. The authors report that this instruction-conditioned diversity beats attention-based, similarity-based, and attention-plus-similarity baselines at every tested reduction ratio, with the advantage widening as the budget shrinks. On LLaVA-NeXT-7B, retaining 640 of 2,880 tokens slightly exceeds the unpruned model's average accuracy; on LLaVA-1.5-7B, keeping 128 of 576 tokens preserves 99.0% of original performance; on LLaVA-Video, keeping 32 tokens per frame preserves 95%. The paper also observes that pruned models can match or exceed the unpruned baseline on the POPE hallucination benchmark, suggesting pruning can remove noise that encourages object hallucination.

Load-bearing premise

The load-bearing premise is that a single cosine-similarity score between each visual token and one text embedding of the question reliably marks which image regions the question asks about; when that alignment is weak, the conditional term stops helping and the method falls back on raw diversity.

Editorial extensions

If this is right

  • Attention-score-free pruning means the method stays compatible with efficient attention implementations that avoid materializing attention maps, so acceleration is not blocked by the pruning step.
  • Relative accuracy is highest for high-resolution images and long videos, where token redundancy is largest and inference cost is most severe.
  • Because selection operates on embeddings and needs no training, CDPruner can be dropped into any open-source MLLM whose visual tokens are accessible, including architectures with built-in token compression such as Qwen2.5-VL and InternVL3.
  • On the POPE hallucination benchmark, pruned models can match or slightly exceed the unpruned baseline, which the authors take as evidence that pruning may mitigate object hallucination.
  • The margin over similarity-only baselines grows at aggressive ratios (around 90% pruning), so the instruction-conditioning term matters most when the token budget is tight.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the single-text-embedding relevance score may be a bottleneck for multi-object or compositional questions; decomposing the instruction into per-phrase embeddings and combining the resulting DPP selections is a natural extension the paper does not test.
  • Beyond the paper: the POPE gains suggest an anti-hallucination effect, but the paper does not isolate it; replacing the real question with a fixed prompt in a hallucination benchmark would test whether conditioning or diversity itself suppresses hallucinated objects.
  • Beyond the paper: the same conditional-DPP kernel could be applied hierarchically to long videos, first selecting diverse frames and then diverse tokens within frames, which the video experiments hint at but do not implement as a separate stage.
  • Beyond the paper: the appendix's balance factor $\theta$ implies the default weighting of relevance versus diversity may not be optimal per benchmark; a per-dataset tuning study could show whether fine-grained questions need more relevance weight and vague questions more diversity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CDPruner, a training-free, model-agnostic visual token pruning method for MLLMs. It augments a determinantal point process (DPP) kernel with instruction-conditioned relevance scores, so that retained tokens maximize a combination of feature diversity and instruction relevance. The method is evaluated on LLaVA-1.5, LLaVA-NeXT, LLaVA-Video, Qwen2.5-VL, and InternVL3 across multiple image and video benchmarks, with reported state-of-the-art retention at high pruning ratios (e.g., 99.0% relative accuracy at 77.8% pruning on LLaVA-1.5-7B, Table 1). The DPP formulation (Eqs. 7-8) is mathematically standard, and the greedy MAP inference with O(nm^2) complexity is correctly described. The main technical concern is the unspecified embedding space used to compute the instruction relevance in Eq. (5), which affects whether the central conditioning mechanism is meaningful and reproducible.

Significance. If the reported results hold, CDPruner would be a strong practical contribution: it is training-free, model-agnostic, compatible with efficient attention implementations, and it consistently outperforms attention- and similarity-based baselines at aggressive token reduction ratios. The paper also provides a useful efficiency analysis (Table 5) covering FLOPs, latency, KV cache, and GPU memory, and it releases code. The use of DPP with a diagonal quality term is principled and clearly derived, and the ablation against DPPruner (Figure 4) supports the general benefit of non-uniform weighting. However, the current manuscript does not pin down the exact representation used for visual and text embeddings in Eq. (5), which is load-bearing for the claim that instruction-conditioned diversity drives the gains.

major comments (3)
  1. [Sec. 3.3, Eq. (5)] The paper must specify the embedding space in which Eq. (5) is computed. In Sec. 3.1, H_v is defined as the post-projector features g(f_v(X_v)) that the LLM consumes, but Sec. 3.3 says that for CLIP/SigLIP-based models it uses features from the paired text encoder and visual encoder. A cosine similarity between a CLIP text embedding and a post-projector LLM feature is not mathematically meaningful. The only coherent reading is that the visual side uses pre-projector CLIP patch features, yet the text never states this, and it also does not address that CLIP's contrastive alignment is at the pooled image/text level rather than at the patch level. Please state explicitly, for each of LLaVA, LLaVA-Video, Qwen2.5-VL, and InternVL3, whether r is computed from pre-projector visual features, post-projector visual features, and which exact text embeddings are used. Without this, the reported gains from instruction conditioning are not reproducible.
  2. [Abstract / Sec. 1 / Table 5] The abstract and Sec. 1 claim that CDPruner 'reduces FLOPs by 95% and CUDA latency by 78%' when applied to LLaVA. The efficiency table for LLaVA-NeXT-7B (Table 5) shows FLOPs going from 41.7T to 4.2T, which is a 89.9% reduction, not 95%, and per-stage CUDA latency reductions of x6.6 (prefill) and x1.3 (decode), which do not add up to 78% in any way stated in the paper. Either the abstract numbers come from a different configuration (e.g., LLaVA-1.5 at 32 tokens) and should be attributed to that configuration, or the claims should be corrected to match Table 5.
  3. [Sec. 4.7, Figure 4] The ablation CDPruner versus DPPruner demonstrates that adding a non-uniform weight vector improves over uniform DPP, but it does not isolate whether the instruction-specific content of r is responsible. A control with a fixed or random non-uniform weight vector (e.g., one derived from an image-only saliency map, or a constant relevance vector) would separate 'instruction-conditioned' from 'non-uniform weighting'. This matters because the authors themselves note in Sec. 4.2 that conditioning is unhelpful on VizWiz, so the claim that instruction relevance carries signal is central and should be tested more directly.
minor comments (5)
  1. [Table 7] The upper-bound row of Table 7 is labeled 'LLaV A-NeXT-7B' although the table title and context indicate LLaVA-NeXT-13B; the label is likely a copy-paste error.
  2. [Sec. 3.1 vs Sec. 3.2] The symbol L is used both for the loss function in Eq. (1) and for the DPP kernel matrix in Eq. (3). These should use distinct symbols to avoid confusion.
  3. [Sec. 3.3] The notation H_v is overloaded: in Eq. (1) it denotes post-projector features, while in Sec. 3.3 it denotes the visual encoder output for CLIP-based models. Use different symbols, e.g., H_v^enc and H_v^proj, throughout.
  4. [Sec. 1] In the first sentence of the Introduction, 'large language models' is abbreviated as 'LMMs'; the standard abbreviation is LLMs.
  5. [Sec. 4.1] There is a typo in 'comparsion methods' in the Comparison methods paragraph; it should be 'comparison'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CDPruner's selection objective is a standard DPP quality-diversity decomposition, and all gains are measured on held-out external benchmarks without fitting to the targets.

full rationale

The derivation chain is self-contained. Eq. (3) defines the DPP kernel from visual-token cosine similarities; Eq. (5) defines instruction relevance via cosine similarity to a text embedding from a pretrained external encoder; Eqs. (7)-(8) combine them using the standard DPP quality term log det(diag(r) L diag(r)) = sum log r_i^2 + log det(L). No parameter in the main method is fitted to the evaluation benchmarks, and the claimed accuracy retention is measured on external held-out benchmarks (VQAv2, GQA, POPE, MME, etc.) against the unpruned model and prior methods. The optional balance factor theta in Appendix C.4 is presented as a per-benchmark ablation, not as the main predictive claim, and the main results use the version without it. The only author-overlapping citation (Zhang et al. 2024b on attention shift) appears in related work and is not load-bearing: CDPruner does not rely on that assertion for its construction, and the citation is corroborated by an independent external reference (Wen et al. 2025a). Hence there is no self-definitional, fitted-input, uniqueness-imported, or renamed-known-result circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical or architectural entities are introduced; the method combines existing DPP machinery with existing text encoders.

free parameters (1)
  • balance factor theta (optional) = not used in main experiments; ablated in Appendix C.4 with values 0.0-0.8
    Controls trade-off between instruction relevance and diversity in the variant of Eq. (18). Main results use the default kernel without this factor.
assumptions (3)
  • standard math Cosine-similarity matrix of visual tokens is a positive semi-definite kernel, so DPP sampling applies.
    Required for Eq. (3) and the greedy Cholesky algorithm in Appendix A.
  • domain assumption Pretrained text encoders (CLIP/SigLIP) or averaged LLM token embeddings produce instruction representations aligned with visual token embeddings.
    Section 3.3 assumes cosine similarity between Hv_i and Hq_bar measures instruction relevance; misalignment would weaken the conditional kernel.
  • domain assumption Min-max normalization of relevance scores per image preserves ranking of token relevance.
    Eq. (6) scales r to [0,1]; it is monotone so preserves per-image order but makes scores instance-relative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs." pith.science (2026). https://pith.science/paper/FN6NJHTG

@misc{pith2026250610967,
  author       = {Pith},
  title        = {Pith review of: Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FN6NJHTG}},
  note         = {Machine review of arXiv:2506.10967}
}
read the original abstract

In multimodal large language models (MLLMs), the length of input visual tokens is often significantly greater than that of their textual counterparts, leading to a high inference cost. Many works aim to address this issue by removing redundant visual tokens. However, current approaches either rely on attention-based pruning, which retains numerous duplicate tokens, or use similarity-based pruning, overlooking the instruction relevance, consequently causing suboptimal performance. In this paper, we go beyond attention or similarity by proposing a novel visual token pruning method named CDPruner, which maximizes the conditional diversity of retained tokens. We first define the conditional similarity between visual tokens conditioned on the instruction, and then reformulate the token pruning problem with determinantal point process (DPP) to maximize the conditional diversity of the selected subset. The proposed CDPruner is training-free and model-agnostic, allowing easy application to various MLLMs. Extensive experiments across diverse MLLMs show that CDPruner establishes new state-of-the-art on various vision-language benchmarks. By maximizing conditional diversity through DPP, the selected subset better represents the input images while closely adhering to user instructions, thereby preserving strong performance even with high reduction ratios. When applied to LLaVA, CDPruner reduces FLOPs by 95\% and CUDA latency by 78\%, while maintaining 94\% of the original accuracy. Our code is available at https://github.com/Theia-4869/CDPruner.

Figures

Figures reproduced from arXiv: 2506.10967 by the authors.

Figure 1
Figure 1. Comparison of different token pruning methods. Attention-based methods retain numerous duplicate tokens, failing to achieve effective visual token compression. Similarity-based methods neglect user instructions, always pruning the same tokens and paying insufficient attention to regions most relevant to the question. Our CDPruner considers the conditional diversity of the selected subset, dynamically adjusting pruni… view at source ↗
Figure 2
Figure 2. Overview of CDPruner. We first calculate the similarity between visual tokens conditioned on their relevance to the current instruction. Then, CDPruner uses a DPP to select the subset to keep. As a training-free and model-agnostic method, it ensures both the diversity and quality of the selected token subset, significantly reducing computational cost while maintaining considerable performance. 3.1 Visual token pruni… view at source ↗
Figure 3
Figure 3. Visualization of relevance scores. We compute the relevance scores for several samples from the POPE benchmark using LLaVA-1.5-7B, with the instruction following the template: “Is there a {object} in the image?” Red indicates high relevance, while blue indicates low relevance. 3.3 Instruction relevance The above only considers the feature similarity among visual tokens, resulting in the same pruning result regardles… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Ablation study of CDPruner design. DPPruner denotes applying DPP to visual token pruning without conditioning on instruction relevance, as a degraded variant of CDPruner. visual tokens remained, competing methods suffer from severe performance degradation. In contrast,…
Figure 5
Figure 5. Figure 5: Additional visualizations of relevance scores. We compute the relevance scores for several samples from the POPE benchmark using LLaVA-1.5-7B, with the instruction following the template: “Is there a {object} in the image?” Red indicates high relevance, while blue indi…

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs

    cs.CV 2026-08 conditional novelty 7.0 of 10

    PhyCheck is a 69,825-pair video QA benchmark that tests and improves Video-LLMs' ability to judge whether events obey physical laws, with fine-grained evidence questions and a context-sensitivity pilot.

  2. Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs

    cs.CV 2026-07 conditional novelty 7.0 of 10

    SFPruner performs redundancy-aware visual token pruning in a single non-iterative pass using semantics-guided ridge leverage scores and directional masking, cutting selection latency from 112.4 ms to 2.5 ms at 512 tok...

  3. RADIO1D: Elastic Representations for Condensed Vision Modeling

    cs.CV 2026-07 accept novelty 7.0 of 10

    RADIO1D produces elastic hierarchical 1D visual tokens via multi-teacher distillation that match or beat fixed 2D encoders in VLMs at lower token counts.

  4. Hi-Token: Hierarchical Coordinate Tokenization for Generative Visual Grounding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Representing bounding-box coordinates as axis-specific hundreds, tens, and ones tokens, plus a geometry-aware GRPO reward, improves generative visual grounding accuracy.

  5. WaveZip: Wavelet-Driven Space-Time Decoupling for Video Token Condensation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    WaveZip, a training-free wavelet method for video token condensation, reports about 10x token reduction while retaining roughly 98-99% accuracy on multi-benchmark LVLM evaluation.

  6. IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Training a coordinate-generating VLM with an IoU-aware distillation loss from a teacher that sees the ground-truth box marked on the image improves referring-expression grounding by ~3-4 accuracy points.

  7. SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    SAFE-Pruner forecasts deep-layer visual-token saliency from historical attention maps and refreshes at subtask boundaries, enabling up to 1.89x faster VLA inference with minimal success-rate drop.

  8. Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics

    cs.RO 2025-12 conditional novelty 6.0 of 10

    A 3D-aware VLM, RoboTracer, generates metric-grounded spatial traces for robot manipulation using scale supervision and metric-sensitive reinforcement rewards.

  9. SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Using the separator token as a position-free query, SepPrune prunes 80% of vision tokens while retaining 96.3% of Qwen2.5-VL-7B's original accuracy.

  10. EgoPrune: Efficient Token Pruning for Egomotion Video Reasoning in Embodied Agent

    cs.CV 2025-07 conditional novelty 5.0 of 10

    EgoPrune prunes egomotion video tokens by homography-based frame alignment and MMR selection, keeping accuracy close to the full-token baseline while reducing compute.

  11. A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models

    cs.CV 2025-08 unverdicted novelty 4.0 of 10

    The paper claims 92.6% dynamic visual-token pruning with retained (or 110% of) baseline VQA performance, but the full text supplied is a different paper, so the claim is unverifiable here.

Reference graph

Works this paper leans on

34 extracted references · 5 canonical work pages · cited by 11 Pith papers

  1. [1]

    Divprune: Diversity-based visual token pruning for large multimodal models

    Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity-based visual token pruning for large multimodal models. arXiv preprint arXiv:2503.02175,

    arXiv

  2. [6]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    URL https://lmsys.org/blog/ 2023-03-30-vicuna/ . Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359,

    work page 2023

  3. [8]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024a. Wenbo Hu, Zi-Yi Dou, Liunian Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query tran...

    arXiv

  4. [10]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  5. [11]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. Springer,

    2016

  6. [13]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122,

    arXiv

  7. [14]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer,

    work page 2014

  8. [15]

    Improved baselines with visual instruction tuning

    11 Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024a. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world ...

    arXiv 2024

Show all 34 references
  1. [16]

    Feast your eyes: Mixture- of-resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture- of-resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003,

    arXiv

  2. [19]

    Less is more: A simple yet effective token reduction method for efficient multi-modal llms

    Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, and Benyou Wang. Less is more: A simple yet effective token reduction method for efficient multi-modal llms. arXiv preprint arXiv:2409.10994,

    arXiv

  3. [21]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295,

    arXiv

  4. [22]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. Hugo Touvron, Lo...

    arXiv

  5. [24]

    Token pruning in multimodal large language models: Are we solving the right problem? arXiv preprint arXiv:2502.11501, 2025a

    Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. Token pruning in multimodal large language models: Are we solving the right problem? arXiv preprint arXiv:2502.11501, 2025a. Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui ...

    arXiv

  6. [25]

    Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247,

    arXiv

  7. [26]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024a. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Ji...

    arXiv

  8. [27]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Li- juan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490,

    arXiv

  9. [28]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858,

    arXiv

  10. [29]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Hao- ran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024a. Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minq...

    arXiv

  11. [30]

    Sparsevlm: Visual token sparsification for efficient vision-language model inference

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417, 2024c. Yuanhan...

    arXiv

  12. [31]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479,

    arXiv

  13. [32]

    Appendix B provides some details of the experimental setup, including information about model architectures, evaluation benchmarks, comparison methods and implemen- tation

    13 Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs Appendix Appendix A describes the fast greedy MAP inference algorithm used in this work, along with the corresponding pseudocode. Appendix B provides some details of the experimental...

    2018

  14. [33]

    Therefore, the greedy algorithm runs in O(nm2) time

    In the k-th iteration, for each item i ∈ Z \ S, updating ci and di involve the inner product of two vectors of length k, resulting in overall complexity O(kn). Therefore, the greedy algorithm runs in O(nm2) time. After parallelizing the for-loop over i using CUDA, the addition...

    2021

  15. [34]

    Here, we fix the input resolution to 896×896, yielding 1,280 visual tokens. Notably, unlike its performance on the LLaV A series, DivPrune exhibits a significant performance drop on InternVL3, as it does not account for the relevance to user instructions during pruning. In con...

    2024

  16. [1975]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244,

    arXiv

  17. [2012]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024a. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wa...

    arXiv

  18. [2015]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609,

    arXiv

  19. [2017]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    12 Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191,

    arXiv

  20. [2018]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer, 2024a...

    arXiv

  21. [2019]

    Similarity-aware token pruning: Your vlm but faster

    Ahmadreza Jeddi, Negin Baghbanzadeh, Elham Dolatabadi, and Babak Taati. Similarity-aware token pruning: Your vlm but faster. arXiv preprint arXiv:2503.11549,

    arXiv

  22. [2021]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388,

  23. [2022]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024a

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024a. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li...

    arXiv

  24. [2023]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923,

    arXiv

  25. [2024]

    Mdp3: A training-free approach for list-wise frame selection in video-llms

    Hui Sun, Shiyin Lu, Huanyu Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Ming Li. Mdp3: A training-free approach for list-wise frame selection in video-llms. arXiv preprint arXiv:2501.02885,

  26. [2025]

    Matryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. In Workshop on Video-Language Models@ NeurIPS 2024, 2024a. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 techni...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.