Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

FoPru: Focal Pruning for Efficient Large Vision-Language Models

T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that large vision-language models contain many redundant visual tokens, and that the vision encoder's attention map alone can identify which tokens are worth keeping before they enter the language model.

desk verdict A simple, training-free token-pruning method with real experimental breadth, but Table 1 mixes per-dataset best-of-two strategies with rank-only speedups, so the headline accuracy claim is not tied to one reproducible configuration. read the letter →

arxiv 2411.14164 v1 pith:J2INVTDX submitted 2024-11-21 cs.CV cs.AI

Lei Jiang , Weizhe Huang , Tongxuan Liu , Yuting Zeng , Jing Li , Lechao Cheng , Xiaohua Xu This is my paper · ORCID
classification cs.CVcs.AI
keywords visualtokenpruninglargevision-languagemodelsattentionmaptraining-freeinferenceaccelerationsignificancemodecollapseCLIPencoderreordering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FoPru is a training-free method that prunes visual tokens in large vision-language models before they reach the language model. It computes a significance score for each visual token from the vision encoder's attention distribution, then keeps the top-scoring tokens either globally (rank strategy) or row-by-row (row strategy), and reorders them to preserve spatial layout. With only 25% of the visual tokens, the method keeps accuracy within one point on MMMU, SQA, and POPE, while cutting time-to-first-token by up to 2.52x on LLaVA-1.6-13B. The paper motivates this by showing that deep encoder layers concentrate over 80% of attention on less than 25% of tokens, a mode collapse that makes a small subset of tokens carry the critical visual information. The reason to care is that this offers a retraining-free inference speedup that can be applied to existing LVLMs.

What carries the argument

The central object is the attention-based token significance score. Starting from the multi-head attention map of the encoder's penultimate layer, FoPru averages attention over heads, then computes the average attention per token along both the row and column directions; it picks the direction with the larger variance as the significance vector, reasoning that a more dispersed distribution makes important tokens stand out. This score is used in two pruning strategies: rank pruning, which keeps the top $N \times r\%$ tokens globally, and row pruning, which reshapes the score into a grid and keeps the highest-scoring rows. Finally, token reordering sorts the kept indices back into ascending order so the pruned sequence preserves the original spatial arrangement. The mechanism works because, as the paper shows, deep encoder layers exhibit an imbalanced attention distribution (mode collapse), with most attention concentrated on a small subset of tokens that carry the critical visual information.

What would settle it

A direct test: construct a dataset of images in which the answer depends on a small object or text string located in a region that the CLIP encoder's attention scores rank low, and measure whether FoPru at 25% retention drops sharply relative to keeping all tokens. Alternatively, compare FoPru's selected tokens against the tokens the LLM's own cross-attention would select for the given prompt; if the two sets disagree on a large fraction of queries, the text-free significance assumption is the weak link.

Watch

Extended reading notes

Core claim

FoPru establishes that visual token pruning for LVLMs can be guided entirely by the attention distribution inside the CLIP vision encoder, without any training or access to the LLM. The token significance score is computed by averaging multi-head attention from the penultimate encoder layer, then taking the row or column marginal that has the larger variance; this score, computed without seeing the question text, is used to select the most important tokens. The rank strategy keeps the globally top-scoring tokens, while the row strategy keeps entire image rows with the highest summed significance, preserving horizontal continuity for text-heavy content. After selection, tokens are reordered by their original indices to keep relative spatial positions. The paper's experiments on three LLaVA-1.6 variants and seven benchmarks show that at 25% retention the method stays within one accuracy point of the full-token baseline on MMMU, SQA, and POPE, and that on GQA, MMMU, SQA, and POPE pruning can even slightly exceed the unfiltered baseline, suggesting that redundant tokens can actively distract the model.

Load-bearing premise

The significance score comes from the vision encoder alone, without seeing the question or instruction text, so the method assumes that the tokens that matter for the final answer are the same ones the encoder's attention highlights on the image itself.

Editorial extensions

If this is right

  • At 25% visual-token retention, FoPru cuts time-to-first-token by up to 2.52x (LLaVA-1.6-13B) and time-per-output-token by up to 1.24x, with GPU memory also reduced, enabling faster deployment of LVLMs without retraining.
  • Because FoPru prunes before the projector, it composes with LLM-level pruning like FastV; the paper's FoPru+FastV combination at 25% retention beats FastV alone on six datasets.
  • On GQA, MMMU, SQA, and POPE, pruning can slightly improve accuracy over the full-token baseline, implying that redundant visual tokens can add noise that interferes with the model's judgments.
  • The optimal retention ratio is task-dependent: text-heavy datasets like TextVQA and OCRbench need more tokens, while SQA and Ai2D remain near 60% accuracy even at 0.2% retention (about five tokens), so a practical deployment would tune the ratio per dataset or per task family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The text-free significance score could be made more robust by conditioning on the prompt: for example, using the LLM's first-layer cross-attention with the question text as a second scoring signal, then fusing the two ranks. The paper does not explore this, and the degradation on TextVQA/OCRbench suggests where the gain would show up.
  • The row strategy's advantage on text-heavy images at high retention suggests a hybrid scheme: use row pruning when the image has dense horizontal content (e.g., documents or scenes with text) and rank pruning for natural images with a clear focal object; the paper reports the two separately but does not combine them adaptively.
  • If the mode-collapse observation holds across other vision encoders (e.g., SigLIP, DINOv2), FoPru's scoring rule should transfer directly, but the optimal significance formula might need re-tuning since attention distributions differ; this is a testable extension beyond the LLaVA/CLIP family reported.
  • The near-60% accuracy at 0.2% retention on SQA and Ai2D hints that those benchmarks contain images where the answer is localized in a single small region; measuring the correlation between FoPru's selected patch and human-annotated answer regions would turn this hint into a diagnostic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. FoPru proposes a training-free visual token pruning method for LVLMs. It computes per-token significance from the penultimate-layer attention maps of the frozen vision encoder, selects tokens either by global rank or by row-preserving local structure, and then reorders the survivors to retain spatial positions. The method is evaluated on LLaVA-NeXT-8B, LLaVA-1.6-7B, and LLaVA-1.6-13B across seven multimodal benchmarks at retention ratios of 25%, 50%, and 75%, with additional comparisons to FastV, PruMerge, and PruMerge+ on LLaVA-1.5-7B. The headline claims are that 25% visual token retention stays within a 1% accuracy margin on MMMU, SQA, and POPE while yielding up to 2.52x TTFT speedup, and that FoPru generally maintains accuracy while pruning at 50% and 75% retention.

Significance. If the claims hold, FoPru is a simple, model-agnostic, training-free optimization that could be plugged into existing LVLMs without retraining or architecture changes. The paper's strengths are its breadth of models and datasets, its use of the LMMS-Eval platform for consistent evaluation, direct comparison with FastV and PruMerge/PruMerge+, and an honest limitations section that admits retention ratios and strategies are task-dependent. The method is not fitted to evaluation data, so circularity is not a concern. However, the empirical support is weakened by the oracle-style reporting in the main table, the absence of any repeated-run statistics, and the ad hoc variance rule in Eq. (4). These issues affect the central claims about accuracy retention and inference speedup, so they need to be resolved before the paper can be recommended for publication.

major comments (3)
  1. [§5.2, Table 1] Table 1 reports accuracy as the per-dataset best of the rank and row strategies (cross-referencing with Table 4), while the inference-efficiency columns are explicitly presented for the rank strategy only. For example, LLaVA-NeXT-8B at 75% retention takes Ai2D 70.69, SQA 79.91, TextVQA 64.14, and Ocrbench 53.20 from row pruning, but GQA 65.21, MMMU 39.78, and POPE 87.87 from rank pruning. Consequently no single FoPru configuration attains both the reported accuracy and the reported speedup at 50% and 75% retention, so the claim that FoPru maintains high accuracy at those ratios is not supported as stated. Please report a single consistent strategy with its own speedups, or clearly label the oracle selection and provide efficiency numbers for the exact configuration that produced each accuracy column.
  2. [§5.2, Table 1; §5.4, Table 3] No error bars, confidence intervals, or repeated-run statistics are reported for any accuracy number, and several benchmarks are small (MMMU-val is roughly 150 questions). Under that noise scale, a 1.0-point drop corresponds to about one or two items, which makes the abstract's 'within a 1% margin' claim statistically fragile; moreover, for LLaVA-NeXT-8B at 25% retention the MMMU drop is exactly 1.00 point (40.22 to 39.22), which contradicts the bullet's 'less than 1%' wording. Please add repeated-run statistics or at least an explicit sample-size caveat, and correct the wording.
  3. [§4.2, Eq. (4); §5.4, Table 3] The variance rule for choosing between s1 and s2 is presented without a principled justification, and the ablation labeled 'w/o Variance' tests selecting the low-variance direction, not whether the variance heuristic itself adds value. It is therefore unknown whether Eq. (4) improves over always using s1 or always using s2. Please compare the fixed choices and the variance rule across models, datasets, and retention ratios, or provide a principled basis for the heuristic.
minor comments (3)
  1. [§6, Conclusion] The limitation paragraph already concedes that optimal ratios vary by task and model; consider moving this observation into the main-results discussion so the dataset-specific behavior in Figure 5 is framed as a design constraint rather than an aside.
  2. [Figure 6; Table 1] There are typos in the figure axis ('Relative Accrucy') and in the Table 1 header ('Inference Efficience'); in §5.2 the phrase 'accuracy accuracy after pruning' should be corrected.
  3. [§2.2] The related-work discussion of 'mode collapse' cites [11] and uses it as motivation, but the connection to FoPru's design is descriptive; consider stating explicitly where the concept is used (Eq. (4)) so the reader can assess the strength of the link.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FoPru's pruning scores come from the frozen encoder and are evaluated on held-out benchmarks; the sole self-citation is motivational.

full rationale

FoPru's derivation chain is not circular. The token significance score (Eq. 4) is computed directly from the CLIP encoder's penultimate-layer attention maps (Eqs. 2-3), and the rank/row pruning strategies select tokens by those fixed scores given a user-set retention ratio r; no component of the score, strategy, or ratio is fitted to the evaluation benchmarks. The accuracy and efficiency claims are then measured on held-out datasets (MMMU, SQA, POPE, etc.) against unpruned baselines, so the claim that pruning maintains accuracy is an empirical result rather than a consequence of how the pruning input was defined. The paper's limitation statements, such as the observation that optimal pruning ratios vary across tasks and that text-heavy or position-sensitive tasks need more tokens, are honest acknowledgments of where the static encoder-saliency proxy fails, and they further confirm that the method is being tested rather than asserted. The only self-citation is [11] in Section 3.1, used as 'inspiration' for visualizing attention maps; the mode-collapse observation is the paper's own Figure 1, and the central claims do not rest on that citation, so it is not load-bearing. The Table 1 practice of reporting the per-dataset best of two strategies alongside rank-only speedups is a reporting or statistical concern rather than circularity: the reported numbers are measured outputs, not fitted inputs renamed as predictions. Accordingly, no circular step is identified.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The contribution rests on the empirical premise that frozen encoder attention indicates downstream usefulness; this is a domain assumption, not a mathematical axiom, and it is only validated on a subset of benchmarks.

free parameters (1)
  • token retention ratio r
    User-selected hyperparameter swept across 0.2% to 75%; the central efficiency/accuracy trade-off depends on it.
assumptions (3)
  • domain assumption Attention scores in the penultimate layer of the vision encoder are a reliable proxy for token importance to the downstream LLM.
    Invoked in Sec. 4.2 (Token Significance) and motivated by Fig. 1; the entire pruning signal is derived from this, without text-conditioning.
  • ad hoc to paper The dimension (s1 vs s2) with higher variance better separates important tokens.
    Eq. 4 in Sec. 4.2; the choice is justified only by the ablation (w/o Variance degrades), not by a principled derivation.
  • domain assumption The penultimate layer of the CLIP encoder provides the primary visual representation for LLaVA-style models.
    Sec. 4.2 states this to justify using that layer; no systematic comparison across layers is shown for the significance signal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FoPru: Focal Pruning for Efficient Large Vision-Language Models." pith.science (2026). https://pith.science/paper/J2INVTDX

@misc{pith2026241114164,
  author       = {Pith},
  title        = {Pith review of: FoPru: Focal Pruning for Efficient Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2INVTDX}},
  note         = {Machine review of arXiv:2411.14164}
}
read the original abstract

Large Vision-Language Models (LVLMs) represent a significant advancement toward achieving superior multimodal capabilities by enabling powerful Large Language Models (LLMs) to understand visual input. Typically, LVLMs utilize visual encoders, such as CLIP, to transform images into visual tokens, which are then aligned with textual tokens through projection layers before being input into the LLM for inference. Although existing LVLMs have achieved significant success, their inference efficiency is still limited by the substantial number of visual tokens and the potential redundancy among them. To mitigate this issue, we propose Focal Pruning (FoPru), a training-free method that prunes visual tokens based on the attention-based token significance derived from the vision encoder. Specifically, we introduce two alternative pruning strategies: 1) the rank strategy, which leverages all token significance scores to retain more critical tokens in a global view; 2) the row strategy, which focuses on preserving continuous key information in images from a local perspective. Finally, the selected tokens are reordered to maintain their original positional relationships. Extensive experiments across various LVLMs and multimodal datasets demonstrate that our method can prune a large number of redundant tokens while maintaining high accuracy, leading to significant improvements in inference efficiency.

Figures

Figures reproduced from arXiv: 2411.14164 by the authors.

Figure 1
Figure 1. The attention map of CLIP in different layers. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The framework of Focal Pruning for LVLMs. First, we obtain the attention map in the vision encoder and calculate the token [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The proportion of visual tokens and textual tokens in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The CLIP model processes the input image (d) to gen [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Performance metrics across visual token retention ratios [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The comparison of different token pruning strategies across different LVLMs and datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Performance metrics across visual token retention ratios [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    METEOR is a three-stage token pruning framework that reduces visual tokens in multi-encoder MLLMs by 76% with only a 0.3% average accuracy drop.

Reference graph

Works this paper leans on

37 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1, 2

    arXiv 2023

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023. 2

    2023

  4. [4]

    To- ken merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. To- ken merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 1

    arXiv 2022

  5. [5]

    Vip- llava: Making large multimodal models understand arbitrary visual prompts

    Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip- llava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12914– 12923, 2024. 1

    work page 2024

  6. [6]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 13817–13827, 2024. 1, 2

    work page 2024

  7. [7]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Jun- yang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models. arXiv preprint arXiv:2403.06764, 2024. 1, 2, 3

    arXiv 2024

  8. [8]

    Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. 1, 2

    arXiv 2023

Show all 37 references
  1. [9]

    Mobilevlm v2: Faster and stronger baseline for vision language model

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024. 1, 2

    2024 arXiv

  2. [10]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  3. [11]

    Mitigating undisci- plined over-smoothing in transformer for weakly supervised semantic segmentation

    Jingxuan He, Lechao Cheng, Chaowei Fang, Dingwen Zhang, Zhangye Wang, and Wei Chen. Mitigating undisci- plined over-smoothing in transformer for weakly supervised semantic segmentation. arXiv preprint arXiv:2305.03112 ,

    arXiv

  4. [12]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 5

    2019

  5. [13]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 235–

    2016

  6. [14]

    Llava-next: Stronger llms supercharge multimodal capa- bilities in the wild, 2024

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capa- bilities in the wild, 2024. 3, 5

    2024

  7. [15]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 1, 2

    2023

  8. [16]

    Tokenpacker: Effi- cient visual projector for multimodal llm

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. Tokenpacker: Effi- cient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392, 2024. 2

    2024 arXiv

  9. [17]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 2, 5

    2023 arXiv

  10. [18]

    Improved baselines with visual instruction tuning, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024. 2, 7

    2024

  11. [19]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 5

    2024

  12. [20]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 2

    2024

  13. [21]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895, 2023. 5

    2023 arXiv

  14. [22]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  15. [23]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu 9 Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arX...

    2024 arXiv

  16. [24]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388,

  17. [25]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 5

    2019

  18. [26]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1

    2023 arXiv

  19. [27]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  20. [28]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 2

    2023 arXiv

  21. [29]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1):1, 2023. 2

    2023 arXiv

  22. [30]

    Dense connector for mllms

    Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang. Dense connector for mllms. arXiv preprint arXiv:2405.13800, 2024. 1

    2024 arXiv

  23. [31]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. 2

    2023 arXiv

  24. [32]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  25. [33]

    Dockylin: A large multimodal model for visual document understanding with efficient visual slim- ming

    Jiaxin Zhang, Wentao Yang, Songxuan Lai, Zecheng Xie, and Lianwen Jin. Dockylin: A large multimodal model for visual document understanding with efficient visual slim- ming. arXiv preprint arXiv:2406.19101, 2024. 1, 2

    2024 arXiv

  26. [34]

    Lmms-eval: Re- ality check on the evaluation of large multimodal models

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Re- ality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772, 2024. 5

    2024 arXiv

  27. [35]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 2 10 0 10 20 30 40 50 60 70 80 90 100 Ratio (%) 0 10 20 30 40 50 60 70 80 90...

    2023 arXiv

  28. [37]

    The trend is smoother compared to rank pruning, where accuracy often rises more sharply at lower ratios (e.g., Ocrbench)

    Appendix Token Retention Ratios in Row Pruning Figure 7 shows that accuracy increases steadily with higher token reten- tion ratios for both models under row pruning. The trend is smoother compared to rank pruning, where accuracy often rises more sharply at lower ratios (e.g.,...

  29. [251]

    Springer, 2016. 2, 5

    2016

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.