Pith. sign in

REVIEW 4 major objections 7 minor 84 references

METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read METEOR cuts 76% of visual tokens in multi-encoder vision-language models while losing only 0.3% average accuracy.

desk verdict Solid new framework for pruning tokens in multi-encoder MLLMs; the results look credible but several key hyperparameters are unreported and the rank-based budget allocation needs a stronger analysis. read the letter →

arxiv 2507.20842 v1 pith:WCMZYE6A submitted 2025-07-28 cs.CV

Yuchen Liu , Yaoming Wang , Bowen Shi , Xiaopeng Zhang , Wenrui Dai , Chenglin Li , Hongkai Xiong , Qi Tian This is my paper
classification cs.CV
keywords visualtokenpruningmulti-encoderMLLMefficientvisionlanguagemodelrank-basedsparsityallocationinstance-adaptiveattentionheadselectionOCRmultimodalefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that multi-encoder vision-language models, which fuse visual features from several specialist encoders, carry far more visual tokens than they need, and that this redundancy can be removed in three coordinated stages. METEOR prunes tokens inside each vision encoder, again when encoder features are merged, and finally in the LLM's decoding layers under the guidance of the text prompt. On the EAGLE architecture the method removes 76% of visual tokens, saves 49% of TFLOPS, and raises throughput by 46% with only a 0.3% average accuracy drop across 11 benchmarks. The significance is that multi-encoder MLLMs need not be prohibitively slow; their accuracy can be delivered at a fraction of the visual-token cost.

What carries the argument

A three-stage progressive pruning pipeline. Stage one uses the cosine similarity to the average token (shallow blocks) and class-token attention (deep blocks) to drop redundant tokens within each encoder, with per-encoder budgets allocated in proportion to feature-map rank $r^l_b$ (from an offline SVD-based estimate) so that information-poor encoders give up more tokens. Stage two projects each encoder's tokens through its own MLP, concatenates them, and prunes the tokens with the highest summed cross-encoder similarity, reducing mutual redundancy. Stage three computes a visual attention value $VAV_h$ per attention head, keeps the top-$k$ heads, scores tokens by those heads' attention to the last instruction token, and sets the retained count $K = \lambda \sum_{h=1}^k \sum_{i=1}^N a_{i,I(h)}$ so the pruning ratio adapts to instance complexity. The design rests on five empirical findings about where redundancy lives; the rank-is-information finding carries the budget allocation.

What would settle it

Run METEOR on an OCR-heavy benchmark, then swap the rank-based allocation for its inverse (most tokens to the lowest-rank encoder) while keeping everything else fixed; if the inverse allocation matches or exceeds the rank-based one, the rank-is-information premise is false. Alternatively, construct images where the task-critical content lives in a low-rank subspace, such as a few large flat text regions, and check whether rank allocation starves the encoder that actually reads the text.

Watch

Extended reading notes

Core claim

METEOR is the first token-pruning framework designed for multi-encoder MLLMs, eliminating redundant visual tokens progressively across encoding, fusion, and decoding. In encoding, token significance is measured by similarity to the average token in shallow layers and by class-token attention in deep layers, while each encoder's token budget is allocated in proportion to the rank of its feature maps, on the evidence that rank is a stable measure of information richness. In fusion, each encoder keeps its own projector and mutually redundant tokens across encoders are pruned cooperatively, which raises feature diversity more effectively than pruning each encoder separately. In decoding, redundant visual tokens are identified by the attention values of the top-k most reliable heads rather than by averaging all heads, and the number of retained tokens is adjusted per instance according to the visual attention value, so that fine-grained tasks like OCR keep more tokens than coarse general questions. Compared with EAGLE, METEOR reduces visual tokens from 1024 to about 242, a 76% reduction, while keeping average accuracy within 0.3%.

Load-bearing premise

The rank of a feature map is a stable and faithful measure of how much task-relevant visual information an encoder carries, so giving higher-rank encoders bigger token budgets is the right allocation.

Editorial extensions

If this is right

  • Multi-encoder MLLMs such as EAGLE can retain their accuracy with roughly a quarter of the visual tokens, so the compute bottleneck of fusing several vision encoders is not inherent.
  • Fixed-ratio pruning harms fine-grained tasks; instance-adaptive retention tuned by visual attention value gives OCR-oriented benchmarks a relative edge of 8.8-12.3 percentage points over earlier pruning methods.
  • The framework transfers beyond EAGLE: with Cambrian-1's encoder set it records an average gain over Cambrian-1 while using 44% fewer visual tokens.
  • Token budgets can be pushed further, to about 126 tokens per image, with the model still beating larger single-encoder baselines like Mini-Gemini on average.
  • Because decoding-stage pruning is training-free, the adaptive ratio can be deployed on an already fine-tuned model without additional data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Rank-guided budget allocation could serve other settings where several encoders or views are fused, such as multi-camera or multi-modal sensor inputs, wherever feature-map rank is cheap to estimate offline.
  • The correlation between top-k visual attention value and instance complexity suggests the same signal could be used as a proxy for task difficulty or as a confidence estimate for when the model needs more visual detail.
  • A testable extension would apply the adaptive ratio at the per-instance rather than per-dataset level; the paper shows dataset-level variation in retained tokens but does not establish that per-image budgets outperform per-dataset budgets given the same average.
  • The method's reliance on rank and similarity scores assumes redundancy is linear; nonlinear redundancy, such as complementary coding of the same semantic content by different encoders, might go undetected, which could set an upper bound on how far token counts can fall.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes METEOR, a multi-stage visual token pruning framework for multi-encoder vision-language models. It prunes tokens inside each vision encoder using a similarity/attention criterion with per-encoder budgets allocated by feature-map rank, then reduces cross-encoder redundancy during fusion, and finally applies a text-prompt-aware, instance-adaptive pruning scheme during LLM decoding that selects the most relevant attention heads and adjusts the retained token count via a learned scaling constant. Experiments on 11 benchmarks with two LLM backbones and two encoder sets show that METEOR reduces visual tokens by 76% relative to EAGLE, saves 49% TFLOPS, and increases throughput by 46% while suffering only a 0.3% average accuracy drop, and it outperforms existing efficient MLLM methods, especially on OCR tasks.

Significance. If the empirical claims hold, the paper demonstrates a practical route to running multi-encoder MLLMs at a fraction of their usual visual-token budget, which is a relevant and timely contribution given the high overhead of multi-encoder fusion. The strengths include broad benchmark coverage (11 benchmarks, two LLMs, two encoder sets), comparisons against several training-free and training-based pruning methods, an extension to Cambrian-1 encoders showing generality, and release of the code. The main limitations are that several load-bearing components are under-specified (the rank criterion, the scaling constant λ, the hand-set hyperparameters) and the headline '0.3% drop' is reported without variance estimates, making the precise accuracy-efficiency trade-off difficult to assess.

major comments (4)
  1. [Section 3.2, Eq. (3), Finding 2, Figure 6(b)] The manuscript never defines how the rank of a feature map is computed numerically (e.g., a threshold on singular values), and the SVD argument in Eq. (3) only shows that a low-rank approximation exists; it does not establish that rank is a measure of task-relevant information. Figure 4(a) reports only average ranks without variance or per-encoder values, and Figure 6(b) reports only aggregate accuracies for the three allocation strategies, not the resulting per-encoder token budgets. Consequently, the evidence does not rule out that rank is saturated near min(N,D) for all encoders, in which case the allocation k_l^b = k_b · r_l^b / Σ r_c^b becomes approximately uniform and the claimed collaborative assignment is inert. Please report actual rank values, the resulting budget fractions per encoder, and a variance/spectra analysis, and clarify how rank is computed numerically.
  2. [Section 3.4, Eq. (6)] The scaling constant λ is never specified anywhere in the paper or the experiments. The retained token counts marked with an asterisk (312*, 242*, 126*) are described as 'adaptive,' but they appear to be controlled by λ, which is not reported. Without the value of λ or a description of how it is selected for each configuration, Stage 3 is not reproducible, and the claim that pruning ratios are 'dynamically adjusted' for task demands is underdetermined. Please report λ for each configuration, state whether a single global λ is used across all datasets, and provide a sensitivity analysis over λ.
  3. [Section 3.4, Table 5] The method depends on two hand-set choices: the number of top attention heads k and the LLM pruning layer indices [4, 12, 20]. The ablation in Table 5 compares head filtering to no filtering but does not vary k, and no experiment varies the pruning layer indices. These choices are load-bearing for the Stage 3 results, and without sensitivity studies the reported improvements could be specific to the selected hyperparameter values. Please add ablations over k and over the pruning layer indices.
  4. [Section 4.2, Tables 1 and 2] All benchmark numbers are reported from single runs, with no error bars or multiple seeds. The headline claim of 'only 0.3% average performance drop' is within typical benchmark noise—the per-benchmark differences are mixed, some positive and some negative—so the current evidence does not statistically support the claim that the drop is negligible. Reporting variance across at least three seeds for the main EAGLE† versus METEOR comparison would substantiate the central efficiency-accuracy trade-off.
minor comments (7)
  1. [Section 3.2, Figure 3] The text says 'As shown in Figure 3(a)' when discussing the high entropy of attention values in shallow layers, but the entropy and Kendall tau plots appear in Figure 3(b); the figure cross-reference is inconsistent.
  2. [Related Work, Section 2] The phrase 'Prompt-agonistic methods' should read 'Prompt-agnostic methods'.
  3. [Section 3.2, Eq. (1)] The top-k selection expression 'sort(-S(...)) < k' is ambiguous; it should be clarified that the k tokens with the smallest cosine similarity to the average token are retained (or alternatively, the k largest similarities are discarded).
  4. [Figure 1 caption and Section 4] The label 'EAGLE-X4' in Figure 1 is not defined in the text; please use consistent terminology with the EAGLE model described in Section 4.
  5. [Section 3.2, Eq. (3)] The SVD in Eq. (3) reuses the subscript b for both the block index and the singular-value index, which is confusing; use separate indices for the singular components.
  6. [Section 3.2, Figure 4(a) and Finding 2] The paper claims the variance of rank is 'negligible' but provides no numerical measure of variance; please report the variance or a confidence interval to support this claim.
  7. [Section 3.2, Stage 1] The phrase 'we equally split the model into three phases' is unclear—what model, and how are the phases defined? Please specify the layer ranges for the shallow and deep phases.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline efficiency/accuracy results are measured on external benchmarks, and the rank/VAV allocation rules are explicit design choices with external precedent rather than predictions reduced to fitted inputs.

full rationale

The paper's central efficiency claims (76% visual-token reduction, 49% TFLOPS saving, 46% throughput gain at a 0.3% average accuracy drop) are empirical measurements on 11 external benchmarks against EAGLE and other baselines; they are not derived from the method's definitions. The rank-based budget k_l^b = k_b * r_l^b / sum_l r_l^b (Section 3.2) is a stated design rule, not a fitted prediction: its premise that higher rank means more information is argued from SVD, grounded in external work HRank [37], and tested against alternatives in the Figure 6(b) ablation. The adaptive retained-token rule K = lambda * sum_i a_{i,I(h)} (Eq. 6) explicitly defines the retained-token count as a linear function of observed visual-attention values; lambda is presented as a scaling constant rather than a hidden fitted parameter, and the headline performance is a measured outcome of applying that rule. Table 5 further compares adaptive versus fixed ratios at the same average token budget, so the reported gains are not an artifact of the scaling constant. Self-citations such as [18] and [24] appear only in motivation and related work and are not load-bearing for the main results. No uniqueness theorem is imported, and the rank criterion is attributed to external prior work rather than to the authors' own unverified claims. The weakest assumption, that rank tracks task-relevant information richness, is an empirical/correctness concern about saturation and calibration, not a circular reduction: the benchmark results would remain independent evidence even if the rank heuristic were suboptimal.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; the 'visual contribution level' is a reweighted sum of existing attention values. The central claim rests on several domain assumptions about token significance and information richness, and on a few hand-set hyperparameters.

free parameters (5)
  • lambda (scaling constant in Eq. 6) = not reported; chosen to yield average retained token counts of ~312, ~242, ~126
    In Eq. (6), K = lambda * sum(VAV). The paper says lambda is a scaling constant but does not report its value or sensitivity. Different token-budget configurations appear to be obtained by tuning lambda, so the adaptive pruning ratio is partly a fitted quantity.
  • top-k attention heads k = not reported
    Stage 3 selects the top-k most significant attention heads for computing visual attention values. The value of k is not specified, yet performance depends on it.
  • pruning layer indexes in LLM = [4, 12, 20]
    Table 5 shows pruning at layers 4, 12, and 20. These indexes are manually chosen; no sensitivity analysis is provided for other layer choices.
  • vision encoder 3-phase split = first phase shallow, last two phases deep
    The split between using average-token similarity (shallow) and class attention (deep) is a design choice, not derived. It affects which tokens are pruned in Stage 1.
  • fusion-stage retained token budget = 576 tokens
    The number of tokens kept after cross-encoder cooperative pruning during SFT is fixed at 576. This is a hand-set hyperparameter, and the paper does not study its sensitivity extensively.
assumptions (4)
  • domain assumption The rank of a feature map measures information richness: higher rank means more information.
    Borrowed from HRank [37] for CNN filter pruning and extended to ViT tokens. It is an empirical heuristic, not a proven theorem for token pruning.
  • domain assumption Average token similarity identifies redundant tokens in shallow layers, while class-token attention identifies redundancy in deep layers.
    Based on prior single-encoder pruning works; the paper validates on its own encoders but provides no formal justification for the phase split.
  • domain assumption Visual attention values of the top-k heads correlate with instance complexity, so a larger sum requires more retained tokens.
    Empirical observation on a small set of datasets (Figure 5c); no statistical test or theoretical justification is given.
  • domain assumption The expectation of feature-map rank is stable across input images, making offline computation valid.
    Reported as negligible variance on a small batch (Figure 4a), without quantitative bounds or a distributional argument.

how reviews work

0 comments
Cite this review

Pith. "Pith review of METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models." pith.science (2026). https://pith.science/paper/WCMZYE6A

@misc{pith2026250720842,
  author       = {Pith},
  title        = {Pith review of: METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WCMZYE6A}},
  note         = {Machine review of arXiv:2507.20842}
}
read the original abstract

Vision encoders serve as the cornerstone of multimodal understanding. Single-encoder architectures like CLIP exhibit inherent constraints in generalizing across diverse multimodal tasks, while recent multi-encoder fusion methods introduce prohibitive computational overhead to achieve superior performance using complementary visual representations from multiple vision encoders. To address this, we propose a progressive pruning framework, namely Multi-Encoder collaboraTivE tOken pRuning (METEOR), that eliminates redundant visual tokens across the encoding, fusion, and decoding stages for multi-encoder MLLMs. For multi-vision encoding, we discard redundant tokens within each encoder via a rank guided collaborative token assignment strategy. Subsequently, for multi-vision fusion, we combine the visual features from different encoders while reducing cross-encoder redundancy with cooperative pruning. Finally, we propose an adaptive token pruning method in the LLM decoding stage to further discard irrelevant tokens based on the text prompts with dynamically adjusting pruning ratios for specific task demands. To our best knowledge, this is the first successful attempt that achieves an efficient multi-encoder based vision language model with multi-stage pruning strategies. Extensive experiments on 11 benchmarks demonstrate the effectiveness of our proposed approach. Compared with EAGLE, a typical multi-encoder MLLMs, METEOR reduces 76% visual tokens with only 0.3% performance drop in average. The code is available at https://github.com/YuchenLiu98/METEOR.

Figures

Figures reproduced from arXiv: 2507.20842 by the authors.

Figure 1
Figure 1. Overview of METEOR, which collaboratively prune re [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall architecture of METEOR. Redundant visual tokens are pruned within each encoder during encoding in stage 1. The rank [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. (a) Vision encoding and LLM prefilling latency (fed [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: (a) Rank statistics of feature maps across different phases [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: (a) Accuracy and throughput of full model, pruning with [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Final retained visual token number for different datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

84 extracted references · 52 canonical work pages

  1. [24]

    From CLIP to DINO: Visual encoders shout in multi-modal large language models

    Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin’e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong. From CLIP to DINO: Visual encoders shout in multi-modal large language models. arXiv preprint arXiv:2310.08825, 2023. 2, 3

    arXiv

  2. [1]

    HiRED: Attention-guided token dropping for efficient inference of high-resolution vision-language models

    Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S Nikolopou- los, Hans Vandierendonck, Deepu John, and Bo Ji. HiRED: Attention-guided token dropping for efficient inference of high-resolution vision-language models. In Proceedings of the 39th AAAI Conference on Artificial Intelligence , pages 1773–1781, 2025. 3

    2025

  3. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Day- iheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfe...

    arXiv 2023

  4. [3]

    Qwen-VL: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 ,

    arXiv

  5. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S...

    1901

  6. [5]

    Matryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. In The Thirteenth Interna- tional Conference on Learning Representations, 2025. 1, 3

    2025

  7. [6]

    Open-LLaV A-NeXT: An open- source implementation of LLaV A-NeXT series for facilitat- ing the large multi-modal model community

    Lin Chen and Long Xing. Open-LLaV A-NeXT: An open- source implementation of LLaV A-NeXT series for facilitat- ing the large multi-modal model community. https:// github . com / xiaoachen98 / Open - LLaVA - NeXT,

  8. [7]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceler- ation for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceler- ation for large vision-language models. In Proceedings of the 18th European Conference on Computer Vision (ECCV), pages 19–35, 2024. 1, 3, 5, 6, 7

    work page 2024

Show all 84 references
  1. [8]

    How far are we to GPT- 4V? Closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhan...

    2024

  2. [9]

    Intern VL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Intern VL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In 2024 IEE...

    2024

  3. [10]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. https://lmsys.org/ blog/2023-0...

    2023

  4. [11]

    InstructBLIP: Towards general-purpose vision- language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning. In Advances in Neural Information Processing Systems 36 , pages 49250...

    2023

  5. [12]

    Transformer-XL: Attentive language models beyond a fixed-length context

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, 2019. 1

    2019

  6. [13]

    Prune spatio-temporal to- kens by semantic-aware temporal accumulation

    Shuangrui Ding, Peisen Zhao, Xiaopeng Zhang, Rui Qian, Hongkai Xiong, and Qi Tian. Prune spatio-temporal to- kens by semantic-aware temporal accumulation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16945–16956, 2023. 1

    2023

  7. [14]

    MouSi: Poly-visual-expert vision-language models

    Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, Ming Zhang, Caishuang Huang, Rui Zheng, Zhiheng Xi, Yuhao Zhou, Shihan Dou, Junjie Ye, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and...

    2024 arXiv

  8. [15]

    MME: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 6

    2023 arXiv

  9. [16]

    Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images

    Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images. InProceedings of the 18th European Con- ference on Computer Vision (ECCV), pages 390–406, 2...

    2024

  10. [17]

    Re- thinking token reduction in MLLMs: Towards a unified paradigm for training-free acceleration

    Yuhang Han, Xuyang Liu, Pengxiang Ding, Donglin Wang, Honggang Chen, Qingsen Yan, and Siteng Huang. Re- thinking token reduction in MLLMs: Towards a unified paradigm for training-free acceleration. arXiv preprint arXiv:2411.17686, 2024. 3

    2024

  11. [18]

    In- corporating visual experts to resolve the information loss in multimodal large language models

    Xin He, Longhui Wei, Lingxi Xie, and Qi Tian. In- corporating visual experts to resolve the information loss in multimodal large language models. arXiv preprint arXiv:2401.03105, 2024. 3

    2024 arXiv

  12. [19]

    ZipVL: Efficient large vision-language models with dynamic token spar- sification and KV cache compression

    Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. ZipVL: Efficient large vision-language models with dynamic token spar- sification and KV cache compression. arXiv preprint arXiv:2410.08584, 2024. 3

    arXiv

  13. [20]

    iLLaV A: An image is worth fewer than 1/3 input tokens in large multimodal models

    Lianyu Hu, Fanhua Shang, Liang Wan, and Wei Feng. iLLaV A: An image is worth fewer than 1/3 input tokens in large multimodal models. arXiv preprint arXiv:2412.06263,

  14. [21]

    Matryoshka query Trans- former for large vision-language models

    Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query Trans- former for large vision-language models. In Advances in Neural Information Processing Systems 37 , pages 50168– 50188, 2024. 3

    2024

  15. [22]

    IVTP: Instruction-guided visual token pruning for large vision-language models

    Kai Huang, Hao Zou, Ye Xi, BoChen Wang, Zhen Xie, and Liang Yu. IVTP: Instruction-guided visual token pruning for large vision-language models. InProceedings of the 18th Eu- ropean Conference on Computer Vision (ECCV), pages 214– 230, 2024. 3

    2024

  16. [23]

    GQA: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In 2019 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 6700– 6709, 2019. 6

    2019

  17. [25]

    FoPru: Focal pruning for efficient large vision-language models

    Lei Jiang, Weizhe Huang, Tongxuan Liu, Yuting Zeng, Jing Li, Lechao Cheng, and Xiaohua Xu. FoPru: Focal pruning for efficient large vision-language models. arXiv preprint arXiv:2411.14164, 2024. 1, 3, 4

    2024 arXiv

  18. [26]

    Devils in middle layers of large vision- language models: Interpreting, detecting and mitigating ob- ject hallucinations via attention lens

    Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision- language models: Interpreting, detecting and mitigating ob- ject hallucinations via attention lens. In 2025 IEEE/CVF Conference on Computer Vision and Pattern...

    2025

  19. [27]

    BRA VE: Broadening the visual encoding of vision-language models

    O ˘guzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. BRA VE: Broadening the visual encoding of vision-language models. In Proceedings of the 18th European Conference on Com- puter Vision (ECCV), pages 113–132, 2024. 2, 3, 4, 6

    2024

  20. [28]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Proceedings of the 14th Euro- pean Conference on Computer Vision (ECCV) , pages 235– 251, 2016. 6

    2016

  21. [29]

    MoAI: Mixture of all intelligence for large language and vision models

    Byung-Kwan Lee, Beomchan Park, Chae Won Kim, and Yong Man Ro. MoAI: Mixture of all intelligence for large language and vision models. In Proceedings of the 18th Eu- ropean Conference on Computer Vision (ECCV), pages 273– 302, 2024. 3

    2024

  22. [30]

    Pix2Struct: Screenshot parsing as pretraining for visual lan- guage understanding

    Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandel- wal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2Struct: Screenshot parsing as pretraining for visual lan- guage understanding. In Proceedings of the 40...

    2023

  23. [31]

    SEED-Bench: Benchmarking multi- modal large language models

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. SEED-Bench: Benchmarking multi- modal large language models. In 2024 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 13299–13308, 2024. 6

    2024

  24. [32]

    RedundancyLens: Revealing and exploiting visual token processing redundancy for efficient decoder-only MLLMs

    Hongliang Li, Jiaxin Zhang, Wenhui Liao, Dezhi Peng, Kai Ding, and Lianwen Jin. RedundancyLens: Revealing and exploiting visual token processing redundancy for efficient decoder-only MLLMs. arXiv preprint arXiv:2501.19036 ,

    arXiv

  25. [33]

    TokenPacker: Efficient visual projector for multimodal LLM.International Journal of Computer Vision, pages 1–19, 2025

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. TokenPacker: Efficient visual projector for multimodal LLM.International Journal of Computer Vision, pages 1–19, 2025. 3, 6

    2025

  26. [34]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pages 292–305, 2023. 6

    2023

  27. [35]

    Mini-Gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-Gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  28. [36]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),...

    2024

  29. [37]

    HRank: Filter pruning using high-rank feature map

    Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. HRank: Filter pruning using high-rank feature map. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1529–1538, 2020. 4

    2020

  30. [38]

    SPHINX: A mixer of weights, visual em- beddings and image scales for multi-modal large language models

    Ziyi Lin, Dongyang Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Yu Qiao, and Hongsheng Li. SPHINX: A mixer of weights, visual em- beddings and image scales for multi-modal large languag...

    2024

  31. [39]

    Boosting multimodal large language models with visual to- kens withdrawal for rapid inference

    Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual to- kens withdrawal for rapid inference. In Proceedings of the 39th AAAI Conference on Artificial Intelligence , pages 5334–5342, 2025. 7

    2025

  32. [40]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems 36, pages 34892–34916, 2023. 1, 3

    2023

  33. [41]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, 2024. 6

    2024

  34. [42]

    LLaV A-NeXT: Im- proved reasoning, OCR, and world knowledge

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A-NeXT: Im- proved reasoning, OCR, and world knowledge. https: / / llava - vl . github . io / blog / 2024 - 01 - 30 - llava-next/, 2024. 2, 3, 6

    2024

  35. [43]

    Prismer: A vision-language model with multi-task experts

    Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Prismer: A vision-language model with multi-task experts. Transactions on Machine Learning Resear, 2024. 2, 3

    2024

  36. [44]

    Multi-stage vision token dropping: Towards efficient multimodal large language model

    Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.10803, 2024. 3

    2024 arXiv

  37. [45]

    On the hidden mystery of OCR in large multimodal models

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of OCR in large multimodal models. arXiv preprint arXiv:2305.07895, 2023. 6

    2023 arXiv

  38. [46]

    MMBench: Is your multi-modal model an all-around player? In Proceedings of the 18th European Conference on Computer Vision (ECCV), pages 216–233, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? In Proceedings of the 18th European Conference on Computer Vision (ECCV), ...

    2024

  39. [47]

    A ConvNet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11976–11986, 2022. 6

    2022

  40. [48]

    DeepSeek-VL: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. DeepSeek-VL: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  41. [49]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems 35 , pag...

    2022

  42. [50]

    Feast your eyes: Mixture- of-resolution adaptation for multimodal large language mod- els

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xi- aoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture- of-resolution adaptation for multimodal large language mod- els. In The Thirteenth International Conference on Learning Representations, 2025. 4, 6

    2025

  43. [51]

    OK-VQA: A visual question answer- ing benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answer- ing benchmark requiring external knowledge. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3190–3199, 2019. 6

    2019

  44. [52]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, 2022. 6

    2022

  45. [53]

    DocVQA: A dataset for VQA on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. DocVQA: A dataset for VQA on document images. In 2021 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2200–2209, 2021. 6

    2021

  46. [54]

    MM1: methods, analysis and insights from multimodal LLM pre- training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu H`e, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang...

    2024

  47. [55]

    DeepStack: Deeply stacking visual tokens is surprisingly simple and ef- fective for LMMs

    Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zux- uan Wu, Jianfeng Gao, and Yu-Gang Jiang. DeepStack: Deeply stacking visual tokens is surprisingly simple and ef- fective for LMMs. In Advances in Neural Information Pro- cessing Systems 37, pages 23464–23487, 2024. 6

    2024

  48. [56]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  49. [57]

    LLaV A-PruMerge: Adaptive token reduc- tion for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaV A-PruMerge: Adaptive token reduc- tion for efficient large multimodal models. arXiv preprint arXiv:2403.15388, 2024. 1, 3, 4

    2024

  50. [58]

    Eagle: Exploring the design space for multimodal LLMs with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catan- zaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal LLMs wi...

    2025

  51. [59]

    Towards VQA models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8317–8326, 2019. 6

    2019

  52. [60]

    EV A-CLIP: Improved training techniques for CLIP at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EV A-CLIP: Improved training techniques for CLIP at scale. arXiv preprint arXiv:2303.15389, 2023. 3, 6

    2023 arXiv

  53. [61]

    Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In Ad...

    2024

  54. [62]

    Eyes wide shut? exploring the vi- sual shortcomings of multimodal LLMs

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the vi- sual shortcomings of multimodal LLMs. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9568–9578, 2024. 3

    2024

  55. [63]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation langua...

    2023 arXiv

  56. [64]

    [CLS] token tells everything needed for training-free efficient MLLMs

    Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. [CLS] token tells everything needed for training-free efficient MLLMs. arXiv preprint arXiv:2412.05819, 2024. 3, 4

    2024 arXiv

  57. [65]

    FOLDER: Accelerat- ing multi-modal large language models with enhanced per- formance

    Haicheng Wang, Zhemeng Yu, Gabriele Spadaro, Chen Ju, Victor Qu´etu, and Enzo Tartaglione. FOLDER: Accelerat- ing multi-modal large language models with enhanced per- formance. arXiv preprint arXiv:2501.02430, 2025. 3

    2025 arXiv

  58. [66]

    Vary: Scaling up the vision vocabulary for large vision-language model

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language model. In Proceedings of the 18th Euro- pean Conference on Computer Vision (ECCV) , pages 40...

    2024

  59. [67]

    PyramidDrop: Accelerating your large vision-language models via pyramid visual redundancy reduction

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and Dahua Lin. PyramidDrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247, 2024. 1, 3...

    arXiv

  60. [68]

    Mitigat- ing hallucination in large vision-language models via mod- ular attribution and intervention

    Tianyun Yang, Ziniu Li, Juan Cao, and Chang Xu. Mitigat- ing hallucination in large vision-language models via mod- ular attribution and intervention. In The Thirteenth Interna- tional Conference on Learning Representations, 2025. 5

    2025

  61. [69]

    DeCo: Decoupling token compres- sion from semantic abstraction in multimodal large language models

    Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. DeCo: Decoupling token compres- sion from semantic abstraction in multimodal large language models. arXiv preprint arXiv:2405.20985, 2024. 1, 3, 7

    2024 arXiv

  62. [70]

    mPLUG- Owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mPLUG- Owl2: Revolutionizing multi-modal large language model with modality collaboration. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1...

    2024

  63. [71]

    Fit and prune: Fast and training-free visual token pruning for multi- modal large language models

    Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training-free visual token pruning for multi- modal large language models. In Proceedings of the 39th AAAI Conference on Artificial Intelligence , pages 22128– 22136, 2025. 1, 3, 5

    2025

  64. [72]

    ATP-LLaV A: Adaptive token pruning for large vision language models

    Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. ATP-LLaV A: Adaptive token pruning for large vision language models. In 2025 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 24972–24982, 2025. 3, 5

    2025

  65. [73]

    V oCo-LLaMA: Towards vision compression with large language models

    Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, and Yansong Tang. V oCo-LLaMA: Towards vision compression with large language models. In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 29836–29846, 2025. 3

    2025

  66. [74]

    Lifting the veil on visual information flow in MLLMs: Unlocking pathways to faster inference

    Hao Yin, Guangzong Si, and Zilei Wang. Lifting the veil on visual information flow in MLLMs: Unlocking pathways to faster inference. In 2025 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 9382– 9391, 2025. 1, 3

    2025

  67. [75]

    [CLS] attention is all you need for training-free visual token pruning: Make VLM inference faster

    Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shang- hang Zhang. [CLS] attention is all you need for training-free visual token pruning: Make VLM inference faster. arXiv preprint arXiv:2412.01818, 2024. 1, 3, 4

    2024 arXiv

  68. [76]

    LLaV A-Mini: Efficient image and video large multimodal models with one vision token

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. LLaV A-Mini: Efficient image and video large multimodal models with one vision token. In The Thirteenth Interna- tional Conference on Learning Representations, 2025. 3

    2025

  69. [77]

    Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in LVLMs

    Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, and Jieping Ye. Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in LVLMs. arXiv preprint arXiv:2411.09968, 2024. 5

    2024 arXiv

  70. [78]

    SparseVLM: Visual token sparsification for efficient vision- language model inference

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. SparseVLM: Visual token sparsification for efficient vision- language model inference. In Proceedings of the 42nd In- t...

    2025

  71. [79]

    Treat visual tokens as text? but your MLLM only needs fewer efforts to see

    Zeliang Zhang, Phu Pham, Wentian Zhao, Kun Wan, Yu-Jhe Li, Jianing Zhou, Daniel Miranda, Ajinkya Kale, and Chen- liang Xu. Treat visual tokens as text? but your MLLM only needs fewer efforts to see. arXiv preprint arXiv:2410.06169,

    arXiv

  72. [80]

    Accelerating multimodel large language models by searching optimal vision token re- duction

    Shiyu Zhao, Zhenting Wang, Felix Juefei-Xu, Xide Xia, Miao Liu, Xiaofang Wang, Mingfu Liang, Ning Zhang, Dim- itris N Metaxas, and Licheng Yu. Accelerating multimodel large language models by searching optimal vision token re- duction. arXiv preprint arXiv:2412.00556, 2024. 3

    2024 arXiv

  73. [81]

    AIM: Adaptive inference of multi-modal LLMs via token merging and pruning

    Yiwu Zhong, Zhuoming Liu, Yin Li, and Liwei Wang. AIM: Adaptive inference of multi-modal LLMs via token merging and pruning. arXiv preprint arXiv:2412.03248, 2024. 3

    2024 arXiv

  74. [82]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 3

    2023 arXiv

  75. [83]

    FocusLLaV A: A coarse-to-fine approach for effi- cient and effective visual token compression

    Yuke Zhu, Chi Xie, Shuang Liang, Bo Zheng, and Sheng Guo. FocusLLaV A: A coarse-to-fine approach for effi- cient and effective visual token compression. arXiv preprint arXiv:2411.14228, 2024. 3

    2024 arXiv

  76. [84]

    MoV A: Adapting mixture of vision experts to multimodal context

    Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. MoV A: Adapting mixture of vision experts to multimodal context. In Advances in Neural Information Processing Sys- tems 37, pages 103305–103333, 2024. 2, 3

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.