Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Learning Compact Vision Tokens for Efficient Large Multimodal Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fusing neighboring vision tokens with a learnable convolution and supplementing features from eight frozen-encoder layers lets a LLaVA-1.5 model match its full-token baseline using only 25% of the vision tokens, at 1.9 TFLOPs instead of…

desk verdict A solid efficiency result with a mislabeled 'lossless' fusion and a missing TokenPacker comparison; worth refereeing after those are fixed. read the letter →

arxiv 2506.07138 v1 pith:KVJTAX3T submitted 2025-06-08 cs.CV cs.AIcs.CLcs.MM

Hao Tang , Chengchao Shen This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.MM
keywords largemultimodalmodelsvisiontokencompressionspatialfusionmulti-blockreductioninferenceaccelerationLLaVAvision-languagebenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the long vision-token sequences fed into large multimodal models are heavily spatially redundant, so a learnable fusion of neighboring tokens can cut the sequence to a quarter of its original length without sacrificing benchmark performance. The authors show that even plain average pooling to 25% of the tokens loses little on most benchmarks, and that their two-stage fusion—concatenating features from eight layers of the vision encoder, whose weights remain frozen, then fusing 2x2 neighborhoods with a learnable convolution—matches or exceeds the full-token LLaVA-1.5-7B on the average of eight vision-language benchmarks while dropping inference cost from 7.6 to 1.9 TFLOPs. If correct, this means much of the compute spent on long vision prefixes is unnecessary, and that fixed vision encoders can be compensated by multi-layer feature fusion instead of being retrained.

What carries the argument

Spatial Token Fusion (STF) is a convolution with kernel and stride 2x2 that maps a 24x24x1024 token map to 12x12x4096, so each compact token absorbs four adjacent tokens and its 4096-channel width matches the text-embedding width of the LLM, making the reduction lossless in the channel dimension. Multi-Block Token Fusion (MBTF) concatenates the outputs of blocks 3, 6, 9, 12, 15, 18, 21, and 24 of the frozen CLIP ViT-L/14 encoder and fuses them with two 1x1 convolutions (4096 then 1024 channels) before STF runs. A reshape step allows E fused tokens per neighborhood, with E=1 by default, and the reported gains use that setting.

What would settle it

A concrete test: on a dense fine-grained benchmark—say TextVQA at higher resolution or a small-object counting set—compare the 25%-token STC model against the full-token LLaVA-1.5 baseline and against the MBTF-only model at full FLOPs. If STC drops substantially below the baseline while MBTF-only does not, the spatial-redundancy premise fails; if both drop together, the loss lies in multi-block feature fusion rather than in the 4x spatial reduction.

Watch

Extended reading notes

Core claim

The central claim is that spatial redundancy, not token count, dominates the vision-token cost of LLaVA-style models. Concretely, the paper's combined model (STC) compresses the 576 vision tokens of LLaVA-1.5-7B to 144 tokens—25% of the original—and still averages 66.3% across GQA, SQA, TextVQA, POPE, MMBench, MMBench-CN, VQAv2, and VisWiz, against 65.5% for the full-token LLaVA-1.5 full-tuning baseline, at 1.9 TFLOPs versus 7.6 TFLOPs. The authors attribute the result to two complementary modules: MBTF, which fuses intermediate-block features and alone lifts the full-token baseline by 1.1 points, and STF, which alone holds the baseline at 25% tokens. They caution that on some individual benchmarks the compressed model is still inferior even though the average is higher.

Load-bearing premise

The method assumes that the fine visual details lost when four neighboring image tokens are squeezed into one are already stored redundantly across the eight sampled layers of the vision encoder, whose weights are left unchanged; if a task needs a detail that lives in only one layer and the squeezing destroys it, the quarter-token model will fall behind the full-token one.

Editorial extensions

If this is right

  • A 4x reduction in vision tokens cuts the measured inference cost of LLaVA-1.5-7B from 7.6 to 1.9 TFLOPs while holding the 8-benchmark average at or above the full-token baseline.
  • Plain average pooling to 25% of the tokens already outperforms several published efficient-LLaVA methods, indicating that the redundancy is structural and not specific to the learnable fusion.
  • MBTF's standalone improvement over the full-token baseline implies that intermediate-block features carry information the final block alone misses, so token compression can be paired with layer fusion rather than encoder unfreezing.
  • Aggressive fusion (k=4 or k=8) degrades accuracy, so the paper's claim is for a moderate 4x spatial reduction, not for arbitrary token compression.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not run: if 4x spatial fusion is nearly lossless at 336x336 resolution, the same fusion ratio at higher input resolutions (e.g., 672x672) should preserve proportionally more detail and may push the average further above baseline.
  • Because STF removes spatial redundancy before the LLM while text-guided methods remove task-irrelevant tokens, combining both families should compound savings; the paper's analysis does not rule this out.
  • MBTF's success suggests that fusing a few intermediate blocks of a frozen encoder could serve as a general drop-in replacement for unfreezing the encoder in other LMMs, not just LLaVA-1.5.
  • The 4096-to-4096 channel match means the fusion bottleneck is purely spatial; if that is right, further gains at 25% tokens must come from better multi-block fusion, not from keeping more tokens per neighborhood.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a two-stage vision token compression method for LLaVA-style large multimodal models. A Multi-Block Token Fusion (MBTF) module concatenates features from eight evenly sampled CLIP ViT-L/14 blocks and fuses them with 1x1 convolutions, and a Spatial Token Fusion (STF) module applies a strided k x k convolution (default k=2) to fuse adjacent tokens, reducing the token count to 25% of the original. The reduced model is trained with the standard LLaVA-1.5 two-stage procedure and evaluated on eight vision-language benchmarks. The authors report an average of 66.3% at 1.9 TFLOPs, slightly above the full-token LLaVA-1.5 full-tuning baseline (65.5%) and above several other efficient LMM methods, while ablations examine the contribution of each module, the kernel size k, the number of fused tokens E, and alternative fusion strategies.

Significance. If the reported results are robust, the paper provides evidence for strong spatial redundancy in CLIP visual features and offers a practical way to cut the compute of LMM inference by roughly 4x with minimal accuracy change. The release of code and weights is a concrete strength, as is the inclusion of a simple average-pooling baseline that helps calibrate how much redundancy exists. The ablations are fairly thorough, covering module contributions, fusion kernel size, number of output tokens, and fusion strategies. The main weaknesses are that the central 'lossless' claim is mathematically unsupported and internally contradicted by Table 3, and that all headline comparisons rest on single runs without variance estimates, so the claimed superiority over the full-token baseline is not yet established with statistical confidence.

major comments (3)
  1. [Sections 3.1 and 3.4, Eq. (2)] The claim that token reduction is 'lossless' is not justified. Section 3.1 states the aim of reducing aligned tokens 'in a lossless manner', and Section 3.4 argues that a 2x2 convolution with C2 = k^2 * C1 achieves 'lossless token reduction'. However, Eq. (2) defines STF as a strided 2x2 convolution followed by GeLU; a nonlinear strided convolution is not an information-preserving map in general. Moreover, the paper's own ablation in Table 3 shows that adding STF to MBTF decreases the average score from 66.6 to 66.3, which is direct evidence that the spatial reduction is lossy. The authors should reframe the contribution as learned lossy compression that preserves task-relevant information, and remove the 'lossless' terminology.
  2. [Tables 2 and 3] The central empirical claim of 'comparable or even superior performance' to LLaVA-1.5 relies on single training runs with no error bars or significance testing. The headline 0.8-point average gain over full-token LLaVA-1.5 full tuning (66.3 vs. 65.5) could plausibly be within run-to-run variance, especially since several individual benchmarks are lower for the proposed method (e.g., GQA 61.9 vs. 62.0, TextVQA 57.4 vs. 58.2, POPE 86.0 vs. 85.9 is a tie within rounding). The authors should report results over multiple seeds or at least provide a statistical significance test (e.g., paired bootstrap) to support the superiority claim.
  3. [Tables 2 and 4] The best configuration (k=2, E=1) is selected by looking at performance on the same eight benchmarks used for the final comparison, which introduces selection bias. The reported 66.3 average is therefore an optimistic estimate of the method's expected performance on those benchmarks. The authors should either use a validation split for hyperparameter choice or report performance across a small grid of hyperparameters without cherry-picking the best row, to make the comparison in Table 2 fair.
minor comments (4)
  1. [Table 2] The FastV row has a missing SQA score and a missing VQAT score, yet an average is reported; the authors should state how averages are computed when entries are missing, or fill in the missing values, so that the comparison is apples-to-apples.
  2. [Section 4.3.2] The explanation that larger kernel sizes overfit 'due to the deficit of training data' is speculative; no training loss, generalization gap, or variance evidence is provided. It would be better to report the number of parameters added by each kernel size and, if possible, training curves.
  3. [Section 4.1.2 and Table 2] MME-Perception is listed as an evaluation benchmark but does not appear in Table 2; either add the corresponding result or remove the benchmark from the setup description.
  4. [Figure 3 and Section 4.4] The text refers to the cattle-counting example as 'Figure 3 (d)', but in the figure the cattle image is panel (e); the panel references should be corrected. There are also several typos, including 'pretrainig', 'finetuing', and 'acheves'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper's central claim is an empirical architecture evaluated on external benchmarks, and its 'lossless' framing is mathematically unsupported but not circular.

full rationale

The paper does not derive its performance from a fitted quantity or from a self-citation chain. The central claim, that LLaVA-1.5 with 25% vision tokens reaches comparable or better benchmark accuracy, is an empirical result reported in Table 2 against pre-existing baselines on eight external vision-language benchmarks. The two proposed modules, MBTF (Eq. 1) and STF (Eq. 2), are architecturally defined operations, not quantities fit to the benchmarks. The passage in Section 3.4 that describes the STF convolution as 'lossless token reduction' because C2 = C3 = 4096 is an unsupported mathematical equivalence, but it is not circular: the method's reported accuracy does not presuppose that the reduction is lossless, and Table 3 even shows that adding STF to MBTF slightly lowers the average score (66.3 vs. 66.6), which is consistent with the view that the reduction is lossy. There are self-citations ([30] and [32]), but they appear only in related-work enumerations and are not load-bearing for the method or its results. Hyperparameters k and E are ablated in Table 4, yet the reported model is not obtained by optimizing against the benchmark targets; the ablations are discrete architecture choices, and the final scores are honest evaluations. No prediction reduces by construction to an input, and no load-bearing premise is justified only by a self-citation. The paper is self-contained as an empirical systems contribution.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a set of architectural hyperparameters chosen via ablation on the same benchmarks, plus standard assumptions about transferability of LLaVA training and the informativeness of multi-block CLIP features. No new physical or conceptual entities are introduced.

free parameters (5)
  • kernel size k = 2
    Chosen by ablation on the 8 benchmark suite (Table 4); larger k degrades average performance.
  • target fused tokens E = 1
    Chosen by ablation (Table 4); E=1 gives the best average score.
  • number of selected blocks M = 8
    Set by design, sampling every 3 from 24 ViT blocks; no ablation is shown for this choice.
  • selected block indices = {3,6,9,12,15,18,21,24}
    Evenly sampled; not ablated.
  • STF intermediate channel width = 16384
    Architecture choice in Table 1; no justification is provided for this expansion.
assumptions (4)
  • domain assumption CLIP ViT-L/14 features from early blocks are useful for vision-language tasks.
    MBTF relies on features from blocks 3, 6, 9, etc. (Section 3.3).
  • domain assumption The LLaVA-1.5 training data and recipe are sufficient to train the added fusion modules without degrading the LLM.
    Section 4.1.3 describes using the same data and schedule as LLaVA-1.5.
  • domain assumption Average pooling baseline accurately measures the degree of spatial redundancy.
    Figure 1 and Section 1 use average pooling as evidence that 75% of vision tokens are redundant.
  • domain assumption Benchmark scores are deterministic enough that single runs support the parity claim.
    Section 4.2 reports one run per configuration with no variance estimates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Compact Vision Tokens for Efficient Large Multimodal Models." pith.science (2026). https://pith.science/paper/KVJTAX3T

@misc{pith2026250607138,
  author       = {Pith},
  title        = {Pith review of: Learning Compact Vision Tokens for Efficient Large Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KVJTAX3T}},
  note         = {Machine review of arXiv:2506.07138}
}
abstract

Large multimodal models (LMMs) suffer significant computational challenges due to the high cost of Large Language Models (LLMs) and the quadratic complexity of processing long vision token sequences. In this paper, we explore the spatial redundancy among vision tokens and shorten the length of vision token sequences for inference acceleration. Specifically, we propose a Spatial Token Fusion (STF) method to learn compact vision tokens for short vision token sequence, where spatial-adjacent tokens are fused into one. Meanwhile, weight-frozen vision encoder can not well adapt to the demand of extensive downstream vision-language tasks. To this end, we further introduce a Multi-Block Token Fusion (MBTF) module to supplement multi-granularity features for the reduced token sequence. Overall, we combine STF and MBTF module to balance token reduction and information preservation, thereby improving inference efficiency without sacrificing multimodal reasoning capabilities. Experimental results demonstrate that our method based on LLaVA-1.5 achieves comparable or even superior performance to the baseline on 8 popular vision-language benchmarks with only $25\%$ vision tokens of baseline. The source code and trained weights are available at https://github.com/visresearch/LLaVA-STF.

Figures

Figures reproduced from arXiv: 2506.07138 by the authors.

Figure 1
Figure 1. Spatial token redundancy in the visual context of LMMs. The performance gap between [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of our method. Our method follows LLaVA-style architecture and introduces [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Case study of LLaVA-1.5 and our proposed method. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    SDMPrune combines a self-distillation loss with Taylor-based importance scoring to prune only MLP neurons, improving zero-shot performance of compressed LLaMA models over existing pruning methods.

Reference graph

Works this paper leans on

41 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. 2023

    work page 2023

  2. [2]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Fu Chaoyou, Chen Peixian, Shen Yunhang, Qin Yulei, Zhang Mengdan, Lin Xu, Yang Jinrui, Zheng Xiawu, Li Ke, Sun Xing, Wu Yunsheng, and Ji Rongrong. Mme: A comprehensive evaluation benchmark for multimodal large language models. 2024

    work page 2024

  3. [3]

    Llavolta: Efficient multi-modal models via stage-wise visual context compression

    Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yang Wang, Daniel Khashabi, and Alan Yuille. Llavolta: Efficient multi-modal models via stage-wise visual context compression. 2024

    work page 2024

  4. [4]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. InECCV, pages 19–35, 2024

    work page 2024

  5. [5]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. 3(5), 2023

    work page 2023

  6. [6]

    Funnel-transformer: Filtering out sequential redundancy for efficient language processing.NeurIPS, 33:4271–4282, 2020

    Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. Funnel-transformer: Filtering out sequential redundancy for efficient language processing.NeurIPS, 33:4271–4282, 2020

    work page 2020

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. 2020

    work page 2020

  8. [8]

    Eva: Exploring the limits of masked visual representation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. InCVPR, pages 19358–19369, 2023

    work page 2023

Show all 41 references
  1. [9]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. pages 10323–10337, 2023

    2023

  2. [11]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InCVPR, pages 6904–6913, 2017

    2017

  3. [12]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images

    Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. InECCV, pages 390–406, 2024

    2024

  4. [13]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InCVPR, pages 3608–3617, 2018

    2018

  5. [14]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Liu Haotian, Li Chunyuan, Li Yuheng, Li Bo, Zhang Yuanhan, Shen Sheng, and Lee Yong Jae. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  6. [15]

    Gaussian error linear units (gelus).arXiv preprint arXiv:1606.08415, 2016

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus).arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  7. [16]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InCVPR, pages 6700–6709, 2019. 10

    2019

  8. [17]

    Phi-2: The surprising power of small language models

    Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. 1(3):3, 2023

    2023

  9. [18]

    Tokenpacker: Efficient visual projector for multimodal llm

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm. 2024

    2024

  10. [19]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. 2024

    2024

  11. [20]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. 2023

    2023

  12. [21]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR, pages 26296–26306, 2024

    2024

  13. [22]

    Visual instruction tuning.NeurIPS, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.NeurIPS, 36:34892–34916, 2023

    2023

  14. [23]

    Mmbench: Is your multi-modal model an all-around player? In ECCV, pages 216–233, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, pages 216–233, 2024

    2024

  15. [24]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.NeurIPS, 35:2507–2521, 2022

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.NeurIPS, 35:2507–2521, 2022

    2022

  16. [25]

    Are sixteen heads really better than one? 32, 2019

    Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? 32, 2019

    2019

  17. [26]

    Efficient transformers with dynamic token pooling

    Piotr Nawrot, Jan Chorowski, Adrian Ła´ncucki, and Edoardo M Ponti. Efficient transformers with dynamic token pooling. 2022

    2022

  18. [27]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. pages 8748–8763, 2021

    2021

  19. [28]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 35...

    2020

  20. [29]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. 2024

    2024

  21. [30]

    Inter-instance similarity modeling for contrastive learning.arXiv preprint arXiv:2306.12243, 2023

    Chengchao Shen, Dawei Liu, Hao Tang, Zhe Qu, and Jianxin Wang. Inter-instance similarity modeling for contrastive learning.arXiv preprint arXiv:2306.12243, 2023

    2023 arXiv

  22. [31]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InCVPR, pages 8317–8326, 2019

    2019

  23. [32]

    Data-efficient multi-scale fusion vision transformer.Pattern Recognition, 161:111305, 2025

    Hao Tang, Dawei Liu, and Chengchao Shen. Data-efficient multi-scale fusion vision transformer.Pattern Recognition, 161:111305, 2025

    2025

  24. [33]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. 2023

    2023

  25. [34]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. 2023

    2023

  26. [35]

    Attention is all you need.NeurIPS, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.NeurIPS, 30, 2017

    2017

  27. [36]

    Calflops: A flops and params calculate tool for neural networks in pytorch framework, 2023

    X Ye. Calflops: A flops and params calculate tool for neural networks in pytorch framework, 2023

    2023

  28. [37]

    Texthawk: Exploring efficient fine-grained perception of multimodal large language models

    Ya-Qi Yu, Minghui Liao, Jihao Wu, Yongxin Liao, Xiaoyu Zheng, and Wei Zeng. Texthawk: Exploring efficient fine-grained perception of multimodal large language models. 2024. 11

    2024

  29. [38]

    Vcc: scaling transformers to 128k tokens or more by prioritizing important tokens.NeurIPS, 36:20260–20286, 2023

    Zhanpeng Zeng, Cole Hawkins, Mingyi Hong, Aston Zhang, Nikolaos Pappas, Vikas Singh, and Shuai Zheng. Vcc: scaling transformers to 128k tokens or more by prioritizing important tokens.NeurIPS, 36:20260–20286, 2023

    2023

  30. [39]

    Tinychart: Efficient chart understanding with visual token merging and program-of-thoughts learning

    Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, and Fei Huang. Tinychart: Efficient chart understanding with visual token merging and program-of-thoughts learning. 2024

    2024

  31. [40]

    Llava-mini: Efficient image and video large multimodal models with one vision token

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. 2025

    2025

  32. [41]

    Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts

    Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts. 2024

    2024

  33. [42]

    Treat visual tokens as text? but your mllm only needs fewer efforts to see

    Zeliang Zhang, Phu Pham, Wentian Zhao, Kun Wan, Yu-Jhe Li, Jianing Zhou, Daniel Miranda, Ajinkya Kale, and Chenliang Xu. Treat visual tokens as text? but your mllm only needs fewer efforts to see. 2024. 12

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.