Pith. sign in

REVIEW 6 major objections 5 minor 36 references

Efficient Multi-modal Large Language Models via Visual Token Grouping

T0 review · 6 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that grouping similar image patches into semantic tokens preserves 98.1% of a vision-language model's accuracy while cutting inference time by over 27%.

desk verdict A solid, incremental token-reduction method for MLLMs; the headline PRT is slightly overstated and the 'semantic grouping' claim outruns the evidence, but the core result and ablations are worth a serious look. read the letter →

arxiv 2411.17773 v2 pith:ZEOLA2IP submitted 2024-11-26 cs.CV

Minbin Huang , Runhui Huang , Han Shi , Yimeng Chen , Chuanyang Zheng , Xiangguo Sun , Xin Jiang , Zhenguo Li
show 1 more author
Hong Cheng
This is my paper · ORCID
classification cs.CV
keywords multi-modallargelanguagemodelsvisualtokencompressiongroupinginferenceaccelerationvisiontransformersemantictokensGumbel-Softmaxassignmentinstruction-aware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VisToG is a visual-token compression method for multimodal large language models that tries to cut inference cost without retraining the vision encoder and without segmentation masks. The paper's claim is that adding a small set of learnable semantic tokens inside the frozen CLIP vision encoder lets the model group similar image patches, merge each group into a single token, and still answer questions nearly as well as the full 576-token model. On six standard benchmarks, the 128-token version retains 98.1% of the baseline's average performance while reducing inference time by more than 27%. The reason this matters is that the language model's attention cost grows with the number of input tokens, so cheaper, smaller visual token sets make high-resolution and video inputs more practical.

What carries the argument

The load-bearing mechanism is the grouping layer placed inside the vision encoder, together with an isolated-attention mask. The grouping layer introduces N learnable semantic tokens, computes a similarity matrix between semantic tokens and image patch tokens, applies a Gumbel-Softmax with a straight-through estimator to make the one-hot assignment differentiable, and forms each output token as a weighted sum of the patches assigned to its group, added to the semantic token. The mask M is set to False whenever an image token attends to a semantic token, so the frozen image representations are unchanged while semantic tokens can aggregate information from all patches. This design transfers the clustering burden to the pretrained encoder's prior knowledge and makes the grouping trainable under instruction supervision in the second training stage.

What would settle it

Run VisToG and the full 576-token baseline on a benchmark of dense small text or fine-grained detail, such as a page with many tiny labels, and compare per-image accuracy: if grouping drops the answer region into a large averaged group and the model fails on questions the baseline answers, the central claim that grouping is near-lossless for instruction-relevant detail is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the redundancy of image tokens can be exploited inside the pretrained vision encoder rather than in the downstream connector. VisToG concatenates N learnable semantic tokens with the image patch tokens before the transformer layers of a frozen CLIP encoder, uses a Gumbel-Softmax assignment to softly allocate each patch to one semantic group, and then replaces each group by a weighted average of its patches, producing 128 (or 64) visual tokens instead of 576. An isolated-attention mask forbids the original image patches from attending to the semantic tokens, so the pretrained representations stay intact while the semantic tokens learn to collect patches into instruction-relevant groups during visual instruction tuning. With this mechanism, the paper reports the 128-token model keeping average performance at 98.1% of the reproduced LLaVA-1.5 baseline and cutting inference time by over 27%, and the 64-token model staying close behind. The largest observed drop is on TextVQA, where fine-grained text reading suffers, which the paper treats as a sign of redundancy limits rather than a refutation.

Load-bearing premise

The load-bearing assumption is that a frozen, pre-trained vision encoder can split an image into meaningful semantic groups using only a few learned semantic tokens, with no segmentation labels, and that replacing every patch in a group by the group average leaves intact whatever detail the question needs.

Editorial extensions

If this is right

  • If the claim holds, visual token budgets for MLLMs can be cut by roughly a factor of 4 (576 to 128) with only a small benchmark-quality loss, making longer inputs such as high-resolution images more affordable.
  • Because the grouping layer is trained during instruction tuning, the token groups become task-dependent rather than fixed, so the same image can be compressed differently for different questions.
  • The isolated-attention result implies that adding auxiliary tokens to a frozen vision transformer does not have to disturb its original representations, a general recipe for injecting learnable structure into pretrained encoders.
  • The 64-token variant's near-parity on most benchmarks suggests the effective information content of many images is far below 576 tokens, and that the bottleneck is coverage of semantic regions rather than sheer token count.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same grouping layer could be applied to video by treating frames as additional patches or by inserting a temporal grouping step, since the paper explicitly leaves video experiments for future work.
  • Editorial inference: if the TextVQA drop reflects a real limitation, a hybrid token allocation that keeps fine-grained tokens for high-detail regions and groups only low-detail background would likely recover much of the loss while keeping most of the speed gain.
  • Editorial inference: the method's reliance on CLIP's patch embeddings suggests it will work best on images whose semantic structure matches CLIP's image-text alignment; domain-shifted inputs such as medical or satellite imagery may need re-trained semantic tokens.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. This paper proposes VisToG, a method to reduce the number of visual tokens fed to a multimodal large language model (MLLM). VisToG inserts learnable semantic tokens into a frozen CLIP ViT encoder, uses an isolated attention mask to keep image patch tokens unaffected, and applies a Gumbel-softmax grouping layer that assigns patch tokens to semantic groups and merges them into a smaller set of tokens before the visual projector. Training is done in two stages: first the visual connector is aligned on image-text pairs without grouping, then the grouping layer, connector, and LLM are fine-tuned on visual instruction data. The paper evaluates VisToG on GQA, ScienceQA, TextVQA, POPE, MME, and MMB, reporting that it maintains 98.1% of the LLaVA-1.5 performance while reducing inference time by over 27%.

Significance. If the reported results hold, VisToG offers a practical way to cut MLLM inference cost with a modest performance drop, and the use of a reproduced LLaVA-1.5 baseline is a strength that makes the main comparisons fair. However, the key mechanistic claim—that the benefit comes from semantic grouping rather than from generic learned token reduction—is not yet supported, because no matched learnable-pooling control is reported. The claimed PRT and inference-time numbers also need clarification. The contribution is incremental but potentially useful; the paper contains falsifiable predictions that the grouping structure matters, which additional experiments can test.

major comments (6)
  1. [Abstract, Table 2, Sec. 4.3] The abstract and conclusion claim that VisToG maintains 98.1% of the original performance, but Table 2, which defines PRT over GQA, TextVQA, POPE, and MME, reports 97.5% for the 128-token model. The 98.1% figure matches an average over all six datasets in Table 1 (including SQA and MMB), so the manuscript is not consistent about which datasets the headline PRT uses; please state the exact set and report the corresponding value.
  2. [Sec. 3.2, Eq. (4)] Equation (4) defines the merged token as VISi = Semi + Wo * (Σ_j \hat A_{i,j} Wv Img_j) / (Σ_j \hat A_{i,j} Wv). The denominator is a sum of projected vectors, not a scalar, so the normalization is ill-defined; with the one-hot assignment from Eq. (3) the expression degenerates. Please correct the denominator to a scalar normalization (e.g., Σ_j \hat A_{i,j}) or provide the intended operation, because this equation is the core of the grouping mechanism.
  3. [Sec. 4.4, Fig. 5] The paper's central claim is that VisToG works by grouping image patches into semantic segments, but no controlled baseline with a learnable pooling mechanism (e.g., a learned attention-pooling or cross-attention layer with the same number of output tokens and the same training budget) is provided. LLaVA-AvgPool is a non-learned adaptive average pooling and is compared only on GQA and POPE in Fig. 5, not on the full benchmark suite at 128 tokens. Without such a control, the results are consistent with any learned token-reduction mechanism, and the specific contribution of semantic grouping is not established.
  4. [Sec. 4.4, Eq. (7)] The claimed 'over 27% inference time reduction' is not supported with sufficient detail: the paper does not report the actual inference time of VisToG, nor the measurement protocol (batch size, number of generated tokens, warmup, input resolution, prefill vs. decode time). Please provide the raw times in a table and describe the protocol so the efficiency claim can be reproduced.
  5. [Sec. 4.3, Table 1] No variance or number of seeds is reported for any result, including the stochastic components (Gumbel noise in grouping and random token sampling for LLaVA-rand). The paper itself notes in Sec. 4.3 that LLaVA-rand has 'high variance,' so reporting mean and standard deviation across multiple runs is necessary to assess the significance of the differences, especially the small gaps between VisToG and the baseline.
  6. [Table 1] The LLaVA-1.5-Q-Former row lists #Tokens=576, while the text in Sec. 4.2 states that the number of queries is set to 64. If Q-Former is used to compress tokens, the token count should be 64; if it is 576, the comparison is not a compression setting. Please correct this inconsistency and ensure the comparison is at the claimed token count.
minor comments (5)
  1. [Sec. 3.1] 'Large Lanugae Model' is a typo for 'Large Language Model'.
  2. [Fig. 3] Fig. 3 visualizes LLaVA-rand, not VisToG; adding a visualization of VisToG's learned group assignments would directly support the semantic grouping claim.
  3. [Table 2] Table 2 reports PRT only for the 128-token VisToG; the corresponding 64-token PRT can be inferred from Table 1 but should be reported explicitly for completeness.
  4. [Sec. 3.2, Eq. (2)-(3)] Eq. (2) is called a Gumbel-Softmax operation, but Eq. (3) uses a one-hot straight-through estimator; please clarify the relationship between the soft assignment used for gradients and the hard assignment used for the merged output.
  5. [Sec. 4.4, Eq. (7)] The notation t_ij/|D_i| in Eq. (7) is unclear; specifying the averaging over samples and datasets would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VisToG's reported performance and efficiency gains are measured against external benchmarks and a reproduced baseline, not constructed from the method's own definitions.

full rationale

The paper's central claim—that VisToG reduces visual tokens from 576 to 128/64 while retaining about 98% of LLaVA-1.5 performance—is supported by evaluations on external benchmarks (GQA, SQA, TextVQA, POPE, MME, MMB) against a reproduced LLaVA-1.5 baseline in Table 1. The Performance Retain Rate in Eq. (6) is a normalized ratio of measured scores to baseline scores; it is a reporting metric, not a fitted parameter, and the 98.1% figure is not defined in terms of VisToG's own outputs. The grouping layer, Eqs. (2)-(4), is trained with the LLM's next-token objective in Stage 2, and no benchmark score is used to select or fit the learned weights or semantic tokens. The 27% inference-time reduction is a measured wall-clock comparison on an L40S, independent of the benchmark scores. The only self-citations ([15], [26]) appear in general related-work context and are not load-bearing for VisToG's derivation. The statement 'Building upon the design in [28]' cites GroupViT, an external previous work; the paper does not invoke a self-authored uniqueness theorem or otherwise forbid alternatives through self-citation. The limitation that video experiments were not conducted is a scope caveat, not a circular step. A concern that the learned groups are not visually verified as semantic would be a correctness or evidence issue, not an instance of the derivation reducing to its inputs; no evidence in the paper shows the benchmark results are forced by construction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The ledger shows the claim rests on an inductive bias about CLIP features and on a reproduced baseline, plus one hyperparameter (N). No constants are fitted to the evaluation benchmarks, and no new entities are introduced.

free parameters (1)
  • N (number of semantic groups / output visual tokens) = 128 or 64
    The number of semantic tokens is set to the target token budget in the experiments; it is a hyperparameter rather than a constant fitted to the test set, but the paper does not provide a principled rule for choosing it.
assumptions (4)
  • domain assumption The pre-trained CLIP vision encoder's patch features are semantically clusterable without segmentation masks.
    Motivates the grouping approach; invoked in Sec. 1 and Sec. 3.2 where similarity between semantic tokens and image patches is computed. The TextVQA drop suggests this is only partially true for fine-grained text.
  • domain assumption LLM inference dominates total MLLM inference cost, so reducing visual token count yields wall-clock speedup.
    Stated in Sec. 1 and Sec. 3.1; the speedup is then measured in Sec. 4.4, so this assumption is validated empirically rather than remaining hidden.
  • domain assumption The reproduced LLaVA-1.5 baseline trained on 624K instruction samples is a valid reference for the 'original performance' the paper claims to retain.
    Sec. 4.1 notes that only 624K of the intended 665K instruction samples were available; the authors reproduce LLaVA-1.5 under the same data, but equivalence to the original 665K checkpoint is assumed.
  • standard math The straight-through Gumbel-Softmax estimator provides usable gradients for learning discrete token assignments.
    Equation 3 uses the straight-through estimator from [25]; this is a standard technique and not a novel risk, but the grouping quality depends on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Multi-modal Large Language Models via Visual Token Grouping." pith.science (2026). https://pith.science/paper/ZEOLA2IP

@misc{pith2026241117773,
  author       = {Pith},
  title        = {Pith review of: Efficient Multi-modal Large Language Models via Visual Token Grouping},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZEOLA2IP}},
  note         = {Machine review of arXiv:2411.17773}
}
read the original abstract

The development of Multi-modal Large Language Models (MLLMs) enhances Large Language Models (LLMs) with the ability to perceive data formats beyond text, significantly advancing a range of downstream applications, such as visual question answering and image captioning. However, the substantial computational costs associated with processing high-resolution images and videos pose a barrier to their broader adoption. To address this challenge, compressing vision tokens in MLLMs has emerged as a promising approach to reduce inference costs. While existing methods conduct token reduction in the feature alignment phase. In this paper, we introduce VisToG, a novel grouping mechanism that leverages the capabilities of pre-trained vision encoders to group similar image segments without the need for segmentation masks. Specifically, we concatenate semantic tokens to represent image semantic segments after the linear projection layer before feeding into the vision encoder. Besides, with the isolated attention we adopt, VisToG can identify and eliminate redundant visual tokens utilizing the prior knowledge in the pre-trained vision encoder, which effectively reduces computational demands. Extensive experiments demonstrate the effectiveness of VisToG, maintaining 98.1% of the original performance while achieving a reduction of over 27\% inference time.

Figures

Figures reproduced from arXiv: 2411.17773 by the authors.

Figure 1
Figure 1. Overview of of our proposed VisToG. Semantic tokens are concatenated with the image patch tokens after linear projection and fed into the pre-trained vision encoder. Before the visual projector to LLM, a grouping layer is applied to group similar image segment tokens into semantically abstraction tokens of image. Besides, isolated attention is applied to ensure a better abstraction. conduct visual token reduction on… view at source ↗
Figure 2
Figure 2. (a) Structure of the grouping layer. (b) Comparison of Inference time and Average Performance between different models. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the image tokens selected of the LLaVA-rand. The instruction is ”What is the main focus of the image?”. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance Comparison between standard attention and isolated attention. The numbers are the relative performance compared [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: (a) Ablation on the number of image tokens on the POPE dataset. (b) Ablation on the number of image tokens on GQA dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 4 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 1, 2

    arXiv 2023

  3. [3]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 2

    2023

  4. [4]

    To- ken merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. To- ken merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 3

    arXiv 2022

  5. [5]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 13817–13827, 2024. 1, 3

    2024

  6. [6]

    Auroracap: Efficient, performant video detailed captioning and a new benchmark

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng-Neng Hwang, Sain- ing Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051, 2024. 3

    arXiv

  7. [7]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Jun- yang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models. arXiv preprint arXiv:2403.06764, 2024. 1

    arXiv 2024

  8. [8]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6,

    2023

Show all 36 references
  1. [9]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems 36...

    2023

  2. [10]

    MME: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306.13394, 2023. 5

    2023 arXiv

  3. [11]

    Model tells you what to discard: Adaptive kv cache compression for llms

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023. 3

    2023 arXiv

  4. [12]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 5

    2019

  5. [13]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2, 3

    2023

  6. [14]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 5

    2023 arXiv

  7. [15]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding. arXiv preprint arXiv:2405.087...

    2024 arXiv

  8. [16]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024. 3

    2024 arXiv

  9. [17]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 3, 5, 6

    2024

  10. [18]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 5

    2023 arXiv

  11. [19]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  12. [20]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  13. [21]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 5

    2019

  14. [22]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 3

    2024 arXiv

  15. [23]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of 9 the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 3

    2024

  16. [24]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1

    2023 arXiv

  17. [25]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 4

    2017

  18. [26]

    Boosting visual- language models by exploiting hard samples

    Haonan Wang, Minbin Huang, Runhui Huang, Lanqing Hong, Hang Xu, Tianyang Hu, Xiaodan Liang, Zhenguo Li, Hong Cheng, and Kenji Kawaguchi. Boosting visual- language models by exploiting hard samples. arXiv preprint arXiv:2305.05208, 2023. 3

    2023 arXiv

  19. [27]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. 3

    2023 arXiv

  20. [28]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 18134–18144, 2022. 4

    2022

  21. [29]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1):1, 2023. 2

    2023 arXiv

  22. [30]

    Deco: Decoupling token compres- sion from semantic abstraction in multimodal large language models

    Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. Deco: Decoupling token compres- sion from semantic abstraction in multimodal large language models. arXiv preprint arXiv:2405.20985, 2024. 1, 3, 7

    2024 arXiv

  23. [31]

    V oco-llama: Towards vision compression with large language models

    Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, and Yansong Tang. V oco-llama: Towards vision compression with large language models. arXiv preprint arXiv:2406.12275, 2024. 3

    2024 arXiv

  24. [32]

    Tinygpt-v: Efficient multimodal large language model via small back- bones

    Zhengqing Yuan, Zhaoxu Li, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small back- bones. arXiv preprint arXiv:2312.16862, 2023. 3

    2023 arXiv

  25. [33]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 3

    2023

  26. [34]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output. arXiv preprint arXiv:2407.03320, 2024. 2, 3

    2024 arXiv

  27. [35]

    Tinyllava: A frame- work of small-scale large multimodal models.arXiv preprint arXiv:2402.14289, 2024

    Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A frame- work of small-scale large multimodal models.arXiv preprint arXiv:2402.14289, 2024. 3

    2024 arXiv

  28. [36]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 3 10

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.