Pith. sign in

REVIEW 5 major objections 5 minor 36 references

Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adaptive per-layer allocation of sparsity and KV-cache bit widths, found by a Tree-structured Parzen Estimator using text perplexity as the objective, lets LLaVA-1.5 models shed half their weights and compress the KV cache while keeping…

desk verdict A credible but overclaimed compression paper: TPE-based layer-wise sparsity and KV bandwidth allocation works on most benchmarks, but the search proxy is unvalidated and one table row contradicts the 'outperforms all' claim. read the letter →

arxiv 2507.20613 v1 pith:K6FL53FW submitted 2025-07-28 cs.AI cs.LG

Te Zhang , Yuheng Li , Junxiang Wang , Lujun Li This is my paper · ORCID
classification cs.AIcs.LG
keywords largemultimodalmodelsmodelcompressionpruningKVcachequantizationlayer-wisesparsityallocationtree-structuredparzenestimatorpost-trainingLLaVA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that uniform compression is the wrong default for large multimodal models. It shows that a Bayesian search over layer-wise sparsity ratios and KV-cache quantization bit widths, guided only by text perplexity, produces compressed LLaVA-1.5 models (7B and 13B) that outperform uniform SparseGPT and Wanda at 50% overall sparsity on VQAv2, ScienceQA, TextVQA, and POPE. If true, this means substantial memory savings for edge deployment without any fine-tuning or weight adjustment, and it reveals where compression hurts least: keep early and late layers dense, and keep early KV caches in higher precision. The caveat to keep in mind is that the search objective is Wikitext-2 perplexity, and the paper assumes this tracks multimodal task accuracy.

What carries the argument

The load-bearing mechanism is the trial loop in Algorithm 1: a Tree-structured Parzen Estimator proposes a full layer-wise sparsity profile (and, for the KV cache, a bit-width profile), the model is pruned or quantized with that profile, and Wikitext-2 perplexity is measured; the estimator then models which profiles produced low perplexity and proposes the next trial. This loop is cheap because the pruning metric in Equation 4 is post-training and needs no fine-tuning: each weight is scored by a log-scaled combination of its row and column L2-normalized magnitudes multiplied by the input activation norm. The search runs about 30 to 50 trials, with each trial taking one to two minutes on an A100 for LLaVA-1.5 7B.

What would settle it

On LLaVA-1.5 7B, take a set of random sparsity profiles at 50% overall sparsity, measure each profile's Wikitext-2 perplexity and VQAv2 accuracy, and compute the rank correlation between the two. If the correlation is not strongly negative, the search objective is not selecting for the downstream task. More directly, run the TPE search twice, once optimizing Wikitext-2 perplexity and once optimizing a held-out VQAv2 split, and compare the two profiles' test-set VQAv2 scores; if the perplexity-optimized profile does not match the VQA-optimized one, the paper's claim that the searched allocation is the best one fails.

Watch

Extended reading notes

Core claim

The central claim is that adaptively allocating compression resources across layers beats applying the same sparsity ratio or the same quantization bit width to every layer. With an overall 50% unstructured sparsity budget, the TPE-searched layer-wise sparsity profile combined with a new pruning metric beats Magnitude, SparseGPT, and Wanda on all four benchmarks for LLaVA-1.5 Vicuna-7B, and on all but POPE for Vicuna-13B. Under a constraint that half the layers use 8-bit and half use 6-bit KV cache quantization, the searched placement of those bit widths beats the reversed ('Opposite') placement on all four benchmarks, often by several points on ScienceQA and TextVQA. The paper also reports the allocation patterns it finds: sparsity follows a U-shape across depth (denser at early and late layers), while KV cache precision favors earlier layers.

Load-bearing premise

The search chooses profiles by minimizing Wikitext-2 perplexity, and the paper assumes that ordering carries over to visual question answering and hallucination benchmarks; if that correlation is weak, the selected profiles are arbitrary and the claimed edge over uniform allocation is not established.

Editorial extensions

If this is right

  • At 50% overall sparsity, the 7B model loses about 2.4 points on VQAv2 (78.5 to 76.1) and 1.2 points on ScienceQA (66.8 to 65.6), far less than the 13-plus-point drops from magnitude pruning.
  • Adding searched 8/6-bit KV cache quantization on top of the pruned 7B model costs less than one point on most benchmarks, whereas a naive 6-bit-everywhere allocation collapses ScienceQA.
  • The discovered profiles give a reusable heuristic for other transformer-based multimodal models: spend sparsity budget in the middle layers and KV precision in the early layers.
  • The same search transfers to single-modality LLaMA-1 and LLaMA-2 models, so the benefit is not specific to vision-language training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The search objective is text-only Wikitext-2 perplexity; if perplexity does not rank-order profiles by their visual-question-answering accuracy, the searched profiles are arbitrary. This could be tested by re-running the search with a small VQAv2 validation split as the objective and comparing the resulting profiles and accuracies.
  • The same TPE machinery could tune other per-layer choices, such as weight quantization precision or attention-head counts, since nothing in Algorithm 1 is specific to sparsity or KV bits; the conclusion already gestures at mixed precision.
  • The 'Opposite' allocation row in Table IV shows the performance gradient across layer placements is steep, so an even finer-grained search over mixes beyond half-8-bit/half-6-bit might recover some of the remaining loss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes an adaptive compression framework for large multimodal models (LMMs) that jointly searches layer-wise unstructured sparsity ratios and KV cache quantization bit-widths using the Tree-structured Parzen Estimator (TPE), together with a new pruning metric defined in Eq. (4). The method is evaluated on LLaVA-1.5 7B and 13B on VQAv2, SQA, TextVQA, POPE, and Wikitext-2 perplexity, and the authors claim that it outperforms uniform SparseGPT and Wanda at 50% overall sparsity, and that the adaptive KV cache bandwidth allocation outperforms the reverse allocation.

Significance. If the results hold, the framework would offer a practical recipe for deploying compressed LMMs on resource-constrained devices, and the combination of pruning and KV cache quantization with a TPE-based search is a timely and relevant direction. The paper also reports a new pruning metric that is computationally cheap and potentially useful. However, the current evidence is insufficient to support the central claims: the search objective (PPL on Wikitext-2) is not shown to transfer to multimodal benchmarks, the pruning metric is not ablated separately from the allocation, and the claims of universal superiority are directly contradicted by the paper's own Table II.

major comments (5)
  1. [Section IV.B, Table II] The text states that 'our method outperforms others on all benchmarks,' but in the Vicuna-13B part of Table II, the POPE column shows Ours = 83.94 while SparseGPT = 86.90 and Wanda = 85.01, i.e., our method is worse than both baselines on this benchmark. This is a direct factual contradiction of the claim. The authors must either correct the claim to acknowledge exceptions or explain why this particular benchmark is less indicative of the method's value.
  2. [Section III.C, Algorithm 1, and Table II] The search objective in Algorithm 1 is Wikitext-2 perplexity, yet the headline results are on multimodal VQA benchmarks. The paper asserts that 'PPL is chosen as it reflects model performance' but never provides any evidence of a correlation between text-only PPL and multimodal task accuracy. In fact, Table II provides a counterexample: for Vicuna-13B, Ours has the best PPL (6.66) but the worst POPE among the three compression methods (83.94 vs. SparseGPT 86.90 and Wanda 85.01). Without a correlation analysis or a multimodal validation split in the search, the selected sparsity and bandwidth profiles may be overfit to PPL, and the observed VQA gains cannot be attributed to the adaptive allocation rather than to the new pruning metric or other confounds. Please add such an analysis or adapt the search objective to a multimodal metric.
  3. [Section III.E and Table II] The pruning metric in Eq. (4) is never ablated separately from the sparsity allocation search. As a result, it is impossible to determine whether the improvements over SparseGPT/Wanda come from the new metric, from the adaptive layer-wise allocation, or from both. The central claim that the framework's allocation is beneficial requires an ablation with at least three settings: (i) uniform sparsity + Wanda metric, (ii) uniform sparsity + proposed metric, (iii) adaptive allocation + proposed metric. Only then can the reader assess the contribution of each component.
  4. [Algorithm 1, line 4] The pseudocode in Algorithm 1 rejects profiles with 'mean(Pt) > σoverall' (continue to next trial), while the text in Section III.C states that 'we ensure the average sparsity meets or exceeds the specified overall sparsity before proceeding.' These two statements are inconsistent. If the code is interpreted literally, the accepted profiles satisfy mean(Pt) ≤ σoverall, i.e., the actual sparsity is below the stated 50% target, which would give the method an unfair advantage over baselines that are pruned to exactly 50%. If the text is correct, then the pseudocode condition should be 'mean(Pt) < σoverall' instead. Please clarify and verify that the effective overall sparsity is exactly (or at least) the same for all methods in the comparison.
  5. [Section III.C] The search space for each layer's sparsity is limited to ±5% around the overall sparsity in 2.5% increments (e.g., {45%, 47.5%, 50%, 52.5%, 55%} for 50% overall). This is a narrow range, and the paper does not compare the TPE search against a random search over the same space or against a uniform allocation with the same pruning metric. Without such a comparison, the claimed advantage of 'adaptive' allocation over uniform allocation is not isolated; the observed differences could be within the noise of the restricted search space.
minor comments (5)
  1. [Abstract and Section IV.B] The abstract contains grammatical errors, e.g., 'our framework automatic allocation' should be 'our framework's automatic allocation,' and the phrases 'sets a new standard in LMM optimization' and 'superiority over state-of-the-art techniques' are promotional and not supported by the data, especially given the POPE exception in Table II.
  2. [Section III.C] The statement 'PPL is chosen as it reflects model performance' conflates text-only perplexity with overall model performance, including multimodal capabilities. It would be more precise to say that PPL is a convenient proxy and to provide evidence for its correlation with the downstream benchmarks used in the paper.
  3. [Table IV] The 'Opposite' row is described as 'the reverse allocation,' but it is not specified whether this means the exact reversed profile of 'Ours' or simply swapping the layer groups that receive 6-bit and 8-bit. Please define this clearly for reproducibility.
  4. [Eq. (4)] The activation term in the proposed metric is sqrt(||X_j||_2), while Wanda uses ||X_j||_2 without the square root. No justification is given for this square root, and it is not discussed in the ablation. A brief explanation or a sensitivity analysis would strengthen the metric's description.
  5. [Table I] Table I compares compressed LLaVA models to uncompressed models such as BLIP-2 and InstructBLIP. Since the compressed models start from a strong pretrained baseline, the comparison is not apples-to-apples. The text should clarify that this table is only a reference to show the compressed models remain competitive with existing uncompressed LMMs, not a head-to-head comparison of compression methods.

Circularity Check

1 steps flagged · score 2.0 of 10

Wikitext PPL column is the search objective itself; the multimodal benchmark claim is independent and not circular.

  1. fitted input called prediction [Algorithm 1 line 8; Section III.C; Tables II/III/V ('Wikitext PPL' rows)]
    "PPLt ← Evaluate pruned model on D ... PPL is chosen as it reflects model performance and allows quick trial completion."

    Algorithm 1 selects the layer-wise sparsity profile Pt that minimizes Wikitext-2 PPL on D (line 8) and returns the best-PPL profile. The same Wikitext-2 PPL of that selected profile is then reported as the 'Ours' row in Tables II, III and V. Thus those PPL numbers are the optimization objective itself: the 'improvement' over uniform SparseGPT/Wanda in this column is by construction the TPE search result, not an independent prediction. The four VQA benchmarks are not part of this objective, so the central multimodal claim is not reduced to its input; the circularity is confined to the PPL tables.

full rationale

The paper's central claim is that its pruning metric plus TPE-selected layer-wise sparsity allocation outperforms uniform SparseGPT and Wanda at 50% overall sparsity on VQAv2, SQA, TextVQA, and POPE. Those benchmarks are not used in Algorithm 1; the search objective is Wikitext-2 perplexity. Therefore the multimodal comparison is not fitted to the evaluation metric, and the headline result has independent content. The only identified issue is that the Wikitext PPL columns in Tables II, III, and V report the exact quantity that the TPE search was optimizing, so those numbers are selected rather than predicted. The self-citations [13]-[15] are cited as motivation for layer-wise sparsity and are not load-bearing in the derivation. The 13B POPE row (83.94 vs. SparseGPT 86.90 and Wanda 85.01) contradicts the Section IV.B statement that the method 'outperforms others on all benchmarks', but that is a correctness/consistency problem, not circularity. Overall the derivation is largely self-contained against external benchmarks, with only a minor fitted-objective-reported-as-result issue in the PPL tables.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the choice of Wikitext-2 perplexity as the search objective and on the ad hoc pruning metric. Neither is independently validated: the PPL-VQA correlation is never measured, and the metric is never ablated alone against Wanda's. The search itself is constrained by hand-set ranges and trial counts. No new entities are introduced.

free parameters (4)
  • Sparsity search range = ±5% around overall sparsity, 2.5% step
    Chosen by hand in Section IV.A; the paper does not justify why this range or granularity is sufficient to capture optimal allocation.
  • Number of search trials = 50
    Set in Section IV.A ('best sparsity profile found over 50 trials'); no convergence analysis or budget sensitivity.
  • Quantization bit constraint = Half layers 8-bit, half 6-bit
    Section IV.A constrains the bandwidth search to equal numbers of 6-bit and 8-bit layers, which is an arbitrary constraint to prevent the search from favoring all 8-bit.
  • Metric design constants = 1 and 2 inside log; sqrt on activation norm
    Equation 4 is an ad hoc combination of row/column L2 norms, log scaling, and sqrt of activation norm, without derivation or ablation to justify each element.
assumptions (4)
  • domain assumption Wikitext-2 perplexity is a valid proxy for downstream multimodal task performance
    Algorithm 1 and Section III.C use PPL as the objective to select sparsity profiles; the paper never validates the PPL-VQA correlation.
  • domain assumption TPE with 50 trials converges to a near-optimal allocation
    Section III.D assumes TPE efficiently narrows the search space; no evidence of convergence or multiple restarts is provided.
  • ad hoc to paper The pruning metric in Equation 4 ranks weight importance better than Wanda's metric
    The metric is introduced in Section III.E with motivational text, but no ablation isolates the metric from the allocation search, so its superiority is assumed rather than shown.
  • domain assumption Post-training unstructured pruning at 50% sparsity preserves enough accuracy for edge deployment
    The paper's framing assumes 50% sparsity models remain useful; the actual drops (e.g., 2.4 points on VQAv2, 4.9 on TextVQA for 7B) are presented without a deployment threshold.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression." pith.science (2026). https://pith.science/paper/K6FL53FW

@misc{pith2026250720613,
  author       = {Pith},
  title        = {Pith review of: Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K6FL53FW}},
  note         = {Machine review of arXiv:2507.20613}
}
read the original abstract

Large multimodal models (LMMs) have advanced significantly by integrating visual encoders with extensive language models, enabling robust reasoning capabilities. However, compressing LMMs for deployment on edge devices remains a critical challenge. In this work, we propose an adaptive search algorithm that optimizes sparsity and KV cache compression to enhance LMM efficiency. Utilizing the Tree-structured Parzen Estimator, our method dynamically adjusts pruning ratios and KV cache quantization bandwidth across different LMM layers, using model performance as the optimization objective. This approach uniquely combines pruning with key-value cache quantization and incorporates a fast pruning technique that eliminates the need for additional fine-tuning or weight adjustments, achieving efficient compression without compromising accuracy. Comprehensive evaluations on benchmark datasets, including LLaVA-1.5 7B and 13B, demonstrate our method superiority over state-of-the-art techniques such as SparseGPT and Wanda across various compression levels. Notably, our framework automatic allocation of KV cache compression resources sets a new standard in LMM optimization, delivering memory efficiency without sacrificing much performance.

Figures

Figures reproduced from arXiv: 2507.20613 by the authors.

Figure 1
Figure 1. A brief schematic overview of our framework. We utilize the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 22 canonical work pages

  1. [1]

    Mistral 7b,

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825, 2023

    arXiv

  2. [2]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, and T. ... & Scialom, “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288 , 2023

    arXiv 2023

  3. [3]

    Visual instruction tuning,

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, “Visual instruction tuning,” arXiv:2304.08485, 2023

    arXiv 2023

  4. [4]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed El- hoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592 , 2023

    arXiv 2023

  5. [5]

    Detkds: Knowledge distillation search for object detectors,

    Lujun Li, Yufan Bao, Peijie Dong, Chuanguang Yang, Anggeng Li, Wenhan Luo, Qifeng Liu, Wei Xue, and Yike Guo, “Detkds: Knowledge distillation search for object detectors,” in ICML, 2024

    work page 2024

  6. [6]

    Kd-zero: Evolving knowledge distiller for any teacher-student pairs,

    Lujun Li, Peijie Dong, Anggeng Li, Zimian Wei, and Ya Yang, “Kd-zero: Evolving knowledge distiller for any teacher-student pairs,” NeuIPS, 2024

    work page 2024

  7. [7]

    Diswot: Student architecture search for distillation without training,

    Peijie Dong, Lujun Li, and Zimian Wei, “Diswot: Student architecture search for distillation without training,” in CVPR, 2023

    work page 2023

  8. [8]

    Automated knowl- edge distillation via monte carlo tree search,

    Lujun Li, Peijie Dong, Zimian Wei, and Ya Yang, “Automated knowl- edge distillation via monte carlo tree search,” in ICCV, 2023

    work page 2023

Show all 36 references
  1. [9]

    Norm: Knowledge distillation via n-to-one representation matching,

    Liu Xiaolong, Li Lujun, Li Chao, and Anbang Yao, “Norm: Knowledge distillation via n-to-one representation matching,” in ICLR, 2023

    2023

  2. [10]

    Self-regulated feature learning via teacher-free feature distillation,

    Lujun Li, “Self-regulated feature learning via teacher-free feature distillation,” in ECCV, 2022

    2022

  3. [11]

    Attnzero: efficient attention discovery for vision transformers,

    Lujun Li, Zimian Wei, Peijie Dong, Wenhan Luo, Wei Xue, Qifeng Liu, and Yike Guo, “Attnzero: efficient attention discovery for vision transformers,” in ECCV, 2024

    2024

  4. [12]

    Auto-gas: Automated proxy discovery for training-free generative architecture search,

    Lujun Li, Haosen Sun, Shiwen Li, Peijie Dong, Wenhan Luo, Wei Xue, Qifeng Liu, and Yike Guo, “Auto-gas: Automated proxy discovery for training-free generative architecture search,” ECCV , 2024

    2024

  5. [13]

    Discovering sparsity allocation for layer-wise pruning of large language models,

    Lujun Li, Peijie, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo, “Discovering sparsity allocation for layer-wise pruning of large language models,” in NeuIPS, 2024

    2024

  6. [14]

    Als: Adaptive layer sparsity for large language models via activation correlation assessment,

    Wei Li, Lujun Li, Mark Lee, and Shengjie Sun, “Als: Adaptive layer sparsity for large language models via activation correlation assessment,” in NeuIPS, 2024

    2024

  7. [15]

    Pruner-zero: Evolving symbolic pruning metric from scratch for large language models,

    Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu, “Pruner-zero: Evolving symbolic pruning metric from scratch for large language models,” in ICML, 2024

    2024

  8. [16]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” 2023

    2023

  9. [17]

    A survey on multimodal large language models,

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen, “A survey on multimodal large language models,” arXiv preprint arXiv:2306.13549, 2023

    2023 arXiv

  10. [18]

    Mm-llms: Recent advances in multimodal large language models,

    Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu, “Mm-llms: Recent advances in multimodal large language models,” arXiv preprint arXiv:2401.13601 , 2024

    2024 arXiv

  11. [19]

    Sparsegpt: Massive language models can be accurately pruned in one-shot,

    Elias Frantar and Dan Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” ArXiv, vol. abs/2301.00774, 2023

    2023 arXiv

  12. [20]

    A simple and effective pruning approach for large language models,

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter, “A simple and effective pruning approach for large language models,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  13. [21]

    Llm-pruner: On the structural pruning of large language models,

    Xinyin Ma, Gongfan Fang, and Xinchao Wang, “Llm-pruner: On the structural pruning of large language models,” ArXiv, vol. abs/2305.11627, 2023

    2023 arXiv

  14. [22]

    Emq: Evolving training-free proxies for automated mixed precision quantization,

    Peijie Dong, Lujun Li, Zimian Wei, Xin Niu, Zhiliang Tian, and Hengyue Pan, “Emq: Evolving training-free proxies for automated mixed precision quantization,” arXiv preprint arXiv:2307.10554 , 2023

    2023 arXiv

  15. [23]

    Stbllm: Breaking the 1-bit barrier with structured binary llms,

    Peijie Dong, Lujun Li, Yuedong Zhong, Dayou Du, Ruibo Fan, Yuhan Chen, Zhenheng Tang, Qiang Wang, Wei Xue, Yike Guo, et al., “Stbllm: Breaking the 1-bit barrier with structured binary llms,” in ICLR, 2025

    2025

  16. [24]

    Quantization and training of neural networks for efficient integer- arithmetic-only inference,

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko, “Quantization and training of neural networks for efficient integer- arithmetic-only inference,” in Proceedings of the IEEE conference on computer vision...

    2018

  17. [25]

    Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance,

    Shuhei Watanabe, “Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance,” arXiv preprint arXiv:2304.11127 , 2023

    2023 arXiv

  18. [26]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597 , 2023

    2023 arXiv

  19. [27]

    Instructblip: Towards general-purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,” 2023

    2023

  20. [28]

    Shikra: Unleashing multimodal llm’s referential dialogue magic,

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao, “Shikra: Unleashing multimodal llm’s referential dialogue magic,” arXiv preprint arXiv:2306.15195 , 2023

    2023 arXiv

  21. [29]

    Introducing idefics: An open reproduction of state-of-the-art visual language model,

    IDEFICS, “Introducing idefics: An open reproduction of state-of-the-art visual language model,” https://huggingface.co/blog/idefics, 2023

    2023

  22. [30]

    Qwen-vl: A frontier large vision-language model with versatile abilities,

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, “Qwen-vl: A frontier large vision-language model with versatile abilities,” arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  23. [31]

    Improved baselines with visual instruction tuning,

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, “Improved baselines with visual instruction tuning,” 2023

    2023

  24. [32]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 6904– 6913

    2017

  25. [33]

    Learn to explain: Multimodal reasoning via thought chains for science question answering,

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” Advances in Neural Information Processing Systems , 2022

    2022

  26. [34]

    Towards vqa models that can read,

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach, “Towards vqa models that can read,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 8317–8326

    2019

  27. [35]

    Evaluating object hallucination in large vision-language models,

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen, “Evaluating object hallucination in large vision-language models,” arXiv preprint arXiv:2305.10355 , 2023

    2023 arXiv

  28. [36]

    Pointer sentinel mixture models,

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher, “Pointer sentinel mixture models,” 2016

    2016

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.