Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that compressing visual tokens inside the vision encoder, using a pixel-shuffle merge plus a parameter-free residual shortcut, outperforms post-encoder compression by a wide margin while cutting training time over 20%.

desk verdict The residual shortcut in LaCo is literally 2D average pooling, and the paper never benchmarks against average pooling, so the headline >30% margin over 'all existing methods' is not yet secured; the paper is still a useful systematic study of inner-encoder compression. read the letter →

arxiv 2507.02279 v1 pith:PTDDNMDW submitted 2025-07-03 cs.CV

Juntao Liu , Liqiang Niu , Wenchao Chen , Jie Zhou , Fandong Meng This is my paper
classification cs.CV
keywords multimodallargelanguagemodelsvisualtokencompressionlayer-wisepixel-shuffleresidualconnectionpatchmergelayertrainingefficiencyinferencethroughput
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that visual-token compression for multimodal large language models should happen inside the vision encoder, not after it, and that this becomes viable when the compression layer is paired with a parameter-free residual shortcut. It introduces LaCo, a Patch Merge Layer inserted at an intermediate encoder layer that merges adjacent tokens with pixel-shuffle and a two-layer MLP, then adds a residual path that rearranges channels and averages them so the model learns only the lost detail. On single-image benchmarks, the paper reports LaCo averaging around 53 versus roughly 36 for Pixel-Shuffle, LDPv2, and TokenPacker relocated to the same layer, while also improving training efficiency beyond 20% and inference throughput over 15% relative to external compression. If the claim holds, efficiency gains in multimodal LLMs do not require a separate compression stage after the encoder; the encoder itself can operate on a reduced token set from an intermediate layer onward.

What carries the argument

The carrying mechanism is the Patch Merge Layer (PML) with a residual shortcut. For token embeddings $E_v$ and compression ratio $r$, the main path is $\hat{E}_v = \mathrm{MLP}(\mathrm{PS}(E_v, r))$, where $\mathrm{PS}$ is pixel-shuffle: it reshapes the token grid from $H \times W \times C$ to $H/r \times W/r \times r^2 C$, merging adjacent tokens into channels, and the MLP maps the $r^2 C$ dimension back to $C$. The second path is the residual connection $\mathrm{RC}(E_v, r) = \mathrm{CA}(\mathrm{PS}(E_v, r))$, a channel-averaging shortcut that introduces no parameters; the layer output is the sum of the two paths. This residual path is what lets the model carry visual information forward during compression while the MLP learns only the residual detail lost in the merge.

What would settle it

Train LaCo at the 1/4 layer with the residual shortcut removed, leaving a plain pixel-shuffle plus MLP patch merge, and re-run the three relocated baselines with per-method hyperparameter sweeps and warmup; if the plain patch merge and tuned baselines match LaCo's average, the residual connection is not the cause of the reported gap.

Watch

Extended reading notes

Core claim

The central claim is that visual-token compression belongs inside the vision encoder rather than after it, provided the compression layer preserves information through a residual shortcut. LaCo inserts a Patch Merge Layer after the first quarter of the encoder, merging adjacent tokens with pixel-shuffle and a two-layer MLP, then adds a non-parametric shortcut that applies the same space-to-channel rearrangement followed by channel averaging, so the trainable MLP only models the residual information lost in merging. With this design, the paper reports single-image benchmark averages of about 53 for LaCo versus about 36 for Pixel-Shuffle, LDPv2, and TokenPacker placed at the same quarter-layer, improvements of roughly 10 percent on multi-image benchmarks and 23 to 27 percent on video benchmarks, and, compared with applying LaCo after the full encoder, over 20 percent faster training and over 15 percent higher inference throughput. The experiments are run with a 0.5B language model across the AIMv2, SigLIP, and InternViT vision encoders, following the three-stage training recipe of a recent open multimodal model.

Load-bearing premise

The large reported gaps depend on the assumption that Pixel-Shuffle, LDPv2, and TokenPacker were just as carefully configured and tuned as LaCo when all four were moved to the 1/4 encoder layer.

Editorial extensions

If this is right

  • Vision encoders can be made to operate on a reduced token set from an intermediate layer onward, so the efficiency gain compounds through the remaining encoder layers, the projector, and the language model.
  • The residual shortcut is the load-bearing difference: without it, pixel-shuffle at an inner layer loses fine-grained detail on OCR-style and document-understanding benchmarks, so any inner-layer compression method needs an information-preserving path.
  • The reported efficiency gains make inner-layer compression attractive in compute-limited settings: over 20% faster training and over 15% higher inference throughput with a modest accuracy trade-off on detail-heavy single-image tasks.
  • There is a usable depth trade-off: compressing at the 1/2 layer loses little performance, while 1/4 gives larger efficiency gains, and the paper recommends the 1/2 layer as a balance between accuracy and speed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The residual idea is method-agnostic: any token-reduction operator inserted early in an encoder, including attention-based pruning or learned merges, could be paired with a parameter-free space-to-channel shortcut, so the principle likely transfers beyond pixel-shuffle.
  • The non-monotonic depth results, where 1/12 outperforms 1/6, hint that the earliest layers produce features too immature to compress while later layers refine a compact representation; an adaptive layer-selection policy could pick the compression point per image or task.
  • A hybrid design that compresses inside the encoder but passes a small set of high-resolution tokens through a parallel branch could recover the document and OCR accuracy lost at the 1/4 layer while keeping most of the speed gain.
  • Because the residual shortcut is parameter-free, the compression ratio $r$ could be made adaptive per image or per layer without adding parameters, which the paper lists as future work but does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes LaCo, a layer-wise visual token compression method for multimodal LLMs. LaCo inserts a Patch Merge Layer (PML) after an intermediate layer of the vision encoder; the PML uses pixel-shuffle followed by a two-layer MLP, and adds a non-parametric residual shortcut based on channel-averaged pixel-shuffle (Eq. 4). The method is trained with LLaVA-OneVision stages and evaluated on a broad set of single-image, multi-image, and video benchmarks using AIMv2, SigLIP, and InternViT encoders. The authors report that LaCo outperforms Pixel-Shuffle, LDPv2, and TokenPacker when all are placed at the 1/4 encoder layer, and that inner-layer compression improves training efficiency by over 20% and inference throughput by over 15% relative to external compression.

Significance. If the comparative results are sound, the paper would make a useful contribution by demonstrating that token compression inside the vision encoder is feasible and efficient, and by providing a simple architectural recipe (pixel-shuffle plus residual) that works across multiple encoders and tasks. The paper is strong in breadth: it covers three vision encoders, five compression depths, three benchmark groups, and reports training and inference efficiency metrics. However, the central comparative claim is currently under-supported because the only baselines in Tables 1–3 are three re-implemented methods, and the residual branch is mathematically equivalent to standard 2D average pooling, which is not included as a baseline. The work is therefore significant conditional on a fair and complete comparison.

major comments (3)
  1. [§3.3, Eq. (4), Tables 1–3] The residual shortcut defined in Eq. (4), RC(E,r)=CA(PS(E,r)), is mathematically equivalent to 2D average pooling with kernel size r over the spatial feature map: pixel-shuffle rearranges the tensor to (H/r)×(W/r)×(r²C), and the channel-averaging step collapses the r² subpixel channels to C channels. Since average pooling is itself an existing compression method (the paper cites Gemma3's use of it in §1), the comparison in Tables 1–3 is incomplete: LaCo contains an average-pooling path plus a trainable MLP branch, yet no average-pooling baseline appears, and no ablation removes the RC branch. The claim in the Abstract and §4.2 that LaCo 'outperforms all existing methods' therefore needs, at minimum, (i) an average-pooling baseline inserted at the same 1/4 layer, and (ii) an ablation of LaCo without the residual branch, to separate the contribution of the MLP branch from that of the existing pooling operation.
  2. [§4.1, Tables 1–3] The baseline comparison is not adequately controlled. The paper states that Pixel-Shuffle, LDPv2, and TokenPacker are 'implemented' at the 1/4 layer of AIMv2, but no details are given for how these methods were adapted from their original post-encoder design: no hyperparameter search, no warmup schedule, no learning-rate adjustments, and no description of how each method's internal components (e.g., TokenPacker's point-to-region attention) behave at intermediate encoder features. Since all three baselines were originally designed for external compression, the large reported gaps (over 30% average in Table 1) could substantially reflect untuned or misconfigured baselines rather than an intrinsic advantage of LaCo's residual design. The authors should either provide a clear adaptation and tuning protocol for the relocated baselines, or report the best known configurations with evidence that the comparison is fair.
  3. [§4.1, Tables 1, 4, 8] Efficiency claims are not verifiable from the reported numbers. The metrics PT, IT, VT, and TPS are presented without any measurement methodology: no hardware specification, framework version, batch size, image resolution, input sequence length, or number of repeated runs. Without this information, the headline claims of 'training efficiency beyond 20% and inference throughput over 15%' (Abstract; also §4.3) cannot be reproduced or compared across methods. Please add a clear description of the measurement setup and, ideally, report standard deviations across runs.
minor comments (4)
  1. [§4.3] The sentence 'AIMv2-LaCo@1/4 achieves 7.7% improvement in PT, 26.5% in IT, 48.8% in VT, and 15.0% in TPS compared to compared to AIMv2-LaCo@1' contains a duplicated 'compared to'.
  2. [Figure 3] Figure 3 contains non-English text ('图文连接层对齐训练') and the frozen/trainable status of components is indicated by icons that are not explained in the caption; please translate the labels and replace the icons with a clear legend.
  3. [Tables 1, 4, 8] Several benchmarks report two numbers separated by a slash (e.g., '63.8/65.0' for DocVQA in Table 1) with no explanation of what the two values represent; please define the dual scores in a table footnote.
  4. [References] The reference for ScienceQA (Saikh et al., 2022) appears to be a different dataset (a question-answering resource for scholarly articles) rather than the ScienceQA benchmark used in the evaluation; please verify and correct the citation.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity; evaluation is external and the residual branch is non-parametric. Mild flag: Eq. 4's residual shortcut is 2D average pooling in new notation, and Tables 1–3 omit an average-pooling baseline and any residual ablation.

  1. renaming known result [Section 3.3, Eq. (4); compare Section 2.2 and Tables 1–3]
    "the residual pathway consists of a non-parametric shortcut realized through a space-to-channel operation, followed by a channel averaging step to align the channel dimension. The process is formulated as: ˆEv = RC(Ek v , r) =CA(P S(Ev, r))"

    By construction, PS rearranges H×W×C into (H/r)×(W/r)×(r²C) and CA averages the r² sub-pixel groups back to C channels, exactly 2D average pooling. The paper itself attributes '2D average pooling operations' to Gemma3 in Section 2.2, so the 'non-parametric shortcut' is an existing compression method renamed as residual learning. Tables 1–3 compare LaCo only against Pixel-Shuffle, LDPv2, and TokenPacker, none of which contain this pooled branch, and no ablation isolates the MLP branch without RC. The reported 'over 30%' single-image margin is therefore computed against models lacking the very term Eq. 5 adds, so the 'outperforms all existing methods' claim is unsecured against the existing method LaCo contains by construction.

full rationale

The core derivation and evaluation are self-contained: LaCo is measured on held-out public benchmarks, so no fitted parameter is relabeled as a prediction and no score is implied by the training objective by construction. The reported training/inference efficiency numbers (PT, IT, VT, TPS) are measured, not derived from the compression definition. The residual branch in Eq. 4 is a fixed non-parametric operation, not a parameter fit to a subset of the evaluation data. The two self-citations that share the authors' group (Lan et al. 2024; Lan et al. 2025) appear only in related-work surveys and carry no load for the method's claims. The only reduction worth flagging is Eq. 4: RC(E,r) = CA(PS(E,r)) is 2D average pooling in pixel-shuffle notation, and Tables 1–3 omit an average-pooling baseline while no ablation removes the residual term, so part of the reported margin over 'all existing methods' is not isolated. That is a baseline-completeness and attribution gap, not a claim that reduces by definition to its inputs, which keeps the score low (2).

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical or architectural entities are invented; the components are standard (pixel-shuffle, MLP, residual shortcut). The key uncharged assumptions are fair baseline implementation and the adequacy of channel-averaging as the residual.

free parameters (3)
  • compression ratio r = 2 (implied by 1/4 token count and N/r² formula)
    The paper states the compression ratio r is used in the PML, and experiments compress to 1/4 of tokens, implying r=2, but the paper does not report ablations over r.
  • insertion depth (1/12, 1/6, 1/4, 1/2, 1) = chosen by hand: 1/4 for main comparisons
    The layer at which PML is inserted is a design choice, and the paper evaluates several depths. The main tables use 1/4, while the conclusion suggests 1/2 is a good balance; this choice is empirical and not derived.
  • PML MLP hidden dimension and initialization = two-layer MLP, unspecified hidden size
    The MLP that maps merged r²C channels back to C is a learned component whose size and initialization are not specified, yet it is central to the method.
assumptions (3)
  • domain assumption The LLM (Qwen2.5-0.5B), vision encoders (AIMv2, SigLIP, InternViT), and LLaVA-OneVision training pipelines are used as given.
    Section 4.1 specifies these components. The paper's results depend on their pre-trained weights and training data behavior.
  • ad hoc to paper The three baselines (Pixel-Shuffle, LDPv2, TokenPacker) were correctly re-implemented and fairly configured at the 1/4 insertion point.
    Section 4.1 states all methods are implemented by the authors at the 1/4 layer with the same data; no tuning or adaptation details are given, yet the central comparison depends on this.
  • domain assumption Channel averaging (CA) of pixel-shuffled tokens is a sufficient non-parametric approximation of the merged representation for the residual path.
    Section 3.3 asserts the residual shortcut preserves information, but there is no analysis or ablation isolating the channel-average choice against other non-parametric options.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/PTDDNMDW

@misc{pith2026250702279,
  author       = {Pith},
  title        = {Pith review of: LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PTDDNMDW}},
  note         = {Machine review of arXiv:2507.02279}
}
read the original abstract

Existing visual token compression methods for Multimodal Large Language Models (MLLMs) predominantly operate as post-encoder modules, limiting their potential for efficiency gains. To address this limitation, we propose LaCo (Layer-wise Visual Token Compression), a novel framework that enables effective token compression within the intermediate layers of the vision encoder. LaCo introduces two core components: 1) a layer-wise pixel-shuffle mechanism that systematically merges adjacent tokens through space-to-channel transformations, and 2) a residual learning architecture with non-parametric shortcuts that preserves critical visual information during compression. Extensive experiments indicate that our LaCo outperforms all existing methods when compressing tokens in the intermediate layers of the vision encoder, demonstrating superior effectiveness. In addition, compared to external compression, our method improves training efficiency beyond 20% and inference throughput over 15% while maintaining strong performance.

Figures

Figures reproduced from arXiv: 2507.02279 by the authors.

Figure 1
Figure 1. (a) The solid and the dotted represent vision [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison between encoder-level and layer-level visual token compression. The left presents compress [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Detailed overview of the training stage. The left figure represents stage-1, which only learn the projector [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The percentages of performance and training [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visual comparison between Pixel-Shuffle( [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    CRAFT recursively merges video tokens with training-free similarity selection plus learnable gated fusion, retaining ~97% of average accuracy at 8x compression across six benchmarks.

  2. Stateful Token Reduction for Long-Video Hybrid VLMs

    cs.CV 2026-02 conditional novelty 6.0 of 10

    For hybrid Mamba–Transformer video models, keeping 25% of visual tokens with a query-based progressive schedule gives 3.8–4.2x prefilling speedups with near-baseline accuracy; the paper attributes this to stateful com...

Reference graph

Works this paper leans on

61 extracted references · 19 canonical work pages · cited by 2 Pith papers

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, and 8 others. 2022. Flamingo: a visual language model for few-shot le...

    arXiv 2022

  2. [2]

    Daich Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. 2021. Scanqa: 3d question answering for spatial scene understanding. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19107--19117

    work page 2021

  3. [3]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv, abs/2308.12966

    arXiv 2023

  4. [4]

    Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yuxin Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yuxin Pan, Giancarlo Baldan, and Oscar Beijbom. 2019. nuscenes: A multimodal dataset for autonomous driving. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11618--11628

    work page 2019

  5. [5]

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. 2023. Honeybee: Locality-enhanced projector for multimodal llm. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13817--13827

    work page 2023

  6. [6]

    Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han. 2024 a . Deep compression autoencoder for efficient high-resolution diffusion models. ArXiv, abs/2410.10733

    arXiv 2024

  7. [7]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024 b . An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision

    work page 2024

  8. [8]

    Lin Chen, Jinsong Li, Xiao wen Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision

    work page 2023

Show all 61 references
  1. [9]

    Lin Chen, Jinsong Li, Xiao wen Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. 2024 c . Are we on the right way for evaluating large vision-language models? ArXiv, abs/2403.20330

    2024 arXiv

  2. [10]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiao wen Dong, Hang Yan, Hewei Guo, Conghui He, Zhenjiang Jin, Chaochao Xu, Bin Wang, and 9 others. 2024 d . How far are we to gpt-4v? clo...

    2024 arXiv

  3. [11]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2024 e . https://arxiv.org/abs/2312.14238 Internvl: Scaling up vision foundation models and aligning for gen...

    2024 arXiv

  4. [12]

    Yew Ken Chia, Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. 2024. Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns. ArXiv, abs/2403.13315

    2024 arXiv

  5. [13]

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, and Chunhua Shen. 2023. Mobilevlm : A fast, strong and open vision language assistant for mobile devices. ArXiv, abs/2312.16886

    2023 arXiv

  6. [14]

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, and Chunhua Shen. 2024. Mobilevlm v2: Faster and stronger baseline for vision language model. ArXiv, abs/2402.03766

    2024 arXiv

  7. [15]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305.06500

    2023 arXiv

  8. [16]

    Susskind, and Alaaeldin El-Nouby

    Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guilherme Turrisi da Costa, Louis B'ethune, Zhe Gan, Alexander Toshev, Marcin Eichner, Moin Nabi, Yinfei Yang, Joshua M. Susskind, and Alaaeldin El-Nouby. 2024. Multimo...

    2024 arXiv

  9. [17]

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. 2024 a . Video-mme: The first-eve...

    2024 arXiv

  10. [18]

    Smith, Wei-Chiu Ma, and Ranjay Krishna

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024 b . Blink: Multimodal large language models can see but not perceive. ArXiv, abs/2404.12390

    2024 arXiv

  11. [19]

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023. 3d-llm: Injecting the 3d world into large language models. ArXiv, abs/2307.12981

    2023 arXiv

  12. [20]

    Harsh Jhamtani and Taylor Berg-Kirkpatrick. 2018. Learning to describe differences between pairs of similar images. ArXiv, abs/1808.10584

    2018 arXiv

  13. [21]

    Gemma Team Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram'e, Morgane Rivi \`e re, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casb...

    2025 arXiv

  14. [22]

    Aniruddha Kembhavi, Michael Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. ArXiv, abs/1603.07396

    2016 arXiv

  15. [23]

    Zhibin Lan, Liqiang Niu, Fandong Meng, Wenbo Li, Jie Zhou, and Jinsong Su. 2024. Avg-llava: A large multimodal model with adaptive visual granularity. ArXiv, abs/2410.02745

    2024 arXiv

  16. [24]

    Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. 2025. Llave: Large language and vision embedding models with hardness-weighted contrastive learning. ArXiv, abs/2503.04812

    2025

  17. [25]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . Llava-onevision: Easy visual task transfer. ArXiv, abs/2408.03326

    2024 arXiv

  18. [26]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023 a . Seed-bench: Benchmarking multimodal llms with generative comprehension. ArXiv, abs/2307.16125

    2023 arXiv

  19. [27]

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024 b . Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. ArXiv, abs/2407.07895

    2024 arXiv

  20. [28]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023 b . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning

    2023

  21. [29]

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. 2024 c . Tokenpacker: Efficient visual projector for multimodal llm. ArXiv, abs/2407.02392

    2024 arXiv

  22. [30]

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. 2023. Vila: On pre-training for visual language models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26679--26689

    2023

  23. [31]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023 a . Improved baselines with visual instruction tuning. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286--26296

    2023

  24. [32]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  25. [33]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 b . Visual instruction tuning. ArXiv, abs/2304.08485

    2023 arXiv

  26. [34]

    Xuejing Liu, Wei Tang, Xinzhe Ni, Jinghui Lu, Rui Zhao, Zechao Li, and Fei Tan. 2023 c . What large language models bring to text-rich vqa? ArXiv, abs/2311.07306

    2023 arXiv

  27. [35]

    Yuanzhan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023 d . Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision

    2023

  28. [36]

    Dongchen Lu, Yuyao Sun, Zilu Zhang, Leping Huang, Jianliang Zeng, Mao Shu, and Huo Cao. 2025. Internvl-x: Advancing and accelerating internvl series with efficient visual token compression. ArXiv, abs/2503.21307

    2025 arXiv

  29. [37]

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. Egoschema: A diagnostic benchmark for very long-form video language understanding. ArXiv, abs/2308.09126

    2023 arXiv

  30. [38]

    Joty, and Enamul Hoque

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. ArXiv, abs/2203.10244

    2022 arXiv

  31. [39]

    Minesh Mathew, Viraj Bagal, Rub \`e n P \'e rez Tito, Dimosthenis Karatzas, Ernest Valveny, and C.V. Jawahar. 2021. Infographicvqa. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2582--2591

    2021

  32. [40]

    Manmatha, and C

    Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, and C. V. Jawahar. 2020. Docvqa: A dataset for vqa on document images. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2199--2208

    2020

  33. [41]

    Koster, Junlin Zhang, Stephanie Winkler, and 5 others

    Viorica Puatruaucean, Lucas Smaira, Ankush Gupta, Adri \`a Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yezhou Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alexander Fr \'e chette, Hanna Klimc...

    2023 arXiv

  34. [42]

    Harsh Raj, Janhavi Dadhania, and Akhilesh Bhardwaj. 2021. Multi-image visual question answering. ArXiv, abs/2112.13706

    2021 arXiv

  35. [43]

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. 2022. Scienceqa: a novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23:289 -- 301

    2022

  36. [44]

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. ArXiv, abs/2403.15388

    2024

  37. [45]

    Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang

    Wenzhe Shi, Jose Caballero, Ferenc Husz \'a r, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. 2016 IEEE Conference on Computer Visi...

    2016

  38. [46]

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2019. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2019

  39. [47]

    Alane Suhr and Yoav Artzi. 2019. Nlvr2 visual bias analysis. ArXiv, abs/1909.10411

    2019 arXiv

  40. [48]

    Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell

    Ting-Hao, Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. 2016. https://arxiv.org/abs/1...

    2016 arXiv

  41. [49]

    Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, and 2 others. 2024 a . Muirbench: A com...

    2024 arXiv

  42. [50]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Ke-Yang Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 b . Qwen2-vl: Enhancing vision-language...

    2024 arXiv

  43. [51]

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2023. Cogvlm: Visual expert for pretrained language models. ArXiv, abs/2311.03079

    2023 arXiv

  44. [52]

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin. 2023. Q-bench: A benchmark for general-purpose foundation models on low-level vision. ArXiv, abs/2309.14181

    2023 arXiv

  45. [53]

    xAI. 2024. Grok-1.5. A Large Language Model (LLM) developed by xAI

    2024

  46. [54]

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9772--9781

    2021

  47. [55]

    Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. 2024. Fit and prune: Fast and training-free visual token pruning for multi-modal large language models. ArXiv, abs/2409.10197

    2024 arXiv

  48. [56]

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. 2024. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. ArXiv, abs/2409.02813

    2024 arXiv

  49. [57]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941--11952

    2023

  50. [58]

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. 2024. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision

    2024

  51. [59]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv, abs/2304.10592

    2023 arXiv

  52. [60]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  53. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.