Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

LinVT: Empower Your Image-level Large Language Model to Understand Videos

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A linear video tokenizer turns image LLMs into video LLMs

desk verdict Useful plug-and-play video tokenizer, but the headline zero-shot numbers are contaminated because the test benchmarks appear in the training set. read the letter →

arxiv 2412.05185 v2 pith:INYRK5U6 submitted 2024-12-06 cs.CV cs.LGcs.MM

Lishuai Gao , Yujie Zhong , Yingsen Zeng , Haoxian Tan , Dengjie Li , Zheng Zhao This is my paper · ORCID
classification cs.CVcs.LGcs.MM
keywords videounderstandinglargelanguagemodelstokenizerlineartransformationtokencompressionvision-languagealignmentmulti-scalepoolinginstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LinVT, a plug-and-play module that bolts onto an existing image-based large language model and, after training on video data only, gives that model video understanding without erasing its image skills. The central bet is that if the video tokenizer's outputs are weighted averages of the image model's own visual tokens, the original image–language alignment survives training, so the model can be taught video on video data alone. The authors test the module on six different image-LLMs, including some never trained on video, and report state-of-the-art scores across short, long, and open-ended video QA benchmarks, while image benchmarks stay at baseline levels. If the bet holds, any mature image-LLM can be upgraded to video with a small, compatible adapter instead of retraining a video model from scratch.

What carries the argument

The central object is the linear aggregation map $\Phi_{\mathrm{Lin}}(T_v, Q_{v,t}) = \mathrm{softmax}(W_q Q_{v,t} (W_k T_v)^\top) \cdot T_v$, where $T_v$ are the multi-scale visual tokens and $Q_{v,t}$ are text-conditioned scale-specific queries. Because the value matrix is the identity, with no residual connection and no value projection, every output token is a convex combination of input visual tokens, which is the linearity that is supposed to protect the image-LLM's existing visual–language alignment during video-only training. The complementary mechanism is representative information condensation: spatio-temporal significance scoring plus hard top-$k$ selection reduces thousands of frame tokens to a manageable set, and shifted-window average pooling creates multiple temporal scales so both short events and long dependencies survive. Together the two mechanisms, linearity for knowledge preservation and condensation for tractability, are what let a frozen image-LLM take on video tasks.

What would settle it

Run the paper's alignment-preservation ablation, original versus randomly initialized intermediate layer, on a second image-LLM such as Molmo or Qwen2-VL with the same video-only recipe; if image-benchmark scores drop substantially even with the original projector kept, the linearity premise is falsified for that host. A sharper test: because top-k selection makes the map non-linear, replace the softmax scoring with a fixed linear weighting and observe whether image preservation and video accuracy both change.

Watch

Extended reading notes

Core claim

LinVT claims that a frozen image-LLM can become a strong video-LLM through a tokenizer whose outputs are strictly convex combinations, weighted averages, of part of the frame-level visual tokens the image model already produces. The module has two parts: a Spatio-Temporal Visual Token Refiner (SVR), which scores tokens by spatio-temporal significance, keeps the top-k, and pools them at multiple temporal scales, and a Text-conditioned Token Aggregation (TTA), which uses scale-specific learnable queries that attend to visual tokens and to the user's text, then aggregates via a linear layer that computes softmax attention maps directly over the input values with no residual connections. Because each output is a weighted sum of input tokens, the argument goes, the original vision–language alignment is preserved; because only the informative tokens survive top-k selection, the video is condensed to a fixed, LLM-friendly length. Trained in two stages on video data only, alignment then instruction tuning, LinVT-based models, with LinVT-Qwen2-VL in particular, reach top accuracy on MSVD-QA, MSRVTT-QA, ActivityNet-QA, TGIF-QA, MVBench, VideoMME and long-video benchmarks, while the image-benchmark ablation shows performance roughly equal to the untouched image-LLM.

Load-bearing premise

The load-bearing premise is that outputting weighted averages of the image model's own visual tokens will preserve its image–language alignment through video-only training, so image ability is never traded away for video ability, a premise the paper supports with a single model (Mipha-1.6B) rather than a proof or a multi-model ablation.

Editorial extensions

If this is right

  • Any mature image-LLM can be upgraded to video understanding by inserting LinVT and training only on video data, without a from-scratch video pretraining pipeline.
  • Because the base LLM, vision encoder, and intermediate layers stay frozen during alignment, with only LoRA on the LLM during instruction tuning, the upgrade cost is small relative to training a video-LLM from scratch.
  • Video-only training with LinVT preserves image-level ability, so a single model can serve both image and video tasks after the upgrade.
  • The multi-scale and text-conditioned design lets the same module handle events of different durations and question-relevant content, which is why long-video benchmarks improve along with short-video QA.
  • LinVT also boosts models that already have some video capability, such as InternVL2, Aquila, and Qwen2-VL, showing it is an add-on rather than merely an adapter for image-only models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper tests the alignment-preservation premise only on Mipha-1.6B, a natural extension is to run the same original-versus-randomized-projector ablation on the other five base models; if image scores drop on any of them, the linearity premise is host-specific rather than general.
  • The full tokenizer is not literally linear in all input tokens because hard top-k selection and softmax scoring are discrete and nonlinear steps; replacing the softmax scoring with a fixed linear weighting would test whether the preservation effect comes from the convex-combination form or simply from freezing the image backbone.
  • The state-of-the-art comparisons are against open-source video-LLMs, so a direct comparison with commercial closed models such as GPT-4o or Gemini on the same video benchmarks would clarify how close video-only-trained LinVT gets to much larger systems.
  • The recipe suggests a broader principle: any modality adapter whose outputs are convex combinations of a frozen backbone's tokens may inherit that backbone's alignment, a hypothesis that could be tested on other modality transfers such as audio or point clouds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes LinVT, a plug-and-play video tokenizer module that converts image-level LLMs into video-LLMs after training on video data. The module is built on two principles: linearity (output tokens are convex combinations of input visual tokens) and representative information condensation (achieved via spatio-temporal significance scoring, top-k selection, multi-scale pooling, and text-conditioned aggregation). The authors integrate LinVT into six recent image-LLMs (Aquila, BLIP-3, InternVL2, Mipha, Molmo, Qwen2-VL) and report state-of-the-art results on a range of short, long, and open-ended video QA benchmarks, along with evidence that image understanding is preserved.

Significance. If the claims are validated, LinVT offers a lightweight, broadly compatible method for adapting image-LLMs to video without destroying their image-level competence. The ablations are systematic (SVR/TTA roles, multi-scale variants, text conditioning, design choices) and the compatibility across six diverse backbones is a strength. Independent evidence from benchmarks not present in the training mixture (MVBench, Video-MME, EgoSchema, MLVU, LongVideoBench) is reported, which partially supports the method's generality. However, the headline 'zero-shot' results in Table 7 and the associated SOTA claims are undermined by the fact that the four evaluation datasets also appear in the training set (Table 1), so the strength of the central contribution is not adequately validated as presented.

major comments (3)
  1. [Section 4.2, Table 7 vs Table 1] The four 'zero-shot' open-ended QA benchmarks in Table 7 (MSVD-QA, MSRVTT-QA, ActivityNet-QA, TGIF-QA) are all listed in the Stage-2 instruction-tuning dataset in Table 1. The model is therefore trained on those exact datasets before being evaluated on them, so the results are in-domain, not zero-shot. This invalidates the 'zero-shot' label and materially weakens the state-of-the-art claims in the abstract and introduction. The authors must either re-run the evaluation on truly held-out benchmarks or clearly disclose that the train splits of these benchmarks were used and remove the 'zero-shot' terminology; the same issue affects NExT-QA, which appears both in Table 1 and Table 8.
  2. [Section 3.2, Eq. (1) and Table 3] The paper's central design principle, that linear (convex) combination of the image-LLM's visual tokens preserves visual-language alignment, is not directly tested. The full LinVT encoder includes nonlinear operations (top-k token selection and attention scoring), so the overall map is not linear. The ablation in Table 3 compares the original intermediate layer against a randomly initialized one, which changes many factors beyond linearity; it does not isolate linearity as the cause of preserved image ability. Please provide a more controlled comparison, e.g., against a nonlinear tokenizer of similar capacity, or soften the claim to state that the specific linear-aggregation design preserves the empirical value distribution and is conjectured to help alignment.
  3. [Section 3.3, Training recipe] The description of Stage-1 training is underspecified: 'Training is conduct under contrastive loss and cross-entropy loss' is vague. Which contrastive loss is used, over what pairs, and how is the cross-entropy loss applied? This level of detail is necessary for reproducibility, especially since the stage is claimed to be essential for the final performance.
minor comments (6)
  1. [Section 1] Typo: 'at rish' should be 'at risk' in the paragraph on excessive visual tokens.
  2. [Section 4.2] Heading 'Sate-of-The-Art' should be 'State-of-the-Art'.
  3. [Tables 3 and 10] The caption 'Importantce' is a typo for 'Importance'.
  4. [Table 7 and other tables] The notation '80.2 / 4.4' is used throughout but is not explained in any caption; please define that the first number is accuracy and the second is a score (e.g., CIDEr or consistency) in the table notes.
  5. [Section 6 (supplementary)] Claiming 267M parameters is 'negligible compared to the original billion-size image-LLMs' is misleading for the 1.6B Mipha model, where it is a substantial fraction; please report the relative overhead per backbone.
  6. [Section 3.2.1] The description of spatio-temporal significance scoring is incomplete: 'applying self-attention within the corresponding visual token' does not explain how scalar significance scores are produced from the attention maps; please specify the exact computation.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported zero-shot SOTA on MSVD-QA, MSRVTT-QA, ActivityNet-QA and TGIF-QA is circular because those benchmarks appear in the Stage-2 training set (Table 1).

  1. fitted input called prediction [Sec. 4.2 / Table 7 vs Table 1 (Stage-2 training data)]
    ""we evaluate the zero-shot performance on the open-ended video QA benchmark on MSVD-QA (MVD-QA) [70], MSRVTT-QA (MTT-QA) [70], ActivityNet-QA (Act-QA) [77] and TGIF-QA [22]." Table 1 Stage-2 rows: "ActivityNet-QA [77] 29k, MSVD-QA [70] 30k, MSRVTT-QA [70] 109k, TGIF-QA [22] 71k.""

    The four benchmarks presented as zero-shot predictions in Sec. 4.2 / Table 7 are exactly the instruction-tuning datasets listed in Table 1 (Stage 2). LinVT and the LoRA adapter were trained on those very benchmarks, so the reported numbers (e.g., LinVT-Qwen2-VL 80.2 on MVD-QA, 66.2 on MTT-QA, 60.1 on Act-QA, 81.3 on TGIF-QA) are in-domain supervised evaluations, not zero-shot generalizations. Even if only the train split of each dataset was used for tuning, the evaluation is not zero-shot. The 'zero-shot' label is therefore supplied by the training set itself, not by held-out prediction; the claim of state-of-the-art zero-shot performance on these four tasks reduces to having trained on those tasks.

full rationale

The architectural derivation of LinVT is not circular. LinVT's linear aggregation (Eq. 1) is a design choice, and the claim that weighted averages of input visual tokens preserve image-LLM knowledge is an empirical hypothesis tested in Table 3; it is not obtained by assuming the conclusion. There is no load-bearing self-citation or imported uniqueness theorem; references to co-authored prior work (e.g., UniMD, DetAL, React, TriDet) only motivate multi-scale design and are not used to force LinVT's form. The one genuine circularity is evaluation-level: Sec. 4.2 labels MSVD-QA, MSRVTT-QA, ActivityNet-QA and TGIF-QA as 'zero-shot' benchmarks, but Table 1 lists all four datasets in the Stage-2 video instruction-tuning mixture. Therefore the Table 7 SOTA numbers on these benchmarks are fitted/in-domain rather than independent predictions. NExT-QA, also in Table 1, is additionally reported in Table 8. Because the method still has independent support from benchmarks not in the training mixture (Video-MME, EgoSchema, VideoVista, TempCompass, MLVU, LongVideoBench), the circularity is partial and confined to the zero-shot QA claim; the core linear-tokenizer derivation itself is not circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the design principle that linear (weighted-average) token outputs preserve image-language alignment, plus practical assumptions about token selection, video segmentation with AutoShot, and evaluation fairness. No new physical entities are introduced; the module is an implemented architecture, so invented_entities is empty. The most fragile items are the alignment-preservation premise and the assumption that training splits of evaluation benchmarks do not affect zero-shot results.

free parameters (4)
  • top-k token count k = 2048
    Number of visual tokens retained by top-k selection in SVR; chosen via ablation in Table 12, where 4096 gives marginal gains, 1024 hurts, and 2048 balances performance and compute.
  • number of scales and scale-specific query counts = 3 scales: 64, 32, 16
    Set in Sec 6 and ablated in Table 12: {128, 64, 32} gives slight gains with more compute, {32, 16, 8} drops performance, so {64, 32, 16} is selected.
  • number of SVR and TTA layers L = 4
    Selected via ablation in Table 12: doubling to 8 gives marginal gains, reducing to 2 degrades performance.
  • number of attention heads = 8
    Stated in Sec 6 as an implementation detail; not ablated.
assumptions (4)
  • ad hoc to paper A convex combination (weighted average) of the frozen image-LLM's visual tokens preserves the original visual-language alignment, so training only on video data will not destroy image ability.
    Introduced as a design principle in Sec 3.2 and relied on for the no-image-data training recipe; supported only by the Mipha-1.6B ablation in Table 3, with no proof or larger-scale verification.
  • domain assumption Top-k significance scoring plus multi-scale pooling retains enough information for video understanding across temporal scales.
    Core condensation assumption in Sec 3.2.1; if important tokens are discarded, downstream video QA degrades.
  • domain assumption AutoShot temporal boundary detection provides correct clip segmentation for the video tokenizer.
    Stated in Sec 6 as a preprocessing step; no ablation runs without AutoShot, so performance depends on this external tool.
  • ad hoc to paper The instruction-tuning datasets are used only via their train splits, so evaluating on the same benchmarks is considered zero-shot.
    Assumed in Sec 4.2; this is the contested premise flagged by the selection_bias red flag.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LinVT: Empower Your Image-level Large Language Model to Understand Videos." pith.science (2026). https://pith.science/paper/INYRK5U6

@misc{pith2026241205185,
  author       = {Pith},
  title        = {Pith review of: LinVT: Empower Your Image-level Large Language Model to Understand Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/INYRK5U6}},
  note         = {Machine review of arXiv:2412.05185}
}
read the original abstract

Large Language Models (LLMs) have been widely used in various tasks, motivating us to develop an LLM-based assistant for videos. Instead of training from scratch, we propose a module to transform arbitrary well-trained image-based LLMs into video-LLMs (after being trained on video data). To better adapt image-LLMs for processing videos, we introduce two design principles: linear transformation to preserve the original visual-language alignment and representative information condensation from redundant video content. Guided by these principles, we propose a plug-and-play Linear Video Tokenizer(LinVT), which enables existing image-LLMs to understand videos. We benchmark LinVT with six recent visual LLMs: Aquila, Blip-3, InternVL2, Mipha, Molmo and Qwen2-VL, showcasing the high compatibility of LinVT. LinVT-based LLMs achieve state-of-the-art performance across various video benchmarks, illustrating the effectiveness of LinVT in multi-modal video understanding.

Figures

Figures reproduced from arXiv: 2412.05185 by the authors.

Figure 1
Figure 1. By being trained on video data, LinVT can endow an [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework of the LinVT-based video-LLM. The LinVT module takes visual tokens corresponding to individual frames of a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The left part represents single-scale token processing, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Visualization of the patches corresponding to the se [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Visualization of the patches corresponding to the selected tokens in video frames. The selection is achieved by the spatio [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Visualization of VQA on long videos. The captions generated by LinVT exhibit a greater focus on fundamental facts. The [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DisTime: Distribution-based Time Representation for Video Large Language Models

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A single learnable time token, decoded into a probability distribution over time bins, improves temporal grounding in Video-LLMs and is trained partly on a new 1.25M-event pseudo-labeled dataset.

  2. AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multi-modal Information Between Reality and Videos

    cs.HC 2025-07 conditional novelty 6.0 of 10

    AROMA pairs a blind cook's spoken descriptions of what they feel, smell, and taste with a wearable camera and a video recipe to answer questions and issue proactive alerts, and eight participants rated it usable despi...

  3. FlexSelect: Flexible Token Selection for Efficient Long Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    FlexSelect selects a small fraction of query-relevant visual tokens using attention from an intermediate layer, improving long-video accuracy and inference speed across multiple VideoLLMs.

  4. ${\mu}^2$Tokenizer: Differentiable Multi-Scale Multi-Modal Tokenizer for Radiology Report Generation

    cs.LG 2025-06 reject novelty 4.0 of 10

    A tokenizer that combines multi-scale CT image features with text questions, plus DPO training on a clinical metric, is claimed to improve automated radiology report generation.

Reference graph

Works this paper leans on

91 extracted references · 25 canonical work pages · cited by 4 Pith papers

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Es- sam Sleiman, Deyao Zhu, Jian Ding, and Mohamed El- hoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.03413, 2024. 8

    arXiv 2024

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1, 8

    arXiv 2023

  4. [4]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1728–1738,

  5. [5]

    Revisiting the” video” in video-language understanding

    Shyamal Buch, Crist ´obal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the” video” in video-language understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2917–2927, 2022. 5

    2022

  6. [6]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceed- ings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015. 3

    2015

  7. [7]

    Sharegpt4video: Improving video understand- ing and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 8

    arXiv 2024

  8. [8]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 1, 2, 3, 6

    arXiv 2024

Show all 91 references
  1. [9]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 2, 8

    2024 arXiv

  2. [10]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6,

    2023

  3. [11]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  4. [12]

    Molmo and pixmo: Open weights and open data for state-of-the-art 9 multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art 9 multimodal models. arXiv preprint arXiv:2409.17146,...

    2024 arXiv

  5. [13]

    Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing. arXiv preprint arXiv:2406.14515, 2024. 6

    2024 arXiv

  6. [14]

    Video-ccam: Enhancing video-language un- derstanding with causal cross-attention masks for short and long videos

    Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language un- derstanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408.14023, 2024. 8

    2024 arXiv

  7. [15]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 6

    2023 arXiv

  8. [16]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 6

    2024 arXiv

  9. [17]

    Lazyllm: Dynamic token pruning for efficient long context llm inference

    Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mo- hammad Rastegari, and Mahyar Najibi. Lazyllm: Dynamic token pruning for efficient long context llm inference. arXiv preprint arXiv:2407.14057, 2024. 3

    2024 arXiv

  10. [18]

    Saliency-guided detr for mo- ment retrieval and highlight detection

    Aleksandr Gordeev, Vladimir Dokholyan, Irina Tolstykh, and Maksim Kuprashevich. Saliency-guided detr for mo- ment retrieval and highlight detection. arXiv preprint arXiv:2410.01615, 2024. 4

    2024

  11. [19]

    Infinity-mm: Scaling multimodal perfor- mance with large-scale and high-quality instruction data,

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, Zhenchong Hu, Bo-Wen Zhang, Jijie Li, Dong Liang, Yingli Zhao, Yulong Ao, Yaoqi Liu, Fangxiang Feng, and Guang Liu. Infinity-mm: Scaling multimodal pe...

  12. [20]

    Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models. In P...

    2024

  13. [21]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 6

    2021 arXiv

  14. [22]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2758–2766, 2017. 6

    2017

  15. [23]

    Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 137...

    2024

  16. [24]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 235–

    2016

  17. [25]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems , 34: 11846–11858, 2021. 4

    2021

  18. [26]

    Mimic- it: Multi-modal in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic- it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023. 3

    2023 arXiv

  19. [27]

    Seed-bench: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13299–13308, 2024. 6

    2024

  20. [28]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 8

    2024 arXiv

  21. [29]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 1, 2, 5

    2023

  22. [30]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2, 3, 8

    2023 arXiv

  23. [31]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  24. [32]

    M3it: A large-scale dataset towards multi-modal multilingual instruction tuning

    Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. M3it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387, 2023. 3

    2023 arXiv

  25. [33]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 6

    2023 arXiv

  26. [34]

    Videovista: A versatile bench- mark for video understanding and reasoning

    Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile bench- mark for video understanding and reasoning. arXiv preprint arXiv:2406.11303, 2024. 6

    2024 arXiv

  27. [35]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2025. 8

    2025

  28. [36]

    Detal: Open-vocabulary temporal action 10 localization with decoupled networks

    Zhiheng Li, Yujie Zhong, Ran Song, Tianjiao Li, Lin Ma, and Wei Zhang. Detal: Open-vocabulary temporal action 10 localization with decoupled networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 4

    2024

  29. [37]

    Video-llava: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 8

    2023 arXiv

  30. [38]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 8

    2024

  31. [39]

    Univtg: Towards unified video- language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shra- man Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video- language temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2794–...

    2023

  32. [40]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 2, 3

    2024

  33. [41]

    Kangaroo: A powerful video-language model supporting long-context video input

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xi- aoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542,

    arXiv

  34. [42]

    Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024. 6

    2024 arXiv

  35. [43]

    Et bench: Towards open-ended event-level video-language understanding

    Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Ying Shan, and Chang Wen Chen. Et bench: Towards open-ended event-level video-language understanding. arXiv preprint arXiv:2409.18111, 2024. 3

    2024 arXiv

  36. [44]

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Ji- wen Lu, and Yongming Rao. Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961, 2024. 8

    2024 arXiv

  37. [45]

    Valley: Video assistant with large language model enhanced ability

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207 ,

    arXiv

  38. [46]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 3, 6, 8

    2023 arXiv

  39. [47]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural In- formation Processing Systems, 36:46212–46244, 2023. 6

    2023

  40. [48]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 6

    2021

  41. [49]

    Snag: Scalable and accurate video grounding

    Fangzhou Mu, Sicheng Mo, and Yin Li. Snag: Scalable and accurate video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18930–18940, 2024. 4

    2024

  42. [50]

    4v (ision) system card

    GPT OpenAI. 4v (ision) system card. preprint, 2023. 8

    2023

  43. [51]

    Gpt-4 technical report

    R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023. 3

    2023 arXiv

  44. [52]

    Scanning only once: An end-to-end framework for fast temporal grounding in long videos

    Yulin Pan, Xiangteng He, Biao Gong, Yiliang Lv, Yujun Shen, Yuxin Peng, and Deli Zhao. Scanning only once: An end-to-end framework for fast temporal grounding in long videos. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 13767–13777, 2023. 4

    2023

  45. [53]

    Per- ception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Re- casens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Per- ception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing ...

    2024

  46. [54]

    Testa: Temporal-spatial token aggregation for long-form video-language understanding

    Shuhuai Ren, Sishuo Chen, Shicheng Li, Xu Sun, and Lu Hou. Testa: Temporal-spatial token aggregation for long-form video-language understanding. arXiv preprint arXiv:2310.19060, 2023. 3

    2023 arXiv

  47. [55]

    Timechat: A time-sensitive multimodal large lan- guage model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large lan- guage model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14313–14323, 2024. 5, 8

    2024

  48. [56]

    xgen-mm-vid (blip-3- video): You only need 32 tokens to represent a video even in vlms

    Michael S Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles. xgen-mm-vid (blip-3- video): You only need 32 tokens to represent a video even in vlms. arXiv preprint arXiv:2410.16267, 2024. 8

    arXiv

  49. [57]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388,

  50. [58]

    Tempme: Video temporal token merging for efficient text-video re- trieval

    Leqi Shen, Tianxiang Hao, Sicheng Zhao, Yifeng Zhang, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Tempme: Video temporal token merging for efficient text-video re- trieval. arXiv preprint arXiv:2409.01156, 2024. 3

    2024 arXiv

  51. [59]

    React: Temporal action detection with relational queries

    Dingfeng Shi, Yujie Zhong, Qiong Cao, Jing Zhang, Lin Ma, Jia Li, and Dacheng Tao. React: Temporal action detection with relational queries. In European conference on computer vision, pages 105–121. Springer, 2022. 4

    2022

  52. [60]

    Tridet: Temporal action detection with relative boundary modeling

    Dingfeng Shi, Yujie Zhong, Qiong Cao, Lin Ma, Jia Li, and Dacheng Tao. Tridet: Temporal action detection with relative boundary modeling. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 18857–18866, 2023. 4

    2023

  53. [61]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 6

    2019

  54. [62]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530, 2024. 8

    2024 arXiv

  55. [63]

    Look-m: Look- 11 once optimization in kv cache for efficient multimodal long- context inference

    Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look- 11 once optimization in kv cache for efficient multimodal long- context inference. arXiv preprint arXiv:2406.18139, 2024. 3

    2024 arXiv

  56. [64]

    Tarsier: Recipes for training and evaluating large video description models

    Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634, 2024. 8

    2024 arXiv

  57. [65]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 3, 6, 8

    2024 arXiv

  58. [66]

    Non-local neural networks

    Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim- ing He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 7794–7803, 2018. 5

    2018

  59. [67]

    Longvideobench: A benchmark for long-context inter- leaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context inter- leaved video-language understanding. arXiv preprint arXiv:2407.15754, 2024. 6

    2024 arXiv

  60. [68]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9777–9786, 2021. 6

    2021

  61. [69]

    Can i trust your answer? visually grounded video question answering

    Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13204– 13214, 2024. 3

    2024

  62. [70]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. In Proceedings of the 25th ACM international confer- ence on Multimedia, pages 1645–1653, 2017. 6

    2017

  63. [71]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 2, 8

    2024 arXiv

  64. [72]

    Slowfast-llava: A strong training-free base- line for video large language models

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free base- line for video large language models. arXiv preprint arXiv:2407.15841, 2024. 8

    2024 arXiv

  65. [73]

    Multiinstruct: Im- proving multi-modal zero-shot learning via instruction tun- ing

    Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Im- proving multi-modal zero-shot learning via instruction tun- ing. arXiv preprint arXiv:2212.10773, 2022. 3

    2022 arXiv

  66. [74]

    Longvila: Scaling long-context visual language models for long videos

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024. 8

    2024 arXiv

  67. [75]

    xgen-mm (blip-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024. 1, 2, 3, 6

    2024

  68. [76]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference on machine learn- ing. PMLR, 2024. 6

    2024

  69. [77]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yuet- ing Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 9127–9134, 2019. 6

    2019

  70. [78]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  71. [79]

    Unimd: Towards unifying moment retrieval and temporal ac- tion detection

    Yingsen Zeng, Yujie Zhong, Chengjian Feng, and Lin Ma. Unimd: Towards unifying moment retrieval and temporal ac- tion detection. In European Conference on Computer Vision, pages 286–304. Springer, 2025. 4

    2025

  72. [80]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 2, 8

    2023 arXiv

  73. [81]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 8

    2024 arXiv

  74. [82]

    Direct preference optimiza- tion of video large multimodal models from language model reward

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimiza- tion of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01258, 2024. 6

    2024 arXiv

  75. [83]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 8

    2024

  76. [84]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264,

    arXiv

  77. [85]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 3

    2023 arXiv

  78. [86]

    A comprehensive overhaul of multimodal assistant with small language models

    Minjie Zhu, Yichen Zhu, Xin Liu, Ning Liu, Zhiyuan Xu, Chaomin Shen, Yaxin Peng, Zhicai Ou, Feifei Feng, and Jian Tang. A comprehensive overhaul of multimodal assistant with small language models. arXiv preprint arXiv:2403.06199, 2024. 2, 3, 6

    2024 arXiv

  79. [87]

    Openai’s gpt-4o in surgical oncology: revolu- tionary advances in generative artificial intelligence

    Ning Zhu, Nan Zhang, Qipeng Shao, Kunming Cheng, and Haiyang Wu. Openai’s gpt-4o in surgical oncology: revolu- tionary advances in generative artificial intelligence. Euro- pean Journal of Cancer, 2024. 8

    2024

  80. [88]

    12 Autoshot: A short video dataset and state-of-the-art shot boundary detection

    Wentao Zhu, Yufang Huang, Xiufeng Xie, Wenxian Liu, Jin- can Deng, Debing Zhang, Zhangyang Wang, and Ji Liu. 12 Autoshot: A short video dataset and state-of-the-art shot boundary detection. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition ...

    2023

  81. [89]

    64, 32, and 16)

    Implementation details Unless specified in the ablation study, SVR consists of L = 4 spatio-temporal layers, while the TTA module incor- porates multi-scale processing with l = 3 different scales (i.e. 64, 32, and 16). The number of multi-heads in the TTA module is set to 8. P...

  82. [90]

    More validation results are presented in Tab

    More Experiments for Ablation Study Importantce of maintaining the original vision-language alignment. More validation results are presented in Tab. 10. The previous conclusion in Tab. 3 holds in the additional benchmarks. Role of LinVT in video-compatible image-LLMs. As illus...

  83. [91]

    Image patches of the selected tokens

    Qualitative Analyses We present visualizations of selected tokens using the spatio-temporal significance scoring and selection mech- anism, alongside examples of visual question answering (VQA) on long videos utilizing LinVT-InternVL2 (8B). Image patches of the selected tokens...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.