Pith. sign in

REVIEW 2 major objections 4 minor 2 cited by

TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models

T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a training-free video LLM can match video-trained giants by feeding a frozen image LLM a low-resolution thumbnail grid of a few frames plus uniformly sampled tokens from all frames, establishing a new state of the…

desk verdict A strong, well-controlled empirical baseline whose 'SOTA among training-free video LLMs' claim is undermined by the paper's own MVBench table if PLLaVA is training-free, as its title suggests. read the letter →

arxiv 2411.11066 v1 pith:SSV4M5DL submitted 2024-11-17 cs.CV

Tingyu Qu , Mingxiao Li , Tinne Tuytelaars , Marie-Francine Moens This is my paper · ORCID
classification cs.CV
keywords training-freevideoLLMvisualtokencompressionthumbnail-and-samplingquestionansweringMVBenchMLVUefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a powerful image-only LLM can understand video without any video-specific training, and answers yes if the visual tokens are organized correctly. It compares five token-compression strategies and finds that spatial pooling, the most common choice, is not very effective, while grid thumbnails and uniform token sampling each capture complementary strengths. The proposed TS-LLaVA combines both: a grid-view thumbnail made from a few equidistant frames gives a global summary, and uniformly sampled tokens from all frames preserve detail. On multiple-choice video QA benchmarks this training-free recipe sets a new state of the art, with the 34B version outperforming GPT-4V on MVBench and reaching accuracy comparable to Video-LLaMA2-72B on MLVU. The significance is that video understanding may not require large curated video-text datasets if token compression is designed well.

What carries the argument

Thumbnail-and-Sampling token construction. Given N frames, select N_T equidistant frames (N_T ≪ N, even) to build one grid-view thumbnail image I_T, encoded to F_T ∈ R^V; all N frames are encoded and uniformly sampled to F_S ∈ $R^{{M−V}}$; the LLM receives [F_S, F_T]. The thumbnail supplies a low-resolution global summary that shines on spatial and temporal layout reasoning, while uniform sampling preserves high-resolution details from every frame that support action reasoning; combining them at the same total compression rate outperforms either stream alone.

What would settle it

Run TS-LLaVA-34B and GPT-4V on the same MVBench instances with identical prompt templates and decoding settings, and TS-LLaVA-34B against Video-LLaMA2-72B on MLVU with identical settings; if the reported average gaps (52.6 vs 43.5 on MVBench and 43.0 vs 45.6 on MLVU) shrink, invert, or change qualitatively, the comparative claim fails while the internal ablations still stand.

Watch

Extended reading notes

Core claim

TS-LLaVA establishes that a frozen image LLM (LLaVA-v1.6, 7B or 34B) can be turned into a strong video QA system with no fine-tuning by replacing the visual token stream with a Thumbnail-and-Sampling schedule. From N uniformly sampled frames, a few equidistant frames are arranged into a single low-resolution grid image (the Thumbnail), whose 576 tokens summarize the video; the remaining token budget comes from uniformly sampling tokens across all N frames at full resolution. The two token groups are concatenated and passed to the LLM. The paper reports that this hybrid outperforms Concat, Pooling, Grid, Grids, and Sampling alone on Video-MME, and on NExT-QA, EgoSchema, IntentQA, MVBench, and MLVU it exceeds prior training-free methods, with the 34B model beating GPT-4V on MVBench average accuracy and matching Video-LLaMA2-72B on MLVU.

Load-bearing premise

The headline comparisons to GPT-4V and Video-LLaMA2-72B rest on benchmark numbers reported by other papers or official repositories rather than re-runs with identical prompts, frame sampling, and decoding settings; if those numbers are not directly comparable, the claimed superiority is not established.

Editorial extensions

If this is right

  • A frozen, image-only LLM can do competitive long-form video question answering without any video-text fine-tuning.
  • Token organization, not raw token count, drives training-free video performance: TS-LLaVA beats SF-LLaVA with roughly 60% of its visual tokens (2304 vs 3680).
  • The hybrid schedule inherits Grid's spatial and temporal reasoning and Sampling's action reasoning, improving both over either strategy on Video-MME.
  • The 34B model's MVBench average (52.6) and MLVU average (43.0) show that a smaller training-free model can stand comparison with much larger video-trained systems (GPT-4V and Video-LLaMA2-72B) on multiple-choice QA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline comparisons to GPT-4V and Video-LLaMA2-72B are taken from published numbers, not matched re-runs; identical-prompt runs could shift the gaps, so a direct head-to-head is the natural next check.
  • The success of uniform sampling over pooling suggests spatial averaging throws away information that frozen LLMs need for temporal and action reasoning; this may extend to other long-context or multi-image settings.
  • Choosing thumbnail frames by content (e.g., keyframes) rather than equidistant spacing is a straightforward extension the paper does not explore.
  • Because the method is training-free and backbone-agnostic, it can be transplanted to newer image LLMs; transferring the schedule is a cheap test of whether the compression principle generalizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes TS-LLaVA, a training-free video LLM that compresses visual tokens by combining a thumbnail image (a grid of a small number of equidistant frames) with uniformly sampled visual tokens from all frames. The authors first compare five token-compression strategies on Video-MME, then introduce Thumbnail-and-Sampling and evaluate it on Multiple Choice VideoQA benchmarks (NExT-QA, EgoSchema, IntentQA), MVBench, and MLVU, also reporting Open-Ended VideoQA and text-generation results in an appendix. The central claims are that TS-LLaVA achieves new state-of-the-art among training-free video LLMs, that TS-LLaVA-34B outperforms GPT-4V on MVBench, and that it is comparable to Video-LLaMA2-72B on MLVU.

Significance. If the SOTA claim is substantiated, the paper is a strong empirical contribution: it offers a simple, token-efficient compression schedule that works with frozen image LLMs, with well-matched ablations at fixed token budgets. The authors provide code, and their repeated GPT-evaluation case study in Sec. 7.2 is a useful methodological caution about open-ended video QA scoring. Even if the SOTA framing needs revision, the controlled comparisons among compression strategies and the positive MVBench/MLVU results are informative for the training-free video LLM literature.

major comments (2)
  1. [Sec. 4.2, Table 5] The paper classifies PLLaVA as a training-based video LLM ("is further trained on video data," Sec. 4.2) and places it above the training-free dashed line in Table 5, but the cited reference [44] is titled "PLLaVA: Parameter-free LLaVA Extension from Images to Videos" and presents a pooling-based method without a video-training stage. If PLLaVA is training-free, then on the paper's own Table 5a, TS-LLaVA-7B (45.5) and TS-LLaVA-34B (52.6) are both below PLLaVA-7B (46.6) and PLLaVA-34B (58.1), respectively, and the abstract's "new state-of-the-art among training-free video LLMs" claim is false. The authors must either provide evidence of a video training stage in PLLaVA or reclassify it and revise the SOTA statements accordingly.
  2. [Sec. 4.2, Tables 5 and 7] The headline comparisons to PLLaVA, GPT-4V, and Video-LLaMA2-72B rely on accuracy numbers taken from prior papers (footnote in Table 5) rather than on re-runs under the authors' own frame-sampling, prompt, and decoding protocol. Because the SOTA claim depends on these cross-paper numbers, the authors should either re-run the closest competitor (PLLaVA) under their protocol or explicitly restrict the SOTA claim to methods evaluated in a controlled setting; otherwise the comparison is not apples-to-apples.
minor comments (4)
  1. [Sec. 3.1, Table 1] The Grid strategy uses 576 visual tokens while the other strategies use 2304 tokens; the text notes this, but the comparison would be cleaner if the token-budget mismatch were presented as a separate analysis rather than within the same matched-budget table.
  2. [Sec. 4.3 and Appendix Sec. 7.3] Many design choices (50 input frames, 6 frames per thumbnail, one thumbnail, 3456 visual tokens) appear to be selected using the same benchmarks on which the final results are reported; the paper should disclose whether any held-out validation was used, and report variance or multiple seeds where feasible.
  3. [Eq. (1) and Fig. 3] The notation RM−V is slightly ambiguous because M is the total visual-token budget and V is the number of thumbnail tokens; defining M_s = M−V as the sampled-token budget would make the equation easier to follow.
  4. [Throughout] There are minor spelling inconsistencies (e.g., "TS-LLaV A" in the header vs. "TS-LLaVA" in the text; "Video-LLAMA2" vs. "Video-LLaMA2"), and the appendix Table 8 refers to "colored text" that is not visible in the grayscale version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TS-LLaVA is an empirical token-scheduling method validated against external benchmarks; the central claims do not reduce to the method's own inputs by construction.

full rationale

The paper is an empirical systems paper: it proposes a Thumbnail-and-Sampling visual token compression strategy for frozen image LLMs and evaluates it on external benchmarks (Video-MME, NExT-QA, EgoSchema, IntentQA, MVBench, MLVU). There is no derivation chain in which an output quantity is defined in terms of the quantity it is said to predict. The compression schedule is specified by construction (Eq. 1 is just uniform sampling of already-encoded visual features), and the benchmark accuracies are measured externally, not produced by the method. Hyperparameters such as number of frames, thumbnail size, and token budget are selected using held-out benchmark results, which is a model-selection concern, not circularity. The only self-citation is a pointer to the authors' prior work on routing functions for parameter-efficient fine-tuning ([32]), mentioned in the Limitations section as future potential; it is not load-bearing for the central claim. The paper's headline SOTA claim rests on the placement of PLLaVA above the 'trained on video data' line in Table 5 and on numbers taken from prior papers/repositories; this is a factual comparability and classification issue (the cited PLLaVA title says 'Parameter-free'), which could invalidate the SOTA comparison, but it is not a case of a prediction reducing to a fitted input or a self-citation chain. The authors also explicitly disclose unreliability of GPT-assisted open-ended evaluation and provide a statistical case study, which is an honest limitation rather than circular reasoning. No step in the paper exhibits the required form: an equation, fitted parameter, or uniqueness argument that is equivalent to its own input by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new mathematical axioms; its load-bearing assumptions are domain assumptions about generalization of frozen image LLMs, the fidelity of uniform token sampling, and the comparability of cross-paper benchmark numbers. The free parameters are design choices, each ablated in Sec. 4.3 and the appendix, but ultimately selected using the evaluation benchmarks themselves.

free parameters (5)
  • Number of input frames N = 50
    Set as maximum and ablated (Fig. 4a); 34B benefits from more frames while 7B saturates.
  • Frames per thumbnail NT = 6
    Chosen after ablations (Fig. 4b, Table 11); 6 gives the best EgoSchema accuracy for both model sizes.
  • Number of thumbnail images k = 1
    Chosen after ablations (Fig. 4c, Table 10); more thumbnails degrade EgoSchema due to higher compression of sampled tokens.
  • Total visual token budget M_total = 3456 (576 thumbnail + 2880 sampled)
    Set to stay below LLM context length and below SF-LLaVA's 3680; token-efficiency ablation in Fig. 4d.
  • Thumbnail layout = 2 columns x 3 rows
    Sets each frame to 112x168 px; stated in supplementary Sec. 6 and ablated indirectly through NT.
assumptions (4)
  • domain assumption A frozen image LLM (LLaVA-v1.6) can perform video understanding without any video-specific training.
    Core premise of training-free video LLMs; stated in Sec. 1 and acknowledged in limitations (Sec. 4.4, Sec. 7.2 of appendix): action and moving-object tasks remain hard.
  • domain assumption Uniformly sampling visual tokens across all frames preserves enough spatiotemporal information when combined with a thumbnail.
    The method's effectiveness rests on this; supported by ablations in Tables 1-2 but not by an independent theoretical or measurement argument.
  • domain assumption Results of baseline models taken from prior papers are directly comparable to the authors' runs.
    Tables 3-5 compare against GPT-4V, Video-LLaMA2, and PLLaVA using numbers from official repositories or papers without re-running under identical prompts and frame sampling.
  • domain assumption Multiple-choice VideoQA accuracy is a reliable proxy for video understanding quality.
    The paper restricts headline results to multiple-choice benchmarks because GPT-assisted open-ended evaluation is shown to be noisy (appendix Sec. 7.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models." pith.science (2026). https://pith.science/paper/SSV4M5DL

@misc{pith2026241111066,
  author       = {Pith},
  title        = {Pith review of: TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SSV4M5DL}},
  note         = {Machine review of arXiv:2411.11066}
}
read the original abstract

Recent advances in multimodal Large Language Models (LLMs) have shown great success in understanding multi-modal contents. For video understanding tasks, training-based video LLMs are difficult to build due to the scarcity of high-quality, curated video-text paired data. In contrast, paired image-text data are much easier to obtain, and there is substantial similarity between images and videos. Consequently, extending image LLMs for video understanding tasks presents an appealing alternative. Developing effective strategies for compressing visual tokens from multiple frames is a promising way to leverage the powerful pre-trained image LLM. In this work, we explore the limitations of the existing compression strategies for building a training-free video LLM. The findings lead to our method TS-LLaVA, which constructs visual tokens through a Thumbnail-and-Sampling strategy. Given a video, we select few equidistant frames from all input frames to construct a Thumbnail image as a detailed visual cue, complemented by Sampled visual tokens from all input frames. Our method establishes the new state-of-the-art performance among training-free video LLMs on various benchmarks. Notably, our 34B model outperforms GPT-4V on the MVBench benchmark, and achieves performance comparable to the 72B training-based video LLM, Video-LLaMA2, on the challenging MLVU benchmark. Code is available at https://github.com/tingyu215/TS-LLaVA.

Figures

Figures reproduced from arXiv: 2411.11066 by the authors.

Figure 1
Figure 1. Illustration of training-free video LLM. Vision Tower: [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Visual token compression strategies illustrated. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of our TS-LLaVA. The vision tower includes vision encoder and projection module in image LLM. The dashed lines [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Design choices of TS-LLaVA. In (a), (b) (c), IntentQA shows similar pattern as NExT-QA, please refer to the Appendix. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Results from different ways of positioning visual tokens. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 25 citations worldwide. Full citation record

  1. IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A new multi-shot video dataset and an instance-prompt video LLM report large gains, but the main benchmark is built by the same authors and the model is not released.

  2. TreeSoc: Tree-Structured Dynamic Reasoning and Tool Synergy for Soccer Video Understanding

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Dynamic DFS tree reasoning with adaptive tool routing yields SOTA SoccerBench scores (85.2/87.4/82.2%) and 74.16% on NExT-QA.

Reference graph

Works this paper leans on

56 extracted references · 26 canonical work pages · cited by 2 Pith papers

  1. [44]

    Pllava : Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava : Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 1, 3, 6, 7, 2

    arXiv 2024

  2. [1]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Day- iheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfe...

    arXiv 2023

  3. [2]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 1, 2

    arXiv 2023

  4. [3]

    Matryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. arXiv preprint arXiv:2405.17430, 2024. 6

    arXiv 2024

  5. [4]

    Collecting highly parallel data for paraphrase evaluation

    David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Lin- guistics: Human Language Technologies , pages 190–200, Portland, Oregon, USA, 2011. Association for Computa- tional Linguistics. 5, 1

    work page 2011

  6. [5]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video- llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video- llms. arXiv preprint arXiv:2406.07476, 2024. 2, 3, 6, 7

    arXiv 2024

  7. [6]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 1, 2, 5

    work page 2023

  8. [7]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty- seventh Conference on Neural Information Processing Sys- tems, 2023. 2

    2023

Show all 56 references
  1. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  2. [9]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 2, 7

    2019

  3. [10]

    Video-mme: The first- ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first- ever compr...

    arXiv

  4. [11]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. arXiv preprint arXiv:2403.19046, 2024. 2, 3, 7

    2024 arXiv

  5. [12]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guil- laume Lample, L ´elio Renard Lavaud, Lucile Saulnier, Mari...

    2024 arXiv

  6. [13]

    An image grid can be worth a video: Zero- shot video question answering using a vlm

    Wonkyun Kim, Changin Choi, Wonseok Lee, and Won- jong Rhee. An image grid can be worth a video: Zero- shot video question answering using a vlm. arXiv preprint arXiv:2403.18406, 2024. 1, 2, 3, 5, 6, 8

    2024 arXiv

  7. [14]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Pro- ceedings of the 40th International Conference on Machine Learning, pages 19730–19742. PMLR, 2023. 2

    2023

  8. [15]

    Inten- tqa: Context-aware video intent reasoning

    Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. Inten- tqa: Context-aware video intent reasoning. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11963–11974, 2023. 5

    2023

  9. [16]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2024. 2, 7

    2024 arXiv

  10. [17]

    Mvbench: A comprehensive multi- modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  11. [18]

    Tgif: A new dataset and benchmark on animated gif description

    Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 5, 1

    2016

  12. [19]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In Pro- ceedings of the 18th European Conference on Computer Vi- sion (ECCV), 2024. 1, 3, 7, 2

    2024

  13. [20]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1, 2, 6, 7 9

    2023 arXiv

  14. [21]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems, pages 34892–34916. Curran Associates, Inc., 2023. 1, 2

    2023

  15. [22]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, 2024. 1, 2, 8

    2024

  16. [23]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 3, 5

    2024

  17. [24]

    St-llm: Large language models are effective tem- poral learners

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective tem- poral learners. In Computer Vision – ECCV 2024, pages 1– 18, Cham, 2025. Springer Nature Switzerland. 7

    2024

  18. [25]

    Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens

    Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13151–1...

    2024

  19. [26]

    Video-ChatGPT: Towards detailed video un- derstanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChatGPT: Towards detailed video un- derstanding via large vision and language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages...

    2024

  20. [27]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. In Thirty-seventh Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. 5

    2023

  21. [28]

    Deepstack: Deeply stacking visual tokens is surprisingly simple and ef- fective for lmms

    Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zux- uan Wu, Jianfeng Gao, and Yu-Gang Jiang. Deepstack: Deeply stacking visual tokens is surprisingly simple and ef- fective for lmms. arXiv preprint arXiv:2406.04334, 2024. 6, 2

    2024 arXiv

  22. [29]

    Nous-hermes-2-yi-34b model card, 2023

    NousResearch. Nous-hermes-2-yi-34b model card, 2023. 5

    2023

  23. [30]

    Gpt-4v(ision) system card, 2023

    OpenAI. Gpt-4v(ision) system card, 2023. 2

    2023

  24. [31]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 2

    2023 arXiv

  25. [32]

    Introducing routing functions to vision-language parameter- efficient fine-tuning with low-rank bottlenecks

    Tingyu Qu, Tinne Tuytelaars, and Marie-Francine Moens. Introducing routing functions to vision-language parameter- efficient fine-tuning with low-rank bottlenecks. In Computer Vision – ECCV 2024, pages 291–308, Cham, 2025. Springer Nature Switzerland. 8

    2024

  26. [33]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  27. [34]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural In- formation Processing Systems, 2023. 2

    2023

  28. [35]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Con...

    2024

  29. [36]

    Moviechat+: Question-aware sparse memory for long video question answering

    Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Moviechat+: Question-aware sparse memory for long video question answering. arXiv preprint arXiv:2404.17176, 2024. 6, 2

    2024 arXiv

  30. [37]

    Llama: Open and efficient foundation lan- guage models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aure- lien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. Llama: Open and efficient foundation la...

    2023 arXiv

  31. [38]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung- Levy. Videoagent: Long-form video understanding with large language model as agent. In Proceedings of the 18th European Conference on Computer Vision (ECCV), 2024. 6

    2024

  32. [39]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. arxiv, 2024. 6

    2024

  33. [40]

    Freeva: Offline mllm as training-free video assistant

    Wenhao Wu. Freeva: Offline mllm as training-free video assistant. arXiv preprint arXiv:2405.07798, 2024. 1, 2, 3

    2024 arXiv

  34. [41]

    Audiovisual slowfast networks for video recognition

    Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740 ,

    2001 arXiv

  35. [42]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 9777–9786, 2021. 5

    2021

  36. [43]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 5, 1

    2016

  37. [45]

    Slowfast-llava: A strong training-free base- line for video large language models

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free base- line for video large language models. arXiv preprint arXiv:2407.15841, 2024. 1, 2, 3, 5, 6, 7

    2024 arXiv

  38. [46]

    10 mplug-owl: Modularization empowers large language mod- els with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Jun- feng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 10 mplug-owl: Modularization empowers large language mod...

    arXiv

  39. [47]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, pages 9127–9134, 2019. 5, 1

    2019

  40. [48]

    A sim- ple llm framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A sim- ple llm framework for long-range video question-answering. arXiv preprint arXiv:2312.17235, 2024. 6

    2024 arXiv

  41. [49]

    Video-LLaMA: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video un- derstanding. In Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations, pages 543–553, Singapore, 2023. Ass...

    2023

  42. [50]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 3

    2024 arXiv

  43. [51]

    LLaMA-adapter: Efficient fine-tuning of large language models with zero- initialized attention

    Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. LLaMA-adapter: Efficient fine-tuning of large language models with zero- initialized attention. In The Twelfth International Conference on Learning Representations, 2024. 1

    2024

  44. [52]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 2, 7

    2024

  45. [53]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264,

    arXiv

  46. [54]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2 11 TS-LLaV A: Constructing Visual Tokens through Thumbnail-and-Sampling for T...

    2023 arXiv

  47. [55]

    Each frame is resized accordingly to fit within the resulting 336 ×336 thumbnail image

    Additional Implementation Details To construct the thumbnail image, we arrange the selected frames in a 2-column by 3-row grid. Each frame is resized accordingly to fit within the resulting 336 ×336 thumbnail image. MVBench Details We report the detailed classification of each...

  48. [56]

    We start with additional experiments conducted for the study on compression strategies

    Additional Results We report additional experimental results in this section. We start with additional experiments conducted for the study on compression strategies. Then we report the re- sults on Open-Ended VideoQA and video-based Text Gen- eration. Finally, we conclude this...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.