Pith. sign in

REVIEW 4 major objections 5 minor 49 references

Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DYTO shows that a frozen image-based multimodal model, with dynamic frame clustering and bipartite token merging, can match or surpass video-fine-tuned models on zero-shot video question answering.

desk verdict A sensible training-free recipe, but the paper's own tables contradict its SOTA claim and contain conflicting numbers; needs major revision before it can be trusted. read the letter →

arxiv 2411.14401 v2 pith:OQUHLTYZ submitted 2024-11-21 cs.CV cs.LG

Yiming Zhang , Zhuokai Zhao , Zhaorun Chen , Zenghui Ding , Xianjun Yang , Yining Sun This is my paper · ORCID
classification cs.CVcs.LG
keywords zero-shotvideounderstandingmultimodallargelanguagemodelstokenmerginghierarchicalclusteringkeyframeselectionvisualcompressiontraining-freequestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DYTO is a training-free framework that performs zero-shot video question answering by dynamically selecting which frames to keep and how many visual tokens each selected frame contributes to the language model. The paper argues that fixed uniform sampling and pooled token averaging, used by prior training-free methods, lose both event-level and spatial detail; instead DYTO segments the sampled frames into event clusters using per-frame [CLS] tokens with temporal weighting, then compresses the visual tokens of each cluster to a fixed total budget through bipartite token merging. Evaluated on ten video QA benchmarks, DYTO reports accuracy gains over the strongest training-free baselines and, on several benchmarks, higher accuracy than models fine-tuned on video data. The claim matters because it implies that a frozen image-based MLLM, with clever input selection and token compression, can approach or exceed video-specific training, and that the deciding factor is how tokens are allocated, not additional supervision.

What carries the argument

The machinery is a two-stage dynamic token allocation. Stage one computes a temporally weighted frame-distance matrix $W_t(i,j)=(1-\langle v_i,v_j\rangle)\cdot|t_i-t_j|/N$ from L2-normalized [CLS] tokens, thresholds it into a 1-NN graph, takes connected components as candidate clusters, recursively merges clusters by feature-average similarity, and selects the second-largest clustering as the final segmentation, so the number of keyframes K adapts to the video's event structure. Stage two, inspired by token merging (ToMe), partitions each sampled frame's patch tokens into two sets, scores token pairs by head-averaged cosine similarity, and greedily pairs and pools the most similar tokens, leaving $R_0 - Z/K$ tokens per frame so that the total output to the language model is exactly $Z$ tokens. The dynamic per-cluster budget is the load-bearing device: videos with few distinct events keep more tokens per frame, while videos with many events trade per-frame detail for temporal coverage.

What would settle it

Take a video with two visually distinct but semantically one continuous event and compare DYTO's accuracy against a uniform-sampling baseline on a QA task about that event; if the clustering splits the video into two clusters and the token budget is halved per frame, accuracy on the continuous event should drop below the uniform baseline, which would show that the dynamic clustering does not always help. Alternatively, replace the second-largest rule by an oracle that tries all cluster counts on a benchmark such as VideoMME; if the oracle does not beat the fixed rule, the heuristic is not the driver of the gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that zero-shot video understanding improves when token allocation is dynamic rather than uniform. Using the [CLS] token of each frame and a temporally weighted distance that multiplies cosine dissimilarity by relative timestamp distance, DYTO builds a 1-NN graph whose connected components segment the video into K event clusters; it then samples representative frames from each cluster and runs a bipartite token-merging step that keeps $R_0 - Z/K$ tokens per frame, holding the total number of visual tokens at a constant $Z$. The method is completely training-free and model-agnostic, and on NExTQA, EgoSchema, IntentQA, VideoMME, and MVBench the paper reports that DYTO systematically beats the training-free baselines IG-VLM and SlowFast-LLaVA and at 34B scale also exceeds several video-fine-tuned models, with additional gains on open-ended MSVD-QA, MSRVTT-QA, TGIF-QA, ANet-QA, and Video-ChatGPT benchmarks.

Load-bearing premise

The method's gains rest on the assumption that a frame's [CLS] token, together with temporally weighted cosine distance and the choice of the second-largest connected-component clustering, reliably divides a video into the events that matter for the question being answered.

Editorial extensions

If this is right

  • Zero-shot video understanding can be improved through input selection and token compression alone, removing the need for video-level fine-tuning data and computation.
  • Sampling a larger number of frames (100 or more) and compressing them to a fixed token budget helps long-video tasks, because the clustering selects frames that cover distinct events instead of spreading tokens uniformly.
  • The accuracy gap between training-free and video-fine-tuned models narrows: at 34B scale DYTO outperforms several fine-tuned models on structured video QA benchmarks.
  • The method transfers across different image-based MLLM backbones and scales with model size, with larger backbones showing larger gains when combined with DYTO.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'second-largest clustering' rule is a heuristic; DYTO does not derive it from the downstream task, and a cluster-count selection criterion could potentially improve segmentation further.
  • The token-budgeting principle generalizes: any video LLM could allocate its context window proportionally to event complexity, which may also help streaming or real-time settings with hard token limits.
  • Most open-ended gains are reported via GPT-assisted quality scores; human evaluation of temporal consistency would test whether the qualitative event-coverage advantage translates to perceived quality.
  • Because DYTO is a front-end only, it could in principle be stacked on top of video-fine-tuned models to improve their input selection as well, which the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DYTO, a training-free framework for zero-shot video understanding built on image-based multimodal large language models. It first uniformly samples N=100 frames, uses per-frame [CLS] tokens with a temporally weighted distance (Eq. 1) to build a 1-NN graph (Eq. 2), recursively merges clusters, and selects the 'second-largest' clustering as the event segmentation. It then applies a bipartite token merging scheme (inspired by ToMe) to compress each cluster's visual tokens to a fixed budget Z, with a merge ratio r=Z/K. The method is evaluated on structured VQA (NExTQA, EgoSchema, IntentQA, VideoMME, MVBench) and open-ended VQA (MSVD-QA, MSRVTT-QA, TGIF-QA, ANet-QA, Video-ChatGPT) benchmarks across several base model families and sizes, with the central claim that DYTO achieves state-of-the-art zero-shot performance, even surpassing video-fine-tuned models.

Significance. If the empirical claims were reliable, DYTO would be a useful contribution: it is a training-free, conceptually simple method that adaptively selects frames and compresses tokens, with code available and fixed hyperparameters reported. The idea of using hierarchical clustering for keyframe selection and bipartite token merging is plausible and could inform future work on efficient video MLLM inference. However, the paper's headline claim of superiority over fine-tuned models is contradicted by its own tables, and the reported numbers are internally inconsistent across duplicate rows. These issues prevent the paper from making a trustworthy case for its main contribution.

major comments (4)
  1. [Abstract and Section 4.3, Table 1] The abstract and Section 4.3 claim that DYTO achieves 'superior performance compared to both fine-tuned and training-free methods' and 'outperforms all the training-free approaches as well as all the fine-tuned models across all benchmarks.' This claim is directly contradicted by Table 1: the fine-tuned LLaVA-OneVision-7B scores 79.4 on NExTQA, while DYTO-7B scores 65.7; similarly, in Table 2 the training-free SlowFast-34B scores 67.1 on MSRVTT-QA, while DYTO-34B scores 66.2. The headline statement is therefore false by the paper's own reported numbers, and the manuscript should either be substantially revised to a more modest claim or the experiments must be re-run and reported accurately.
  2. [Tables 1 and 3] The experimental record is internally inconsistent. DYTO-7B is reported to score 41.2 on VideoMME in Table 1 but 42.7 in Table 3 for the same configuration (LLaVA-NeXT-image-7B, dynamic, Z=3680). Likewise, DYTO-34B scores 67.5 on IntentQA in Table 1 but 67.3 in Table 3. These are not different evaluation settings or ablations; they are duplicate rows with conflicting values. This inconsistency means the head-to-head comparisons supporting the state-of-the-art claim cannot be taken at face value, and the authors must reconcile the numbers or provide the raw evaluation logs to establish which values are correct.
  3. [Section 3.1, Eq. (2) and Table 3] The central design choice of selecting the 'second-largest' clustering result as the final segmentation is asserted without justification or sensitivity analysis. The ablation in Table 3 shows that removing the clustering module ('w/o Clustering') yields nearly identical performance to the full DYTO model, and even slightly better on NExTQA-7B (65.6 vs 65.7). This suggests that the claimed benefit of the hierarchical clustering is not clearly demonstrated, and the paper should provide an analysis of how the choice of clustering selection affects downstream accuracy, or temper the claim that dynamic keyframe selection is the source of the reported gains.
  4. [Section 4.3 and Tables 1-2] The comparison with fine-tuned models is not apples-to-apples and the paper does not disclose the computational cost of the fine-tuned baselines in the same setup. While the authors compare against methods with different frame counts and base models, the abstract's blanket statement that DYTO 'outperforms both fine-tuned and training-free methods' goes beyond what the table actually shows. Even where DYTO beats a fine-tuned model, the margins are often small and not accompanied by variance or significance testing, so the claim of a 'new state-of-the-art' is not statistically grounded.
minor comments (5)
  1. [Section 5.1 title] The title 'Scalabity on Base Model Size' contains a typo; it should be 'Scalability on Base Model Size'.
  2. [Table 2] The table header contains 'LLaVA-NeXT-iamge' instead of 'LLaVA-NeXT-image', and several entries are missing spaces between scores (e.g., '3.482.96' and '3.842.70').
  3. [Section 5.3] The text contains an incomplete word: 'verall, DyTo consistently outperforms' should be 'Overall, DYTO consistently outperforms'.
  4. [Throughout] The method name appears inconsistently as 'DYTO', 'DyTo', and 'D YTO' across the paper; please standardize.
  5. [Figure 2] The text in the figure includes an artifact '1st-IN' and a partially merged word 'NIteration'; these should be cleaned up to make the pipeline diagram interpretable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DYTO's reported gains are empirical comparisons on external benchmarks with fixed hyperparameters, and no prediction reduces to a fitted quantity or to a load-bearing self-citation.

full rationale

DYTO's method is a training-free inference procedure with fixed design choices: N=100 uniformly sampled frames, CLS-token distances in Eq. (1), a 1-NN graph in Eq. (2), hierarchical clustering with the second-largest segmentation, and bipartite token merging to a fixed budget Z. The token budget Z is set by hardware constraints (3680 or 7200), and r=Z/K is a bookkeeping identity that enforces the output length; it is not a parameter fitted to the benchmark answers or to the claimed accuracy numbers. The paper does not derive its SOTA result from an equation whose output equals its input; the SOTA claim rests on externally evaluated benchmark comparisons. The self-citations in the reference list, such as HALC [5] and RankCLIP [47], appear only as ordinary examples of MLLM-related methods and are not used to justify the clustering or merging design, nor does the paper import any uniqueness theorem from the authors. The noted Table 1/Table 3 discrepancies (VideoMME 41.2 vs. 42.7; IntentQA 67.5 vs. 67.3) are experimental reproducibility concerns, but an inconsistent reported number is not the same as a derivation that reduces to its own inputs. No circular step is present, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method contributes a heuristic integration of two existing techniques (Sarfraz clustering and ToMe merging) plus several hand-chosen hyperparameters (N, Z, the second-largest rule, RoPE scaling). It does not fit parameters to the test benchmarks, so circularity burden is low, but the design choices are under-justified.

free parameters (5)
  • Sampled frame count N = 100
    Set in Section 4.2: 'we uniformly sample each video to N=100 frames'. Chosen by hand; performance varies with N as shown in Figure 3.
  • Visual token budget Z = 3680 (7B) / 7200 (34B)
    Set in Section 4.2: 'we set the visual token sequence length Z to either 3680 or 7200, corresponding to model sizes of 7B and 34B'. Chosen due to hardware limits.
  • Merge ratio r = r = R0 - Z/K (dynamic per video)
    Defined in Section 3.2; depends on the cluster count K, which is itself determined by the clustering heuristic and not independently validated.
  • RoPE scaling factor = 2
    Appendix B: 'apply a scaling factor of 2, extending the original context length from 4096 to 8192 tokens.' Chosen to fit longer token sequences.
  • Final clustering selection rule = second-largest clustering result
    Section 3.1: 'we identify multiple clustering results and select the second-largest one as the final segmentation'. Presented without justification or sensitivity analysis.
assumptions (5)
  • domain assumption The CLS token of each frame's visual encoder is a sufficient frame-level semantic summary for clustering.
    Invoked in Section 3.1 to define the compressed representation v and the distance W_t; no validation that CLS tokens are discriminative across event boundaries.
  • domain assumption The temporally weighted cosine distance and 1-NN graph produce clusters that correspond to video events.
    Assumed in Section 3.1; used to partition frames into K clusters. Qualitative figures are given, but no quantitative validation against event annotations.
  • ad hoc to paper Recursive merging with selection of the 'second-largest' clustering yields the correct event segmentation.
    Section 3.1; the rule is stated without a principled justification or ablation across alternative selections.
  • domain assumption Bipartite token merging preserves the semantic information needed for VQA after compression to Z tokens.
    Adopted in Section 3.2 from ToMe; DYTO relies on this to avoid semantic loss, but the fixed budget Z may discard task-relevant details.
  • domain assumption Image-based MLLMs with RoPE scaling to 8192 tokens can process the compressed video representation without degradation.
    Required by the experimental setup in Section 4.2 and Appendix B; not separately verified across all backbones.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding." pith.science (2026). https://pith.science/paper/OQUHLTYZ

@misc{pith2026241114401,
  author       = {Pith},
  title        = {Pith review of: Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OQUHLTYZ}},
  note         = {Machine review of arXiv:2411.14401}
}
read the original abstract

Recent advancements in multimodal large language models (MLLMs) have opened new avenues for video understanding. However, achieving high fidelity in zero-shot video tasks remains challenging. Traditional video processing methods rely heavily on fine-tuning to capture nuanced spatial-temporal details, which incurs significant data and computation costs. In contrast, training-free approaches, though efficient, often lack robustness in preserving context-rich features across complex video content. To this end, we propose DYTO, a novel dynamic token merging framework for zero-shot video understanding that adaptively optimizes token efficiency while preserving crucial scene details. DYTO integrates a hierarchical frame selection and a bipartite token merging strategy to dynamically cluster key frames and selectively compress token sequences, striking a balance between computational efficiency with semantic richness. Extensive experiments across multiple benchmarks demonstrate the effectiveness of DYTO, achieving superior performance compared to both fine-tuned and training-free methods and setting a new state-of-the-art for zero-shot video understanding.

Figures

Figures reproduced from arXiv: 2411.14401 by the authors.

Figure 1
Figure 1. Comparison with two SOTA training-free video-based LLM decoding methods over three different model backbones across five [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of DYTO, a training-free model built upon image-based MLLM without any fine-tuning. Specifically, DYTO first divides the video into K clusters using the [CLS] token (pink block). Then the dynamic bipartite merging module samples frames from each cluster and controls the final output length as Z, resulting in better balance between computational efficiency and semantic richness. ing (VQA) capabilities of… view at source ↗
Figure 3
Figure 3. Top: Performance comparison of baseline method under various video lengths. Bottom: Effect of different input sampling lengths under various video lengths. increases, DYTO’s accuracy on longer videos notably im￾proves, demonstrating its capacity to capture detailed tem￾poral and contextual information more effectively than competing methods. This improvement highlights the ad￾vantage of DYTO’s hierarchical clusterin… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The sampling method and clustering module output visualization on a video. Our method offers more comprehensive video [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Example from NExTQA benchmark. red part denotes the incorrect or hallucination content. green part denotes the cor￾rect object content. DYTO demonstrates significantly better per￾formance compared to the other two methods. holding a cup of cereal respectively). Unlike …
Figure 7
Figure 7. Figure 7: Results of merging on video frames.From top to bottom is original images,after pooling, patchfied, and the proposed bipartite [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Example from MVBench benchmark. red part denotes the incorrect or hallucination content. green part denotes the correct object content. 3 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Example from Egoschema benchmark. green part denotes the correct object content. blue part denotes the missing content in IG-VLM and SlowFast-LLaVA answers. DYTO can effectively capture the events in the video. 4 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 33 canonical work pages

  1. [1]

    Is space-time attention all you need for video understanding? In ICML, page 4, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, page 4, 2021. 1

    work page 2021

  2. [2]

    Token merging: Your ViT but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023. 3

    2023

  3. [3]

    Collecting highly parallel data for paraphrase evaluation

    David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Lin- guistics: Human Language Technologies , pages 190–200, Portland, Oregon, USA, 2011. Association for Computa- tional Linguistics. 4, 5

    work page 2011

  4. [4]

    Fewer tokens and fewer videos: Extending video understanding abilities in large vision-language mod- els

    Shimin Chen, Yitian Yuan, Shaoxiang Chen, Zequn Jie, and Lin Ma. Fewer tokens and fewer videos: Extending video understanding abilities in large vision-language mod- els. arXiv preprint arXiv:2406.08024, 2024. 1

    arXiv 2024

  5. [5]

    Halc: Object hallucination re- duction via adaptive focal-contrast decoding

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination re- duction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425, 2024. 1

    arXiv 2024

  6. [6]

    VideoLLaMA 2: Advanc- ing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. VideoLLaMA 2: Advanc- ing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs, 2024. 1, 2, 5, 6

    work page 2024

  7. [7]

    Video-MME: The First- Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis, 2024

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-MME: The First- Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis, 2024. 4, 5, 6

    work page 2024

  8. [8]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. arXiv preprint arXiv:2410.05160, 2024. 1

    arXiv 2024

Show all 49 references
  1. [9]

    Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding, 2024

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding, 2024. 1, 2

    2024

  2. [10]

    Evaluating open-domain question answer- ing in the era of large language models

    Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiei. Evaluating open-domain question answer- ing in the era of large language models. arXiv preprint arXiv:2305.06984, 2023. 1

    2023 arXiv

  3. [11]

    An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM, 2024

    Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee. An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM, 2024. 1, 2, 3, 5, 6, 7

    2024

  4. [12]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 5

    2024 arXiv

  5. [13]

    Inten- tqa: Context-aware video intent reasoning

    Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. Inten- tqa: Context-aware video intent reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 11963–11974, 2023. 4, 5

    2023

  6. [14]

    Uniformer: Unified transformer for efficient spatiotemporal representation learning, 2022

    Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning, 2022. 3

    2022

  7. [15]

    VideoChat: Chat-Centric Video Understanding, 2024

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. VideoChat: Chat-Centric Video Understanding, 2024. 1, 2

    2024

  8. [16]

    MVBench: A Comprehensive Multi- modal Video Understanding Benchmark, 2024

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. MVBench: A Comprehensive Multi- modal Video Understanding Benchmark, 2024. 4, 5, 6

    2024

  9. [17]

    Tgif: A new dataset and benchmark on animated gif description

    Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4641–4650, 2016. 4, 6

    2016

  10. [18]

    LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models, 2023

    Yanwei Li, Chengyao Wang, and Jiaya Jia. LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models, 2023. 1, 2

    2023

  11. [19]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. 2024. 5, 6

    2024

  12. [20]

    Video-LLaV A: Learning United Visual Representation by Alignment Before Projection, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaV A: Learning United Visual Representation by Alignment Before Projection, 2023. 1, 2, 5, 6

    2023

  13. [21]

    Tsm: Temporal shift module for efficient video understanding

    Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7083–7093, 2019. 1

    2019

  14. [22]

    Video swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202–3211, 2022. 1

    2022

  15. [23]

    Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens, 2023

    Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens, 2023. 1, 2, 6

    2023

  16. [24]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 1, 4, 6

    2023 arXiv

  17. [25]

    Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Mod- els, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Mod- els, 2023. 6

    2023

  18. [26]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024. 1, 2, 6

    2024

  19. [27]

    Foundation mod- els for video understanding: A survey

    Neelu Madan, Andreas Møgelmose, Rajat Modi, Yogesh S Rawat, and Thomas B Moeslund. Foundation mod- els for video understanding: A survey. arXiv preprint arXiv:2405.03770, 2024. 1

    2024 arXiv

  20. [28]

    Egoschema: A diagnostic benchmark for very long- 9 form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- 9 form video language understanding. Advances in Neural In- formation Processing Systems, 36:46212–46244, 2023. 4, 5, 6

    2023

  21. [29]

    Mm1: Methods, analysis & insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024. 1

    2024 arXiv

  22. [30]

    Less is more: Pay less attention in vision transform- ers

    Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jian- fei Cai. Less is more: Pay less attention in vision transform- ers. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2035–2043, 2022. 3

    2022

  23. [31]

    Effi- cient parameter-free clustering using first neighbor relations

    Saquib Sarfraz, Vivek Sharma, and Rainer Stiefelhagen. Effi- cient parameter-free clustering using first neighbor relations. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 8934–8943, 2019. 5

    2019

  24. [32]

    MovieChat: From Dense Token to Sparse Memory for Long Video Understanding, 2024

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. MovieChat: From Dense Token to Sparse Memory for Long Video Understanding, 2024. 2, 6

    2024

  25. [33]

    Video understanding with large language models: A survey

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. Video understanding with large language models: A survey. arXiv preprint arXiv:2312.17432, 2023. 1

    2023

  26. [34]

    Internvideo2: Scaling foundation models for mul- timodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for mul- timodal video understanding. In European Conference on Computer Vision, pages 396–416. Springer, 2024. 5

    2024

  27. [35]

    Freeva: Offline mllm as training-free video assistant

    Wenhao Wu. Freeva: Offline mllm as training-free video assistant. 2024. 1, 2, 3, 4

    2024

  28. [36]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9777–9786, 2021. 4, 5, 6

    2021

  29. [37]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5288–5296, 2016. 4, 6

    2016

  30. [38]

    PLLaV A : Parameter-free LLaV A Extension from Images to Videos for Video Dense Captioning, 2024

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. PLLaV A : Parameter-free LLaV A Extension from Images to Videos for Video Dense Captioning, 2024. 2, 6

    2024

  31. [39]

    SlowFast-LLaV A: A Strong Training-Free Baseline for Video Large Language Models, 2024

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin De- hghan. SlowFast-LLaV A: A Strong Training-Free Baseline for Video Large Language Models, 2024. 1, 2, 3, 5, 6, 7

    2024

  32. [40]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36:76749–76771, 2023. 5

    2023

  33. [41]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36, 2024. 1, 2

    2024

  34. [42]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yuet- ing Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 9127–9134, 2019. 4, 6

    2019

  35. [43]

    Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding, 2023. 1, 2

    2023

  36. [44]

    Ferret- v2: An improved baseline for referring and grounding with large language models

    Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret- v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07973 ,

    arXiv

  37. [45]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 5, 6

    2024

  38. [46]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 2

    2024

  39. [47]

    Rankclip: Ranking- consistent language-image pretraining

    Yiming Zhang, Zhuokai Zhao, Zhaorun Chen, Zhili Feng, Zenghui Ding, and Yining Sun. Rankclip: Ranking- consistent language-image pretraining. arXiv preprint arXiv:2404.09387, 2024. 1

    2024 arXiv

  40. [48]

    Multimodal guidance network for missing- modality inference in content moderation

    Zhuokai Zhao, Harish Palani, Tianyi Liu, Lena Evans, and Ruth Toner. Multimodal guidance network for missing- modality inference in content moderation. In 2024 IEEE In- ternational Conference on Multimedia and Expo Workshops (ICMEW), pages 1–4. IEEE, 2024. 1

    2024

  41. [49]

    A survey on gen- erative ai and llm for video generation, understanding, and streaming

    Pengyuan Zhou, Lin Wang, Zhi Liu, Yanbin Hao, Pan Hui, Sasu Tarkoma, and Jussi Kangasharju. A survey on gen- erative ai and llm for video generation, understanding, and streaming. arXiv preprint arXiv:2404.16038, 2024. 1 10 A. Time Consumption Experiment we conducted the exper...

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.