Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that merging video tokens along a quadtree hierarchy plus temporal chains cuts video-LLM prefill cost 2-3x while keeping accuracy nearly intact.

desk verdict STTM is a genuinely useful training-free token-merging method for video LLMs with strong empirical results, but its core similarity-to-redundancy assumption has a real gap and the evaluation needs error bars and code. read the letter →

arxiv 2507.07990 v1 pith:QS225IU3 submitted 2025-07-10 cs.CV cs.AI

Jeongseok Hyun , Sukjun Hwang , Su Ho Han , Taeoh Kim , Inwoong Lee , Dongyoon Wee , Joon-Young Lee , Seon Joo Kim
show 1 more author
Minho Shim
This is my paper
classification cs.CVcs.AI
keywords videolargelanguagemodelstokenmergingtraining-freeaccelerationspatio-temporalredundancyquadtreeKVcachereusequestionansweringinferencespeed-up
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video large language models are slow because every frame contributes many visual tokens and attention cost grows quadratically. The paper proposes STTM, a training-free plug-in that merges redundant tokens before the LLM's heavy attention layers: it first merges spatially similar tokens within each frame using a coarse-to-fine quadtree search, then merges similar tokens across consecutive frames into their earliest occurrence. The claim is that this two-step, query-agnostic merging preserves the information needed for video question answering while cutting the token count roughly in half or to a third. If correct, STTM would make long-video prefill 2-3x cheaper without retraining, and because the reduction does not depend on the question, the same key-value cache can be reused for multiple queries on one video.

What carries the argument

The carrying object is a quadtree-structured, multi-granular token hierarchy built per frame. At each level a parent token is compared by cosine similarity with its four 2x2 child tokens; if all four similarities exceed the spatial threshold tau_S, the parent represents the region and the children are pruned, otherwise the search descends to a finer level. After spatial merging, temporal merging compares tokens that spatially overlap between consecutive frames, links pairs whose similarity exceeds tau_T, and computes connected components with a vectorized union-find algorithm so each chain of similar tokens merges into the earliest frame. The method is applied once in an early transformer layer, before the heavy attention computation, and reorders the surviving tokens by spatial Z-scan and temporal order.

What would settle it

Run STTM at a 30% token budget on a synthetic video whose frames are identical except for a small answer-carrying patch, and ask the question that depends on that patch; if cosine similarity in the patch stays above the merging thresholds yet accuracy collapses, the similarity-to-redundancy mapping fails.

Watch

Extended reading notes

Core claim

The central claim is that local spatio-temporal redundancy in video is sufficient to support aggressive, training-free token reduction in video LLMs. STTM represents each frame at multiple spatial granularities, retaining coarse tokens only where all four child tokens are sufficiently similar to their parent, and subdividing where details differ; then it chains overlapping tokens across frames into directed graphs and merges each chain into the earliest token. Across six video QA benchmarks the paper reports a 2x speed-up with a 0.5% relative accuracy drop under a 50% token budget, a 3x speed-up with about a 2% drop under a 30% token budget, and better or comparable results than query-aware and query-agnostic reduction baselines, including on needle-in-a-haystack tasks requiring fine-grained detail. The same merged-token sequence is independent of the user's question, so the key-value cache can be reused across questions for the same video.

Load-bearing premise

The load-bearing premise is that high cosine similarity between a quadtree parent token and its four child tokens, and between spatially overlapping tokens across consecutive frames, reliably indicates information redundancy that can be removed without hurting downstream question answering.

Editorial extensions

If this is right

  • At a 50% token budget, STTM reports a 2x reduction in time-to-first-token with only a 0.5% average relative accuracy drop across the six benchmarks.
  • At a 30% token budget, it reports a 3x speed-up with roughly a 2% drop, and on the needle-in-a-haystack benchmark the drop is about 2% while baseline query-agnostic methods lose around 18%.
  • Because merging is query-agnostic, video key-value states computed once can be reused for later questions, eliminating repeated prefill computation in multi-turn and multi-query deployment.
  • The method transfers across different video LLM families, including a 72B model, and at times improves accuracy while using fewer tokens than the uncompressed video.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves threshold selection manual; an automatic, budget-aware rule for tau_S and tau_T would remove the main tuning knob and is the most direct next step.
  • The same redundancy signal could in principle drive decoder-side key-value cache eviction after prefill, extending the speed-up to the generation phase, though STTM itself only reduces prefill tokens.
  • The top-left destination approximation for one-to-many temporal merging is a vectorization choice; replacing it with an approximate-nearest-neighbor merge could recover some fine-grained accuracy on dynamic scenes without losing parallelism.
  • Merging before attention is compatible with later query-aware pruning, but per-query pruning would forfeit the key-value cache reuse that is STTM's main deployment advantage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes STTM, a training-free, query-agnostic spatio-temporal token merging method for video LLMs. It first builds a multi-granular spatial representation per frame via a quadtree, pruning child tokens when all four are sufficiently similar to their averaged parent, and then merges tokens across frames by chaining spatially overlapping tokens into directed graphs that collapse into their earliest occurrence. The method is inserted at an early LLM layer, enabling KV-cache reuse across queries. The authors evaluate on six video QA benchmarks (VNBench, VideoMME, LongVideoBench, MLVU, EgoSchema, NExT-QA) with LLaVA-Video 7B, LLaVA-OneVision 7B, Qwen2VL 7B, and LLaVA-Video 72B, reporting that a 50% token budget yields about a 2x speed-up with a 0.5% relative accuracy drop and a 30% budget yields about a 3x speed-up with a 2% drop, outperforming several training-free baselines. Ablations cover spatial granularity, decomposed vs. joint merging, temporal destination choice, positional embedding handling, and merging layer position.

Significance. If the empirical results hold, STTM is a practically valuable contribution: it is training-free, query-agnostic, and directly compatible with KV-cache reuse, addressing a deployment-relevant bottleneck for long-video LLMs. The paper's strengths include evaluation across six benchmarks and four model families (including a 72B model), a clear decomposed spatial-plus-temporal design, and ablations that isolate the contributions of the quadtree, temporal chaining, positional embeddings, and layer position. The complexity analysis (O(HW) per frame spatially and O(THW) temporally) is also a plus. However, the significance is tempered by the absence of statistical repeatability (no error bars or multiple runs) and by the fact that the central merge criterion—parent-child cosine similarity—is heuristic, with thresholds manually tuned to hit token budgets. The claim that fine-grained details (e.g., VNBench needles) are preserved rests on aggregate benchmark averages, which could hide systematic over-merging on particular content types.

major comments (4)
  1. [§3.2, Fig. 3, Eq. (1)–(2)] The quadtree pruning criterion compares each child token to the averaged parent token. Because the parent is the arithmetic mean of the four children, high parent-child cosine similarity can occur even when the children are mutually dissimilar—for example, when a small task-relevant patch (a needle) lies in one quadrant over a large common background. The manuscript does not provide an argument or experiment showing that this criterion bounds the information lost by replacing children with their mean. The strong VNBench result is evidence on the tested distribution, but not evidence that a common-background-dominant video would not cause systematic over-merging. Please add a pairwise-merge ablation (merging only when all pairwise child-child similarities exceed a threshold) on VNBench, or a worst-case analysis, to show whether the quadtree criterion retains task-critical residuals that a pairwise criterion would retain.
  2. [§4.1, Tables 1–4 and Appendix Tables 11–14] No error bars, seed variance, or multiple runs are reported. The headline statement that STTM incurs only a 0.5% relative accuracy drop at a 50% budget corresponds to an average relative accuracy of 99.5 versus 100.0; without an estimate of run-to-run or item-level variance, this difference could be within noise on some benchmarks. Please report at least three runs with standard deviations for the main comparisons, or a paired bootstrap over benchmark items, to make the accuracy-preservation claim authoritative.
  3. [§4.1 and Conclusion] The paper states that the spatial and temporal thresholds (τS and τT) are 'empirically adjusted' to approximately meet each token budget. This means the two key hyperparameters are tuned per token budget and per benchmark family, and the conclusion explicitly acknowledges that performance depends on manually adjusted thresholds. The comparison with fixed-budget baselines is therefore not strictly apples-to-apples. Please report the exact threshold values used for each budget, show a sensitivity sweep around the chosen operating points, and include an ablation where a single threshold set is used across all budgets to assess how much accuracy depends on per-budget tuning.
  4. [§3.3 and Table 7] The temporal merging ablation in Table 7 compares 'Optimal' destination selection with the 'Top-left' approximation, but the procedure for the 'Optimal' baseline is not described. If 'Optimal' selects the most similar destination among all candidates using the same τT, the comparison is informative; if it uses a different similarity measure or threshold, the result is not a clean comparison. Please clarify the exact algorithm used for the 'Optimal' row, and report whether the vectorized union-find implementation changes the number of tokens or only the runtime.
minor comments (5)
  1. [§4.2] Typo: 'at the ipnut' should read 'at the input'.
  2. [Figure 1 caption] The caption uses 'NIAH' to refer to the needle-in-a-haystack dataset, but the text uses 'VNBench'; please define the abbreviation at first use in the caption.
  3. [Table 3 and Appendix Table 13] For Qwen2VL at the 50% budget, STTM reports NV above 50% on VideoMME (52.3%) and LongVideoBench (51.5%); the paper says thresholds are adjusted to 'approximately meet' budgets, but the table does not flag these entries. Please add a footnote or marker indicating which rows exceed the nominal budget.
  4. [§3.4 and Table 8] The text says that for Qwen2VL, which uses M-RoPE, reassignment is not applicable and the survival strategy is used instead; however, Table 8 shows reassignment is best for LLaVA. Please clarify whether the positional-embedding choice is model-dependent and whether the reported results for OneVision and Qwen2VL use the survival strategy throughout.
  5. [Title page] The affiliation 'NA VER Cloud' should be 'NAVER Cloud'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: STTM's accuracy and speed claims are measured against external benchmarks, and the merging criterion is an empirical design choice rather than a fitted prediction.

full rationale

The paper's central claims are empirical: STTM achieves certain relative accuracy and latency numbers on six external video QA benchmarks (VNBench, VideoMME, LongVideoBench, MLVU, EgoSchema, NExT-QA) under 50% and 30% token budgets. These numbers are measured after token reduction, not derived from the method's definition. The spatial and temporal merging rules in Sections 3.2 and 3.3 use cosine similarity thresholds (tau_S and tau_T) that are manually set to approximately meet a token budget, as stated in Section 4.1: 'Our merging threshold values (tau_S and tau_T) are empirically adjusted to approximately meet specific token budget for fair comparisons with other methods.' This is hyperparameter selection, not a fit of the reported accuracy values; the accuracy outcomes remain free empirical results that could in principle come out worse. The paper does not invoke a self-citation chain for any load-bearing premise: the quadtree structure is cited to standard external references (Finkel and Bentley; Samet), the VideoMAE redundancy motivation is cited to external work, and the union-find implementation is cited to Tarjan. There is no imported uniqueness theorem, no ansatz smuggled in via a same-author citation, and no renaming of a known result presented as a new unification. The skeptic's concern that high parent-child cosine similarity does not mathematically guarantee that merging loses no task-relevant information is a correctness and robustness critique, not a circularity: the paper's own ablation studies, including the VNBench results and the trade-off curves in Fig. 4, provide external evidence that the criterion preserves accuracy on the tested distribution. Whether that guarantee is strong enough for all content classes is an empirical risk, not a case of the derivation reducing to its own inputs. The paper is appropriately cautious in its conclusion, noting that 'its performance currently depends on manually adjusted threshold values.' Because every headline result is benchmark-measured and no equation in the paper is shown to contain its own conclusion by construction, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method rests on similarity-based redundancy assumptions that are validated only empirically. Thresholds and architectural choices (root scale, layer) are free parameters tuned to achieve target token budgets. No code or configs are released, so these choices are not independently verifiable.

free parameters (4)
  • spatial threshold tau_S = not reported
    Empirically adjusted to meet token budget; determines how aggressively spatial tokens are merged. Values not disclosed in the paper.
  • temporal threshold tau_T = not reported
    Empirically adjusted to meet token budget; controls cross-frame token merging. Values not disclosed.
  • quadtree root scale (Lv.1 scale) = 4x4
    Selected via ablation in Table 5; 4x4 outperforms 2x2 for accuracy retention.
  • merging layer position = layer 3 for 7B, layer 1 for 72B
    Selected via ablation in Table 9; balances accuracy and latency.
assumptions (4)
  • domain assumption Cosine similarity between a quadtree parent token (average of children) and its four child tokens is a reliable indicator of spatial redundancy.
    Used in Section 3.2 to decide whether to prune child nodes. If similarity is high, children are merged into the parent. This assumes local feature homogeneity implies redundancy for the downstream video LLM.
  • domain assumption Tokens representing the same spatial region across consecutive frames that are similar can be merged into the earlier token without loss of task-relevant information.
    Core temporal merging assumption in Section 3.3. Relies on temporal continuity of video content.
  • domain assumption The top-left token approximation for one-to-many temporal merging is a sufficient approximation to the most-similar destination.
    Used in Section 3.3 for vectorization; validated empirically in Table 7, but not derived.
  • domain assumption Reassigning RoPE based on the new token order preserves the model's positional semantics.
    Chosen in Section 3.4 via ablation (Table 8); valid for LLaVA models with 1D RoPE, not for Qwen2VL with M-RoPE.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs." pith.science (2026). https://pith.science/paper/QS225IU3

@misc{pith2026250707990,
  author       = {Pith},
  title        = {Pith review of: Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QS225IU3}},
  note         = {Machine review of arXiv:2507.07990}
}
abstract

Video large language models (LLMs) achieve strong video understanding by leveraging a large number of spatio-temporal tokens, but suffer from quadratic computational scaling with token count. To address this, we propose a training-free spatio-temporal token merging method, named STTM. Our key insight is to exploit local spatial and temporal redundancy in video data which has been overlooked in prior work. STTM first transforms each frame into multi-granular spatial tokens using a coarse-to-fine search over a quadtree structure, then performs directed pairwise merging across the temporal dimension. This decomposed merging approach outperforms existing token reduction methods across six video QA benchmarks. Notably, STTM achieves a 2$\times$ speed-up with only a 0.5% accuracy drop under a 50% token budget, and a 3$\times$ speed-up with just a 2% drop under a 30% budget. Moreover, STTM is query-agnostic, allowing KV cache reuse across different questions for the same video. The project page is available at https://www.jshyun.me/projects/sttm.

Figures

Figures reproduced from arXiv: 2507.07990 by the authors.

Figure 1
Figure 1. Comparison of training-free token reduction methods us [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (Left) Our spatio-temporal token merging method is a training-free, plug-and-play module that produces spatio-temporally multi￾granular tokens. (Middle) In step 1, tokens are merged based on spatial locality, where similar tokens within a 2D grid are combined into a single token. (Right) In step 2, spatially multi-granular tokens are further merged along the temporal dimension, where similar tokens across frames are… view at source ↗
Figure 3
Figure 3. A coarse-to-fine spatial search is performed using a quadtree structure. If all four fine child nodes exhibit high simi￾larity with the coarse parent node, the search process terminates, and the parent node is used to represent the corresponding region. Otherwise, the search continues until the finest level is reached. Here, the scale for each level is an example for illustration. connected across frames when they o… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Trade-off of accuracy and visual token retention ratio. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Visualization of spatial token merging results. Each image patch within a green box represents a single token. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualization of spatio-temporal token merging results on VideoMME. (a) The first eight consecutive frames are sampled. (b) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    CRAFT recursively merges video tokens with training-free similarity selection plus learnable gated fusion, retaining ~97% of average accuracy at 8x compression across six benchmarks.

Reference graph

Works this paper leans on

63 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1

    arXiv 2023

  2. [2]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In ICLR, 2023. 3, 5, 7

    work page 2023

  3. [3]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Jun- jie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069, 2024. 7

    arXiv 2024

  4. [4]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, pages 6299–6308, 2017. 2

    work page 2017

  5. [5]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In CVPR, pages 13817–13827, 2024. 2

    work page 2024

  6. [6]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In ECCV, pages 19–35. Springer, 2024. 2, 3, 5, 6, 7

    work page 2024

  7. [7]

    Longvila: Scaling long-context visual language models for long videos

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Yihui He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling long-context visual language models for long videos. In ICLR, 2025. 1, 3

    work page 2025

  8. [8]

    vid-tldr: Training free token merging for light-weight video transformer

    Joonmyung Choi, Sanghyeok Lee, Jaewon Chu, Minhyuk Choi, and Hyunwoo J Kim. vid-tldr: Training free token merging for light-weight video transformer. In CVPR, pages 18771–18781, 2024. 3

    work page 2024

Show all 63 references
  1. [9]

    FlashAttention-2: Faster attention with better par- allelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better par- allelism and work partitioning. In ICLR, 2024. 1, 2

    2024

  2. [10]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R´e

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher R´e. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In NeurIPS, 2022. 1, 2

    2022

  3. [11]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  4. [12]

    Masked autoencoders as spatiotemporal learners

    Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. NeurIPS, 35:35946–35958, 2022. 2

    2022

  5. [13]

    Quad trees a data structure for retrieval on composite keys

    Raphael A Finkel and Jon Louis Bentley. Quad trees a data structure for retrieval on composite keys. Acta informatica, 4:1–9, 1974. 2, 4

    1974

  6. [14]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In CVPR, pages 24108–24118, 2025. 5

    2025

  7. [15]

    Framefusion: Combining similarity and importance for video token reduction on large visual language models.arXiv preprint arXiv:2501.01986, 2024

    Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shen- gen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. Framefusion: Combining similarity and importance for video token reduction on large visual language models.arXiv preprint arXiv:2501.01986, 2024. 2, 3, 5

    2024 arXiv

  8. [16]

    Caching — google ai, 2024

    Google. Caching — google ai, 2024. Accessed: 2025-07-09. 1

    2024

  9. [17]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 1

    2024 arXiv

  10. [18]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016. 2

    2016

  11. [19]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000–16009, 2022. 2

    2022

  12. [20]

    Prunevid: Visual to- ken pruning for efficient video large language models

    Xiaohu Huang, Hao Zhou, and Kai Han. Prunevid: Visual to- ken pruning for efficient video large language models. arXiv preprint arXiv:2412.16117, 2024. 2, 3

    2024 arXiv

  13. [21]

    Chat-univi: Unified visual representation em- powers large language models with image and video under- standing

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video under- standing. In CVPR, pages 13700–13710, 2024. 3

    2024

  14. [22]

    Needle in a haystack – pressure testing llms

    G Kamradt. Needle in a haystack – pressure testing llms. https://github.com/gkamradt/LLMTest_ NeedleInAHaystack, 2023. 5

    2023

  15. [23]

    Handwritten digit recognition with a back- propagation network

    Yann LeCun, Bernhard Boser, John Denker, Donnie Hen- derson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back- propagation network. NeurIPS, 2, 1989. 2

    1989

  16. [24]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pages 12888–12900. PMLR, 2022. 2

    2022

  17. [25]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742. PMLR, 2023. 1, 2

    2023

  18. [26]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 1, 2

    2023 arXiv

  19. [27]

    Mvbench: A comprehensive multi- modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark. In CVPR, pages 22195–22206, 2024. 1

    2024

  20. [28]

    Video-llava: Learning united visual repre- sentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual repre- sentation by alignment before projection. In EMNLP, 2024. 2

    2024

  21. [29]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2023. 1

    2023

  22. [30]

    World model on million-length video and language with blockwise ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.08268 ,

    arXiv

  23. [31]

    Ring atten- tion with blockwise transformers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring atten- tion with blockwise transformers for near-infinite context. In ICLR, 2024. 1, 3 9

    2024

  24. [32]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. InACL,

  25. [33]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. NeurIPS, 36:46212– 46244, 2023. 5

    2023

  26. [34]

    Efficiently scaling transformer inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shiv- ani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. MLSys, 5:606–624, 2023. 1

    2023

  27. [35]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. InCVPR, pages 14313– 14323, 2024. 1, 2

    2024

  28. [36]

    xgen-mm-vid (blip-3- video): You only need 32 tokens to represent a video even in vlms

    Michael S Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles. xgen-mm-vid (blip-3- video): You only need 32 tokens to represent a video even in vlms. arXiv preprint arXiv:2410.16267, 2024. 2

    arXiv

  29. [37]

    The quadtree and related hierarchical data structures

    Hanan Samet. The quadtree and related hierarchical data structures. ACM CSUR, 16(2):187–260, 1984. 2, 4

    1984

  30. [38]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388,

  31. [39]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In CVPR, pages 18221–18232, 2024. 1, 2

    2024

  32. [40]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  33. [41]

    Efficient quadtree cod- ing of images and video

    Gary J Sullivan and Richard L Baker. Efficient quadtree cod- ing of images and video. TIP, 3(3):327–331, 1994. 2

    1994

  34. [42]

    Overview of the high efficiency video coding (hevc) standard

    Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (hevc) standard. TCSVT, 22(12):1649–1668, 2012. 2

    2012

  35. [43]

    Dycoke: Dynamic compression of tokens for fast video large language models

    Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. In CVPR, 2025. 2, 3, 5, 6, 7

    2025

  36. [44]

    Efficiency of a good but not linear set union algorithm

    Robert Endre Tarjan. Efficiency of a good but not linear set union algorithm. JACM, 22(2):215–225, 1975. 2, 5, 7

    1975

  37. [45]

    Gpt-4o system card

    OpenAI Team. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 1

    2024 arXiv

  38. [46]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. NeurIPS, 35:10078– 10093, 2022. 2

    2022

  39. [47]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1, 2

    2023 arXiv

  40. [48]

    Learning spatiotemporal features with 3d convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torre- sani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. InICCV, pages 4489–4497,

  41. [49]

    Look-m: Look- once optimization in kv cache for efficient multimodal long- context inference

    Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look- once optimization in kv cache for efficient multimodal long- context inference. arXiv preprint arXiv:2406.18139, 2024. 2, 3

    2024 arXiv

  42. [50]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1, 7

    2024 arXiv

  43. [51]

    Videollamb: Long-context video understanding with recur- rent memory bridges

    Yuxuan Wang, Cihang Xie, Yang Liu, and Zilong Zheng. Videollamb: Long-context video understanding with recur- rent memory bridges. arXiv preprint arXiv:2409.01071 ,

    arXiv

  44. [52]

    Sullivan, Gisle Bjontegaard, and Ajay Luthra

    Thomas Wiegand, Gary J. Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h.264/avc video coding stan- dard. TCSVT, 13(7):560–576, 2003. 2

    2003

  45. [53]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. NeurIPS, 37:28828–28857,

  46. [54]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In CVPR, pages 9777–9786, 2021. 5

    2021

  47. [55]

    Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy re- duction

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy re- duction. In CVPR, 2025. 3, 7

    2025

  48. [56]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 1

    2024 arXiv

  49. [57]

    Pyramidinfer: Pyramid kv cache com- pression for high-throughput llm inference

    Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. Pyramidinfer: Pyramid kv cache com- pression for high-throughput llm inference. arXiv preprint arXiv:2405.12532, 2024. 7

    2024 arXiv

  50. [58]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. EMNLP Demo Track, 2023. 1, 2

    2023

  51. [59]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 1

    2024 arXiv

  52. [60]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 1

    2024 arXiv

  53. [61]

    Needle in a video haystack: A scalable synthetic evalua- tor for video mllms

    Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in a video haystack: A scalable synthetic evalua- tor for video mllms. arXiv preprint arXiv:2406.09367, 2024. 5 10

    2024 arXiv

  54. [62]

    Mlvu: Benchmarking multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, et al. Mlvu: Benchmarking multi-task long video understanding. In CVPR, pages 13691–13701,

  55. [2025]

    11 to 14 show the absolute values for the main comparison results

    5 11 Appendix Tabs. 11 to 14 show the absolute values for the main comparison results. Token BudgetMethod Q. Agn. VNBench VideoMME LongVideoBench MLVU EgoSchema NExT-QA Avg. Acc↑TTFT↓ NV↓ Acc↑TTFT↓ NV↓ Acc↑TTFT↓ NV↓ Acc↑TTFT↓ NV↓ Acc↑TTFT↓ NV↓ Acc↑TTFT↓NV↓Acc↑TTFT↓ NV↓ 100%LLa...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.