Pith. sign in

REVIEW 4 major objections 5 minor 8 cited by

Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Video-XL-2 claims to beat every open-source lightweight MLLM on long-video benchmarks while cutting prefill FLOPs roughly in half and running 10,000 frames on a single A100 GPU.

desk verdict Solid efficiency engineering, but the headline SOTA claim is contradicted by the paper's own table and the core oracle is a black box. read the letter →

arxiv 2506.19225 v1 pith:7NR5HXTO submitted 2025-06-24 cs.CV cs.AI

Minghao Qin , Xiangrui Liu , Zhengyang Liang , Yan Shu , Huaying Yuan , Juenjie Zhou , Shitao Xiao , Bo Zhao
show 1 more author
Zheng Liu
This is my paper
classification cs.CVcs.AI
keywords longvideounderstandingmultimodallargelanguagemodelKVcachesparsificationchunk-basedpre-fillingbi-leveldecodingtemporalgroundinginferenceefficiencyquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to settle a practical trade-off: long-video understanding models are either accurate but too expensive, or fast but lossy. Video-XL-2's wager is that most of a long video is irrelevant to any particular question, so most of the video should be stored and read at coarse grain, with fine detail reserved for the few chunks a query actually touches. To that end it introduces two mechanisms, chunk-based pre-filling and bi-level KV decoding, that together claim to halve prefill compute, cut decoding KV cache by nearly forty percent, and still improve benchmark scores. If the claims hold, hour-scale video understanding becomes feasible on a single 80GB GPU, and open 7B/8B models can compete with much larger and closed models.

What carries the argument

The load-bearing object is task-aware KV sparsification, realized as two mechanisms. Chunk-based pre-filling splits the visual token sequence into equal chunks; each chunk attends fully to itself and sparsely to earlier chunks through historical timestamp tokens and a sliding window of overlap, so prefill cost scales with chunk length and roughly linearly with frame count instead of quadratically. Bi-level KV decoding keeps, for every video chunk, both the original dense key-value pairs and a downsampled sparse version; at query time a relevance oracle scores each chunk and the decoder loads dense KVs for high-scoring chunks and sparse KVs for the rest, producing a mixed KV set that is lighter and, the paper argues, more focused. The oracle is the component that decides where fine detail is preserved.

What would settle it

Pick a benchmark question whose answer depends on a detail in one brief moment, then deliberately make the relevance scorer rank that moment lowest; if the model still answers, the dense-versus-sparse distinction is not what preserves accuracy, and if it fails, the published results hinge entirely on scorer quality. A companion check is to replace the scorer with a random number generator and rerun MLVU, VideoMME, and LongVideoBench; if accuracies stay near the reported 74.8, 66.6, and 61.0, the query-aware selection is not doing the work claimed.

Watch

Extended reading notes

Core claim

Video-XL-2 argues that the reason long-video MLLMs are expensive is not that they need all their visual tokens at full resolution everywhere; it is that they keep dense key-value state for the entire input even when a specific question only needs fine detail in a few places. The paper's claim is that by running pre-filling in chunks, with full attention inside each chunk and sparse attention across chunks through timestamp tokens and a sliding window, and by maintaining two resolutions of KV cache per chunk, dense and downsampled, then loading dense KVs only for chunks a relevance oracle ranks as important to the query, a 7B model can match or beat all open-source lightweight rivals on long-video benchmarks while using 48.8% of the original prefill FLOPs and 61.2% of the original decoding KV cache. State-of-the-art results are reported on MLVU dev/test, VideoMME without subtitles, LVBench, and VideoEval-Pro among lightweight open models, with competitive temporal grounding on Charades-STA and V-STaR.

Load-bearing premise

The method's accuracy rests on an assumption about the scoring step: it must rarely rank the one chunk that contains the answer as irrelevant, because chunks marked irrelevant are stored and read only at coarse resolution.

Editorial extensions

If this is right

  • Long-video question answering on hour-scale footage becomes feasible on a single 80GB GPU: the paper reports processing up to 10,000 frames on one A100 and thousands of frames in seconds.
  • Prefill cost grows nearly linearly rather than quadratically with frame count, so the same recipe should keep working on still longer inputs, limited mainly by hardware rather than attention complexity.
  • Keeping dense detail only where the query points and coarse detail elsewhere gives other multimodal models a new design point: efficiency and accuracy need not be traded off, as Table 3 shows FLOPs falling to 48.8% with small benchmark gains.
  • Temporal grounding in long video should improve because the timestamp tokens that carry cross-chunk history give the model explicit time anchors; the paper reports competitive results on Charades-STA and V-STaR.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline efficiency numbers obscure one degree of freedom: Table 3 compares the full oracle-based pipeline against no sparsification, so the contribution of the relevance ranking itself is not isolated. A random-selection ablation would separate downsampling from task-awareness.
  • The near-linear prefill scaling suggests the 10,000-frame ceiling is a hardware boundary rather than a method boundary; multi-GPU or stronger pooling could plausibly reach feature-length video, but correctness at that scale is untested.
  • The historical timestamp tokens function as a compressed summary of past chunks, so the relevance oracle could be replaced by an actual video-to-text retriever, connecting this decoding scheme with retrieval-augmented long-video systems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Video-XL-2, a 7B-parameter video MLLM built on Qwen2.5-7B, with a vision encoder, dynamic token synthesis (DTS), MLP projector, and an LLM. Its efficiency contributions are chunk-based pre-filling (full attention within chunks, sparse attention across chunks with historical timestamp tokens) and bi-level KV decoding (dense KVs for query-relevant chunks, downsampled sparse KVs for the rest, selected via a relevance oracle). The authors claim state-of-the-art results on long-video understanding benchmarks among open-source lightweight models, processing over 10,000 frames on an 80GB A100, and report substantial FLOPs and KV-cache reductions. The paper includes training stages, main benchmark results in Table 2, efficiency ablations in Table 3, and a needle-in-haystack evaluation.

Significance. The efficiency mechanisms, especially chunk-based pre-filling with timestamp tokens and task-aware KV selection, are practically motivated and, if correctly evaluated, would be a useful contribution to long-video MLLM inference. The authors have publicly released the model and provide concrete training and efficiency numbers, which supports reproducibility of the method's architecture. However, the paper's central comparative claim is currently undercut by its own Table 2, and the unspecified relevance oracle makes the main efficiency-performance tradeoff difficult to verify. As a result, the headline significance claims are not yet established in the submitted version.

major comments (4)
  1. [Table 2 and Section 1 (Contribution 1)] The claim that Video-XL-2 'outperforms all existing open-source lightweight (7B or 8B) MLLMs' is contradicted by the paper's own table. Seed1.5-VL-8B and Eagle2.5-8B are listed under 'Closed-source Models' even though both are publicly released open-weight 8B models. On MLVU Dev, Seed1.5-VL-8B scores 82.1 and Eagle2.5-8B scores 77.6, while Video-XL-2 scores 74.8; on VideoMME, Eagle2.5-8B scores 72.4 versus Video-XL-2's 66.6. Under the paper's own classification criterion, these baselines belong in the open-source lightweight comparison set, so the abstract and contribution 1 overstate the result. This must be corrected by regrouping the baselines or by revising the superiority claims.
  2. [Section 3.4 (Bi-level KVs Decoding)] The relevance oracle that selects dense versus sparse KV chunks is never specified for the main results. The text says the oracle 'can be implemented via various methods such as an MLLM embedder, attention score mechanism or a more complex, custom-designed relevance computation pipeline,' but no concrete implementation, hyperparameters, or dense-chunk selection ratio are given for the experiments in Table 2 and Table 3. Without this information, the reported 38.8% average KV-cache reduction and the performance preservation in Table 3 cannot be reproduced or audited. Please disclose the exact oracle used, its parameters, and an ablation showing sensitivity to oracle quality.
  3. [Section 4.1 and Table 2 (Efficiency metrics)] The efficiency comparison uses 'Visual Input Length (k)' and 'FLOPs (G)' computed from 'the average of the maximum frame counts utilized across all long video understanding benchmarks.' This definition is ambiguous: it is unclear whether FLOPs include vision encoder and DTS computations, how the maximum frame count is determined for each model, and how different sampling strategies affect the comparison. Since the paper's efficiency advantage is a central contribution, please provide a precise methodology for both metrics and, ideally, report the underlying frame counts per benchmark.
  4. [Section 4.2 and MLVU benchmark] MLVU [49] is authored by members of the same research group, and the paper highlights MLVU Dev as the headline result where Video-XL-2 surpasses GPT-4o. To rule out evaluation-protocol bias or accidental training-data overlap, please disclose whether any MLVU data was used in training and provide a detailed evaluation protocol, including how the subset of MLVU Dev used for Table 2 was selected. Independent confirmation on benchmarks not authored by the group would substantially strengthen the claim.
minor comments (5)
  1. [Section 2] There are several typos in the related work, such as 'LLaV A' instead of 'LLaVA,' and 'MLP projetor' instead of 'MLP projector' in Section 3.3.
  2. [Table 2] The column headers 'M-avg' and 'M-avg w/o sub' are not defined; please spell out the metrics and specify which benchmarks contribute to each average.
  3. [Section 3.2] The frame sampling strategy depends on a 'pre-defined frame upper bound' and a 'pre-defined maximum sampling rate,' but neither value is stated; please report the actual values used in the experiments.
  4. [Section 4.4 and Figure 5] The needle-in-haystack evaluation is described only qualitatively; please provide the number of questions, the scoring rule, and numeric results for Video-XL-2 and Video-XL so the claimed ability to process 10,000 frames can be assessed.
  5. [Table 3] The rows 'Video-XL2 100% 100%' and '+Bi-level KVs Decoding 48.2% 61.2%' are inconsistent with the reported average FLOPs and KV-cache values: please clarify whether the first row is the dense baseline and how the percentages are normalized.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the paper's empirical claims rest on direct measurements and public benchmark comparisons, not on inputs that are equivalent to the reported outputs.

full rationale

Video-XL-2 is an empirical systems paper: its main claims are benchmark scores and measured efficiency numbers. There is no equation in which a fitted parameter is relabeled as a prediction, and no quantity is defined in terms of the target result. The chunk-based pre-filling and bi-level KV decoding are described operationally and evaluated in Table 3 against the model's own full-precision configuration; these are ablations, not circular constructions. The DTS module is inherited from prior work by the same group (Video-XL-Pro [16]), but its use here is architectural adoption rather than a derivation that reduces the current paper's results to that citation. The relevance oracle in Section 3.4 is indeed underspecified, which is a reproducibility weakness, but the paper does not state that the oracle is trained on the target benchmarks or that its scores are derived from the final answer; it is an inference-time selector. Similarly, the MLVU benchmark [49] is co-authored by members of this group, raising a valid concern about evaluation independence, but the model's scores are measured against a fixed public benchmark, not constructed from that benchmark's definition. No step in the paper exhibits the specific reduction pattern required for a circularity finding, so the appropriate verdict is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central efficiency claims rest on hand-set hyperparameters that are never stated (frame bound, chunk size, window step, dense-chunk ratio), on the inherited DTS module from the authors' own Video-XL-Pro, and on an undisclosed relevance oracle. The oracle is the most fragile piece because it is introduced for this paper and carries no independent evidence of reliability.

free parameters (4)
  • Frame sampling upper bound = not stated
    The frame sampling strategy (Section 3.2) resamples up to a 'pre-defined frame upper bound' and maximum sampling rate; both are hand-set design choices that set input length, FLOPs, and benchmark scores, yet their values are never given.
  • Chunk size (visual tokens per chunk) = not stated
    Chunk length determines pre-filling attention cost, cross-chunk timestamp coverage, and the granularity of bi-level KV selection (Section 3.4); the paper never states it.
  • Sliding chunk window step size = not stated
    The sliding chunk window (Figure 2, Section 3.4) sets how much detail from the previous chunk is visible; the step size is unspecified, and it directly controls the FLOPs reduction.
  • Dense-chunk selection ratio in bi-level decoding = implied 38.8% KV-cache reduction
    Table 3 reports an average KV-cache reduction of 38.8% during decoding, but the paper does not state how many chunks receive dense KVs or the threshold used; the headroom of the whole method depends on this unstated ratio.
assumptions (4)
  • domain assumption Sparsity of attention in VLMs transfers to the chunked video setting without retraining
    Section 3.4 relies on prior observations [17,22] that KV sparsity exists in LLM and VLM attention, and assumes the same sparsity patterns justify dropping cross-chunk computation in Video-XL-2; the paper adds no measurements of attention sparsity for its own model.
  • domain assumption DTS module compresses temporal redundancy without losing needed information
    The Dynamic Token Synthesize module is adopted wholesale from Video-XL-Pro [16], same author group; its effectiveness is asserted, not re-derived or independently validated here.
  • domain assumption Timestamp tokens supply sufficient coarse-grained cross-chunk context
    Chunk-based pre-filling lets each chunk attend only to its own tokens, historical timestamp tokens, and the system prompt (Section 3.4); the adequacy of timestamp-only history for later questions is assumed without analysis.
  • ad hoc to paper Relevance oracle scores correlate with where the answer lives in the video
    The bi-level decoding method's performance preservation depends on the oracle (Section 3.4) ranking chunks correctly; the oracle is an unspecified component introduced for this paper, with no evidence provided for its reliability.
invented entities (2)
  • Relevance oracle
    purpose: Assigns each video chunk a relevance score against the query, deciding whether dense or sparse KVs are loaded during decoding
    The paper treats the oracle as an interchangeable abstraction ('can be implemented via various methods') but never discloses which one produced the main results, so its reliability has no falsifiable handle outside this paper.
  • Bi-level KV representation (dense and pooled-sparse KVs per chunk)
    purpose: Stores two compression levels of each chunk's KV cache so decoding can trade detail against memory per chunk
    This is a new representational construct whose value rests entirely on the paper's own ablations (Table 3); no external benchmark or independent implementation confirms the 38.8% memory saving with stable scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification." pith.science (2026). https://pith.science/paper/7NR5HXTO

@misc{pith2026250619225,
  author       = {Pith},
  title        = {Pith review of: Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7NR5HXTO}},
  note         = {Machine review of arXiv:2506.19225}
}
read the original abstract

Multi-modal large language models (MLLMs) models have made significant progress in video understanding over the past few years. However, processing long video inputs remains a major challenge due to high memory and computational costs. This makes it difficult for current models to achieve both strong performance and high efficiency in long video understanding. To address this challenge, we propose Video-XL-2, a novel MLLM that delivers superior cost-effectiveness for long-video understanding based on task-aware KV sparsification. The proposed framework operates with two key steps: chunk-based pre-filling and bi-level key-value decoding. Chunk-based pre-filling divides the visual token sequence into chunks, applying full attention within each chunk and sparse attention across chunks. This significantly reduces computational and memory overhead. During decoding, bi-level key-value decoding selectively reloads either dense or sparse key-values for each chunk based on its relevance to the task. This approach further improves memory efficiency and enhances the model's ability to capture fine-grained information. Video-XL-2 achieves state-of-the-art performance on various long video understanding benchmarks, outperforming existing open-source lightweight models. It also demonstrates exceptional efficiency, capable of processing over 10,000 frames on a single NVIDIA A100 (80GB) GPU and thousands of frames in just a few seconds.

Figures

Figures reproduced from arXiv: 2506.19225 by the authors.

Figure 1
Figure 1. The architecture of Video-XL-2. The proposed Video-XL-2 comprises four main components: (1) Vision encoder to encode images and videos, (2) DTS to compress and make initial temporal modeling on visual features from vision encoder. (3) an MLP projector to project visual features into LLM embedding, and (4) a Large Language Model to process multi-modal inputs. Video-XL-2 interleaves timestamp tokens within the visual … view at source ↗
Figure 2
Figure 2. Chunk-based Pre-filling Illustration. In chunk-based pre-filling, the current processing chunk only attend to itself, historical timestamp tokens and the system prompt, as depicted in the left subfigure. And the right subfigure illustrates the current chunks for processing are decided by a sliding chunk window. Chunk-based Pre-filling. The computational complexity of attention operations scales quadratically with th… view at source ↗
Figure 3
Figure 3. Bi-level KVs decoding. Bi-level KVs comprise both dense KVs (derived from the full video input) and sparse KVs, where the latter are obtained by downsampling the former at a chunk level. During decoding, it selectively reloads either dense KVs for video chunks highly relevant to the specific task query text, or sparse KVs for less relevant chunks, optimizing memory while preserving critical information. Bi-level KVs… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Efficiency Analysis Illustration. All results measured using eager attention for fair compari [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Needle in Haystack Evaluation. information in these lengthy inputs. To evaluate this, we utilized question-answer pairs from [56] and videos exceeding one hour from the VideoMME as our “haystack" videos to deliver a “Needle in Haystack” Evaluation. As depicted in [PIT…

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GROVE: Growing and Reasoning over Temporally Stratified Memory from Streaming Video Experience

    cs.CV 2026-08 conditional novelty 6.0 of 10

    GROVE shows that a streaming video memory stratified into four temporal scales, each with its own retrieval skill, improves both question answering and proactive assistance across five benchmarks.

  2. Modularized Dynamic-Granularity Video LLM for Multi-Event Long Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An iterative grounding-and-granularity video LLM outperforms strong baselines on multi-event long-video QA and on a new MEventBench benchmark.

  3. VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An open 4B video MLLM with inflated-3D ViT tokenization and adaptive streaming perception outperforms comparable open models on general, long-video, and streaming benchmarks while using fewer visual tokens.

  4. FOLIO: Focused Semantic Memory for Streaming Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Entity-centered focus-guided streaming memory lifts Qwen3-VL-8B to 82.0/69.1 Perception/Backward on OVO-Bench and 74.5 on StreamingBench while cutting writer tokens by ~32%.

  5. $M^3-Verse$: A "Spot the Difference" Challenge for Large Multimodal Models

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A new benchmark tests whether large multimodal models can compare paired 'before and after' videos to detect scene changes, and finds current models perform near random.

  6. cuRAMSES: Scalable AMR Optimizations for Large-Scale Cosmological Simulations

    astro-ph.GA 2026-04 conditional novelty 5.0 of 10

    Recursive k-section domain decomposition, Morton-key hashing, and GPU dispatch cut communication and memory bottlenecks in RAMSES while preserving conservation to ~0.5%.

  7. Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Models

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Identical video questions get different accuracy when placed at the start, middle, or end of a long context, and the new benchmark maps this bias across 27 video-language models.

  8. Infinite Video Understanding

    cs.CV 2025-07 conditional novelty 3.0 of 10

    The paper argues that video understanding research should aim at processing streams of arbitrary, unbounded duration and outlines the challenges, directions, and metrics needed.

Reference graph

Works this paper leans on

56 extracted references · 6 canonical work pages · cited by 8 Pith papers

  1. [49]

    Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    arXiv 2024

  2. [1]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  3. [2]

    Gpt-4o.https://openai.com/index/hello-gpt-4o/, May 2024

    OpenAI. Gpt-4o.https://openai.com/index/hello-gpt-4o/, May 2024

    2024

  4. [3]

    Claude 3.https://www.anthropic.com/news/claude-3-family, March 2024

    Anthropic. Claude 3.https://www.anthropic.com/news/claude-3-family, March 2024

    work page 2024

  5. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [5]

    Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    arXiv 2025

  7. [6]

    Videochat-flash: Hierarchical compression for long-context video modeling.arXiv preprint arXiv:2501.00574, 2024

    Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flash: Hierarchical compression for long-context video modeling.arXiv preprint arXiv:2501.00574, 2024

    arXiv 2024

  8. [7]

    Nvila: Efficient frontier visual language models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 4122–4134, 2025

    work page 2025

Show all 56 references
  1. [8]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  2. [9]

    Eagle 2.5: Boosting long-context post-training for frontier vision-language models.arXiv preprint arXiv:2504.15271, 2025

    Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, et al. Eagle 2.5: Boosting long-context post-training for frontier vision-language models.arXiv preprint arXiv:2504.15271, 2025

    2025

  3. [10]

    Longllava: Scaling multi- modal llms to 1000 images efficiently via hybrid architecture.arXiv preprint arXiv:2409.02889, 2024

    Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi- modal llms to 1000 images efficiently via hybrid architecture.arXiv preprint arXiv:2409.02889, 2024

    2024

  4. [11]

    Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  5. [12]

    Llama-vid: An image is worth 2 tokens in large language models.arXiv preprint arXiv:2311.17043, 2023

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models.arXiv preprint arXiv:2311.17043, 2023

    2023 arXiv

  6. [13]

    Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  7. [14]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  8. [15]

    Moviechat: From dense token to sparse memory for long video understanding.arXiv preprint arXiv:2307.16449, 2023

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding.arXiv preprint arXiv:2307.16449, 2023

    2023 arXiv

  9. [16]

    Video-xl-pro: Reconstructive token compression for extremely long video understanding.arXiv preprint arXiv:2503.18478, 2025

    Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, and Bo Zhao. Video-xl-pro: Reconstructive token compression for extremely long video understanding.arXiv preprint arXiv:2503.18478, 2025

    2025 arXiv

  10. [17]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention, 2024.URL https://arxiv

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention, 2024.URL https://arxiv. org/abs/2407.02490

    2024 arXiv

  11. [18]

    Snapkv: Llm knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947–22970, 2024

    2024

  12. [19]

    Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  13. [20]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing System...

    2023

  14. [21]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089, 2025

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089, 2025

    2025 arXiv

  15. [22]

    Moba: Mixture of block attention for long-context llms.arXiv preprint arXiv:2502.13189, 2025

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms.arXiv preprint arXiv:2502.13189, 2025

    2025 arXiv

  16. [23]

    Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  17. [24]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  18. [25]

    Flamingo: a visual language model for few-shot learning.NeurIPS, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.NeurIPS, 2022

    2022

  19. [26]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.ICML, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.ICML, 2023

    2023

  20. [27]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  21. [28]

    Vidtext: Towards comprehensive evaluation for video text understanding.arXiv preprint arXiv:2505.22810, 2025

    Zhoufaran Yang, Yan Shu, Zhifei Yang, Yan Zhang, Yu Li, Keyang Lu, Gangyan Zeng, Shaohui Liu, Yu Zhou, and Nicu Sebe. Vidtext: Towards comprehensive evaluation for video text understanding.arXiv preprint arXiv:2505.22810, 2025

    2025

  22. [29]

    Vid-sme: Membership inference attacks against large video understanding models.arXiv preprint arXiv:2506.03179, 2025

    Qi Li, Runpeng Yu, and Xinchao Wang. Vid-sme: Membership inference attacks against large video understanding models.arXiv preprint arXiv:2506.03179, 2025

    2025 arXiv

  23. [30]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. arXiv preprint arXiv:2404.05726, 2024

    2024 arXiv

  24. [31]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. InEuropean Conference on Computer Vision, pages 323–340. Springer, 2024

    2024

  25. [32]

    Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos.arXiv preprint arXiv:2408.14023, 2024

    Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos.arXiv preprint arXiv:2408.14023, 2024

    2024 arXiv

  26. [33]

    Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022

    2022 arXiv

  27. [34]

    Long context compression with activation beacon.arXiv preprint arXiv:2401.03462, 2024

    Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Long context compression with activation beacon.arXiv preprint arXiv:2401.03462, 2024

    2024 arXiv

  28. [35]

    Lighter and better: Towards flexible context adaptation for retrieval augmented generation

    Chenyuan Wu, Ninglu Shao, Zheng Liu, Shitao Xiao, Chaozhuo Li, Chen Zhang, Senzhang Wang, and Defu Lian. Lighter and better: Towards flexible context adaptation for retrieval augmented generation. In Proceedings of the Eighteenth ACM International Conference on Web Search and ...

    2025

  29. [36]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. InEuropean Conference on Computer Vision, pages 19–35. Springer, 2024

    2024

  30. [37]

    Zipvl: Efficient large vision-language models with dynamic token sparsification and kv cache compression.arXiv preprint arXiv:2410.08584, 2024

    Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipvl: Efficient large vision-language models with dynamic token sparsification and kv cache compression.arXiv preprint arXiv:2410.08584, 2024

    arXiv

  31. [38]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069, 2024

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069, 2024

    2024 arXiv

  32. [39]

    Mminference: Accelerating pre-filling for long-context vlms via modality-aware permutation sparse attention.arXiv preprint arXiv:2504.16083, 2025

    Yucheng Li, Huiqiang Jiang, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Amir H Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, et al. Mminference: Accelerating pre-filling for long-context vlms via modality-aware permutation sparse attention.arXiv preprint arXiv:2504...

    2025 arXiv

  33. [40]

    Video-xl: Extra-long vision language model for hour-scale video understanding

    Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 26160–26169, 2025

    2025

  34. [41]

    Retake: Reducing temporal and knowledge redundancy for long video understanding.arXiv preprint arXiv:2412.20504, 2024

    Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. Retake: Reducing temporal and knowledge redundancy for long video understanding.arXiv preprint arXiv:2412.20504, 2024

    2024 arXiv

  35. [42]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and ...

    2025 arXiv

  36. [43]

    Qwen2.5 technical report

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025

  37. [44]

    Longvila: Scaling long-context visual language models for long videos

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024

    2024 arXiv

  38. [45]

    Languagebind: Extending video-language pretraining to n-modality by language- based semantic alignment.arXiv preprint arXiv:2310.01852, 2023

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language- based semantic alignment.arXiv preprint arXiv:2310.01852, 2023

    arXiv

  39. [46]

    Internvideo2: Scaling foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for multimodal video understanding. InEuropean Conference on Computer Vision, pages 396–416. Springer, 2024

    2024

  40. [47]

    Megapairs: Massive data synthesis for universal multimodal retrieval.arXiv preprint arXiv:2412.14475, 2024

    Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval.arXiv preprint arXiv:2412.14475, 2024

    2024 arXiv

  41. [48]

    Memory- enhanced retrieval augmentation for long video understanding.arXiv preprint arXiv:2503.09149, 2025

    Huaying Yuan, Zheng Liu, Minhao Qin, Hongjin Qian, Y Shu, Zhicheng Dou, and Ji-Rong Wen. Memory- enhanced retrieval augmentation for long video understanding.arXiv preprint arXiv:2503.09149, 2025

    2025 arXiv

  42. [50]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  43. [51]

    Longvideobench: A benchmark for long-context interleaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024

    2024 arXiv

  44. [52]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024

    2024 arXiv

  45. [53]

    Videoeval-pro: Robust and realistic long video understanding evaluation.arXiv preprint arXiv:2505.14640, 2025

    Wentao Ma, Weiming Ren, Yiming Jia, Zhuofeng Li, Ping Nie, Ge Zhang, and Wenhu Chen. Videoeval-pro: Robust and realistic long video understanding evaluation.arXiv preprint arXiv:2505.14640, 2025

    2025 arXiv

  46. [54]

    Tall: Temporal activity localization via language query

    Zhenheng Yang Jiyang Gao, Chen Sun and Ram Nevatia. Tall: Temporal activity localization via language query. InIn Proceedings of the IEEE international conference on computer vision, page 5267–5275, 2017

    2017

  47. [55]

    V-star: Benchmarking video-llms on video spatio-temporal reasoning.arXiv preprint arXiv:2503.11495, 2025

    Zixu Cheng, Jian Hu, Ziquan Liu, Chenyang Si, Wei Li, and Shaogang Gong. V-star: Benchmarking video-llms on video spatio-temporal reasoning.arXiv preprint arXiv:2503.11495, 2025

    2025 arXiv

  48. [56]

    Videorope: What makes for good video rotary position embedding?arXiv preprint arXiv:2502.05173, 2025

    Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, et al. Videorope: What makes for good video rotary position embedding?arXiv preprint arXiv:2502.05173, 2025. 12

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.