Pith. sign in

REVIEW 5 major objections 5 minor 55 references

AdaCM$^2$: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AdaCM2 keeps visual tokens that the text query pays attention to and discards the rest, so long videos can be processed with bounded memory and without losing—often improving—accuracy.

desk verdict A plausible cross-modal KV-eviction method with a real memory bound, but the key ablation lacks a no-eviction control, so the 4.5% accuracy attribution is underdetermined. read the letter →

arxiv 2411.12593 v3 pith:IFMKWMDL submitted 2024-11-19 cs.CV cs.AI

Yuanbin Man , Ying Huang , Chengming Zhang , Bingzhe Li , Wei Niu , Miao Yin This is my paper · ORCID
classification cs.CVcs.AI
keywords long-termvideounderstandingcross-modalattentionmemoryreductiontokenevictionquestionansweringcaptioningvision-languagemodelscachecompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AdaCM2 is a memory-reduction method that lets LLM-based video models watch very long videos by keeping only the visual tokens that the text prompt actually attends to. At each layer of the visual-textual alignment stage, it splits the running video cache into a recent part, which is always kept, and an older part, from which only the tokens with the highest cross-modality attention scores survive. The authors show this gives a bounded cache size no matter how many frames arrive, and report that it improves average accuracy by 4.5% on the seven-task LVU benchmark while reducing GPU memory by up to 65% compared with the prior MA-LMM system. If the claim holds, memory cost no longer scales with video duration, so hour-long and even unbounded video streams become practical for question answering, captioning, and classification.

What carries the argument

The load-bearing object is the cross-modality attention score $S_t^c(i)=\sum_j S_t(j,i)$, the total attention that text/query tokens give to a visual token in a Q-Former layer. AdaCM2 treats this scalar as an importance rank for eviction: the video cache is partitioned into a recent cache and a previous cache, the recent cache is kept intact to preserve temporal continuity, and the previous cache is trimmed to its top-$\beta$ scoring tokens. Because the trimming ratios $\alpha$ and $\beta$ are set per layer, the method adapts to the observation that deep layers are more redundant than shallow ones. Regressive frame-by-frame query updates make the learnable query tokens carry temporal context, and the final length-limited query vector is what the frozen LLM decodes from.

What would settle it

Take a long-video VQA item whose answer depends on a small or background visual detail, e.g., a license plate or a brief object near the frame edge, run AdaCM2 with its default $\alpha=0.1$ and $\beta=0.1$, and check whether the correct answer survives after eviction. A systematic version would replace the cross-attention ranker with random eviction at matched token budgets: if random eviction ever matches or beats the attention-based rule on a task, that task provides a counterexample to the assumption that the attention score is a dependable importance signal.

Watch

Extended reading notes

Core claim

The paper's central claim is that redundancy in video-text alignment is query-dependent and layer-dependent, and that this redundancy can be exploited without losing accuracy. By inspecting the Q-Former's cross-attention between text tokens and visual tokens, AdaCM2 observes that only a small subset of visual tokens in a frame receive high attention from the text, and that adjacent frames' attention patterns are highly similar, with similarity increasing in deeper layers. It therefore scores every visual token by the sum of cross-attention it receives from text tokens (Eq. 5), always keeps the most recent $\alpha$ fraction of the cache, and from the older $(1-\alpha)$ fraction keeps only the top-$\beta$ scoring tokens (Eq. 7). The resulting cache size follows a geometric bound, converging to $P r/(1-r)$ with $r=\alpha+(1-\alpha)\beta$ as video length grows. On this basis the paper claims 4.5% average accuracy improvement over the strongest prior memory-bank method on LVU, gains on Breakfast/COIN and on MSVD captioning, and up to 65% lower GPU memory with near-flat growth as frames increase.

Load-bearing premise

The method assumes that a visual token's total cross-attention score from the text query correctly identifies which visual details are needed to answer that query, in every layer and every video; if the needed information sits in a low-attention background token, the eviction rule could delete the evidence needed for the right answer.

Editorial extensions

If this is right

  • With $T\to\infty$, the video cache converges to the constant size $P r/(1-r)$, so videos of any length can in principle be streamed through the model with bounded memory.
  • On the LVU benchmark the method raises average Top-1 accuracy from 63.0 to 67.5 over the prior MA-LMM baseline, with the largest gains on scene, writer, and year prediction.
  • Because the visual encoder and LLM stay frozen and only the Q-Former is fine-tuned, the reduction module can be dropped into existing BLIP-based video-language pipelines without retraining the large backbones.
  • The layer-wise adaptive ratios mean the model spends more memory where information is denser and less where frames are redundant, making memory use track the video's content rather than its duration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same query-guided eviction rule could be applied to other long-context modalities, such as audio or sensor streams, wherever a cross-attention score is available between a query and a per-timestep memory.
  • A testable prediction follows: queries about small or background details (e.g., a number on a jersey, an object behind the main action) should show a larger accuracy drop than queries about salient foreground content, because such details are more likely to live in low-attention tokens that get evicted.
  • The bounded-cache result suggests an online or streaming deployment where each frame is consumed once and expired tokens never need to be revisited; that would make AdaCM2 a candidate for real-time long-term video assistants, though the paper does not evaluate latency or streaming.
  • Layer-wise $\alpha$ and $\beta$ could be made question-adaptive by inspecting the entropy of the attention distribution per layer, rather than fixed global values; the paper's hyperparameter sweeps show a mid-range optimum but do not explore per-video adaptation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes AdaCM^2, a video-language model that reduces GPU memory for long-video understanding by selecting which visual tokens to keep in a growing video cache using cross-modality attention scores from a Q-Former. The visual encoder and LLM are frozen; only Q-Former parameters are fine-tuned. The method processes frames regressively, partitions the cache into recent and previous parts, and evicts previous-cache tokens with low accumulated cross-modality attention scores. The authors report state-of-the-art or competitive results on LVU, Breakfast, COIN, MSRVTT-QA, MSVD-QA, and captioning benchmarks, and claim up to 65% GPU memory reduction with near-constant memory as video length grows. The central derivation, Eq. (8), gives a constant asymptotic cache size under a fixed per-step reduction ratio.

Significance. If the claims hold, the contribution is practically significant: it addresses a real bottleneck in LLM-based video understanding, where memory grows with frame count. The paper has several genuine strengths: the memory bound in Eq. (8) is a correct mathematical consequence of the reduction algorithm; the random-eviction comparison in Figure 7 is a useful sanity check for the selection signal; and the plug-and-play framing around InstructBLIP is attractive. The method is also evaluated across multiple datasets and tasks. However, the central mechanistic claim — that cross-modality attention scores preserve answer-critical tokens — is under-supported: the evidence against random eviction is not an upper-bound control, there are no error bars or run counts, and the adaptive layer-wise component is not actually exercised in the experiments because alpha and beta are fixed to 0.1 for all layers. The significance will be much clearer once these missing controls are supplied.

major comments (5)
  1. [§4.2 / Eq. (8), §5.4] The paper motivates a layer-wise adaptive reduction from Observation 2 (Section 3.2), but Eq. (8) derives the memory bound using a single per-step ratio r = alpha + (1-alpha)beta, and Section 5.4 states that alpha and beta are set to 0.1 for all layers. A fixed alpha=beta=0.1 means the reduction is uniform across layers, so the central 'adaptive layer-wise' claim is not actually exercised in any experiment. Please either generalize Eq. (8) to layer-dependent r_l (with an appropriate product/sum bound) and report experiments with distinct per-layer ratios, or soften the layer-wise claim to match the evaluated configuration.
  2. [§5.4 / Figure 7] The only behavioral evidence that cross-modality attention scores identify answer-critical tokens is a comparison with random eviction. Beating random eviction does not establish that the retained tokens are the answer-critical ones; a selection rule can beat random while still discarding important information. The paper should add a no-eviction (or beta=1) control on the same LVU tasks to quantify how much accuracy is sacrificed by the memory reduction itself, and ideally ablate the regressive Q-Former query and the cache-partition design separately. Without these controls, the 4.5% LVU improvement over MA-LMM cannot be attributed specifically to the cross-modality ranking.
  3. [Tables 1–3] All accuracy numbers are reported as single values without the number of runs, standard deviations, or significance tests. Several differences against the strongest baseline are small (e.g., COIN 93.3 vs. 93.2, MSRVTT METEOR 33.0 vs. 33.4, YouCook2 CIDEr 125.6 vs. 131.2), and these could easily be within run-to-run variation for fine-tuned multimodal models. Please report at least three runs with mean and standard deviation, and state whether the reported numbers are averages or best runs.
  4. [Abstract / §5.3 / Figure 6] The abstract's 'up to 65% GPU memory reduction' claim is not supported by any table or numerical protocol. Figure 6 shows qualitative curves, but the paper does not report the measured memory values, the batch size, the number of frames at which the 65% figure is computed, or how the memory footprint is measured (e.g., peak CUDA memory vs. model plus cache). Please add a table with memory measurements and the exact measurement conditions, and state whether the 65% reduction is relative to MA-LMM, InstructBLIP, or another baseline.
  5. [§4.2 / Eq. (5)] Equation (5) defines the cross-modality attention score S_c^t(i) by summing over j = 1..N, where N is the number of query and text tokens. The surrounding text says the score measures correlation with 'text tokens,' but the sum includes the learnable Q-Former query tokens, which are trained parameters, not text. This conflation matters because the eviction rule in Eq. (7) is driven by S_c^t. Please either restrict the sum to the text-token subset, or explicitly justify why query-token attention belongs in the scoring signal.
minor comments (5)
  1. [Throughout] There are multiple typos and inconsistencies: 'weight matrics' (Section 4.2), 'MSVT' instead of 'MSRVTT' (Section 5.4), inconsistent 'AdaCM 2' / 'AdaCM2' spacing, and the Table 1 title 'The underlinednumber means the second best.' A careful proofread is needed.
  2. [Figure 2] Figure 2 is presented as a qualitative zero-shot case study on Ego4D, but the paper does not state whether this video is part of any quantitative benchmark or how it was selected. A brief sentence clarifying its role would prevent confusion.
  3. [Figure 3c] Observation 2 is supported by a single MSR-VTT video. The claim that redundancy varies across layers would be strengthened by reporting statistics over multiple videos and datasets, with error bars or box plots.
  4. [Table 3] The caption and text claim state-of-the-art performance, but mPLUG-2 achieves higher METEOR and CIDEr on MSRVTT. The dismissal of mPLUG-2 due to pretraining overhead should be made in the text as a qualifying statement rather than in the table caption only.
  5. [§4.3 / Eq. (9)] The loss in Eq. (9) uses symbols V and P that are not defined in the text generation section; please define all notation, including the role of the video features V in a text-generation loss.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the token-selection rule is not fitted to the answer, and the memory bound is a deductive consequence of the eviction rule.

full rationale

The paper's central claim is that cross-modality attention scores (Eq. 5) identify visual tokens worth keeping in the video cache (Eq. 7), and that layer-wise eviction keeps memory bounded (Eq. 8). None of these steps reduces to the paper's own inputs by construction. The attention scores are computed from the current Q-Former and text query, not from the target answers or from the final evaluation metrics; the eviction rule discards low-score tokens before the LLM generates text, so the reported accuracy is an empirical outcome rather than an identity. The bounded-memory formula |KT| = P r(1-r^T)/(1-r) is a direct algebraic consequence of the split and conserve ratios α and β, not a fitted result. The only tuned quantities are the hyperparameters α and β, which are chosen from accuracy/memory curves in Figure 8; this is conventional hyperparameter selection, and the paper does not present α or β as a prediction. There are no load-bearing self-citations: the motivating references [14,46] and backbone models [11,23] are external and independently developed. The absence of a no-eviction control and the reliance on a single visualization for Observation 1 are legitimate concerns about empirical support, but they concern evidence strength rather than circularity. Accordingly, no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The central claims rest on the Q-Former's cross-attention scores being a valid importance signal, and on fixed ratios alpha and beta for the memory bound. The two hyperparameters are tuned on the evaluation datasets themselves via accuracy/memory curves, which adds a mild fitting component to the reported numbers.

free parameters (2)
  • split ratio alpha = 0.1 (per layer)
    Fraction of the video cache retained as 'recent'; tuned on MSRVTT-QA and MSVD-QA accuracy/memory curves (Figure 8); set to 0.1 for all layers despite the text claiming layer-wise values.
  • conserve ratio beta = 0.1 (per layer)
    Fraction of the previous-cache tokens kept by top cross-modality attention score; tuned jointly with alpha in Figure 8; set to 0.1 for all layers.
assumptions (2)
  • domain assumption Cross-modality attention scores from the Q-Former correctly rank visual tokens by their importance for the text-conditioned output.
    Observations in Section 3 are based on one randomly sampled MSR-VTT video; the whole method assumes these scores generalize across videos, layers, and tasks.
  • ad hoc to paper A constant per-step cache reduction ratio r = alpha + (1-alpha)*beta applies across all layers and time steps for the memory bound in Eq. 8.
    Section 4.2 derives convergence of |KT| using a single r; Section 5.4 sets identical alpha and beta for all layers, so the bound holds, but the paper's claim of layer-wise adaptivity is not reflected in distinct ratios.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AdaCM$^2$: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction." pith.science (2026). https://pith.science/paper/IFMKWMDL

@misc{pith2026241112593,
  author       = {Pith},
  title        = {Pith review of: AdaCM$^2$: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IFMKWMDL}},
  note         = {Machine review of arXiv:2411.12593}
}
abstract

The advancements in large language models (LLMs) have propelled the improvement of video understanding tasks by incorporating LLMs with visual models. However, most existing LLM-based models (e.g., VideoLLaMA, VideoChat) are constrained to processing short-duration videos. Recent attempts to understand long-term videos by extracting and compressing visual features into a fixed memory size. Nevertheless, those methods leverage only visual modality to merge video tokens and overlook the correlation between visual and textual queries, leading to difficulties in effectively handling complex question-answering tasks. To address the challenges of long videos and complex prompts, we propose AdaCM$^2$, which, for the first time, introduces an adaptive cross-modality memory reduction approach to video-text alignment in an auto-regressive manner on video streams. Our extensive experiments on various video understanding tasks, such as video captioning, video question answering, and video classification, demonstrate that AdaCM$^2$ achieves state-of-the-art performance across multiple datasets while significantly reducing memory usage. Notably, it achieves a 4.5% improvement across multiple tasks in the LVU dataset with a GPU memory consumption reduction of up to 65%.

Figures

Figures reproduced from arXiv: 2411.12593 by the authors.

Figure 1
Figure 1. (Left) Existing approaches compress visual features of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The case study of AdaCM2 zero-shot on Ego4D[15] dataset. As shown, AdaCM2 can 1) summarize an extremely long video lasting over 2 hours with limited memory consumption and identify the number on a person’s back at the end accurately, 2) answer questions related to a mid-length video spanning more than 20 minutes. the text prompt, particularly in Visual Question Answer￾ing (VQA) [2, 12, 20, 50] tasks with complex tex… view at source ↗
Figure 3
Figure 3. Visualization for cross-modality attention, generated using a randomly sampled video from the MSR-VTT [ [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The framework of AdaCM2 . With video and text query as input, AdaCM2 first utilizes a visual encoder to extract visual features from video frames. Then, video Q-Former embeds the correlation between visual features and the text prompt into a learnable query in a regres…
Figure 5
Figure 5. Figure 5: Illustration for our video memory reduction. The [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Practical memory consumption analysis compared to ex [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Performance Comparison between two memory reduc [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 39 canonical work pages

  1. [1]

    Keyformer: Kv cache reduction through key tokens selection for efficient generative inference

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 7, 2024. 3

    work page 2024

  2. [2]

    Visual question an- swering, 2015

    Aishwarya Agrawal, Dhruv Batra, Marcus Rohrbach, Jiasen Lu, Michael Bernstein, and Devi Parikh. Visual question an- swering, 2015. 2

    work page 2015

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  4. [4]

    METEOR: An auto- matic metric for MT evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. METEOR: An auto- matic metric for MT evaluation with improved correlation with human judgments. InProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan, 2005. Association for Computational Linguistics. 6

    2005

  5. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz...

    arXiv 2005

  6. [6]

    Chen and William B

    David L. Chen and William B. Dolan. Collecting highly par- allel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL-2011), Portland, OR, 2011. 6, 7

    work page 2011

  7. [7]

    Videollm-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In CVPR, 2024. 3

    work page 2024

  8. [8]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 6, 8

    work page 2023

Show all 55 references
  1. [9]

    Colwell, and Adrian Weller

    Krzysztof Choromanski, Valerii Likhosherstov, David Do- han, Xingyou Song, Andreea Gane, Tam ´as Sarl ´os, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy J. Colwell, and Adrian Weller. Re- thinking attention with performers. CoRR, abs/2009.14794,

    2009 arXiv

  2. [10]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robin...

    2022

  3. [11]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. 2, 3, 6, 7, 8

    2023

  4. [12]

    Visual question answering: A survey on techniques and common trends in recent literature, 2023

    Ana Claudia Akemi Matsuki de Faria, Felype de Castro Bas- tos, Jose Victor Nogueira Alves da Silva, Vitor Lopes Fabris, Valeska de Sousa Uchoa, Decio Gonc alves de Aguiar Neto, and Claudio Filipi Goncalves dos Santos. Visual question answering: A survey on techniques and commo...

    2023

  5. [13]

    Im- pact of green human resource management (ghrm) practices on organizational performance

    Naglaa Fathy El Dessouky and Hakeem Omar Alquaiti. Im- pact of green human resource management (ghrm) practices on organizational performance. In 2020 Second Interna- tional Sustainability and Resilience Conference: Technol- ogy and Innovation in Building Designs(51154) , pages 1–4,

    2020

  6. [14]

    Model tells you what to dis- card: Adaptive KV cache compression for LLMs

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to dis- card: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representa- tions, 2024. 4

    2024

  7. [15]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InPro- ceedings of the IEEE/CVF Conference on Computer Vision...

    2022

  8. [16]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. CVPR, 2024. 1, 3, 7

    2024

  9. [17]

    Noureldien Hussein, Efstratios Gavves, and Arnold W. M. Smeulders. Timeception for complex action recognition,

  10. [18]

    Noureldien Hussein, Efstratios Gavves, and Arnold W. M. Smeulders. Videograph: Recognizing minutes-long human activities in videos, 2019. 7

    2019

  11. [19]

    Long movie clip classification with state-space video models

    Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. arXiv preprint arXiv:2204.01692, 2022. 3, 7

    2022 arXiv

  12. [20]

    Show, ask, attend, and answer: A strong baseline for visual question answering

    Vahid Kazemi and Ali Elqursh. Show, ask, attend, and answer: A strong baseline for visual question answering. CoRR, abs/1704.03162, 2017. 2

    2017 arXiv

  13. [21]

    Kuehne, A

    H. Kuehne, A. B. Arslan, and T. Serre. The language of ac- tions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of Computer Vision and Pattern Recognition Conference (CVPR), 2014. 6, 7

    2014

  14. [22]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML,

  15. [23]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. 2, 3, 4, 5, 8

    2023

  16. [24]

    Swinbert: End-to-end transformers with sparse attention for video cap- tioning, 2022

    Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse attention for video cap- tioning, 2022. 7

    2022

  17. [25]

    Learning to recognize procedural activities with distant supervision

    Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with distant supervision. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 13853–13863,

  18. [26]

    Minicache: Kv cache com- pression in depth dimension for large language models

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffaria, and Bohan Zhuang. Minicache: Kv cache com- pression in depth dimension for large language models. In NIPS, 2024. 3

    2024

  19. [27]

    Vil- bert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vil- bert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2019. 1

    2019

  20. [28]

    Univl: A unified video and language pre-training model for multimodal understanding and generation, 2020

    Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation, 2020. 7

    2020

  21. [29]

    Howto100m: Learning a text-video embedding by watching hundred mil- lion narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Howto100m: Learning a text-video embedding by watching hundred mil- lion narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV...

    2019

  22. [30]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 3

    2023 arXiv

  23. [31]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, a...

    2022 arXiv

  24. [32]

    Keeping your eye on the ball: Tra- jectory attention in video transformers

    Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and Joao F Henriques. Keeping your eye on the ball: Tra- jectory attention in video transformers. Advances in neural information processing systems, 34:12493–12506...

    2021

  25. [33]

    Eva-clip: Improved training techniques for clip at scale

    Ledell Wu Quan Sun, Yuxin Fang, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 6

    2023 arXiv

  26. [34]

    Language models are unsuper- vised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsuper- vised multitask learners. 2019. Technical report. 1, 3

    2019

  27. [35]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  28. [36]

    Videobert: A joint model for video and language representation learning

    Chen Sun, Austin Myers, Carl V ondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. CoRR, abs/1904.01766,

    1904 arXiv

  29. [37]

    Plummer, Bryan Russell, and Kate Saenko

    Reuben Tan, Ximeng Sun, Ping Hu, Jui-Hsien Wang, Hanieh Deilamsalehy, Bryan A. Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-llm. In CVPR, 2024. 1, 3

    2024

  30. [38]

    Coin: A large-scale dataset for comprehensive instructional video analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), 2019. 6, 7

    2019

  31. [39]

    Llama: Open and efficient foundation lan- guage models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aure- lien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. Llama: Open and efficient foundation la...

    2023

  32. [40]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. CoRR, abs/1411.5726, 2014. 6

    2014 arXiv

  33. [41]

    Git: A generative image-to-text transformer for vision and language, 2022

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language, 2022. 7

    2022

  34. [42]

    Selective structured state-spaces for long-form video understanding

    Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In CVPR 2023, 2023. 3

    2023

  35. [43]

    Selective structured state-spaces for long-form video understanding

    Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6387–6397, 2023. 7

    2023

  36. [44]

    Temporal segment networks for action recognition in videos, 2017

    Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos, 2017. 7

    2017

  37. [45]

    Towards Long- Form Video Understanding

    Chao-Yuan Wu and Philipp Kr ¨ahenb¨uhl. Towards Long- Form Video Understanding. In CVPR, 2021. 1, 2, 6, 7

    2021

  38. [46]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv, 2023. 4

    2023

  39. [47]

    mplug-2: A modularized multi-modal foundation model across text, image and video, 2023

    Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, Guohai Xu, Ji Zhang, Songfang Huang, Fei Huang, and Jin- gren Zhou. mplug-2: A modularized multi-modal foundation model across text, image and video, 2023. 7

    2023

  40. [48]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5288–5296, 2016. 3, 4, 6, 7

    2016

  41. [49]

    Videococa: Video- text modeling with zero-shot transfer from contrastive cap- tioners, 2023

    Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, So- ham Ghosh, Yonghui Wu, and Jiahui Yu. Videococa: Video- text modeling with zero-shot transfer from contrastive cap- tioners, 2023. 7 10

    2023

  42. [50]

    Stacked attention networks for image question answering

    Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2016. 2

    2016

  43. [51]

    Scaling vision transformers, 2022

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lu- cas Beyer. Scaling vision transformers, 2022. 6

    2022

  44. [52]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding, 2023. 7

    2023

  45. [53]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 3

    2023 arXiv

  46. [54]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou andTianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R ´e, Clark Barrett andZhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In NIPS. NIPS, 2023. 3

    2023

  47. [55]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. In AAAI Conference on Artificial Intelligence, pages 7590–7598, 2018. 6 11

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.