Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Combining audio cues with shot-aware token compression lets a multimodal LLM localize video moments more accurately than prior systems.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:38 UTC pith:NA3HUZEU

load-bearing objection A credible incremental MLLM moment-retrieval paper—audio and shot compression give real but modest gains; one ablation passage is internally contradictory. the 4 major comments →

arxiv 2511.14143 v2 pith:NA3HUZEU submitted 2025-11-18 cs.CV cs.AI

SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM

An Yu , Weiheng Lu , Jian Li , Zhenfei Zhang , Yunhang Shen , Felix X.-F. Ye , Ming-Ching Chang This is my paper
classification cs.CV cs.AI
keywords video moment retrievalaudio-visual learningmultimodal large language modelsshot-aware token compressiontemporal groundingCharades-STAQVHighlightsLoRA fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that video moment retrieval -- finding the segment of an untrimmed video that matches a natural-language query -- is held back by two things: models rely on vision alone, and they jam many redundant frames into the language model. SMART adds an audio branch (BEATs) and a shot-aware token compression step that keeps only the visually dynamic tokens per shot. On Charades-STA and QVHighlights, this raises recall and mAP above previous state-of-the-art results, with gains like +2.59 in R1@0.7 on Charades-STA. If right, it means audio is a cheap, untapped signal for temporal grounding, and shot structure is a natural unit for compression.

Core claim

On its own terms, the paper claims that an MLLM-based moment-retrieval pipeline can achieve state-of-the-art accuracy by jointly using audio and a shot-aware token compression scheme. The model encodes frames with EVA-CLIP and audio with BEATs, concatenates the projected features into a prompt (with timestamps for visual tokens), and fine-tunes a LLM with LoRA. The key claim is that compressing non-keyframe tokens within detected shots -- keeping only tokens whose variance across frames is high -- removes redundancy without losing the fine-grained temporal cues the queries need. The reported results set new high marks, e.g., R1@0.5 of 78.15 and R1@0.7 of 63.16 on QVHighlights test, and R1@0.

What carries the argument

The load-bearing component is Shot-aware Token Compression (STC), which first segments the video into shots using TransNetV2, selects keyframes by the smoothed L2 norm of inter-frame patch differences, then, within each shot, computes variance of Q-Former output tokens across the temporal axis. High-variance tokens (dynamic content) are retained; low-variance tokens (static background) are discarded from non-keyframes. This is what lets the model feed long videos into a fixed LLM context with less quadratic-attention cost. The other component is the audio branch: BEATs embeddings are average-pooled, projected, and appended to the visual tokens in the prompt, following an Overall Concatenatio

Load-bearing premise

The claim that tokens that barely change across frames within a shot are redundant and can be safely discarded, so that no query-relevant information ever lives only in a low-variance token.

What would settle it

A video where the queried moment is identified by a small static object or text (e.g., 'the sign that says OPEN') that appears only in a non-keyframe. If SMART misses it while a no-compression baseline finds it, the central assumption fails. Concretely, construct such videos, run SMART with and without STC, and compare recall on those queries.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Moment retrieval on long, multi-minute videos becomes more feasible because STC shrinks the visual token sequence while keeping dynamic details.
  • Queries whose answer depends on speech or ambient sound -- 'talking', a reporter naming a location -- can be grounded when audio is appended as a global context.
  • The same audio-plus-compression recipe can be transferred to other video grounding tasks, such as highlight detection or dense video captioning.
  • Because only 0.63% of parameters are updated via LoRA, the gains are attributable to the input representation and prompt structure, not to a larger model.
  • Datasets that omit audio (like silent ActivityNet) are not adequate to evaluate multimodal moment retrieval, so the field may need audio-inclusive benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If STC is the reason for the gains, a natural extension is to make token selection query-aware: keep tokens that are attended to by the query text, rather than only variance-based tokens, which could recover subtle static details.
  • The success of simply concatenating pooled audio suggests that fine-grained audio-text alignment may be unnecessary for retrieval, hinting that global audio context carries most of the signal.
  • Variance-based token pruning is a general principle that could be applied to other vision-language tasks (e.g., video QA) where long videos exceed context windows.
  • The paper's own qualitative example of 'two men' being missed by compression hints that variance is not a perfect proxy for query relevance; a testable fix is to keep a small residual of low-variance tokens.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SMART, an MLLM-based video moment retrieval framework that integrates audio via BEATs and a shot-aware token compression (STC) mechanism. STC detects shot boundaries with TransNetV2, identifies keyframes by frame-difference magnitudes, and discards low-variance Q-Former tokens from non-keyframes within each shot, aiming to reduce sequence length while preserving fine-grained temporal cues. The model is evaluated on Charades-STA and QVHighlights, reporting state-of-the-art results (e.g., R1@0.5 of 78.15 on QVHighlights test, 72.26 on Charades-STA). Ablations on QVHighlights decompose the gains into audio integration and STC, and additional studies examine audio concatenation strategies, audio length L, and frame/keyframe counts N and k.

Significance. If the empirical results are robust, SMART makes a meaningful contribution: it demonstrates that audio cues can improve moment retrieval in an MLLM framework and that shot-level token compression can reduce sequence length with only modest accuracy changes. The paper provides extensive ablations (Tables II–III, Figs. 5–6), qualitative examples, and an honest limitations section. The methodological components are simple and reproducible in principle. However, the reported gains are small (1–2 points), and the lack of statistical validation plus an internal inconsistency in the audio-length study undermine confidence in the attribution of these gains. The contribution is incremental but within the scope of a top-tier venue if the issues are addressed.

major comments (4)
  1. [§IV-C, Study of Audio Compression Length L] The text states that this study excludes shot-aware token compression, yet the reported results at L=150 (R1@0.5 78.65, R1@0.7 65.03, mAP@0.5 70.46, mAP@0.75 56.72) are identical to row (d) of Table II, which is the full SMART model with both audio and STC. If STC were excluded, the L=150 numbers should match row (b) of Table II (77.23, 64.52, 70.02, 56.66). This contradiction means the study does not isolate the effect of L, and the conclusion that L=150 is optimal for the overall-concatenation strategy is unsupported. Please clarify which variant was actually used and correct the reported numbers accordingly.
  2. [Table II] The ablations report single-run numbers without error bars, standard deviations, or significance tests. The individual gains are +0.71 R1@0.5 for audio (row b vs a) and +0.51 R1@0.5 for STC (row c vs a). Given the stochasticity of LLM fine-tuning, these small differences may be within run-to-run noise. The central claim that both components individually contribute—and that the full model (+2.13) exceeds the sum of parts—requires repeated runs (e.g., 3–5 seeds) or at least a paired evaluation to demonstrate significance. Without this, the attribution of the SOTA improvement to the proposed mechanisms is not established.
  3. [§III-C, Stage 2 and §V Limitations] The low-variance-token discarding rule rests on the premise that low variance implies redundancy and that the information is recoverable from keyframes. The paper's own Limitations section concedes that STC 'can still discard subtle visual details in highly dynamic sequences,' and the qualitative Example 2 discusses a failure of a baseline due to losing 'two men' during compression. The ablation only reports aggregate metrics; it does not stratify by query type (e.g., queries requiring static objects, background text, or subtle visual details). A per-query or error-bar analysis bounding the risk of STC-induced failures is needed to substantiate the claim that STC 'preserves fine-grained temporal cues' rather than merely trading one type of error for another.
  4. [§IV-C, Figs. 5–6 and Table I] The optimal hyperparameters L=150, N=80, and k=32 are chosen from validation-set curves on QVHighlights, and the same validation set is used for the ablation tables and the comparison in Table I. While selection on validation is standard, reporting validation numbers as the primary evidence for component contributions and SOTA status introduces optimistic bias. The test-set results in Table I are less affected, but the ablations in Tables II–III and the hyperparameter studies should be reported on a held-out split (or at least acknowledge that validation numbers may be overfit). Please state explicitly whether any held-out validation was used for early stopping or model selection.
minor comments (5)
  1. [Table I, Charades-STA rows] For Charades-STA, the text claims 'clear performance gains' and 'new best mIoU,' but SMART's mAP@0.5 (69.47) is slightly lower than LLaVA-MR (69.96). Please temper the wording to avoid implying improvement on all metrics, or add a footnote explaining the trade-off.
  2. [Table III] The 'Audio-Visual Fusion' row drops to 24.84 R1@0.5. This is a drastic degradation that seems out of proportion to 'large number of added parameters.' Please provide details on the fusion architecture and any debugging; as presented, this result may indicate a configuration issue rather than a genuine comparison.
  3. [§III-C, Stage 1] The keyframe identification computes inter-frame differences on 'visual features F_v', but the notation is not fully specified—are these raw EVA-CLIP patch embeddings or Q-Former outputs? Clarify what exactly is differenced and how the L2 norm is taken over the patch dimension.
  4. [Equation (1)] The input sequence notation uses 't_i' for time tokens but later mentions 'V_E' and 'A_E' separators. The sequence shown omits the audio time alignment; please define whether audio tokens have position information or are purely sequential, as this affects reproducibility.
  5. [References] References [18] and [10] appear twice (LLaVA-MR and Mr.BLIP are cited both in the main text and in the references with different spellings, e.g., 'LLaV A-MR' vs 'LLaVA-MR'). Please unify the spelling and deduplicate if necessary.

Circularity Check

0 steps flagged

No equation-level circularity; central SMART gains are empirical benchmark results. Minor self-citation of LLaVA-MR is not load-bearing; validation-set tuning/reporting inconsistencies are evaluation concerns, not circular reasoning.

full rationale

The SMART paper's central claims are empirical benchmark results rather than derived quantities: the architecture feeds Q-Former visual tokens and BEATs audio tokens into a LoRA-tuned LLM, and its claimed improvements are read from Table I on public benchmarks. No equation in §III defines retrieval performance directly in terms of the method's hyperparameters or token-variance quantities, so there is no construction-level equivalence between inputs and outputs. The STC rule in §III-C uses motion magnitude and token variance to discard low-variance tokens from non-keyframes; whether this 'preserves fine-grained temporal cues' is an empirical claim tested by the ablation in Table II, not an assertion that follows by definition from variance. The paper's own limitation statement—'Shot-aware token compression reduces redundancy but can still discard subtle visual details in highly dynamic sequences'—is a correctness/robustness risk, not a circular reduction. The only self-citation, LLaVA-MR [18], shares authors with this paper but is used as a comparison baseline in Table I, not as the load-bearing justification for any design choice; the architectural inspiration Mr.BLIP [10] is an external work. Two evaluation concerns do not amount to circularity: (1) L, N, and k are tuned on the QVHighlights validation set and the same validation numbers are reported as the model's performance; (2) the §IV-C 'Study of Audio Compression Length L' text says the study was run 'excluding shot-aware token compression' yet reports the full-model numbers (R1@0.5 78.65, R1@0.7 65.03), which match Table II row (d). These are validation-selection and internal-consistency issues, not demonstrations that a prediction reduces to a fitted input. Honest non-finding: no significant circularity in the claimed derivation chain.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The central empirical claim rests on three domain assumptions about shot boundaries, token-variance redundancy, and audio complementarity, plus three validation-set-tuned hyperparameters (L=150, N=80, k=32). No new physical or conceptual entities are introduced, and the gains are not derived from first principles.

free parameters (3)
  • Audio pooling length L = 150 (QVHighlights)
    Chosen by validation sweep in Fig. 5; reported as optimal for all metrics.
  • Frame count N = 80 (QVHighlights)
    Chosen by validation sweep in Fig. 6; reported as optimal trade-off between richness and sequence length.
  • Keyframe count k = 32 (QVHighlights)
    Chosen by validation sweep in Fig. 6; too few or too many keyframes reduce R1@0.5.
axioms (3)
  • domain assumption TransNetV2 shot boundaries are accurate and semantically meaningful for this task.
    Invoked in §III-C to segment videos into shots before compression; errors in shot detection could group unrelated frames or split coherent actions.
  • domain assumption Low-variance Q-Former tokens in non-keyframes are redundant and can be discarded without losing query-relevant information.
    Core of STC in §III-C Stage 2. The paper's own Limitations admits compression “can still discard subtle visual details in highly dynamic sequences.”
  • domain assumption BEATs audio embeddings are complementary to visual cues and aligned with semantic query content.
    Audio integration assumes acoustic cues help grounding, but no per-query analysis is provided; the paper excludes datasets without audio (ActivityNet) precisely because the method depends on this premise.

pith-pipeline@v1.3.0-alltime-deepseek · 13395 in / 12967 out tokens · 127067 ms · 2026-08-03T21:38:40.503967+00:00 · methodology

0 comments
read the original abstract

Video Moment Retrieval is a task in video understanding that aims to localize a specific temporal segment in an untrimmed video based on a natural language query. Despite recent progress in moment retrieval from videos using both traditional techniques and Multimodal Large Language Models (MLLM), most existing methods still rely on coarse temporal understanding and a single visual modality, limiting performance on complex videos. To address this, we introduce \textit{S}hot-aware \textit{M}ultimodal \textit{A}udio-enhanced \textit{R}etrieval of \textit{T}emporal \textit{S}egments (SMART), an MLLM-based framework that integrates audio cues and leverages shot-level temporal structure. SMART enriches multimodal representations by combining audio and visual features while applying \textbf{Shot-aware Token Compression}, which selectively retains high-information tokens within each shot to reduce redundancy and preserve fine-grained temporal details. We also refine prompt design to better utilize audio-visual cues. Evaluations on Charades-STA and QVHighlights show that SMART achieves significant improvements over state-of-the-art methods, including a 1.61\% increase in R1@0.5 and 2.59\% gain in R1@0.7 on Charades-STA.

Figures

Figures reproduced from arXiv: 2511.14143 by An Yu, Felix X.-F. Ye, Jian Li, Ming-Ching Chang, Weiheng Lu, Yunhang Shen, Zhenfei Zhang.

Figure 1
Figure 1. Figure 1: Overview: (a) Traditional models rely on low-level visual features and generalize poorly. (b) MLLM-based models improve semantic understanding but ignore audio and incur high computational cost. (c) Our SMART model integrates audio and shot-coherent token compression for efficient, accurate, and generalizable multimodal moment retrieval. traditional approaches typically adopt Transformer-based ar￾chitectur… view at source ↗
Figure 2
Figure 2. Figure 2: The Shot-aware Multimodal Audio-enhanced Retrieval of Temporal Segments (SMART) architecture: (a) The SMART pipeline integrates a pretrained MLLM with frozen visual and audio encoders and a lightweight LoRA-tuned LLM to predict temporal segments relevant to a query. Visual and audio features are projected, temporally encoded, and concatenated into a unified multimodal prompt, with shot-aware token compress… view at source ↗
Figure 3
Figure 3. Figure 3: Shot-aware token compression: Key frames are detected via inter￾frame differences. Within each shot, high-variance tokens from dynamic regions are preserved, while low-variance tokens from non-key frames are discarded, effectively minimizing redundancy and retaining essential temporal cues for accurate moment retrieval. the vision-language framework. Raw audio is extracted and resampled to a standardized 1… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results on QVHighlights [4]: This figure visualizes the predicted and ground-truth segments for query events. SMART outperforms the visual-text MLLM baseline by preserving shot-level consistency and leveraging audio cues for fine-grained temporal understanding. See text for details. TABLE II COMPREHENSIVE ABLATION STUDY OF THE TWO KEY COMPONENTS IN SMART ON THE QVHIGHLIGHTS DATASET. EACH MODULE… view at source ↗
Figure 5
Figure 5. Figure 5: Impact of audio compression length L of the Overall Concatenation strategy on QVHighlights, where the circular marker at L = 150 indicates the optimal setting for all metrics [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Determining the optimal frame number N and top-k keyframe for Shot-aware Token Compression (STC) on QVHighlights, where red stars mark the best configuration (N = 80, k = 32). L that balances context preservation with sequence efficiency. Performance was measured by R1@0.5, R1@0.7, mAP@0.5, and mAP@0.75. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. StoryTR: Narrative-Centric Video Temporal Retrieval with Theory of Mind Reasoning

    cs.AI 2026-04 unverdicted novelty 7.0

    StoryTR is a new benchmark and agentic data pipeline that adds explicit Theory of Mind reasoning chains to train smaller video retrieval models, yielding a 15% relative IoU gain over larger baselines on narrative content.

Reference graph

Works this paper leans on

58 extracted references · 23 linked inside Pith · cited by 1 Pith paper

  1. [1]

    TALL: Temporal activity lo- calization via language query,

    J. Gao, C. Sun, Z. Yang, and R. Nevatia, “TALL: Temporal activity lo- calization via language query,” inProceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 5267–5275

    2017

  2. [2]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  3. [3]

    TransNet V2: An effective deep network architecture for fast shot transition detection,

    T. Sou ˇcek and J. Loko ˇc, “TransNet V2: An effective deep network architecture for fast shot transition detection,” inProc. of the 32nd ACM International Conference on Multimedia (ACM MM), pp. 11218–11221, 2024

    2024

  4. [4]

    Detecting moments and highlights in videos via natural language queries,

    J. Lei, T. L. Berg, and M. Bansal, “Detecting moments and highlights in videos via natural language queries,”Advances in Neural Information Processing Systems, vol. 34, pp. 11846–11858, 2021

    2021

  5. [5]

    UnLoc: A unified framework for video localization tasks,

    S. Yan, X. Xiong, A. Nagrani, A. Arnab, Z. Wang, W. Ge, D. Ross, and C. Schmid, “UnLoc: A unified framework for video localization tasks,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 13623–13633

    2023

  6. [6]

    Knowing where to focus: Event-aware transformer for video grounding,

    J. Jang, J. Park, J. Kim, H. Kwon, and K. Sohn, “Knowing where to focus: Event-aware transformer for video grounding,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 13846–13856

    2023

  7. [7]

    Query-dependent video representation for moment retrieval and highlight detection,

    W. Moon, S. Hyun, S. Park, D. Park, and J.-P. Heo, “Query-dependent video representation for moment retrieval and highlight detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 23023–23033

    2023

  8. [8]

    UniMD: Towards unifying mo- ment retrieval and temporal action detection,

    Y . Zeng, Y . Zhong, C. Feng, and L. Ma, “UniMD: Towards unifying mo- ment retrieval and temporal action detection,” inEuropean Conference on Computer Vision (ECCV), 2024, pp. 286–304

    2024

  9. [9]

    Correlation-guided query-dependency calibration for video temporal grounding,

    W. Moon, S. Hyun, S. Lee, and J.-P. Heo, “Correlation-guided query-dependency calibration for video temporal grounding,” arXiv:2311.08835, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    The surprising effectiveness of multimodal large language models for video moment retrieval,

    B. Meinardus, A. Batra, A. Rohrbach, and M. Rohrbach, “The surprising effectiveness of multimodal large language models for video moment retrieval,”arXiv:2406.18113, 2024

    arXiv 2024

  11. [11]

    R 2- Tuning: Efficient image-to-video transfer learning for video temporal grounding,

    Y . Liu, J. He, W. Li, J. Kim, D. Wei, H. Pfister, and C.-W. Chen, “R 2- Tuning: Efficient image-to-video transfer learning for video temporal grounding,” inEuropean Conference on Computer Vision (ECCV), 2024, pp. 421–438

    2024

  12. [12]

    Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection,

    Y . Xiao, Z. Luo, Y . Liu, Y . Ma, H. Bian, Y . Ji, Y . Yang, and X. Li, “Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 18709–18719

    2024

  13. [13]

    VideoLights: Feature refinement and cross-task alignment transformer for joint video highlight detection and moment retrieval,

    D. Paul, M. R. Parvez, N. Mohammed, and S. Rahman, “VideoLights: Feature refinement and cross-task alignment transformer for joint video highlight detection and moment retrieval,”arXiv:2412.01558, 2024

    arXiv 2024

  14. [14]

    FlashVTG: Feature layering and adaptive score handling network for video temporal grounding,

    Z. Cao, B. Zhang, H. Du, X. Yu, X. Li, and S. Wang, “FlashVTG: Feature layering and adaptive score handling network for video temporal grounding,”arXiv:2412.13441, 2024

    Pith/arXiv arXiv 2024

  15. [16]

    InternVideo2: Scaling foundation models for mul- timodal video understanding,

    Y . Wanget al., “InternVideo2: Scaling foundation models for mul- timodal video understanding,” inEuropean Conference on Computer Vision (ECCV), 2024, pp. 396–416

    2024

  16. [17]

    UniVTG: Towards unified video-language temporal grounding,

    K. Q. Lin, P. Zhang, J. Chen, S. Pramanick, D. Gao, A. J. Wang, R. Yan, and M. Z. Shou, “UniVTG: Towards unified video-language temporal grounding,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 2794–2804. 9

    2023

  17. [18]

    LLaV A- MR: Large language-and-vision assistant for video moment retrieval,

    W. Lu, J. Li, A. Yu, M.-C. Chang, S. Ji, and M. Xia, “LLaV A- MR: Large language-and-vision assistant for video moment retrieval,” arXiv:2411.14505, 2024

    Pith/arXiv arXiv 2024

  18. [19]

    TransNet V2: An effective deep network architecture for fast shot transition detection,

    T. Soucek and J. Lokoc, “TransNet V2: An effective deep network architecture for fast shot transition detection,” inProceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 11218– 11221

    2024

  19. [20]

    Localizing moments in video with natural language,

    L. A. Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, and B. Russell, “Localizing moments in video with natural language,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 5803–5812

    2017

  20. [21]

    Semantic conditioned dynamic modulation for temporal sentence grounding in videos,

    Y . Yuan, L. Ma, J. Wang, W. Liu, and W. Zhu, “Semantic conditioned dynamic modulation for temporal sentence grounding in videos,”Ad- vances in Neural Information Processing Systems, vol. 32, 2019

    2019

  21. [22]

    MAN: Moment alignment network for natural language moment retrieval via iterative graph adjustment,

    D. Zhang, X. Dai, X. Wang, Y .-F. Wang, and L. S. Davis, “MAN: Moment alignment network for natural language moment retrieval via iterative graph adjustment,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 1247– 1257

    2019

  22. [23]

    Temporal convolutional networks for action segmentation and detection,

    C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager, “Temporal convolutional networks for action segmentation and detection,” inPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 156–165

    2017

  23. [24]

    Temporal action detection with structured segment networks,

    Y . Zhao, Y . Xiong, L. Wang, Z. Wu, X. Tang, and D. Lin, “Temporal action detection with structured segment networks,” inProceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2914–2923

    2017

  24. [25]

    Graph convolutional networks for temporal action localization,

    R. Zeng, W. Huang, M. Tan, Y . Rong, P. Zhao, J. Huang, and C. Gan, “Graph convolutional networks for temporal action localization,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019, pp. 7094–7103

    2019

  25. [26]

    ActionFormer: Localizing moments of actions with transformers,

    C.-L. Zhang, J. Wu, and Y . Li, “ActionFormer: Localizing moments of actions with transformers,” inEuropean Conference on Computer Vision (ECCV), 2022, pp. 492–510

    2022

  26. [27]

    UMT: Unified multi-modal transformers for joint video moment retrieval and highlight detection,

    Y . Liu, S. Li, Y . Wu, C.-W. Chen, Y . Shan, and X. Qie, “UMT: Unified multi-modal transformers for joint video moment retrieval and highlight detection,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 3042–3051

    2022

  27. [28]

    EV A-CLIP: Improved training techniques for CLIP at scale,

    Q. Sun, Y . Fang, L. Wu, X. Wang, and Y . Cao, “EV A-CLIP: Improved training techniques for CLIP at scale,”arXiv:2303.15389, 2023

    Pith/arXiv arXiv 2023

  28. [29]

    BEATs: Audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei, “BEATs: Audio pre-training with acoustic tokenizers,” arXiv:2212.09058, 2022

    Pith/arXiv arXiv 2022

  29. [30]

    Flamingo: A visual language model for few-shot learning,

    J.-B. Alayracet al., “Flamingo: A visual language model for few-shot learning,”Advances in Neural Information Processing Systems, vol. 35, pp. 23716–23736, 2022

    2022

  30. [31]

    OpenFlamingo: An open-source frame- work for training large autoregressive vision-language models,

    A. Awadallaet al., “OpenFlamingo: An open-source frame- work for training large autoregressive vision-language models,” arXiv:2308.01390, 2023

    Pith/arXiv arXiv 2023

  31. [32]

    Mimic-It: Multi-modal in-context instruction tuning,

    B. Liet al., “Mimic-It: Multi-modal in-context instruction tuning,” arXiv:2306.05425, 2023

    Pith/arXiv arXiv 2023

  32. [33]

    LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention,

    R. Zhanget al., “LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention,”arXiv:2303.16199, 2023

    Pith/arXiv arXiv 2023

  33. [34]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning,

    W. Daiet al., “InstructBLIP: Towards general-purpose vision-language models with instruction tuning,”arXiv:2305.06500, 2023

    Pith/arXiv arXiv 2023

  34. [35]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” arXiv:2304.08485, 2023

    Pith/arXiv arXiv 2023

  35. [36]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “MiniGPT-4: Enhancing vision-language understanding with advanced large language models,”arXiv:2304.10592, 2023

    Pith/arXiv arXiv 2023

  36. [37]

    VideoChat: Chat-centric video understanding,

    K. Liet al., “VideoChat: Chat-centric video understanding,” arXiv:2305.06355, 2024

    Pith/arXiv arXiv 2024

  37. [38]

    Valley: Video assistant with large language model enhanced ability,

    R. Luoet al., “Valley: Video assistant with large language model enhanced ability,”arXiv:2306.07207, 2023

    Pith/arXiv arXiv 2023

  38. [39]

    Video-ChatGPT: Towards detailed video understanding via large vision and language models,

    M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video-ChatGPT: Towards detailed video understanding via large vision and language models,”arXiv:2306.05424, 2024

    Pith/arXiv arXiv 2024

  39. [40]

    Video-LLaMA: An instruction- tuned audio-visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-LLaMA: An instruction- tuned audio-visual language model for video understanding,” arXiv:2306.02858, 2023

    Pith/arXiv arXiv 2023

  40. [41]

    VideoLLaMA 2: Advancing spatial-temporal modeling and audio understanding in video-LLMs,

    Z. Chenget al., “VideoLLaMA 2: Advancing spatial-temporal modeling and audio understanding in video-LLMs,”arXiv:2406.07476, 2024

    Pith/arXiv arXiv 2024

  41. [42]

    PandaGPT: One model to instruction-follow them all,

    Y . Suet al., “PandaGPT: One model to instruction-follow them all,” arXiv:2305.16355, 2023

    Pith/arXiv arXiv 2023

  42. [43]

    NeXT-GPT: Any-to-any multimodal LLM,

    S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua, “NeXT-GPT: Any-to-any multimodal LLM,” inProceedings of the International Conference on Machine Learning (ICML), 2024, pp. 53366–53397. (PMLR, vol. 235.)

    2024

  43. [44]

    LoRA: Low-rank adaptation of large language models,

    E. J. Huet al., “LoRA: Low-rank adaptation of large language models,” arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  44. [45]

    Dense- captioning events in videos,

    R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles, “Dense- captioning events in videos,” inProceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 706–715

    2017

  45. [46]

    Efficient multimodal large language models: A survey,

    Y . Jinet al., “Efficient multimodal large language models: A survey,” arXiv:2405.10739, 2024

    arXiv 2024

  46. [47]

    A survey on benchmarks of multimodal large language models,

    J. Liet al., “A survey on benchmarks of multimodal large language models,”arXiv:2408.08632, 2024

    Pith/arXiv arXiv 2024

  47. [48]

    Context-enhanced video moment retrieval with large language models,

    W. Liu, B. Miao, J. Cao, X. Zhu, B. Liu, M. Nasim, and A. Mian, “Context-enhanced video moment retrieval with large language models,” arXiv:2405.12540, 2024

    Pith/arXiv arXiv 2024

  48. [49]

    VideoAgent: Long- form video understanding with large language model as agent,

    X. Wang, Y . Zhang, O. Zohar, and S. Yeung-Levy, “VideoAgent: Long- form video understanding with large language model as agent,” in European Conference on Computer Vision (ECCV), 2024, pp. 58–76

    2024

  49. [50]

    CenterCLIP: Token clus- tering for efficient text-video retrieval,

    S. Zhao, L. Zhu, X. Wang, and Y . Yang, “CenterCLIP: Token clus- tering for efficient text-video retrieval,” inProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022, pp. 970–981

    2022

  50. [51]

    Reducing the footprint of multi- vector retrieval with minimal performance impact via token pooling,

    B. Clavi ´e, A. Chaffin, and G. Adams, “Reducing the footprint of multi- vector retrieval with minimal performance impact via token pooling,” arXiv:2409.14683, 2024

    Pith/arXiv arXiv 2024

  51. [52]

    Fast learning of temporal action proposal via dense bound- ary generator,

    C. Linet al., “Fast learning of temporal action proposal via dense bound- ary generator,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 34, 2020, pp. 11499–11506

    2020

  52. [53]

    WavChat: A survey of spoken dialogue models,

    S. Jiet al., “WavChat: A survey of spoken dialogue models,” arXiv:2411.13577, 2024

    Pith/arXiv arXiv 2024

  53. [54]

    Coherent multi-sentence video description with variable level of detail,

    A. Rohrbach, M. Rohrbach, W. Qiu, A. Friedrich, M. Pinkal, and B. Schiele, “Coherent multi-sentence video description with variable level of detail,” inGerman Conference on Pattern Recognition (GCPR), 2014, pp. 184–195

    2014

  54. [55]

    MAD: A scalable dataset for language grounding in videos from movie audio descriptions,

    M. Soldan, A. Pardo, J. L. Alc ´azar, F. Caba, C. Zhao, S. Giancola, and B. Ghanem, “MAD: A scalable dataset for language grounding in videos from movie audio descriptions,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5026–5035

    2022

  55. [56]

    Self-chained image-language model for video localization and question answering,

    S. Yu, J. Cho, P. Yadav, and M. Bansal, “Self-chained image-language model for video localization and question answering,”Advances in Neural Information Processing Systems, vol. 36, pp. 76749–76771, 2023

    2023

  56. [57]

    In- sightVideo: Toward hierarchical video content organization for efficient browsing, summarization and retrieval,

    X. Zhu, A. K. Elmagarmid, X. Xue, L. Wu, and A. C. Catlin, “In- sightVideo: Toward hierarchical video content organization for efficient browsing, summarization and retrieval,”IEEE Transactions on Multime- dia, vol. 7, no. 4, pp. 648–666, 2005

    2005

  57. [58]

    A learned lexicon-driven paradigm for interactive video retrieval,

    C. G. M. Snoek, M. Worring, D. C. Koelma, and A. W. M. Smeulders, “A learned lexicon-driven paradigm for interactive video retrieval,”IEEE Transactions on Multimedia, vol. 9, no. 2, pp. 280–292, 2007. ChatGPT said:

    2007

  58. [59]

    Cross-modality knowledge calibration network for video corpus moment retrieval,

    T. Chen, W. Wang, Z. Jiang, R. Li, and B. Wang, “Cross-modality knowledge calibration network for video corpus moment retrieval,”IEEE Transactions on Multimedia, vol. 26, pp. 3799–3813, 2023

    2023