Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TimeExpert claims that routing timestamp, saliency, and caption tokens to different experts in a mixture-of-experts decoder improves video temporal grounding across three tasks while activating fewer parameters.

desk verdict Table 5 undercuts the paper's own central claim: TimeExpert's dynamic routing matches a static top-6 MoE on every metric, so the reported gains are plausibly noise. read the letter →

arxiv 2508.01699 v1 pith:TS6KWSIW submitted 2025-08-03 cs.CV

Zuhao Yang , Yingchen Yu , Yunqing Zhao , Shijian Lu , Song Bai This is my paper
classification cs.CV
keywords videotemporalgroundingmixtureofexpertslargelanguagemodelsmomentretrievaldensecaptioninghighlightdetectiondynamicroutingtasktokenspecialization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video temporal grounding asks a model to turn a video and a text query into a structured list of events, each with a timestamp, a saliency score, and a caption. TimeExpert's thesis is that these three output types are different enough that a single shared parameter path hurts performance, so it routes each type of token to a different specialist inside a mixture-of-experts decoder. The paper reports state-of-the-art zero-shot results on dense video captioning (YouCook2), moment retrieval (Charades-STA), and highlight detection (QVHighlights), and improved fine-tuned results on the first two, all with fewer activated parameters than dense 7B baselines. A sympathetic reader would take the contribution to be evidence that task-token-aware expert routing is a useful design axis for fine-grained video understanding.

What carries the argument

The load-bearing object is the task-token-aware dynamic MoE decoder. Its gating computes $s(x) = \cos(x, W_g)$ and then applies a sigmoid to $(s(x) + \alpha A_t)/(1+\alpha)$ before comparing with a learnable threshold $G$; the term $A_t$ is the historical activation rate of the token type, so timestamp tokens can develop a preference for experts that have served timestamps well. The routing layer records per-expert activation rates $R_E$ and the average embedding $R_S$ of tokens that activate no expert; when too many tokens go unmatched, a new expert is inserted with $W_{g,K+1} = R_S/\|R_S\|$ and $G_{K+1}=0$, and experts with activation below $\tau_{\min}$ are pruned. Finally, the auxiliary loss $L_{\text{aux}}$ combines a task-aware concentration term, which matches the fraction of activations an expert receives to the fraction of task tokens assigned to it, with a weight-norm regularizer that discourages one expert from dominating. Together these pieces are what let the model specialize timestamps, saliency scores, and captions instead of processing all tokens identically.

What would settle it

Rerun the same three-stage training with a fixed top-6 router and the same data across several random seeds, and compare the distributions of the seven metrics; if the 0.1 to 0.4 point differences fall inside one standard error, the central claim that dynamic expert allocation helps is not supported on these benchmarks.

Watch

Extended reading notes

Core claim

The paper's central claim is that token type—timestamp, saliency score, or caption text—should determine which experts process a token, rather than all tokens flowing through the same static pathway. To realize this, TimeExpert replaces the decoder of a multimodal mixture-of-experts Video-LLM with three coordinated mechanisms: a task-aware dynamic gating function that activates an expert only if the cosine similarity between token and expert exceeds a learnable threshold adjusted by that token type's historical activation rate; a token-adaptive routing scheme that logs per-type activation statistics, creates a new expert initialized from the mean embedding of tokens no expert will accept, and deletes experts whose activation stays below $\tau_{\min}$; and a task-dependent auxiliary loss that pulls frequently activated experts toward their preferred token types while regularizing expert weights. On the three VTG tasks, the paper reports the best zero-shot numbers among the compared Video-LLMs—for example 23.6 F1 on YouCook2, 42.8 R@1 IoU=0.5 on Charades-STA, and 46.9 HIT@1 on QVHighlights—using an average of roughly 9 to 15 activated experts per layer rather than a full 7B-parameter dense pass. The paper also shows the same data-recipe variant still beats the strongest prior baseline on all seven zero-shot metrics, which it reads as evidence that the architecture, not extra data, drives the improvement.

Load-bearing premise

The load-bearing premise is that the improvements come from the task-aware dynamic routing and auxiliary loss rather than from extra expert capacity or chance, but the paper's own ablation shows a vanilla mixture-of-experts model with six activated experts matching TimeExpert within 0.1 to 0.4 points on all seven metrics, with no error bars reported.

Editorial extensions

If this is right

  • On the three VTG benchmarks (YouCook2, Charades-STA, QVHighlights), TimeExpert reports the best zero-shot numbers among the compared Video-LLMs, with adaptive expert counts averaging 14.5, 9.1, and 11.8 and fewer activated parameters than dense 7B baselines.
  • Fine-tuned on YouCook2 and Charades-STA, it improves over the prior best model on both dense video captioning (CIDEr 39.0 vs 35.5, F1 33.5 vs 31.8) and moment retrieval (R@1 IoU=0.5 64.1 vs 61.7).
  • The same data-recipe variant, trained only on the prior model's data, still outperforms that baseline on all seven zero-shot metrics, so the paper argues the gains are not simply from a larger training set.
  • Ablation results show the token-adaptive routing and the task-dependent auxiliary loss each contribute, with the auxiliary loss's removal costing the most on highlight detection (mAP 28.1 vs 29.6, HIT@1 45.2 vs 46.9).
  • Increasing input frames from 64 to 128 improves moment retrieval and highlight detection, indicating that longer temporal context helps even with expert routing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gap between TimeExpert and a vanilla MoE with $k=6$ is 0.1 to 0.4 points on every one of the seven metrics, and the paper reports no error bars; the dynamic routing may be buying efficiency and stability more than accuracy, so the state-of-the-art claim should be read as tentative until repeated-seed variance is reported.
  • The paper's implicit prediction is that the benefit of task-token-aware routing grows with the share of non-text tokens in the output; a direct test would be to compare routing-aware versus fixed-$k$ MoE on tasks with very long timestamp or score sequences.
  • Because removing the separate time and score encoders makes the model fail to follow instructions, the structured-encoding component is validated by collapse rather than by graded ablation; an informative follow-up would be to train with shared encoders but separate heads to isolate the contribution of token-type separation.
  • The adaptive add-and-prune scheme generalizes beyond video: any structured generation task whose outputs mix coordinates, confidence scores, and natural language (for example GUI grounding or embodied instruction following) could use the same task-token-aware routing policy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes TimeExpert, a Mixture-of-Experts Video-LLM for video temporal grounding. The model builds on the ARIA backbone and introduces task-aware dynamic gating (Eq. 6), token-adaptive routing with expert addition/removal (Eqs. 7-8), separate encoders and decoding heads for timestamps, saliency scores, and captions, and a task-dependent auxiliary loss (Eq. 9). The authors evaluate zero-shot and fine-tuned performance on Dense Video Captioning (YouCook2), Moment Retrieval (Charades-STA), Video Highlight Detection (QVHighlights), and ActivityNet Captions, reporting state-of-the-art results over TRACE and other VTG-specific Video-LLMs. The central claim is that dynamic, task-token-aware expert allocation improves accuracy and computational efficiency.

Significance. If the attribution to dynamic routing were established, TimeExpert would be a useful demonstration that MoE specialization can be applied to structured output tokens in VTG, and the benchmark results over TRACE are potentially interesting. The paper has strengths: it evaluates across three VTG tasks and several benchmarks, includes a data-recipe variant to control for training data, and reports a number of ablations including frame count and number of experts. However, the key ablation in Table 5 shows that a vanilla top-k MoE with k=6 matches TimeExpert to within 0.1-0.4 points on all seven metrics, and no error bars or multiple seeds are reported. Because the paper's central scientific claim is that adaptive routing is the cause of the gains, the current evidence does not support that claim; the contribution is therefore currently more architectural than empirical.

major comments (3)
  1. [§4.3, Table 5] The bottom block of Table 5 directly undercuts the central attribution. Vanilla MoE with k=6 produces 2.5/8.2/23.5/42.8/20.2/29.6/46.9 on the seven metrics, while TimeExpert produces 2.5/8.2/23.6/42.8/20.3/29.6/46.9; the k=8 row is higher than TimeExpert on mAP (29.8 vs 29.6). No standard deviations, confidence intervals, or multiple-seed results are reported anywhere. Since the only control that isolates the adaptive mechanism is indistinguishable from static top-k MoE, the paper's claim that 'dynamic gating and adaptive routing' (Section 4.2) drive the improvements is unsupported. The 'w/o token-adaptive routing' row is not an adequate substitute: it changes the gating recipe rather than comparing a static-k model trained under the identical task encoders, auxiliary loss, and data recipe.
  2. [§4.2, Table 2 (efficiency claim)] The abstract and Section 4.2 claim 'increased computational efficiency' and fewer activated parameters. Section 4.2 reports that the adaptive-k variant activates on average 14.5, 9.1, and 11.8 experts on YouCook2, Charades-STA, and QVHighlights, respectively, which is more than the k=6 static-MoE control that matches its accuracy in Table 5. No FLOPs, inference latency, or memory measurements are reported. The only defensible efficiency statement is that TimeExpert activates fewer parameters than dense 7B baselines; the comparison to a static MoE of the same family, which is the relevant control, shows no efficiency benefit and possibly a cost.
  3. [Table 2, 'TimeExpert (TRACE's data)' row] The 'TimeExpert (TRACE's data)' row is intended to rule out data-recipe effects, but it does not isolate the routing mechanism. Relative to TRACE, the model still differs in the base LLM (ARIA), the separate task encoders/heads, and the MoE decoder; no static-k MoE trained on the same data and encoders is reported. Given that the gains over TRACE are small (0.0 to 2.3 points on the seven metrics in Table 2) and no variance estimates are given, the improvements over TRACE could be due to the backbone or data preprocessing rather than dynamic routing. I would ask for seed-level results and a static-k baseline under the identical recipe before accepting the attribution.
minor comments (6)
  1. [Table 5, 'w/o separate encoders' row] The 'w/o separate encoders' row is reported only as dashes, with the text saying the model 'fails to follow instructions.' A qualitative statement is not a quantitative ablation; please report the actual failure mode or a measurable proxy (e.g., formatting accuracy) and state whether evaluation was impossible.
  2. [Eq. (9)] The notation 'AePKj=1Aj' and 'NePKj=1Nj' is ambiguous; it appears to intend (Ae / ΣAj) and (Ne / ΣNj), but the fraction is not typeset. Please clarify.
  3. [§3.5, Training Data] The text states that data are 'systematically processed, filtered, and re-annotated' but no details, statistics, or filtering criteria are given; this makes the data recipe hard to reproduce.
  4. [Table 2, 'No. of Activated Parameters'] The table reports three different approximate values (≈5.9B/3.5B/4.8B) for the same model, and it is unclear why these differ across benchmarks and how they were computed; please clarify the accounting and consider reporting FLOPs or inference latency.
  5. [§3.2-§3.4, hyperparameters] The hyperparameters α, λ1, λ2, τmin, and the initial number of experts K are never given values, and no sensitivity analysis is reported for them; since the proposed gating mechanism depends on these, at least the chosen values should be stated.
  6. [Reproducibility] No code or checkpoints are provided, and the training recipe uses roughly 5.1M samples; releasing the model or code would materially aid reproducibility and would also help the community verify the small reported gains.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TimeExpert's claims are supported by external benchmark evaluations rather than by definitional or self-cited reductions.

full rationale

TimeExpert is an empirical systems paper. Its central claims—improved DVC/MR/VHD performance and reduced activated parameters—are evaluated on external benchmarks (YouCook2, Charades-STA, QVHighlights, ActivityNet Captions) against external baselines. No parameter is fitted to a subset of these benchmarks and then reported as a prediction of a closely related quantity; the model is trained on separate instruction data and evaluated zero-shot or fine-tuned on the benchmarks. The only passage that could look self-fulfilling is the task-dependent auxiliary loss (Eq. 9), which by construction drives expert activation shares to match task-token assignment shares, and Section 4.3 states that removing this loss 'improves expert specialization by forcing task-aware routing.' This is a design objective rather than a circular derivation: the loss enforces specialization by definition, but the paper's quantitative claims rest on the measured benchmark deltas in Table 5, not on the loss identity. The causal event modeling and token separator conventions are adopted from TRACE [16], an external prior work, with no author overlap; there is no load-bearing self-citation chain. Possible concerns about the static-MoE control in Table 5 (k=6 nearly matches TimeExpert, no error bars) are statistical-evidence concerns, not circularity, and therefore do not affect this score.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method introduces several user-set hyperparameters whose values are not reported, and it relies on the domain assumptions listed above. No new physical or ontological entities are postulated. The model builds on the ARIA MoE base and public video datasets.

free parameters (4)
  • alpha (scaling coefficient in Eq. 6) = not reported
    Controls the influence of the historical task activation rate on gating. No value or sensitivity analysis is provided.
  • lambda_1, lambda_2 in auxiliary loss (Eq. 9) = not reported
    Weights balancing task-aware concentration and activation regularization. Values are not given, and no ablation varies them.
  • tau_min (expert removal threshold in Eq. 8) = not reported
    Threshold below which experts are pruned. The chosen value is not reported.
  • Initial number of experts K = 64
    The MoE layer uses 64 experts. This is a design choice that affects capacity and routing behavior.
assumptions (3)
  • domain assumption Task tokens (timestamps, saliency scores, captions) are separable and benefit from processing by different experts.
    This is the central premise of Sections 3.2 to 3.4, motivating task-aware gating and the auxiliary loss. It is supported only by the Figure 4 visualization and ablations, not by independent evidence.
  • domain assumption The auxiliary loss Laux (Eq. 9) increases expert specialization without degrading language generation quality.
    Ablations show that removing the loss slightly hurts benchmark scores, but the mechanism by which it improves generation is assumed, and no generation quality metrics are reported separately.
  • domain assumption The filtered and re-annotated training data provide labels of sufficient quality for the evaluation benchmarks.
    Section 3.5 states that data were 'systematically processed, filtering, and re-annotating' but the filtering and re-annotation rules are not specified, so label quality is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding." pith.science (2026). https://pith.science/paper/TS6KWSIW

@misc{pith2026250801699,
  author       = {Pith},
  title        = {Pith review of: TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TS6KWSIW}},
  note         = {Machine review of arXiv:2508.01699}
}
read the original abstract

Video Temporal Grounding (VTG) aims to precisely identify video event segments in response to textual queries. The outputs of VTG tasks manifest as sequences of events, each defined by precise timestamps, saliency scores, and textual descriptions. Despite recent advances, a fundamental limitation persists in existing Video Large Language Models (Video-LLMs): they process all task tokens through identical and static pathways, failing to recognize that temporal localization, saliency assessment, and textual generation represent fundamentally distinct tasks requiring specialized processing. To address this, we introduce TimeExpert, a Mixture-of-Experts (MoE)-based Video-LLM that effectively decomposes VTG tasks by dynamically routing task-specific tokens (e.g., timestamps, saliency scores) to specialized experts, with increased computational efficiency. Our design choices enable precise handling of each subtask, leading to improved event modeling across diverse VTG applications. Extensive experiments demonstrate that TimeExpert consistently achieves state-of-the-art performance on various VTG tasks such as Dense Video Captioning, Moment Retrieval, and Video Highlight Detection.

Figures

Figures reproduced from arXiv: 2508.01699 by the authors.

Figure 1
Figure 1. Left: Video Temporal Grounding (VTG) is a fine-grained video understanding task that aims to accurately locate content along with event timestamps based on natural language queries. In this work, we mainly consider three major types of VTG tasks: (1) Moment Retrieval (MR), (2) Video Highlight Detection (VHD), and (3) Dense Video Captioning (DVC). The outputs of VTG often contain textual captions, timestamps, and sal… view at source ↗
Figure 2
Figure 2. A Glimpse of Comparison across VTG Approaches. (a): VTG-specific Video-LLM [16, 40] relies on a single static model with shared parameters for all tasks, limiting their ability to specialize in diverse VTG subtasks. (b): Vanilla MoE improves upon this by activating a fixed set (e.g., k=2) of experts, enabling some degree of task specialization. (c): Our TimeExpert takes a step further by implementing an adaptive rou… view at source ↗
Figure 3
Figure 3. Architecture Overview of TimeExpert. Our model leverages independent encoders and decoding heads to process time, score, and text inputs and outputs. The timestamps and saliency scores of sampled frames are encoded into special tokens and integrated into the corresponding visual tokens. During infer￾ence, the generated response follows a structured format, sequen￾tially incorporating time tokens, score tokens, and t… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: of expert assignments. In particular, we measure the ratio of activated task-specific tokens to the number of processed text tokens in that layer. Our findings reveal that certain experts, such as expert #7, consistently activate in re￾sponse to specific subtasks (e.g.…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Towards Temporal Compositional Reasoning in Long-Form Sports Videos

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    SportsTime benchmark and CoTR method improve multimodal AI's temporal compositional reasoning and evidence grounding in long-form sports videos.

  2. Towards Temporal Compositional Reasoning in Long-Form Sports Videos

    cs.CV 2026-04 conditional novelty 5.5 of 10

    SportsTime plus Chain-of-Time Reasoning (temporal-reward GRPO and anchor-observe-infer) modestly lifts open-ended sports VideoQA and step-wise temporal grounding over 4B–8B MLLM baselines.

Reference graph

Works this paper leans on

65 extracted references · 52 canonical work pages · cited by 1 Pith paper

  1. [1]

    Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In ACLW, pages 65–72, 2005. 6

    work page 2005

  2. [2]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961–970, 2015. 3, 5

    work page 2015

  3. [3]

    Sharegpt4video: Improving video understand- ing and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understand- ing and generation with better captions. In NeurIPS, pages 19472–19495, 2024. 5

    work page 2024

  4. [4]

    Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset. In NeurIPS, pages 72842–72866, 2023. 3

    work page 2023

  5. [5]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 2, 3

    arXiv 2024

  6. [6]

    Uni- fied scaling laws for routed language models

    Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Men- sch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Uni- fied scaling laws for routed language models. InICML, pages 4057–4086. PMLR, 2022. 4

    work page 2022

  7. [7]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021. 6

    work page 2021

  8. [8]

    Learning factored representations in a deep mixture of ex- perts

    David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of ex- perts. arXiv preprint arXiv:1312.4314, 2013. 3

    arXiv 2013

Show all 65 references
  1. [9]

    To- wards an empirical understanding of moe design choices

    Dongyang Fan, Bettina Messmer, and Martin Jaggi. To- wards an empirical understanding of moe design choices. In ICLRW, 2024. 4

    2024

  2. [10]

    Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity. JMLR, 23(120):1–39, 2022. 5

    2022

  3. [11]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In CVPR, pages 24108–24118, 2025. 2

    2025

  4. [12]

    Soda: Story oriented dense video captioning evaluation framework

    Soichiro Fujita, Tsutomu Hirao, Hidetaka Kamigaito, Man- abu Okumura, and Masaaki Nagata. Soda: Story oriented dense video captioning evaluation framework. In ECCV, pages 517–531. Springer, 2020. 6

    2020

  5. [13]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In ICCV, pages 5267–5275, 2017. 1, 2, 3, 6, 7, 8

    2017

  6. [14]

    Dynamic mixture of experts: An auto- tuning approach for efficient transformer models

    Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin. Dynamic mixture of experts: An auto- tuning approach for efficient transformer models. In ICLR,

  7. [15]

    Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding

    Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xi- aoying Tang, Dianbo Sui, Qingbin Liu, Xi Chen, and Kevin Zhao. Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding. InAAAI, 2025. 2, 3, 5, 6, 7

    2025

  8. [16]

    Trace: Temporal grounding video llm via causal event modeling

    Yongxin Guo, Jingyu Liu, Mingda Li, Xiaoying Tang, Qing- bin Liu, and Xi Chen. Trace: Temporal grounding video llm via causal event modeling. In ICLR, 2025. 1, 2, 3, 6, 7

    2025

  9. [17]

    Creating summaries from user videos

    Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. Creating summaries from user videos. In ECCV, pages 505–520. Springer, 2014. 3

    2014

  10. [18]

    Unleash the potential of clip for video highlight detection

    Donghoon Han, Seunghyeon Seo, Eunhwan Park, Seong-Uk Nam, and Nojun Kwak. Unleash the potential of clip for video highlight detection. In CVPR, pages 8275–8279, 2024. 2

    2024

  11. [19]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. In CVPR, pages 14271–14280, 2024. 2, 3, 6, 7

    2024

  12. [20]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. In ECCV, pages 202–218. Springer, 2024. 2, 3

    2024

  13. [21]

    Harder tasks need more experts: Dynamic routing in moe models

    Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder tasks need more experts: Dynamic routing in moe models. In ACL, 2024. 3, 4

    2024

  14. [22]

    Do you remember? dense video captioning with cross-modal memory retrieval

    Minkuk Kim, Hyeon Bae Kim, Jinyoung Moon, Jinwoo Choi, and Seong Tae Kim. Do you remember? dense video captioning with cross-modal memory retrieval. In CVPR, pages 13894–13904, 2024. 7

    2024

  15. [23]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, pages 706–715, 2017. 6, 7

    2017

  16. [24]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. In NeurIPS, pages 11846–11858, 2021. 1, 2, 3, 6, 8

    2021

  17. [25]

    Aria: An open multimodal native mixture-of- experts model

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of- experts model. arXiv preprint arXiv:2410.05993, 2024. 6

    2024 arXiv

  18. [26]

    Unmasked teacher: Towards training-efficient video foundation models

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, pages 19948–19960, 2023. 3

    2023

  19. [27]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In CVPR, pages 22195–22206, 2024. 2, 3

    2024

  20. [28]

    Uni- moe: Scaling unified multimodal llms with mixture of ex- perts

    Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni- moe: Scaling unified multimodal llms with mixture of ex- perts. TPAMI, 2025. 3

    2025

  21. [29]

    Video-llava: Learning united visual repre- sentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual repre- sentation by alignment before projection. In EMNLP, 2024. 2, 3

    2024

  22. [30]

    Univtg: Towards unified video- language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shra- man Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video- language temporal grounding. In ICCV, pages 2794–2804,

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, pages 34892–34916,

  24. [32]

    Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection

    Ye Liu, Siyuan Li, Yang Wu, Chang-Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In CVPR, pages 3042–3051, 2022. 2, 3

    2022

  25. [33]

    Valley: Video assistant with large language model enhanced ability

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207 ,

    arXiv

  26. [34]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. InACL,

  27. [35]

    Correlation-guided query-dependency calibra- tion for video temporal grounding

    WonJun Moon, Sangeek Hyun, SuBeen Lee, and Jae- Pil Heo. Correlation-guided query-dependency calibra- tion for video temporal grounding. arXiv preprint arXiv:2311.08835, 2023. 3

    2023 arXiv

  28. [36]

    Query-dependent video representa- tion for moment retrieval and highlight detection

    WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo. Query-dependent video representa- tion for moment retrieval and highlight detection. In CVPR, pages 23023–23033, 2023. 3

    2023

  29. [37]

    En- coding and controlling global semantics for long-form video question answering

    Thong Thanh Nguyen, Zhiyuan Hu, Xiaobao Wu, Cong- Duy T Nguyen, See-Kiong Ng, and Anh Tuan Luu. En- coding and controlling global semantics for long-form video question answering. In EMNLP, 2024. 5

    2024

  30. [38]

    Queryd: A video dataset with high-quality text and audio narrations

    Andreea-Maria Oncescu, Joao F Henriques, Yang Liu, An- drew Zisserman, and Samuel Albanie. Queryd: A video dataset with high-quality text and audio narrations. In ICASSP, pages 2265–2269. IEEE, 2021. 3

    2021

  31. [39]

    Momen- tor: Advancing video large language model with fine-grained temporal reasoning

    Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat- Seng Chua, Yueting Zhuang, and Siliang Tang. Momen- tor: Advancing video large language model with fine-grained temporal reasoning. In ICML, 2024. 3, 5, 6, 7

    2024

  32. [40]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. InCVPR, pages 14313– 14323, 2024. 1, 2, 3, 6, 7

    2024

  33. [41]

    Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer. In ICLR, 2017. 3

    2017

  34. [42]

    Tvsum: Summarizing web videos using titles

    Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tvsum: Summarizing web videos using titles. In CVPR, pages 5179–5187, 2015. 3

    2015

  35. [43]

    Coin: A large-scale dataset for comprehensive instructional video analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In CVPR, pages 1207–1216, 2019. 3

    2019

  36. [44]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In CVPR, pages 4566–4575, 2015. 6

    2015

  37. [45]

    Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models

    Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yu- fan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models. arXiv preprint arXiv:2410.03290, 2024. 2

    arXiv

  38. [46]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 3

    2024 arXiv

  39. [47]

    End-to-end dense video captioning with parallel decoding

    Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. In ICCV, pages 6847–6857, 2021. 7

    2021

  40. [48]

    Internvideo: General video foundation models via generative and discriminative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022. 3, 5

    2022 arXiv

  41. [49]

    Hawkeye: Training video- text llms for grounding text in videos

    Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan Zhao. Hawkeye: Training video- text llms for grounding text in videos. arXiv preprint arXiv:2403.10228, 2024. 2, 6

    2024 arXiv

  42. [50]

    Star: A benchmark for situated reasoning in real-world videos

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In NeurIPS, 2021. 5

    2021

  43. [51]

    A large cross- modal video retrieval dataset with reading comprehension

    Weijia Wu, Yuzhong Zhao, Zhuang Li, Jiahong Li, Hong Zhou, Mike Zheng Shou, and Xiang Bai. A large cross- modal video retrieval dataset with reading comprehension. Pattern Recognition, 157:110818, 2025. 5

    2025

  44. [52]

    Multi-head mixture-of-experts

    Xun Wu, Shaohan Huang, Wenhui Wang, Shuming Ma, Li Dong, and Furu Wei. Multi-head mixture-of-experts. In NeurIPS, pages 94073–94096, 2024. 5

    2024

  45. [53]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In CVPR, pages 9777–9786, 2021. 5

    2021

  46. [54]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In EMNLP, 2021. 3

    2021

  47. [55]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 2, 3

    2024 arXiv

  48. [56]

    M6-t: Exploring sparse expert models and beyond

    An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, et al. M6-t: Exploring sparse expert models and beyond. arXiv preprint arXiv:2105.15082, 2021. 4

    2021 arXiv

  49. [57]

    Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning. In CVPR, pages 10714–10726, 2023. 7

    2023

  50. [58]

    Xmoe: Sparse models with fine-grained and adaptive expert selection

    Yuanhang Yang, Shiyi Qi, Wenchao Gu, Chaozheng Wang, Cuiyun Gao, and Zenglin Xu. Xmoe: Sparse models with fine-grained and adaptive expert selection. In Findings of ACL, 2024. 3

    2024

  51. [59]

    Hierarchical video-moment retrieval and step-captioning

    Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Bar- las Oguz, Yashar Mehdad, and Mohit Bansal. Hierarchical video-moment retrieval and step-captioning. InCVPR, pages 23056–23065, 2023. 3

    2023

  52. [60]

    Unimd: Towards unifying moment retrieval and temporal ac- tion detection

    Yingsen Zeng, Yujie Zhong, Chengjian Feng, and Lin Ma. Unimd: Towards unifying moment retrieval and temporal ac- tion detection. In ECCV, pages 286–304. Springer, 2024. 3

    2024

  53. [61]

    Adamoe: Token-adaptive routing with null ex- perts for mixture-of-experts language models

    Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. Adamoe: Token-adaptive routing with null ex- perts for mixture-of-experts language models. In Findings of ACL, 2024. 3

    2024

  54. [62]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11975–11986, 2023. 6

    2023

  55. [63]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 5

    2024 arXiv

  56. [64]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018. 1, 2, 3, 6, 7, 8

    2018

  57. [65]

    St- moe: Designing stable and transferable sparse expert mod- els

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St- moe: Designing stable and transferable sparse expert mod- els. arXiv preprint arXiv:2202.08906, 2022. 5

    2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.