Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

DisTime: Distribution-based Time Representation for Video Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DisTime claims that one learnable token, decoded into a probability distribution over the timeline, gives Video-LLMs state-of-the-art temporal grounding without heavy modules.

desk verdict Solid distribution-based time token for Video-LLMs, but the zero-shot Charades-STA claim is undercut by likely benchmark leakage in the pseudo-label pipeline. read the letter →

arxiv 2505.24329 v2 pith:VUOGB27B submitted 2025-05-30 cs.CV

Yingsen Zeng , Zepeng Huang , Yujie Zhong , Chengjian Feng , Jie Hu , Lin Ma , Yang Liu This is my paper · ORCID
classification cs.CV
keywords videolargelanguagemodelstemporallocalizationmomentretrievaldistribution-basedtimetokendensecaptioninggroundedquestionansweringautomatedannotationgroundingdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DisTime sets out to give Video-LLMs a precise sense of when events happen. Its core proposal is to represent time with a single learnable token whose hidden state is decoded into two probability distributions over the normalized timeline, so start and end times emerge as expectation values rather than as text numerals or special-token classifications. On top of that, the authors build InternVid-TG, a 1.25M-event automatically annotated dataset produced by combining GPT-4o event captions with boundary proposals from three dedicated temporal localizers and an ensemble scoring step. They report state-of-the-art results on moment retrieval, dense video captioning, and grounded video QA—including a zero-shot 81.0 R@1 IoU=0.3 on Charades-STA with an 8B model—while keeping general video QA competitive.

What carries the argument

The Distribution-based Time Token (<TIME STAMP>) is the load-bearing object: one extra vocabulary token whose hidden state the time decoder (MLP + softmax) turns into start/end probability vectors over regmax+1 anchors at positions i/regmax on the normalized timeline. Predictions are the expectation values of those distributions, which keeps time continuous and lets the distribution shape absorb boundary uncertainty; its inverse, the time encoder, converts any timestamp into a Gaussian-smoothed discretized distribution and then an MLP token, so decoded times can be re-injected for the next autoregressive step. Training couples distribution focal loss, a 1D-IoU regression loss, and next-token prediction, with LoRA used for the LLM backbone.

What would settle it

Retrain DisTime on InternVid-TG annotations produced solely by TFVTG, the training-free annotator that has never seen Charades-STA or ANet-Caption, and measure Charades-STA R@1 IoU=0.3; if the score drops far below 78–81, the zero-shot result comes from the other annotators' benchmark exposure rather than the time-token mechanism. Also audit InternVid-TG for videos or queries overlapping the two benchmarks.

Watch

Extended reading notes

Core claim

The central claim is that the dominant failure modes of time in Video-LLMs—confusing time with numbers, under-trained special tokens, and boundary ambiguity—can all be handled by one Distribution-based Time Token. When the LLM emits <TIME STAMP>, its final-layer hidden state passes through a three-layer MLP with softmax to produce start and end distributions over regmax+1 uniformly spaced anchor bins on [0,1]; the predicted times are the anchor-weighted sums of those distributions. A companion time encoder maps any timestamp back into a Gaussian-regularized distribution and then an LLM embedding, and re-encodes the decoded times into the autoregressive context, giving what the paper calls Iterative Time Refinement. Trained with distribution focal loss plus a 1D-IoU regression loss and standard next-token prediction, the scheme lifts Charades-STA R@1 IoU=0.3 from 3.1 (baseline InternVL2.5-1B) to 78.1, and to 81.0 at 8B in a zero-shot setting, while also improving dense video captioning (YouCook2 F1 from 2.2 to 20.5 in ablation) and grounded VQA.

Load-bearing premise

The zero-shot scores on Charades-STA and ANet-Caption rest on the assumption that InternVid-TG pseudo-labels do not count as benchmark training—yet the three annotating models were trained on moment-retrieval data that includes those benchmarks, so the indirect supervision may be the real source of the gains.

Editorial extensions

If this is right

  • Video-LLMs can express precise continuous timestamps with a single added token, avoiding text-number confusion and the long-tail problem of many sparse special tokens.
  • The same time token transfers across backbone families—InternVL2.5 (1B, 8B) and LLaVA-OneVision (7B)—so the mechanism is model-agnostic rather than tied to one architecture.
  • Temporal supervision can be scaled automatically: 1.25M events over 179k videos, roughly 55 times ActivityNet-Caption, generated without shot-boundary or fixed-interval constraints.
  • Zero-shot moment retrieval on Charades-STA (81.0 R@1 IoU=0.3 at 8B) surpasses dedicated grounding models that were fine-tuned on the benchmark.
  • Iterative time re-encoding gives the largest gains in high-precision metrics; in ablations Charades-STA R@1 IoU=0.7 improves by 3 points and YouCook2 CIDEr by 11.6.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the zero-shot number survives a data-leakage audit, the same distribution-decoder pattern should generalize to other continuous quantities that LLMs currently type as text—bounding-box coordinates, audio timestamps, or confidence scores.
  • The Gaussian re-encoding of decoded times behaves like a consistency regularizer between decoder and encoder; using learned anchors instead of the fixed i/regmax grid is a natural next probe.
  • Because InternVid-TG pseudo-labels are selected among three annotators, the dataset's ceiling is the union of those annotators; training on low-confidence ensemble cases may be where the model learns beyond any single annotator.
  • Charades-STA results are reached with only 16 sampled frames, suggesting the benchmark's queries are answerable from coarse temporal layout; the same method on denser sampling might close the ANet-Caption gap against dedicated models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DisTime, a lightweight temporal representation framework for Video-LLMs. A dedicated <TIME_STAMP> token is decoded by a small MLP into probability distributions over time bins, and timestamps are re-encoded into Gaussian-regularized token embeddings for input and for autoregressive refinement. Training combines next-token prediction, an IoU regression loss, and distribution focal loss. To address the scarcity of temporally labeled data, the authors construct InternVid-TG (1.25M events from 179k videos) by extracting events with GPT-4o, obtaining event boundaries from UniMD, Mr.Blip, and TFVTG, and ensembling their outputs via an InternVideo2-based similarity score. Experiments cover moment retrieval (Charades-STA, ANet-Caption, QVHighlights), dense video captioning (ANet-Caption, YouCook2), grounded VQA (NExT-GQA), and general video QA. The headline claim is state-of-the-art moment retrieval on Charades-STA in a zero-shot setting, e.g., 81.0 R@1 IoU=0.3 for the 8B model.

Significance. If the claims hold, DisTime is a clean, parameter-efficient alternative to dedicated temporal heads: distribution-based decoding is well motivated by boundary ambiguity, the single-token design is simple, and the re-encoding mechanism is a plausible way to maintain temporal context during autoregression. The authors provide clear ablations supporting the main architectural choices, and they release code and data. The InternVid-TG dataset is a potentially useful resource. However, the central zero-shot claim is not adequately supported because the pseudo-label generation pipeline uses models trained on the target benchmarks, and the composition of the other training data is not audited. The absence of a contamination audit, combined with hyperparameter selection on the evaluation benchmarks and no error bars, means the SOTA claim is currently not established.

major comments (4)
  1. [Sec. 4.1, Table 5] The Charades-STA zero-shot claim is not established. Section 4.1 states that InternVid-TG pseudo-labels are generated by Mr.Blip, a model trained on Charades-STA, ANet-Caption, and QVHighlight, and by UniMD, a dedicated trained grounding model; the ensemble scoring then selects among these outputs. This supervision is used to train DisTime (Table 2). The Charades-STA evaluation in Table 5 is therefore not zero-shot with respect to benchmark-derived annotation knowledge. The 1k-query manual validation in Appendix A measures pseudo-label quality, not contamination. Please provide a training-data contamination audit, or a control experiment that removes this leakage path, or reframe the claim as a distillation result rather than zero-shot.
  2. [Table 2, Appendix D] The direct training data may also leak Charades-STA annotations. Table 2 lists ET-Instruct (126k multi-event samples) as training data, but its composition is never disclosed. Appendix D says the authors expanded on ET-Instruct instruction data but does not state whether Charades-STA-derived samples are excluded. If ET-Instruct contains any Charades-STA-derived instructions, the zero-shot claim is false for that path as well. The manuscript must specify the provenance of every training source or demonstrate exclusion of target-benchmark samples.
  3. [Sec. 3.2, Eq. (6)] The time encoder uses N(st, δ^2) with δ=1 on normalized timestamps in [0,1]. A Gaussian with standard deviation 1 over the unit interval is nearly uniform, which would make re-encoded time tokens almost indistinguishable across different timestamps. This appears to contradict the claimed continuous temporal embedding in Sec. 3.1. Please clarify the intended value of δ and how discriminative temporal information survives this encoding; if δ=1 is a typo, state the correct value and rerun the affected ablations.
  4. [Appendix B, Tables B and C] Important hyperparameters (number of bins regmax, number of encoder/decoder layers) are selected using the evaluation benchmarks Charades-STA and YouCook2, with no held-out validation set reported. No error bars or multiple-seed results are given. Consequently, the reported margins over prior methods in Tables 5 and 6, and the ablation differences in Table 3 (e.g., 78.1 vs. 77.0 in R@1 IoU=0.3), may reflect selection bias or noise. Please report results with multiple seeds or a held-out validation split for hyperparameter selection.
minor comments (5)
  1. [Table 3] The rows with checkmarks for Direct/Dist./Re-Enc. are not fully self-explanatory; please add a one-line description of each configuration, especially how the Direct regression baseline is implemented.
  2. [Table 2] There is a typo in the table header: "ET-Instcuct" should be "ET-Instruct."
  3. [Fig. 5] The pipeline figure contains "InternVideo23" in the middle block; this appears to be a typo for "InternVideo2."
  4. [Sec. 3.2] The paper states δ=1 for boundary uncertainty but provides no sensitivity analysis for δ; since the Gaussian width is central to the time-encoder design, please add an ablation over δ or justify the fixed value.
  5. [Sec. 5.4] The phrase "in a zero-shot setting" in Table 5 and the main text should be revised consistently once the contamination question is resolved, because the paper otherwise trains on ANet-Caption and InternVid-TG data that is closely related to the evaluation task.

Circularity Check

1 steps flagged · score 4.0 of 10

The architectural derivation is self-contained, but the headline zero-shot Charades-STA result is not independent: InternVid-TG pseudo-labels are produced by models trained on Charades-STA, and the scoring/ensemble rule is validated on Charades-STA itself, so the benchmark's annotations enter the training pipeline indirectly.

  1. fitted input called prediction [Sec. 4.1 (Event boundary detection / Similarity scoring and ensemble), Table 5, Appendix B]
    "Mr.Blip is a dedicated Video-LLM trained for temporal grounding, and it is one of the state-of-the-art (SOTA) models in the Charades-STA [9], ANet-Caption [16], and QVHighlight [17] benchmarks. ... A scoring strategy evaluates the grounding results, selecting the highest-scoring model output as the ensemble result for each event. ... To validate the effectiveness of the scoring, we explore using the query and video data from the Charades-STA dataset, including 3720 queries."

    The Table 5 Charades-STA result is presented as 'in a zero-shot setting,' but the model is trained on InternVid-TG, whose pseudo-labels are generated by Mr.Blip, a model explicitly trained on Charades-STA. Appendix B additionally uses Charades-STA queries and ground truth to validate the ensemble scoring rule that selects those pseudo-labels. Thus the benchmark's annotations enter the training pipeline twice: through the annotator's training and through scoring calibration. The Charades-STA score is therefore an indirect transfer of the benchmark's supervision into the training data, so the zero-shot claim is reduced to a distillation setup rather than an independent prediction.

full rationale

The core representation method is not circular: the distribution-based time decoder is compared against direct regression on the same training/evaluation data in Table 3, and the time-encoder/decoder design, loss terms, bin counts, and layer counts are ablated in a self-contained way. Those experiments do not presuppose the SOTA claim. The main circularity concern is data provenance in the claim of zero-shot moment retrieval. InternVid-TG is constructed using Mr.Blip, which the paper itself states is trained on Charades-STA and ANet-Caption, and the scoring/ensemble strategy used to create the pseudo-labels is validated on Charades-STA ground truth in Appendix B. Consequently, the zero-shot Charades-STA result in Table 5 is not an evaluation against an external, unseen benchmark; it is an evaluation on data whose supervision was indirectly injected through the pseudo-labeling pipeline. The paper also lists ET-Instruct as multi-event training data without disclosing whether it contains Charades-STA-derived samples. The use of UniMD (same authors) and TFVTG (coauthor) as annotators is self-citation, but it is not the primary load-bearing reduction; Mr.Blip's explicit benchmark training and the Charades-STA-based scoring calibration are. No mathematical derivation in the paper reduces to its own inputs, and the architectural contributions retain independent content; the circularity is limited to the zero-shot evaluation framing, so a moderate score is appropriate.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method introduces a token-level distribution representation; the only hand-set values are the Gaussian spread delta, the number of time bins (regmax), the depth of the time encoder/decoder, and the loss weights, chosen via ablation or by hand. The main domain assumptions are that boundaries are ambiguous, that one token carries enough temporal information, and that the pseudo-label ensemble provides adequate supervision. No new physical entities are postulated.

free parameters (4)
  • Gaussian spread delta (time encoder) = 1
    Hand-set to 1 to model boundary uncertainty; controls spread of the Gaussian used to encode timestamps, affecting how much ambiguity is injected into training targets.
  • Number of time bins regmax = 32
    Selected via ablation on Charades-STA and YouCook2 (Appendix Tab. B); performance peaks at 32 bins, so the final architecture uses 32.
  • Number of layers in time encoder/decoder = 3
    Selected via ablation (Appendix Tab. C); 3 layers give best Charades-STA and YouCook2 results.
  • Loss weights lambda1, lambda2, lambda3 = 1, 1, 1
    Hand-set to 1 for balanced multi-task optimization; no search over these weights is reported.
assumptions (4)
  • domain assumption Temporal boundaries are inherently ambiguous, making direct regression to absolute timestamps harder than distribution prediction.
    Motivates the Distribution-based Time Decoder; stated in Sec. 3.2 and attributed to DFL [22].
  • domain assumption The hidden state of the single <TIME_STAMP> token from the LLM's last layer contains enough information for a lightweight MLP decoder to recover accurate start and end times.
    Underlies the whole single-token architecture; Sec. 3.1, Eq. (3).
  • ad hoc to paper Re-encoding decoded timestamps into fixed-variance Gaussians standardizes the temporal representation and improves autoregressive accuracy.
    Introduced in Sec. 3.3 as Iterative Time Refinement; no independent theoretical justification is given, only ablation support.
  • domain assumption Pseudo-labels from the ensemble of UniMD, Mr.Blip, and TFVTG, selected by InternVideo2-based cosine similarity, provide supervision of sufficient quality.
    Validated only on 1k manually annotated queries (Appendix Tab. A); if rejected, the value of the 1.25M-event dataset for training is uncertain.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DisTime: Distribution-based Time Representation for Video Large Language Models." pith.science (2026). https://pith.science/paper/VUOGB27B

@misc{pith2026250524329,
  author       = {Pith},
  title        = {Pith review of: DisTime: Distribution-based Time Representation for Video Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VUOGB27B}},
  note         = {Machine review of arXiv:2505.24329}
}
read the original abstract

Despite advances in general video understanding, Video Large Language Models (Video-LLMs) face challenges in precise temporal localization due to discrete time representations and limited temporally aware datasets. Existing methods for temporal expression either conflate time with text-based numerical values, add a series of dedicated temporal tokens, or regress time using specialized temporal grounding heads. To address these issues, we introduce DisTime, a lightweight framework designed to enhance temporal comprehension in Video-LLMs. DisTime employs a learnable token to create a continuous temporal embedding space and incorporates a Distribution-based Time Decoder that generates temporal probability distributions, effectively mitigating boundary ambiguities and maintaining temporal continuity. Additionally, the Distribution-based Time Encoder re-encodes timestamps to provide time markers for Video-LLMs. To overcome temporal granularity limitations in existing datasets, we propose an automated annotation paradigm that combines the captioning capabilities of Video-LLMs with the localization expertise of dedicated temporal models. This leads to the creation of InternVid-TG, a substantial dataset with 1.25M temporally grounded events across 179k videos, surpassing ActivityNet-Caption by 55 times. Extensive experiments demonstrate that DisTime achieves state-of-the-art performance across benchmarks in three time-sensitive tasks while maintaining competitive performance in Video QA tasks. Code and data are released at https://github.com/josephzpng/DisTime.

Figures

Figures reproduced from arXiv: 2505.24329 by the authors.

Figure 1
Figure 1. Comparison of temporal expression paradigms in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Video-LLM with DisTime. Our DisTime method employs a single token (denoted as <TIME STAMP>) to represent continuous time, transformed by a distribution-based time decoder and time encoder. Initially, sampled timestamps are converted into time tokens and integrated into the LLM’s input sequence. During autoregressive generation, when a <TIME STAMP> is encountered, its corresponding LLM last-layer embe… view at source ↗
Figure 3
Figure 3. The illustration of time decoder and time encoder. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The autoregression of text token and time token. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The pipeline of our temporal-aware data expansion paradigm, which utilizes multimodal LLM and dedicated temporal models to [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation

    cs.CV 2026-02 unverdicted novelty 7.0 of 10

    Video-OPD uses on-policy distillation from a frontier teacher to turn sparse episode rewards into dense step-wise signals for more efficient post-training of MLLMs on temporal video grounding.

  2. TimePLE: Rethinking Temporal Representation for Video Temporal Grounding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    TimePLE predicts a whole video interval as a joint distribution over a position-duration square, rather than predicting start and end separately, and reports higher mIoU across four VTG benchmarks.

Reference graph

Works this paper leans on

59 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1728–1738,

  3. [3]

    Cg- bench: Clue-grounded question answering benchmark for long video understanding

    Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, and Limin Wang. Cg- bench: Clue-grounded question answering benchmark for long video understanding. arXiv preprint arXiv:2412.12075,

    arXiv

  4. [4]

    Timemarker: A versatile video-llm for long and short video understanding with superior temporal localiza- tion ability

    Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video-llm for long and short video understanding with superior temporal localiza- tion ability. arXiv preprint arXiv:2411.18211 , 2024. 1, 3, 8

    arXiv 2024

  5. [5]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 4, 7, 8

    arXiv 2024

  6. [6]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101,

  7. [7]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023. 1, 3

    work page 2023

  8. [8]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 7

    arXiv 2024

Show all 59 references
  1. [9]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on com- puter vision, pages 5267–5275, 2017. 1, 2, 6, 7

    2017

  2. [10]

    Linvt: Empower your image- level large language model to understand videos

    Lishuai Gao, Yujie Zhong, Yingsen Zeng, Haoxian Tan, Dengjie Li, and Zheng Zhao. Linvt: Empower your image- level large language model to understand videos. arXiv preprint arXiv:2412.05185, 2024. 4

    2024 arXiv

  3. [11]

    Saliency-guided detr for mo- ment retrieval and highlight detection

    Aleksandr Gordeev, Vladimir Dokholyan, Irina Tolstykh, and Maksim Kuprashevich. Saliency-guided detr for mo- ment retrieval and highlight detection. arXiv preprint arXiv:2410.01615, 2024. 2, 5, 6

    2024

  4. [12]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vi- sio...

    2022

  5. [13]

    Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding

    Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xi- aoying Tang, Dianbo Sui, Qingbin Liu, Xi Chen, and Kevin Zhao. Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding. arXiv preprint arXiv:2405.13382, 2024. 2, 3, 8

    2024 arXiv

  6. [14]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 5, 1

    2022

  7. [15]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14271–14280, 2024. 1, 2, 3, 5, 6, 7, 8

    2024

  8. [16]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on com- puter vision, pages 706–715, 2017. 1, 2, 6, 7

    2017

  9. [17]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems , 34: 11846–11858, 2021. 6, 7

    2021

  10. [18]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 3, 4, 7, 8

    2024 arXiv

  11. [19]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 1, 3

    2023

  12. [20]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 1

    2023 arXiv

  13. [21]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  14. [22]

    Generalized focal loss: Learning qualified and distributed bounding boxes for dense 9 object detection

    Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense 9 object detection. Advances in Neural Information Processing Systems, 33:21002–21012, 2020. 2, 4, 5

    2020

  15. [23]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2024. 3

    2024

  16. [24]

    Groundinggpt: Language enhanced multi-modal grounding model

    Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Van Tu Vu, et al. Groundinggpt: Language enhanced multi-modal grounding model. arXiv preprint arXiv:2401.06071, 2024. 1, 3

    2024 arXiv

  17. [25]

    Detal: Open-vocabulary temporal action localization with decoupled networks

    Zhiheng Li, Yujie Zhong, Ran Song, Tianjiao Li, Lin Ma, and Wei Zhang. Detal: Open-vocabulary temporal action localization with decoupled networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):7728– 7741, 2024. 1

    2024

  18. [26]

    Univtg: Towards unified video- language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shra- man Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video- language temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2794–...

    2023

  19. [27]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 1, 3

    2023

  20. [28]

    Et bench: Towards open-ended event-level video-language understanding

    Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Ying Shan, and Chang Wen Chen. Et bench: Towards open-ended event-level video-language understanding. arXiv preprint arXiv:2409.18111, 2024. 7, 2

    2024 arXiv

  21. [29]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 1

    2017 arXiv

  22. [30]

    Llava-mr: Large language-and- vision assistant for video moment retrieval

    Weiheng Lu, Jian Li, An Yu, Ming-Ching Chang, Sheng- peng Ji, and Min Xia. Llava-mr: Large language-and- vision assistant for video moment retrieval. arXiv preprint arXiv:2411.14505, 2024. 8, 2

    2024 arXiv

  23. [31]

    Valley: Video assistant with large language model enhanced ability

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207 ,

    arXiv

  24. [32]

    The surprising effectiveness of multimodal large language models for video moment retrieval

    Boris Meinardus, Anil Batra, Anna Rohrbach, and Marcus Rohrbach. The surprising effectiveness of multimodal large language models for video moment retrieval. arXiv preprint arXiv:2406.18113, 2024. 2, 6, 8

    2024

  25. [33]

    Correlation-guided query-dependency calibra- tion for video temporal grounding

    WonJun Moon, Sangeek Hyun, SuBeen Lee, and Jae- Pil Heo. Correlation-guided query-dependency calibra- tion for video temporal grounding. arXiv preprint arXiv:2311.08835, 2023. 2

    2023 arXiv

  26. [34]

    Perceptiongpt: Effectively fusing visual perception into llm

    Renjie Pi, Lewei Yao, Jiahui Gao, Jipeng Zhang, and Tong Zhang. Perceptiongpt: Effectively fusing visual perception into llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 27124– 27133, 2024. 1

    2024

  27. [35]

    Momen- tor: Advancing video large language model with fine-grained temporal reasoning

    Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat- Seng Chua, Yueting Zhuang, and Siliang Tang. Momen- tor: Advancing video large language model with fine-grained temporal reasoning. arXiv preprint arXiv:2402.11435, 2024. 2, 3, 5, 6, 7, 8

    2024 arXiv

  28. [36]

    Chatvtg: Video temporal grounding via chat with video dialogue large language models

    Mengxue Qu, Xiaodong Chen, Wu Liu, Alicia Li, and Yao Zhao. Chatvtg: Video temporal grounding via chat with video dialogue large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1847–1856, 2024. 8

    2024

  29. [37]

    Timechat: A time-sensitive multimodal large lan- guage model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large lan- guage model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14313–14323, 2024. 3, 8, 2

    2024

  30. [38]

    xgen-mm-vid (blip-3- video): You only need 32 tokens to represent a video even in vlms

    Michael S Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles. xgen-mm-vid (blip-3- video): You only need 32 tokens to represent a video even in vlms. arXiv preprint arXiv:2410.16267, 2024. 4

    arXiv

  31. [39]

    React: Temporal action detection with relational queries

    Dingfeng Shi, Yujie Zhong, Qiong Cao, Jing Zhang, Lin Ma, Jia Li, and Dacheng Tao. React: Temporal action detection with relational queries. In European conference on computer vision, pages 105–121. Springer, 2022. 1

    2022

  32. [40]

    Temporal action localization with enhanced instant discriminability

    Dingfeng Shi, Qiong Cao, Yujie Zhong, Shan An, Jian Cheng, Haogang Zhu, and Dacheng Tao. Temporal action localization with enhanced instant discriminability. arXiv preprint arXiv:2309.05590, 2023

    2023 arXiv

  33. [41]

    Tridet: Temporal action detection with relative boundary modeling

    Dingfeng Shi, Yujie Zhong, Qiong Cao, Lin Ma, Jia Li, and Dacheng Tao. Tridet: Temporal action detection with relative boundary modeling. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 18857–18866, 2023. 1

    2023

  34. [42]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942, 2023. 5

    2023 arXiv

  35. [43]

    Internvideo2: Scaling foundation models for mul- timodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for mul- timodal video understanding. In European Conference on Computer Vision, pages 396–416. Springer, 2024. 6

    2024

  36. [44]

    Internvideo2

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xi- angyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2. 5: Empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386, 2025. 2, 6, 8

    2025 arXiv

  37. [45]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Informa- tion Processing Systems, 37:28828–28857, 2025. 7

    2025

  38. [46]

    Can i trust your answer? visually grounded video question answering

    Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13204– 13214, 2024. 1, 2, 7, 8

    2024

  39. [47]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 1

    2024 arXiv

  40. [48]

    Zero-shot video question answering via 10 frozen bidirectional language models

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via 10 frozen bidirectional language models. Advances in Neural Information Processing Systems, 35:124–141, 2022. 8

    2022

  41. [49]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 1

    2024 arXiv

  42. [50]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36:76749–76771, 2023. 8

    2023

  43. [51]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yuet- ing Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 9127–9134, 2019. 2

    2019

  44. [52]

    Unimd: Towards unifying moment retrieval and temporal ac- tion detection

    Yingsen Zeng, Yujie Zhong, Chengjian Feng, and Lin Ma. Unimd: Towards unifying moment retrieval and temporal ac- tion detection. In European Conference on Computer Vision, pages 286–304. Springer, 2024. 2, 6, 8

    2024

  45. [53]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 1, 3

    2023 arXiv

  46. [54]

    Ld-detr: Loop decoder detection transformer for video moment retrieval and highlight detection

    Pengcheng Zhao, Zhixian He, Fuwei Zhang, Shujin Lin, and Fan Zhou. Ld-detr: Loop decoder detection transformer for video moment retrieval and highlight detection. arXiv preprint arXiv:2501.10787, 2025. 6

    2025 arXiv

  47. [55]

    Training-free video temporal grounding using large-scale pre-trained models

    Minghang Zheng, Xinhao Cai, Qingchao Chen, Yuxin Peng, and Yang Liu. Training-free video temporal grounding using large-scale pre-trained models. In European Conference on Computer Vision, pages 20–37. Springer, 2024. 2, 6

    2024

  48. [56]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence, 2018. 1, 2, 7

    2018

  49. [57]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 3 11 DisTime: Distribution-based Time Representation for Video Large Languag...

    2023 arXiv

  50. [58]

    Dedicated

    was applied with a cosine learning rate decay and an initial warm-up period. During training, we used a single epoch with a learning rate set to 4 × 10−5. The LoRA [14] parameters were configured with r = 16 and α = 32. We complete the model training process on 8 A100 GPUs. Th...

  51. [59]

    Give you the textual query: ‘thereis an orange barrier out of which people can stand’

    can’t go nearby. Give you the textual query: ‘thereis an orange barrier out of which people can stand’. When does the described content occur in the video? It occurs at 0.38s - 11.26s. Why do we stay far away from landing planes? We stay far away from landing planes to avoid b...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.