Pith. sign in

REVIEW 4 major objections 5 minor 35 references

Temporally-Grounded Language Generation: A Benchmark for Real-Time Vision-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A vision-language model that interleaves video and text on a shared timeline beats turn-based rivals at generating utterances in real time.

desk verdict The TGLG benchmark and TRACE metric are real contributions, but the paper's central architecture claim is undermined by an unacknowledged training asymmetry in the HoloAssist comparison. read the letter →

arxiv 2505.11326 v1 pith:DNI5FHUA submitted 2025-05-16 cs.CV cs.AI

Keunwoo Peter Yu , Joyce Chai This is my paper · ORCID
classification cs.CVcs.AI
keywords temporally-groundedlanguagegenerationreal-timevision-languagemodelsperceptualupdatingcontingencyawarenessTRACEmetrictime-synchronizedinterleavingstreamingvideounderstandingbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that real-time interactive vision-language models need two capabilities that turn-based systems lack: perceptual updating, or revising an interpretation as new frames arrive, and contingency awareness, or adjusting output in response to the effects of one's own words. To measure these, it introduces a benchmark task, Temporally-Grounded Language Generation (TGLG), which asks a model to produce utterances that match human reference utterances in both content and timing while watching streaming video. It also introduces TRACE, a metric that scores semantic similarity and temporal alignment together. The central experimental claim is that a model which interleaves vision tokens and text tokens on a shared timeline, VLM-TSI, outperforms the turn-based VideoLLM-Online on both benchmark capabilities, while overall scores remain modest. If true, this points toward architectures that keep perceiving while speaking rather than freezing the world during generation.

What carries the argument

The load-bearing mechanism is time-synchronized interleaving: VLM-TSI encodes each incoming video frame as a vision token and inserts it into a single timestamp-ordered sequence with text tokens, so every generated token is conditioned on all visual and linguistic context observed up to that moment. At inference, a predicted BOS token triggers utterance generation and non-BOS tokens are discarded, letting the model start and stop speaking frame-by-frame without pausing the video stream. TRACE is the evaluation instrument: it aligns generated and ground-truth utterances by temporal proximity, refines the alignment by semantic similarity, then combines semantic accuracy and timing scores scaled by an F1-based alignment factor.

What would settle it

A live on-policy evaluation in which VLM-TSI and VideoLLM-Online guide a real user or a simulator through the same tasks, measuring task success and utterance timing, would settle the claim: if the turn-based model matches or beats VLM-TSI there despite losing on TRACE, the benchmark's proxy would be shown to be misleading.

Watch

Extended reading notes

Core claim

The paper's central claim is that time-synchronized interleaving of visual and linguistic tokens is a more effective architecture for real-time, temporally-grounded language generation than the turn-based assumption embodied by VideoLLM-Online. On the TGLG benchmark, VLM-TSI scores 39.1 versus 27.1 on perceptual updating and 18.8 versus 9.6 on contingency awareness under TRACE. The largest component gain is in the overlap score, which nearly doubles, because the interleaved design enforces non-overlapping utterances. The authors also report that both models are better at knowing when to start speaking than when to stop, and that contingency awareness is harder than perceptual updating for both systems. Absolute scores remain low enough that the benchmark is presented as an open challenge rather than a solved problem.

Load-bearing premise

Matching pre-recorded human utterances, scored by TRACE, is a valid stand-in for how well the model would perform in a live interaction; the evaluation never puts the model in a feedback loop with a real user or environment.

Editorial extensions

If this is right

  • Turn-based models that pause the environment while generating are structurally mismatched to real-time interaction; a shared-timeline interleaving avoids the compounding delays and overlaps illustrated in the paper.
  • VLM-TSI's near-doubling of the overlap score means that enforcing non-overlapping generation by design carries a measurable timing benefit.
  • Because both models do much worse on contingency awareness than on perceptual updating, real-time instruction following that must account for the effects of one's own utterances remains the harder open problem.
  • TGLG and TRACE give future work a common protocol for comparing real-time VLMs, including decomposable diagnostics of start-time, end-time, and semantic failures.
  • The benchmark's modest absolute scores suggest that improving utterance termination, knowing when to stop, is a concrete next target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same interleaving principle could extend to other streaming tasks such as dense video captioning and embodied task guidance, where the model's own past utterances are part of the visual context.
  • A natural test of TRACE's validity would be to compare its scores with human judgments or downstream task success in a live interactive loop; the paper's off-policy design makes this an open verification.
  • The BOS-as-onset training rule may remove the need for hand-set EOS thresholds, which could make real-time systems easier to deploy across domains with different silence-to-activity ratios.
  • If end-alignment remains the weakest component, a model with explicit utterance-duration prediction or learned stopping cues could improve TRACE more than a better semantic generator.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Temporally-Grounded Language Generation (TGLG), a benchmark task for real-time vision-language models in which models must generate utterances that match human timing and content while processing streaming video. It introduces the TRACE metric, which combines semantic similarity with temporal alignment, curates datasets from SoccerNet (perceptual updating) and HoloAssist (contingency awareness), and presents VLM-TSI, a model that interleaves vision and text tokens on a shared timeline. Experiments compare VLM-TSI against VideoLLM-Online and report higher TRACE scores on both capabilities, while acknowledging modest absolute performance and the off-policy nature of the evaluation in Appendix F.

Significance. If the benchmark and metric are reliable, this is a valuable step toward evaluating temporally-grounded language generation, an underexplored area relative to offline video QA and captioning. The paper ships code and data, provides per-category analyses, and includes qualitative examples, all of which are strengths. However, the central architectural claim is weakened by asymmetric fine-tuning on the contingency-awareness split and by the absence of statistical uncertainty estimates. These issues need to be resolved before the benchmark's conclusions can be accepted.

major comments (4)
  1. [Sec. 6.1 / Table 2 (Contingency Awareness)] The comparison between VLM-TSI and VideoLLM-Online on HoloAssist is confounded by asymmetric training. The text states that VLM-TSI is fine-tuned for 2 epochs on Ego4D Goal-Step streaming narration/dialogue data, whereas VideoLLM-Online receives no further fine-tuning because it is 'already pre-trained on this dataset.' This contradicts the stated protocol in Section 6.1 that 'modifying only the token interleaving strategy during fine-tuning' ensures a fair comparison. The additional in-domain gradient updates could improve instruction-generation behavior independently of time-synchronized interleaving, so the 18.8 vs 9.6 TRACE gap in Table 2 does not establish an architectural advantage. Please add an ablation or matched training condition and report contingency-awareness results for both conditions.
  2. [Abstract, Sec. 6.2.1, Table 2] The claim that VLM-TSI 'significantly outperforms' VideoLLM-Online is not supported by any statistical evidence. All scores in Tables 2-4 are single-run point estimates; there are no confidence intervals, significance tests, or multiple-seed results. Given that the per-task difference in Table 4 is even negative for 'Repair Machinery' (-0.01), the significance claim is overreaching. Please provide repeated runs with variance estimates or apply bootstrap or permutation tests at the utterance level.
  3. [Appendix B, Eqs. (12)-(18)] The TRACE metric depends on six manually set parameters (α, αstart, αend, τtime, τwin, τpen) and an assumed speech rate of 150 words per minute with 1.3 tokens per word to estimate end times (Appendix A). No sensitivity analysis is reported, and several sub-scores are exponential functions of these thresholds. It is therefore not established that the reported ranking is robust to reasonable parameter changes. Please include a sensitivity analysis over the parameter ranges or justify the chosen values on independent data.
  4. [Appendix F / Sec. 3] Appendix F correctly acknowledges that the evaluation is off-policy: models are not placed in an interactive feedback loop. This directly affects the construct validity of the 'contingency awareness' label, which Section 3 defines as the ability to adjust actions based on their effects. The HoloAssist results may measure instruction-following in recorded interactions rather than true contingency awareness. I recommend either reframing the capability name or providing a small on-policy validation to support the construct interpretation.
minor comments (5)
  1. [Abstract / Sec. 4.3] The metric name is inconsistent: the Abstract says 'Temporal Responsiveness and Alignment Coherence Evaluation' while Section 4.3 says 'Temporal Responsiveness and Alignment Evaluation.' Please unify the name.
  2. [References] References [29] and [30] are identical (both cite 'Streaming Dense Video Captioning'); remove the duplicate.
  3. [Sec. 3] There is a redundant phrasing: 'VideoLLM-Online still assumes that the environment is assumed to pause during language generation' should be simplified.
  4. [Appendix A] The definition of 'evaluation clusters' is terse; please clarify how clusters are constructed and justify the 5-second window used for grouping utterances.
  5. [Sec. 6.2.3, Table 4] The negative difference for 'Repair Machinery' is not discussed until the following paragraph; consider moving this discussion into the results paragraph for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the comparison is empirical and TRACE is not a training objective.

full rationale

The paper's derivation chain is not circular. The TGLG task and the TRACE metric are defined in Section 4 before VLM-TSI is introduced in Section 5, so the model is not an input to the benchmark definition. TRACE is used only for evaluation, never as a training loss, and no fitted parameter is relabeled as a prediction. The SoccerNet comparison is matched, with both models fine-tuned for 5 epochs on the same split, and the HoloAssist subset is fully held out, so test labels do not enter training. The self-citations ([2], [25]) appear in related-work framing and in a standard semantic-similarity choice for the metric; they are not load-bearing and do not supply a uniqueness theorem or a smuggled ansatz. The overlap component of TRACE rewards non-overlapping output, and VLM-TSI's interleaved inference tends to avoid overlaps by design, but this is an evaluation-design alignment rather than circularity: the model is not optimized against TRACE, and the overall TRACE advantage also holds on semantic accuracy and start/end alignment components. The training asymmetry for contingency awareness (extra Ego4D Goal-Step fine-tuning for VLM-TSI) and the off-policy nature acknowledged in Appendix F are validity concerns, not circularity, because the reported advantage is not equivalent to the benchmark inputs by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the benchmark and metric design. The free parameters are manually chosen constants in TRACE and baseline thresholds that affect all reported scores. The axioms are assumptions about data quality, classifier accuracy, domain transfer, the validity of off-policy evaluation, and the adequacy of sentence embeddings for semantic scoring. No new physical or conceptual entities are introduced.

free parameters (5)
  • TRACE weight alpha = 0.5
    Balance between semantic accuracy and timing accuracy in final TRACE score; set manually in Appendix B, Equation 18. All reported scores depend on this balance.
  • TRACE timing weights alpha_start and alpha_end = 0.4, 0.4
    Weights for start and end alignment in the timing score decomposition; manual choice in Equation 18.
  • TRACE time constants tau_time, tau_win, tau_pen = 3.0, 5.0, 1.0
    Time-scale parameter for alignment cost, pruning window for matched pairs, and penalty scale for timing errors; set in Appendix B.1 and B.3. These values control how much the metric rewards temporal closeness.
  • VideoLLM-Online EOS threshold = 0.725 (perceptual updating), 0.8 (contingency awareness)
    Threshold for emitting the EOS token in the baseline model; raised for contingency awareness because the model otherwise generated too few utterances (Section 6.1). This tuning changes baseline behavior and could affect the comparison.
  • Speech rate for end-time estimation = 150 words/min, 1.3 tokens/word
    Used to convert generated token counts into utterance end times for models that do not emit explicit end timestamps (Appendix A). Assumes a fixed speaking rate across all generated content.
assumptions (5)
  • domain assumption WhisperX transcriptions of SoccerNet commentary are time-accurate enough to align utterances with video events.
    The perceptual-updating dataset is built from these transcriptions; errors in transcription timecodes would misalign ground-truth utterance timings (Section 4.1.1).
  • domain assumption The LSTM classifier with 94% held-out accuracy sufficiently isolates play-by-play commentary from analysis, banter, and advertisements.
    The classifier filters 58,031 of 148,533 utterances; residual misclassification would introduce non-grounded utterances into the benchmark (Section 4.1.1).
  • domain assumption Ego4D Goal-Step streaming data is similar enough to HoloAssist to transfer when fine-tuning VLM-TSI for the contingency-awareness task.
    VLM-TSI is fine-tuned on Ego4D Goal-Step and evaluated on HoloAssist without additional HoloAssist fine-tuning, assuming the domains share enough structure for useful transfer (Section 6.1).
  • domain assumption Off-policy human utterances are a valid proxy for the ideal real-time model output.
    The benchmark evaluates against pre-recorded human commentary and instructions, not in a live interactive loop. This is explicitly acknowledged in Appendix F and limits the validity of the contingency-awareness and perceptual-updating measurements.
  • domain assumption Sentence embeddings from all-mpnet-base-v2 capture semantic similarity in these domains well enough for TRACE to measure semantic accuracy.
    TRACE computes Sa as cosine similarity between sentence embeddings (Equation 7). If the embeddings miss domain-specific meaning, especially short instruction phrases, the semantic scores could be unreliable (Appendix B.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temporally-Grounded Language Generation: A Benchmark for Real-Time Vision-Language Models." pith.science (2026). https://pith.science/paper/DNI5FHUA

@misc{pith2026250511326,
  author       = {Pith},
  title        = {Pith review of: Temporally-Grounded Language Generation: A Benchmark for Real-Time Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DNI5FHUA}},
  note         = {Machine review of arXiv:2505.11326}
}
abstract

Vision-language models (VLMs) have shown remarkable progress in offline tasks such as image captioning and video question answering. However, real-time interactive environments impose new demands on VLMs, requiring them to generate utterances that are not only semantically accurate but also precisely timed. We identify two core capabilities necessary for such settings -- $\textit{perceptual updating}$ and $\textit{contingency awareness}$ -- and propose a new benchmark task, $\textbf{Temporally-Grounded Language Generation (TGLG)}$, to evaluate them. TGLG requires models to generate utterances in response to streaming video such that both content and timing align with dynamic visual input. To support this benchmark, we curate evaluation datasets from sports broadcasting and egocentric human interaction domains, and introduce a new metric, $\textbf{TRACE}$, to evaluate TGLG by jointly measuring semantic similarity and temporal alignment. Finally, we present $\textbf{Vision-Language Model with Time-Synchronized Interleaving (VLM-TSI)}$, a model that interleaves visual and linguistic tokens in a time-synchronized manner, enabling real-time language generation without relying on turn-based assumptions. Experimental results show that VLM-TSI significantly outperforms a strong baseline, yet overall performance remains modest -- highlighting the difficulty of TGLG and motivating further research in real-time VLMs. Code and data available $\href{https://github.com/yukw777/tglg}{here}$.

Figures

Figures reproduced from arXiv: 2505.11326 by the authors.

Figure 1
Figure 1. Turn-based VLMs fail to operate effectively in real-time environments, because they cannot [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Sports broadcast datasets like SoccerNet [ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Egocentric interaction datasets like HoloAssist [ [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: VLM-TSI interleaves vision tokens vt and text tokens xτ in a temporally synchronized manner. For simplicity, each frame ft is encoded as a single vision token vt. 5.3 Inference At inference time, VLM-TSI receives one visual token per timestep (or more if video frames a…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 17 canonical work pages

  1. [1]

    Whisperx: Time-accurate speech transcription of long-form audio

    Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. Whisperx: Time-accurate speech transcription of long-form audio. INTERSPEECH 2023, 2023

    2023

  2. [2]

    Can foundation models watch, talk and guide you step by step to make a cake? arXiv preprint arXiv:2311.00738, 2023

    Yuwei Bao, Keunwoo Peter Yu, Yichi Zhang, Shane Storks, Itamar Bar-Yossef, Alexander De La Iglesia, Megan Su, Xiao Lin Zheng, and Joyce Chai. Can foundation models watch, talk and guide you step by step to make a cake? arXiv preprint arXiv:2311.00738, 2023

    arXiv 2023

  3. [3]

    Collecting highly parallel data for paraphrase evaluation

    David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 190–200, Portland, Oregon, USA, June 2011. Association for Computational Linguistics

    work page 2011

  4. [4]

    Videollm-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18407–18418, June 2024

    2024

  5. [5]

    Scaling up soccernet with multi-view spatial localization and re-identification

    Anthony Cioppa, Adrien Deliège, Silvio Giancola, Bernard Ghanem, and Marc Van Droogen- broeck. Scaling up soccernet with multi-view spatial localization and re-identification. Scientific data, 9(1):355, 2022

    work page 2022

  6. [6]

    Soccernet-v2: A dataset and benchmarks for holistic understanding of broadcast soccer videos

    Adrien Deliege, Anthony Cioppa, Silvio Giancola, Meisam J Seikavandi, Jacob V Dueholm, Kamal Nasrollahi, Bernard Ghanem, Thomas B Moeslund, and Marc Van Droogenbroeck. Soccernet-v2: A dataset and benchmarks for holistic understanding of broadcast soccer videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 450...

    work page 2021

  7. [7]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    arXiv 2024

  8. [8]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

Show all 35 references
  1. [9]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758–2766, 2017

    2017

  2. [10]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017

    2017 arXiv

  3. [11]

    Kuehne, H

    H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: A large video database for human motion recognition. In 2011 International Conference on Computer Vision, pages 2556–2563, 2011

    2011

  4. [12]

    Revealing single frame bias for video-and-language learning

    Jie Lei, Tamara Berg, and Mohit Bansal. Revealing single frame bias for video-and-language learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–507, 2023

    2023

  5. [13]

    Streamingbench: Assessing the gap for mllms to achieve streaming video understanding

    Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for mllms to achieve streaming video understanding. CoRR, abs/2411.03628, 2024

    2024 arXiv

  6. [14]

    Video-ChatGPT: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Comput...

    2024

  7. [15]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    2023

  8. [16]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019

    2019

  9. [17]

    Ego4d goal-step: Toward hierarchical understanding of procedural activities

    Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: Toward hierarchical understanding of procedural activities. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Informat...

    2023

  10. [18]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012

    2012 arXiv

  11. [19]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024

    2024 arXiv

  12. [20]

    Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world

    Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, et al. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. InProceedings of the IEEE/C...

    2023

  13. [21]

    Streaming video understanding and multi-round interaction with memory-enhanced knowl- edge

    Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Jiawen Zhu, and Huchuan Lu. Streaming video understanding and multi-round interaction with memory-enhanced knowl- edge. CoRR, abs/2501.13468, 2025

    2025 arXiv

  14. [22]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Conference on Multimedia , MM ’17, page 1645–1653, New York...

    2017

  15. [23]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    2016

  16. [24]

    Svbench: A benchmark with temporal multi-turn dialogues for streaming video understanding

    Zhenyu Yang, Yuhang Hu, Zemin Du, Dizhan Xue, Shengsheng Qian, Jiahong Wu, Fan Yang, Weiming Dong, and Changsheng Xu. Svbench: A benchmark with temporal multi-turn dialogues for streaming video understanding. CoRR, abs/2502.10810, 2025

    2025

  17. [25]

    Eliciting in-context learning in vision-language models for videos through curated data distributional properties

    Keunwoo Yu, Zheyuan Zhang, Fengyuan Hu, Shane Storks, and Joyce Chai. Eliciting in-context learning in vision-language models for videos through curated data distributional properties. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Confe...

    2024

  18. [26]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):9127–9134, Jul. 2019

    2019

  19. [27]

    Flash-vstream: Memory-based real-time understanding for long video streams

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.08085, 2024

    2024 arXiv

  20. [28]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024. 11

    2024 arXiv

  21. [29]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18243–18252, 2024

    2024

  22. [30]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18243– 18252, June 2024. A Ta...

    2024

  23. [31]

    Assistant: Now remove the indicated component that’s damaged,

    33.2-43.3: “Assistant: Now remove the indicated component that’s damaged, . . . ”

  24. [32]

    User: Oh, this thing?

    45.3-46.6: “User: Oh, this thing?”

  25. [33]

    Assistant: To the right

    46.6-47.4: “Assistant: To the right.”

  26. [34]

    Assistant: The small cube

    47.9-49.2: “Assistant: The small cube.”

  27. [35]

    Assistant: Yes

    49.3-49.8: “Assistant: Yes.” Each utterance ui = (si,ei,{xt| si≤ t≤ ei}) includes a start time si, an end time ei, and a sequence of text tokens xt timestamped att. The complete set of utterances in a video forms the interaction historyU ={ui| 1≤i≤N}. We define evaluation clus...

    2015

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.