{"id":"942ab4aa-3e49-422b-afe0-dc9633508c70","arxiv_id":"2505.11326","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"VLM-TSI, which interleaves vision and text tokens along a shared timeline, outperforms the turn-based VideoLLM-Online on TGLG, a new benchmark for real-time temporally-grounded language generation.","lead":"The paper introduces TGLG, a benchmark that tests whether vision-language models can describe streaming video with both accurate content and correct timing. The authors also propose VLM-TSI, which interleaves visual and text tokens on a shared timeline, and report that it outperforms a turn-based baseline on the new benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The HoloAssist comparison is confounded by asymmetric training: VLM-TSI receives two extra epochs of Ego4D Goal-Step fine-tuning while VideoLLM-Online does not, so the claimed architecture advantage on contingency awareness is not established.","rationale":"The reader's conditional verdict is well-motivated, but I would anchor it on a different, more actionable flaw. The paper's central claim is that time-synchronized interleaving yields better real-time language generation. On SoccerNet, the comparison is reasonably controlled: both models start from the same checkpoint and are fine-tuned on the same data for the same number of epochs, so the 39.1 vs 27.1 gap is a credible (though not statistically validated) architecture effect. On HoloAssist, however, the protocol diverges: VLM-TSI is fine-tuned for 2 epochs on Ego4D Goal-Step, a domain closely related to HoloAssist, while VideoLLM-Online receives no further training. Additional in-domain data and gradient updates are a well-known source of large performance gains, so the 18.8 vs 9.6 gap cannot be attributed to the interleaving mechanism. Because the paper's contributions and conclusions cover both capabilities, this confound is load-bearing: it undermines the generalization claim, not just a secondary analysis. The fix is straightforward (an ablation or matched fine-tuning control), which keeps the verdict conditional rather than reject. I also agree with the reader that the off-policy design limits construct validity for contingency awareness; Appendix F acknowledges this, and it compounds the problem, but the training asymmetry is the more immediate threat to the empirical claim. If the control shows the gap persists, the paper's central claim would be substantially strengthened.","tokens_in":14865,"tokens_out":10325,"duration_ms":95393,"concrete_test":"Run a control where a VLM-TSI variant is evaluated on HoloAssist without the 2-epoch Ego4D Goal-Step fine-tuning, starting from the same VideoLLM-online-8B-v1+ checkpoint and with identical inference settings; if its TRACE drops toward VideoLLM-Online's 9.6, the reported improvement is attributable to extra training. Complementarily, fine-tune VideoLLM-Online on the same Ego4D Goal-Step data for 2 epochs with the same recipe and re-evaluate; if the gap narrows substantially or reverses, the architecture claim for contingency awareness is not supported. Report multiple seeds and error bars for whichever control is run.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.1 states that for contingency awareness, VLM-TSI is fine-tuned for 2 epochs on Ego4D Goal-Step streaming narration/dialogue data, while VideoLLM-Online is not fine-tuned further because it is 'already pre-trained on this dataset.' This violates the paper's claim to modify 'only the token interleaving strategy during fine-tuning.' VLM-TSI receives additional gradient updates on in-domain egocentric interaction data, plausibly improving its instruction-generation behavior independently of the time-synchronized interleaving architecture. Therefore the 18.8 vs 9.6 TRACE gap in Table 2 (contingency awareness) may be a training-data effect rather than an architectural advantage. The only unconfounded comparison is SoccerNet (both models fine-tuned on the same split for 5 epochs), which narrows the scope of the central claim. The off-policy limitation in Appendix F is real, but the training asymmetry is a more immediate internal-validity threat that is not acknowledged.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Temporally-Grounded Language Generation (TGLG), a benchmark task for real-time vision-language models in which models must generate utterances that match human timing and content while processing streaming video. It introduces the TRACE metric, which combines semantic similarity with temporal alignment, curates datasets from SoccerNet (perceptual updating) and HoloAssist (contingency awareness), and presents VLM-TSI, a model that interleaves vision and text tokens on a shared timeline. Experiments compare VLM-TSI against VideoLLM-Online and report higher TRACE scores on both capabilities, while acknowledging modest absolute performance and the off-policy nature of the evaluation in Appendix F.","tokens_in":15065,"tokens_out":5811,"duration_ms":53328,"significance":"If the benchmark and metric are reliable, this is a valuable step toward evaluating temporally-grounded language generation, an underexplored area relative to offline video QA and captioning. The paper ships code and data, provides per-category analyses, and includes qualitative examples, all of which are strengths. However, the central architectural claim is weakened by asymmetric fine-tuning on the contingency-awareness split and by the absence of statistical uncertainty estimates. These issues need to be resolved before the benchmark's conclusions can be accepted.","major_comments":[{"comment":"The comparison between VLM-TSI and VideoLLM-Online on HoloAssist is confounded by asymmetric training. The text states that VLM-TSI is fine-tuned for 2 epochs on Ego4D Goal-Step streaming narration/dialogue data, whereas VideoLLM-Online receives no further fine-tuning because it is 'already pre-trained on this dataset.' This contradicts the stated protocol in Section 6.1 that 'modifying only the token interleaving strategy during fine-tuning' ensures a fair comparison. The additional in-domain gradient updates could improve instruction-generation behavior independently of time-synchronized interleaving, so the 18.8 vs 9.6 TRACE gap in Table 2 does not establish an architectural advantage. Please add an ablation or matched training condition and report contingency-awareness results for both conditions.","section":"Sec. 6.1 / Table 2 (Contingency Awareness)"},{"comment":"The claim that VLM-TSI 'significantly outperforms' VideoLLM-Online is not supported by any statistical evidence. All scores in Tables 2-4 are single-run point estimates; there are no confidence intervals, significance tests, or multiple-seed results. Given that the per-task difference in Table 4 is even negative for 'Repair Machinery' (-0.01), the significance claim is overreaching. Please provide repeated runs with variance estimates or apply bootstrap or permutation tests at the utterance level.","section":"Abstract, Sec. 6.2.1, Table 2"},{"comment":"The TRACE metric depends on six manually set parameters (α, αstart, αend, τtime, τwin, τpen) and an assumed speech rate of 150 words per minute with 1.3 tokens per word to estimate end times (Appendix A). No sensitivity analysis is reported, and several sub-scores are exponential functions of these thresholds. It is therefore not established that the reported ranking is robust to reasonable parameter changes. Please include a sensitivity analysis over the parameter ranges or justify the chosen values on independent data.","section":"Appendix B, Eqs. (12)-(18)"},{"comment":"Appendix F correctly acknowledges that the evaluation is off-policy: models are not placed in an interactive feedback loop. This directly affects the construct validity of the 'contingency awareness' label, which Section 3 defines as the ability to adjust actions based on their effects. The HoloAssist results may measure instruction-following in recorded interactions rather than true contingency awareness. I recommend either reframing the capability name or providing a small on-policy validation to support the construct interpretation.","section":"Appendix F / Sec. 3"}],"minor_comments":[{"comment":"The metric name is inconsistent: the Abstract says 'Temporal Responsiveness and Alignment Coherence Evaluation' while Section 4.3 says 'Temporal Responsiveness and Alignment Evaluation.' Please unify the name.","section":"Abstract / Sec. 4.3"},{"comment":"References [29] and [30] are identical (both cite 'Streaming Dense Video Captioning'); remove the duplicate.","section":"References"},{"comment":"There is a redundant phrasing: 'VideoLLM-Online still assumes that the environment is assumed to pause during language generation' should be simplified.","section":"Sec. 3"},{"comment":"The definition of 'evaluation clusters' is terse; please clarify how clusters are constructed and justify the 5-second window used for grouping utterances.","section":"Appendix A"},{"comment":"The negative difference for 'Repair Machinery' is not discussed until the following paragraph; consider moving this discussion into the results paragraph for readability.","section":"Sec. 6.2.3, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper would be a reasonable contribution if the training-asymmetry confound is resolved and statistical uncertainty is added. Given the current evidence, the architectural superiority claim is not yet established. Please also check that the benchmark and metric names are consistent across versions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TGLG and TRACE are real contributions. TGLG formalizes a task that existing benchmarks ignore: generating utterances that are both semantically right and temporally right under streaming video. The SoccerNet and HoloAssist curation is thoughtful, and TRACE, with its decomposable semantic/timing/overlap components, is a reasonable first metric. VLM-TSI's idea of interleaving vision and text tokens on a shared timeline is clean and worth trying. The SoccerNet comparison, where both models get the same fine-tuning data, does show a genuine advantage for VLM-TSI, especially on the overlap subscore. The biggest problem is in Section 6.1. For contingency awareness, VLM-TSI gets two extra epochs of Ego4D Goal-Step fine-tuning that VideoLLM-Online does not receive. That breaks the stated 'only token interleaving' manipulation. The 18.8 vs 9.6 gap on HoloAssist could easily be a data exposure effect rather than an architectural win. This needs fixing - fine-tune both on the same data or make the asymmetry an explicit experimental variable. The second issue is statistics: all results are single runs with no error bars or significance tests, so even the unconfounded SoccerNet result is not robust. The manual TRACE weights and the baseline threshold tuning are also under-justified. The off-policy limitation is acknowledged in Appendix F, and it is real, but the training asymmetry is the more immediate internal-validity threat. To be fair, the paper is honest about some limits: it explicitly says TRACE parameters are manually set, and the off-policy issue is in print. The code and data link suggests reproducibility, though no commit hash is provided. The citation pattern looks fine. Verdict: this deserves a serious referee. The benchmark and metric are useful enough that the community should see them, but the model comparison needs major experimental rework. I would send it out and insist on fixing the training asymmetry and adding some form of variance reporting.","headline":"The TGLG benchmark and TRACE metric are real contributions, but the paper's central architecture claim is undermined by an unacknowledged training asymmetry in the HoloAssist comparison.","tokens_in":697,"tokens_out":1432,"would_cite":true,"duration_ms":32426,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model that interleaves video and text on a shared timeline beats turn-based rivals at generating utterances in real time.","keywords":["temporally-grounded language generation","real-time vision-language models","perceptual updating","contingency awareness","TRACE metric","time-synchronized interleaving","streaming video understanding","benchmark"],"falsifier":"A live on-policy evaluation in which VLM-TSI and VideoLLM-Online guide a real user or a simulator through the same tasks, measuring task success and utterance timing, would settle the claim: if the turn-based model matches or beats VLM-TSI there despite losing on TRACE, the benchmark's proxy would be shown to be misleading.","tokens_in":14638,"feed_emoji":"⏱️","tokens_out":7082,"duration_ms":61830,"temperature":0.7,"pith_summary":"This paper proposes that real-time interactive vision-language models need two capabilities that turn-based systems lack: perceptual updating, or revising an interpretation as new frames arrive, and contingency awareness, or adjusting output in response to the effects of one's own words. To measure these, it introduces a benchmark task, Temporally-Grounded Language Generation (TGLG), which asks a model to produce utterances that match human reference utterances in both content and timing while watching streaming video. It also introduces TRACE, a metric that scores semantic similarity and temporal alignment together. The central experimental claim is that a model which interleaves vision tokens and text tokens on a shared timeline, VLM-TSI, outperforms the turn-based VideoLLM-Online on both benchmark capabilities, while overall scores remain modest. If true, this points toward architectures that keep perceiving while speaking rather than freezing the world during generation.","feed_headline":"Watch-and-speak model beats turn-based AI at timed narration","feed_subtitle":"A model that watches while it speaks outscores a turn-based rival 39-27 on timed commentary.","key_machinery":"The load-bearing mechanism is time-synchronized interleaving: VLM-TSI encodes each incoming video frame as a vision token and inserts it into a single timestamp-ordered sequence with text tokens, so every generated token is conditioned on all visual and linguistic context observed up to that moment. At inference, a predicted BOS token triggers utterance generation and non-BOS tokens are discarded, letting the model start and stop speaking frame-by-frame without pausing the video stream. TRACE is the evaluation instrument: it aligns generated and ground-truth utterances by temporal proximity, refines the alignment by semantic similarity, then combines semantic accuracy and timing scores scaled by an F1-based alignment factor.","core_discovery":"The paper's central claim is that time-synchronized interleaving of visual and linguistic tokens is a more effective architecture for real-time, temporally-grounded language generation than the turn-based assumption embodied by VideoLLM-Online. On the TGLG benchmark, VLM-TSI scores 39.1 versus 27.1 on perceptual updating and 18.8 versus 9.6 on contingency awareness under TRACE. The largest component gain is in the overlap score, which nearly doubles, because the interleaved design enforces non-overlapping utterances. The authors also report that both models are better at knowing when to start speaking than when to stop, and that contingency awareness is harder than perceptual updating for both systems. Absolute scores remain low enough that the benchmark is presented as an open challenge rather than a solved problem.","pith_inferences":["The same interleaving principle could extend to other streaming tasks such as dense video captioning and embodied task guidance, where the model's own past utterances are part of the visual context.","A natural test of TRACE's validity would be to compare its scores with human judgments or downstream task success in a live interactive loop; the paper's off-policy design makes this an open verification.","The BOS-as-onset training rule may remove the need for hand-set EOS thresholds, which could make real-time systems easier to deploy across domains with different silence-to-activity ratios.","If end-alignment remains the weakest component, a model with explicit utterance-duration prediction or learned stopping cues could improve TRACE more than a better semantic generator."],"forward_implications":["Turn-based models that pause the environment while generating are structurally mismatched to real-time interaction; a shared-timeline interleaving avoids the compounding delays and overlaps illustrated in the paper.","VLM-TSI's near-doubling of the overlap score means that enforcing non-overlapping generation by design carries a measurable timing benefit.","Because both models do much worse on contingency awareness than on perceptual updating, real-time instruction following that must account for the effects of one's own utterances remains the harder open problem.","TGLG and TRACE give future work a common protocol for comparing real-time VLMs, including decomposable diagnostics of start-time, end-time, and semantic failures.","The benchmark's modest absolute scores suggest that improving utterance termination, knowing when to stop, is a concrete next target."],"supporting_citations":[{"why":"Supplies the turn-based baseline that VLM-TSI is compared against and whose training recipe the comparison follows.","marker":"[4]"},{"why":"Provides the SoccerNet sports broadcast videos and commentary used to benchmark perceptual updating.","marker":"[5]"},{"why":"Provides the HoloAssist egocentric interaction data used to benchmark contingency awareness.","marker":"[20]"},{"why":"Provides the sentence embedding model that measures semantic similarity inside the TRACE metric.","marker":"[16]"},{"why":"Supplies the Ego4D Goal-Step streaming narration and dialogue data used to fine-tune VLM-TSI for contingency awareness.","marker":"[17]"},{"why":"Provides the SoccerNet action-spotting annotations used to group and analyze per-action performance differences.","marker":"[6]"},{"why":"Supplies the low-rank adaptation method used to fine-tune both models efficiently.","marker":"[8]"}],"fun_headline_variants":["Time-synced VLM beats turn-based at real-time narration","Interleaved tokens let AI narrate live video on beat","Watch-while-talk model wins timed language benchmark","New VLM times speech to video, doubles overlap score","Real-time VLM outpaces turn-based on contingency awareness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Matching pre-recorded human utterances, scored by TRACE, is a valid stand-in for how well the model would perform in a live interaction; the evaluation never puts the model in a feedback loop with a real user or environment.","fun_headline_variants_meta":{"raw":{"variants":["Time-synced VLM beats turn-based at real-time narration","Interleaved tokens let AI narrate live video on beat","Watch-while-talk model wins timed language benchmark","New VLM times speech to video, doubles overlap score","Real-time VLM outpaces turn-based on contingency awareness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000656,"raw_usage":{"total_tokens":3019,"prompt_tokens":975,"completion_tokens":2044,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":1962}},"tokens_in":591,"tokens_out":2044,"duration_ms":14986,"temperature":1.0,"reasoning_tokens":1962,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:53:25.076891+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A live on-policy evaluation in which VLM-TSI and VideoLLM-Online guide a real user or a simulator through the same tasks, measuring task success and utterance timing, would settle the claim: if the turn-based model matches or beats VLM-TSI there despite losing on TRACE, the benchmark's proxy would be shown to be misleading.","supporting_citations":[{"cited_title":"Scaling up soccernet with multi-view spatial localization and re-identification","cited_arxiv_id":null,"evidence_quote":"Provides the SoccerNet sports broadcast videos and commentary used to benchmark perceptual updating."},{"cited_title":"Ego4d goal-step: Toward hierarchical understanding of procedural activities","cited_arxiv_id":null,"evidence_quote":"Supplies the Ego4D Goal-Step streaming narration and dialogue data used to fine-tune VLM-TSI for contingency awareness."},{"cited_title":"Soccernet-v2: A dataset and benchmarks for holistic understanding of broadcast soccer videos","cited_arxiv_id":null,"evidence_quote":"Provides the SoccerNet action-spotting annotations used to group and analyze per-action performance differences."}],"review_version":1}