{"id":"afc40b70-4942-4cda-85de-972936a75c45","arxiv_id":"2411.12951","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Most open-source Video-LLMs are near chance-level at verifying their own temporal predictions, and a proposed event-temporal verification tuning improves both grounding and consistency.","lead":"Video language models often give inconsistent answers when asked to re-verify or re-ground their own temporal predictions, with many open-source models performing near chance. The authors release new consistency benchmarks and a verification tuning method that improves both grounding accuracy and consistency.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All headline scores depend on a GPT-4o-mini generator-judge loop: it writes the test queries, writes VTune's training queries, and scores model answers, while only query quality (not answer scoring) is human-validated.","rationale":"I agree with the reader's weakest-assumption diagnosis. The most load-bearing concern is not any single equation but the absence of an external anchor for the answer-scoring step. The paper's human study is real evidence for the validity of the generated queries, and the open-sourced code/data are a plus; the qualitative examples in Table 15 make it plausible that open-source models do fail at self-verification. But the quantitative claims—especially 'near chance' and 'significant improvements'—depend on GPT-4o-mini's judgments at both ends: it produces the curriculum VTune is tuned on and the test questions it is evaluated on, and it grades the answers. Without a human-judged subset or an independently written test set, the same-generator loop remains unresolved. I do not think this warrants rejection; the finding is plausible and partially supported by examples, and the reader's CONDITIONAL verdict is appropriate. My proposed human-judgment check would settle whether the concern lands. If it fails to land (high agreement, persistent VTune gain), the paper should move to ACCEPT (modulo error bars).","tokens_in":23372,"tokens_out":8581,"duration_ms":106620,"concrete_test":"Select a stratified random sample of 200 model responses for each of H-Verify and C-Verify per dataset from Table 1, plus 100 responses per condition from the VTune/IT rows of Table 2, covering both positive and negative templates. Two annotators, with access to the relevant video segments, independently score each response as correct/incorrect using the rubric in Figure 13; a third annotator adjudicates disagreements. Recompute relative consistency scores from the human labels. If human-judged open-source H-Verify scores rise from ~50% to above ~65%, or the VTune advantage shrinks to within noise on human-scored responses, the GPT loop is responsible for the reported findings; if human-GPT agreement is ≥95% on the sampled responses and the VTune advantage persists, the central claims stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The numbers that carry the paper's two central claims—open-source Video-LLMs are near chance at self-verification (Table 1) and VTune produces significant gains (Tables 2–4)—all flow through GPT-4o-mini in the same pipeline. GPT-4o-mini generates the aligned/misaligned/compositional test queries (§3.1), generates the VTune verification queries and corrections (Appendix D, Figure 11), and judges whether a model's verification response matches the expected answer (§3.3, Figure 13). The human study in §3.1 validates only the generated sentences (92.2% 'well-matched'), not the answer-judge's decisions. Consequently, if GPT-4o-mini's judge favors responses that resemble its own phrasing or output format, two things follow: the ~50% H-Verify scores of open-source models could be depressed by judge bias rather than by genuine inconsistency, and VTune—trained on GPT-4o-mini's own query/correction style and tested on GPT-4o-mini-generated questions—would be artificially advantaged. This is the weakest load-bearing assumption in the paper: it is external validity rather than internal logic, but it directly determines the magnitude (and even sign) of the headline metrics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies the consistency of Video-LLMs in temporal grounding by introducing two evaluation sets, Charades-CON and ActivityNet-CON, and four probes: rephrased grounding, shifted grounding, holistic verification, and compositional verification. The authors evaluate ten open- and closed-source models and report that open-source Video-LLMs often achieve near chance-level verification consistency while closed-source models are more consistent. They further analyze prompting and instruction tuning, find these approaches unstable or ineffective for consistency, and propose event temporal verification tuning (VTune), which reformulates temporal grounding as a verification task. VTune is reported to improve both grounding and consistency over instruction-tuning baselines on Charades-STA and ActivityNet-Captions.","tokens_in":23592,"tokens_out":6867,"duration_ms":71483,"significance":"If the findings hold, the paper provides a useful diagnostic benchmark and a practical training recipe for improving temporal consistency in Video-LLMs. The study's strengths are its broad model coverage, including time-aware models, the human validation of generated query quality, the clear presentation of absolute and relative consistency scores, and the open-sourced data and code. The central negative result---that many open-source Video-LLMs cannot reliably confirm their own moment predictions---would be an important community-wide signal. However, the headline numbers currently rest on a single GPT-4o-mini generator-judge pipeline, a consistency metric computed only on IoU-filtered correct predictions, and training-data comparisons that are not matched in scale. These issues constrain the external validity of the near-chance and VTune claims until addressed.","major_comments":[{"comment":"The claim that open-source models perform \"near chance-level\" in holistic verification is not statistically supported. Relative consistency is computed only for predictions with IoU≥0.5, and for low-recall models this leaves a small subset; for example, Video-ChatGPT on Charades-CON has Ground=14.4 and H-Verify relative=44.8, corresponding to roughly 100 samples, which gives a standard error of about 5 percentage points and makes 44.8% statistically indistinguishable from 50%. No confidence intervals, multiple runs, or significance tests are reported for any of the scores in Tables 1–4. Please add confidence intervals or error bars and report the sample sizes behind each relative score.","section":"§3.3, Table 1"},{"comment":"The evaluation pipeline is a single-LLM loop: GPT-4o-mini generates the aligned/misaligned/compositional test queries (§3.1), generates VTune's verification queries and corrections (Appendix D, Figure 11), and judges whether a model's verification answer matches the expected answer (§3.3, Figure 13). The human study in §3.1 validates only the quality of the generated sentences, not the judge's decisions. This creates a risk that the near-chance scores for open-source models and the VTune gains reflect familiarity with GPT-4o-mini's phrasing and judged response style rather than genuine temporal consistency. Please provide a human evaluation of the judge's decisions, report agreement with an alternative judge model, and/or evaluate on an independently curated test set.","section":"§3.1, §3.3, Appendix D"},{"comment":"The comparison between instruction tuning (IT) and VTune is confounded by training-data scale. According to Table 12, VTune uses 99,244 queries on Charades-STA versus 12,408 for IT, and 205,510 versus 37,330 on ActivityNet-Captions. The gains in Table 2 could therefore be due to more training data rather than to the verification objective. Please include an IT baseline trained on the same total number of queries (for example, by adding additional grounding queries), or provide a VTune ablation matched to IT's data size, to isolate the effect of event temporal verification.","section":"§5.3, Tables 2 and 12"},{"comment":"For VTG-LLM, the verification scores appear to mix consistency with instruction-following ability. Appendix B.3 states that VTG-LLM \"frequently generates timestamps as answers, even when asked for verification probes,\" and such outputs are presumably scored as inconsistent. The unusually low relative H-Verify scores (23.1 on Charades-CON and 10.9 on ActivityNet-CON in Table 1) may therefore reflect an instruction-following failure rather than a temporal-consistency failure. Please report the frequency of invalid responses separately and analyze consistency on the subset of valid responses.","section":"Appendix B.3, Table 1"}],"minor_comments":[{"comment":"The S-Ground (0.7) column contains entries that look like typos; for example, Video-LLaMA on Charades-CON shows 8.3, which is higher than the corresponding IoU=0.5 value of 5.3. Please verify all entries in this table.","section":"Table 10"},{"comment":"The word-order modification rule is abbreviated as \"WO\" in the main text but as \"OC\" in Table 14; please harmonize the abbreviations.","section":"§3.1, Table 14"},{"comment":"The frame-count selection for GPT-4o is performed on 200-video development subsets; please clarify whether a similar selection was performed for Gemini and whether the chosen frame counts are the ones listed in Table 1.","section":"Appendix B.2, Table 9"},{"comment":"The statement that TimeChat achieves \"state-of-the-art results on both datasets\" is based on a limited set of baselines (BM-DETR, Mr.BLIP, HawkEye, VTG-LLM, Video-LLaMA, and TimeChat); consider wording this as \"best among compared methods\" or adding more recent state-of-the-art results.","section":"§5.3, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The core tension is the external validity of the GPT-4o-mini generator-judge loop. If the authors can supply human judge validation, an alternative judge or independently curated test set, and a matched-scale IT baseline, the paper would be much stronger. The near-chance claim for open-source models is the most influential result and currently lacks statistical support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper gives the video-LLM field a new axis—temporal grounding consistency under rephrasing, shifting, and self-verification—and it backs the idea with two datasets, a probing suite, and a tuning method (VTune) that actually moves numbers. I think the central qualitative claim is right: most open-source Video-LLMs are near chance (around 50%) at verifying their own temporal predictions, and closed models are substantially better. That is worth knowing.\n\nWhat is actually new: prior work looked at temporal understanding via QA benchmarks (TempCompass, MVBench, ViTaTeC) but not at whether a model agrees with its own moment prediction when you rephrase the query, shift the moment, or ask it to verify components. The probe design is sensible, the rephrasing rules are concrete, and they do a human study on the generated queries (92.2% well-matched). The VTune idea—reformulating grounding as verification with aligned/misaligned queries and temporal shifts—is a reasonable, clearly described extension of instruction tuning, and the gains over plain instruction tuning in Table 2 are large. They also release data and code, which is real credit.\n\nSoft spots, in order. First, the generator-judge loop. GPT-4o-mini writes the test queries, writes VTune's training queries, and scores model answers; the human validation covers only the generated sentences, not the answer-scoring decisions. That means part of the VTune advantage could be adaptation to one LLM's phrasing and judging style, and part of the open-source near-chance scores could be judge bias rather than genuine inconsistency. This is the strongest reason to treat the headline numbers as provisional. It is not fatal—the direction of the finding is plausible—but it is a real external-validity gap. Second, no error bars or multiple runs anywhere, so near-chance and significant-improvements claims lack statistical support. Third, the main VTune vs IT comparison does not match total training data volume; Table 12 shows VTune has far more queries, so scale is confounded with method. The ablation in Table 4 partially addresses this but not fully. Fourth, consistency is measured only on predictions that already pass IoU>=0.5, which can discard most of a model's outputs; the relative scores are thus conditional on a narrow subset.\n\nThe math and data work are otherwise clean, and the citation pattern is honest; self-citation to the authors' own prior VTG work is ordinary. Who this is for: anyone evaluating or improving temporal grounding in Video-LLMs. It deserves a serious referee; the right outcome is likely a revise-and-resubmit asking for a human-judged subset, an independently written test set, error bars, and a matched-data-scale baseline. I would not desk-reject it.","headline":"A genuinely useful consistency benchmark and tuning recipe for Video-LLMs, with a real caveat: the same LLM writes the test questions, trains the method, and scores the answers.","tokens_in":859,"tokens_out":747,"would_cite":true,"duration_ms":25062,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source video language models verify their own temporal predictions at near chance, and the authors' verification-based tuning improves both consistency and grounding.","keywords":["video large language models","temporal grounding","prediction consistency","self-verification","VTune","Charades-CON","ActivityNet-CON","moment retrieval"],"falsifier":"Re-run the full protocol with probe paraphrases written by humans or a second independent LLM and with answers graded by human raters instead of GPT-4o-mini; if open-source models' holistic verification consistency rises well above 50%, or if VTune's advantage over plain instruction tuning disappears, the central claim of near-chance self-verification and the reported gains would be artifacts of the GPT-4o-mini evaluation scaffold.","tokens_in":23096,"feed_emoji":"🎬","tokens_out":9195,"duration_ms":84989,"temperature":0.7,"pith_summary":"This paper tries to establish that current open-source Video-LLMs are not consistent in temporal comprehension: after a model predicts a video moment, its behavior under paraphrased queries, shifted video content, and direct verification questions often contradicts its own initial prediction. To make the claim measurable, the authors build Charades-CON and ActivityNet-CON from two standard grounding datasets, adding aligned, misaligned, and compositional queries, and define four probes: rephrased grounding, shifted grounding, holistic verification, and compositional verification. Evaluated on ten models, most open-source systems, including time-aware models built specifically for temporal grounding, show near chance-level holistic verification (about 50% relative consistency), while GPT-4o and Gemini 1.5 Flash are far more self-consistent. The paper then proposes event temporal verification tuning (VTune), which trains a model to confirm matching query-moment pairs and correct mismatched ones, and reports that VTune improves both grounding accuracy and consistency. If the finding is right, accuracy on standard temporal grounding benchmarks overstates how reliably open-source Video-LLMs understand video.","feed_headline":"Open video models score near chance at self-checking","feed_subtitle":"Rephrasing, shifting, or verifying a moment exposes inconsistent temporal grounding in eight open-source systems.","key_machinery":"The machinery is a verification-based evaluation protocol. Formally, grounding is written as $m=\\mathrm{TempG}(v,q)$ and verification as $a=\\mathrm{TempV}(v,q,m)\\in\\{\\text{Yes},\\text{No}\\}$; the paper probes whether these two operations cohere. Four probes do the work: Rephrased Grounding measures the intersection-over-union overlap between predictions for a query and its paraphrases; Shifted Grounding moves the ground-truth moment to another temporal location and tests whether the model follows; Holistic Verification asks whether the whole query occurs in the model's own predicted moment, with misaligned queries as negative controls; Compositional Verification decomposes the query into subject, action, and relation and tests each component. All probe data are generated by GPT-4o-mini in aligned, misaligned, and compositional forms, and free-form answers are graded by GPT-4o-mini as Yes/No. The proposed fix, VTune, is the same verification viewpoint turned into training data: aligned queries paired with positive moments, misaligned queries paired with positive moments (requiring the model to identify and correct what changed), and aligned queries paired with negative moments (requiring relocation).","core_discovery":"The central discovery is that a Video-LLM's ability to localize a moment and its ability to stand behind that localization come apart. On the paper's evaluation, models that pass the grounding bar (their predicted moment overlaps the ground truth by at least 0.5) still answer yes/no questions about whether the queried event is inside their own predicted moment at roughly chance level, often around 50% relative consistency. The same models also largely fail to re-ground the same visual content when it is shifted to a different temporal position, while they handle rephrased queries somewhat better; this pattern suggests reliance on language priors rather than visual comprehension. The paper reports that time-aware models such as TimeChat, VTimeLLM, and VTG-LLM improve grounding but not verification consistency, and that closed-source models are considerably more consistent. Its proposed repair, VTune, converts grounding into a verification task with aligned queries paired to positive moments and misaligned or displaced query-moment pairs, and the reported result is simultaneous improvement in both consistency and grounding on both datasets.","pith_inferences":["Consistency probes could serve as a pre-deployment sanity check for any Video-LLM asked to justify its own temporal answers in applications such as video search, highlight detection, or surveillance review, since a model that cannot confirm its own prediction cannot be trusted to explain it.","The near-chance verification scores may hide a systematic Yes/No bias rather than pure randomness; a balanced-probe audit that separates biased models from genuinely uncertain ones would make the failure mode precise.","VTune's recipe of confirming unchanged queries, correcting altered ones, and relocating displaced ones is not specific to video and could transfer to image grounding or referring-expression tasks, although the paper only tests temporal grounding.","If language-prior reliance is the cause of these inconsistencies, then models trained with VTune should show less performance drop when visual content is degraded, a testable prediction the paper does not run."],"forward_implications":["Future temporal-grounding benchmarks should report consistency as a separate axis, because high grounding accuracy can coexist with chance-level self-verification.","Standard fixes such as chain-of-thought prompting, description prompting, and instruction tuning on target datasets should not be assumed to repair consistency; the paper finds their gains unstable and sometimes negative.","Training that treats grounding as verification (VTune) is a viable way to improve both consistency and grounding, with TimeChat-VT reaching state-of-the-art results on the Charades-STA and ActivityNet-Captions test sets.","Specializing a model for temporal grounding does not by itself make its temporal comprehension trustworthy, since the time-aware models still fail verification.","Closed-source models, though not designed for grounding, set the consistency bar that open-source Video-LLMs need to reach."],"supporting_citations":[{"why":"Supplies the Charades-STA moments and queries from which Charades-CON is built.","marker":"[7]"},{"why":"Supplies the ActivityNet-Captions moments and queries from which ActivityNet-CON is built.","marker":"[14]"},{"why":"Video-LLaVA serves as a general-purpose open-source baseline with near-chance holistic verification.","marker":"[20]"},{"why":"Video-ChatGPT serves as a general-purpose open-source baseline evaluated across all probes.","marker":"[24]"},{"why":"Video-LLaMA is a general-purpose baseline and one of the two models fine-tuned with VTune.","marker":"[41]"},{"why":"VTG-LLM is a time-aware baseline showing strong grounding but poor verification consistency.","marker":"[8]"},{"why":"VTimeLLM is a time-aware baseline with strong grounding but limited verification gains.","marker":"[9]"},{"why":"TimeChat is a time-aware baseline and the other model fine-tuned with VTune.","marker":"[31]"},{"why":"GPT-4o is the closed-source comparison model that shows relatively high consistency.","marker":"[1]"},{"why":"Gemini 1.5 Flash is the closed-source comparison model with the strongest shifted-grounding consistency.","marker":"[30]"}],"fun_headline_variants":["Video-LLMs fail self-check: grounding ≠ understanding","Video-LLMs can't verify their own moment picks","VTune closes video models' temporal self-check gap","Self-check blind spot: Video-LLMs near chance on verification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's weakest point is that both the probe questions and the grading of model answers come from GPT-4o-mini, and consistency is measured only on predictions that already passed a 0.5 overlap accuracy filter, so if that assistant's wording or judging preferences favor some models over others, the near-chance scores and VTune gains could be partly artifacts of the evaluation scaffold.","fun_headline_variants_meta":{"raw":{"variants":["Video-LLMs fail self-check: grounding ≠ understanding","Video-LLMs can't verify their own moment picks","VTune closes video models' temporal self-check gap","Self-check blind spot: Video-LLMs near chance on verification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000525,"raw_usage":{"total_tokens":2524,"prompt_tokens":925,"completion_tokens":1599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":1531}},"tokens_in":541,"tokens_out":1599,"duration_ms":12879,"temperature":1.0,"reasoning_tokens":1531,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:00:57.153972+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full protocol with probe paraphrases written by humans or a second independent LLM and with answers graded by human raters instead of GPT-4o-mini; if open-source models' holistic verification consistency rises well above 50%, or if VTune's advantage over plain instruction tuning disappears, the central claim of near-chance self-verification and the reported gains would be artifacts of the GPT-4o-mini evaluation scaffold.","supporting_citations":[{"cited_title":"Dense-captioning events in videos","cited_arxiv_id":null,"evidence_quote":"Supplies the ActivityNet-Captions moments and queries from which ActivityNet-CON is built."},{"cited_title":"Video-chatgpt: Towards detailed video understanding via large vision and language models","cited_arxiv_id":null,"evidence_quote":"Video-ChatGPT serves as a general-purpose open-source baseline evaluated across all probes."},{"cited_title":"Vtimellm: Empower llm to grasp video moments","cited_arxiv_id":null,"evidence_quote":"VTimeLLM is a time-aware baseline with strong grounding but limited verification gains."},{"cited_title":"Timechat: A time-sensitive multimodal large lan- guage model for long video understanding","cited_arxiv_id":null,"evidence_quote":"TimeChat is a time-aware baseline and the other model fine-tuned with VTune."}],"review_version":1}