{"id":"47283e27-ee87-4ae6-b161-9eaa452afbb9","arxiv_id":"2412.13845","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey arguing that video-LLMs rely on pretrained encoders and short-biased datasets, leaving them weak at long-term temporal reasoning such as causality and event progression.","lead":"This paper reviews whether AI systems that watch videos truly understand time, and argues they mostly track short visual patterns without grasping causality or long event chains. It maps current models, datasets, and evaluation gaps, which is useful for researchers deciding where to invest in temporal reasoning.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim that LLMs lack direct temporal awareness is asserted with no direct test and conflicts with the paper's own Section 3 text and cited work.","rationale":"The reader's weakest assumption targets the comparability of Figures 4-6. I agree that the charts lack protocol detail, but that concern attacks a supporting observation ('no single video-LLM excels across all tasks'), not the paper's central negative thesis. The thesis that LLMs lack direct temporal awareness is the premise on which the paper's conclusions and future directions rest. That premise is asserted rather than demonstrated, and the paper's own references (e.g., [64], [106]) undercut it. The proposed text-only and shuffled-frame tests would settle whether the premise is empirically true. The reader's CONDITIONAL verdict remains appropriate: the survey components (Tables 1-3, fusion taxonomy) are useful, but the central claim needs either direct evidence or a precise, falsifiable definition before it can be accepted. Hence the verdict is unchanged, though the load-bearing concern differs from the reader's weakest assumption.","tokens_in":31623,"tokens_out":5641,"duration_ms":49050,"concrete_test":"Run the LLM backbone of a representative video-LLM (e.g., Vicuna-7B as used in Video-LLaVA) on a text-only temporal reasoning benchmark, with no image or video input. Use TempCompass's text-only split, or convert TGIF-QA frame-order questions to text (e.g., 'Which happened first: a man pours water, then a man drinks?'). If the text-only backbone exceeds chance on order and causality questions, the claim that LLMs lack direct temporal awareness is contradicted. Second, run Video-LLaVA on a video temporal-order task with (a) normal frames and (b) frame order shuffled. If (a) significantly outperforms (b), the video-LLM system demonstrably uses temporal order information, so the blanket conclusion 'lack temporal awareness' requires qualification. This separates 'the encoder provides temporal structure' from 'the LLM cannot reason about time'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assertion is in Section 3 ('Analysis and Discussion'): 'LLMs lack direct temporal awareness' and 'fall short in understanding long-term temporal dependencies.' This negative thesis is what makes the paper a critique rather than a survey, and the entire Future Directions section rests on it. Yet the paper provides no experiment that isolates temporal understanding; the claim is supported only by architectural observation. It is also internally tensioned: the same section states that LLMs 'can infer temporal relationships through contextual cues such as \"first\", \"then\", and \"after\"' from textual pretraining, which is itself a form of temporal awareness. The paper cites Gurnee & Tegmark (Ref [64], 'Language Models Represent Space and Time') and TempCompass (Ref [106]) without reconciling their findings with the negative claim. The term 'direct temporal awareness' is never operationalized: if a model uses frame-order tokens or positional embeddings, is that direct or indirect? Without a definition, the claim is unfalsifiable. The benchmark figures (Figures 4-6) are a secondary issue: even if the plotted numbers are exactly right, they show task performance, not the absence of temporal reasoning. The observation that no single video-LLM excels across tasks does not establish that LLMs fail at temporal reasoning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a survey/critical review of video-based large language models (video-LLMs) and their temporal reasoning capabilities. It reviews recent video-LLM architectures and their interaction mechanisms with pretrained visual encoders (Tables 1–2), catalogs video datasets across tasks such as action recognition, video QA, captioning, retrieval, and anomaly detection (Table 3), and presents benchmark comparisons of video-LLMs (Figures 4–6). The central claim is that LLMs, despite achieving strong task performance when paired with pretrained video encoders, lack direct temporal awareness and fall short in understanding long-term temporal dependencies. The paper identifies dataset limitations (lack of temporal annotations, short-term bias, low diversity) and proposes future directions including joint encoder-LLM training, richer temporal annotations, and new fusion architectures.","tokens_in":31815,"tokens_out":4742,"duration_ms":44147,"significance":"The paper is a broad and timely survey that usefully organizes a fast-moving literature on video-LLMs, fusion mechanisms, and video datasets. Its strength is the critical framing: it explicitly argues that benchmark comparisons are often inconsistent and that dataset design shapes temporal reasoning, which is a valuable message for the community. The GitHub repository and the extensive tables (Tables 1–3) provide a convenient reference. However, the paper's central claim about the absence of direct temporal awareness is asserted rather than demonstrated, and the benchmark figures are presented without a documented extraction protocol. If the central claim were supported by controlled evidence or a systematic benchmark analysis, the paper would be a significant position piece; as it stands, its conclusions outrun its evidence. The paper contains no new experiments, code, or machine-checked derivations, so its value depends on the accuracy and representativeness of the plotted numbers and the force of its conceptual argument.","major_comments":[{"comment":"The central thesis that LLMs 'lack direct temporal awareness' is load-bearing for the paper, but it is neither operationalized nor tested. The same section states that LLMs 'can infer temporal relationships through contextual cues such as \"first\", \"then\", and \"after\"', which is itself a form of temporal inference, and the paper cites Ref. [64] (Gurnee and Tegmark) and Ref. [106] (TempCompass) without reconciling their evidence of temporal representation in LLMs with the negative claim. Please define what would count as 'direct' temporal awareness (e.g., whether positional embeddings, frame-order tokens, or learned temporal projections count), state the claim as a falsifiable hypothesis, and support it with either a controlled experiment or a systematic analysis of existing temporal-reasoning benchmarks such as TempCompass, TGIF-QA, or MVBench.","section":"Section 3, 'Analysis and Discussion'"},{"comment":"The benchmark figures are presented without the extraction protocol needed to support the claim that no video-LLM excels across tasks. For each plotted number, the paper should report the exact model checkpoint, prompt template, subtitles-on/off condition, frame sampling, metric version, and original source. As written, the numbers mix models, tasks, and possibly evaluation settings, so the performance gaps in Figures 4–6 do not logically follow. This is a particular problem because the paper itself (Section 'Fair evaluation is needed') argues that inconsistent evaluations produce misleading conclusions; the charts should not reproduce the very inconsistency they criticize.","section":"Figures 4–6"},{"comment":"The conclusion states as a finding that LLMs 'fall short in understanding long-term temporal dependencies', but the evidence consists of architectural observations and aggregate benchmark charts rather than a comparison that isolates temporal reasoning from other factors. A model could fail on long videos because of context-window limits, token subsampling, or dataset bias rather than because it lacks temporal awareness. The causal attribution to 'the encoders' focus on short-term patterns' is underdetermined. Please either add an analysis that isolates the temporal component (e.g., comparing shuffled vs. ordered frames, short vs. long clips, or temporal-order questions vs. content questions) or explicitly soften the conclusion to a research hypothesis.","section":"Section 5, Conclusion"}],"minor_comments":[{"comment":"The keyword 'Language language models' contains a typo; it should be 'Large language models'.","section":"Keywords"},{"comment":"Some dataset facts appear questionable: MSVD-QA is listed as having 'Start and end timestamps provided', but MSVD-QA questions are not typically timestamped; please verify and clarify the annotation type. Also, the average length listed for EPIC-KITCHENS (~458 s) refers to full raw videos rather than the annotated segments used in most benchmarks; please clarify.","section":"Table 3"},{"comment":"References [26], [170], and [187] lack complete publication data (venue/year); please complete these entries.","section":"References"},{"comment":"The caption says 'Performance (accuracy) comparison', but the exact metric (overall accuracy vs. per-category accuracy, with or without subtitles) should be specified in the caption or text, and the definition should match the source benchmark.","section":"Figure 4"},{"comment":"The description of ActionFormer as part of 'video LLMs' is imprecise; ActionFormer is a temporal action localization transformer, not an LLM-based video model. Please tighten terminology to avoid conflating video transformers with video LLMs.","section":"Section 3, 'State-of-the-Art video LLMs'"},{"comment":"The discussion of dataset limitations appears twice with overlapping content (Section 2 'Datasets for video understanding' and Section 3 'Video datasets: an enabler or bottleneck?'); consider consolidating to reduce redundancy.","section":"Section 2 and Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper contains many self-citations (e.g., Refs. [21, 26, 125, 157, 160–169, 212]), which is not a problem per se, but the authors should ensure that Table 1 and the benchmark figures include the latest models beyond their own group's work. There is no circularity by construction, since the paper fits no model and derives no equations; however, the central claim needs to be either tested or reframed as a hypothesis before publication. The scope as a WWW Companion survey is acceptable, but the review should be self-consistent with its own methodological warnings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a survey/position piece, not a new result. The curated tables of video-LLMs, encoders, and datasets are genuinely useful, and the discussion of inconsistent evaluation practices is sensible. What is not solid is the paper's central negative thesis — that LLMs lack direct temporal awareness — which is asserted from architectural observation rather than tested, and the benchmark figures are plotted without enough protocol detail to support the conclusions drawn from them.\n\nThe paper does well by aggregation. Table 1 collects 66 video-LLMs with their interaction/fusion mechanisms, Table 2 lists encoders and pretraining data, and Table 3 maps datasets to temporal annotation availability. That is real reference value, especially for someone entering the field. Figure 3's distribution of fusion mechanisms and modalities is a fair descriptive summary. The 'fair evaluation is needed' section makes a legitimate point: across-paper comparisons are frequently apples-to-oranges, and within-paradigm benchmarking would be more informative.\n\nThe soft spots are real and load-bearing. Section 3 states without a direct test that 'LLMs lack direct temporal awareness.' The term is never operationalized — if positional embeddings or frame-order tokens count as temporal awareness, then many current systems have it; if not, the claim is unfalsifiable. The same section concedes LLMs can infer temporal relations from textual cues like 'first' and 'then,' which is a form of temporal awareness, creating an internal tension the paper doesn't resolve. Cited work on learned representations of time (Gurnee & Tegmark) and on video-LLM temporal benchmarks (TempCompass) is not reconciled with the negative claim. Figures 4-6 compare models across Video-MME, MSVD-QA, MSRVTT-QA, etc., but the paper doesn't specify the checkpoint, prompt, subtitles-on/off, sampling, or metric version for each number. Given the paper's own argument that evaluations are inconsistent, the performance gaps in those figures could be artifacts of protocol differences. The conclusion that no video-LLM excels across tasks may hold, but the charts don't establish it.\n\nWho this is for: readers wanting a quick map of video-LLM architectures, fusion mechanisms, and dataset temporal annotations. The survey value is real. Who it is not for: anyone expecting a controlled study of temporal reasoning in LLMs.\n\nIf I were the editor: send it to peer review, but as a position/survey paper, and tell the referee to push for either a controlled temporal-reasoning experiment or a careful restatement of the central claim as a hypothesis about current encoder-LLM interfaces. It shouldn't be accepted in its current form, but it deserves referee time.","headline":"A useful survey of video-LLMs and datasets whose central claim about temporal awareness is asserted rather than tested.","tokens_in":32329,"tokens_out":2975,"would_cite":false,"duration_ms":27586,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that today's video-LLMs achieve benchmark success without genuinely understanding time, because temporal structure is supplied by pretrained encoders and datasets, not learned by the LLM.","keywords":["temporal reasoning","video understanding","large language models","video-LLMs","pretrained visual encoders","temporal annotations","video benchmarks","multimodal fusion"],"falsifier":"Run one controlled experiment: take a current video-LLM and ask the same set of order-sensitive questions on the same videos in correct and in temporally shuffled frame order, with subtitles off and a fixed prompt. If accuracy on the shuffled version stays at the same level, the model is not using order and the claim of lacking temporal awareness is supported; if accuracy collapses on the shuffled version, the claim that LLMs lack direct temporal awareness would be falsified.","tokens_in":31405,"feed_emoji":"⏳","tokens_out":5804,"duration_ms":53693,"temperature":0.7,"pith_summary":"The paper sets out to answer whether large language models used for video understanding actually understand time. Its answer is no: LLMs paired with pretrained visual encoders can label actions, answer questions, and caption clips, but they lack direct temporal awareness, in the sense that they never see a sequence they have to reason about as a sequence; the encoder supplies tokens and motion features, and the LLM attends over them without an inherent model of event order, causality, or duration. The paper supports this by reviewing recent video-LLMs, their encoder-LLM fusion mechanisms, and video datasets, arguing that both the encoders and the datasets are biased toward short-term motion and static appearance, and that evaluations mix incompatible protocols. If this is right, current benchmark success is not evidence of temporal reasoning, and progress depends on joint encoder-LLM training, explicit temporal annotations, and long-context evaluation.","feed_headline":"No video LLM truly understands time, review argues","feed_subtitle":"Benchmark scores mask a gap: models and datasets favor short clips and static cues, not event order or causality.","key_machinery":"The load-bearing object is the interaction/fusion mechanism between a pretrained visual encoder and an LLM: projection layers, cross-attention modules, Q-Formers, and temporal-specific connectors such as temporal samplers and scene-level sequential alignment. The paper's argument is that this coupling is where temporal information is filtered or lost. Encoders such as CLIP, I3D, TimeSformer, and Video Swin supply frame-level tokens and short-range motion features; the LLM then aligns those with language without any inherent model of sequence order, so any true temporal structure must come from outside. The distribution of fusion mechanisms across recent models, and the benchmark plots built on this coupling, are the evidence the review leans on.","core_discovery":"On the paper's own terms, the discovery is a negative result about current architectures: state-of-the-art video-LLMs achieve competitive task performance through the interaction of pretrained visual encoders and LLMs, but this performance does not amount to temporal comprehension. The paper claims that LLMs lack direct temporal awareness, that encoders focus on short-term patterns and fragmented cues, and that datasets rarely carry the temporal annotations—event order, duration, causality—that would let either component learn long-term dependencies. The conclusion is that no current video-LLM excels across video tasks and that the apparent progress is partly an artifact of inconsistent evaluation.","pith_inferences":["If the paper is right, a cheap diagnostic follows: shuffling frame order or reversing event order in existing test videos should produce a large accuracy drop on order-sensitive questions, and this could be measured today without new training.","The same short-term bias would likely carry over to video generation and long-context models, because a longer context window does not by itself supply training signal about causality or event progression.","One way to test the dataset bottleneck directly is to take a fixed video-LLM and fine-tune it on a small set of temporally annotated long videos; if temporal reasoning improves substantially, the bottleneck is data rather than architecture.","The paper's critique generalizes to time-series and event-forecasting uses of LLMs, where surface task success may again come from textual priors rather than from an internal model of time."],"forward_implications":["Benchmark numbers on Video-MME, MSVD-QA, MSRVTT-QA, ActivityNet-QA, and retrieval or captioning sets should not be read as measures of temporal reasoning, because they mix encoder quality, language priors, and short-term cues.","Improving temporal understanding requires training encoders and LLMs jointly on temporally annotated data, not just scaling model size or context length.","New benchmarks must include explicit temporal labels such as event order, duration, and causality, and must use long-form videos; otherwise progress in temporal reasoning will be mismeasured.","Most current models use projection layers and cross-attention rather than temporal-specific fusion mechanisms, so temporal structure is imposed only weakly and inconsistently.","Comparisons of video-LLMs against traditional video models such as I3D, SlowFast, and Video Swin are unfair; within-paradigm benchmarking against other video-LLMs is needed to reveal real progress."],"supporting_citations":[{"why":"The canonical video-LLM that fuses visual and textual modalities; the paper uses it as the baseline example of encoder-LLM integration.","marker":"[5]"},{"why":"A prior study that attempts to instill a sense of time in video-language models, providing the baseline against which the paper claims temporal awareness is missing.","marker":"[7]"},{"why":"The Video-MME benchmark whose numbers underpin the Figure 4 comparison and the claim that evaluation protocols are inconsistent.","marker":"[49]"},{"why":"VideoChat2 and the MVBench dataset are central examples in the state-of-the-art comparison and the argument that no video-LLM excels across tasks.","marker":"[94]"},{"why":"A prior study questioning whether video LLMs really understand videos, cited to support the paper's skepticism about temporal reasoning.","marker":"[106]"},{"why":"EgoSchema, a long-form egocentric benchmark, illustrates the long-term dependency gap that the paper argues current models and datasets cannot address.","marker":"[116]"},{"why":"CLIP is the standard pretrained image encoder used by many video-LLMs, and its spatial bias is central to the claim that temporal structure must be borrowed from elsewhere.","marker":"[122]"}],"fun_headline_variants":["LLMs fake video time understanding, study finds","Video LLMs lack true temporal reasoning, review shows","Time-blind LLMs: video models miss causality","Do video LLMs know time? Hard no, says review"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the benchmark numbers plotted across Figures 4-6 being comparable; if those models were evaluated under different prompts, subtitles settings, sampling, or metric versions, the performance gaps and the conclusion that no video-LLM excels across tasks do not follow.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fake video time understanding, study finds","Video LLMs lack true temporal reasoning, review shows","Time-blind LLMs: video models miss causality","Do video LLMs know time? Hard no, says review"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1328,"prompt_tokens":943,"completion_tokens":385,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":322}},"tokens_in":559,"tokens_out":385,"duration_ms":4101,"temperature":1.0,"reasoning_tokens":322,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:43:53.416188+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run one controlled experiment: take a current video-LLM and ask the same set of order-sensitive questions on the same videos in correct and in temporally shuffled frame order, with subtitles off and a fixed prompt. If accuracy on the shuffled version stays at the same level, the model is not using order and the claim of lacking temporal awareness is supported; if accuracy collapses on the shuffled version, the claim that LLMs lack direct temporal awareness would be falsified.","supporting_citations":[],"review_version":1}