FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

Changsong Li; Jinlan Fu; Min Zhang; Qian Chen; See-kiong Ng; Xipeng Qiu

REVIEW 4 major objections 5 minor 2 cited by

FutureOmni is the first benchmark that tests whether multimodal LLMs can forecast what happens next from video plus audio, and it finds the best model tops out at 64.8% accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:23 UTC pith:OLCX4ZUO

load-bearing objection A genuinely useful new benchmark for audio-visual future forecasting, but the OFF training results rest on an undocumented data split that needs to be cleaned up before the paper is publishable. the 4 major comments →

arxiv 2601.13836 v2 pith:OLCX4ZUO submitted 2026-01-20 cs.CL cs.CVcs.MM

FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

Qian Chen , Jinlan Fu , Changsong Li , Min Zhang , See-Kiong Ng , Xipeng Qiu This is my paper

classification cs.CL cs.CVcs.MM

keywords future forecastingomni-modal reasoningaudio-visual integrationmultimodal LLM benchmarkcausal reasoninginstruction tuningvideo understanding

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces FutureOmni, a benchmark of 919 videos and 1,034 multiple-choice questions designed to test whether multimodal language models can forecast what happens next from combined audio and visual context. It argues that previous benchmarks only test retrospective understanding, so the field lacks a measure of omni-modal future prediction — a capability relevant to autonomous driving, safety monitoring, and assistive systems. Evaluating 20 models, the paper finds that even the best system, Gemini 3 Flash, reaches only 64.8% accuracy, and that removing either audio or video costs roughly five accuracy points, evidence that genuine cross-modal integration is required rather than single-modality shortcuts. The paper also proposes an Omni-Modal Future Forecasting (OFF) training strategy built on a rationale-augmented 7K-sample dataset, reporting gains on FutureOmni that transfer to other audio-visual and video-only benchmarks. A reader should care because the benchmark defines a new capability axis for MLLMs and the training result suggests a concrete path toward improving it.

Core claim

On its own terms, the paper's central discovery is that future forecasting from audio-visual context is a distinct and underserved capability of MLLMs, and the performance gap is driven mainly by visual perception errors and cross-modal reasoning failures rather than missing world knowledge. The benchmark's four adversarial distractor types force models to reason across modalities and about the direction of time. Under these conditions, omni-modal models outperform video-only models, and the best accuracy is 64.8%. Fine-tuning on FutureOmni-7K with OFF improves open-source models by up to +3.87% overall, with the largest gains in speech-heavy scenarios, and gains carry over to out-of-domain

What carries the argument

The load-bearing component is the audio-coordinated video selection filter, which keeps only the 50% of videos whose captions change most when the audio track is removed, operationalizing 'audio causally matters' as 'caption similarity drops without audio.' The rest of the pipeline — temporal grounding via Gemini 2.5 Flash, causal-pair discovery and audio-causal scoring via DeepSeek-V3.2, distractor generation with four adversarial heuristics, and dual-stage GPT-4o plus human verification — turns those videos into forecasting QA pairs. The OFF training strategy wraps the same rationales into instruction-tuning samples so models learn the reasoning chain rather than just the answer.

Load-bearing premise

The benchmark's claim that audio is causally important rests on the assumption that videos where removing audio most changes an automatic caption are exactly the videos where audio genuinely drives future events — a proxy that may select unusual clips rather than representative ones.

What would settle it

Run the same 20-model evaluation on the 50% of collected videos that were discarded for showing low caption change when audio was removed; if the omni-modal advantage over video-only models shrinks or disappears there, the core 'audio-visual integration matters' conclusion is an artifact of the selection filter.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

If the reported gap is real, future forecasting should be tracked as its own benchmark dimension in MLLM development, separate from retrospective video QA.
Models that pass FutureOmni would, by construction, be able to use speech, sound effects, and music to anticipate immediate next events, a capability directly relevant to robotics, safety monitoring, and autonomous driving.
Short clips being hardest implies forecasting depends on accumulated context, so model builders should evaluate with variable context windows rather than fixed-length inputs.
OFF-style rationale-augmented instruction tuning is a reproducible recipe: roughly 7K samples and one LoRA epoch yield gains on the benchmark and transfer to other tasks, indicating future-forecasting skill is trainable.
The four distractor types double as a diagnostic: a model that fails specifically on audio-only or reverse-causal options can be fingerprinted as over-relying on one modality or confusing temporal direction.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Because the selection filter keeps only audio-dependent videos, FutureOmni likely overstates both the average difficulty of audio-visual forecasting and the benefit of audio; re-weighting results by the unfiltered video distribution would give a more calibrated ability estimate.
The attention-visualization result suggests a generic mechanism — forecasting training changes where the model looks, not just what it knows. A direct test would apply OFF to a model without an audio branch and check whether visual keyframe attention alone explains the gains on video-only benchmarks.
The pipeline itself is reusable: audio-coordinated filtering plus LLM causal-pair mining could generate forecasting benchmarks for specific high-stakes domains such as driving or factory safety, provided the causal-pair step is audited for hallucinated timestamps — a risk the paper's own correction prompt acknowledges.
A natural next step the paper does not take is to compare OFF against training on the same QA pairs without rationales, isolating whether the rationale component drives generalization or whether the gains come simply from more task-specific data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A genuinely useful new benchmark for audio-visual future forecasting, but the OFF training results rest on an undocumented data split that needs to be cleaned up before the paper is publishable.

read the letter

FutureOmni fills a real gap: no existing benchmark tests future-event prediction from joint audio-visual input, and the paper's core finding—that even the best proprietary model only hits 64.8%, with video-only models below 50%—is robust and interesting. The four adversarial distractor types (visual-only, audio-only, delayed, reverse-causal) are well designed, and the 20-model evaluation is broad. The modality ablation is also convincing: A+V beats video-only and audio-only substantially, and the speech-harder-than-music pattern holds across models. This is a solid benchmark contribution.

The main problem is the training half. The paper fine-tunes on FutureOmni-7K and evaluates on FutureOmni (1,034 QAs), but never says where FutureOmni-7K comes from, whether it overlaps with the evaluation videos or QAs, or whether any deduplication was done. If the 7K set contains eval samples or clips from the same videos, the reported gains (e.g., +3.87 for video-SALMONN2) could just be memorization. This is not a minor omission—it decides whether F2 is interpretable. The authors need to state the provenance of FutureOmni-7K, confirm the eval set is excluded, and ideally run a video-level dedup check.

Other soft spots are smaller. No human accuracy or inter-annotator agreement is reported, so we don't know how ambiguous the 1,034 QAs are; a human baseline would contextualize the 64.8% ceiling. The audio-dependency filter (retaining only the top-50% most audio-dependent videos) is disclosed, so it's a defensible design choice, but it means the benchmark is deliberately skewed toward audio-relevant scenarios—results shouldn't be read as representative of all video. The self-reference of using Gemini 2.5 Flash, DeepSeek-V3.2, and GPT-4o to build data that those same model families are evaluated on is worth a caveat, though not fatal. Table 5's numbers also look off at first glance; probably the averages are weighted by per-domain sample counts, but the paper doesn't say so, and the large per-domain swings (e.g., Qwen2.5-Omni dropping from 59.86 to 41.10 on Surveillance after OFF) need checking.

Overall: the benchmark itself is a contribution worth refereeing. The OFF training claim is conditional on the split issue being resolved. Send it to review, but make the authors clarify data provenance and add human validation. I'd bring this to a reading group and likely cite it once the data hygiene is documented.

Referee Report

4 major / 5 minor

Summary. The paper introduces FutureOmni, a benchmark for evaluating multimodal LLMs on future-event forecasting from joint audio-visual context. It contains 919 videos and 1,034 multiple-choice QA pairs across 8 domains, built by filtering YouTube videos for audio dependency, using Gemini 2.5 Flash for temporal grounding, DeepSeek-V3.2 for causal-pair extraction, and GPT-4o plus human verification for QA generation. The authors evaluate 20 open-source and proprietary MLLMs, report that all models struggle (best 64.8% by Gemini 3 Flash), and observe that omni-modal models outperform video-only ones. They then introduce a 7K instruction-tuning dataset and an OFF training strategy, reporting improvements on FutureOmni and on six out-of-domain benchmarks, plus attention visualizations suggesting better keyframe focus. The paper claims to be the first benchmark for omni-modal future forecasting and makes both the code and data publicly available.

Significance. If the claims hold, FutureOmni fills a genuine gap: it targets an understudied task, includes adversarial distractor types designed to force cross-modal reasoning, and evaluates a broad set of 20 models. The core finding that current models are far from solving audio-visual future forecasting is likely robust and useful to the community. The proposed OFF training strategy and the released artifacts are potentially valuable. However, the training-set provenance and its overlap with the evaluation set are not documented, which makes the OFF generalization claims currently unverifiable. The paper would be a solid benchmark contribution after these issues are resolved.

major comments (4)

[Sec. 5.1 / Sec. 3] The provenance of FutureOmni-7K is not specified. Section 3 describes a pipeline producing 1,034 evaluation QAs, while Section 5 introduces a separate 7K instruction-tuning set without stating whether it was generated from the same filtered videos, whether the 1,034 evaluation QAs were excluded, or whether any video-level deduplication was applied. Without this information, the reported gains in Tables 4-6 (e.g., +3.87 for video-SALMONN2) could reflect memorization of evaluation videos rather than improved forecasting. This is load-bearing for claim F2. Please document the construction of FutureOmni-7K, report the split and deduplication procedure, and provide overlap statistics.
[Sec. 4.2 / Table 3] The claim that text-based augmentations 'still fall short of the full A+V setting' is contradicted by MiniCPM-o 2.6, where V+S achieves 49.80 vs. A+V 48.54. Thus the conclusion that raw audio is necessary for best performance is not uniformly supported by the presented data. Please correct the claim, or analyze and explain this exception. The current wording overstates the modality-ablation result.
[Sec. 3.3 / Appendix C] Human verification is claimed, but no human accuracy, inter-annotator agreement, or number of QAs rejected or revised is reported; Appendix C contains prompts only. Without a human upper bound, the answerability and quality of the benchmark are unverified. This is particularly important because the dataset was constructed with LLM assistance (Gemini 2.5 Flash, DeepSeek-V3.2, GPT-4o), so model bias may be embedded in the questions. Please add human evaluation statistics and any mitigation against LLM artifacts.
[Sec. 3.1 / Appendix B.5] The audio-coordinated filtering retains only the top 50% of videos by caption-similarity drop. This is a disclosed design choice, but it limits the benchmark to audio-centric events; the finding that omni-modal models outperform video-only models may be partly a consequence of this selection. Please state this limitation explicitly and, if feasible, provide a sensitivity analysis on the excluded videos to estimate how much the omni-vs-video gap depends on the filtering ratio.

minor comments (5)

[Sec. 3.1 vs Appendix B.5] The wording is inconsistent: Sec. 3.1 says 'we only select top-50% videos', while Appendix B.5 says 'we retain the bottom 50% with the lowest scores.' Please clarify the ranking direction.
[Throughout] There are numerous typos and grammatical errors (e.g., 'forcasting', 'evlauating', 'typocally', 'auido-video', 'verfication'). A careful proofreading pass is needed.
[Sec. 4.1] Inference details are underspecified: number of sampled frames, sampling strategy, maximum token length, and decoding settings are not reported. Since models are compared on accuracy, these details matter for reproducibility.
[Sec. 5.2 / Figure 7] The attention visualization is qualitative and lacks error bars or statistical aggregation. The claim of 'consistent elevation' should be supported by quantitative measures.
[Table 4] Qwen2.5-Omni improves by about 10 points on Speech but drops by 7 points on Sound and by 3.4 points on Music; the text highlights only the Speech gain. Please qualify the claim that OFF 'significantly boosts performance in the challenging Speech category' with the full pattern.

Circularity Check

0 steps flagged

No demonstrable circularity: the benchmark results and OFF generalization claims rest on external measurements, and the only self-citation is unrelated related-work context.

full rationale

FutureOmni is a dataset/evaluation paper rather than a derived mathematical result, so there is no equation-level chain in which an output is equivalent to an input by construction. The audio-coordinated video selection (Sec. 3.1) is a disclosed sampling criterion: videos are retained when captions change most under audio removal, but this does not by itself determine any model's accuracy, and the headline finding (best accuracy 64.8%, models struggle on speech-heavy scenarios) is an empirical measurement on the resulting benchmark. The OFF training result is also externally checked: Table 6 evaluates the fine-tuned models on independent benchmarks such as WorldSense, DailyOmni, JointAVBench, OmniVideoBench, Video-MME, and MLVU, so the generalization claim does not reduce to in-sample fitting. A potential overlap between FutureOmni-7K and the 1,034 evaluation QAs is not documented, and if it exists it would be a data-integrity/leakage problem rather than a demonstrated circular step; the paper provides no quoted text establishing that the training set contains evaluation samples. The only self-citation is FutureX (ref. [48]), which shares co-author Xipeng Qiu, but it appears solely in related-work positioning ('FutureBench, ForecastBench, FutureX, and MIRAI predict real-world future events...') and is not load-bearing for any result. No imported uniqueness theorem, no ansatz smuggled through citation, and no renamed empirical pattern were found. The reader's concern about audio-filter representativeness is a validity caveat about benchmark design, not circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central benchmark rests on hand-set dataset construction thresholds and on LLM annotations treated as ground truth. None of these are fitted to the final model scores, but the 50% audio-dependency filter and the 30-second causal window directly shape which videos and questions constitute the benchmark, and therefore shape the conclusions about audio's importance.

free parameters (5)

Inter-frame similarity cutoff = 70%
Hand-set threshold in Sec. 3.1 to remove videos with limited scene changes; affects which videos enter the benchmark.
Audio-dependency retention ratio = top 50%
Sec. 3.1 keeps only the half of videos with largest caption similarity drop when audio is removed; directly shapes the audio-centric nature of the benchmark.
MFCC boundary threshold = 2.0
Sec. 3.2 uses a pre-defined threshold to decide whether a timestamp boundary is acoustically valid; no sensitivity analysis is reported.
Maximum cause-effect time gap = 30 seconds
Sec. 3.3 restricts causal pairs to a 30-second gap between premise and target event; a hand-set definition of what counts as predictable.
Audio causal contribution score threshold = score 2 = causality
Sec. 3.3 asks DeepSeek to score audio causal factor from 0 to 2, where 2 denotes causality; used to classify audio-driven future forecasting pairs.

axioms (5)

domain assumption LLM-generated captions, causal pairs, rationales, and distractors are sufficiently accurate ground truth after human verification.
Introduced in Sec. 3.2-3.3; Appendix C is referenced for human verification, but no inter-annotator agreement or human error rate is reported in the main text.
domain assumption The audio-dependency filtering step identifies videos where audio is causally relevant.
Sec. 3.1/B.5 assumes that a large caption cosine-similarity drop when audio is removed is a valid proxy for audio's causal role in the video.
domain assumption MFCC discontinuities with a fixed threshold of 2.0 mark true event boundaries.
Sec. 3.2 Time Boundary Checking uses this assumption for validating timestamps, without calibration or error analysis.
domain assumption DeepSeek-V3.2 can reliably extract causal relations from chronological event descriptions within a 30-second window.
Sec. 3.3 Causal Pair Discovery relies on an LLM to decide which adjacent events are logical consequences; this is a model-dependent annotation step.
domain assumption Official model implementations and LoRA fine-tuning settings reproduce the intended behavior of evaluated models.
Sec. 4.1 and B.1 assume official inference code and standard hyperparameters faithfully represent each model's capability.

pith-pipeline@v1.3.0-alltime-deepseek · 22740 in / 11117 out tokens · 112804 ms · 2026-08-03T09:23:37.494366+00:00 · methodology

0 comments

read the original abstract

Although Multimodal Large Language Models (MLLMs) demonstrate strong omni-modal perception, their ability to forecast future events from audio-visual cues remains largely unexplored, as existing benchmarks focus mainly on retrospective understanding. To bridge this gap, we introduce FutureOmni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments. The evaluated models are required to perform cross-modal causal and temporal reasoning, as well as effectively leverage internal knowledge to predict future events. FutureOmni is constructed via a scalable LLM-assisted, human-in-the-loop pipeline and contains 919 videos and 1,034 multiple-choice QA pairs across 8 primary domains. Evaluations on 13 omni-modal and 7 video-only models show that current systems struggle with audio-visual future prediction, particularly in speech-heavy scenarios, with the best accuracy of 64.8% achieved by Gemini 3 Flash. To mitigate this limitation, we curate a 7K-sample instruction-tuning dataset and propose an Omni-Modal Future Forecasting (OFF) training strategy. Evaluations on FutureOmni and popular audio-visual and video-only benchmarks demonstrate that OFF enhances future forecasting and generalization. We publicly release all code (https://github.com/OpenMOSS/FutureOmni) and datasets (https://huggingface.co/datasets/OpenMOSS-Team/FutureOmni).

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videos
cs.CV 2026-05 unverdicted novelty 8.0

TraceAV-Bench is the first benchmark for multi-hop trajectory reasoning over long audio-visual videos, showing top models reach only 51-68% accuracy with substantial room for improvement.
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
cs.CL 2026-04 unverdicted novelty 6.0

MiniCPM-o 4.5 uses the Omni-Flow streaming framework to deliver real-time full-duplex omni-modal interaction with proactive behavior in a 9B model that approaches Gemini 2.5 Flash performance.

Reference graph

Works this paper leans on

71 extracted references · 20 linked inside Pith · cited by 2 Pith papers

[1]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkow...

2022
[2]

Claude haiku 4.5.https://www.anthropic.com/news/claude-haiku-4-5, 2025

Anthropic. Claude haiku 4.5.https://www.anthropic.com/news/claude-haiku-4-5, 2025. Accessed: 2025-10-16

2025
[3]

Qwen3-vl technical report, 2025

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

Pith/arXiv arXiv 2025
[4]

Qwen2.5-vl technical report, 2025

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shĳie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 20...

Pith/arXiv arXiv 2025
[5]

Jointavbench: A benchmark for joint audio-visual reasoning evaluation, 2025

Jianghan Chao, Jianzhang Gao, Wenhui Tan, Yuchong Sun, Ruihua Song, and Liyun Ru. Jointavbench: A benchmark for joint audio-visual reasoning evaluation, 2025. URLhttps://arxiv.org/abs/2512.12772

Pith/arXiv arXiv 2025
[6]

Avocado: An audiovisual video captioner driven by temporal orchestration, 2025

Xinlong Chen, Yue Ding, Weihong Lin, Jingyun Hua, Linli Yao, Yang Shi, Bozhou Li, Yuanxing Zhang, Qiang Liu, Pengfei Wan, Liang Wang, and Tieniu Tan. Avocado: An audiovisual video captioner driven by temporal orchestration, 2025. URLhttps://arxiv.org/abs/2510.10395

arXiv 2025
[7]

Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms, 2024

Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms, 2024. URLhttps://arxiv.org/abs/2406.07476

Pith/arXiv arXiv 2024
[8]

Avtrustbench: Assessing and enhancing reliability and robustness in audio-visual llms.arXiv preprint arXiv:2501.02135, 2025

SanjoyChowdhury,SayanNag,SubhrajyotiDasgupta,YaotingWang,MohamedElhoseiny,RuohanGao,andDinesh Manocha. Avtrustbench: Assessing and enhancing reliability and robustness in audio-visual llms.arXiv preprint arXiv:2501.02135, 2025

Pith/arXiv arXiv 2025
[9]

Qwen2-audio technical report, 2024

Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024. URLhttps://arxiv.org/abs/ 2407.10759

Pith/arXiv arXiv 2024
[10]

Deepseek-v3.2: Pushing the frontier of open large language models, 2025

DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, and Bingxuan Wang. Deepseek-v3.2: Pushing the frontier of open large language models, 2025. URLhttps://arxiv.org/abs/2512.02556

Pith/arXiv arXiv 2025
[11]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conferenceon Learning Representations, 2021

2021
[12]

Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InP...

2025
[13]

Longvale: Vision-audio- language-event benchmark towards time-aware omni-modal perception of long videos

Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio- language-event benchmark towards time-aware omni-modal perception of long videos. InProceedings of the IEEE/CVF ConferenceonComputerVisionand PatternRecognition(CVPR), June 2025. 11

2025
[14]

Gemini 3 flash: frontier intelligence built for speed

Google. Gemini 3 flash: frontier intelligence built for speed. https://blog.google/products/gemini/ gemini-3-flash/, 2025. Accessed: 2025-12-17

2025
[15]

Gemini2.5: Pushingthefrontierwithadvancedreasoning,multimodality,longcontext,andnextgeneration agentic capabilities, 2025

Google. Gemini2.5: Pushingthefrontierwithadvancedreasoning,multimodality,longcontext,andnextgeneration agentic capabilities, 2025. URLhttps://arxiv.org/abs/2507.06261

Pith/arXiv arXiv 2025
[16]

Worldsense: Evaluating real-world omnimodal understanding for multimodal llms, 2025

Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms, 2025. URLhttps://arxiv.org/abs/2502.04326

Pith/arXiv arXiv 2025
[17]

LoRA: Low-rank adaptation of large language models

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conferenceon Learning Representations,
[18]

WavLLM: Towards robust and adaptive speech large language model

Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei. WavLLM: Towards robust and adaptive speech large language model. InFindingsofthe AssociationforComputationalLinguistics: EMNLP 2024, 2024

2024
[19]

Forecastbench: A dynamic benchmark of AI forecasting capabilities

Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tetlock. Forecastbench: A dynamic benchmark of AI forecasting capabilities. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https:// openreview.net/forum?id=lfPkGWXLLf

2025
[20]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URLhttps://arxiv.org/abs/2309.06180

Pith/arXiv arXiv 2023
[21]

What is more likely to happen next? video-and-language future event prediction

Jie Lei, Licheng Yu, Tamara Berg, and Mohit Bansal. What is more likely to happen next? video-and-language future event prediction. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020

2020
[22]

Omnivideobench: Towards audio-visual understanding evaluation for omni mllms, 2025

Caorui Li, Yu Chen, Yiyan Ji, Jin Xu, Zhenyu Cui, Shihao Li, Yuanxing Zhang, Jiafu Tang, Zhenghao Song, Dingling Zhang, Ying He, Haoxiang Liu, Yuxuan Wang, Qiufeng Wang, Zhenhe Wu, Jiehui Luo, Zhiyu Pan, Weihao Xie, Chenchen Zhang, Zhaohui Wang, Jiayi Tian, Yanghai Wang, Zhe Cao, Minxin Dai, Ke Wang, Runzhe Wen, Yinghao Ma, Yaning Pan, Sungkyun Chang, Ter...

arXiv 2025
[23]

Learning to answer questions in dynamic audio-visual scenarios

Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. InIEEE Conferenceon ComputerVisionand PatternRecognition(CVPR), 2022

2022
[24]

Haoxuan Li, Zhengmao Yang, Yunshan Ma, Yi Bin, Yang Yang, and Tat-Seng Chua. Mm-forecast: A multimodal approachtotemporaleventforecastingwithlargelanguagemodels.InJianfeiCai,MohanS.Kankanhalli,Balakrishnan Prabhakaran, Susanne Boll, Ramanathan Subramanian, Liang Zheng, Vivek K. Singh, Pablo César, Lexing Xie, and Dong Xu, editors,Proceedings of the 32nd A...

arXiv 2024
[25]

Intentqa: Context-aware video intent reasoning

Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. Intentqa: Context-aware video intent reasoning. In2023 IEEE/CVF International Conferenceon ComputerVision(ICCV), 2023

2023
[26]

BLIP-2: Bootstrapping language-image pre-training with frozenimageencodersandlargelanguagemodels

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozenimageencodersandlargelanguagemodels. In Proceedingsofthe40thInternationalConferenceonMachine Learning, 2023

2023
[27]

Llama-vid: An image is worth 2 tokens in large language models

Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ComputerVision–ECCV2024: 18thEuropeanConference,Milan,Italy,September29–October4,2024,Proceedings, PartXLVI, 2024

2024
[28]

Video-llava: Learning united visual representation by alignment before projection

Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conferenceon EmpiricalMethods in NaturalLanguageProcessing, EMNLP 2024, 12 Miami, FL, USA,November12-16, 2024, page...

doi:10.18653/v1/2024.emnlp-main.342 2024
[29]

Visual instruction tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvancesin Neural InformationProcessingSystems, 2023

2023
[30]

Ola: Pushing the frontiers of omni-modal language model, 2025

Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model, 2025. URLhttps://arxiv.org/abs/2502.04328

Pith/arXiv arXiv 2025
[31]

See, hear, and understand: Benchmarking audiovisual human speech understanding in multimodal large language models, 2025

Le Thien Phuc Nguyen, Zhuoran Yu, Samuel Low Yu Hang, Subin An, Jeongik Lee, Yohan Ban, SeungEun Chung, Thanh-Huy Nguyen, JuWan Maeng, Soochahn Lee, and Yong Jae Lee. See, hear, and understand: Benchmarking audiovisual human speech understanding in multimodal large language models, 2025. URL https://arxiv.org/abs/2512.02231

Pith/arXiv arXiv 2025
[32]

Gpt-4o system card, 2024

OpenAI. Gpt-4o system card, 2024. URLhttps://arxiv.org/abs/2410.21276

Pith/arXiv arXiv 2024
[33]

Learning transferable visual models from natural language supervision

AlecRadford,JongWookKim,ChrisHallacy,AdityaRamesh,GabrielGoh,SandhiniAgarwal,GirishSastry,Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings ofthe 38th International Conferenceon MachineLearning, 2021

2021
[34]

Robust speech recognition via large-scale weak supervision

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InProceedings of the 40th International Conference on Machine Learning, pages 28492–28518, 2023

2023
[35]

Timechat: A time-sensitive multimodal large language model for long video understanding

Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In2024 IEEE/CVF Conferenceon ComputerVisionand PatternRecognition (CVPR), 2024

2024
[36]

SALMONN: Towards generic hearing abilities for large language models

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. InThe TwelfthInternational Conference on Learning Representations, 2024

2024
[37]

video-salmonn 2: Caption-enhanced audio-visual large language models, 2025

Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. video-salmonn 2: Caption-enhanced audio-visual large language models, 2025. URLhttps://arxiv.org/abs/ 2506.15220

arXiv 2025
[38]

Empoweringllmswithpseudo- untrimmed videos for audio-visual temporal understanding

YunlongTang,DaikiShimada,JingBi,MingqianFeng,HangHua,andChenliangXu. Empoweringllmswithpseudo- untrimmed videos for audio-visual temporal understanding. InProceedingsofthe Thirty-NinthAAAI Conference on Artificial Intelligence and Thirty-SeventhConference on InnovativeApplications of Artificial Intelligence and Fifteenth Symposiumon EducationalAdvancesinA...

2025
[39]

Fostering video reasoning via next-event prediction, 2025

Haonan Wang, Hongfu Liu, Xiangyan Liu, Chao Du, Kenji Kawaguchi, Ye Wang, and Tianyu Pang. Fostering video reasoning via next-event prediction, 2025. URLhttps://arxiv.org/abs/2505.22457

Pith/arXiv arXiv 2025
[40]

Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers, 2021

Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers, 2021. URLhttps://arxiv.org/abs/2012.15828

Pith/arXiv arXiv 2021
[41]

Ugc-videocaptioner: An omni ugc video detail caption model and new benchmarks, 2025

Peiran Wu, Yunze Liu, Zhengdong Zhu, Enmin Zhou, and Junxiao Shen. Ugc-videocaptioner: An omni ugc video detail caption model and new benchmarks, 2025. URLhttps://arxiv.org/abs/2507.11336

arXiv 2025
[42]

Qwen2.5-omni technical report, 2025

Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025. URL https://arxiv.org/abs/2503.20215

Pith/arXiv arXiv 2025
[43]

Qwen3-omni technical report, 2025

Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo ...

Pith/arXiv arXiv 2025
[44]

Avqa: A dataset for audio-visual question answering on videos

Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. InProceedings of the 30th ACM International Conference on Multimedia, 2022. 13

2022
[45]

Audio-centric video understanding benchmark without text shortcut

Yudong Yang, Jimin Zhuang, Guangzhi Sun, Changli Tang, Yixuan Li, Peihan Li, Yifan Jiang, Wei Li, Zejun Ma, and Chao Zhang. Audio-centric video understanding benchmark without text shortcut. InProceedingsofthe 2025 Conferenceon EmpiricalMethods inNaturalLanguageProcessing, 2025

2025
[46]

Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprintarXiv:2408.01800, 2024

YuanYao, TianyuYu, AoZhang, ChongyiWang, JunboCui, HongjiZhu, TianchiCai, HaoyuLi, WeilinZhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprintarXiv:2408.01800, 2024

Pith/arXiv arXiv 2024
[47]

MIRAI: evaluating LLM agents for event forecasting.CoRR, abs/2407.01231, 2024

Chenchen Ye, Ziniu Hu, Yihe Deng, Zĳie Huang, Mingyu Derek Ma, Yanqiao Zhu, and Wei Wang. MIRAI: evaluating LLM agents for event forecasting.CoRR, abs/2407.01231, 2024. doi: 10.48550/ARXIV.2407.01231. URL https://doi.org/10.48550/arXiv.2407.01231

Pith/arXiv arXiv doi:10.48550/arxiv.2407.01231 2024
[48]

Futurex: An advanced live benchmark for LLM agents in future prediction

Zhiyuan Zeng, Jiashuo Liu, Siyuan Chen, Tianci He, Yali Liao, Jinpeng Wang, Zaiyuan Wang, Yang Yang, Lingyue Yin, Mingren Yin, Zhenwei Zhu, Tianle Cai, Zehui Chen, Jiecao Chen, Yantao Du, Xiang Gao, Jiacheng Guo, Liang Hu, Jianpeng Jiao, Xiangsheng Li, Jingkai Liu, Shuang Ni, Zhoufutu Wen, Ge Zhang, Kaiyuan Zhang, Xin Zhou, Jose Blanchet, Xipeng Qiu, Meng...

Pith/arXiv arXiv doi:10.48550/arxiv.2508.11987 2025
[49]

Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025

Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025. URLhttps://arxiv.org/abs/2501. 13106

2025
[50]

Video-LLaMA: An instruction-tuned audio-visual language model for video understanding

Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: SystemDemonstrations, 2023

2023
[51]

Timelens: Rethinking video temporal grounding with multimodal llms, 2025

Jun Zhang, Teng Wang, Yuying Ge, Yixiao Ge, Xinhao Li, Ying Shan, and Limin Wang. Timelens: Rethinking video temporal grounding with multimodal llms, 2025. URLhttps://arxiv.org/abs/2512.14698

arXiv 2025
[52]

Llava-next: A strong zero-shot video understanding model, April 2024

Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024. URLhttps://llava-vl.github.io/blog/ 2024-04-30-llava-next-video/

2024
[53]

Llamafactory: Unified efficient fine-tuning of 100+ language models, 2024

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models, 2024. URLhttps://arxiv.org/abs/2403. 13372

2024
[54]

Mlvu: Benchmarkingmulti-tasklongvideounderstanding

Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, BoZhang,TiejunHuang,andZhengLiu. Mlvu: Benchmarkingmulti-tasklongvideounderstanding. In Proceedings ofthe IEEE/CVFConferenceon ComputerVisionandPatternRecognition(CVPR), 2025

2025
[56]

audio event

Ziwei Zhou, Rui Wang, and Zuxuan Wu. Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities, 2025. URLhttps://arxiv.org/abs/2505.17862. 14 Appendix Appendix Contents A Audio Type and Duration Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B Experiment Details . . . . . . . . . . . . . ....

arXiv 2025
[58]

John walks down the street

Identify a Causal Pair (The Premise & The Effect):Scan the text to find two events (Event A and Event B) where Event A clearly leads to or causes Event B. Constraint 1 (Temporal Proximity):The time gap between the END of Event A (Cause) and the START of Event B (Effect) must be less than 30 seconds. Ideally, it should be immediate (< 10s). Constraint2(Cau...
[59]

Given the premise event: ’[Description of Event A]’, which event is its most direct conclusion?

Draft the Question: Format:"Given the premise event: ’[Description of Event A]’, which event is its most direct conclusion?"
[60]

decisive

Draft the Options (1 Correct, 4 Distractors): Correct Answer:A precise description of Event B as it appears in the caption. Constraint 3 (Distractor Logic):Distractors must be "decisive" (plausible within the scene’s context) but logically inferior to the correct answer given the premise. Do not create a distractor that is a generic "common sense" outcome...
[61]

motor jam

Output Generation: Return the result in the strict JSON format provided below. Input Caption: 20 INSERT_CAPTION_HERE Example Input Caption: 00:00 - 00:05: A shot of a building at night, with one light on in a window.Modality: Video. 00:05 - 00:07: A hand presses the down arrow button on an elevator panel, and the button lights up.Modality: Video, Audio (b...
[62]

Understand the sequence of actions, the characters involved, and the overall flow of the narrative

Comprehend the Timeline: First, carefully read the entire ### TIMELINE OF EVENTS ###. Understand the sequence of actions, the characters involved, and the overall flow of the narrative. Pay attention to both visual and auditory cues.Generate five distractor effect events based on the provided cause event. You must use the specific heuristics below to guid...
[63]

A glass shatters

Foreachdistractoryoucreate,youmustexplicitlyselectandapplyoneofthefollowingstrategies: 1).Reverse-Causal:Proposeaneventthatcouldhave*caused*theobservedcauseevent,effectively reversing the true temporal-causal order. (e.g., Cause: "A glass shatters." -> Distractor: "A ball hits the glass."). 2).Delayed or Premature:Propose an event that is part of the same...
[64]

Premise: The description of the video/audio context before the prediction
[65]

Question: The question asked to the model
[66]

Correct Answer: The ground truth future event
[67]

Model Prediction: The incorrect answer generated by the model
[68]

# Classification Criteria: You must classify the error into exactly one of the following four categories

Key Modality: The primary modality (Visual, Audio, or Both) required to solve this specific question. # Classification Criteria: You must classify the error into exactly one of the following four categories. Use the hierarchy below to decide:
[69]

## Indicator: The visual context alone is insufficient or misleading, and the model failed because it missed the acoustic cue (e.g., a doorbell ringing off-screen)

Audio Perception Error: ## Definition: The correct prediction relies heavily on a specific sound (speech, event, or music) that the model clearly ignored or hallucinated. ## Indicator: The visual context alone is insufficient or misleading, and the model failed because it missed the acoustic cue (e.g., a doorbell ringing off-screen)
[70]

## Indicator: The model’s prediction contradicts clear visual evidence (e.g., predicting "driving" when the car is visually parked)

Video Perception Error: ## Definition: The correct prediction relies on a visual detail (object, text, or action) that the model failed to recognize. ## Indicator: The model’s prediction contradicts clear visual evidence (e.g., predicting "driving" when the car is visually parked)
[71]

## Indicator: Understanding the scene requires prior knowledge (e.g., knowing that mixing specific chemicals causes an explosion, or knowing the rules of Chess)

Lack of Knowledge: ## Definition: The model likely perceived the sensory data correctly but lacked the external world knowledge, physics, or domain expertise required to predict the outcome. ## Indicator: Understanding the scene requires prior knowledge (e.g., knowing that mixing specific chemicals causes an explosion, or knowing the rules of Chess)
[72]

he is exercising

Audio-Video Joint Reasoning Failure: ## Definition: The model correctly perceives both the visual and audio elements individually but fails to combine them logically to derive the causal future. ## Indicator: The error is not due to missing a sound or object, but failing to link them (e.g., seeing a man run + hearing a siren -> predicting "he is exercisin...
[2022]

URLhttps://openreview.net/forum?id=nZeVKeeFYf9

[1] [1]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkow...

2022

[2] [2]

Claude haiku 4.5.https://www.anthropic.com/news/claude-haiku-4-5, 2025

Anthropic. Claude haiku 4.5.https://www.anthropic.com/news/claude-haiku-4-5, 2025. Accessed: 2025-10-16

2025

[3] [3]

Qwen3-vl technical report, 2025

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

Pith/arXiv arXiv 2025

[4] [4]

Qwen2.5-vl technical report, 2025

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shĳie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 20...

Pith/arXiv arXiv 2025

[5] [5]

Jointavbench: A benchmark for joint audio-visual reasoning evaluation, 2025

Jianghan Chao, Jianzhang Gao, Wenhui Tan, Yuchong Sun, Ruihua Song, and Liyun Ru. Jointavbench: A benchmark for joint audio-visual reasoning evaluation, 2025. URLhttps://arxiv.org/abs/2512.12772

Pith/arXiv arXiv 2025

[6] [6]

Avocado: An audiovisual video captioner driven by temporal orchestration, 2025

Xinlong Chen, Yue Ding, Weihong Lin, Jingyun Hua, Linli Yao, Yang Shi, Bozhou Li, Yuanxing Zhang, Qiang Liu, Pengfei Wan, Liang Wang, and Tieniu Tan. Avocado: An audiovisual video captioner driven by temporal orchestration, 2025. URLhttps://arxiv.org/abs/2510.10395

arXiv 2025

[7] [7]

Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms, 2024

Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms, 2024. URLhttps://arxiv.org/abs/2406.07476

Pith/arXiv arXiv 2024

[8] [8]

Avtrustbench: Assessing and enhancing reliability and robustness in audio-visual llms.arXiv preprint arXiv:2501.02135, 2025

SanjoyChowdhury,SayanNag,SubhrajyotiDasgupta,YaotingWang,MohamedElhoseiny,RuohanGao,andDinesh Manocha. Avtrustbench: Assessing and enhancing reliability and robustness in audio-visual llms.arXiv preprint arXiv:2501.02135, 2025

Pith/arXiv arXiv 2025

[9] [9]

Qwen2-audio technical report, 2024

Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024. URLhttps://arxiv.org/abs/ 2407.10759

Pith/arXiv arXiv 2024

[10] [10]

Deepseek-v3.2: Pushing the frontier of open large language models, 2025

DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, and Bingxuan Wang. Deepseek-v3.2: Pushing the frontier of open large language models, 2025. URLhttps://arxiv.org/abs/2512.02556

Pith/arXiv arXiv 2025

[11] [11]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conferenceon Learning Representations, 2021

2021

[12] [12]

Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InP...

2025

[13] [13]

Longvale: Vision-audio- language-event benchmark towards time-aware omni-modal perception of long videos

Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio- language-event benchmark towards time-aware omni-modal perception of long videos. InProceedings of the IEEE/CVF ConferenceonComputerVisionand PatternRecognition(CVPR), June 2025. 11

2025

[14] [14]

Gemini 3 flash: frontier intelligence built for speed

Google. Gemini 3 flash: frontier intelligence built for speed. https://blog.google/products/gemini/ gemini-3-flash/, 2025. Accessed: 2025-12-17

2025

[15] [15]

Gemini2.5: Pushingthefrontierwithadvancedreasoning,multimodality,longcontext,andnextgeneration agentic capabilities, 2025

Google. Gemini2.5: Pushingthefrontierwithadvancedreasoning,multimodality,longcontext,andnextgeneration agentic capabilities, 2025. URLhttps://arxiv.org/abs/2507.06261

Pith/arXiv arXiv 2025

[16] [16]

Worldsense: Evaluating real-world omnimodal understanding for multimodal llms, 2025

Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms, 2025. URLhttps://arxiv.org/abs/2502.04326

Pith/arXiv arXiv 2025

[17] [17]

LoRA: Low-rank adaptation of large language models

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conferenceon Learning Representations,

[18] [18]

WavLLM: Towards robust and adaptive speech large language model

Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei. WavLLM: Towards robust and adaptive speech large language model. InFindingsofthe AssociationforComputationalLinguistics: EMNLP 2024, 2024

2024

[19] [19]

Forecastbench: A dynamic benchmark of AI forecasting capabilities

Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tetlock. Forecastbench: A dynamic benchmark of AI forecasting capabilities. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https:// openreview.net/forum?id=lfPkGWXLLf

2025

[20] [20]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URLhttps://arxiv.org/abs/2309.06180

Pith/arXiv arXiv 2023

[21] [21]

What is more likely to happen next? video-and-language future event prediction

Jie Lei, Licheng Yu, Tamara Berg, and Mohit Bansal. What is more likely to happen next? video-and-language future event prediction. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020

2020

[22] [22]

Omnivideobench: Towards audio-visual understanding evaluation for omni mllms, 2025

Caorui Li, Yu Chen, Yiyan Ji, Jin Xu, Zhenyu Cui, Shihao Li, Yuanxing Zhang, Jiafu Tang, Zhenghao Song, Dingling Zhang, Ying He, Haoxiang Liu, Yuxuan Wang, Qiufeng Wang, Zhenhe Wu, Jiehui Luo, Zhiyu Pan, Weihao Xie, Chenchen Zhang, Zhaohui Wang, Jiayi Tian, Yanghai Wang, Zhe Cao, Minxin Dai, Ke Wang, Runzhe Wen, Yinghao Ma, Yaning Pan, Sungkyun Chang, Ter...

arXiv 2025

[23] [23]

Learning to answer questions in dynamic audio-visual scenarios

Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. InIEEE Conferenceon ComputerVisionand PatternRecognition(CVPR), 2022

2022

[24] [24]

Haoxuan Li, Zhengmao Yang, Yunshan Ma, Yi Bin, Yang Yang, and Tat-Seng Chua. Mm-forecast: A multimodal approachtotemporaleventforecastingwithlargelanguagemodels.InJianfeiCai,MohanS.Kankanhalli,Balakrishnan Prabhakaran, Susanne Boll, Ramanathan Subramanian, Liang Zheng, Vivek K. Singh, Pablo César, Lexing Xie, and Dong Xu, editors,Proceedings of the 32nd A...

arXiv 2024

[25] [25]

Intentqa: Context-aware video intent reasoning

Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. Intentqa: Context-aware video intent reasoning. In2023 IEEE/CVF International Conferenceon ComputerVision(ICCV), 2023

2023

[26] [26]

BLIP-2: Bootstrapping language-image pre-training with frozenimageencodersandlargelanguagemodels

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozenimageencodersandlargelanguagemodels. In Proceedingsofthe40thInternationalConferenceonMachine Learning, 2023

2023

[27] [27]

Llama-vid: An image is worth 2 tokens in large language models

Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ComputerVision–ECCV2024: 18thEuropeanConference,Milan,Italy,September29–October4,2024,Proceedings, PartXLVI, 2024

2024

[28] [28]

Video-llava: Learning united visual representation by alignment before projection

Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conferenceon EmpiricalMethods in NaturalLanguageProcessing, EMNLP 2024, 12 Miami, FL, USA,November12-16, 2024, page...

doi:10.18653/v1/2024.emnlp-main.342 2024

[29] [29]

Visual instruction tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvancesin Neural InformationProcessingSystems, 2023

2023

[30] [30]

Ola: Pushing the frontiers of omni-modal language model, 2025

Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model, 2025. URLhttps://arxiv.org/abs/2502.04328

Pith/arXiv arXiv 2025

[31] [31]

See, hear, and understand: Benchmarking audiovisual human speech understanding in multimodal large language models, 2025

Le Thien Phuc Nguyen, Zhuoran Yu, Samuel Low Yu Hang, Subin An, Jeongik Lee, Yohan Ban, SeungEun Chung, Thanh-Huy Nguyen, JuWan Maeng, Soochahn Lee, and Yong Jae Lee. See, hear, and understand: Benchmarking audiovisual human speech understanding in multimodal large language models, 2025. URL https://arxiv.org/abs/2512.02231

Pith/arXiv arXiv 2025

[32] [32]

Gpt-4o system card, 2024

OpenAI. Gpt-4o system card, 2024. URLhttps://arxiv.org/abs/2410.21276

Pith/arXiv arXiv 2024

[33] [33]

Learning transferable visual models from natural language supervision

AlecRadford,JongWookKim,ChrisHallacy,AdityaRamesh,GabrielGoh,SandhiniAgarwal,GirishSastry,Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings ofthe 38th International Conferenceon MachineLearning, 2021

2021

[34] [34]

Robust speech recognition via large-scale weak supervision

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InProceedings of the 40th International Conference on Machine Learning, pages 28492–28518, 2023

2023

[35] [35]

Timechat: A time-sensitive multimodal large language model for long video understanding

Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In2024 IEEE/CVF Conferenceon ComputerVisionand PatternRecognition (CVPR), 2024

2024

[36] [36]

SALMONN: Towards generic hearing abilities for large language models

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. InThe TwelfthInternational Conference on Learning Representations, 2024

2024

[37] [37]

video-salmonn 2: Caption-enhanced audio-visual large language models, 2025

Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. video-salmonn 2: Caption-enhanced audio-visual large language models, 2025. URLhttps://arxiv.org/abs/ 2506.15220

arXiv 2025

[38] [38]

Empoweringllmswithpseudo- untrimmed videos for audio-visual temporal understanding

YunlongTang,DaikiShimada,JingBi,MingqianFeng,HangHua,andChenliangXu. Empoweringllmswithpseudo- untrimmed videos for audio-visual temporal understanding. InProceedingsofthe Thirty-NinthAAAI Conference on Artificial Intelligence and Thirty-SeventhConference on InnovativeApplications of Artificial Intelligence and Fifteenth Symposiumon EducationalAdvancesinA...

2025

[39] [39]

Fostering video reasoning via next-event prediction, 2025

Haonan Wang, Hongfu Liu, Xiangyan Liu, Chao Du, Kenji Kawaguchi, Ye Wang, and Tianyu Pang. Fostering video reasoning via next-event prediction, 2025. URLhttps://arxiv.org/abs/2505.22457

Pith/arXiv arXiv 2025

[40] [40]

Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers, 2021

Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers, 2021. URLhttps://arxiv.org/abs/2012.15828

Pith/arXiv arXiv 2021

[41] [41]

Ugc-videocaptioner: An omni ugc video detail caption model and new benchmarks, 2025

Peiran Wu, Yunze Liu, Zhengdong Zhu, Enmin Zhou, and Junxiao Shen. Ugc-videocaptioner: An omni ugc video detail caption model and new benchmarks, 2025. URLhttps://arxiv.org/abs/2507.11336

arXiv 2025

[42] [42]

Qwen2.5-omni technical report, 2025

Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025. URL https://arxiv.org/abs/2503.20215

Pith/arXiv arXiv 2025

[43] [43]

Qwen3-omni technical report, 2025

Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo ...

Pith/arXiv arXiv 2025

[44] [44]

Avqa: A dataset for audio-visual question answering on videos

Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. InProceedings of the 30th ACM International Conference on Multimedia, 2022. 13

2022

[45] [45]

Audio-centric video understanding benchmark without text shortcut

Yudong Yang, Jimin Zhuang, Guangzhi Sun, Changli Tang, Yixuan Li, Peihan Li, Yifan Jiang, Wei Li, Zejun Ma, and Chao Zhang. Audio-centric video understanding benchmark without text shortcut. InProceedingsofthe 2025 Conferenceon EmpiricalMethods inNaturalLanguageProcessing, 2025

2025

[46] [46]

Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprintarXiv:2408.01800, 2024

YuanYao, TianyuYu, AoZhang, ChongyiWang, JunboCui, HongjiZhu, TianchiCai, HaoyuLi, WeilinZhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprintarXiv:2408.01800, 2024

Pith/arXiv arXiv 2024

[47] [47]

MIRAI: evaluating LLM agents for event forecasting.CoRR, abs/2407.01231, 2024

Chenchen Ye, Ziniu Hu, Yihe Deng, Zĳie Huang, Mingyu Derek Ma, Yanqiao Zhu, and Wei Wang. MIRAI: evaluating LLM agents for event forecasting.CoRR, abs/2407.01231, 2024. doi: 10.48550/ARXIV.2407.01231. URL https://doi.org/10.48550/arXiv.2407.01231

Pith/arXiv arXiv doi:10.48550/arxiv.2407.01231 2024

[48] [48]

Futurex: An advanced live benchmark for LLM agents in future prediction

Zhiyuan Zeng, Jiashuo Liu, Siyuan Chen, Tianci He, Yali Liao, Jinpeng Wang, Zaiyuan Wang, Yang Yang, Lingyue Yin, Mingren Yin, Zhenwei Zhu, Tianle Cai, Zehui Chen, Jiecao Chen, Yantao Du, Xiang Gao, Jiacheng Guo, Liang Hu, Jianpeng Jiao, Xiangsheng Li, Jingkai Liu, Shuang Ni, Zhoufutu Wen, Ge Zhang, Kaiyuan Zhang, Xin Zhou, Jose Blanchet, Xipeng Qiu, Meng...

Pith/arXiv arXiv doi:10.48550/arxiv.2508.11987 2025

[49] [49]

Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025

Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025. URLhttps://arxiv.org/abs/2501. 13106

2025

[50] [50]

Video-LLaMA: An instruction-tuned audio-visual language model for video understanding

Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: SystemDemonstrations, 2023

2023

[51] [51]

Timelens: Rethinking video temporal grounding with multimodal llms, 2025

Jun Zhang, Teng Wang, Yuying Ge, Yixiao Ge, Xinhao Li, Ying Shan, and Limin Wang. Timelens: Rethinking video temporal grounding with multimodal llms, 2025. URLhttps://arxiv.org/abs/2512.14698

arXiv 2025

[52] [52]

Llava-next: A strong zero-shot video understanding model, April 2024

Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024. URLhttps://llava-vl.github.io/blog/ 2024-04-30-llava-next-video/

2024

[53] [53]

Llamafactory: Unified efficient fine-tuning of 100+ language models, 2024

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models, 2024. URLhttps://arxiv.org/abs/2403. 13372

2024

[54] [54]

Mlvu: Benchmarkingmulti-tasklongvideounderstanding

Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, BoZhang,TiejunHuang,andZhengLiu. Mlvu: Benchmarkingmulti-tasklongvideounderstanding. In Proceedings ofthe IEEE/CVFConferenceon ComputerVisionandPatternRecognition(CVPR), 2025

2025

[55] [56]

audio event

Ziwei Zhou, Rui Wang, and Zuxuan Wu. Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities, 2025. URLhttps://arxiv.org/abs/2505.17862. 14 Appendix Appendix Contents A Audio Type and Duration Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B Experiment Details . . . . . . . . . . . . . ....

arXiv 2025

[56] [58]

John walks down the street

Identify a Causal Pair (The Premise & The Effect):Scan the text to find two events (Event A and Event B) where Event A clearly leads to or causes Event B. Constraint 1 (Temporal Proximity):The time gap between the END of Event A (Cause) and the START of Event B (Effect) must be less than 30 seconds. Ideally, it should be immediate (< 10s). Constraint2(Cau...

[57] [59]

Given the premise event: ’[Description of Event A]’, which event is its most direct conclusion?

Draft the Question: Format:"Given the premise event: ’[Description of Event A]’, which event is its most direct conclusion?"

[58] [60]

decisive

Draft the Options (1 Correct, 4 Distractors): Correct Answer:A precise description of Event B as it appears in the caption. Constraint 3 (Distractor Logic):Distractors must be "decisive" (plausible within the scene’s context) but logically inferior to the correct answer given the premise. Do not create a distractor that is a generic "common sense" outcome...

[59] [61]

motor jam

Output Generation: Return the result in the strict JSON format provided below. Input Caption: 20 INSERT_CAPTION_HERE Example Input Caption: 00:00 - 00:05: A shot of a building at night, with one light on in a window.Modality: Video. 00:05 - 00:07: A hand presses the down arrow button on an elevator panel, and the button lights up.Modality: Video, Audio (b...

[60] [62]

Understand the sequence of actions, the characters involved, and the overall flow of the narrative

Comprehend the Timeline: First, carefully read the entire ### TIMELINE OF EVENTS ###. Understand the sequence of actions, the characters involved, and the overall flow of the narrative. Pay attention to both visual and auditory cues.Generate five distractor effect events based on the provided cause event. You must use the specific heuristics below to guid...

[61] [63]

A glass shatters

Foreachdistractoryoucreate,youmustexplicitlyselectandapplyoneofthefollowingstrategies: 1).Reverse-Causal:Proposeaneventthatcouldhave*caused*theobservedcauseevent,effectively reversing the true temporal-causal order. (e.g., Cause: "A glass shatters." -> Distractor: "A ball hits the glass."). 2).Delayed or Premature:Propose an event that is part of the same...

[62] [64]

Premise: The description of the video/audio context before the prediction

[63] [65]

Question: The question asked to the model

[64] [66]

Correct Answer: The ground truth future event

[65] [67]

Model Prediction: The incorrect answer generated by the model

[66] [68]

# Classification Criteria: You must classify the error into exactly one of the following four categories

Key Modality: The primary modality (Visual, Audio, or Both) required to solve this specific question. # Classification Criteria: You must classify the error into exactly one of the following four categories. Use the hierarchy below to decide:

[67] [69]

## Indicator: The visual context alone is insufficient or misleading, and the model failed because it missed the acoustic cue (e.g., a doorbell ringing off-screen)

Audio Perception Error: ## Definition: The correct prediction relies heavily on a specific sound (speech, event, or music) that the model clearly ignored or hallucinated. ## Indicator: The visual context alone is insufficient or misleading, and the model failed because it missed the acoustic cue (e.g., a doorbell ringing off-screen)

[68] [70]

## Indicator: The model’s prediction contradicts clear visual evidence (e.g., predicting "driving" when the car is visually parked)

Video Perception Error: ## Definition: The correct prediction relies on a visual detail (object, text, or action) that the model failed to recognize. ## Indicator: The model’s prediction contradicts clear visual evidence (e.g., predicting "driving" when the car is visually parked)

[69] [71]

## Indicator: Understanding the scene requires prior knowledge (e.g., knowing that mixing specific chemicals causes an explosion, or knowing the rules of Chess)

Lack of Knowledge: ## Definition: The model likely perceived the sensory data correctly but lacked the external world knowledge, physics, or domain expertise required to predict the outcome. ## Indicator: Understanding the scene requires prior knowledge (e.g., knowing that mixing specific chemicals causes an explosion, or knowing the rules of Chess)

[70] [72]

he is exercising

Audio-Video Joint Reasoning Failure: ## Definition: The model correctly perceives both the visual and audio elements individually but fails to combine them logically to derive the causal future. ## Indicator: The error is not due to missing a sound or object, but failing to link them (e.g., seeing a man run + hearing a siren -> predicting "he is exercisin...

[71] [2022]

URLhttps://openreview.net/forum?id=nZeVKeeFYf9