REVIEW 6 cited by
Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The hallucination of large multimodal models (LMMs), providing responses that appear correct but are actually incorrect, limits their reliability and applicability. This paper aims to study the hallucination problem of LMMs in video modality, which is dynamic and more challenging compared to static modalities like images and text. From this motivation, we first present a comprehensive benchmark termed HAVEN for evaluating hallucinations of LMMs in video understanding tasks. It is built upon three dimensions, i.e., hallucination causes, hallucination aspects, and question formats, resulting in 6K questions. Then, we quantitatively study 7 influential factors on hallucinations, e.g., duration time of videos, model sizes, and model reasoning, via experiments of 16 LMMs on the presented benchmark. In addition, inspired by recent thinking models like OpenAI o1, we propose a video-thinking model to mitigate the hallucinations of LMMs via supervised reasoning fine-tuning (SRFT) and direct preference optimization (TDPO)-- where SRFT enhances reasoning capabilities while TDPO reduces hallucinations in the thinking process. Extensive experiments and analyses demonstrate the effectiveness. Remarkably, it improves the baseline by 7.65% in accuracy on hallucination evaluation and reduces the bias score by 4.5%. The code and data are public at https://github.com/Hongcheng-Gao/HAVEN.
Forward citations
Cited by 6 Pith papers
-
VADER: Adaptive Debiasing for Hallucination Mitigation in Video Large Language Models
A training-free decoding framework that adaptively reweights attention toward video tokens and erases key visual evidence per frame to suppress hallucinated predictions, achieving 72.60% accuracy on EventHallusion wit...
-
Enhancing Video Representations with Spatiotemporal-Semantic Residual to Mitigate Hallucinations in Video Large Multimodal Models
A learned residual disruptor creates hallucination-prone negative video features, and subtracting their logits during decoding reduces hallucination in two 7B video LLMs.
-
Position: Reasoning After Perception Means Reasoning Without Vision
Reasoning in text space cannot recover visual information that was collapsed during perception, so multimodal models need architectures that reason within the visual representation.
-
Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark
Video-MMLU: a 1,065-video lecture benchmark where most AI video models score 10-50%, but text-only models answer 40% of quiz questions without video.
-
Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models
A survey organizes multimodal reasoning research into a staged roadmap and proposes native large multimodal reasoning models that unify perception, generation, and agentic planning.
-
FlowReasoner: Reinforcing Query-Level Meta-Agents
FlowReasoner, a reasoning meta-agent trained by R1 distillation and reinforcement learning, generates a query-specific multi-agent workflow per query and improves code benchmark accuracy over fixed workflows and direc...
Discussion (0). Continue with ORCID to comment.