REVIEW 8 cited by
video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Speech understanding as an element of the more generic video understanding using audio-visual large language models (av-LLMs) is a crucial yet understudied aspect. This paper proposes video-SALMONN, a single end-to-end av-LLM for video processing, which can understand not only visual frame sequences, audio events and music, but speech as well. To obtain fine-grained temporal information required by speech understanding, while keeping efficient for other video elements, this paper proposes a novel multi-resolution causal Q-Former (MRC Q-Former) structure to connect pre-trained audio-visual encoders and the backbone large language model. Moreover, dedicated training approaches including the diversity loss and the unpaired audio-visual mixed training scheme are proposed to avoid frames or modality dominance. On the introduced speech-audio-visual evaluation benchmark, video-SALMONN achieves more than 25\% absolute accuracy improvements on the video-QA task and over 30\% absolute accuracy improvements on audio-visual QA tasks with human speech. In addition, video-SALMONN demonstrates remarkable video comprehension and reasoning abilities on tasks that are unprecedented by other av-LLMs. Our training code and model checkpoints are available at \texttt{\url{https://github.com/bytedance/SALMONN/}}.
Forward citations
Cited by 8 Pith papers
-
Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion
PPAD injects MLLM semantic feedback into diffusion denoising via lookahead sketches and ping-pong-ahead resampling, improving text-to-image alignment.
-
OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models
A training-free, query-anchored, modality-decoupled token compression method outperforms unidirectional audio/video compression baselines on Qwen2.5-Omni while cutting prefill cost.
-
Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos
AV-Flamingo is an open audio-visual language model that outperforms similarly sized open models on many long-video understanding benchmarks, using a new 7M-question dataset and timestamped reasoning.
-
ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships
Writing editing instructions that explicitly bind each attribute to a reference image via `<Image_N>` tokens substantially improves multi-reference video editing, and a specialized MLLM trained with GRPO generates the...
-
MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
AVHaystacks is a new 3100-question benchmark for audio-visual QA across 500 videos, and the MAGNET multi-agent pipeline beats current baselines on it.
-
AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs
A clue-grounded audio-visual counting benchmark over 497 long videos and an RL-trained counting model, whose headline result is undermined by training on the DVD-Counting evaluation benchmark.
-
Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
A delayed bidirectional audio-visual alignment framework with bank-grounded disentangled audio queries achieves state-of-the-art segmentation results on AVS-Objects and VPO benchmarks.
-
ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts
A 7B multimodal model that fuses audio and visual signals with explicit timestamps achieves strong measured comprehension of real-world short videos on the authors' new ShortVid-Bench benchmark.
Discussion (0). Sign in to comment.