REVIEW 9 cited by
Valley: Video Assistant with Large Language model Enhanced abilitY
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs), with remarkable conversational capability, have emerged as AI assistants that can handle both visual and textual modalities. However, their effectiveness in joint video and language understanding has not been extensively explored. In the paper, we introduce Valley, a multi-modal foundation model that is designed to enable enhanced video comprehension and instruction-following capabilities. To this end, we construct two datasets, namely Valley-702k and Valley-instruct-73k, to cover a diverse range of video-text alignment and video-based instruction tasks, such as multi-shot captions, long video descriptions, action recognition, causal inference, etc. Then, we adopt ViT-L/14 as the vision encoder and explore three different temporal modeling modules to learn multifaceted features for enhanced video understanding. In addition, we implement a two-phase training approach for Valley: the first phase focuses solely on training the projection module to facilitate the LLM's capacity to understand visual input, and the second phase jointly trains the projection module and the LLM to improve their instruction following ability. Extensive experiments demonstrate that Valley has the potential to serve as an effective video assistant, simplifying complex video-understanding scenarios. Our code and data are published anonymously at https://github.com/valley-vl/Valley.
Forward citations
Cited by 9 Pith papers
-
Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors
LangDC compresses video tokens dynamically by converting clips into captions from a small language model, cutting compute by 49% with near-parity accuracy.
-
IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes
A new multi-shot video dataset and an instance-prompt video LLM report large gains, but the main benchmark is built by the same authors and the model is not released.
-
Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey
The paper offers the first focused review of MLLM-based video translation organized by a three-role taxonomy of Semantic Reasoner, Expressive Performer, and Visual Synthesizer, plus open challenges.
-
VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
Fine-tuning Qwen2.5-VL on VisReason, a 489K-example multi-round visual chain-of-thought dataset (165K with pseudo-depth), modestly improves LLM-judged visual reasoning scores, with caveats about self-referential 3D ev...
-
SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM
SMART, an audio-enhanced MLLM with shot-aware token compression, reports new state-of-the-art moment retrieval accuracy on Charades-STA and QVHighlights.
-
TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding
TimeExpert applies dynamic mixture-of-experts routing to video temporal grounding, reporting small state-of-the-art gains over TRACE on dense video captioning, moment retrieval, and highlight detection.
-
Task-Aware KV Compression For Cost-Effective Long Video Understanding
Video-X2L uses bi-level KV compression with task-aware selective reloading to improve long-video QA accuracy and reduce decode-time memory versus uniform KV compression.
-
Video Understanding by Design: How Datasets Shape Video Models
A dataset-centric framework that explains video architectures as responses to structural properties of benchmark datasets.
- MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answering
Discussion (0). Sign in to comment.