REVIEW 19 cited by
Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Visual data comes in various forms, ranging from small icons of just a few pixels to long videos spanning hours. Existing multi-modal LLMs usually standardize these diverse visual inputs to a fixed resolution for visual encoders and yield similar numbers of tokens for LLMs. This approach is non-optimal for multimodal understanding and inefficient for processing inputs with long and short visual contents. To solve the problem, we propose Oryx, a unified multimodal architecture for the spatial-temporal understanding of images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to seamlessly and efficiently process visual inputs with arbitrary spatial sizes and temporal lengths through two core innovations: 1) a pre-trained OryxViT model that can encode images at any resolution into LLM-friendly visual representations; 2) a dynamic compressor module that supports 1x to 16x compression on visual tokens by request. These design features enable Oryx to accommodate extremely long visual contexts, such as videos, with lower resolution and high compression while maintaining high recognition precision for tasks like document understanding with native resolution and no compression. Beyond the architectural improvements, enhanced data curation and specialized training on long-context retrieval and spatial-aware data help Oryx achieve strong capabilities in image, video, and 3D multimodal understanding simultaneously. Our work is open-sourced at https://github.com/Oryx-mllm/Oryx.
Forward citations
Cited by 19 Pith papers
-
VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videos
A new dataset of 420 math video-question pairs with step-by-step reasoning annotations shows that current multimodal AI models, including the best proprietary system, answer fewer than half of the multi-binary questio...
-
Stateful Token Reduction for Long-Video Hybrid VLMs
For hybrid Mamba–Transformer video models, keeping 25% of visual tokens with a query-based progressive schedule gives 3.8–4.2x prefilling speedups with near-baseline accuracy; the paper attributes this to stateful com...
-
Social Caption: Evaluating Social Understanding in Multimodal Models
A three-part benchmark for multimodal LLMs separates question-answering about social videos from holistic and directed scene description, and shows AI judges can approximate human quality ratings.
-
MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models
MESH, a three-layer video hallucination benchmark, shows LVMs ace basic objects and coarse traits but slip badly on fine character details and multi-subject actions in longer clips.
-
HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.
-
Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
Benchmarks seven open-source audio-video-text MLLMs on six affective datasets and shows a generative-knowledge prompting step improves fine-tuned emotion recognition.
-
Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding
A new 1,000-video benchmark with adversarial question variants shows video LLMs remain far below human accuracy and robustness on short real-world videos.
-
HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentation
HRSeg combines high-resolution image crops, region attention, and cross-attention mask enhancement, improving reasoning segmentation over LLM-Seg by up to 13.2 gIoU points on LLM-Seg40K.
-
Flash-VStream: Efficient Real-Time Understanding for Long Video Streams
Flash-VStream uses a two-memory design (clustered low-res summaries plus high-res key-frame details) to answer questions about long videos with state-of-the-art benchmark scores and sub-second latency.
-
LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs
A training-free token compression method using semantic connected components in space and time keeps video understanding accuracy high even when retaining only 5-10% of visual tokens.
-
Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning
A 3B agent trained with supervised tool-use traces and reinforcement learning answers week-long egocentric video questions by dynamically selecting hierarchical retrieval, video-LLM, and VLM tools.
-
How Important are Videos for Training Video LLMs?
Training video LLMs on ordered static images with temporal questions nearly matches real video training on TVBench, suggesting video data is underused.
-
SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
SparseMM shows that under 5% of attention heads carry visual understanding in MLLMs and uses OCR-derived head scores to allocate KV-cache budgets asymmetrically, preserving accuracy at low cache sizes.
-
VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation
VCapsBench is a video caption quality benchmark with 109,796 QA pairs across 21 fine-grained dimensions on 5,677 videos, evaluating caption accuracy, inconsistency, and coverage.
-
Vision-Language Memory for Spatial Reasoning
A video-based vision-language model with 3D-aligned visual features and bounded dual memory achieves state-of-the-art scores on four spatial reasoning benchmarks.
-
Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark
MTMEUR is a new multimodal emotion reasoning benchmark where the best single model scores 71.19% and a four-agent reasoning framework tops 72.93%.
-
Enhancing Sports Strategy with Video Analytics and Data Mining: Assessing the effectiveness of Multimodal LLMs in tennis video analysis
VideoLLaMA2's tennis sequence edit score jumps from 39.7 to 76.0 when text coordinates from detection models are included in the prompt, and a separately fine-tuned CLIP encoder raises single-event accuracy from 0.41 to 0.56.
-
Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs
Manager aggregates multi-layer unimodal representations and improves both two-tower VLMs (ManagerTower) and MLLMs (LLaVA-OV-Manager) on 24 downstream tasks.
-
FlexSelect: Flexible Token Selection for Efficient Long Video Understanding
FlexSelect selects a small fraction of query-relevant visual tokens using attention from an intermediate layer, improving long-video accuracy and inference speed across multiple VideoLLMs.
Discussion (0). Sign in to comment.