Pith. sign in

REVIEW 9 cited by

LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.11703 v1 pith:WDES6ZZL submitted 2024-03-18 cs.CV cs.AI

Ruyi Xu , Yuan Yao , Zonghao Guo , Junbo Cui , Zanlin Ni , Chunjiang Ge , Tat-Seng Chua , Zhiyuan Liu
show 2 more authors
Maosong Sun Gao Huang
This is my paper
classification cs.CVcs.AI
keywords imagesllava-uhdvisualencodingllava-1lmmsmodelaspect
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Visual encoding constitutes the basis of large multimodal models (LMMs) in understanding the visual world. Conventional LMMs process images in fixed sizes and limited resolutions, while recent explorations in this direction are limited in adaptivity, efficiency, and even correctness. In this work, we first take GPT-4V and LLaVA-1.5 as representative examples and expose systematic flaws rooted in their visual encoding strategy. To address the challenges, we present LLaVA-UHD, a large multimodal model that can efficiently perceive images in any aspect ratio and high resolution. LLaVA-UHD includes three key components: (1) An image modularization strategy that divides native-resolution images into smaller variable-sized slices for efficient and extensible encoding, (2) a compression module that further condenses image tokens from visual encoders, and (3) a spatial schema to organize slice tokens for LLMs. Comprehensive experiments show that LLaVA-UHD outperforms established LMMs trained with 2-3 orders of magnitude more data on 9 benchmarks. Notably, our model built on LLaVA-1.5 336x336 supports 6 times larger (i.e., 672x1088) resolution images using only 94% inference computation, and achieves 6.4 accuracy improvement on TextVQA. Moreover, the model can be efficiently trained in academic settings, within 23 hours on 8 A100 GPUs (vs. 26 hours of LLaVA-1.5). We make the data and code publicly available at https://github.com/thunlp/LLaVA-UHD.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Counterfactual present/removed teacher views attribute visually supported corrections and reconstruct student-anchored distillation targets that beat source-mixed multimodal OPD.

  2. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

  3. E-VLA: Event-Augmented Vision-Language-Action Model for Dark and Blurred Scenes

    cs.CV 2026-04 conditional novelty 6.0 of 10

    E-VLA integrates event streams directly into VLA models via lightweight fusion, raising Pick-Place success from 0% to 60-90% at 20 lux and from 0% to 20-25% under severe motion blur.

  4. LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LIRA improves referring segmentation and reduces hallucination in multimodal LLMs by fusing semantic and pixel features and interleaving local image regions with text descriptions.

  5. LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMs

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LLaVA-SP adds six spatial tokens produced by multi-scale cropping or pooling and cross-attention to MLLMs, improving 10/11 benchmarks over LLaVA-1.5 with nearly unchanged latency.

  6. SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    SparseMM shows that under 5% of attention heads carry visual understanding in MLLMs and uses OCR-derived head scores to allocate KV-cache budgets asymmetrically, preserving accuracy at low cache sizes.

  7. VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning

    cs.CV 2025-11 conditional novelty 5.0 of 10

    Fine-tuning Qwen2.5-VL on VisReason, a 489K-example multi-round visual chain-of-thought dataset (165K with pseudo-depth), modestly improves LLM-judged visual reasoning scores, with caveats about self-referential 3D ev...

  8. MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning

    cs.CV 2026-07 reject novelty 4.0 of 10

    A multi-scale gated fusion plus question-conditioned token router is specified to cut VLM visual tokens ~80% while improving accuracy, but all reported gains are simulated placeholders.

  9. Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A unified image understanding and generation model with decoupled visual encoders achieves competitive benchmark scores on both tasks.

Pith tools