A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models
read the original abstract
Visual token compression is critical for Large Vision-Language Models (LVLMs) to efficiently process high-resolution inputs. Existing methods that typically adopt fixed compression ratios cannot adapt to scenes of varying complexity, often causing imprecise pruning that discards informative visual tokens and results in degraded model performance. To address this issue, we introduce a dynamic pruning framework, GlimpsePrune, inspired by human cognition. It takes a data-driven ''glimpse'' and prunes irrelevant visual tokens in a single forward pass before answer generation. This approach prunes 92.6% of visual tokens while on average fully retaining the baseline performance on free-form VQA tasks. The reduced computational cost also enables more effective fine-tuning: an enhanced GlimpsePrune+ achieves 110% of the baseline performance while maintaining a similarly high pruning rate. Our work paves a new way for building more powerful and efficient LVLMs.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
Semantic-Geometric Dual Compression: Training-Free Visual Token Reduction for Ultra-High-Resolution Remote Sensing Understanding
DualComp uses a lightweight router to split visual token compression into a semantic stream with size-adaptive clustering and a geometric stream with path-tracing recovery, enabling low-cost high-fidelity UHR remote s...
-
Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models
CLSE prunes tokens in MLLMs by quantifying cross-layer spectral redistribution in the frequency domain to preserve semantically active tokens and reduce compute.
-
Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction
PriorTR estimates model-induced prior attention via a null token in one forward pass and contrasts it with task-conditioned attention to improve visual token pruning accuracy-efficiency trade-offs in MLLMs.
-
Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Models
DiffPrune reformulates visual token pruning as continuous control of token information using an Information Throttler with importance-conditioned variance-preserving noise, enabling fully differentiable learning of sc...
-
See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding
SWIM aligns cross-attention maps from object nouns to ground-truth masks during training on the new NL-Refer dataset to enable text-only fine-grained video object understanding in MLLMs.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.