REVIEW 21 cited by
TokenPacker: Efficient Visual Projector for Multimodal LLM
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The visual projector serves as an essential bridge between the visual encoder and the Large Language Model (LLM) in a Multimodal LLM (MLLM). Typically, MLLMs adopt a simple MLP to preserve all visual contexts via one-to-one transformation. However, the visual tokens are redundant and can be considerably increased when dealing with high-resolution images, impairing the efficiency of MLLMs significantly. Some recent works have introduced resampler or abstractor to reduce the number of resulting visual tokens. Unfortunately, they fail to capture finer details and undermine the visual reasoning capabilities of MLLMs. In this work, we propose a novel visual projector, which adopts a coarse-to-fine scheme to inject the enriched characteristics to generate the condensed visual tokens. In specific, we first interpolate the visual features as a low-resolution point query, providing the overall visual representation as the foundation. Then, we introduce a region-to-point injection module that utilizes high-resolution, multi-level region-based cues as fine-grained reference keys and values, allowing them to be fully absorbed within the corresponding local context region. This step effectively updates the coarse point query, transforming it into an enriched one for the subsequent LLM reasoning. Extensive experiments demonstrate that our approach compresses the visual tokens by 75%~89%, while achieves comparable or even better performance across diverse benchmarks with significantly higher efficiency. The source codes can be found at https://github.com/CircleRadon/TokenPacker.
Forward citations
Cited by 21 Pith papers
-
When Do Fewer Visual Tokens Accelerate Multimodal Inference? A Break-Even Study Across Decision Locations and Hardware
A break-even study of Qwen2.5-VL on two GPUs finds that a post-vision pruner and a pre-vision resolution rule both cut end-to-end latency, but pre-vision routing wins by skipping vision encoding, not by removing the m...
-
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
CRAFT recursively merges video tokens with training-free similarity selection plus learnable gated fusion, retaining ~97% of average accuracy at 8x compression across six benchmarks.
-
Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors
LangDC compresses video tokens dynamically by converting clips into captions from a small language model, cutting compute by 49% with near-parity accuracy.
-
CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification
CogVLA pairs instruction-conditioned visual-token aggregation (EFA-Routing) with transformer-layer pruning (LFP-Routing) and bidirectional action decoding (CAtten), reporting LIBERO 97.4%, real-world 70.0%, 2.5x less ...
-
Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models
ACCM recovers information lost in high-rate visual token pruning by generating a question-guided caption from discarded tokens and selecting the best candidate, improving pruned LVLM accuracy with fewer FLOPs.
-
HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentation
HRSeg combines high-resolution image crops, region attention, and cross-attention mask enhancement, improving reasoning segmentation over LLM-Seg by up to 13.2 gIoU points on LLM-Seg40K.
-
DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs
DisCo assigns each visual token to a unique concept from the caption and aligns its attention across frames, improving video MLLM accuracy and token efficiency.
-
FACap: A Large-scale Fashion Dataset for Fine-grained Composed Image Retrieval
FACap contributes 227,680 fashion CIR triplets with VLM/LLM-generated modification texts, and FashionBLIP-2 trained on it reaches 44.63 average Recall on FashionIQ without downstream fine-tuning and 65.97 with fine-tuning.
-
EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?
EOC-Bench evaluates MLLMs on egocentric object cognition across past, present, and future temporal dimensions, finding large gaps versus humans, especially in absolute time perception.
-
EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models
EffiVLM-Bench is a benchmark study showing token compression is task- and model-dependent, KV cache methods are more loyal, and parameter compression preserves accuracy better at typical ratios.
-
PixelThink: Towards Efficient Chain-of-Pixel Reasoning
A soft token-budget penalty driven by task difficulty and model confidence roughly halves the reasoning tokens used by segmentation MLLMs while slightly improving mask accuracy on ReasonSeg-derived benchmarks.
-
Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding
Fast3D prunes up to 90% of object-centric visual tokens in 3D MLLMs while preserving about 96.8% of original benchmark performance, using a trained attention predictor and adaptive layer-wise pruning.
-
LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models
Inserting a pixel-shuffle plus residual patch-merge layer inside the vision encoder compresses visual tokens more efficiently than post-encoder compression, at modest accuracy cost.
-
Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models
A new resolution-focused benchmark and an open-source native-resolution training framework show that preserving original image resolution improves VLM performance on fine-grained visual tasks.
-
Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs
Manager aggregates multi-layer unimodal representations and improves both two-tower VLMs (ManagerTower) and MLLMs (LLaVA-OV-Manager) on 24 downstream tasks.
-
HSENet: Hybrid Spatial Encoding Network for 3D Medical Vision-Language Understanding
HSENet improves 3D CT vision-language understanding by combining global and local 3D encoders with a centroid-based spatial token compressor, posting state-of-the-art results on CT-RATE and RadGenome-ChestCT.
-
Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings
Semantic clustering of patch embeddings at the final model layer, plus fine-tuning, preserves 94.6 to 98.2% of retrieval performance while reducing stored embedding memory to 1.8 to 11.8% of original.
-
MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning
A multi-scale gated fusion plus question-conditioned token router is specified to cut VLM visual tokens ~80% while improving accuracy, but all reported gains are simulated placeholders.
-
MAGE: Multimodal Alignment and Generation Enhancement via Bridging Visual and Semantic Spaces
A multimodal LLM with a learnable alignment projector and a combined cross-entropy and mean-squared-error loss reports state-of-the-art scores on several vision-language benchmarks using only 144 visual tokens.
-
Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.
-
DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding
DynTok dynamically merges similar adjacent visual tokens into groups, reducing video token counts to 44.4% with comparable or better video understanding accuracy.
Discussion (0). Sign in to comment.