REVIEW 12 cited by
From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multi-modal Large Language Models (MLLMs) have made significant strides in expanding the capabilities of Large Language Models (LLMs) through the incorporation of visual perception interfaces. Despite the emergence of exciting applications and the availability of diverse instruction tuning data, existing approaches often rely on CLIP or its variants as the visual branch, and merely extract features from the deep layers. However, these methods lack a comprehensive analysis of the visual encoders in MLLMs. In this paper, we conduct an extensive investigation into the effectiveness of different vision encoders within MLLMs. Our findings reveal that the shallow layer features of CLIP offer particular advantages for fine-grained tasks such as grounding and region understanding. Surprisingly, the vision-only model DINO, which is not pretrained with text-image alignment, demonstrates promising performance as a visual branch within MLLMs. By simply equipping it with an MLP layer for alignment, DINO surpasses CLIP in fine-grained related perception tasks. Building upon these observations, we propose a simple yet effective feature merging strategy, named COMM, that integrates CLIP and DINO with Multi-level features Merging, to enhance the visual capabilities of MLLMs. We evaluate COMM through comprehensive experiments on a wide range of benchmarks, including image captioning, visual question answering, visual grounding, and object hallucination. Experimental results demonstrate the superior performance of COMM compared to existing methods, showcasing its enhanced visual capabilities within MLLMs.
Forward citations
Cited by 12 Pith papers
-
InCoM: Intent-Driven Perception and Structured Coordination for Mobile Manipulation
InCoM reports 23–28 percentage-point success-rate gains in mobile manipulation benchmarks by dynamically reweighting multi-scale perception via inferred motion intent and decoupling base-arm action generation with flo...
-
Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders
Coverage of sparse-autoencoder-identified task features predicts post-training performance and can guide synthesis of small, high-impact datasets (2,000 vs. 300,000 samples).
-
Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference
Harmonia runs LLM inference with all activations in block floating point (BFP) and a 4-bit KV cache, reporting 3.08x average speedup, 2.03x energy savings, and under 1% accuracy loss on LongBench.
-
Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Models
Language models accept misleading endorsements more readily from higher-expertise persona sources, and the effect can be partially steered away by subtracting a residual-stream vector.
-
VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
A 1,680-question video benchmark shows leading multimodal models lag humans by ~15 points on visual knowledge, and a See-Think-Answer RL-trained model narrows the gap.
-
Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations
A new hypothesis test for binary LLM responses treats semantically equivalent query perturbations as an unknown null set and gives asymptotic validity and consistency guarantees under a uniformity assumption.
-
TMUAD: Enhancing Logical Capabilities in Unified Anomaly Detection Models with a Text Memory Bank
TMUAD detects both structural and logical anomalies by comparing a query image's textual object descriptions, object crops, and image patches against three normal memory banks, reaching state-of-the-art AUROC on seven...
-
Decoding Memories: An Efficient Pipeline for Self-Consistency Hallucination Detection
A decoding pipeline reuses cached tokens and anneals sampling temperature to accelerate self-consistency hallucination detection by up to 3x without meaningful AUROC loss.
-
CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
A method that uses a handful of 'semantic retrieval heads' instead of all attention heads to decide which key-value cache entries can be dropped, plus layer-wise cache budgeting, reportedly beats prior KV compression ...
-
METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models
METEOR is a three-stage token pruning framework that reduces visual tokens in multi-encoder MLLMs by 76% with only a 0.3% average accuracy drop.
-
Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor
Stable Diffusion features, especially when conditioned on the question, improve vision-centric multimodal question answering when fused with CLIP.
-
HEAL: A Hypothesis-Based Preference-Aware Analysis Framework
HEAL evaluates preference optimization by measuring ranking accuracy and strength correlation between model likelihoods and proxy reward scores over multi-response hypothesis spaces.
Discussion (0). Sign in to comment.