REVIEW 9 cited by
Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Seeing clearly with high resolution is a foundation of Large Multimodal Models (LMMs), which has been proven to be vital for visual perception and reasoning. Existing works usually employ a straightforward resolution upscaling method, where the image consists of global and local branches, with the latter being the sliced image patches but resized to the same resolution as the former. This means that higher resolution requires more local patches, resulting in exorbitant computational expenses, and meanwhile, the dominance of local image tokens may diminish the global context. In this paper, we dive into the problems and propose a new framework as well as an elaborate optimization strategy. Specifically, we extract contextual information from the global view using a mixture of adapters, based on the observation that different adapters excel at different tasks. With regard to local patches, learnable query embeddings are introduced to reduce image tokens, the most important tokens accounting for the user question will be further selected by a similarity-based selector. Our empirical results demonstrate a `less is more' pattern, where \textit{utilizing fewer but more informative local image tokens leads to improved performance}. Besides, a significant challenge lies in the training strategy, as simultaneous end-to-end training of the global mining block and local compression block does not yield optimal results. We thus advocate for an alternating training way, ensuring balanced learning between global and local aspects. Finally, we also introduce a challenging dataset with high requirements for image detail, enhancing the training of the local compression layer. The proposed method, termed LMM with Sophisticated Tasks, Local image compression, and Mixture of global Experts (SliME), achieves leading performance across various benchmarks with only 2 million training data.
Forward citations
Cited by 9 Pith papers
-
BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception
BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.
-
HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Framework
HART uses a closed-loop 'crop-and-answer' training scheme plus a dynamic-weight GRPO variant to improve LMM grounding and high-resolution reasoning without bounding-box annotations.
-
CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification
CogVLA pairs instruction-conditioned visual-token aggregation (EFA-Routing) with transformer-layer pruning (LFP-Routing) and bidirectional action decoding (CAtten), reporting LIBERO 97.4%, real-world 70.0%, 2.5x less ...
-
Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models
ACCM recovers information lost in high-rate visual token pruning by generating a question-guided caption from discarded tokens and selecting the best candidate, improving pruned LVLM accuracy with fewer FLOPs.
-
GenRecal: Generation after Recalibration from Large to Small Vision-Language Models
A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.
-
AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs
An agent-driven framework adaptively selects a small subset of benchmark questions for MLLMs, preserving over 90% ranking accuracy with roughly 4-5% of the data.
-
Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention
HyCo2 combines soft global compression with hard local token selection, reporting QA performance near uncompressed retrieval while cutting context tokens by about 88.8%.
-
MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
A human-annotated multimodal preference dataset plus critique-based reward modeling and reward-margin-weighted DPO improves MLLM performance across many benchmarks.
-
Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models
A new resolution-focused benchmark and an open-source native-resolution training framework show that preserving original image resolution improves VLM performance on fine-grained visual tasks.
Discussion (0). Continue with ORCID to comment.