REVIEW 5 cited by
Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multimodal Large Language Models (MLLMs) have recently achieved promising zero-shot accuracy on visual question answering (VQA) -- a fundamental task affecting various downstream applications and domains. Given the great potential for the broad use of these models, it is important to investigate their limitations in dealing with different image and question properties. In this work, we investigate whether MLLMs can perceive small details as well as large details in images. In particular, we show that their zero-shot accuracy in answering visual questions is very sensitive to the size of the visual subject of the question, declining up to 46% with size. Furthermore, we show that this effect is causal by observing that human visual cropping can significantly mitigate their sensitivity to size. Inspired by the usefulness of human cropping, we then propose five automatic visual cropping methods -- leveraging either external localization models or the decision process of the given MLLM itself -- as inference time mechanisms to improve the zero-shot performance of MLLMs. We study their effectiveness on four popular VQA datasets, and a subset of the VQAv2 dataset tailored towards fine visual details. Our findings suggest that MLLMs should be used with caution in detail-sensitive VQA applications, and that visual cropping is a promising direction to improve their zero-shot performance. To facilitate further investigation of MLLMs' behaviors, our code and data are publicly released.
Forward citations
Cited by 5 Pith papers
-
Look Where It Matters: Adaptive Visual Refinement for Vision-Language-Action Models
Inserting register tokens into a VLA encoder plus uncertainty-gated, attention-guided cropping raises π0's success from 94.2% to 98.4% on LIBERO and 46.5% to 69.0% on a real-world benchmark, at 1.4–1.6× compute.
-
Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
A new counterfactual benchmark, PriVE-Bench, plus a controlled tool-based extension, PriVE-Tools, shows that VLMs often answer from priors and that tool-derived visual evidence helps some models but does not reliably ...
-
Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model
Folding visual evidence acquisition into the conformal score and re-calibrating on post-acquisition scores preserves the hallucination-rate guarantee while recovering coverage in LVLM selective prediction.
-
LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
A confidence-based when-to-look gate plus semantic-guided attention where-to-look module improves training-free fine-grained visual reasoning accuracy and reduces inference cost versus search-based baselines.
-
MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval
MCERF delivers a 41.1% relative accuracy gain on the DesignQA benchmark by combining ColPali vision-language retrieval with four specialized reasoning modes and dynamic routing.
Discussion (0). Sign in to comment.