REVIEW 12 cited by
Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The rapid evolution of Multi-modality Large Language Models (MLLMs) has catalyzed a shift in computer vision from specialized models to general-purpose foundation models. Nevertheless, there is still an inadequacy in assessing the abilities of MLLMs on low-level visual perception and understanding. To address this gap, we present Q-Bench, a holistic benchmark crafted to systematically evaluate potential abilities of MLLMs on three realms: low-level visual perception, low-level visual description, and overall visual quality assessment. a) To evaluate the low-level perception ability, we construct the LLVisionQA dataset, consisting of 2,990 diverse-sourced images, each equipped with a human-asked question focusing on its low-level attributes. We then measure the correctness of MLLMs on answering these questions. b) To examine the description ability of MLLMs on low-level information, we propose the LLDescribe dataset consisting of long expert-labelled golden low-level text descriptions on 499 images, and a GPT-involved comparison pipeline between outputs of MLLMs and the golden descriptions. c) Besides these two tasks, we further measure their visual quality assessment ability to align with human opinion scores. Specifically, we design a softmax-based strategy that enables MLLMs to predict quantifiable quality scores, and evaluate them on various existing image quality assessment (IQA) datasets. Our evaluation across the three abilities confirms that MLLMs possess preliminary low-level visual skills. However, these skills are still unstable and relatively imprecise, indicating the need for specific enhancements on MLLMs towards these abilities. We hope that our benchmark can encourage the research community to delve deeper to discover and enhance these untapped potentials of MLLMs. Project Page: https://q-future.github.io/Q-Bench.
Forward citations
Cited by 12 Pith papers
-
Tunable Polariton Canalization in Natural van der Waals Oxide
Untreated alpha-V2O5 exhibits frequency-tunable in-plane polariton canalization with unidirectional Poynting-vector flow, mapped by infrared nano-imaging and a permittivity phase diagram.
-
Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
MMRB is the first benchmark combining multi-image inputs with chain-of-thought reasoning annotations, and its evaluation shows open-source MLLMs trail commercial models while multi-image reward models are unstable.
-
TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types
The authors release TaskGalaxy, a 413k-sample multimodal instruction dataset with 19,227 hierarchically organized task types, and report consistent benchmark gains when it is added to LLaVA-v1.5 and InternVL-Chat-v1.0.
-
SVGEval: A Vision-Grounded Framework for Perceptual-Quality Benchmarking and Evaluation in Text-to-SVG Generation
SVGEval benchmarks and trains an explainable multimodal scorer for perceptual quality of text-to-SVG generation, showing a consistent gap on spatial and structural judgments.
-
AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models
AIGI-Holmes combines visual expert pretraining, SFT on explanation data, and direct preference optimization to deliver human-verifiable explanations and top detection accuracy on unseen AI generators.
-
Q-Ponder: A Unified Training Pipeline for Reasoning-based Visual Quality Assessment
Q-Ponder is a two-stage pipeline (distill-then-reinforce) that makes a 7B multimodal model both more accurate at image quality scoring and better at explaining its judgments.
-
Affordance Benchmark for MLLMs
A new 2,000-question benchmark finds multimodal AI models recognize object affordances far worse than humans, with top model Gemini-2.0-Pro at 18.05% versus 85.34% human best.
-
Parameter-Efficient Adaptation of mPLUG-Owl2 via Pixel-Level Visual Prompts for NR-IQA
With a learned 30-pixel border prompt added to input images, a frozen mPLUG-Owl2-7B reaches 0.932 SRCC on KADID-10k using about 156K trainable parameters.
-
LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models
Inserting a pixel-shuffle plus residual patch-merge layer inside the vision encoder compresses visual tokens more efficiently than post-encoder compression, at modest accuracy cost.
-
VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos
A new benchmark, VF-Eval, measures how well multimodal LLMs check, detect, and reason about errors in AI-generated videos, and shows frontier models remain far below human performance.
-
Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion
Instructify converts image metadata into visual instruction-tuning conversations with open LLMs, matching or exceeding GPT-4-generated data quality on LMM benchmarks.
-
Position: Agentic Systems Constitute a Key Component of Next-Generation Intelligent Image Processing
Image processing should move from monolithic deep models to agentic systems that orchestrate multiple tools, with a proposed six-level autonomy ladder.
Discussion (0). Sign in to comment.