REVIEW 12 cited by
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Large Vision-Language Models (LVLMs) show significant strides in general-purpose multimodal applications such as visual dialogue and embodied navigation. However, existing multimodal evaluation benchmarks cover a limited number of multimodal tasks testing rudimentary capabilities, falling short in tracking LVLM development. In this study, we present MMT-Bench, a comprehensive benchmark designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, reasoning, and planning. MMT-Bench comprises $31,325$ meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering $32$ core meta-tasks and $162$ subtasks in multimodal understanding. Due to its extensive task coverage, MMT-Bench enables the evaluation of LVLMs using a task map, facilitating the discovery of in- and out-of-domain tasks. Evaluation results involving $30$ LVLMs such as the proprietary GPT-4V, GeminiProVision, and open-sourced InternVL-Chat, underscore the significant challenges posed by MMT-Bench. We anticipate that MMT-Bench will inspire the community to develop next-generation multimodal foundation models aimed at achieving general-purpose multimodal intelligence.
Forward citations
Cited by 12 Pith papers
-
FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning
FeynmanBench is the first benchmark for evaluating multimodal LLMs on diagrammatic reasoning with Feynman diagrams, revealing systematic failures in enforcing physical constraints and global topology.
-
CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts
CoLT teaches multi-modal models to reason in a few latent steps with temporary dual-mode decoder supervision, cutting inference time ~10× versus text CoT while beating prior latent methods.
-
Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents
Argos is an agentic verifier that adaptively picks scoring functions to evaluate accuracy, localization, and reasoning quality, enabling stronger multimodal RL training for AI agents.
-
Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis
MMOral is a large new dental X-ray instruction dataset and benchmark, but the proposed model's 24.73% improvement is from fine-tuning and then testing on the same data pool.
-
SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
SEAM measures VLM reasoning consistency across modalities using paired semantically equivalent textual and visual notations, and finds systematic vision-language imbalance.
-
HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.
-
MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Models
A new 2,700-task benchmark with budget, time, and distance constraints shows that even the best multimodal LLMs produce feasible plans less than 22% of the time.
-
COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark
COREVQA introduces a 5,608-pair true/false visual entailment benchmark for crowd images on which the strongest tested vision-language models reach only 77.57% accuracy.
-
Large Multi-modal Model Cartographic Map Comprehension for Textual Locality Georeferencing
Grid-based prompting lets GPT-4o georeference textual locality descriptions from natural history records using map excerpts, averaging about 1.03 km error on a 25-example dataset.
-
MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
MMReason is an open-ended multimodal reasoning benchmark that filters out guessable and memorized questions and scores model answers both by final answer and by intermediate steps.
-
CoMemo: LVLMs Need Image Context with Image Memory
CoMemo adds a cross-attention image-memory path and thumbnail-anchored position encoding to reduce visual neglect in long-context and multi-image LVLM tasks.
-
ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models
Interleaving key video frames into step-by-step reasoning improves video question answering by 1.7 to 5.5 points over text-only chain-of-thought on a new self-built benchmark.
Discussion (0). Sign in to comment.