REVIEW 9 cited by
Grounded Chain-of-Thought for Multimodal Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite great progress, existing multimodal large language models (MLLMs) are prone to visual hallucination, greatly impeding their trustworthy applications. In this paper, we study this problem from the perspective of visual-spatial reasoning, and propose a new learning task for MLLMs, termed Grounded Chain-of-Thought (GCoT). Different from recent visual CoT studies, which focus more on visual knowledge reasoning, GCoT is keen to helping MLLMs to recognize and ground the relevant visual cues step by step, thereby predicting the correct answer with grounding coordinates as the intuitive basis. To facilitate this task, we also carefully design and construct a dataset called multimodal grounded chain-of-thought (MM-GCoT) consisting of 24,022 GCoT examples for 5,033 images. Besides, a comprehensive consistency evaluation system is also introduced, including the metrics of answer accuracy, grounding accuracy and answer-grounding consistency. We further design and conduct a bunch of experiments on 12 advanced MLLMs, and reveal some notable findings: i. most MLLMs performs poorly on the consistency evaluation, indicating obvious visual hallucination; ii. visual hallucination is not directly related to the parameter size and general multimodal performance, i.e., a larger and stronger MLLM is not less affected by this issue. Lastly, we also demonstrate that the proposed dataset can help existing MLLMs to well cultivate their GCoT capability and reduce the inconsistent answering significantly. Moreover, their GCoT can be also generalized to exiting multimodal tasks, such as open-world QA and REC.
Forward citations
Cited by 9 Pith papers
-
Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs
AGS uses the ratio of attention to image tokens vs text tokens to switch multimodal LLMs between latent and explicit reasoning, improving accuracy and cutting decoding steps.
-
OPLD: On-Policy Latent Distillation for Multimodal Reasoning
On-policy distillation of a CoT-privileged teacher’s token preferences and latent trajectories yields a CoT-free student that beats prior visual-latent methods on several multimodal benchmarks.
-
Visual Access Boundaries in Vision-Language Model Reasoning
CoT in VLMs extends language-side computation over early image-derived states rather than prolonging direct image-token access, with gains limited by perceptual readout reliability.
-
Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
Selectively attacking the visual input with a gradient of response entropy during GRPO sampling increases policy entropy and improves visual reasoning accuracy by roughly 1–2 points.
-
Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture
Spatial understanding in multimodal LLMs plateaus quickly as training data grows, and position encoding in the visual encoder is the more influential factor.
-
Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning
A 3B agent trained with supervised tool-use traces and reinforcement learning answers week-long egocentric video questions by dynamically selecting hierarchical retrieval, video-LLM, and VLM tools.
-
VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
Fine-tuning Qwen2.5-VL on VisReason, a 489K-example multi-round visual chain-of-thought dataset (165K with pseudo-depth), modestly improves LLM-judged visual reasoning scores, with caveats about self-referential 3D ev...
-
Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation
Injecting self-verified bounding boxes into Chain-of-Thought data improves few-shot adaptation of multimodal LLMs on charts, tables, receipts, and reports.
-
A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.
Discussion (0). Continue with ORCID to comment.