REVIEW 30 cited by
Improve Vision Language Model Chain-of-thought Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial for improving interpretability and trustworthiness. However, current training recipes lack robust CoT reasoning data, relying on datasets dominated by short annotations with minimal rationales. In this work, we show that training VLM on short answers does not generalize well to reasoning tasks that require more detailed responses. To address this, we propose a two-fold approach. First, we distill rationales from GPT-4o model to enrich the training data and fine-tune VLMs, boosting their CoT performance. Second, we apply reinforcement learning to further calibrate reasoning quality. Specifically, we construct positive (correct) and negative (incorrect) pairs of model-generated reasoning chains, by comparing their predictions with annotated short answers. Using this pairwise data, we apply the Direct Preference Optimization algorithm to refine the model's reasoning abilities. Our experiments demonstrate significant improvements in CoT reasoning on benchmark datasets and better generalization to direct answer prediction as well. This work emphasizes the importance of incorporating detailed rationales in training and leveraging reinforcement learning to strengthen the reasoning capabilities of VLMs.
Forward citations
Cited by 30 Pith papers
-
Internalized Reasoning for Long-Context Visual Document Understanding
Synthetic page-ranked reasoning traces plus low-strength model merging give a 32B VLM 58.3 on MMLongBenchDoc, beating a 235B teacher while cutting output tokens ~12× versus explicit reasoning.
-
More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
Reasoning models trade visual grounding for language-based inference, and this paper measures that trade-off with a new metric and benchmark.
-
Egocentric Bias in Vision-Language Models
Most vision-language models fail Level-2 visual perspective taking: they report the camera's view rather than the 180°-rotated string, even though they often recognize that another agent sees differently.
-
VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning
A vision-language model learns via reinforcement learning when to upscale a low-resolution image, cutting visual tokens roughly in half while preserving accuracy on most benchmarks.
-
The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs
Long-CoT SFT and RL trade off strengths in reasoning VLMs, and five combination strategies fail to produce additive gains.
-
MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning
MagiC evaluates answer correctness, reasoning validity, grounding fidelity, and self-correction on about 900 hand-annotated visual questions across 15 vision-language models.
-
Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models
Vision-language models can handle some 2D shape puzzles but nearly all fail at multi-step 3D spatial deformation reasoning.
-
MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
MMReason is an open-ended multimodal reasoning benchmark that filters out guessable and memorized questions and scores model answers both by final answer and by intermediate steps.
-
ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs
A verifiable RL proxy task that asks VLMs to locate a single injected hallucination in a 200-word caption improves visual perception and transfers to math and abstract reasoning benchmarks.
-
VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking
GRPO post-training with structured thinking and dual think/caption rewards improves Qwen2-VL-7B video captioning over the base model and SFT on DREAM-1K, VDC, and CAREBENCH.
-
GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking
A cue-tagging and rethinking training recipe lifts a 7B multimodal model to 81.5% on M3CoT, though the gain may reflect training on the same benchmark.
-
Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks
Agent-X, a new 828-task benchmark with step-level scoring, finds that the best multimodal agents still fail over half of real-world visual tool-use tasks.
-
CrashAgent: Crash Scenario Generation via Multi-modal Reasoning
A multi-agent vision-language framework converts NHTSA crash reports into simulation-ready road layouts and collision scenarios, with modest accuracy gains over direct VLM baselines.
-
Integrating Visual Interpretation and Linguistic Reasoning for Math Problem Solving
A decoupled vision-language reasoning system, where a VLM writes image descriptions and an LLM reasons from them, beats end-to-end LVLMs on geometry benchmarks, especially vision-heavy variants.
-
ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark
ARB provides 1,356 Arabic multimodal questions with 5,119 human-reviewed reasoning steps and shows leading models score much higher on reasoning fluency than on correct answers.
-
R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO
Share-GRPO creates paraphrased and visually augmented versions of reasoning questions and shares answers and reward signals across versions, improving multimodal reasoning without cold-start SFT.
-
Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs
Tree-based reasoning prompts consistently underperform standard zero-shot prompting for VLM image classification on GTSRB and CIFAR-10 across three models.
-
Salience Adjustment for Context-Based Emotion Recognition
A salience-weighted Bayesian cue integration formula improves automated emotion recognition on the Split-Steal corpus, but the weighting is fit to the evaluation set and no held-out validation is provided.
-
Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization
Tactile-VLA fuses tactile sensing into a vision-language-action model so force-related instructions and corrective reasoning transfer to new contact-rich tasks with few demonstrations.
-
ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoning
An event-stream scene text recognizer trained with LLM-generated chain-of-thought rationales improves BLEU-1 on EventSTR from 0.638 to 0.648 and accuracy on WordArt* and IC15* by about half a point.
-
Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought
Argus adds explicit language-guided visual attention to multimodal LLMs by grounding questions to bounding boxes and re-engaging those regions, improving vision-centric reasoning and grounding accuracy.
-
Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models
Infi-MMR-3B trains a 3B visual language model with text-first, caption-augmented, then caption-free reinforcement learning, reaching 43.68% on MathVerse testmini.
-
Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start
Using a two-stage SFT-plus-GRPO pipeline, the authors train 3B and 7B multimodal models that beat prior open-source models on four math reasoning benchmarks, while showing that reflective 'aha moment' text is not a re...
-
Training-Free Reasoning and Reflection in MLLMs
Training-free, layer-wise weight merging of an MLLM with a reasoning LLM, using attention-derived priors, raises MMMU accuracy from 63.9 to 69.2 at the 38B scale.
-
Toward Effective Reinforcement Learning Fine-Tuning for Medical VQA in Vision-Language Models
On a PMC-VQA subset, GRPO-based RL fine-tuning of Qwen2-VL-2B-Instruct outperforms SFT in accuracy, but the study has no error bars and several prose claims conflict with its own table.
-
A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.
-
Towards Video to Piano Music Generation with Chain-of-Perform Support Benchmarks
The paper proposes an open-source 10-hour video-to-piano benchmark with four-level Chain-of-Perform annotations, but supplies only preliminary, incomplete baseline results.
-
Spatial navigation in preclinical Alzheimer's disease: A review
Spatial navigation performance, particularly path integration and wayfinding, correlates with AD biomarkers such as p-tau in cognitively unimpaired at-risk individuals and may enable earlier detection than episodic me...
-
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.
-
ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning
ASPO's adaptive sentence-level loss, by the paper's own definitions, reduces exactly to the standard DPO loss, leaving no difference in the optimization objective.
Discussion (0). Sign in to comment.