Pith. sign in

REVIEW 11 cited by

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.16999 v3 pith:QX5VEDMD submitted 2024-03-25 cs.CV

Hao Shao , Shengju Qian , Han Xiao , Guanglu Song , Zhuofan Zong , Letian Wang , Yu Liu , Hongsheng Li This is my paper
classification cs.CV
keywords visualbenchmarkdatasetmodelsannotatedansweringinputsintroduce
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. However, they often lack interpretability and struggle with complex visual inputs, especially when the resolution of the input image is high or when the interested region that could provide key information for answering the question is small. To address these challenges, we collect and introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions. Additionally, about 98k pairs of them are annotated with detailed reasoning steps. Importantly, we propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts. We also introduce the related benchmark to evaluate the MLLMs in scenarios requiring specific local region identification. Extensive experiments demonstrate the effectiveness of our framework and shed light on better inference strategies. The Visual CoT dataset, benchmark, and pre-trained models are available on https://hao-shao.com/projects/viscot.html to support further research in this area.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

    cs.AI 2026-07 conditional novelty 7.0 of 10

    BRAID jointly optimizes text and image generation in interleaved multi-modal reasoning by casting the full trajectory as a unified MDP with shared advantages and a VLM process reward.

  2. MentalThink: Shaping Thoughts in Mental SVG World

    cs.AI 2026-07 conditional novelty 7.0 of 10

    MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.

  3. Teaching MLLMs to Say No: Generalized Referring Expression Comprehension via Refusal Calibrated GRPO

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A recalibrated GRPO reinforcement learning method lets multimodal LLMs say 'None' for nonexistent referring expressions without sacrificing localization accuracy on objects that do exist.

  4. OPLD: On-Policy Latent Distillation for Multimodal Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    On-policy distillation of a CoT-privileged teacher’s token preferences and latent trajectories yields a CoT-free student that beats prior visual-latent methods on several multimodal benchmarks.

  5. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

  6. Multilingual Multimodal Software Developer for Code Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.

  7. MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MagiC evaluates answer correctness, reasoning validity, grounding fidelity, and self-correction on about 900 hand-annotated visual questions across 15 vision-language models.

  8. Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification

    cs.CV 2025-06 conditional novelty 6.0 of 10

    An iterative, verifier-guided visual token scaling framework improves multi-step visual reasoning in both closed and open multimodal models on BLINK and related benchmarks.

  9. VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning

    cs.CV 2025-11 conditional novelty 5.0 of 10

    Fine-tuning Qwen2.5-VL on VisReason, a 489K-example multi-round visual chain-of-thought dataset (165K with pseudo-depth), modestly improves LLM-judged visual reasoning scores, with caveats about self-referential 3D ev...

  10. Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    Selective fine-tuning of language-specific shallow layers identified by neuron activation analysis improves multilingual vision-language performance with only 14% of parameters tuned.

  11. From Answers to Rationales: Self-Aligning Multimodal Reasoning with Answer-Oriented Chain-of-Thought

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Answer-oriented chain-of-thought prompts that generate both positive and negative reasoning data, combined with iterative DPO, improve multimodal LLM reasoning on several benchmarks.

Pith tools