pith. sign in

arxiv: 2505.15879 · v2 · submitted 2025-05-21 · 💻 cs.CV · cs.AI· cs.CL

GRIT: Teaching MLLMs to Think with Images

Yue Fan , Xuehai He , Diji Yang , Kaizhi Zheng , Ching-Chen Kuo , Yuting Zheng , Sravana Jyothi Narayanaraju , Xinze Guan
show 1 more author
Xin Eric Wang
This is my paper

Pith reviewed 2026-05-22 13:26 UTC · model grok-4.3

classification 💻 cs.CV cs.AIcs.CL
keywords grounded reasoningmultimodal LLMsvisual groundingreinforcement learningdata efficiencybounding boxesreasoning chains
0
0 comments X

The pith

GRIT trains MLLMs to generate reasoning chains that interleave text with explicit bounding box coordinates using only final-answer rewards.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents GRIT as a way to make multimodal large language models produce reasoning that mixes natural language steps with direct references to image regions via bounding boxes. It applies a reinforcement learning procedure called GRPO-GR whose only signals are whether the final answer is correct and whether the output format is right. This setup removes any requirement for datasets that contain step-by-step reasoning annotations or labeled boxes. The result is that useful grounded reasoning emerges after exposure to as few as twenty ordinary image-question-answer examples drawn from existing collections.

Core claim

GRIT establishes a grounded reasoning paradigm in which models generate chains that interleave natural language and explicit bounding box coordinates pointing to consulted image regions, trained by the GRPO-GR reinforcement learning algorithm that supplies rewards exclusively for final answer accuracy and correct output format, thereby unifying reasoning and grounding without any reasoning-chain or bounding-box annotations in the training data.

What carries the argument

The grounded reasoning paradigm that forces interleaved natural-language steps and explicit bounding-box coordinates, optimized by the GRPO-GR reinforcement learning procedure whose rewards depend only on final-answer correctness and output-format compliance.

If this is right

  • Models produce coherent reasoning chains that visibly reference specific image regions.
  • Reasoning and grounding abilities are combined in a single output format without separate training stages.
  • Training succeeds with only twenty image-question-answer triplets from existing datasets.
  • No annotated reasoning chains or bounding-box labels are needed for effective learning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same reward structure might be tested on tasks where the correct box locations are more ambiguous, such as abstract diagrams or multi-object scenes.
  • Inspecting the emitted boxes after training could serve as a lightweight way to audit which image parts the model actually uses for its decisions.
  • Similar format-based rewards could be applied to other multimodal settings that currently require expensive chain-of-thought annotations.

Load-bearing premise

Rewards based solely on final answer accuracy and output format compliance are sufficient to train models to produce correct and meaningful bounding box coordinates without any explicit bounding box labels or reasoning chain annotations.

What would settle it

An evaluation that extracts the bounding boxes the model emits during reasoning and measures their overlap with human-labeled regions that are actually required for the correct answer; low overlap would indicate the boxes are not meaningfully grounded.

Figures

Figures reproduced from arXiv: 2505.15879 by Ching-Chen Kuo, Diji Yang, Kaizhi Zheng, Sravana Jyothi Narayanaraju, Xin Eric Wang, Xinze Guan, Xuehai He, Yue Fan, Yuting Zheng.

Figure 1
Figure 1. Figure 1: Comparison of reasoning with pure natural language and grounded reasoning from GRIT [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Model update via GRPO-GR. During GRPO-GR training, we sample a group of model [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Inference examples of Qwen2.5-VL-GRIT. Results. The results are summarized in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Correlation between image regions and "thoughts" in grounded reasoning eval￾uated by our Vision-Language Reasoning Cross-Modal Correlation metric. The result shows that models trained with GRIT outper￾form baselines [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of scaling training data on model [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt suffix that is appended to the input of models during the training and inference. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt format for GPT-as-judge answer accuracy score and GPT-aided answer-accuracy [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
read the original abstract

Recent studies have demonstrated the efficacy of using Reinforcement Learning (RL) in building reasoning models that articulate chains of thoughts prior to producing final answers. However, despite ongoing advances that aim at enabling reasoning for vision-language tasks, existing open-source visual reasoning models typically generate reasoning content with pure natural language, lacking explicit integration of visual information. This limits their ability to produce clearly articulated and visually grounded reasoning chains. To this end, we propose Grounded Reasoning with Images and Texts (GRIT), a novel method for training MLLMs to think with images. GRIT introduces a grounded reasoning paradigm, in which models generate reasoning chains that interleave natural language and explicit bounding box coordinates. These coordinates point to regions of the input image that the model consults during its reasoning process. Additionally, GRIT is equipped with a reinforcement learning approach, GRPO-GR, built upon the GRPO algorithm. GRPO-GR employs robust rewards focused on the final answer accuracy and format of the grounded reasoning output, which eliminates the need for data with reasoning chain annotations or explicit bounding box labels. As a result, GRIT achieves exceptional data efficiency, requiring as few as 20 image-question-answer triplets from existing datasets. Comprehensive evaluations demonstrate that GRIT effectively trains MLLMs to produce coherent and visually grounded reasoning chains, showing a successful unification of reasoning and grounding abilities.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The manuscript proposes GRIT, a method for training MLLMs to generate reasoning chains that interleave natural language with explicit bounding-box coordinates. It introduces the GRPO-GR reinforcement-learning algorithm, which applies rewards only for final-answer accuracy and output-format compliance (including presence of box tokens). The approach claims to eliminate the need for reasoning-chain or bounding-box annotations and to achieve exceptional data efficiency, succeeding with as few as 20 image-question-answer triplets while unifying reasoning and grounding abilities.

Significance. If the empirical results hold, the work would represent a meaningful step toward data-efficient, visually grounded reasoning in multimodal models. The removal of explicit supervision for chains and boxes, combined with the reported unification of capabilities, could improve interpretability on vision-language tasks that require reference to specific image regions.

major comments (1)
  1. [Abstract] Abstract: the central claim that GRPO-GR produces accurate, task-relevant bounding boxes that participate in the reasoning process rests on the assumption that rewards for final-answer correctness and format compliance are sufficient. The described reward structure contains no term that penalizes hallucinated or irrelevant boxes when the final answer is correct; this is load-bearing for the unification and data-efficiency assertions, because the model could satisfy the rewards by emitting plausible-looking coordinates while relying on non-grounded cues.
minor comments (1)
  1. The abstract states that 'comprehensive evaluations demonstrate' the method's effectiveness but supplies no quantitative metrics, baselines, or dataset names; a brief summary of key results should be added to the abstract.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive feedback on our manuscript. We address the major comment point by point below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that GRPO-GR produces accurate, task-relevant bounding boxes that participate in the reasoning process rests on the assumption that rewards for final-answer correctness and format compliance are sufficient. The described reward structure contains no term that penalizes hallucinated or irrelevant boxes when the final answer is correct; this is load-bearing for the unification and data-efficiency assertions, because the model could satisfy the rewards by emitting plausible-looking coordinates while relying on non-grounded cues.

    Authors: We agree that the GRPO-GR reward consists only of final-answer accuracy and format compliance (including box token presence) with no explicit penalty for irrelevant boxes. This choice was deliberate to remove the need for bounding-box or chain annotations. Our empirical results, including qualitative visualizations and task performance gains, indicate that the model learns to emit relevant boxes that support correct answers rather than spurious ones. To strengthen the presentation, we will revise the abstract for greater precision on the reward design and add a dedicated analysis subsection with box-relevance metrics and ablations. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation is self-contained via independent RL rewards and empirical results.

full rationale

The paper's core chain introduces a grounded reasoning output format and applies GRPO-GR rewards defined solely on final-answer accuracy plus format compliance (including box token presence). These rewards are specified externally to any fitted parameters or prior self-citations, and the data-efficiency claim with 20 examples is presented as an observed training outcome rather than a quantity derived by construction from the inputs. No load-bearing step reduces to self-definition, fitted-input renaming, or an unverified self-citation chain; the unification of reasoning and grounding is therefore an empirical consequence of the stated training procedure.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that format-plus-accuracy rewards in RL can induce correct visual grounding without supervision. No free parameters or invented entities are explicitly introduced in the abstract.

axioms (1)
  • domain assumption Reinforcement learning rewards on answer accuracy and output format can train models to generate correct bounding boxes without explicit labels
    This assumption is required for the data-efficiency claim to hold without annotated reasoning or box data.

pith-pipeline@v0.9.0 · 5803 in / 1243 out tokens · 47718 ms · 2026-05-22T13:26:01.201192+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CaMo: Camera Motion Grounded Evaluation and Training for Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0

    Proposes Spatial Narrative Score (SNS) evaluation for VLMs' camera motion understanding and introduces CaMo model achieving consistent performance on SNS and direct QA.

  2. Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs

    cs.CV 2026-04 unverdicted novelty 7.0

    Mind's Eye benchmark shows top multimodal LLMs score below 50% on visual abstraction, relation, and transformation tasks while humans reach 80%.

  3. Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Space

    cs.CV 2025-12 unverdicted novelty 7.0

    DMLR performs dynamic visual-textual interleaving in latent space using confidence-guided latent policy gradient optimization and a dynamic visual injection strategy, yielding improved multimodal reasoning on benchmarks.

  4. DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning

    cs.AI 2025-09 unverdicted novelty 7.0

    DeFacto trains multimodal models using counterfactual image variants and reinforcement learning rewards to improve both answer accuracy and evidence-answer consistency.

  5. Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming

    cs.CV 2026-05 unverdicted novelty 6.0

    A structured Zoom-then-Diagnose paradigm with uncertainty-aware GRPO rewards improves lesion localization by 39.3% on liver, breast, and thyroid ultrasound VQA datasets by encouraging caution under ambiguity.

  6. Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs

    cs.CV 2026-04 unverdicted novelty 6.0

    Perception Programs rewrite dense visual tool outputs into language-native summaries, boosting MLLM accuracy by 15-45% absolute on BLINK perception tasks and setting new state-of-the-art results.

  7. Scientific Graphics Program Synthesis via Dual Self-Consistency Reinforcement Learning

    cs.CV 2026-04 unverdicted novelty 6.0

    SciTikZer-8B uses a new dataset, benchmark, and dual self-consistency RL to generate TikZ code for scientific graphics, outperforming much larger models like Gemini-2.5-Pro.

  8. Discovering Failure Modes in Vision-Language Models using RL

    cs.CV 2026-04 unverdicted novelty 6.0

    An RL-based questioner agent adaptively generates queries to discover novel failure modes in VLMs without human intervention.

  9. CharTool: Tool-Integrated Visual Reasoning for Chart Understanding

    cs.AI 2026-04 unverdicted novelty 6.0

    CharTool equips MLLMs with cropping and code tools plus agentic RL on DuoChart data to raise chart-reasoning accuracy by up to 9.78 percent on benchmarks.

  10. Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents

    cs.AI 2025-12 unverdicted novelty 6.0

    Argos is an agentic verifier that adaptively picks scoring functions to evaluate accuracy, localization, and reasoning quality, enabling stronger multimodal RL training for AI agents.

  11. DeepEyesV2: Toward Agentic Multimodal Model

    cs.CV 2025-11 unverdicted novelty 6.0

    DeepEyesV2 uses a two-stage cold-start plus reinforcement learning pipeline to produce an agentic multimodal model that adaptively invokes tools and outperforms direct RL on real-world reasoning benchmarks.

  12. Locate-Then-Examine: Grounded Region Reasoning Improves Detection of AI-Generated Images

    cs.CV 2025-10 unverdicted novelty 6.0

    Locate-Then-Examine improves AI-generated image detection by localizing suspicious regions first then performing region-aware re-examination, while releasing the TRACE dataset of 20k annotated images.

  13. DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning

    cs.AI 2025-09 unverdicted novelty 6.0

    DeFacto trains multimodal models with counterfactual image variants and GRPO reinforcement learning to enforce that correct answers are supported by correct visual evidence.

  14. The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

    cs.AI 2025-09 accept novelty 6.0

    Survey that defines agentic RL for LLMs via POMDPs, introduces a taxonomy of planning/tool-use/memory/reasoning capabilities and domains, and compiles open environments from over 500 papers.

  15. Test-time Scaling over Perception: Resolving the Grounding Paradox in Thinking with Images

    cs.CV 2026-04 unverdicted novelty 5.0

    TTSP resolves the Grounding Paradox by treating perception as a scalable test-time process that generates, filters, and iteratively refines multiple visual exploration traces, outperforming baselines on high-resolutio...

  16. Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning

    cs.AI 2025-09 unverdicted novelty 5.0

    MoVT unifies different visual reasoning modes in a single model and uses the AdaVaR two-stage framework with supervised cold-start and RL via AdaGRPO to enable context-adaptive mode selection, yielding consistent gain...

  17. A Survey of Reinforcement Learning for Large Reasoning Models

    cs.CL 2025-09 accept novelty 3.0

    A survey compiling RL methods, challenges, data resources, and applications for enhancing reasoning in large language models and large reasoning models since DeepSeek-R1.

Reference graph

Works this paper leans on

34 extracted references · 34 canonical work pages · cited by 16 Pith papers · 17 internal anchors

  1. [1]

    Introducing openai o1-preview

    OpenAI. Introducing openai o1-preview. https://openai.com/index/ introducing-openai-o1-preview/ , 2024

    work page 2024

  2. [2]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  3. [3]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    work page 2025

  5. [5]

    The Llama 3 Herd of Models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    work page 2022

  7. [7]

    Reasoning models don’t always say what they think

    Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato Carson Denison, John Schul- man, Arushi Somani, Peter Hase, Misha Wagner Fabien Roger Vlad Mikulik, Sam Bowman, Jan Leike Jared Kaplan, et al. Reasoning models don’t always say what they think. Anthropic Research, 2025

    work page 2025

  8. [8]

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    Training Large Language Models to Reason in a Continuous Latent Space

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    Visual spatial reasoning

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 2023. 10

    work page 2023

  15. [15]

    Tallyqa: Answering complex counting questions

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8076–8084, 2019

    work page 2019

  16. [16]

    R1-v: Reinforcing super generaliza- tion ability in vision-language models with less than $3.https://github.com/Deep-Agent/ R1-V, 2025

    Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generaliza- tion ability in vision-language models with less than $3.https://github.com/Deep-Agent/ R1-V, 2025. Accessed: 2025-02-02

    work page 2025

  17. [17]

    Introducing openai o3 and o4-mini

    OpenAI. Introducing openai o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/ , 2025

    work page 2025

  18. [18]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022

    work page 2022

  19. [19]

    Multimodal Chain-of-Thought Reasoning in Language Models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  20. [20]

    Visual chain-of-thought prompting for knowledge-based visual reasoning

    Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Zhiqing Sun, Dan Gutfreund, and Chuang Gan. Visual chain-of-thought prompting for knowledge-based visual reasoning. In AAAI Conference on Artificial Intelligence, 2024

    work page 2024

  21. [21]

    Compositional chain-of- thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of- thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420–14431, 2024

    work page 2024

  22. [22]

    Unsupervised visual chain-of-thought reasoning via preference optimization

    Kesen Zhao, Beier Zhu, Qianru Sun, and Hanwang Zhang. Unsupervised visual chain-of-thought reasoning via preference optimization. arXiv preprint arXiv:2504.18397, 2025

    work page arXiv 2025

  23. [23]

    Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Systems, 37:8612–8642, 2024

    work page 2024

  24. [24]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  25. [25]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    work page 2019

  26. [26]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    work page 2024

  27. [27]

    MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  28. [28]

    How to evaluate the generalization of detection? a benchmark for comprehensive open-vocabulary detection

    Yiyang Yao, Peng Liu, Tiancheng Zhao, Qianqian Zhang, Jiajia Liao, Chunxin Fang, Kyusong Lee, and Qing Wang. How to evaluate the generalization of detection? a benchmark for comprehensive open-vocabulary detection. arXiv preprint arXiv:2308.13177, 2023

    work page arXiv 2023

  29. [29]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    work page 2023

  30. [30]

    Language Models are Few-Shot Learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  31. [31]

    Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023. 11

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  32. [32]

    Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  33. [33]

    OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Open- vlthinker: An early exploration to complex vision-language reasoning via iterative self- improvement. arXiv preprint arXiv:2503.17352, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [34]

    SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models

    Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training r1-like reasoning large vision-language models. arXiv preprint arXiv:2504.11468, 2025. 12 Table 2: Statistics for the testing data used in the experiments. We collect the testing data from seven diverse benchmarks...

    work page internal anchor Pith review Pith/arXiv arXiv 2025