Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RL training on critic data alone turns a vision-language model into a strong policy model, the paper reports.

desk verdict Strong empirical paper on critic-data RL as policy training; the missing decontamination check is the one thing to fix before trusting the headline numbers. read the letter →

arxiv 2509.00676 v1 pith:H43HPSP6 submitted 2025-08-31 cs.CV cs.LG

Xiyao Wang , Chunyuan Li , Jianwei Yang , Kai Zhang , Bo Liu , Tianyi Xiong , Furong Huang This is my paper
classification cs.CVcs.LG
keywords vision-languagemodelscriticpolicyreinforcementlearningpreferenceoptimizationtest-timescalingself-critiqueGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper challenges the standard split between critic models, which evaluate responses, and policy models, which generate them. The authors claim that if preference-labeled critic data is reorganized into a verifiable 'choose the better response' task and used for reinforcement learning on a base vision-language model, the resulting model is both a top-tier critic and a competitive policy: it outscores its base model by 5.7% on average across 26 visual reasoning and understanding benchmarks, and it matches or beats reasoning-specialized VLMs that were trained on in-domain policy data. The same model, used as its own judge at test time through recursive pairwise comparison of 128 samples, gains another 13.8% on average on five representative reasoning tasks with no further training. If this holds, critic data—which only requires pairwise preference labels rather than gold answers—becomes a dual-purpose training signal for stronger generation and self-improvement.

What carries the argument

The load-bearing mechanism is the reformulation of preference data into verifiable RL: each of roughly 40k critic examples is reduced to an image, a question, and two candidate responses, with the instruction to output the better response index or 'Two responses are equally good.' The reward is r = 0.9 times the preference reward plus 0.1 times the format reward, where the preference reward is +1 for matching the ground-truth label and 0 otherwise, and the format reward is +1 for enclosing reasoning in <think>...</think> and boxing the final answer. GRPO optimizes this reward on the base model with no SFT stage. For test-time scaling, the same model generates n=128 candidate answers and then

What would settle it

Perform an exact and near-duplicate match between the images and questions in the 40k critic training set and those in the 26 evaluation benchmarks (plus the five self-critique tasks). If a substantial share of benchmark items are found in training—or if evaluating on a newly released, post-cutoff benchmark erases the +5.7% average gain—the central claim of generalizable dual capability fails. A milder but informative check: ablate by shuffling preference labels; if gains persist, the reward signal itself is not what drives policy improvement.

Watch

Extended reading notes

Core claim

The paper's central claim is that reinforcement learning on critic data does not just make a better judge; it transfers into general generation ability. Starting from Qwen-2.5-VL-7B and using roughly 40k pairwise preference instances (with GPT-written rationales stripped out), the authors train LLaVA-Critic-R1 by rewarding exact preference matches and correct use of a think-then-answer format. The resulting model outperforms its base by +5.7% on average across 26 benchmarks spanning perception, image reasoning, charts, video, and GUI-agent tasks, and it simultaneously reaches top scores on visual reward benchmarks. Applying the same recipe to already-strong reasoning models yields LLaVA-Crit

Load-bearing premise

The paper's core comparison treats the 40k critic training examples and the 26 evaluation benchmarks as separate worlds, but it reports no contamination check; if benchmark images or questions appear in the training pairs, the reported policy and self-critique gains could come from memorization rather than transferable capability.

Editorial extensions

If this is right

  • The same 7B checkpoint can generate and evaluate, so systems can drop separate reward models during inference and use one model for both roles.
  • The recipe turns preference-pair data—labels of relative quality rather than a single ground-truth answer—into a direct policy-improvement signal, matching or beating several in-domain policy-trained reasoning baselines.
  • Starting from a strong reasoning model and then applying critic RL (policy-then-critic) is the best ordering tested: it improves policy and restores the critic ability that policy training erodes.
  • Self-critique scales better than majority voting: Best-of-128 with self-critic improves five reasoning tasks by +13.8% on average and keeps improving past the point where majority voting plateaus.
  • The recipe generalizes across base architectures: Qwen-2.5-VL, ThinkLite-VL, MiMo-VL, and Llama-3.2-Vision all show policy gains, with the strongest reaching 71.9 MMMU at 7B scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the 40k training set is built from preference pairs across a range of VLM-generated responses, the policy gains may come less from 'critic knowledge' than from exposure to diverse, high-variance candidate answers; a direct test would be to re-run the recipe on random pairs with shuffled labels to see if answer diversity alone drives improvement.
  • The recursive pairwise tournament used for self-critique could be replaced by the model scoring all candidates at once; if direct scoring works, test-time scaling cost could drop substantially.
  • The dual-role result suggests a data flywheel: a single model can sample multiple answers, judge them itself, and use its own preference judgments as training signal—an explicit route to self-improvement the paper points to but does not fully close.
  • If contamination is ruled out, the recipe should transfer to text-only LLMs (where preference data is even more abundant) and to larger VLMs; the paper only tests 7B and 11B scales.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes to reorganize pairwise preference-labeled critic data into a verifiable RL task (stripping GPT rationales and using only image, question, two responses, and preference label) and to train a multimodal critic with GRPO. Starting from Qwen-2.5-VL-7B, the resulting LLaVA-Critic-R1 is reported to be both a strong critic on Visual RewardBench and a competitive policy model, averaging +5.7% over its base across 26 visual reasoning/understanding benchmarks. Applying the same procedure to ThinkLite-VL gives LLaVA-Critic-R1+, and to MiMo-VL gives a 7B-scale MMMU score of 71.9. The paper also reports test-time self-critique scaling (Best-of-128) with an average +13.8% gain on five benchmarks, plus ablations on format reward vs. critic reward and on combining critic and policy data.

Significance. If the empirical findings hold, this is a surprising and practically important result: it challenges the standard separation between critic and policy models and suggests that pairwise preference data can serve as a general reasoning curriculum for VLMs, simultaneously improving judgment and generation. The evaluation is broad (26 benchmarks, 6 dimensions), the ablations are well structured, and the paper provides code and model releases. The multi-base-model experiments (Qwen, ThinkLite, MiMo, Llama-3.2) strengthen the generality claim. However, the central empirical claim is currently exposed to two major threats: potential training/evaluation contamination and lack of any variance/significance analysis. Both are addressable and should be fixed before the claims can be accepted at face value.

major comments (3)
  1. [Section 2.1 vs. Section 3.1] The 40K critic training pairs are assembled from VLFeedback, RLHF, and RLHF-V, while evaluation is performed on 26 benchmarks including MMMU, MathVista, ChartQA, MMStar, etc. The manuscript does not report any decontamination analysis between the training pairs and these evaluation benchmarks. Because each training instance is an (image, question, two responses, preference) tuple, any overlap in images/questions would let GRPO memorize correct preference/answer patterns and could substantially inflate the reported +5.7% policy gain and the Table 4 self-critique gains. The Table 5 ablation (format-only RL) and the multi-base experiments in Appendix .1/.2 do not control for this overlap. This is the single most load-bearing threat to the paper's central claim. Please provide per-benchmark overlap statistics at the image and question-text level, and if feasible retrain on a decontaminated s
  2. [Tables 3-10] All results are single runs, reported without variance, confidence intervals, or significance tests. Many of the reported improvements are small in absolute terms (e.g., Table 3: MMBench +0.2, AI2D +0.6, ChartQA +1.4; Table 9: several entries are negative or within ±1 point). The headline +5.7% average is an aggregate over 26 benchmarks, and without repeated seeds or bootstrap intervals it is difficult to determine whether the pattern is systematic or partially noise. I request at least three independent training/evaluation seeds for the main comparison, or bootstrap confidence intervals over benchmark instances, to support the claims of 'consistent' and 'significant' improvement.
  3. [Section 3.2, Table 4] The self-critique test-time scaling comparison uses majority voting and the base model as a critic as baselines. The base model is not trained to judge pairs, so it is a weak critic baseline; the observed gap between self-critic and base-model-critic is therefore partly expected. The recursive pairwise tournament also uses random tie-breaking, but tie rates and tournament variance are not reported. To substantiate the claim that the learned critic ability transfers to test-time scaling, please add a stronger external critic baseline (e.g., SFT-trained LLaVA-Critic or a separately trained reward model) and report tie statistics or multiple tournament seeds.
minor comments (6)
  1. [Section 4.4] Title: 'On the correction between Critic and Policy Ability' should read 'correlation'.
  2. [Table 3] The column headers and arrow notation are confusing: it is not immediately clear which base model each '↑' refers to for LLaVA-Critic-R1 vs. LLaVA-Critic-R1+. Please label the reference base in the table caption or column header.
  3. [Section 3.1.2] The text claims an 'average performance gain of 7.25%' on image reasoning. From Table 3, the average absolute gain is +3.7 on a 42.1 baseline, which is about 8.8% relative. Please reconcile the number.
  4. [Table 9 and Appendix .1] The text reports MMMU as 71.8 while the table and abstract state 71.9. Please correct the inconsistency.
  5. [Tables 1 and 2] The answer parsing is not fully specified. The prompt says to answer with a response number or 'Two responses are equally good.'; the reward computation requires exact match, but the paper does not state how outputs such as 'Response 1' vs. '1' or boxed answers are parsed. Please describe the parsing rule or provide the post-processing code.
  6. [Appendix headings] The appendix section headings appear as '.1', '.2', '.3'; these should be labeled A.1, A.2, A.3.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the policy and self-critic claims are empirical evaluations against external benchmarks, not reductions to training labels or self-citations.

full rationale

The derivation chain is: 40k preference-labeled critic instances are stripped of GPT rationales, reformulated as a verifiable pairwise judgment task (Section 2.1), and optimized with GRPO under r = 0.9*r_pref + 0.1*r_format (Eq. 1, Section 2.2). The reported policy gains (+5.7% average over base across 26 benchmarks, Table 3) and test-time self-critic gains (+13.8%, Table 4) are measured on external benchmarks with ground-truth answers; no parameter is fitted to those benchmark outcomes, and no equation defines the predicted benchmark score in terms of the training preference labels. The explanatory hypotheses in Section 4.1 are post-hoc but are tested with controlled ablations (thinking-only prompting and format-reward-only RL, Table 5), so they are not circular. The only self-citations (LLaVA-Critic dataset, ThinkLite-VL baseline, LLaVA-Critic-113K) are used as data/model provenance, not as load-bearing mathematical or uniqueness claims. The absence of a reported decontamination analysis between the 40k training pairs and the 26 evaluation benchmarks is a validity/memorization concern, not a circularity: even if contamination existed, it would inflate results empirically rather than make the derivation equivalent to its inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central result rests on the reliability of preference labels, the absence of train/test contamination, and the validity of GRPO as the optimizer. No free parameters are fitted to the target benchmarks; the listed hyperparameters are training choices. No new entities are postulated.

free parameters (3)
  • alpha (reward weighting) = 0.9
    Balances preference reward and format reward in Eq. 1. Chosen by hand, not fitted to benchmarks, but affects the training signal.
  • Sampling temperature for test-time scaling = 0.9
    Used in self-critique generation to ensure diversity across the 128 candidates; a manual choice.
  • Number of candidate responses N for self-critique = 128
    Selected based on the test-time scaling curve saturation observed in Figure 2; not fitted to maximize the specific result.
assumptions (4)
  • domain assumption The preference labels in the 40k critic training data are treated as ground truth
    Section 2.1 assumes human/GPT preference labels are correct for reward computation, without manual verification.
  • domain assumption The 26 evaluation benchmarks are free of contamination with the training data
    No decontamination analysis is provided; if benchmark questions or images appear in training pairs, the reported gains would be inflated.
  • standard math GRPO optimizes the verifiable reward effectively for multimodal models
    The paper relies on GRPO as the RL objective (Section 2.2) and demonstrates empirical convergence, but assumes it behaves as expected in this setting.
  • domain assumption Applying the thinking template at inference is a fair intervention across all compared models
    The paper prompts all models with the thinking template in evaluations (Section 3.1.2), assuming this does not disadvantage any baseline relative to critic-trained models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model." pith.science (2026). https://pith.science/paper/H43HPSP6

@misc{pith2026250900676,
  author       = {Pith},
  title        = {Pith review of: LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H43HPSP6}},
  note         = {Machine review of arXiv:2509.00676}
}
read the original abstract

In vision-language modeling, critic models are typically trained to evaluate outputs -- assigning scalar scores or pairwise preferences -- rather than to generate responses. This separation from policy models, which produce the responses, is so entrenched that critics are rarely considered for direct policy use. In this work, we challenge this convention. We propose to reorganize preference-labeled critic datasets into verifiable training signals and perform reinforcement learning directly on a base generative model, producing LLaVA-Critic-R1, a multimodal critic trained to optimize preference judgments while retaining full generation ability. Surprisingly, LLaVA-Critic-R1 emerges not only as a top-performing critic but also as a competitive policy model -- matching or surpassing specialized reasoning VLMs trained with in-domain data across 26 visual reasoning and understanding benchmarks, with an average gain of +5.7% over its base model (Qwen-2.5-VL-7B). Extending this approach to existing strong reasoning VLMs yields LLaVA-Critic-R1+, which further advances policy performance without sacrificing critic quality, achieving a SoTA performance of 71.9 on MMMU at the 7B scale. Finally, we show that the enhanced critic ability benefits inference: applying self-critique at test time yields an average +13.8% improvement on five representative reasoning tasks without additional training. Our results reveal that RL training on critic data can produce a unified model excelling at both evaluation and generation, offering a simple path toward scalable, self-improving multimodal systems.

Figures

Figures reproduced from arXiv: 2509.00676 by the authors.

Figure 1
Figure 1. LLaVA-Critic-R1 is trained on top of the base model Qwen-2.5-VL-7B. Building upon a stronger reasoning VLM, ThinkLite-VL-7B, we further develop LLaVA-Critic-R1+ by applying the same RL critic training procedure. Left: Performance comparison of LLaVA-Critic-R1 with other base and reasoning VLMs on multiple visual reasoning, visual understanding, and visual reward benchmarks. LLaVA-Critic-R1 not only significantly out… view at source ↗
Figure 2
Figure 2. Test-time scaling curve of self-critic and majority vote on 5 visual reasoning and understanding benchmarks with LLaVA-Critic-R1. Self-critic yields significantly greater performance gains than majority vote, and reaches performance plateau at a later sampling scale. Furthermore, despite the clear improvements brought by self-critic scaling, there remains a substantial gap between its performance and the ground trut… view at source ↗
Figure 3
Figure 3. Correlation between LLaVA-Critic-R1’s critic and policy performance. This figure plots the performance of LLaVA-Critic-R1’s two key capabilities as a function of critic training steps. The blue curve represents the policy performance on general visual tasks, while the red curve denotes the critic performance on visual reward benchmarks. The strong positive correlation between the two performance metrics throughout t… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A token-level correctness classifier trained with LoRA and then merged into the model boosts out-of-distribution factuality in summarization and translation.

  2. SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    SIVA-RL uses the observed reward drop between clean and locally edited images to route training toward sensitivity or invariance, improving GRPO/DAPO-based multimodal RL across nine benchmarks.

  3. High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models

    cs.CV 2025-12 unverdicted novelty 6.0 of 10

    High-entropy tokens act as concentrated multimodal failure points in VLMs, enabling sparse Entropy-Guided Attacks that achieve 93-95% success and 30-38% harmful rates with cross-model transfer.

Reference graph

Works this paper leans on

70 extracted references · 9 canonical work pages · cited by 3 Pith papers

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  2. [2]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. Computing Research Repository, arXiv:2407.21787, 2024. URL https://arxiv.org/abs/2407.21787

    arXiv 2024

  3. [3]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    arXiv 2024

  4. [4]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv 2021

  5. [5]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025

    Google Deepmind. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https://arxiv.org/abs/2507.06261

    arXiv 2025

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  7. [7]

    Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement, 2025

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement, 2025. URL https://arxiv.org/abs/2503.17352

    arXiv 2025

  8. [8]

    Smith, Wei-Chiu Ma, and Ranjay Krishna

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive, 2024. URL https://arxiv.org/abs/2404.12390

    arXiv 2024

Show all 70 references
  1. [9]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835--10866. PMLR, 2023

    2023

  2. [10]

    Interpretable contrastive monte carlo tree search reasoning

    Zitian Gao, Boye Niu, Xuzheng He, Haotian Xu, Hongzhang Liu, Aiwei Liu, Xuming Hu, and Lijie Wen. Interpretable contrastive monte carlo tree search reasoning. arXiv preprint arXiv:2410.01707, 2024

    2024 arXiv

  3. [11]

    Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models, 2024

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision...

    2024 arXiv

  4. [12]

    Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444, 2025

    2025 arXiv

  5. [13]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826, 2025

    2025 arXiv

  6. [15]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models, 2025 b

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models, 2025 b . URL https://arxiv.org/abs/2503.06749

    2025 arXiv

  7. [16]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  8. [17]

    A diagram is worth a dozen images, 2016

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images, 2016

    2016

  9. [18]

    Process reward models that think

    Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang. Process reward models that think. arXiv preprint arXiv:2504.16828, 2025

    2025

  10. [19]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  11. [20]

    Vl-rewardbench: A challenging benchmark for vision-language generative reward models, 2025

    Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, Lingpeng Kong, and Qi Liu. Vl-rewardbench: A challenging benchmark for vision-language generative reward models, 2025. URL https://arxiv.org/abs/2411.17451

    2025 arXiv

  12. [21]

    Let’s verify step by step

    Hunter Lightman, Karl Cobbe, Vineet Kosaraju, Yura Burda, Harri Edwards, Jan Leike, and Ilya Sutskever. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. URL https://arxiv.org/abs/2305.20050

    2023 arXiv

  13. [22]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/fo...

    2024

  14. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36: 0 34892--34916, 2023

    2023

  15. [24]

    Noisyrollout: Reinforcing visual reasoning with data augmentation

    Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation. arXiv preprint arXiv:2504.13055, 2025

    2025

  16. [25]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233. Springer, 2024 a

    2024

  17. [26]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67 0 (12): 0 220102, 2024 b

    2024

  18. [27]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representat...

    2024

  19. [28]

    Generative reward models

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fr \"a nken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410.12832, 2024

    2024 arXiv

  20. [29]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  21. [30]

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based rei...

    2025 arXiv

  22. [31]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2023. URL https://api.semanticscholar.org/CorpusID:263218031

    2023

  23. [32]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/learning-to-reason-with-llms/

    2024

  24. [33]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  25. [34]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025. URL https://arxiv.org...

    2025 arXiv

  26. [35]

    Verifierq: Enhancing llm test time compute with q-learning-based verifiers

    Jianing Qi, Hao Tang, and Zhigang Zhu. Verifierq: Enhancing llm test time compute with q-learning-based verifiers. arXiv preprint arXiv:2410.08048, 2024

    2024 arXiv

  27. [36]

    Vision language models are blind

    Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision, pages 18--34, 2024

    2024

  28. [37]

    Atkinson, Aaditya Baranwal, Alexandru Coca, Mikah Dang, Sebastian Dziadzio, Jakob D

    Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, Ioana Croitoru, Simion-Vlad Bogolin, Jialu Tang, Florian Langer, Vyas Raina, Vatsal Raina, Hanyi Xiong, Vishaal Udandarao, Jingyi Lu, Shiyang Chen, Sam Purkis, Tianshuo Yan, Wenye Lin, Gyungin S...

    2025 arXiv

  29. [38]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  30. [39]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  31. [40]

    Aligning large multimodal models with factually augmented rlhf, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf, 2023. URL https://arxiv.org/abs/2309.14525

    2023 arXiv

  32. [41]

    Mimo-vl technical report, 2025 a

    Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang, Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang, Zihan Jiang, Zhixian Zheng, Zhichao S...

    2025 arXiv

  33. [42]

    Glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025 b

    V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Boyan Shi, Changyu Pang, Chenhui Zhang,...

    2025 arXiv

  34. [43]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024. URL https://arxiv.org/abs/2401.06209

    2024 arXiv

  35. [44]

    Srpo: Enhancing multimodal llm reasoning via reflection-aware reinforcement learning

    Zhongwei Wan, Zhihao Dou, Che Liu, Yu Zhang, Dongfei Cui, Qinjian Zhao, Hui Shen, Jing Xiong, Yi Xin, Yifan Jiang, et al. Srpo: Enhancing multimodal llm reasoning via reflection-aware reinforcement learning. arXiv preprint arXiv:2506.01713, 2025

    2025

  36. [45]

    Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning

    Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837, 2025 a

    2025 arXiv

  37. [46]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 202...

    2024

  38. [48]

    Visualprm: An effective process reward model for multimodal reasoning

    Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503.10291, 2025 c

    2025 arXiv

  39. [49]

    Enhancing visual-language modality alignment in large vision language models via self-improvement

    Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, et al. Enhancing visual-language modality alignment in large vision language models via self-improvement. arXiv preprint arXiv:2405.15973, 2024 b

    2024 arXiv

  40. [50]

    Scaling inference-time search with vision value model for improved visual comprehension

    Xiyao Wang, Zhengyuan Yang, Linjie Li, Hongjin Lu, Yuancheng Xu, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Scaling inference-time search with vision value model for improved visual comprehension. arXiv preprint arXiv:2412.03704, 2024 c

    2024 arXiv

  41. [51]

    Vicrit: A verifiable reinforcement learning proxy task for visual perception in vlms

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Yongyuan Liang, Yuhang Zhou, Xiaoyu Liu, Ziyi Zang, Ming Li, Chung-Ching Lin, Kevin Lin, et al. Vicrit: A verifiable reinforcement learning proxy task for visual perception in vlms. arXiv preprint arXiv:2506.10128, 2025 d

    2025 arXiv

  42. [52]

    Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement. arXiv preprint arXiv:2504.07934, 2025 e

    2025 arXiv

  43. [53]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  44. [54]

    Open vision reasoner: Transferring linguistic cognitive behavior for visual reasoning

    Yana Wei, Liang Zhao, Jianjian Sun, Kangheng Lin, Jisheng Yin, Jingcheng Hu, Yinmin Zhang, En Yu, Haoran Lv, Zejia Weng, et al. Open vision reasoner: Transferring linguistic cognitive behavior for visual reasoning. arXiv preprint arXiv:2507.05255, 2025

    2025

  45. [55]

    V*: Guided visual search as a core mechanism in multimodal llms, 2023

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms, 2023. URL https://arxiv.org/abs/2312.14135

    2023 arXiv

  46. [56]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024

    2024 arXiv

  47. [57]

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy Lillicrap, Kenji Kawaguchi, and Michael Q. Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451, 2024. URL https://arxiv.org/abs/2405.00451

    2024 arXiv

  48. [58]

    Llava-critic: Learning to evaluate multimodal models

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. arXiv preprint arXiv:2410.02712, 2024

    2024 arXiv

  49. [59]

    Llava-critic: Learning to evaluate multimodal models

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13618--13628, 2025

    2025

  50. [60]

    Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models, 2025

    Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, Wenhai Wang, Jifeng Dai, and Jinguo Zhu. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models, 2025. URL https://arxi...

    2025 arXiv

  51. [61]

    R1-sharevl: Incentivizing reasoning capability of multimodal large language models via share-grpo

    Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, et al. R1-sharevl: Incentivizing reasoning capability of multimodal large language models via share-grpo. arXiv preprint arXiv:2505.16673, 2025

    2025 arXiv

  52. [62]

    Vl-cogito: Progressive curriculum reinforcement learning for advanced multimodal reasoning

    Ruifeng Yuan, Chenghao Xiao, Sicong Leng, Jianyu Wang, Long Li, Weiwen Xu, Hou Pong Chan, Deli Zhao, Tingyang Xu, Zhongyu Wei, et al. Vl-cogito: Progressive curriculum reinforcement learning for advanced multimodal reasoning. arXiv preprint arXiv:2507.22607, 2025

    2025 arXiv

  53. [63]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  54. [64]

    Internlm-xcomposer2.5-reward: A simple yet effective multi-modal reward model, 2025

    Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, and Jiaqi Wang. Internlm-xcomposer2.5-reward: A simple yet effective multi-modal reward model, 2025. URL https://arxiv.org/abs/2501.12368

    2025 arXiv

  55. [65]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240, 2024 a

    2024 arXiv

  56. [66]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024 b

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024 b

    2024 arXiv

  57. [67]

    Nemotron-research-tool-n1: Exploring tool-using language models with reinforced reasoning

    Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. Nemotron-research-tool-n1: Exploring tool-using language models with reinforced reasoning. arXiv preprint arXiv:2505.00024, 2025 a

    2025 arXiv

  58. [68]

    R1-reward: Training multimodal reward model through stable reinforcement learning, 2025 b

    Yi-Fan Zhang, Xingyu Lu, Xiao Hu, Chaoyou Fu, Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaibing Chen, Kaiyu Tang, Haojie Ding, Jiankang Chen, Fan Yang, Zhang Zhang, Tingting Gao, and Liang Wang. R1-reward: Training multimodal reward model through stable reinforcement le...

    2025 arXiv

  59. [69]

    Mm-rlhf: The next step forward in multimodal llm alignment, 2025 c

    Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, Xue Wang, Yibo Hu, Bin Wen, Fan Yang, Zhang Zhang, Tingting Gao, Di Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Mm-rlhf: The next step forward in multimodal...

    2025 arXiv

  60. [70]

    Genprm: Scaling test-time compute of process reward models via generative reasoning

    Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint arXiv:2504.00891, 2025 a

    2025 arXiv

  61. [71]

    Mmvu: Measuring expert-level multi-discipline video understanding

    Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, et al. Mmvu: Measuring expert-level multi-discipline video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 847...

    2025

  62. [72]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36: 0 46595--46623, 2023

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.