Pith. sign in

REVIEW 4 major objections 5 minor 40 references

APO: Enhancing Reasoning Ability of MLLMs via Asymmetric Policy Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read View-R1-3B, trained with asymmetric policy optimization, lifts MLLM reasoning by 7% without sacrificing general tasks.

desk verdict A plausible GRPO refinement for MLLMs with two new heuristics, but the headline reasoning gains are not trustworthy until train/eval overlap is ruled out and the STCR formula is fixed. read the letter →

arxiv 2506.21655 v1 pith:BEPVCZ7L submitted 2025-06-26 cs.LG cs.AIcs.CV

Minjie Hong , Zirun Guo , Yan Xia , Zehan Wang , Ziang Zhang , Tao Jin , Zhou Zhao This is my paper
classification cs.LGcs.AIcs.CV
keywords multimodallargelanguagemodelsreinforcementlearningwithverifiablerewardsgrouprelativepolicyoptimizationKLdivergenceshapinglengthbiasoverthinkingreasoningGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that two standard failure modes of reinforcement learning for multimodal LLMs—catastrophic forgetting of general skills and overlong, repetitive reasoning chains—can be fixed by treating positive and negative training samples asymmetrically. It proposes Asymmetric Policy Optimization (APO), combining Difficulty-Adaptive Divergence Shaping (DADS) for correct samples and Suboptimal Trajectory Complexity Regularization (STCR) for incorrect ones. Applied to Qwen2.5-VL-3B, the resulting View-R1-3B is reported to gain on average 7% over the base model on reasoning benchmarks and to outperform 7-11B MLLMs, while still improving on general multimodal benchmarks. A sympathetic reader would take the central claim to be that adaptive KL shaping plus length regularization on negative trajectories yields both stronger reasoning and preserved generalization.

What carries the argument

The load-bearing object is the asymmetric treatment of sampled responses inside the GRPO objective. For positive samples, DADS replaces the fixed KL coefficient $\beta$ with $\beta_i = f(d)\beta$ when the group difficulty $d$ (the fraction of incorrect responses in the group) is nonzero, using $f(d)=1-e^{e(d-1)}$ for its low initial slope and fast terminal decay. For negative samples, STCR scales the token-level advantage $\hat{A}_{i,t}$ by $\alpha_i > 1$ when an incorrect response is longer than the mean correct length. The paper's core claim is carried by these two modifications to the per-token advantage of the GRPO loss.

What would settle it

Score View-R1-3B only on MathVista items whose source datasets do not appear in the training list, and compare against the base model: if the gain largely disappears or falls well below the reported 7%, the central claim of improved reasoning over memorization fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the KL penalty in GRPO should not be a single global weight. In DADS, the KL weight for a positive sample is scaled by a difficulty-adaptive function $f(d) = 1 - e^{e(d-1)}$, which maps 1 to 0, so hard correct samples are freed from the reference-model constraint while easy samples stay regularized; groups where every response is correct are filtered out. On the negative side, the paper claims that incorrect responses grow longer than correct ones because GRPO's length-normalized advantage rewards shorter wrong tokens, and it introduces STCR, which multiplies the advantage of incorrect trajectories longer than the mean correct length by $\alpha_i = 2 - \mu^{L_{\mathrm{acc}}^{\mathrm{mean}} - L_i}$, intensifying their penalty. Together these two mechanisms are reported to produce View-R1-3B's gains and to keep training stable.

Load-bearing premise

The reasoning benchmarks are not shown to be disjoint from the RL training data: the training list includes FigureQA, TabMWP, ScienceQA, and GeoQA+, which are also component sources of the MathVista evaluation set, and no deduplication is reported, so part of the reported gain could be memorization.

Editorial extensions

If this is right

  • A fixed KL weight is not necessary: per-sample, difficulty-aware KL shaping can improve both learning speed and peak reasoning accuracy while retaining general knowledge.
  • Length bias in GRPO can be corrected at the objective level, so multimodal RL training need not rely on response truncation or hard length caps to control overthinking.
  • A 3B MLLM can surpass larger open and closed reasoning models on MathVista, MathVerse, and MMK12, changing the compute-accuracy tradeoff for multimodal reasoning.
  • Reasoning-tuned MLLMs need not degrade on general benchmarks such as MMMU; the paper reports consistent improvement instead of the usual drop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same asymmetric scheme should transfer to other MLLM bases and to text-only LLM RL, since both KL-related forgetting and length bias are observed there; a direct test would run APO on a text-only 3B model against standard GRPO baselines.
  • Inference: the specific exponential form of $f(d)$ is a design choice rather than a necessity, because the paper's own comparison shows a cubic variant with similar stability; a broader sweep across benchmarks would clarify how much the shape matters.
  • Inference: the benchmark-overlap question is testable: scoring View-R1-3B separately on MathVista items whose source datasets appear in the training list would reveal whether part of the reported gain is memorization rather than reasoning transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Asymmetric Policy Optimization (APO) for reinforcement learning with verifiable rewards (RLVR) in multimodal large language models. APO combines two modifications to GRPO: Difficulty-Adaptive Divergence Shaping (DADS), which reduces the KL penalty for positive samples whose group failure rate d is high, and Suboptimal Trajectory Complexity Regularization (STCR), which increases the negative advantage of incorrect responses that are longer than the average correct response. Using Qwen2.5-VL-3B-Instruct as the base model, the authors train View-R1-3B on a filtered 20k-sample subset of ten multimodal datasets and report improvements on MathVista, MathVerse, and MMK12, with preserved or improved scores on MMStar and MMMU. The paper includes ablations, training-dynamics plots, and comparisons against larger open- and closed-source MLLMs.

Significance. If the results are reproducible and contamination-free, the paper would make a useful empirical contribution: DADS and STCR are simple, resource-efficient modifications to GRPO that address two known failure modes in multimodal RLVR, namely KL-induced under-exploration and length bias toward overlong incorrect trajectories. The training-dynamics analysis in Figure 3 and the ablation logic in Table 2 are informative, and the proposed mechanisms are mechanistically plausible. However, the manuscript currently does not support the headline claim as strongly as it appears: the primary evaluation benchmark, MathVista, overlaps with the training sources, is also used in the paper to tune the DADS projection function, and all results come from single runs without code, data, seeds, or variance estimates. These issues are fixable, so the contribution merits a revised version rather than rejection.

major comments (4)
  1. [§4.1, Table 1] The training corpus in Section 4.1 includes FigureQA, TabMWP, ScienceQA, and GeoQA+, which are also constituent sources of the MathVista evaluation set (reference [16]); the paper reports no deduplication, no exclusion of evaluation items, and no overlap statistics. Because a random 20k subset is drawn from the 90k filtered training samples, exact-match leakage into the MathVista test set cannot be ruled out, and the headline gain from 57.8 to 66.2 in Table 1 could be inflated by memorization rather than improved reasoning. Please report the overlap between the final training set and each evaluation benchmark, or retrain on a disjoint subset and re-report the headline tables.
  2. [§4.4, Figure 6] The DADS projection function f(d)=1-e^{e(d-1)} is selected by monitoring MathVista accuracy during training (Section 4.4, Figure 6), and MathVista is also the primary benchmark in the headline results and ablations. This makes MathVista a tuning set rather than a held-out test for the paper's central design choice, so the reported gains over KL-Original, KL-Linear, and KL-Cubic partly reflect selection on the evaluation metric. Please disclose the selection procedure explicitly and report primary results on benchmarks that were not used for any design decisions (for example, MathVerse, MMK12, or a held-out split of MathVista); the paper should also state whether any other hyperparameters (for example, β, λ, or μ) were tuned on evaluation benchmarks.
  3. [§1 vs §3.3] There is a direct contradiction about the sign of the length effect: Section 1 states that "the average length of correct responses is consistently longer than that of incorrect ones throughout the training process," while Section 3.3 and Figure 3 state and plot that incorrect responses are always longer than correct ones, with a diverging gap. Because STCR's design premise is that long incorrect responses are harmful and should be penalized, the correct empirical observation must be stated consistently and supported by the actual training curves; otherwise the motivation for STCR is undermined. If the Section 1 sentence is a typo, it should be corrected.
  4. [§4.2, Tables 1-2] All reported numbers come from a single training run with no error bars, and no code, data, or random seeds are provided. In Table 2, several decisive comparisons are separated by only 0.1–0.4 points (for example, 66.1 vs 66.2 for DADS vs the full APO, and 40.0 vs 40.1 on MathVerse), so without variance estimates the marginal contributions of DADS and STCR are not statistically distinguishable. Please provide standard deviations or results from a small number of independent runs for the main configurations, and release code and seeds to make the results reproducible.
minor comments (5)
  1. [§1, Figure 1(a)] The sentence "the performance of the model with KL penalty starts to decrease while the performance with KL penalty still keeps increasing steadily" appears to contain a typo; the second instance should presumably read "without KL penalty," since Figure 1(a) shows the no-KL curve declining at later steps.
  2. [§4.2] The claimed "average 7% gain over the base model" is not defined; please specify which benchmarks are averaged, whether the average is arithmetic, and whether it covers the three reasoning benchmarks or all five benchmarks in Table 1.
  3. [Table 1] The entry "LLaV A-CoT-11B" is a typo for "LLaVA-CoT-11B".
  4. [§3.3, Eq. (6)] The phrase "The μ is a hyperparameter slightly greater than 1" is ungrammatical; more importantly, please state explicitly that α_i lies in (1, 2) for long incorrect responses, since the current description requires the reader to derive this from the formula.
  5. [§4.4] The claim that training without KL "eventually results in performance degradation" is supported only by the plotted MathVista curve, which is the same benchmark used for tuning; please confirm the pattern on at least one benchmark not used for design decisions.

Circularity Check

1 steps flagged · score 4.0 of 10

DADS projection f(d) was selected using MathVista training curves, making the headline MathVista gain partly a fitted choice; other benchmarks provide independent support.

  1. fitted input called prediction [Section 4.4 / Figure 6 (and Equation 5)]
    "We evaluate peak performance (Peak Overall Accuracy on MathVista) and training stability, measured by the Coefficient of Variation (CV) across training steps. ... Our proposed KL-DADS variant, employing the function f(d) = 1−ee(d−1), achieves the highest peak accuracy on MathVista (66.1%) among all tested variants."

    The functional form of DADS (f(d), Eq. 5) is chosen by comparing variants on MathVista accuracy during training (Figure 6). The same MathVista benchmark is then the headline result in Table 1 and the main ablation in Table 2 (66.2 vs 57.8 base), so that specific gain is a selected outcome rather than an independent prediction. This does not make the whole paper circular because MathVerse, MMK12, MMStar, and MMMU were not used for this selection and also improve; those benchmarks independently support the method.

full rationale

The paper's derivation chain is mostly self-contained: APO modifies GRPO by defining test-time difficulty d from group accuracy (Eq. 4), a KL scaling f(d) (Eq. 5), and an STCR advantage scaling alpha_i (Eqs. 6-7); none of these are defined in terms of the reported benchmark scores. The only load-bearing selection step is the choice of f(d): Section 4.4 explicitly selects the exponential DADS variant as the one with the highest peak MathVista accuracy, and MathVista is then the primary benchmark in the abstract and Table 1. That makes the MathVista improvement partly a fitted claim, but the same trained model also improves on MathVerse, MMK12, MMStar, and MMMU, which were not used in the f(d) selection, giving the central claim independent empirical content. The Observe-R1 citation for difficulty-graded dataset construction is a self-citation by overlapping authors, but it is not load-bearing: the APO objective and the STCR length-bias argument do not depend on that citation. The possible overlap between the training datasets listed in Section 4.1 and component sources of MathVista is a data-leakage/correctness concern, not an equation-level circularity; with no deduplication reported it is a serious validity risk, but it is outside the definitional/fitted-prediction circularity scored here. Overall: one partially fitted benchmark result, with independent benchmarks, so score 4 rather than 0 or 6+.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard GRPO/RLVR, a difficulty proxy, a length-bias hypothesis, and an implicit assumption of no train/eval contamination. No new physical or ontological entities are introduced.

free parameters (4)
  • beta = not reported
    Base KL penalty weight in Eq. 3; set by convention or prior work, not reported in the paper.
  • lambda = 0.5
    Reward trade-off coefficient in Eq. 1, set following MM-Eureka.
  • mu = 1.0001
    STCR scaling base in Eq. 6; chosen slightly above 1, but this makes the penalty nearly identity for observed response lengths.
  • f(d) = 1 - e^(e(d-1)) = selected form
    Projection for DADS; selected by comparing variants on MathVista accuracy (Section 4.4), effectively fitted to the primary benchmark.
assumptions (5)
  • standard math GRPO with verifiable rewards is a valid training objective for MLLMs.
    Used as the foundation (Eq. 3), citing DeepSeekMath and DeepSeek-R1; the paper modifies only the advantage and KL weight.
  • domain assumption Answer-match and format rewards (Eq. 1) adequately measure reasoning quality.
    All rewards are exact-match based; no reward for reasoning intermediate steps.
  • ad hoc to paper Group failure rate d (Eq. 4) is a reliable test-time difficulty measure.
    This proxy drives the DADS KL weighting; its noise with G=8 samples is not analyzed.
  • ad hoc to paper Long incorrect responses are harmful and penalizing them improves learning.
    Motivated by observed length trends (Fig. 3), but the causal claim is not isolated in an experiment.
  • domain assumption Evaluation benchmarks are disjoint from RL training data.
    Training set sources overlap with MathVista; no deduplication is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of APO: Enhancing Reasoning Ability of MLLMs via Asymmetric Policy Optimization." pith.science (2026). https://pith.science/paper/BEPVCZ7L

@misc{pith2026250621655,
  author       = {Pith},
  title        = {Pith review of: APO: Enhancing Reasoning Ability of MLLMs via Asymmetric Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BEPVCZ7L}},
  note         = {Machine review of arXiv:2506.21655}
}
read the original abstract

Multimodal Large Language Models (MLLMs) are powerful at integrating diverse data, but they often struggle with complex reasoning. While Reinforcement learning (RL) can boost reasoning in LLMs, applying it to MLLMs is tricky. Common issues include a drop in performance on general tasks and the generation of overly detailed or "overthinking" reasoning. Our work investigates how the KL penalty and overthinking affect RL training in MLLMs. We propose Asymmetric Policy Optimization (APO) to address these issues, which divides the sampled responses into positive and negative groups. For positive samples, Difficulty-Adaptive Divergence Shaping (DADS) is introduced to dynamically adjust the KL divergence weight based on their difficulty. This method prevents policy entropy from dropping sharply, improves training stability, utilizes samples better, and preserves the model's existing knowledge. For negative samples, Suboptimal Trajectory Complexity Regularization (STCR) is proposed to penalize overly long responses. This helps mitigate overthinking and encourages more concise reasoning while preserving the model's explorative capacity. We apply our method to Qwen2.5-VL-3B, creating View-R1-3B. View-R1-3B significantly enhances reasoning capabilities, showing an average 7\% gain over the base model and outperforming larger MLLMs (7-11B) on various reasoning benchmarks. Importantly, unlike other reasoning-tuned MLLMs that often degrade on general tasks, View-R1-3B maintains consistent improvement, demonstrating superior generalization. These results highlight the effectiveness and broad applicability of our DADS and STCR techniques for advancing complex multimodal reasoning in MLLMs. The code will be made available at https://github.com/Indolent-Kawhi/View-R1.

Figures

Figures reproduced from arXiv: 2506.21655 by the authors.

Figure 1
Figure 1. (a) Comparison of the performance on MathVista during the training process. (b) Com [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration GRPO and our method dealing with difficult query conditions. DADS adaptively [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of response length dynamics during training: Original GRPO on difficulty [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Illustration of the length bias in GRPO. When calculating the effective advantage, the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: An example demonstrating how our method, by incorporating DADS and STCR, avoids the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: (a) Different curve variants of f(d) in Difficulty-Adaptive Divergence Shaping (DADS). (b) Performance comparison on MathVista during training. (c) Peak performance and training stability (Coefficient of Variation, CV) for different DADS variants. Lower CV indicates hi…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 11 canonical work pages

  1. [16]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    2024

  2. [1]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, et al. Qwen2.5-vl technical report, 2025

    work page 2025

  3. [2]

    An augmented benchmark dataset for geometric question answering through dual parallel text encoding

    Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1511–1520, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics

    work page 2022

  4. [3]

    Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression, 2022

    Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression, 2022

    2022

  5. [4]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    arXiv 2024

  6. [5]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025

    work page 2025

  7. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  8. [7]

    Observe-r1: Unlocking reasoning abilities of mllms with dynamic progressive reinforcement learning

    Zirun Guo, Minjie Hong, and Tao Jin. Observe-r1: Unlocking reasoning abilities of mllms with dynamic progressive reinforcement learning. arXiv preprint arXiv:2505.12432, 2025

    arXiv 2025

Show all 40 references
  1. [8]

    Polymath: A challenging multi-modal mathematical reasoning benchmark, 2024

    Himanshu Gupta, Shreyas Verma, Ujjwala Anantheswaran, Kevin Scaria, Mihir Parmar, Swaroop Mishra, and Chitta Baral. Polymath: A challenging multi-modal mathematical reasoning benchmark, 2024

    2024

  2. [9]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  3. [10]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  4. [11]

    Figureqa: An annotated figure dataset for visual reasoning, 2018

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning, 2018

    2018

  5. [12]

    Scemqa: A scientific college entrance level multimodal question answering benchmark, 2024

    Zhenwen Liang, Kehan Guo, Gang Liu, Taicheng Guo, Yujun Zhou, Tianyu Yang, Jiajun Jiao, Renjie Pi, Jipeng Zhang, and Xiangliang Zhang. Scemqa: A scientific college entrance level multimodal question answering benchmark, 2024

    2024

  6. [13]

    Clevr-math: A dataset for compositional language, visual and mathematical reasoning, 2022

    Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning, 2022. 10

    2022

  7. [14]

    Othink-mr1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learning

    Zhiyuan Liu, Yuting Zhang, Feng Liu, Changwang Zhang, Ying Sun, and Jun Wang. Othink-mr1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learning. arXiv preprint arXiv:2503.16081, 2025

    2025 arXiv

  8. [15]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  9. [17]

    Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

    2022

  10. [18]

    Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning, 2023

    Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning, 2023

    2023

  11. [19]

    Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  12. [20]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  13. [21]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2023

    2023

  14. [22]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024

    2024

  15. [23]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  16. [24]

    Skywork r1v: pioneering multimodal reasoning with chain-of-thought

    Yi Peng, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, Li Ge, et al. Skywork r1v: pioneering multimodal reasoning with chain-of-thought. arXiv preprint arXiv:2504.05599, 2025

    2025 arXiv

  17. [25]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025

    2025 arXiv

  18. [26]

    We-math: Does your large multimodal model achieve human-like mathematical reasoning?, 2024

    Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong, Yida Xu, Muxi Diao, Zhimin Bao, Chen Li, and Honggang Zhang. We-math: Does your large multimodal model achie...

    2024

  19. [27]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  20. [28]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  21. [29]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  22. [30]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    2024

  23. [31]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  24. [32]

    Llava-o1: Let vision language models reason step-by-step

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440, 2024. 11

    2024 arXiv

  25. [33]

    R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615, 2025

    2025 arXiv

  26. [34]

    Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319, 2024

    2024 arXiv

  27. [35]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  28. [36]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  29. [37]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937, 2025

    2025 arXiv

  30. [38]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024

    2024 arXiv

  31. [39]

    aha moment

    Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1-zero’s" aha moment" in visual reasoning on a 2b non-sft model. arXiv preprint arXiv:2503.05132, 2025

    2025 arXiv

  32. [40]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. 12

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.