Pith. sign in

REVIEW 5 major objections 5 minor 42 references

StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Models

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A learned verifier that scores each sub-question rather than the whole response changes multimodal RL from all-or-nothing to partial credit, and the paper reports state-of-the-art results on six of twelve public benchmarks plus its own STEM

desk verdict A genuine new mechanism for RLVR on multi-question multimodal tasks, but the headline SOTA rests on an unvalidated LLM judge that shares a semantic-equivalence task with the training verifier. read the letter →

arxiv 2508.05383 v1 pith:5B2WAQNK submitted 2025-08-07 cs.AI

Xiangxiang Zhang , Jingxuan Wei , Donghong Zhong , Qi Chen , Caijun Jia , Cheng Tan , Jinming Gu , Xiaobo Qin
show 26 more authors
Zhiping Liu Liang Hu Tong Sun Yuchen Wu Zewei Sun Chenwei Lou Hua Zheng Tianyang Zhan Changbao Wang Shuangzhi Wu Zefa Lin Chang Guo Sihang Yuan Riwei Chen Shixiong Zhao Yingping Zhang Gaowei Wu Bihui Yu Jiahui Wu Zhehui Zhao Qianqian Liu Ruofeng Tang Xingyue Huang Bing Zhao Mengyang Zhang Youqiang Zhou
This is my paper
classification cs.AI
keywords structuredrewardmodelsverifiablerewardsmultimodalreasoningsub-questionscoringpartialcreditreinforcementlearningvision-languageSTEMbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models often fail on multi-part STEM questions because existing RLVR rewards give one binary score for the entire answer, so solving three of four sub-questions earns zero feedback. The paper proposes StructVRM, a two-stage pipeline: SFT on over 50,000 problems with chain-of-thought traces, then PPO reinforcement learning whose reward comes from a trained verifier that scores each sub-question separately, judging semantic and mathematical equivalence rather than exact string match. The authors claim the resulting model, Seed-StructVRM, reaches state-of-the-art results on six of twelve public multimodal benchmarks and sets a high score on their new STEM-Bench (79.23 total), with ablations showing both the verifier and RL contribute. The point of the work is that structured, partial-credit feedback lets RLVR extend beyond multiple-choice and single-number answers to open-ended, hard-to-verify questions.

What carries the argument

The central object is a model-based verifier that turns verification into a trainable subtask. Given a prediction ŷ and reference y, it returns a sub-question-level score vector s = fθ(ŷ,y) = [s1, ..., sk], with each sj being 0 or 1 determined by semantic or numerical equivalence, and sj possibly covering multiple blanks. The RL reward is R_StructVRM = (1/k) Σ_j mean(sj), the average fraction of sub-questions solved. Verifiable multiple-choice questions are handled by a deterministic rule-based reward, while hard-to-verify and open-ended questions go through the learned verifier. Data augmentation (choice-to-fill-in-the-blank, choice-to-true/false) decomposes coarse questions into modular ve

What would settle it

A decisive check would be to have human experts independently grade the STEM-Bench free-form questions and compare their scores with the LLM-judge scores used in the paper; if judge-style agreement rather than correctness drives the results, the reported 79.23 total should drop or re-rank under human grading. A cheaper version: replace the learned verifier's semantic-equivalence judgments with literal exact-match partial credit at the same sub-question granularity and retrain; if the gains survive, semantic equivalence is not the active ingredient.

Watch

Extended reading notes

Core claim

StructVRM's central claim is that reward signal structure, not just reward accuracy, matters for multimodal reasoning. Instead of R=1 when a full response matches the reference and R=0 otherwise, the paper trains a model-based verifier to parse a response into sub-questions and output a score vector s = [s1, ..., sk], where each entry is 0 or 1 based on semantic or mathematical equivalence. The RL reward is the mean of these entries. Training this verifier on over 200,000 LLM-distilled graded examples and using it inside PPO, the paper reports gains on open-ended free-form questions and an overall STEM-Bench total of 79.23 versus 75.51 for the internal baseline. The authors present this as e

Load-bearing premise

The verifier's grading is trustworthy: it is trained on labels distilled by an internal LLM using a strict rubric, and its reported 96.83% agreement with human experts is measured on a held-out set whose size and composition are not described; if those labels or that set are biased, the RL signal and the STEM-Bench evaluation, also performed by an LLM judge, partly reward agreement with the judge's style rather than correctness.

Editorial extensions

If this is right

  • Multipart and open-ended questions become trainable with RLVR, because partial credit guides the policy toward incrementally correct reasoning instead of zero reward.
  • Models trained this way can generalize to single-question reasoning: the paper reports gains on ScienceQA, RealworldQA, and other non-multi-part benchmarks.
  • The combination of SFT on 51,254 chain-of-thought problems and verifier-guided PPO is what produces the reported results; ablations show removing RL costs about 3.76 points and removing the verifier about 2.57 points on STEM-Bench.
  • The method introduces reusable data resources: over 200,000 verifier training examples and augmentation strategies that turn multiple-choice questions into true/false and fill-in-the-blank formats.
  • STEM-Bench provides a new high-difficulty, multi-question evaluation that can serve as a more discriminating testbed than single-answer benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable implication the paper does not pursue: if fine-grained verifier rewards are the active ingredient, the same recipe should transfer to other open-ended verification domains, such as legal reasoning, code repair, or long-form scientific derivation, wherever an LLM judge can produce a structured rubric.
  • Because the final STEM-Bench evaluation is itself performed by an LLM judge, the reported gains may partly reflect alignment between the trained model and LLM-judge conventions; a human-scored subset would settle how much of the margin is genuine correctness.
  • The paper's own error analysis points to visual-structure parsing failures, such as bond counting and heat-distribution symmetry, that sub-question rewards do not fix; future work may need tighter vision-language grounding rather than only finer reward granularity.
  • The method's reliance on an internal base model and LLM-distilled verifier labels means the public contribution is currently the method and benchmark, not a fully reproducible artifact; publishing the verifier and describing the human-agreement set would change that.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes StructVRM, a two-stage post-training pipeline for a multimodal MoE vision-language model. Stage 1 is SFT on ~50k curated multimodal problems with generated chain-of-thought traces. Stage 2 is PPO with a reward mix: rule-based exact matching for deterministic multiple-choice answers, and a trained model-based verifier for hard-to-verify and open-ended multi-sub-question problems. The verifier outputs a sub-question-level score vector, and the reward is the mean across sub-questions (Eq. 2). The authors also construct STEM-Bench, a new benchmark of high-difficulty math/physics/chemistry/biology questions, and report that Seed-StructVRM achieves state-of-the-art results on 6 of 12 public benchmarks and the best total on STEM-Bench (79.23 vs. 77.85 for Gemini-2.5-pro, Table 2). Ablations (Table 3) show lower totals without StructVRM (76.66) or without RL (75.47).

Significance. If the empirical claims hold, the paper contributes a practical technique for extending RLVR to structured, multi-part, open-ended multimodal reasoning, where scalar final-answer rewards are indeed too coarse. The verifier's sub-question score vectors are a reasonable design and the internal ablations support the qualitative conclusion that fine-grained verifier rewards plus RL help. The proposed STEM-Bench could be a useful evaluation resource, provided it is released with transparent provenance. However, the headline results currently rest on an unvalidated LLM judge for STEM-Bench and a verifier whose only reported quality figure is an underspecified 96.83% agreement. The margins over strong baselines and across ablations are small (1–4 points), so judge or verifier bias of even a few points could change the main conclusions. The paper does not release code, data, or the benchmark, and it reports no variance or seed information, which limits independent verification. These are fixable within the manuscript's scope, but they are load-bearing rather than cosmetic.

major comments (5)
  1. [§7.1, Table 2] The STEM-Bench evaluation uses an LLM judge: 'a large language model (LLM) is used to extract and critique both the model-generated answers and the reference answers' with three independent scoring rounds averaged. No human agreement, no inter-annotator reliability, and no per-item variance or error bars are reported. This is load-bearing because the reported SOTA margins are small: 79.23 vs. 77.85 for Gemini-2.5-pro, and the ablation margins in Table 3 are 2.57 and 3.76 points. A systematic judge preference for Seed-StructVRM's output style (length, boxed answers, CoT structure) could account for these differences. Moreover, the judge's semantic-equivalence task is essentially the same judgment that the verifier was trained to make (Section 3.2.1, Figure 3), so the evaluation is not independent of the training signal. Please report human-judge agreement on a random sample of STEM-Bench,
  2. [§3.2.1] The verifier's only quality evidence is '96.83% agreement with expert judgments' on a held-out evaluation set, but the size, composition, annotation procedure, and per-format breakdown of that set are never described. This verifier is the source of the RL reward in Section 6.2 via Eq. (2), and its labels were themselves distilled by an internal LLM using a strict rubric. If the distilled labels are biased, the improvement in Table 3 could reflect reward overoptimization rather than genuine reasoning gains. Please specify the held-out set size and annotator details, and report verifier agreement separately for multiple-choice, fill-in-the-blank, and open-ended free-form questions, especially the FFQ items that drive the STEM-Bench chemistry gains.
  3. [§6.2, §7.4, Table 3] The ablation conditions are not defined precisely. 'w/o StructVRM' presumably means the model-based verifier is not used for hard-to-verify/non-verifiable problems, but the caption does not say whether rule-based rewards still apply to those items, or what replaces the verifier. Without this, the 2.57-point drop cannot be attributed specifically to the structured verifier rather than to removing any reward for open-ended items. Similarly, 'w/o RL' is a training-stage ablation, not a controlled reward-design ablation. No seeds or error bars are reported, so it is impossible to tell whether the 79.23 vs. 76.66 and 79.23 vs. 75.47 differences are statistically distinguishable. Please define the exact training configurations and report repeated-run variation, even if only for a subset.
  4. [§3.1, Table 1] The training data pipeline collects 'large-scale, publicly available multimodal questions' and 'open-source reasoning data' with no decontamination analysis against the public benchmarks evaluated in Table 1 (MathVision, ScienceQA, CMMMU, MMMU, MMMU-pro, etc.). If any of those benchmark test sets appear in the 90w+ collection or the 50k SFT set, the public-benchmark SOTA claims would be inflated. The same concern applies to STEM-Bench if any of its source exams overlap with training data. A standard decontamination report (exact/near-duplicate analysis and handling of detected overlaps) is needed to support the benchmark claims.
  5. [§2.3, §7] The paper's central claim is that fine-grained, sub-question-level verifier rewards improve multimodal reasoning relative to coarse scalar rewards. The experiments compare the full pipeline to an internal baseline and to external models, but they never compare against an alternative fine-grained reward mechanism, such as a scalar model-based verifier with the same model capacity, a process reward model, or a generative reward model. The ablation 'w/o StructVRM' removes the verifier's structured signal, but it does not isolate whether the benefit comes from the verifier being model-based, from sub-question decomposition, or simply from having any reward on open-ended items. An additional comparison with a non-structured model-based verifier would make the 'structured' claim load-bearing.
minor comments (5)
  1. [References] Several citations appear mismatched. For example, the text cites MathVision as [6], but reference [6] is titled 'SoliGeo: Measuring Multimodal Spatial Math Reasoning in Solid Geometry.' Similarly, [10] is cited for science-level benchmarks, but R-Bench is about robustness to corruptions. Please verify all bibliography entries.
  2. [§7.1] The evaluation protocol for public benchmarks says only 'single-pass inference (pass@1).' Details such as temperature, maximum tokens, prompt templates, and whether the '*' results in Table 1 were obtained with identical settings across models are missing. This matters for reproducibility.
  3. [§4, Eq. (2)] The notation is slightly ambiguous: each s_j is defined as 'a one-dimensional list,' but then mean(s_j) is used. Please clarify whether s_j is a vector of binary scores and whether 'mean' is over all elements or over sub-blanks.
  4. [§3.2.2] The number of generated CoT traces per problem and the pass-rate thresholds for the 'target difficulty band' (pass rate > 0.8 and = 0) are mentioned, but the exact thresholds and the number of retained examples per band are not tabulated. A small table would help.
  5. [§7.4] In Table 3, the 'w/o StructVRM' row shows e.g., Math MCQ 43.26 vs. 43.04, which is a small increase. The text says 'lighter but consistent reductions' but this is not consistent across all cells; please acknowledge or explain the non-monotonic pattern.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-level circularity; the unvalidated LLM judge on STEM-Bench is a validity risk, not a circular reduction.

full rationale

The paper's derivation chain is empirical rather than definitional. The verifier is trained on LLM-distilled sub-question labels (Section 3.2.1) and used to compute structured rewards (Eq. 2) for PPO (Section 6.2). The headline STEM-Bench evaluation is performed by a separate LLM that extracts and critiques answers (Section 7.1), not by the fitted verifier itself; the paper never states that the evaluator is the verifier or that the benchmark score is a function of the training reward. Without that identity, there is no equation-level reduction of the evaluation to the training input. Public benchmarks with fixed answers (ScienceQA, MMMU, RealworldQA, etc.) provide independent grounding that does not depend on the LLM judge. The verifier's reported 96.83% agreement with human experts is a human-annotated check, though its size and construction are not reported. The main concern is that the STEM-Bench LLM judge is never validated against human experts and shares the same semantic-equivalence judgment task as the verifier's training labels, which could bias the reported gains; this is a correctness/validity risk rather than a circularity, because the paper does not define the benchmark outcome as the verifier's output. The self-citation to Seed1.5-VL as the base model is not load-bearing for the method's claimed contribution. No circular step meets the threshold of exhibiting a specific reduction of the result to its inputs.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on four unverified premises: unbiased LLM-distilled labels, verifier generalization from an undescribed held-out set, validity of the STEM-Bench LLM judge, and no scale confound versus open baselines. Free parameters are hand-chosen training choices that are not ablated. The paper introduces no new particles, forces, dimensions, or conserved quantities; the model-based verifier is a trained neural network rather than a postulated entity, and its independent evidence is limited to the internal 96.83% agreement claim.

free parameters (2)
  • KL coefficient for verifiable prompts = 0
    Chosen to allow exploration during RL while general prompts use 1e-5 (Section 6.2). This choice is not ablated and directly gates reward-hacking risk.
  • SFT target difficulty band thresholds = pass rate > 0.8 excluded; pass rate = 0 retried
    Hand-chosen filters defining which generated reasoning traces enter the SFT set (Section 6.1); thresholds are not varied in any experiment.
assumptions (5)
  • domain assumption LLM-distilled sub-question labels used to train the verifier are correct and unbiased.
    Section 3.2.1: the verifier's 200k training set is labeled by an internal LLM using a strict rubric; no human audit or re-scoring of training labels is reported.
  • domain assumption The verifier's 96.83% agreement with human experts on its held-out set transfers to unseen question formats.
    Section 3.2.1 states the agreement figure but does not report the held-out set size, subject mix, or annotation procedure.
  • domain assumption The STEM-Bench LLM evaluator's scores reflect true answer correctness.
    Section 7.1: an unnamed LLM averages three scoring rounds per sample; no human-agreement study for this protocol is given, so metric validity is assumed.
  • domain assumption Observed gains over open baselines are not primarily a base-model scale effect.
    Section 7.1: Seed-StructVRM starts from an unreleased 20B/200B MoE while the largest open comparison is Qwen2.5-VL-72B; the comparison is scale-confounded.
  • domain assumption The mean of sub-question scores is a valid RL reward for multi-part problems.
    Equation (2) assigns equal weight to all sub-questions and assumes their scores are independent; no sensitivity analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Models." pith.science (2026). https://pith.science/paper/5B2WAQNK

@misc{pith2026250805383,
  author       = {Pith},
  title        = {Pith review of: StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5B2WAQNK}},
  note         = {Machine review of arXiv:2508.05383}
}
read the original abstract

Existing Vision-Language Models often struggle with complex, multi-question reasoning tasks where partial correctness is crucial for effective learning. Traditional reward mechanisms, which provide a single binary score for an entire response, are too coarse to guide models through intricate problems with multiple sub-parts. To address this, we introduce StructVRM, a method that aligns multimodal reasoning with Structured and Verifiable Reward Models. At its core is a model-based verifier trained to provide fine-grained, sub-question-level feedback, assessing semantic and mathematical equivalence rather than relying on rigid string matching. This allows for nuanced, partial credit scoring in previously intractable problem formats. Extensive experiments demonstrate the effectiveness of StructVRM. Our trained model, Seed-StructVRM, achieves state-of-the-art performance on six out of twelve public multimodal benchmarks and our newly curated, high-difficulty STEM-Bench. The success of StructVRM validates that training with structured, verifiable rewards is a highly effective approach for advancing the capabilities of multimodal models in complex, real-world reasoning domains.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 27 linked inside Pith

  1. [1]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report.arXiv preprint arXiv:2505.07062, 2025

    arXiv 2025

  2. [2]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024

    2024

  3. [3]

    Gemini 2.5: Our most intelligent ai model, 2025

    Google DeepMind. Gemini 2.5: Our most intelligent ai model, 2025

    2025

  4. [4]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv 2023

  5. [5]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer, 2024

    2024

  6. [6]

    Solidgeo: Measuring multimodal spatial math reasoning in solid geometry.arXiv preprint arXiv:2505.21177, 2025

    Peijie Wang, Chao Yang, Zhong-Zhi Li, Fei Yin, Dekang Ran, Mi Tian, Zhilong Ji, Jinfeng Bai, and Cheng-Lin Liu. Solidgeo: Measuring multimodal spatial math reasoning in solid geometry.arXiv preprint arXiv:2505.21177, 2025

    arXiv 2025

  7. [7]

    We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024

    Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024

    arXiv 2024

  8. [8]

    Polymath: A challenging multi-modal mathematical reasoning benchmark.arXiv preprint arXiv:2410.14702, 2024

    Himanshu Gupta, Shreyas Verma, Ujjwala Anantheswaran, Kevin Scaria, Mihir Parmar, Swaroop Mishra, and Chitta Baral. Polymath: A challenging multi-modal mathematical reasoning benchmark.arXiv preprint arXiv:2410.14702, 2024

    arXiv 2024

Show all 42 references
  1. [9]

    Mathscape: Evaluating mllms in multimodal math scenarios through a hierarchical benchmark

    Minxuan Zhou, Hao Liang, Tianpeng Li, Zhiyu Wu, Mingan Lin, Linzhuang Sun, Yaqi Zhou, Yan Zhang, Xiaoqin Huang, Yicong Chen, et al. Mathscape: Evaluating mllms in multimodal math scenarios through a hierarchical benchmark. arXiv preprint arXiv:2408.07543, 2024

    2024

  2. [10]

    R-bench: Are your large multimodal model robust to real-world corruptions?arXiv preprint arXiv:2410.05474, 2024

    Chunyi Li, Jianbo Zhang, Zicheng Zhang, Haoning Wu, Yuan Tian, Wei Sun, Guo Lu, Xiaohong Liu, Xiongkuo Min, Weisi Lin, et al. R-bench: Are your large multimodal model robust to real-world corruptions?arXiv preprint arXiv:2410.05474, 2024

    2024 arXiv

  3. [11]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. InProceedings of the 62nd Annua...

    2024

  4. [12]

    Scibench: Evaluating college-level scientific problem-solving abilities of large language models

    Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. InInternational Conference on Machine Lear...

    2024

  5. [13]

    Scemqa: A scientific college entrance level multimodal question answering benchmark

    Zhenwen Liang, Kehan Guo, Gang Liu, Taicheng Guo, Yujun Zhou, Tianyu Yang, Jiajun Jiao, Renjie Pi, Jipeng Zhang, and Xiangliang Zhang. Scemqa: A scientific college entrance level multimodal question answering benchmark. InProceedings ofthe 62nd AnnualMeeting ofthe Association ...

    2024

  6. [14]

    Vnhsge: Vietnamese high school graduation examination dataset for large language models

    Xuan-Quy Dao, Ngoc-Bich Le, The-Duy Vo, Xuan-Dung Phan, Bac-Bien Ngo, Van-Tien Nguyen, Thi-My-Thanh Nguyen, and Hong-Phuoc Nguyen. Vnhsge: Vietnamese high school graduation examination dataset for large language models. arXiv preprint arXiv:2305.12199, 2023

    2023 arXiv

  7. [15]

    Chemvlm: Exploring the power of multimodal large language models in chemistry area

    Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang, Xinrui Xiong, et al. Chemvlm: Exploring the power of multimodal large language models in chemistry area. In Proceedings of the AAAI Conference on Artificial Intelligence, numb...

    2025

  8. [16]

    Chemrxivquest: A curated chemistry question-answer database extracted from chemrxiv preprints.arXiv preprint arXiv:2505.05232, 2025

    Mahmoud Amiri and Thomas Bocklitz. Chemrxivquest: A curated chemistry question-answer database extracted from chemrxiv preprints.arXiv preprint arXiv:2505.05232, 2025

    2025 arXiv

  9. [17]

    Evaluating the symbol binding ability of large language models for multiple-choice questions in vietnamese general education

    Duc-Vu Nguyen and Quoc-Nam Nguyen. Evaluating the symbol binding ability of large language models for multiple-choice questions in vietnamese general education. InProceedings of the 12th International Symposium on Information and Communication Technology, pages 379–386, 2023. 17

    2023

  10. [18]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    2025 arXiv

  11. [19]

    Reason-rft: Reinforcement fine-tuning for visual reasoning.arXiv preprint arXiv:2503.20752, 2025

    Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning.arXiv preprint arXiv:2503.20752, 2025

    2025

  12. [20]

    Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  13. [21]

    Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  14. [22]

    Perception-r1: Pioneering perception policy with reinforcement learning.arXiv preprint arXiv:2504.07954, 2025

    En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, et al. Perception-r1: Pioneering perception policy with reinforcement learning.arXiv preprint arXiv:2504.07954, 2025

    2025 arXiv

  15. [23]

    aha moment

    Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1-zero’s" aha moment" in visual reasoning on a 2b non-sft model.arXiv preprint arXiv:2503.05132, 2025

    2025 arXiv

  16. [24]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  17. [25]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  18. [26]

    Demystifying long chain-of-thought reasoning in llms.arXiv preprint arXiv:2502.03373, 2025

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms.arXiv preprint arXiv:2502.03373, 2025

    2025 arXiv

  19. [27]

    Open r1: A fully open reproduction of deepseek-r1, january 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, january 2025. URL https://github. com/huggingface/open-r1, page 9, 2025

    2025

  20. [28]

    Rethinking rl scaling for vision language models: A transparent, from-scratch framework and comprehensive evaluation scheme.arXiv preprint arXiv:2504.02587, 2025

    Yan Ma, Steffi Chern, Xuyang Shen, Yiran Zhong, and Pengfei Liu. Rethinking rl scaling for vision language models: A transparent, from-scratch framework and comprehensive evaluation scheme.arXiv preprint arXiv:2504.02587, 2025

    2025 arXiv

  21. [29]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025

    2025 arXiv

  22. [30]

    Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning.arXiv preprint arXiv:2504.08837, 2025

    Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning.arXiv preprint arXiv:2504.08837, 2025

    2025 arXiv

  23. [31]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  24. [32]

    Rlpr: Extrapolating rlvr to general domains without verifiers.arXiv preprint arXiv:2506.18254, 2025

    Tianyu Yu, Bo Ji, Shouli Wang, Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, et al. Rlpr: Extrapolating rlvr to general domains without verifiers.arXiv preprint arXiv:2506.18254, 2025

    2025 arXiv

  25. [33]

    Grpo-lead: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models.arXiv preprint arXiv:2504.09696, 2025

    Jixiao Zhang and Chunsheng Zuo. Grpo-lead: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models.arXiv preprint arXiv:2504.09696, 2025

    2025

  26. [34]

    Math- shepherd: Verify and reinforce llms step-by-step without human annotations.arXiv preprint arXiv:2312.08935, 2023

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math- shepherd: Verify and reinforce llms step-by-step without human annotations.arXiv preprint arXiv:2312.08935, 2023

    2023 arXiv

  27. [35]

    rstar-math: Small llms can master math reasoning with self-evolved deep thinking.arXiv preprint arXiv:2501.04519, 2025

    Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking.arXiv preprint arXiv:2501.04519, 2025

    2025 arXiv

  28. [36]

    R-prm: Reasoning-driven process reward modeling.arXiv preprint arXiv:2503.21295, 2025

    Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, and Shujian Huang. R-prm: Reasoning-driven process reward modeling.arXiv preprint arXiv:2503.21295, 2025. 18

    2025 arXiv

  29. [37]

    Mm-prm: Enhancing multimodal mathematical reasoning with scalable step-level supervision.arXiv preprint arXiv:2505.13427, 2025

    Lingxiao Du, Fanqing Meng, Zongkai Liu, Zhixiang Zhou, Ping Luo, Qiaosheng Zhang, and Wenqi Shao. Mm-prm: Enhancing multimodal mathematical reasoning with scalable step-level supervision.arXiv preprint arXiv:2505.13427, 2025

    2025 arXiv

  30. [38]

    Reasonflux-prm: Trajectory-aware prms for long chain-of-thought reasoning in llms.arXiv preprint arXiv:2506.18896, 2025

    Jiaru Zou, Ling Yang, Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang. Reasonflux-prm: Trajectory-aware prms for long chain-of-thought reasoning in llms.arXiv preprint arXiv:2506.18896, 2025

    2025

  31. [39]

    Judgelm: Fine-tuned large language models are scalable judges

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631, 2023

    2023 arXiv

  32. [40]

    Rm-r1: Reward modeling as reasoning.arXiv preprint arXiv:2505.02387, 2025

    Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-r1: Reward modeling as reasoning.arXiv preprint arXiv:2505.02387, 2025

    2025

  33. [41]

    Gram: A generative foundation reward model for reward generalization.arXiv preprint arXiv:2506.14175, 2025

    Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Qiaozhi He, Murun Yang, Bei Li, Tong Xiao, Chunliang Zhang, Tongran Liu, et al. Gram: A generative foundation reward model for reward generalization.arXiv preprint arXiv:2506.14175, 2025

    2025

  34. [42]

    Reasongrm: Enhancing generative reward models through large reasoning models.arXiv preprint arXiv:2506.16712, 2025

    Bin Chen, Xinzge Gao, Chuanrui Hu, Penghang Yu, Hua Zhang, and Bing-Kun Bao. Reasongrm: Enhancing generative reward models through large reasoning models.arXiv preprint arXiv:2506.16712, 2025. 19

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.