Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a quadratic surface over data-mixture weights predicts multimodal RLVR out-of-distribution accuracy well enough that the optimized mixture beats uniform mixing by 5.24% and the pre-finetuning base model by 20.74%.

desk verdict Useful qualitative result on mixture effects in multimodal RLVR, but the headline gain is selected on the same benchmarks used to fit the surrogate, so the quantitative claim is provisional. read the letter →

arxiv 2505.24871 v2 pith:5GXVPAEU submitted 2025-05-30 cs.CV cs.CLcs.LG

Yiqing Liang , Jielin Qiu , Wenhao Ding , Zuxin Liu , James Tompkin , Mengdi Xu , Mengzhou Xia , Zhengzhong Tu
show 2 more authors
Laixi Shi Jiacheng Zhu
This is my paper
classification cs.CVcs.CLcs.LG
keywords multimodallargelanguagemodelsreinforcementlearningwithverifiablerewardsdatamixtureoptimizationGRPOquadraticresponsesurfaceout-of-distributiongeneralizationvision-languagereasoningonlineRLpost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that how you mix multiple verifiable-reward vision-language datasets during RLVR post-training is a learnable design choice, not a detail. It curates five image-text datasets with different rule-based rewards, post-trains a 2B vision-language model with GRPO on weighted mixtures, and fits a quadratic surface to the out-of-distribution accuracy of a few pilot mixtures. It then uses that surface to pick a new mixture and reports that this mixture improves out-of-distribution benchmarks by an average of 5.24% over uniform mixing and by 20.74% over the pre-finetuning baseline. If this is right, data-mixture selection becomes a cheap lever for multimodal reasoning that does not require more data or a bigger model.

What carries the argument

The load-bearing object is the quadratic response surface $\hat{L}_{\mathrm{test}}(\mathbf{w}) = a + \mathbf{b}^{\top}\mathbf{w} + \mathbf{w}^{\top} C \mathbf{w}$, a second-order model of out-of-distribution test accuracy as a function of the mixture weights $\mathbf{w}$. The linear term captures each dataset's independent contribution, while the quadratic term $\mathbf{w}^{\top} C \mathbf{w}$ captures dataset interactions that linear models miss. Fitted by Eq. (3) over a small set of pilot RLVR runs and validated by five-fold cross-validation, this surface turns an exponentially large mixture search into a one-time optimization: candidate mixtures are sampled around observed mixtures, ranked by the fitted surface, and the top one is used for the final GRPO post-training. The machine also includes the five datasets with their per-domain reward functions (format, accuracy, and IoU) and the GRPO training loop that produces the post-trained checkpoint for each mixture.

What would settle it

Reserve at least one of the four out-of-distribution benchmarks entirely from the fitting loop, select a mixture without seeing it, and post-train the same base model on that mixture. If the chosen mixture does not beat uniform mixing on the reserved benchmark, the generalization claim collapses.

Watch

Extended reading notes

Core claim

The central discovery is that the post-training outcome $L_{\mathrm{test}}(\mathbf{w})$ is a predictable function of the mixture distribution $\mathbf{w}$, and that a multivariate quadratic model $a + \mathbf{b}^{\top}\mathbf{w} + \mathbf{w}^{\top} C \mathbf{w}$ captures the interaction among diverse datasets well enough to rank unseen mixtures. Using a small set of pilot RLVR runs on deliberately chosen seed mixtures, fitting the quadratic by least squares, and then sampling candidate mixtures around the observed ones, the framework selects a data mixture whose post-trained model outperforms a uniformly mixed model on four out-of-distribution benchmarks on average and exceeds the pre-finetuning baseline by a total of 20.74%. The paper presents this as the first systematic treatment of dataset mixtures for multimodal RLVR post-training.

Load-bearing premise

The load-bearing premise is that a quadratic surface fitted to a handful of pilot RLVR runs ranks unseen mixtures correctly, and that the same out-of-distribution benchmarks used for fitting can also serve as the final evaluation of the chosen mixture without inflating the measured gain.

Editorial extensions

If this is right

  • With a small set of pilot RLVR runs, a quadratic mixture-to-performance model can select a data mixture that beats uniform mixing on out-of-distribution benchmarks; in these experiments the average gain is 5.24%.
  • Multi-domain RLVR post-training on verifiable vision-language tasks improves both in-domain scores (LISA, SAT, ScienceQA) and out-of-domain benchmarks (ChartQA, InfoVQA, MathVista, MMMU) over the pre-finetuning base model.
  • Dataset interactions are nonlinear, so independent-weighting heuristics leave performance on the table; a quadratic model with cross terms captures these interactions better than linear regression.
  • The strategy is efficient: full search over the mixture simplex is replaced by a few pilot runs plus one optimized post-training run.
  • Any collection of datasets with verifiable rewards can be plugged into the same mixture-selection procedure, making mixture choice a reusable post-training component.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Iterating the loop is an untested extension: after a selected mixture is run, add it to the pilot set, refit the quadratic surface, and select again; if the surrogate is accurate near the optimum, the gains should compound until diminishing returns.
  • The quadratic surrogate is modality-agnostic, so the same recipe could be transferred to text-only RLVR or to video and audio verifiable domains; this extrapolates beyond the paper's image-text experiments.
  • A stricter validation would reserve one entire out-of-distribution benchmark for final evaluation only, since the paper uses the same four benchmarks both to fit and to judge the surrogate; that would reveal how much of the 5.24% gain is genuinely out-of-sample.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MoDoMoDo, a framework for optimizing data mixture weights in multimodal RLVR post-training. The authors curate five verifiable image-text training datasets (COCO, LISA, GeoQA V, SAT, ScienceQA), train Qwen2-VL-2B-Instruct with GRPO under different mixture recipes, and introduce a model-based strategy that fits a quadratic response surface to mixture-to-performance data, samples candidate mixtures, and selects the top candidates. The claimed headline result is that the best found mixture improves out-of-distribution accuracy by 5.24% over uniform mixing. The paper also reports heuristic mixture baselines and analyzes the effect of single, exclude-one, and all-dataset mixtures.

Significance. The qualitative finding that mixture composition affects OOD performance in multimodal RLVR is useful and is supported by Table 2: different seed and heuristic mixtures produce markedly different Out-Scores, and the authors provide a reasonably comprehensive set of training and evaluation benchmarks. The dataset curation and the formal mixture-optimization framing are potentially valuable for future work on multimodal RL post-training. However, the model-based contribution is not validated by the reported experiments. The evaluation protocol uses the same four OOD benchmarks both to fit the surrogate and to select and report the final mixture, and the experimental scale (11 pilot runs for a 21-parameter surrogate, single seed, no held-out benchmark) is too limited to support the claimed quantitative advantage of the proposed method. If a clean held-out evaluation and repeated-seed analysis were provided, the empirical contribution could be substantial; in its current form the central quantitative claim is not supported.

major comments (4)
  1. [§3.1, Table 1; Algorithm 4] The four 'Out' benchmarks (ChartQA, InfoVQA, MathVista, MMMU) are used to compute Out-Score, which is the target variable y in Eq. (3) and in Algorithm 4. The same Out-Score is then used to rank candidate mixtures and to report the final winner in Table 2. Consequently, the abstract's 5.24% improvement is an in-sample maximum of a response surface fitted to the evaluation benchmarks, not an unbiased estimate of out-of-distribution generalization. A benchmark held out from the mixture-fitting and mixture-selection procedure is needed before the headline claim can be assessed.
  2. [§2.2, Eq. (3); Algorithm 4; Figure 7] The quadratic surrogate has 21 free parameters (a, b, and symmetric C for m=5), while the seed set described in §3.2 consists of 11 runs (5 Single, 5 Exclude-One, All). Even if the heuristic runs are added to the fitting set, fitting 21 parameters to this many points and then selecting the best of 5 random CV splits (Figure 7) does not provide evidence of generalization. The held-out R² in Figure 7 is computed on random splits of the same points and cannot validate extrapolation to the simplex region from which the GMM samples candidate mixtures; the claim that linear models fail while quadratic models generalize is therefore not established.
  3. [Table 2; Abstract] The best Out-Score in Table 2 is achieved by the heuristic Norm row (0.5133 at 2000 steps), not by any model-based run (maximum 0.5104 for run 010). The abstract's 'best mixture' therefore appears to refer to the leave-one-out heuristic rather than to the surrogate-based method, unless the authors intend to claim the best across all strategies, in which case the model-based optimizer is not the source of the headline gain. This ambiguity needs to be resolved and the reported 5.24% needs to be attributed to the correct method.
  4. [§3.3, Figure 6; Table 2] The comparison reports single runs without repeated-seed variance; Section 3.1 states that training uses deterministic seed 42. The claim that the model-based strategy 'reduces variance' refers to variance across mixtures within one seed, not across training stochasticity. The Out-Score differences among the top model-based runs (e.g., 0.5104, 0.5067, 0.5048) are small and no confidence intervals or significance tests are provided, so it is unclear whether the observed ranking is meaningful.
minor comments (5)
  1. [Abstract and Table 2] The 5.24% and 20.74% figures appear to be absolute percentage-point differences (e.g., 0.5133 - 0.4609 = 0.0524), not relative improvements; the paper should state this explicitly to avoid misinterpretation.
  2. [§3.2, Algorithm 4] Section 3.2 describes the model-based strategy as using 'Covariance Matrix Adaptation Evolution Strategy (CMA-ES)', but Algorithm 4 fits a quadratic surface by ordinary least squares and samples candidates from a GMM; the relationship between CMA-ES and the implemented algorithm is not explained.
  3. [Table 2] The labels 'Step-averaged' and '2000-step' are not defined in the main text; clarify whether step-averaged scores average over multiple checkpoints and over which steps.
  4. [Table 2, Model-based rows] The mixture weight vectors for the model-based runs 001-010 are not reported, so the experiments cannot be reproduced; please include the selected mixtures in a supplementary table.
  5. [Figure 7] The label '1@5 Splits' is ambiguous; it appears to mean the best of five random train-test splits, which is a form of selection and should be described as such in the text.

Circularity Check

2 steps flagged · score 6.0 of 10

The headline OOD gain is selected on the same four benchmarks used for evaluation, so the reported 5.24% is an in-sample selection result rather than an out-of-sample validation.

  1. fitted input called prediction [Sec. 2.2 (Eq. 3), Sec. 3.1 Evaluation Metrics, Algorithm 4, Table 2]
    "Using a non-linear second-order model bLtest(w) := a + b^T w + w^T C w ... The parameters {a, b, C} are estimated by minimizing the empirical risk minimization (ERM) objective using the data generated according to the seed samples: min_{a,b,C} 1/k Σ_{i=1}^k L( bLtest(w0_i), Ltest(w0_i) ) ... Finally, computing the optimal weight w* = arg max_{w∈Δ^m} bLtest(w) and using the corresponding training data distribution μ = μ(w*) for RLVR post-training. Out-Score: the weighted average over datasets that were not used at train time."

    The surrogate bLtest(w) is fit, via Eq. (3), to Ltest(w0_i), which is the Out-Score on ChartQA, InfoVQA, MathVista, and MMMU. Algorithm 4 then returns the mixture maximizing that same fitted surface, and Table 2 reports Out-Scores on the same four benchmarks as the headline result. Thus the objective used to rank/select the mixture and the metric used to validate it are identical; the 'predicted optimal mixture' is the in-sample argmax of a response surface fitted to the evaluation labels, not an out-of-sample estimate on a benchmark excluded from the optimization.

  2. fitted input called prediction [Sec. 3.2 Heuristic data mixture, Algorithm 3, Table 2]
    "Leave-One-Out Normalization (Norm): Assumes that the performance gap when a dataset is excluded compared to All is related to its importance in the final mixture. Algorithm 3: Normalize out scores ... s′_r ← 0.2−(0.1·ŝ_r) ▷ Transform: higher score = lower weight ... Derive dataset weights from leave-one-out scores. Table 2: Norm ... 0.5133."

    The Norm mixture weights are computed directly from leave-one-out Out-Scores on ChartQA, InfoVQA, MathVista, and MMMU, and the final Out-Score of that recommended mixture is measured on those same benchmarks. The reported gain in the abstract, 0.5133 − 0.4609 = 0.0524 or 5.24%, is exactly this Norm row versus the uniform All row. The heuristic therefore 'predicts' weights from the evaluation metric and is then evaluated on that same metric, so the headline improvement does not estimate generalization to held-out OOD benchmarks.

full rationale

The central quantitative claim is that the best mixture improves OOD accuracy by 5.24% over uniform. The derivation chain is: define Out-Score as the weighted average over ChartQA, InfoVQA, MathVista, and MMMU; fit the quadratic surrogate bLtest(w) to seed and heuristic runs scored with that Out-Score (Eq. 3); select mixtures by maximizing bLtest (Algorithm 4); then report Out-Scores of the selected runs in Table 2. Because the fitting/selection step and the final evaluation step use the same four benchmarks, the OOD benchmarks are not held out from the mixture-selection procedure. The abstract's exact absolute improvement, 0.0524, matches the heuristic Norm row (0.5133) versus uniform All (0.4609), and Norm's weights are themselves derived from leave-one-out Out-Scores, making the headline gain a direct function of the evaluation metric. This is a selection-bias form of circularity rather than an equation-level identity, and there is no load-bearing self-citation or imported uniqueness theorem, so the appropriate score is 6 rather than 8-10. Notably, the best model-based run (010) reaches only 0.5104, below Norm's 0.5133, so the claimed 'best mixture' is not even produced by the proposed surrogate method.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the assumption that the RLVR objective in Eq. (1) is solved well enough by a single GRPO run per mixture, that the verifiable reward rules are correct for all five tasks, that the four OOD benchmarks are fair, and that the hand-designed 11-point pilot design plus a 21-parameter quadratic surrogate can predict unseen mixture performance. The surrogate coefficients are fitted parameters, as are several heuristic hyperparameters and reward weights. No invented entities are introduced.

free parameters (7)
  • Quadratic surrogate coefficients a, b, C = not reported
    21 values fit by Eq. (3) to 11 pilot RL runs; these determine all predicted scores and the selected optimal mixture.
  • Pilot mixture design (seed set) = 5 Single, 5 Exclude-One, 1 All
    Hand-selected experimental design; the observed response surface depends entirely on these simplex points.
  • Alpha-family trade-off alpha = 0.0, 0.5, 1.0
    Manual hyperparameter balancing In-Score and Out-Score in Algorithm 1; not learned from data.
  • Ridge regularization lambda = 1e-3
    Default value in Algorithm 2 for the Collinearity heuristic.
  • Surrogate candidate hyperparameters nsamples, k, GMM components = not reported
    Algorithm 4 requires these values for GMM sampling and top-k selection; they influence which mixtures are considered and chosen.
  • Reward scaling weights = Acc/IoU 2.0, Format 1.0
    Hand-chosen in Section 3.1; affects the RL update magnitude but not the mixture prediction itself.
  • KL coefficient = 0.04
    Hand-chosen GRPO hyperparameter in Section 3.1; standard RLVR setup.
assumptions (6)
  • domain assumption GRPO training with Eq. (1)'s KL penalty reliably approximates the optimal policy theta*(w) for a given mixture w, so a single training run measures Ltest(w).
    Section 2.1 assumes the trained RLVR output approximates theta*(w0_i) for each seed run; no repeated runs are used to estimate noise.
  • domain assumption Verifiable rewards (format, accuracy, IoU) are sufficient and unambiguous signals for the five tasks; exact string match is a correct measure for natural-language VQA.
    Section 3.1 defines the rewards; exact-match scoring on SAT, ChartQA, and InfoVQA may be brittle to paraphrases but is assumed to be the right reward.
  • domain assumption ChartQA, InfoVQA, MathVista, and MMMU are genuinely out-of-distribution and are not indirectly covered by the five training datasets.
    Section 3.1 lists these as Out benchmarks; the generalization claim depends on them being unseen tasks.
  • ad hoc to paper The quadratic model bLtest(w)=a+b^T w+w^T C w is an adequate response surface for the mixture-to-performance mapping, and linear models fail.
    Section 2.2 and Figure 7 justify this by a top-1-of-5-splits cross-validation selection on 11 points, not by an independent test.
  • domain assumption A single deterministic seed (42) and fixed hyperparameters make run-to-run variation negligible.
    Section 3.1 reports one run per mixture; no error bars or seed ablations are provided.
  • ad hoc to paper Selecting the best of k candidate mixtures using a surrogate fitted to the evaluation benchmarks yields a valid estimate of the method's performance.
    Algorithm 4 ranks candidates by predicted Out-Score and the same Out benchmarks are used for final scores; this assumes no selection bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning." pith.science (2026). https://pith.science/paper/5GXVPAEU

@misc{pith2026250524871,
  author       = {Pith},
  title        = {Pith review of: MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5GXVPAEU}},
  note         = {Machine review of arXiv:2505.24871}
}
read the original abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a powerful paradigm for post-training large language models (LLMs), achieving state-of-the-art performance on tasks with structured, verifiable answers. Applying RLVR to Multimodal LLMs (MLLMs) presents significant opportunities but is complicated by the broader, heterogeneous nature of vision-language tasks that demand nuanced visual, logical, and spatial capabilities. As such, training MLLMs using RLVR on multiple datasets could be beneficial but creates challenges with conflicting objectives from interaction among diverse datasets, highlighting the need for optimal dataset mixture strategies to improve generalization and reasoning. We introduce a systematic post-training framework for Multimodal LLM RLVR, featuring a rigorous data mixture problem formulation and benchmark implementation. Specifically, (1) We developed a multimodal RLVR framework for multi-dataset post-training by curating a dataset that contains different verifiable vision-language problems and enabling multi-domain online RL learning with different verifiable rewards; (2) We proposed a data mixture strategy that learns to predict the RL fine-tuning outcome from the data mixture distribution, and consequently optimizes the best mixture. Comprehensive experiments showcase that multi-domain RLVR training, when combined with mixture prediction strategies, can significantly boost MLLM general reasoning capacities. Our best mixture improves the post-trained model's accuracy on out-of-distribution benchmarks by an average of 5.24% compared to the same model post-trained with uniform data mixture, and by a total of 20.74% compared to the pre-finetuning baseline.

Figures

Figures reproduced from arXiv: 2505.24871 by the authors.

Figure 2
Figure 2. Demonstration of a General Question￾Answer Pair With and Without Reasoning. We begin by describing the experimental setup, including dataset curation, reasoning mode, re￾ward models, training and testing strategies, data sampling, and evaluation metrics in Sec. 3.1. Subsequently, Sec. 3.2 explains the design of 3 groups of data mixture strategies examined in our paper, i.e. seed, heuristic, and model-based. Finally,… view at source ↗
Figure 3
Figure 3. Model Performance before / after GRPO training on All data mixture. Reasoning Mode Rather than permitting the multimodal LLM to emit an answer immediately after observing the image-question pair, we place it in a two-step reasoning mode: (1) generate a free-form chain-of-thought, and (2) commit to a concise final answer. We trigger this mode by appending a specialised reasoning prompt to every question. Sec. 3 is an… view at source ↗
Figure 4
Figure 4. Model Performance Comparison after GRPO training using [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Model Performance Comparison after GRPO training using [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Model-based mixture optimization. Left: PCA on mixture vectors reveals lack of linear separability, motivating nonlinear modeling. Middle-left: Linear regression fails to fit Out-Score accurately, even with all data. Right: Quadratic regression fitted on training folds…
Figure 6
Figure 6. Figure 6: Data Mixture Strategies’ Comparison. Heuristic (middle) and model-based (right) achieve higher min￾ima and medians than Seed (middle). Model-based further reduces variance. Comparison of Mixture Strategies ( [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Comparison of Grounding Question-Answer Pairs [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    RLVR training on 64,000 procedurally generated Trace instances improves Qwen2.5-VL macro-average on 24 external visual reasoning benchmarks by 3.51 points at 3B and 4.06 points at 7B.

  2. Multi-Task GRPO: Reliable LLM Reasoning Across Tasks

    cs.CL 2026-02 conditional novelty 6.0 of 10

    MT-GRPO reweights tasks by reward and improvement and enforces those weights after zero-gradient filtering, improving worst-task accuracy by 6–28% over GRPO/DAPO baselines on 3- and 9-task setups.

  3. Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    A dynamic replay and reweighting scheduler (RECAP) preserves general capabilities during RLVR while keeping reasoning performance at least as good as reasoning-only finetuning.

Reference graph

Works this paper leans on

71 extracted references · 39 canonical work pages · cited by 3 Pith papers

  1. [1]

    Alayrac, J

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    Albalak, L

    A. Albalak, L. Pan, C. Raffel, and W. Y . Wang. Efficient online data mixing for language model pre-training. arXiv preprint arXiv:2312.02406, 2023

    arXiv 2023

  3. [3]

    Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. ...

    arXiv 2022

  4. [5]

    H. Chen, Y . Feng, Z. Liu, W. Yao, A. Prabhakar, S. Heinecke, R. Ho, P. L. Mui, S. Savarese, C. Xiong, and H. Wang. Language models are hidden reasoners: Unlocking latent reasoning capabilities via self- rewarding.OpenReview (ICLR), 2024

    work page 2024

  5. [6]

    H. Chen, H. Tu, F. Wang, H. Liu, X. Tang, X. Du, Y . Zhou, and C. Xie. Sft or rl? an early investigation into training r1-like reasoning large vision-language models, 2025

    work page 2025

  6. [7]

    L. Chen, L. Li, H. Zhao, Y . Song, and Vinci. R1-v: Reinforcing super generalization ability in vision- language models with less than $3. https://github.com/Deep-Agent/R1-V , 2025. Accessed: 2025- 02-02

    work page 2025

  7. [8]

    X. Chen, W. Li, C. Liu, C. Xie, X. Hu, C. Ma, F. Zhu, and R. Zhao. On the suitability of reinforcement fine-tuning to visual tasks, 2025

    work page 2025

  8. [9]

    Y . Chen, Y . Ge, R. Wang, Y . Ge, L. Qiu, Y . Shan, and X. Liu. Exploring the effect of reinforcement learning on video understanding: Insights from seed-bench-r1.ArXiv Preprint, 2025

    work page 2025

Show all 71 references
  1. [10]

    H. W. Chung, N. Constant, X. Garcia, A. Roberts, Y . Tay, S. Narang, and O. Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.arXiv preprint arXiv:2304.09151, 2023

    2023 arXiv

  2. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.ArXiv Preprint, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.ArXiv Preprint, 2025

    2025

  3. [12]

    Y . Deng, H. Bansal, F. Yin, N. Peng, W. Wang, and K.-W. Chang. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement.ArXiv Preprint, 2025

    2025

  4. [13]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,...

    2019

  5. [14]

    K. Feng, K. Gong, B. Li, Z. Guo, Y . Wang, T. Peng, B. Wang, and X. Yue. Video-r1: Reinforcing video reasoning in mllms.ArXiv Preprint, 2025

    2025

  6. [17]

    Havrilla, Y

    A. Havrilla, Y . Du, S. C. Raparthy, C. Nalmpantis, J. Dwivedi-Yu, M. Zhuravinskyi, E. Hambro, S. Sukhbaatar, and R. Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642, 2024. 10

    2024 arXiv

  7. [18]

    W. Held, B. Paranjape, P. S. Koura, M. Lewis, F. Zhang, and T. Mihaylov. Optimizing pretraining data mixtures with llm-estimated utility.arXiv preprint arXiv:2501.11747, 2025

    2025 arXiv

  8. [19]

    J. Hu, Y . Zhang, Q. Han, D. Jiang, X. Zhang, and H.-Y . Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    2025 arXiv

  9. [20]

    Ilyas, S

    A. Ilyas, S. M. Park, L. Engstrom, G. Leclerc, and A. Madry. Datamodels: Predicting predictions from training data.arXiv preprint arXiv:2202.00622, 2022

    2022 arXiv

  10. [21]

    Kazemnejad, M

    A. Kazemnejad, M. Aghajohari, E. Portelance, A. Sordoni, S. Reddy, A. Courville, and N. L. Roux. Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment.OpenReview (ICLR), 2024

    2024

  11. [22]

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  12. [23]

    X. Lai, Z. Tian, Y . Chen, Y . Li, Y . Yuan, S. Liu, and J. Jia. Lisa: Reasoning segmentation via large language model.ArXiv Preprint, 2023

    2023

  13. [24]

    H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V . Carbune, A. Rastogi, and S. Prakash. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2024

    2024 arXiv

  14. [25]

    J. Leng, C. Huang, L. Huang, B. Y . Lin, W. W. Cohen, H. Wang, and J. Huang. Crosswordbench: Evaluating the reasoning capabilities of llms and lvlms with controllable puzzle generation, 2025

    2025

  15. [26]

    B. Li, K. Zhang, and A. Marafioti. Multimodal open r1. https://github.com/EvolvingLMMs-Lab/ open-r1-multimodal, 2025. Accessed: 2025-02-08

    2025

  16. [27]

    Z. Li, X. Wu, H. Du, H. Nghiem, and G. Shi. A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges.ArXiv Preprint, 2025

    2025

  17. [28]

    Liang, W

    W. Liang, W. Gao, W. Yu, and W. Zhang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  18. [29]

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors,European Conference on Computer Vision (ECCV), 2014

    2014

  19. [30]

    Z. Lin, Y . Gao, X. Zhao, Y . Yang, and J. Sang. Mind with eyes: from language reasoning to multimodal reasoning.ArXiv Preprint, 2025

    2025

  20. [31]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  21. [32]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  22. [33]

    Q. Liu, X. Zheng, N. Muennighoff, G. Zeng, L. Dou, T. Pang, J. Jiang, and M. Lin. Regmix: Data mixture as regression for language model pre-training.arXiv preprint arXiv:2407.01492, 2024

    2024 arXiv

  23. [34]

    X. Liu, J. Ni, Z. Wu, C. Du, L. Dou, H. Wang, T. Pang, and M. Q. Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation, 2025

    2025

  24. [35]

    Z. Liu, Z. Sun, Y . Zang, X. Dong, Y . Cao, H. Duan, D. Lin, and J. Wang. Visual-rft: Visual reinforcement fine-tuning.ArXiv Preprint, 2025

    2025

  25. [36]

    Z. Liu, Y . Zhang, F. Liu, C. Zhang, Y . Sun, and J. Wang. Othink-mr1: Stimulating multimodal generalized reasoning capabilities through dynamic reinforcement learning.ArXiv Preprint, 2025

    2025

  26. [37]

    P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. InThe 36th Conference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  27. [38]

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  28. [39]

    Y . Ma, S. Chern, X. Shen, Y . Zhong, and P. Liu. Rethinking rl scaling for vision language models: A transparent, from-scratch framework and comprehensive evaluation scheme.ArXiv Preprint, 2025. 11

    2025

  29. [40]

    Masry, D

    A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  30. [41]

    Mathew, V

    M. Mathew, V . Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1697–1706, January 2022

    2022

  31. [42]

    McKinzie, Z

    B. McKinzie, Z. Gan, J.-P. Fauconnier, S. Dodge, B. Zhang, P. Dufter, D. Shah, X. Du, F. Peng, A. Belyi, et al. Mm1: methods, analysis and insights from multimodal llm pre-training. InEuropean Conference on Computer Vision, pages 304–323. Springer, 2024

    2024

  32. [43]

    Y . Meng, M. Xia, and D. Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  33. [44]

    Y . Mroueh. Reinforcement learning with verifiable rewards: Grpo’s effective loss, dynamics, and success amplification.arXiv preprint arXiv:2503.06639, 2025

    2025

  34. [45]

    Cosmos-reason1: From physical common sense to embodied reasoning.ArXiv Preprint, 2025

    NVIDIA. Cosmos-reason1: From physical common sense to embodied reasoning.ArXiv Preprint, 2025

    2025

  35. [46]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. URL https://arxiv.org/ abs/2303.08774

    2023 arXiv

  36. [47]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions ...

    2022 arXiv

  37. [48]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions ...

    2022

  38. [49]

    Peng, Chris, X

    Y . Peng, Chris, X. Wang, Y . Wei, J. Pei, W. Qiu, A. Jian, Y . Hao, J. Pan, T. Xie, L. Ge, R. Zhuang, X. Song, Y . Liu, and Y . Zhou. Skywork r1v: Pioneering multimodal reasoning with chain-of-thought, 2025. URL https://huggingface.co/Skywork/Skywork-R1V-38B

    2025

  39. [50]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  40. [51]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model.arXiv preprint arXiv:2305.18290, 2023

    2023 arXiv

  41. [52]

    A. Ray, J. Duan, R. Tan, D. Bashkirova, R. Hendrix, K. Ehsani, A. Kembhavi, B. A. Plummer, R. Krishna, K.-H. Zeng, and K. Saenko. Sat: Spatial aptitude training for multimodal language models.ArXiv Preprint, 2024

    2024

  42. [53]

    Sagawa, P

    S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.arXiv preprint arXiv:1911.08731, 2019

    1911 arXiv

  43. [54]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y . Li, Y . Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300

    2024 arXiv

  44. [55]

    H. Shen, P. Liu, J. Li, C. Fang, Y . Ma, J. Liao, Q. Shen, Z. Zhang, K. Zhao, Q. Zhang, R. Xu, and T. Zhao. Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  45. [56]

    Sheng, C

    G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y . Peng, H. Lin, and C. Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256, 2024

    2024 arXiv

  46. [57]

    Y . Song, T. Ou, Y . Kong, Z. Li, G. Neubig, and X. Yue. Visualpuzzles: Decoupling multimodal reasoning evaluation from domain knowledge, 2025

    2025

  47. [58]

    Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y . Shen, C. Gan, L.-Y . Gui, Y .-X. Wang, Y . Yang, et al. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  48. [59]

    F. Wang, W. Zhou, J. Y . Huang, N. Xu, S. Zhang, H. Poon, and M. Chen. mdpo: Conditional preference optimization for multimodal large language models.arXiv preprint arXiv:2406.11839, 2024. 12

    2024 arXiv

  49. [60]

    H. Wang, C. Qu, Z. Huang, W. Chu, F. Lin, and W. Chen. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning.ArXiv Preprint, 2025

    2025

  50. [61]

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y . Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.ArXiv Preprint, 2024

    2024

  51. [62]

    X. Wang, Z. Yang, C. Feng, H. Lu, L. Li, C.-C. Lin, K. Lin, F. Huang, and L. Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement.ArXiv Preprint, 2025

    2025

  52. [63]

    W. Xiao, L. Gan, W. Dai, W. He, Z. Huang, H. Li, F. Shu, Z. Yu, P. Zhang, H. Jiang, and F. Wu. Fast-slow thinking for large vision-language model reasoning.ArXiv Preprint, 2025

    2025

  53. [64]

    S. M. Xie, H. Pham, X. Dong, N. Du, H. Liu, Y . Lu, P. S. Liang, Q. V . Le, T. Ma, and A. W. Yu. Doremi: Optimizing data mixtures speeds up language model pretraining.Advances in Neural Information Processing Systems, 36:69798–69818, 2023

    2023

  54. [65]

    Xiong, H

    W. Xiong, H. Zhang, C. Ye, L. Chen, N. Jiang, and T. Zhang. Self-rewarding correction for mathematical reasoning.arXiv preprint arXiv:2502.19613, 2025

    2025 arXiv

  55. [66]

    W. Xu, J. Wang, W. Wang, Z. Chen, W. Zhou, A. Yang, L. Lu, H. Li, X. Wang, X. Zhu, W. Wang, J. Dai, and J. Zhu. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. ArXiv Preprint, 2025

    2025

  56. [67]

    J. Yan, Y . Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y . Cheng, and Y . Zhang. Learning to reason under off-policy guidance.ArXiv Preprint, 2025

    2025

  57. [68]

    Y . Yang, X. He, H. Pan, X. Jiang, Y . Deng, X. Yang, H. Lu, D. Yin, F. Rao, M. Zhu, B. Zhang, and W. Chen. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization.ArXiv Preprint, 2025

    2025

  58. [69]

    J. Ye, P. Liu, T. Sun, J. Zhan, Y . Zhou, and X. Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance.arXiv preprint arXiv:2403.16952, 2024

    2024 arXiv

  59. [70]

    E. Yu, K. Lin, L. Zhao, J. Yin, Y . Peng, H. Wei, J. Sun, C. Han, Z. Ge, X. Zhang, D. Jiang, J. Wang, and W. Tao. Perception r1: Pioneering perception policy with reinforcement learning.ArXiv Preprint, 2025

    2025

  60. [71]

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y . Liu, W. Huang, H. Sun, Y . Su, and W. Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning bench...

    2024

  61. [72]

    Y . Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y . Yue, S. Song, and G. Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?ArXiv Preprint, 2025

    2025

  62. [73]

    Zhang, B

    K. Zhang, B. Li, P. Zhang, F. Pu, J. A. Cahyono, K. Hu, S. Liu, Y . Zhang, J. Yang, C. Li, and Z. Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024. URL https://arxiv. org/abs/2407.12772

    2024 arXiv

  63. [74]

    aha moment

    H. Zhou, X. Li, R. Wang, M. Cheng, T. Zhou, and C.-J. Hsieh. R1-zero’s "aha moment" in visual reasoning on a 2b non-sft model.ArXiv Preprint, 2025. 13 A Related Work A.1 Reinforcement Learning for LLM Reinforcement Learning from Human Feedback (RLHF) has become a foundational ...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.