Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SynthRL: harder verified math questions add 1 point of VLM accuracy

desk verdict A plausible data-synthesis pipeline for VLM RLVR, but the 'guaranteed' verifier is a 4/16 self-rollout threshold and the headline gains are small and inconsistent across benchmarks. read the letter →

arxiv 2506.02096 v1 pith:YQMUAHGA submitted 2025-06-02 cs.LG cs.CLcs.CV

Zijian Wu , Jinjie Ni , Xiangyan Liu , Zichen Liu , Hang Yan , Michael Qizhe Shieh This is my paper
classification cs.LGcs.CLcs.CV
keywords visualmathreasoningvision-languagemodelsreinforcementlearningwithverifiablerewardsdatasynthesisdifficultycalibrationMonteCarloverificationscalingout-of-domaingeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that RLVR training data for visual math reasoning can be scaled automatically by converting easy seed questions into harder ones, as long as each candidate is verified to preserve the original answer and to be genuinely more difficult. Its pipeline selects seeds the target model already solves reliably (at least 12 of 16 Monte Carlo rollouts), prompts a strong synthesizer to create variants without seeing the answer, and accepts only candidates with a rollout pass count between 4 and c_ori − 2. Applied to 8,072 MMK12 seeds, this produced 3,380 accepted variants, and models trained on seed plus synthesized data averaged 58.0% accuracy across five out-of-domain visual math benchmarks versus 57.0% for seed-only training. The authors interpret the gain, which is largest on medium and hard evaluation items, as evidence that challenge-matched synthetic data elicits deeper reasoning rather than merely adding volume.

What carries the argument

The load-bearing mechanism is the Monte Carlo rollout pass count C_pass(I,Q,A; pi_target), the number of times the target model outputs the ground-truth answer in N=16 stochastic rollouts of a question. The pipeline uses this one metric twice: first to select seeds with c_ori ≥ 12, and then to verify candidates with the two-sided criterion 4 ≤ c_cand ≤ c_ori − 2. The same metric carries both the correctness guarantee and the difficulty guarantee, which is what makes the synthesis automated and model-relative rather than dependent on human labeling or on an external answer key.

What would settle it

Take a random sample of the 3,380 accepted synthesized questions and have independent human solvers, or a stronger held-out verifier, check two things: whether the original answer is the only correct answer, and whether each variant is actually harder than its seed. If a substantial fraction of accepted samples fail either check, the two-threshold verifier is too weak, and the reported benchmark gains would not be attributable to difficulty-matched correct data.

Watch

Extended reading notes

Core claim

The central claim is that, under RLVR, augmenting a seed dataset with synthesized questions that are verified to be harder and answer-preserving improves out-of-domain visual math reasoning beyond what the same amount of additional seed data would provide. The verification uses the target model's Monte Carlo pass count in N=16 rollouts: a candidate passes only if it is solved at least T_min=4 times and its pass count is at least Delta_hard=2 lower than the seed question's pass count. Because the accepted question must be solvable by the very model being trained, the synthetic examples sit in a difficulty band just beyond the model's current reliable reach, which the paper argues supplies a stronger learning signal than the extreme easy and hard samples that dominate the original distribution. The empirical evidence is that A-MMK12 beats seed-only MMK12 at 2K, 4K, and 8K scales, with the 8K gap being +1.0% average accuracy, and that the advantage concentrates in the medium and hard difficulty tiers.

Load-bearing premise

The load-bearing premise is that a candidate the target model solves in 4 of 16 tries is reliably correct and genuinely harder than its seed, even though the same model is both the judge and the learner being trained.

Editorial extensions

If this is right

  • At the 8K scale, seed-plus-synthesized training reaches 58.0% average accuracy on the five out-of-domain benchmarks, compared with 57.0% for seed-only training, with single-benchmark gains of +1.9% on MathVerse and +2.0% on WeMath.
  • The advantage grows with seed data volume: +0.2% at 2K, +0.7% at 4K, and +1.0% at 8K, meaning synthesis complements rather than substitutes for data scaling.
  • At 8K, A-MMK12 is −0.5% on easy, +1.7% on medium, and +1.6% on hard examples averaged across benchmarks, so the benefit concentrates on harder items.
  • Ablations show verification matters: replacing the target-model verifier with a different model drops the 4K average from 57.2% to 55.7%, and removing verification entirely gives 55.8%.
  • Replacing seed questions with synthesized ones at the same dataset size underperforms seed-only training (56.1% vs 56.5%), so synthesized data helps by augmenting the original distribution rather than substituting for it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because difficulty is defined relative to the current target model, the method is a moving target: the same pipeline run after further RL training would select different seeds and accept different variants, so iterative synthesis-and-train loops could compound or saturate the gain. The paper does not test this loop.
  • The T_min=4 threshold accepts questions the model gets wrong 75% of the time, so some accepted candidates may be ambiguous or mis-keyed; one way to isolate the mechanism is to train on accepted candidates with answers flipped and see whether the benefit persists.
  • Synthesis quality and cost were explored with one strong synthesizer and seed scales up to 8K, so the paper does not establish how marginal gain per accepted question depends on synthesizer strength; varying the synthesizer and measuring per-sample gain would test this.
  • Because accepted questions are, by construction, just beyond the model's reliable solving rate, the accepted set could be ordered by pass count and used as an automatic curriculum; whether such ordering outperforms shuffled data is a direct, untested consequence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. SynthRL proposes a three-stage pipeline for scaling RLVR training data for vision-language models: (1) difficulty-based seed selection using Monte Carlo rollout pass counts of the target model, (2) synthesis of more challenging question variants with a strong VLM while withholding the ground-truth answer, and (3) a 'guaranteed' verification stage that accepts a candidate only if the target model solves it at least T_min times out of N rollouts and with a pass count no higher than c_ori - Delta_hard. The method is applied to the MMK12 dataset, producing 3,380 additional questions from roughly 8K seeds, and models are trained with GRPO on the augmented set. The main empirical claim is that at the 8K scale the augmented dataset improves average accuracy from 57.0% to 58.0% across five out-of-domain visual math benchmarks, with the largest gains on harder examples. The paper also reports ablation studies on the verifier and on replacement versus augmentation data strategies.

Significance. If the claims hold, SynthRL would be a useful, low-cost method for automatically expanding RLVR training data with difficulty control, and the public release of code and data is a practical contribution. The paper's strengths include a clear pipeline description, a detailed difficulty-based analysis of where gains occur, and ablation experiments that attempt to isolate the role of verification. However, the central 'guarantee' claim is not supported by the verification procedure, and the reported empirical gains are small, inconsistent across benchmarks, and reported without error bars or multiple seeds. The contribution is therefore real but currently overstated; the paper needs either a substantially strengthened verification argument or a careful re-scoping of its claims.

major comments (5)
  1. [Section 3.4 and Abstract] The abstract and introduction call the verification stage 'guaranteed' and claim 'near-perfect correctness,' but the correctness criterion is ccand >= Tmin with Tmin = 4 out of N = 16 rollouts of the same Qwen2.5-VL-7B-Instruct model that is later trained on the accepted data. A 25% success rate is not a correctness guarantee: it admits candidates for which the model fails 75% of rollouts, which may be ambiguous, malformed, or have a different ground truth. The paper's own Section 3.4 only says the threshold provides 'strong evidence' of validity. This mismatch is load-bearing because the method's name and stated contribution rest on the verification guarantee. Please either replace the 'guaranteed' language with a precise statistical claim, or strengthen the verification (e.g., by human or stronger-model audit of a sample) and report the estimated label-corruption rate.
  2. [Table 1 and Section 5.2] The claim of 'consistent gains across five out-of-domain benchmarks' is contradicted by the reported numbers. At the 8K scale, MathVision degrades from 30.0 to 29.6 and MathVista improves by only +0.3; at 4K, MathVision drops from 29.8 to 29.0; at 2K, MathVerse drops from 51.1 to 50.5 and WeMath from 70.2 to 68.7. The only benchmark that improves at all three scales is DynaMath. The headline +1.0 average gain at 8K is driven primarily by MathVerse (+1.9) and WeMath (+2.0). Please either report per-benchmark consistency with uncertainty estimates or revise the 'consistent' claim to reflect the observed pattern.
  3. [Section 5.2 and Figure 6] The paper reports peak performance over training checkpoints selected by the best average on the five evaluation benchmarks themselves (Section 5.1). With a reported gain of only +1.0 average and individual benchmark differences ranging from -0.4 to +2.0, best-checkpoint selection may inflate the apparent improvement. No error bars, confidence intervals, or multiple seeds are reported. Please provide results averaged over at least three seeds with standard deviations, and either use a fixed number of training steps or clearly separate checkpoint selection from evaluation.
  4. [Section 5.4 and Table 3] The ablation 'w/ non-target verifier' replaces the verifier with Gemini-2.0-Flash-001, which changes both the correctness check and the difficulty calibration relative to the target model. The observed drop from 57.2% to 55.7% cannot be attributed specifically to the loss of the 'guarantee'; it may simply reflect that the non-target verifier accepts different (possibly less well-calibrated) data. Similarly, the comparison between single-pass (56.5%) and full Monte Carlo (57.2%) verification is a 0.7-point difference with no variance estimate. Please decompose the ablation to show which component of the verification (solvability threshold, difficulty threshold, number of rollouts) drives the improvement, and report uncertainty.
  5. [Section 4.2 and Table 2] The difficulty-based performance analysis uses an Elo rating derived from Gemini-2.0-Flash-001 pairwise comparisons on the same evaluation benchmarks, and the partition into easy/medium/hard is defined after the fact. The claim that gains concentrate on medium and hard examples (Table 2) is sensitive to the chosen Elo thresholds, and the reported deltas at 2K are within noise (e.g., easy -0.2, medium -0.3, hard +0.6). Please provide the distribution of Elo ratings, the sensitivity of Table 2 to the thresholds, and an analysis that does not rely on the same judge model used elsewhere in the pipeline.
minor comments (6)
  1. [Figure 1] The caption says the chart compares performance of five different models, but the 'Improvement over Baseline' is not defined for the external models; please state explicitly that the baseline is Qwen2.5-VL-7B-Instruct and specify how improvements were computed for models with reported numbers.
  2. [Figure 6] The caption refers to 'peak performance ... indicated by stars and markers,' but the figure itself does not clearly distinguish stars from markers; please add a legend or label the points directly.
  3. [Algorithm 1 (Appendix H)] The pseudocode includes a quality threshold T_quality and a judge model M_judge, but Section 4 does not describe how the quality score is computed or what value of T_quality was used. Please either remove this step from the algorithm or document its implementation.
  4. [Section 2] There are minor naming inconsistencies: 'MA VIS' should be 'MAVIS', and 'LLaV A-OneVision-7B' should be 'LLaVA-OneVision-7B' consistently throughout the text and Table 1.
  5. [Table 1] The caption mentions color-coding for SFT data, RL data, and synthesized RL data, but no color legend is visible in the table; please add a legend or remove the reference to colors.
  6. [Section 5.1] The setup states that 'we train every dataset with sufficient training steps until convergence' but also reports different training episodes by dataset size (Appendix G). Please clarify the stopping criterion and how convergence was determined.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; verified-synthesis pipeline is an empirical augmentation method whose reported gains rest on external OOD benchmarks, not on a self-referential reduction.

full rationale

The paper's central claim is empirical and evaluated externally: Table 1 reports that training on A-MMK12 (MMK12 plus 3.3K synthesized, verified samples) reaches 58.0% average accuracy versus 57.0% for seed-only MMK12 at the 8K scale across five out-of-domain benchmarks (MathVerse, MathVision, MathVista, WeMath, DynaMath). Those benchmarks are not generated by the pipeline, and the comparison is not an algebraic consequence of any equation in Section 3. The verification criteria in Section 3.4 (c_cand >= 4 and c_cand <= c_ori - 2) are operational thresholds: they define acceptance, and the data-analysis plots in Section 4.2 restate the resulting pass-count distribution, which is descriptive rather than a prediction. The paper does not claim to derive benchmark accuracy from those thresholds. The only self-citation, Liu et al. (2025a) (Noisyrollout), appears in a related-work survey sentence and carries no load-bearing premise. The seed dataset is attributed to Meng et al. (external) and the RL algorithm to Shao et al./GRPO. The use of the target model as its own verifier raises a validity question about the strength of the correctness guarantee, but that is a modeling assumption, not a circular derivation: no equation or fitted parameter is equivalent by construction to the reported result. Consequently no specific circular step can be quoted, and the honest finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The pipeline depends on several hand-set thresholds and on model-based judgment for correctness and difficulty. No new physical entities, forces, or conserved quantities are introduced.

free parameters (6)
  • N (number of rollouts) = 16
    Number of Monte Carlo rollouts used for pass-count estimation in Equations 2 and 3; chosen by default with no sensitivity analysis.
  • Seed selection threshold c_ori = 12
    Questions with pass count at least 12 out of 16 are treated as easy and selected for synthesis; hand-chosen in Section 3.2.
  • Verification solvability threshold T_min = 4
    Minimum number of successful rollouts (out of 16) for a candidate question to be accepted as preserving the answer; hand-chosen in Section 4.1.
  • Difficulty increase threshold Delta_hard = 2
    Required reduction in pass count from original to candidate to accept a difficulty increase; hand-chosen in Section 4.1.
  • Battles per sample k for difficulty ranking = 128
    Number of pairwise Gemini-judged comparisons per benchmark sample in the Elo rating procedure; hand-chosen in Appendix E.
  • Elo difficulty boundaries = 950 and 1050
    Thresholds used to categorize benchmark questions into easy, medium, and hard tiers for the difficulty analysis; hand-chosen in Appendix E.4.
assumptions (4)
  • domain assumption Rollout pass count is a valid proxy for both question difficulty and answer correctness for the target model.
    Sections 3.2 and 3.4 define difficulty and correctness only through Monte Carlo pass counts; no external ground truth is used.
  • domain assumption Gemini-2.5-Flash-Preview can generate meaningfully harder questions that preserve the original answer.
    The synthesizer is prompted to do so, but the only check is the target model's pass count against the stored answer.
  • domain assumption The same base model serving as verifier and training initialization gives a difficulty signal that transfers to OOD benchmarks.
    Used in Sections 3.4 and 5.1; the ablation with a non-target verifier shows alignment matters.
  • domain assumption Gemini-2.0-Flash-001 provides reliable answer parsing and difficulty judgments.
    Used as judge for response evaluation in Section 5.1 and for Bradley-Terry difficulty labels in Appendix E.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis." pith.science (2026). https://pith.science/paper/YQMUAHGA

@misc{pith2026250602096,
  author       = {Pith},
  title        = {Pith review of: SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YQMUAHGA}},
  note         = {Machine review of arXiv:2506.02096}
}
read the original abstract

Vision-language models (VLMs) trained via reinforcement learning with verifiable reward (RLVR) have shown notable progress in scaling test-time compute effectively. In this work, we investigate how synthesized RL data can further improve RLVR. To this end, we propose \textbf{SynthRL}-a scalable and guaranteed pipeline for automatic data scaling in reasoning-oriented RL training. SynthRL comprises three key stages: (1) selecting seed questions with appropriate distribution, (2) augmenting them into more challenging variants while preserving the original answers, and (3) a guaranteed verification stage that ensures near-perfect correctness and difficulty enhancement. Our empirical experiments demonstrate SynthRL's scalability and effectiveness. When applied to the MMK12 dataset, SynthRL synthesizes over 3.3K additional verifiable, challenging questions from approximately 8K seed samples. Models trained with our synthesized data achieve consistent gains across five out-of-domain visual math reasoning benchmarks, with a significant improvement over baseline models trained on seed data alone. Notably, detailed analysis reveals that the gains are more pronounced on the most challenging evaluation samples, highlighting SynthRL's effectiveness in eliciting deeper and more complex reasoning patterns.

Figures

Figures reproduced from arXiv: 2506.02096 by the authors.

Figure 1
Figure 1. Improvement over baseline Qwen2.5-VL-7B-Instruct on five out-of-domain visual mathe [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of our SynthRL pipeline. (1) Difficulty-based Seed Selection identifies suitable questions based on Monte Carlo rollout pass rates, (2) Data Synthesizer transforms selected questions into more challenging variants while preserving the original answer A, and (3) Correctness and Difficulty Guaranteed Verifier ensures both answer preservation and increased difficulty. models like GPT-4o (Hurst et al., 2024… view at source ↗
Figure 3
Figure 3. Distribution of rollout pass count on MMK12. Difficulty-aware selection. For each question-answer pair (I, Qori, A) in the processed dataset Dseed, we compute its rollout pass count cori = Cpass(I, Qori, A; πtarget) using Equation 2 with respect to the target model πtarget. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Pass rate distributions across datasets. The left figure compares the original MMK12 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Distribution of reasoning steps be￾tween selected seed questions and synthesized questions. The most notable difference appears in the distri￾bution shape. The seed dataset shows high con￾centrations at the extreme ends of 0 and 16 passes, while synthesized questions d…
Figure 6
Figure 6. Figure 6: Performance on evaluation benchmarks across training steps for models trained on seed [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Pass rate distributions for the 4K dataset (4096 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Pass rate distributions for the 2K dataset (2048 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Distribution of reasoning steps between selected seed questions and synthesized questions [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Comparison of SynthRL generated harder question and original question, case 1. [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Comparison of SynthRL generated harder question and original question, case 2. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Comparison of SynthRL generated harder question and original question, case 3. [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Comparison of SynthRL generated harder question and original question, case 4. [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    RLVR training on 64,000 procedurally generated Trace instances improves Qwen2.5-VL macro-average on 24 external visual reasoning benchmarks by 3.51 points at 3B and 4.06 points at 7B.

Reference graph

Works this paper leans on

76 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  2. [2]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  3. [3]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  4. [4]

    Phi-4-reasoning technical report

    Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phi-4-reasoning technical report. arXiv preprint arXiv:2504.21318, 2025

    arXiv 2025

  5. [5]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 0 23716--23736, 2022

    2022

  6. [6]

    Claude 3.7 sonnet

    Anthropic. Claude 3.7 sonnet. https://www.anthropic.com, 2025

    work page 2025

  7. [7]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  8. [8]

    A survey of multimodal large language model from a data-centric perspective

    Tianyi Bai, Hao Liang, Binwang Wan, Yanran Xu, Xi Li, Shiyu Li, Ling Yang, Bozhou Li, Yifan Wang, Bin Cui, et al. A survey of multimodal large language model from a data-centric perspective. arXiv preprint arXiv:2405.16640, 2024

    arXiv 2024

Show all 76 references
  1. [9]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  2. [10]

    Sft or rl? an early investigation into training r1-like reasoning large vision-language models

    Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training r1-like reasoning large vision-language models. https://github.com/UCSC-VLAA/VLAA-Thinking, 2025 a

    2025

  3. [11]

    R1-v: Reinforcing super generalization ability in vision-language models with less than \ 3

    Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than \ 3. https://github.com/Deep-Agent/R1-V, 2025 b . Accessed: 2025-02-02

    2025

  4. [12]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  5. [13]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference...

    2024

  6. [14]

    Biomedical visual instruction tuning with clinician preference alignment, 2024

    Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang, Hui Ren, Quanzheng Li, Xiang Li, and Carl Yang. Biomedical visual instruction tuning with clinician preference alignment, 2024. URL https://arxiv.org/abs/2406.13173

    2024 arXiv

  7. [15]

    R-cot: Reverse chain-of-thought problem generation for geometric reasoning in large multimodal models, 2024

    Linger Deng, Yuliang Liu, Bohan Li, Dongliang Luo, Liang Wu, Chengquan Zhang, Pengyuan Lyu, Ziyang Zhang, Gang Zhang, Errui Ding, Yingying Zhu, and Xiang Bai. R-cot: Reverse chain-of-thought problem generation for geometric reasoning in large multimodal models, 2024. URL https...

    2024 arXiv

  8. [16]

    Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement, 2025

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement, 2025. URL https://arxiv.org/abs/2503.17352

    2025 arXiv

  9. [17]

    How abilities in large language models are affected by supervised fine-tuning data composition

    Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:2310.05492, 2023

    arXiv

  10. [18]

    What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning, 2025

    Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, and Ji-Rong Wen. What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning, 2025. URL https://arxiv.org/abs/2311.01487

    2025 arXiv

  11. [19]

    Solution of a ranking problem from binary comparisons

    Lester R Ford Jr. Solution of a ranking problem from binary comparisons. The American Mathematical Monthly, 64 0 (8P2): 0 28--33, 1957

    1957

  12. [20]

    Gemini: a family of highly capable multimodal models

    Gemini Team . Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  13. [21]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  14. [22]

    Bias in large language models: Origin, evaluation, and mitigation

    Yufei Guo, Muzhe Guo, Juntao Su, Zhou Yang, Mengqiu Zhu, Hongfei Li, Mengyang Qiu, and Shuo Shuo Liu. Bias in large language models: Origin, evaluation, and mitigation. arXiv preprint arXiv:2411.10915, 2024

    2024 arXiv

  15. [23]

    Minimax-optimal inference from partial rankings

    Bruce Hajek, Sewoong Oh, and Jiaming Xu. Minimax-optimal inference from partial rankings. Advances in Neural Information Processing Systems, 27, 2014

    2014

  16. [24]

    Multi-modal synthetic data training and model collapse: Insights from vlms and diffusion models, 2025

    Zizhao Hu, Mohammad Rostami, and Jesse Thomason. Multi-modal synthetic data training and model collapse: Insights from vlms and diffusion models, 2025. URL https://arxiv.org/abs/2505.08803

    2025 arXiv

  17. [25]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  18. [26]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  19. [27]

    Kimi k1.5: Scaling reinforcement learning with llms, 2025 a

    Kimi Team . Kimi k1.5: Scaling reinforcement learning with llms, 2025 a . URL https://arxiv.org/abs/2501.12599

    2025 arXiv

  20. [28]

    Kimi-VL technical report, 2025 b

    Kimi Team . Kimi-VL technical report, 2025 b . URL https://arxiv.org/abs/2504.07491

    2025 arXiv

  21. [29]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  22. [30]

    Llava-next: Stronger llms supercharge multimodal capabilities in the wild, May 2024 a

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capabilities in the wild, May 2024 a . URL https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/

    2024

  23. [31]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024 b

    2024 arXiv

  24. [32]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day, 2023 a

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day, 2023 a . URL https://arxiv.org/abs/2306.00890

    2023 arXiv

  25. [33]

    Textbind: Multi-turn interleaved multimodal instruction-following in the wild, 2024 c

    Huayang Li, Siheng Li, Deng Cai, Longyue Wang, Lemao Liu, Taro Watanabe, Yujiu Yang, and Shuming Shi. Textbind: Multi-turn interleaved multimodal instruction-following in the wild, 2024 c . URL https://arxiv.org/abs/2309.08637

    2024 arXiv

  26. [34]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023 b

    2023

  27. [35]

    Vlfeedback: A large-scale ai feedback dataset for large vision-language models alignment, 2024 d

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. Vlfeedback: A large-scale ai feedback dataset for large vision-language models alignment, 2024 d . URL https://arxiv.org/abs/2410.09421

    2024 arXiv

  28. [36]

    Understanding and mitigating the bias inheritance in llm-based data augmentation on downstream tasks

    Miaomiao Li, Hao Chen, Yang Wang, Tingyuan Zhu, Weijia Zhang, Kaijie Zhu, Kam-Fai Wong, and Jindong Wang. Understanding and mitigating the bias inheritance in llm-based data augmentation on downstream tasks. arXiv preprint arXiv:2502.04419, 2025 a

    2025 arXiv

  29. [37]

    Limr: Less is more for rl scaling

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502.11886, 2025 b

    2025 arXiv

  30. [38]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36: 0 34892--34916, 2023 a

    2023

  31. [39]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26296--26306, 2024

    2024

  32. [40]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15685, 2023 b

    2023 arXiv

  33. [41]

    Noisyrollout: Reinforcing visual reasoning with data augmentation, 2025 a

    Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation, 2025 a . URL https://arxiv.org/abs/2504.13055

    2025

  34. [42]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025 b

    2025 arXiv

  35. [43]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  36. [44]

    Ursa: Understanding and verifying chain-of-thought reasoning in multimodal mathematics

    Ruilin Luo, Zhuofan Zheng, Yifan Wang, Yiyao Yu, Xinzhe Ni, Zicheng Lin, Jin Zeng, and Yujiu Yang. Ursa: Understanding and verifying chain-of-thought reasoning in multimodal mathematics. arXiv preprint arXiv:2501.04686, 2025

    2025

  37. [45]

    Mmevol: Empowering multimodal large language models with evol-instruct

    Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, et al. Mmevol: Empowering multimodal large language models with evol-instruct. arXiv preprint arXiv:2409.05840, 2024

    2024 arXiv

  38. [46]

    Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  39. [47]

    Iterative ranking from pair-wise comparisons

    Sahand Negahban, Sewoong Oh, and Devavrat Shah. Iterative ranking from pair-wise comparisons. Advances in neural information processing systems, 25, 2012

    2012

  40. [48]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025

    2025 arXiv

  41. [49]

    We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024

    Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024

    2024 arXiv

  42. [50]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  43. [51]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  44. [52]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  45. [53]

    Math-llava: Bootstrapping mathematical reasoning for multimodal large language models

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294, 2024

    2024 arXiv

  46. [54]

    Some rank order tests which are most powerful against specific parametric alternatives

    Milton E Terry. Some rank order tests which are most powerful against specific parametric alternatives. The Annals of Mathematical Statistics, pp.\ 346--366, 1952

    1952

  47. [55]

    Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving

    Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. Advances in Neural Information Processing Systems, 37: 0 7821--7846, 2024

    2024

  48. [56]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37: 0 95095--95169, 2024 a

    2024

  49. [57]

    Agri-llava: Knowledge-infused large multimodal assistant on agricultural pests and diseases, 2024 b

    Liqiong Wang, Teng Jin, Jinyu Yang, Ales Leonardis, Fangyi Wang, and Feng Zheng. Agri-llava: Knowledge-infused large multimodal assistant on agricultural pests and diseases, 2024 b . URL https://arxiv.org/abs/2412.02158

    2024 arXiv

  50. [58]

    Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement, 2025

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement, 2025. URL https://arxiv.org/abs/2504.07934

    2025 arXiv

  51. [59]

    Qurating: Selecting high-quality data for training language models

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739, 2024

    2024 arXiv

  52. [60]

    Less: Selecting influential data for targeted instruction tuning

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333, 2024

    2024 arXiv

  53. [61]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  54. [62]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024

    2024 arXiv

  55. [63]

    R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615, 2025

    2025 arXiv

  56. [64]

    Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319, 2024

    2024 arXiv

  57. [65]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  58. [66]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks

    Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025

    2025 arXiv

  59. [67]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URL https://arxiv.org/abs/2503.18892

    2025 arXiv

  60. [68]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937, 2025

    2025 arXiv

  61. [69]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp.\ 169--186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp.\ 169--186. Sprin...

    2024

  62. [70]

    Mavis: Mathematical visual instruction tuning

    Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, et al. Mavis: Mathematical visual instruction tuning. arXiv e-prints, pp.\ arXiv--2407, 2024 b

    2024

  63. [71]

    Easyr1: An efficient, scalable, multi-modality rl training framework

    Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyr1: An efficient, scalable, multi-modality rl training framework. https://github.com/hiyouga/EasyR1, 2025

    2025

  64. [72]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36: 0 55006--55021, 2023

    2023

  65. [73]

    Navgpt-2: Unleashing navigational reasoning capability for large vision-language models, 2024

    Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. Navgpt-2: Unleashing navigational reasoning capability for large vision-language models, 2024. URL https://arxiv.org/abs/2407.12366

    2024 arXiv

  66. [74]

    Anyprefer: An agentic framework for preference data synthesis, 2025

    Yiyang Zhou, Zhaoyang Wang, Tianle Wang, Shangyu Xing, Peng Xia, Bo Li, Kaiyuan Zheng, Zijian Zhang, Zhaorun Chen, Wenhao Zheng, Xuchao Zhang, Chetan Bansal, Weitong Zhang, Ying Wei, Mohit Bansal, and Huaxiu Yao. Anyprefer: An agentic framework for preference data synthesis, 2...

    2025 arXiv

  67. [75]

    Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models, 2024

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models, 2024

    2024

  68. [76]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.