Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that verifier-guided inference-time visual token scaling—letting a model zoom, crop, OCR, and re-check an image before answering—beats static single-pass MLLM inference and existing tool pipelines on fine-grained visual…

desk verdict A useful framework and dataset for verifier-guided visual tool use, but the paper's headline claim of significant gains is not supported by the reported comparisons. read the letter →

arxiv 2506.07235 v1 pith:SVSH7KJF submitted 2025-06-08 cs.CV cs.CL

Tianyi Bai , Zengjie Hu , Fupeng Sun , Jiantao Qiu , Yizhen Jiang , Guangxin He , Bohan Zeng , Conghui He
show 2 more authors
Binhang Yuan Wentao Zhang
This is my paper
classification cs.CVcs.CL
keywords inference-timescalingvisualtokenmulti-stepreasoningverifier-guideddirectpreferenceoptimizationvisiontoolusebenchmarksmarkovdecisionprocess
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-modal models usually read an image once into a fixed set of visual tokens and then answer; the paper argues this static reading is the bottleneck for fine-grained visual questions and can be replaced by a dynamic loop in which the model repeatedly decides what to look at next. The authors propose VTS-V, an inference-time framework that pairs a reasoner—which plans steps and invokes vision tools such as cropping, zooming, OCR, depth, and grounding—with a verifier trained by multi-step Direct Preference Optimization (DPO) that scores each step and decides when to stop. They build a dataset of 315K verified tool-use reasoning trajectories (VTS-SFT) and 301K preference pairs (VTS-DPO) to train reasoner and verifier. On BLINK, GPT-4o with VTS-V improves from 62.25 to 69.15 average accuracy, and fine-tuned 7B–11B open models surpass their base versions on BLINK, V*Bench, and MMStar. If the results hold, inference-time visual token scaling offers a way to add grounded, iterative visual search to existing MLLMs without relying on static token budgets.

What carries the argument

The central mechanism is a reasoner–verifier loop operating on visual tool actions. The reasoner is an MLLM that proposes, at each step, a planning sentence and an action from a ten-tool set (grounding, depth, zoom, search, crop, OCR, segmentation, captioning, similarity, overlay); the observation is the deterministic tool output. The verifier is a VLM fine-tuned with multi-step DPO: its reward for a trajectory is the summed log-ratio $r_\phi(\tau)=\eta\sum_h \log[V_\phi(t_h|s_h)/V_{\phi_0}(t_h|s_h)]+\eta\sum_h \log[V_\phi(a_h|t_h)/V_{\phi_0}(a_h|t_h)]+Q(s_1)$, and reasoning continues only while the absolute reward difference between steps is at least $\epsilon$. The same DPO objective supplies the theoretical guarantee that the loop terminates in finite steps.

What would settle it

A concrete test: take a random sample of the 315K SFT trajectories, have independent human annotators check whether the tool calls are correct and the final answers match the gold labels, retrain the verifier on only human-approved pairs, and evaluate on BLINK and V*Bench; if the scores fall substantially, the reported gains came from the self-judged trajectories rather than from verifier-guided visual token scaling.

Watch

Extended reading notes

Core claim

The central claim is that visual reasoning improves when the model is allowed to scale visual tokens at inference time under verifier control, rather than committing to a fixed image encoding up front. Formally, reasoning is cast as a Markov Decision Process whose states are complete interaction histories; at each step the reasoner generates a planning sentence, chooses a tool action from a fixed module set, and receives a deterministic visual observation, and a verifier derived from multi-step DPO decides whether the reward difference between consecutive states exceeds a threshold. The verifier is not a separate learned reward model but the DPO-trained VLM itself, whose log-probability ratios define the reward, and the authors prove the resulting process is a stopping time and terminates almost surely under a mild KL-divergence condition. The empirical claim is that this loop, applied to GPT-4o or to open models fine-tuned on VTS-SFT, outperforms static prompting, chain-of-thought, and existing tool-use pipelines on fine-grained visual reasoning benchmarks.

Load-bearing premise

The load-bearing premise is that the model used to generate and judge the training trajectories, Qwen-2.5-VL-72B, is reliable enough that its self-produced, self-filtered 315K SFT examples and 301K preference pairs teach a correct notion of good visual reasoning; if that model shares systematic visual blind spots, the reasoner and verifier inherit them and the reported gains would not transfer.

Editorial extensions

If this is right

  • On BLINK's 13 subtasks, GPT-4o + VTS-V scores 69.15 average versus 62.25 for plain GPT-4o, with the biggest gains on counting and functional/visual correspondence tasks.
  • Open 7B–11B reasoners fine-tuned on VTS-SFT beat their base versions on BLINK (for example Qwen2.5-VL-7B: 56.65 vs 49.07), V*Bench, and MMStar, indicating the loop transfers across architectures.
  • Without verifier guidance, accuracy drops (Qwen2.5-VL-7B: 54.44 to 47.09 on the reported subtasks), so the verifier's stop-and-continue decisions are load-bearing.
  • Applying the loop to untrained open models can hurt (e.g., Qwen2-VL-7B drops from 48.01 to 38.54), so VTS-SFT fine-tuning is required for open models to emit valid tool actions.
  • The stopping rule is a stopping time and terminates almost surely under the stated KL conditions, so the inference loop is guaranteed not to loop forever.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: the same loop could be applied to non-image inputs with tool-like actions, such as paging through documents or querying a video at selected timestamps; only image tools are tested here.
  • Because the same 72B model both generates and filters the training trajectories, data quality rather than the MDP/DPO machinery may be the true source of the gains; swapping in a different teacher or independent judge would test this.
  • The threshold $\epsilon$ is effectively a test-time compute budget knob; adaptively lowering it for hard questions could trade more tool calls for accuracy, which the paper does not explore.
  • Preference pairs built by deliberately corrupting later reasoning steps teach the verifier to detect local errors; a stronger verifier might come from preference pairs that differ only in their visual actions, isolating visual token scaling from text quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes VTS-V, a framework for inference-time visual token scaling in multimodal LLMs. Visual reasoning is cast as an MDP in which a reasoner proposes actions from a set of visual tools and a verifier, trained with multi-step DPO, decides when to stop. The authors introduce a large synthetic dataset, VTS (315K SFT trajectories and 301K DPO preference pairs), constructed from LLaVA-OneVision by generating tool-based reasoning traces with Qwen-2.5-VL-72B and filtering them with the same model as an LLM judge. They report gains on BLINK for GPT-4o and for fine-tuned Qwen2-VL-7B, Qwen2.5-VL-7B, and LLaMA-3.2-11B, plus results on V*Bench, MMStar, and MathVista. A finite-horizon stopping guarantee is stated in Theorem 3.2 with a proof in Appendix A.4. The main empirical claim is that VTS-V significantly outperforms existing approaches across diverse visual reasoning benchmarks.

Significance. If the empirical results are reproducible, the framework and the VTS dataset are a useful contribution to tool-augmented visual reasoning: the MDP formulation is clean, the verifier-guided stopping mechanism is a sensible design, and releasing code and data is valuable. The paper is also commendable for evaluating on multiple benchmark families and for attempting a theoretical analysis of termination. However, the headline significance claim is not established by the current evidence: the only head-to-head comparison against a strong tool-use baseline on BLINK is a +0.25 point difference without error bars, no tool-use baselines are reported on three of the four benchmark families, and the MathVista drop for LLaMA is dismissed without supporting analysis. The framework is plausible but the evidence is incomplete.

major comments (5)
  1. [Abstract; §5.2, Tables 1–2] The headline claim that VTS-V 'significantly outperforms existing approaches' is not supported by the reported comparisons. In Table 1, GPT-4o + VTS-V (69.15) exceeds GPT-4o + MMFactory (68.90) by only +0.25 points on BLINK, with no error bars, repeated runs, or significance tests. In Table 2, V*Bench, MMStar, and MathVista contain no MMFactory or comparable tool-use baseline, so superiority over 'existing approaches' is untested on three of the four benchmark families. In addition, GPT-4o + VTS-V uses extra tool calls, so the gain is not compute-matched against the baselines; the specific contribution of the verifier-guided mechanism, separate from additional inference-time computation, is therefore unclear.
  2. [§4.1–4.2] The training signal is largely self-referential. Section 4.1 states that the same Qwen-2.5-VL-72B model is used to generate the trajectories and as an LLM-as-a-judge filter, and Section 4.2 constructs rejected trajectories by prompting the model to 'proceed with incorrect reasoning steps' based on the gold answer. Any systematic bias in Qwen-2.5-VL-72B is therefore propagated into both VTS-SFT and VTS-DPO. The external benchmarks prevent circular evaluation, but no independent validation of the 315K SFT examples or 301K DPO pairs is reported: no human agreement study, no different judge, and no contamination check. Please add such an analysis or justify why the self-referential filter is reliable enough to support the training claims.
  3. [§5.2, Table 2] The MathVista result for LLaMA-3.2-11B (30.10 to 18.81) contradicts the text's statement that 'performance improves across all models' and is dismissed as 'likely LLaMA's task bias rather than a limitation of VTS-V' without supporting evidence. No per-category breakdown, no additional baselines, and no ablation with the verifier disabled on MathVista are reported. Because MathVista is one of four benchmark families and the drop is large, the generalization claim needs either a corrected narrative, additional evidence, or a modification of the framework.
  4. [§5.3, Table 3] The ablation table does not state which benchmark or metric it reports. The values 54.44, 37.21, and 38.60 coincide with the BLINK Semantic Correlation sub-scores from Table 1, not with an average over a full benchmark. The 'w/o VTS-V' row is also ambiguous: it removes the whole framework, not only visual token scaling, so it cannot separate the verifier's effect from the effect of tool use. Please report full-benchmark averages with variance over multiple runs and clarify what exactly is removed in each condition.
  5. [§3.4, Contribution bullet, Theorem A.6] The theoretical contribution is overstated relative to what is proved. Theorem 3.2 and its formal version show finiteness almost surely under condition (ii) of Theorem A.6, but the contribution bullet claims 'a bounded number of steps' and 'alignment between reasoning actions and visual content'; neither is proven, and condition (ii) is an assumption on the KL divergences that is not verified for the trained models. Please rephrase the claims or supply the missing derivation, and align Algorithm 1's stopping rule with Lemma 3.1, which uses an absolute value.
minor comments (5)
  1. [§3.3 vs. §5.1] Section 3.3 says the verifier is built on a base VLM 'such as LLaVA-v1.5-7B', while Section 5.1 says the verifier is trained on Qwen2.5-VL-7B-Instruct. Please reconcile this discrepancy, as it affects reproducibility.
  2. [Algorithm 1, Lemma 3.1] Algorithm 1 breaks when 'Delta r < epsilon', but Lemma 3.1 requires the absolute value '|Delta r| < epsilon'. Please correct the inconsistency.
  3. [Table 3] The table caption should specify the benchmark and metric, and the reported numbers should be averages with error bars or at least repeated-run variability.
  4. [Figures 3–11] Several figures in Appendix C appear to contain empty text placeholders in the rendered version, making the examples unreadable. Please re-render the figures so that tool calls, observations, and verifier scores are visible.
  5. [§5.2, Table 2] The text says LLaMA-3.2-11B + VTS-V 'fails to output any valid tool-use actions', but the table only shows dashes. Please add an explicit entry or explanation in the table itself.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central derivation and empirical claims are not reduced to their inputs by construction.

full rationale

The paper's derivation chain is an MDP formulation with standard SFT and multi-step DPO losses (Eqs. 2 and 11), plus a verifier obtained from preference optimization. The verifier is a learned reward model, not a restatement of the benchmark answers: at inference, Lemma 3.1 derives the stopping rule from the DPO reward definition, and Theorem 3.2 gives a martingale-based finiteness argument under an explicit KL assumption (Condition ii in Theorem A.6), which is stated rather than smuggled in as a conclusion. The reported accuracy gains are evaluated on external benchmarks (BLINK, V*Bench, MMStar, MathVista) that are not used for training, so the improvements are empirically falsifiable rather than forced by construction. The data-generation pipeline does use Qwen-2.5-VL-72B as both generator and judge, and DPO preferences are induced using gold answers, which is a legitimate data-quality limitation; however, this does not equate any fitted parameter with the predicted benchmark outcome, nor does it make the benchmark results a tautology. No load-bearing self-citation was found, and the mathematical results either cite external standard results or rest on explicit assumptions. Statistical significance of the headline claim is not established in the paper, but that is an evidentiary weakness, not circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim depends mainly on standard DPO and martingale mathematics plus two unverified empirical assumptions: the KL condition behind finite stopping and the correctness of self-generated, self-judged training data.

free parameters (2)
  • Stopping threshold epsilon = not reported
    Algorithm 1 line 10 uses a predetermined positive threshold for reward difference; the value is not reported and controls reasoning length and accuracy.
  • DPO reward scale eta = not reported
    Appears in the definition of r_phi(tau) and in the multi-step DPO loss; its value is not specified in the main text.
assumptions (5)
  • standard math Martingale convergence theorem (Theorem A.5, Durrett)
    Used in the proof of Theorem A.6 to show that r_phi_hat(s_h) converges almost surely.
  • standard math Bradley-Terry preference model and DPO (Rafailov et al.)
    The verifier is trained by maximizing likelihood under the Bradley-Terry model, following Rafailov et al. and the multi-turn DPO of Xiong et al.
  • domain assumption KL distance condition (ii) in Theorem A.6
    The finite-stopping guarantee requires an unproven inequality between the reasoner and verifier distributions; the paper asserts it is reasonable but does not verify it for trained models.
  • ad hoc to paper LLM-as-a-judge correctness
    The generator and judge are the same Qwen-2.5-VL-72B model; no human or independent-model validation of trajectory correctness is reported.
  • domain assumption Deterministic tool observations
    Assumes o_h = f_a_h(t_h) is deterministic code execution; reasonable for local vision modules, but not verified for all tools such as web search.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification." pith.science (2026). https://pith.science/paper/SVSH7KJF

@misc{pith2026250607235,
  author       = {Pith},
  title        = {Pith review of: Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SVSH7KJF}},
  note         = {Machine review of arXiv:2506.07235}
}
read the original abstract

Multi-modal large language models (MLLMs) have achieved remarkable capabilities by integrating visual perception with language understanding, enabling applications such as image-grounded dialogue, visual question answering, and scientific analysis. However, most MLLMs adopt a static inference paradigm, encoding the entire image into fixed visual tokens upfront, which limits their ability to iteratively refine understanding or adapt to context during inference. This contrasts sharply with human perception, which is dynamic, selective, and feedback-driven. In this work, we introduce a novel framework for inference-time visual token scaling that enables MLLMs to perform iterative, verifier-guided reasoning over visual content. We formulate the problem as a Markov Decision Process, involving a reasoner that proposes visual actions and a verifier, which is trained via multi-step Direct Preference Optimization (DPO), that evaluates these actions and determines when reasoning should terminate. To support this, we present a new dataset, VTS, comprising supervised reasoning trajectories (VTS-SFT) and preference-labeled reasoning comparisons (VTS-DPO). Our method significantly outperforms existing approaches across diverse visual reasoning benchmarks, offering not only improved accuracy but also more interpretable and grounded reasoning processes. These results demonstrate the promise of dynamic inference mechanisms for enabling fine-grained, context-aware visual reasoning in next-generation MLLMs.

Figures

Figures reproduced from arXiv: 2506.07235 by the authors.

Figure 1
Figure 1. Iterative Visual Reasoning with VTS-V. Our framework equips both open-source and closed-source models with dynamic visual token scaling and step-wise verification to solve complex visual tasks. The example shows how VTS-V: (1) decomposes questions into executable steps, (2) invokes vision tools, and (3) iteratively refines answers via verifier feedback, achieving correct results. In contrast, vanilla models fail to … view at source ↗
Figure 2
Figure 2. Pipeline for Synthetic Data Generation and Curation in VTS-V.Our data construction process consists of three stages: (1) generating multi-step reasoning trajectories with visual tool calls, (2) filtering out incorrect trajectories using an LLM-as-a-judge framework, and (3) creating contrastive (correct vs. incorrect) trajectory pairs for multi-step DPO training. rich in detailed visual token information. At the same… view at source ↗
Figure 3
Figure 3. Examples of the generated DPO data. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Additional examples of the generated DPO data. [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: Additional examples of the generated DPO data. [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Examples of generated SFT data. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 9
Figure 9. Figure 9: We demonstrate examples of reasoning on benchmark data using VTS-V. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 7
Figure 7. Figure 7: Additional examples of generated SFT data. [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: Additional examples of generated SFT data. [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 10
Figure 10. Figure 10: We demonstrate examples of reasoning on benchmark data using VTS-V. [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: We demonstrate examples of reasoning on benchmark data using VTS-V. [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

  2. Test-time Scaling over Perception: Resolving the Grounding Paradox in Thinking with Images

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    TTSP samples and filters multiple zoom-in exploration traces and iteratively consolidates validated observations into an Evidence Ledger, improving fine-grained multimodal reasoning on V* Bench, HR-Bench, TreeBench, a...

  3. OpenWorldLib: A Unified Codebase and Definition of Advanced World Models

    cs.CV 2026-04 unverdicted novelty 4.0 of 10

    OpenWorldLib offers a standardized codebase and definition for world models that combine perception, interaction, and memory to understand and predict the world.

Reference graph

Works this paper leans on

38 extracted references · 13 canonical work pages · cited by 3 Pith papers

  1. [1]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. InEuropean Conference on Computer Vision, pages 148–166. Springer, 2025

    work page 2025

  2. [2]

    V∗: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V∗: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084–13094, 2024

    work page 2024

  3. [3]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024

    2024

  4. [4]

    Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale

    Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.05237, 2024

    arXiv 2024

  5. [5]

    Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models.arXiv preprint arXiv:2403.16999, 2024

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models.arXiv preprint arXiv:2403.16999, 2024

    arXiv 2024

  6. [6]

    Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    arXiv 2023

  7. [7]

    Visual program distillation: Distilling tools and programmatic reasoning into vision-language models

    Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9590–9601, 2024

    work page 2024

  8. [8]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962, 2023

    2023

Show all 38 references
  1. [9]

    Vipergpt: Visual inference via python execution for reasoning

    D´ıdac Sur´ıs, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888– 11898, 2023

    2023

  2. [10]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  3. [11]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  4. [12]

    Scaffolding coordinates to promote vision-language coordination in large multi-modal models.arXiv preprint arXiv:2402.12058, 2024

    Xuanyu Lei, Zonghan Yang, Xinrui Chen, Peng Li, and Yang Liu. Scaffolding coordinates to promote vision-language coordination in large multi-modal models.arXiv preprint arXiv:2402.12058, 2024. 11

    2024 arXiv

  5. [13]

    Mm-react: Prompting chatgpt for multimodal reasoning and action

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023

    2023 arXiv

  6. [14]

    Making language models better reasoners with step-aware verifier

    Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 53...

    2023

  7. [15]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations

  8. [16]

    Solving math word problems via cooperative reasoning induced language models

    Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. Solving math word problems via cooperative reasoning induced language models. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V...

    2023

  9. [17]

    Let’s reward step by step: Step-level reward model as the navigators for reasoning.arXiv preprint arXiv:2310.10080, 2023

    Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Let’s reward step by step: Step-level reward model as the navigators for reasoning.arXiv preprint arXiv:2310.10080, 2023

    2023 arXiv

  10. [18]

    Math-shepherd: A label-free step-by-step verifier for llms in mathematical reasoning

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: A label-free step-by-step verifier for llms in mathematical reasoning. arXiv preprint arXiv:2312.08935, 2023

    2023 arXiv

  11. [19]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  12. [20]

    Rlcd: Reinforcement learning from contrastive distillation for lm alignment

    Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rlcd: Reinforcement learning from contrastive distillation for lm alignment. In The Twelfth International Conference on Learning Representations

  13. [21]

    Pretraining language models with human preferences

    Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley , Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. InInterna- tional Conference on Machine Learning, pages 17506–17533. PMLR, 2023

    2023

  14. [22]

    Improving large language model fine-tuning for solving math problems.arXiv preprint arXiv:2310.10047, 2023

    Yixin Liu, Avi Singh, C Daniel Freeman, John D Co-Reyes, and Peter J Liu. Improving large language model fine-tuning for solving math problems.arXiv preprint arXiv:2310.10047, 2023

    2023 arXiv

  15. [23]

    V-star: Training verifiers for self-taught reasoners.arXiv preprint arXiv:2402.06457, 2024

    Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners.arXiv preprint arXiv:2402.06457, 2024

    2024 arXiv

  16. [24]

    Building math agents with multi-turn iterative preference learning

    Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, et al. Building math agents with multi-turn iterative preference learning. arXiv preprint arXiv:2409.02392, 2024

    2024 arXiv

  17. [25]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  18. [26]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  19. [27]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 12

    2025 arXiv

  20. [28]

    Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    2024 arXiv

  21. [29]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  22. [30]

    Mmfactory: A universal solution search engine for vision-language tasks

    Wan-Cyuan Fan, Tanzila Rahman, and Leonid Sigal. Mmfactory: A universal solution search engine for vision-language tasks. arXiv preprint arXiv:2412.18072, 2024

    2024 arXiv

  23. [31]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  24. [32]

    Mathematical analysis of machine learning algorithms

    Tong Zhang. Mathematical analysis of machine learning algorithms. Cambridge University Press, 2023

    2023

  25. [33]

    Probability: theory and examples, volume 49

    Rick Durrett. Probability: theory and examples, volume 49. Cambridge university press, 2019. 13 Appendices A Proofs 15 A.1 Verifier Obtaining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.2 Proof of Lemma 3.1 . . . . . . . . . . . ....

    2019

  26. [34]

    Xn is adapted to Fn,

  27. [35]

    If in the last definition,= is replaced by ≤ or ≥, then Xn is said to be a supermartingale or submartingale, respectively

    E[Xn+1 | Fn] =Xn for all n, then Xn is said to be a martingale (with respect toFn). If in the last definition,= is replaced by ≤ or ≥, then Xn is said to be a supermartingale or submartingale, respectively . Definition A.4 (Stopping time). A random variableN is said to be a st...

  28. [36]

    Hτ ˆϕSDPO,ϕ0;ˆθSFT is a stopping time

  29. [37]

    Assume the following two conditions hold. (i) sup Eτh\{s1}∼R ˆθSFT (·|s1)r+ ˆϕSDPO (τh) < ∞ for any given s1 ∼ D, 17 (ii) for any sh+1, RˆθSFT , Vϕ0 and V ˆϕSDPO satisfy that DKL RˆθSFT (· |sh)||Vϕ0 (· |sh) ≥ DKL RˆθSFT (· |sh)||V ˆϕSDPO (· |sh) , and DKL RˆθSFT (· |th)||Vϕ0 (...

  30. [38]

    log V ˆϕSDPO (th | sh) Vϕ0 (th | sh) +log V ˆϕSDPO (ah | th) Vϕ0 (ah | th) # . Observe that condition (ii) implies that Eth∼R ˆθSFT(·|sh)

    By the definition ofr ˆϕSDPO , we have E h r ˆϕSDPO (sh+1) | sh i =r ˆϕSDPO (sh)+ Eth∼R ˆθSFT(·|sh),ah∼R ˆθSFT (·|th),oh=fah (th) " log V ˆϕSDPO (th | sh) Vϕ0 (th | sh) +log V ˆϕSDPO (ah | th) Vϕ0 (ah | th) # . Observe that condition (ii) implies that Eth∼R ˆθSFT(·|sh) " log V...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.