Pith. sign in

REVIEW 3 major objections 5 minor 56 references

ToolVision: Learning When and How to Use Visual Tools with Capability-Aligned Supervision

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training a small vision-language model to use visual tools well requires filtering teacher demonstrations and rewards by what the student itself can exploit.

desk verdict Genuinely new training recipe with solid internal ablations; the 'beats a 32B model' claim rests on unmatched evaluation protocols and is not yet robust. read the letter →

arxiv 2608.08907 v1 pith:4C2AVCDT submitted 2026-08-09 cs.CV cs.AI

Delin Mao , Chenghao Sun , Jingwei Song , Chishui Chen , Linfeng Zhang This is my paper · ORCID
classification cs.CVcs.AI
keywords visualtoolusethinkingwithimagescapabilityalignmentevidencegainmust-use-toolrewardmultimodallargelanguagemodelssupervisedfine-tuningreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard "distill tool-using behavior from a strong teacher, then reinforce with outcome rewards" recipe fails for small multimodal models in two distinct ways, and that both failures can be traced to a single cause: supervision is not aligned with the learner's own capabilities. During supervised fine-tuning, a student may imitate a teacher's tool calls without being able to reproduce the perception that made them useful; during reinforcement learning, outcome-only rewards suppress tool use while blanket tool bonuses reward ineffective operations. ToolVision fixes each stage with an automatic, capability-conditioned signal: SFT keeps only trajectories whose stepwise evidence gain helps student-scale models, and RL rewards successful tool use only on questions where paired rollouts show the frozen model actually benefits from tools. The resulting 8B model improves over its base on all seven main benchmarks, beats three closely related code-as-tool systems on high-resolution benchmarks, and surpasses a 32B reasoning model on two benchmarks. If these results hold, the recipe offers a general way to align teacher demonstrations and reward signals with any learner's abilities, without extra human annotations.

What carries the argument

The load-bearing object is a pair of capability-sensitive supervision signals built on the same toolbelt. First, the stepwise evidence gain $\Delta_t = J(\mathcal{E}_t;x) - J(\mathcal{E}_{t-1};x)$, where $J$ is the average answerability score of an eight-member cross-scale committee—including student-scale models—over the accumulated visual evidence $\mathcal{E}_t$; branches with negative or repeated zero gain are pruned, so only trajectories whose evidence a student can exploit enter SFT. Second, the must-use-tool (MUT) weight $w(x,\pi_{\text{SFT}})$, computed before RL by comparing the frozen SFT model's rollouts with tools disabled and enabled, and used to multiply a tool-success reward $R_{\text{mut}}$. Both signals are constructed automatically from public task data, and together they make the RL reward depend on both the question and the learner.

What would settle it

Re-run the external baselines (Thyme-7B, CodeVision-8B, CodeDance-7B, and Qwen3-VL-32B-Thinking) under ToolVision's exact avg@4, temperature-0.7 evaluation harness. If, under that protocol, ToolVision-8B no longer beats all of them on the three high-resolution benchmarks, the headline superiority claim collapses, even if the internal ablations still support the method's value.

Watch

Extended reading notes

Core claim

ToolVision-8B's central claim is that capability misalignment, not tool engineering, is what prevents small multimodal models from learning to use visual tools well. The paper proposes replacing unselective imitation and outcome-only reinforcement with two capability-aligned signals: trajectory filtering by stepwise evidence gain during SFT, and model-conditioned must-use-tool rewards during RL. On all seven main benchmarks, ToolVision-8B improves over its base (Qwen3-VL-8B-Thinking); it surpasses Thyme-7B, CodeVision-8B, and CodeDance-7B on all three high-resolution benchmarks (V*, HRBench 4K, HRBench 8K); and it outperforms Qwen3-VL-32B-Thinking on V* and HRBench 8K. The authors also report that the method improves tool-dependent tasks sharply: ArxivQA-2k accuracy rises from 58.3 to 74.5 and FSC-147 MAE drops from 44.7 to 11.6. The paper treats these numbers as evidence that routing perception through the right tool can partly offset model scale.

Load-bearing premise

The headline comparisons assume that the external baseline numbers from Thyme, CodeVision, CodeDance, and the Qwen3-VL report were produced under the same evaluation protocol as ToolVision's avg@4 multi-sample decoding at temperature 0.7; if those baselines used greedy decoding or a different sampling scheme, part of the margin may be an artifact of evaluation rather than of the training method.

Editorial extensions

If this is right

  • An 8B model can beat a 32B model on high-resolution perception by delegating localization to an external detector instead of predicting crop coordinates itself.
  • SFT data should be filtered by the student's own ability to use each step's evidence, not by whether the teacher's trajectory succeeds.
  • Reinforcement learning should reward tool use only where the frozen student model measurably benefits, avoiding both tool collapse and invalid overuse.
  • Because the must-use-tool labels are computed once and fixed, the 716-GPU-hour measurement cost is amortized across every subsequent RL run.
  • A policy trained this way specializes sharply after RL: counting invocations saturate on counting tasks, OCR on text-reading tasks, and detection-guided cropping on high-resolution search.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would extend the paired-rollout necessity measurement to inference-time gating: run the no-tool arm first, and only spend tool compute on questions the model answers incorrectly, mirroring the training filter.
  • The stepwise evidence-gain committee could serve as a dense process reward for RL, replacing the binary tool-success term with a graded signal and possibly removing the need for the offline MUT measurement.
  • The capability-alignment principle—filter demonstrations by whether the student can exploit each step—should transfer to other agentic settings such as text-only tool use or code generation, where teacher trajectories also encode abilities the student lacks.
  • I would test the SFT filter's sensitivity to committee composition by dropping all strong members and keeping only student-scale judges; the paper reports a proposer-sensitivity check but not this particular ablation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ToolVision, a two-stage method for training a small multimodal LLM (Qwen3-VL-8B-Thinking) to use visual tools through code. During SFT, a Planner–Executor multi-agent search generates candidate trajectories, and an eight-model committee scores stepwise evidence gain to prune branches; only trajectories with successful execution, positive evidence gain, and correct final answers are kept. During RL, paired rollouts of the frozen SFT model with and without tools are used to assign per-question must-use-tool (MUT) weights (0.5, 0.2, 0), and the tool-success reward is gated by these weights, with an overuse penalty. The final model is evaluated on nine benchmarks, reporting improvements over its base on all seven main benchmarks, superiority over Thyme-7B, CodeVision-8B, and CodeDance-7B on three high-resolution benchmarks, and better scores than Qwen3-VL-32B-Thinking on V* and HRBench 8K. The appendix contains detailed algorithms, data sources, prompts, and reproducibility settings.

Significance. The manuscript makes a relevant contribution to the emerging 'thinking with images' area. The core conceptual claim — that SFT should filter tool trajectories by evidence usable at the student scale, and that RL should reward tool use only where the frozen learner itself benefits — is well motivated and supported by the internal ablations. The stage-wise comparisons (Distilled SFT vs ToolVision SFT; uniform bonus vs MUT) show large, consistent gaps, and the release of code, data manifests, and detailed settings strengthens reproducibility. If the headline comparisons to external baselines are reconfirmed under a common protocol, the method would be a useful step for teaching small MLLMs to leverage specialized visual tools. However, the external comparison issue (see major comment 1) and the checkpoint-selection weakness (major comment 2) currently block full confidence in the abstract's strongest claims.

major comments (3)
  1. [Section 4.1–4.2, Table 1] The headline claim that ToolVision-8B outperforms Qwen3-VL-32B-Thinking on V* and HRBench 8K is not yet robustly supported because the baseline numbers are not matched to ToolVision's evaluation protocol. ToolVision-8B scores are avg@4 of four sampled trajectories at temperature 0.7 in a multi-turn agent harness with up to 12 turns and a reference-conditioned LLM judge, whereas the Thyme-7B, CodeVision-8B, CodeDance-7B, and Qwen3-VL-32B-Thinking numbers are cited from other reports without evidence that they used the same decoding scheme, turn budget, or semantic-equivalence judge. The reported margins over Qwen3-VL-32B-Thinking are 1.1 points on both benchmarks, while Table 6 gives per-run sampling standard deviations of 0.79 and 0.50 for ToolVision-8B; the additional uncertainty from protocol differences is unquantified and could easily be of the same magnitude. I request that the authors either re-run all external baselines under the common ToolVision harness (or provide evidence that the cited baseline numbers used an equivalent protocol) or explicitly restrict the superiority claims to comparisons made under the same evaluation conditions.
  2. [Section 4.1, Appendix H.4] The reported checkpoint is selected as the one with the highest training reward, which occurs at step 180. Because this reward includes the MUT bonus and the overuse penalty, selecting on it can favor checkpoints that overfit the training mixture rather than generalize to benchmarks. The paper should report the training-reward curve and, more importantly, evaluation results on a held-out validation set for several checkpoints around step 180, and justify that the chosen checkpoint is not an artifact of reward maximization. This is particularly relevant because the final numbers in Table 1 and Table 3 all come from this single reward-selected checkpoint.
  3. [Section 4.4, Table 3] The uniform-bonus ablation is used to support the claim that 'the benefit comes from conditioning the tool-success reward on measured necessity rather than from rewarding successful tool use indiscriminately.' While the comparison is internally consistent, only one constant weight (w=0.2) is studied. A sensitivity analysis over a few constant weights (e.g., 0.1, 0.2, 0.5) would strengthen the conclusion that the improvement is due to per-question conditioning rather than to the particular magnitude of the bonus. Without this, the reader cannot fully separate the effect of conditioning from the effect of the chosen weight scale.
minor comments (5)
  1. [Section 4.2, Table 1] The parenthesized deltas in Table 1 are said to be computed before rounding the displayed scores, but the base-model scores themselves are rounded; a brief note on how these deltas are calculated from the unrounded values would improve transparency.
  2. [Section 4.3, Table 2 and Appendix B.2] The ArxivQA-2k holdout is question-disjoint but not image-disjoint: the paper explicitly states that different questions may reuse the same source figure. This caveat should appear in the main text near Table 2, and ideally the authors should report the number of unique source figures or filter at the image level to rule out image leakage.
  3. [Figure 1] The bar lengths are scaled within each benchmark, so the visual impression of ToolVision's advantage can be misleading across columns; the absolute labels help, but a common-scaled or grouped presentation would be easier to interpret.
  4. [Appendix A.2] The statement that 'the teacher's own perception rarely needs specialist localization' is used to explain why Distilled SFT contains almost no detector-guided cropping. This claim is plausible but could be supported with a simple statistic (e.g., the percentage of teacher trajectories that invoke the detector), which would make the behavioral comparison in Figure 4 more interpretable.
  5. [Section 4.1] The term 'avg@4' is nonstandard; it is a mean over four independently scored samples rather than majority vote or best-of-k. Consider clarifying this in a footnote, since readers may otherwise assume a selection-based protocol.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the training-signal construction is self-contained and the external benchmarks are not fitted.

full rationale

The paper's derivation chain does not reduce to its inputs. The capability-aligned SFT pipeline selects trajectories by a committee evidence-gain score, but the final exported examples must also produce a correct final answer checked against external ground truth; the committee score is a filter, not a target that is later reported as a prediction. The MUT reward labels are indeed computed from rollouts of the frozen SFT model itself, which is self-referential by design, but they are fixed before RL and the benchmark evaluations are entirely external to that measurement. Observing that the trained policy uses tools more on questions where the frozen model already succeeded with tools is an intended effect of the reward definition, not an independent prediction made from fitted parameters. The headline comparisons in Table 1 copy baseline numbers from other papers; this raises a legitimate evaluation-protocol risk (different decoding, turn budgets, or judging procedures) but that is a correctness concern, not circularity. No load-bearing self-citation appears: references to Thyme, CodeVision, CodeDance, and the Qwen3-VL report are external prior work, and no uniqueness theorem or author-derived constraint is invoked to force the method's choices. The central claims are supported by ablations against distilled SFT and uniform-bonus RL on held-out benchmarks, so the derivation remains empirically grounded rather than equivalent to its own supervision by construction.

Assumptions & free parameters 10 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The method adds hand-set thresholds, committee models, and reward weights; the central empirical claim rests on those choices and on external tool services.

free parameters (10)
  • w_strong = 0.5
    Hand-set reward weight for strong MUT questions.
  • w_weak = 0.2
    Hand-set reward weight for weak MUT questions.
  • MUT thresholds = GTC>=2, NTC<=1 strong; GTC=1 weak; candidate c<=8; ordinary 9<=c<=15
    Hand-set cutoffs for labeling tool necessity from paired rollouts.
  • lambda (R_proto weight) = 0.2
    Hand-set weight for the format and protocol reward term.
  • mu (overuse penalty) = 0.05
    Hand-set penalty coefficient for tool-call overuse.
  • tau (overuse threshold) = 6
    Hand-set maximum tool calls before the overuse penalty activates.
  • beam width K = 2
    Search beam width in SFT trajectory synthesis.
  • max depth D = 6
    Maximum executed steps per SFT trajectory.
  • Planner proposal budget B = 2
    Maximum candidate actions proposed per round in the SFT search.
  • rollout counts = 16 no-tool, 8 tool-enabled, 4 eval at temperature 0.7
    Number of samples used for MUT measurement and evaluation.
assumptions (4)
  • domain assumption The committee's average match score against ground truth measures stepwise evidence gain for student-scale models.
    Section 3.2 uses J(E_t;x) to prune branches; no direct validation of the score against the student's actual learning is given, though the ToolVision SFT versus Distilled SFT ablation supports it.
  • domain assumption The contrast between tool-enabled correct trajectories and direct-answer correct trajectories estimates question-level tool necessity.
    Section 3.3 and Algorithm 2 assume the paired rollouts are representative and that the frozen policy is stable enough to define fixed MUT labels.
  • domain assumption External tools (OCR, GroundingDINO, SAM2, Depth Pro, CountGD) return correct outputs under the code interface.
    Appendix H.3 lists the services; execution errors are filtered, but silent semantic failures (for example Case 3) show reliability is not guaranteed.
  • domain assumption The nine benchmarks and their metrics measure the intended capabilities.
    Section 4.2 and Appendix C describe benchmark protocols; baseline numbers are taken from prior papers without re-running them under the same protocol.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ToolVision: Learning When and How to Use Visual Tools with Capability-Aligned Supervision." pith.science (2026). https://pith.science/paper/4C2AVCDT

@misc{pith2026260808907,
  author       = {Pith},
  title        = {Pith review of: ToolVision: Learning When and How to Use Visual Tools with Capability-Aligned Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4C2AVCDT}},
  note         = {Machine review of arXiv:2608.08907}
}
read the original abstract

Thinking with images allows a multimodal model to compensate for limited perception by invoking visual tools through code. Yet the prevailing SFT-then-RL recipe creates a different supervision misalignment at each stage. SFT is expected to teach how to use tools, but trajectories from stronger teachers may succeed through perceptual capabilities that a smaller student cannot reliably reproduce or exploit, causing the student to imitate tool-call patterns without learning how to make them useful. RL is expected to teach when to use tools, but outcome-only rewards make fallible tool execution a liability and suppress tool use, whereas a blanket bonus for every correct tool-using trajectory encourages valid but ineffective operations. To address these two misalignments, we introduce ToolVision. During SFT, a multi-agent pipeline explores candidate trajectories, and a committee including student-scale models scores stepwise evidence gain to rank and prune the search branches. Only successfully executed trajectories with correct final answers are retained for SFT. Before RL, ToolVision compares the learner's performance with and without tools, then rewards successful tool use only on questions where tools provide a clear benefit. Both signals are constructed automatically from public task data without additional human annotations of tool use or necessity. ToolVision-8B improves over its base on all seven main benchmarks, surpasses Thyme-7B, CodeVision-8B, and CodeDance-7B on all three high-resolution benchmarks, and outperforms Qwen3-VL-32B-Thinking on V* and HRBench 8K. We will publicly release the datasets and source code.

Figures

Figures reproduced from arXiv: 2608.08907 by the authors.

Figure 1
Figure 1. ToolVision-8B against its base model (Qwen3-VL [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Two failure modes in the SFT-then-RL recipe. Left: coordinate-crop failures in the distilled-then-RL policy, alongside detector-guided localization in ToolVision. Right, top: total reward and tool calls per trajectory under outcome￾only optimization. Right, bottom: ineffective-operation rates and the seven-benchmark average for the uniform-bonus variant (w ≡ 0.2 for every question) and ToolVision. Ineffective operat… view at source ↗
Figure 3
Figure 3. Overview of the capability-aligned SFT construction pipeline. Base-model successes provide direct-answer dia￾logues, while failures enter Planner–Executor search over the shared visual-tool interface. An eight-model committee scores stepwise evidence gain to retain or prune branches; validated trajectories merge with direct dialogues for SFT. 2 Related Work Visual Tool Interfaces. Systems that implement thinking wit… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Pre-RL comparison of Distilled SFT and ToolVision SFT on three high-resolution benchmarks (left to right: accuracy, detection-guided crop rate, any-crop rate, mean tool turns). Distilled SFT crops more images and takes more tool turns, yet never invokes detection-guide…
Figure 5
Figure 5. Figure 5: Capability-aligned construction of SFT trajectories. Candidate steps are executed before scoring, and only non [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Paired measurement of tool necessity and construction of the fixed RL mixture. The tool-enabled arm allows either [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: The policy system prompt, shared by SFT, RL, and evaluation. The text is verbatim; line breaks are adjusted for [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: The turn-level feedback template appended to every [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: The final-round (MUST_ANSWER) policy block and the budget block of the Planner user prompt (verbatim; the last two policy lines are appended only when the trajec￾tory already executed successfully and when a committee consensus exists, respectively). You are a multimod…
Figure 10
Figure 10. Figure 10: The complete committee-judge system prompt. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Case 1 (SFT training example). The recorded intermediate images (a)–(d) are the model’s visible image timeline. The [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Case 2 (RL rollout, SAT-2). The replayed depth visualizations show the grounded boxes and median depth estimates [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Case 3 (RL rollout, TextVQA). All three manual crops execute without error yet land on the keyboard rather than the [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 44 canonical work pages

  1. [1]

    Amini-Naieni, N.; Han, T.; and Zisserman, A. 2024. CountGD: Multi-Modal Open-World Counting. In Advances in Neural Information Processing Systems, volume 37, 48810--48837. Curran Associates, Inc

    work page 2024

  2. [2]

    Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; Cheng, Z.; Deng, L.; Ding, W.; Gao, C.; Ge, C.; et al. 2025 a . Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631

    arXiv 2025

  3. [3]

    Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025 b . Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  4. [4]

    R.; and Koltun, V

    Bochkovskii, A.; Delaunoy, A.; Germain, H.; Santos, M.; Zhou, Y.; Richter, S. R.; and Koltun, V. 2025. Depth Pro: Sharp Monocular Metric Depth in Less Than a Second. In International Conference on Learning Representations

    work page 2025

  5. [5]

    G.; and Zhang, H

    Chen, J.; Wei, F.; Zhao, J.; Song, S.; Wu, B.; Peng, Z.; Chan, S.-H. G.; and Zhang, H. 2025. Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 513--524

    work page 2025

  6. [6]

    Chen, L.; Li, J.; Dong, X.; Zhang, P.; Zang, Y.; Chen, Z.; Duan, H.; Wang, J.; Qiao, Y.; Lin, D.; and Zhao, F. 2024 a . Are We on the Right Way for Evaluating Large Vision-Language Models? In Advances in Neural Information Processing Systems, volume 37, 27056--27087. Curran Associates, Inc

    work page 2024

  7. [7]

    Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024 b . InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 24185--24198

    work page 2024

  8. [8]

    Cui, C.; Sun, T.; Lin, M.; Gao, T.; Zhang, Y.; Liu, J.; Wang, X.; Zhang, Z.; Zhou, C.; Liu, H.; et al. 2025. PaddleOCR 3.0 Technical Report. arXiv:2507.05595

    arXiv 2025

Show all 56 references
  1. [9]

    Daxberger, E.; Wenzel, N.; Griffiths, D.; Gang, H.; Lazarow, J.; Kohavi, G.; Kang, K.; Eichner, M.; Yang, Y.; Dehghan, A.; and Grasch, P. 2025. MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs. In Proceedings of the IEEE/CVF International Conference on Compute...

    2025

  2. [10]

    S.; Salehi, M.; Muennighoff, N.; Lo, K.; Soldaini, L.; et al

    Deitke, M.; Clark, C.; Lee, S.; Tripathi, R.; Yang, Y.; Park, J. S.; Salehi, M.; Muennighoff, N.; Lo, K.; Soldaini, L.; et al. 2025. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models. In Proceedings of the IEEE/CVF Conference on Computer V...

    2025

  3. [11]

    Fan, Y.; He, X.; Yang, D.; Zheng, K.; Kuo, C.-C.; Zheng, Y.; Guan, X.; and Wang, X. 2025. GRIT: Teaching MLLMs to Think with Images. In Advances in Neural Information Processing Systems, volume 38, 116522--116543. Curran Associates, Inc

    2025

  4. [12]

    Feng, J.; Huang, S.; Qu, X.; Zhang, G.; Qin, Y.; Zhong, B.; Jiang, C.; Chi, J.; and Zhong, W. 2026. ReTool: Reinforcement Learning for Strategic Tool Use in LLM s. In The Fourteenth International Conference on Learning Representations

    2026

  5. [13]

    Fu, L.; Kuang, Z.; Song, J.; Huang, M.; Yang, B.; Li, Y.; Zhu, L.; Luo, Q.; Wang, X.; Lu, H.; et al. 2025. OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning. In Advances in Neural Information Processing Systems,...

    2025

  6. [14]

    Gou, Z.; Shao, Z.; Gong, Y.; Shen, Y.; Yang, Y.; Huang, M.; Duan, N.; and Chen, W. 2024. ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . ...

    2024

  7. [15]

    Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu, R.; Zhang, R.; Ma, S.; Bi, X.; et al. 2025. DeepSeek-R1 Incentivizes Reasoning in LLM s through Reinforcement Learning. Nature, 645(8081): 633--638

    2025

  8. [16]

    Guo, Z.; Hong, M.; Zhang, F.; Jia, K.; and Jin, T. 2026. Thinking with Programming Vision: Towards a Unified View for Thinking with Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 33467--33476

    2026

  9. [17]

    Gupta, T.; and Kembhavi, A. 2023. Visual Programming: Compositional Visual Reasoning Without Training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14953--14962

    2023

  10. [18]

    Hu, Y.; Shi, W.; Fu, X.; Roth, D.; Ostendorf, M.; Zettlemoyer, L.; Smith, N.; and Krishna, R. 2024. Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models. In Advances in Neural Information Processing Systems, volume 37, 139348--139379. Curran ...

    2024

  11. [19]

    A.; and Manning, C

    Hudson, D. A.; and Manning, C. D. 2019. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2019

  12. [20]

    O.; Wang, D.; Zamani, H.; and Han, J

    Jin, B.; Zeng, H.; Yue, Z.; Yoon, J.; Arik, S. O.; Wang, D.; Zamani, H.; and Han, J. 2025. Search-R1: Training LLM s to Reason and Leverage Search Engines with Reinforcement Learning. In Second Conference on Language Modeling

    2025

  13. [21]

    Kembhavi, A.; Salvato, M.; Kolve, E.; Seo, M.; Hajishirzi, H.; and Farhadi, A. 2016. A Diagram is Worth a Dozen Images. In Computer Vision -- ECCV 2016, 235--251. Cham: Springer International Publishing

    2016

  14. [22]

    Li, L.; Wang, Y.; Xu, R.; Wang, P.; Feng, X.; Kong, L.; and Liu, Q. 2024. Multimodal A r X iv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2024

  15. [23]

    Liu, S.; Zeng, Z.; Ren, T.; Li, F.; Zhang, H.; Yang, J.; Jiang, Q.; Li, C.; Yang, J.; Su, H.; Zhu, J.; and Zhang, L. 2024 a . Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection. In Leonardis, A.; Ricci, E.; Roth, S.; Russakovsky, O.; Sattler...

    2024

  16. [24]

    Liu, Y.; Li, Z.; Huang, M.; Yang, B.; Yu, W.; Li, C.; Yin, X.-C.; Liu, C.-L.; Jin, L.; and Bai, X. 2024 b . OCRBench: on the hidden mystery of OCR in large multimodal models. Science China Information Sciences, 67(12)

    2024

  17. [25]

    Liu, Z.; Dong, Y.; Rao, Y.; Zhou, J.; and Lu, J. 2024 c . Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models. arXiv preprint arXiv:2403.12966

    2024 arXiv

  18. [26]

    Liu, Z.; Sun, Z.; Zang, Y.; Dong, X.; Cao, Y.; Duan, H.; Lin, D.; and Wang, J. 2025. Visual-RFT: Visual Reinforcement Fine-Tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2034--2044

    2025

  19. [27]

    X.; Tan, J

    Masry, A.; Long, D. X.; Tan, J. Q.; Joty, S.; and Hoque, E. 2022. C hart QA : A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, 2263--2279. Dublin, Ireland: Association for...

    2022

  20. [28]

    Mathew, M.; Bagal, V.; Tito, R.; Karatzas, D.; Valveny, E.; and Jawahar, C. 2022. InfographicVQA. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 1697--1706

    2022

  21. [29]

    Mathew, M.; Karatzas, D.; and Jawahar, C. 2021. DocVQA: A Dataset for VQA on Document Images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2200--2209

    2021

  22. [30]

    OpenAI . 2025. Introducing OpenAI o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/

    2025

  23. [31]

    C.; He, Q.; Wang, H.; Chen, X.; Hakkani-Tur, D.; Tur, G.; and Ji, H

    Qian, C.; Acikgoz, E. C.; He, Q.; Wang, H.; Chen, X.; Hakkani-Tur, D.; Tur, G.; and Ji, H. 2025. ToolRL: Reward is All Tool Learning Needs. In Advances in Neural Information Processing Systems, volume 38, 105523--105553. Curran Associates, Inc

    2025

  24. [32]

    Ranjan, V.; Sharma, U.; Nguyen, T.; and Hoai, M. 2021. Learning To Count Everything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3394--3403

    2021

  25. [33]

    Ravi, N.; Gabeur, V.; Hu, Y.-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; R\" a dle, R.; Rolland, C.; Gustafson, L.; et al. 2025. SAM 2: Segment Anything in Images and Videos. In International Conference on Learning Representations, volume 2025, 28085--28128

    2025

  26. [34]

    H.; Saha, S.; Khandelwal, N.; Jain, A.; Tarr, M

    Sarch, G. H.; Saha, S.; Khandelwal, N.; Jain, A.; Tarr, M. J.; Kumar, A.; and Fragkiadaki, K. 2025. Grounded Reinforcement Learning for Visual Reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems

    2025

  27. [35]

    Schick, T.; Dwivedi-Yu, J.; Dess \`i , R.; Raileanu, R.; Lomeli, M.; Hambro, E.; Zettlemoyer, L.; Cancedda, N.; and Scialom, T. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems, volume 36, 68539--68551. C...

    2023

  28. [36]

    Shao, H.; Qian, S.; Xiao, H.; Song, G.; Zong, Z.; Wang, L.; Liu, Y.; and Li, H. 2024 a . Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning. In Advances in Neural Information Processing Systems, volume 37...

    2024

  29. [37]

    K.; Wu, Y.; and Guo, D

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024 b . DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300

    2024 arXiv

  30. [38]

    Singh, A.; Natarajan, V.; Shah, M.; Jiang, Y.; Chen, X.; Batra, D.; Parikh, D.; and Rohrbach, M. 2019. Towards VQA Models That Can Read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2019

  31. [39]

    Song, Q.; Li, H.; Yu, Y.; Zhou, H.; Yang, L.; Bai, S.; She, Q.; Huang, Z.; and Zhao, Y. 2026. CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 19186--19195

    2026

  32. [40]

    Su, A.; Wang, H.; Ren, W.; Lin, F.; and Chen, W. 2025 a . Pixel Reasoner: Incentivizing Pixel Space Reasoning via Curiosity-Driven Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 38, 8222--8251. Curran Associates, Inc

    2025

  33. [41]

    Su, Z.; Xia, P.; Guo, H.; Liu, Z.; Ma, Y.; Qu, X.; Liu, J.; Li, Y.; Zeng, K.; Yang, Z.; Li, L.; Cheng, Y.; Ji, H.; He, J.; and Fung, Y. R. 2025 b . Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers. arXiv:2506.23918

    2025 arXiv

  34. [42]

    Sur \' s, D.; Menon, S.; and Vondrick, C. 2023. ViperGPT: Visual Inference via Python Execution for Reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 11888--11898

    2023

  35. [43]

    S.; Grover, R.; Pande, N.; and Yerramilli, S

    Tamarapalli, J. S.; Grover, R.; Pande, N.; and Yerramilli, S. 2025. CountQA: How Well Do MLLMs Count in the Wild? arXiv:2508.06585

    2025 arXiv

  36. [44]

    Wang, W.; Ding, L.; Zeng, M.; Zhou, X.; Shen, L.; Luo, Y.; Yu, W.; and Tao, D. 2025. Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, ...

    2025

  37. [45]

    Wu, J.; Guan, J.; Feng, K.; Liu, Q.; Wu, S.; Wang, L.; Wu, W.; and Tan, T. 2025. Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing. In Advances in Neural Information Processing Systems, volume 38, 143297--143330. Curran Associates, Inc

    2025

  38. [46]

    Wu, P.; and Xie, S. 2024. V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 13084--13094

    2024

  39. [47]

    Yang, J.; Zhang, H.; Li, F.; Zou, X.; Li, C.; and Gao, J. 2023. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. arXiv preprint arXiv:2310.11441

    2023 arXiv

  40. [48]

    R.; and Cao, Y

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K. R.; and Cao, Y. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net

    2023

  41. [49]

    Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Dai, W.; Fan, T.; Liu, G.; Liu, J.; et al. 2025. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. In Advances in Neural Information Processing Systems, volume 38, 113222--113244. Curran Associates, Inc

    2025

  42. [50]

    A.; Hu, K.; Liu, S.; Zhang, Y.; Yang, J.; Li, C.; and Liu, Z

    Zhang, K.; Li, B.; Zhang, P.; Pu, F.; Cahyono, J. A.; Hu, K.; Liu, S.; Zhang, Y.; Yang, J.; Li, C.; and Liu, Z. 2025 a . LMM s-Eval: Reality Check on the Evaluation of Large Multimodal Models. In Findings of the Association for Computational Linguistics: NAACL 2025, 881--916. ...

    2025

  43. [51]

    Zhang, Y.; Lu, X.; Yin, S.; Fu, C.; Chen, W.; Hu, X.; Wen, B.; Jiang, K.; Liu, C.; Zhang, T.; et al. 2026. Thyme: Think Beyond Images. In The Fourteenth International Conference on Learning Representations

    2026

  44. [52]

    Zhang, Y.; Zhang, H.; Tian, H.; Fu, C.; Zhang, S.; Wu, J.; Li, F.; Wang, K.; Wen, Q.; Zhang, Z.; Wang, L.; and Jin, R. 2025 b . MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? In International Conference on...

    2025

  45. [53]

    Zhao, S.; Zhang, H.; Lin, S.; Li, M.; Wu, Q.; Zhang, K.; and Wei, C. 2025. PyVision: Agentic Vision with Dynamic Tooling. In First Workshop on Multi-Turn Interactions in Large Language Models

    2025

  46. [54]

    Zheng, C.; Liu, S.; Li, M.; Chen, X.-H.; Yu, B.; Gao, C.; Dang, K.; Liu, Y.; Men, R.; Yang, A.; Zhou, J.; and Lin, J. 2025. Group Sequence Policy Optimization. arXiv:2507.18071

    2025 arXiv

  47. [55]

    Zheng, Z.; Yang, M.; Hong, J.; Zhao, C.; Xu, G.; Yang, L.; Shen, C.; and Yu, X. 2026. DeepEyes: Incentivizing ``Thinking with Images'' via Reinforcement Learning. In The Fourteenth International Conference on Learning Representations

    2026

  48. [56]

    Zhu, J.; Wang, W.; Chen, Z.; Liu, Z.; Ye, S.; Gu, L.; Tian, H.; Duan, Y.; Su, W.; Shao, J.; et al. 2025. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv:2504.10479

    2025 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.