Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

Visual Agentic Reinforcement Fine-Tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small RL recipe turns open vision models into web-and-code agents

desk verdict RLVR for multimodal tool use is real and promising, but the coding reward is under-verified and the headline GPT-4o comparison is not apples-to-apples. read the letter →

arxiv 2505.14246 v1 pith:Z3YQ75XK submitted 2025-05-20 cs.CV cs.AI

Ziyu Liu , Yuhang Zang , Yushan Zou , Zijian Liang , Xiaoyi Dong , Yuhang Cao , Haodong Duan , Dahua Lin
show 1 more author
Jiaqi Wang
This is my paper · ORCID
classification cs.CVcs.AI
keywords Visual-ARFTreinforcementfine-tuningverifiablerewardsmultimodalagentstoolusevision-languagemodelsGRPOmulti-hopquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a small amount of reinforcement fine-tuning with verifiable rewards can turn an open-source vision-language model into an agent that searches the web and writes and executes Python code to manipulate images. The authors introduce Visual-ARFT, a GRPO-based recipe with modular rewards for format, final-answer F1, search-query semantic similarity, and executable code blocks. On their new MAT benchmark, the 7B model gains +18.6 F1 and +13.0 EM on MAT-Coding and +10.3 F1 and +8.7 EM on MAT-Search, surpassing GPT-4o. The same recipe improves out-of-domain text-only multi-hop QA by +29.3 F1 and +25.9 EM. If true, it means tool-using multimodal agents can be built from small open models with very little annotated data.

What carries the argument

The load-bearing mechanism is a modular verifiable reward function evaluated at every step of the trajectory. A format reward enforces the <think>, <search>, <code>, and <answer> tags; an accuracy reward scores final answers by F1, search queries by semantic similarity to a ground-truth query, and any executable code block with a flat reward of 1, deliberately not checking the code's content. GRPO optimizes the policy against these rewards with a KL penalty to the reference model, and an external environment executes each tool call and returns feedback that the model sees before its next step.

What would settle it

In a control experiment, keep Visual-ARFT's rewards identical but return a fixed, non-informative message after every code block instead of running the image-processing code. If MAT-Coding gains remain, the model is not learning to use code to manipulate images.

Watch

Extended reading notes

Core claim

The central claim is that reinforcement fine-tuning with rule-based verifiable rewards, rather than preference data or learned reward models, is enough to teach a large vision-language model when and how to invoke tools for multimodal reasoning. In the coding setting, the model learns to emit OpenCV code that rotates, crops, brightens, or denoises an image before answering; in the search setting, it learns to decompose a multi-hop question and issue web-search queries. The paper reports that this training transfers across modalities, producing gains on text-only multi-hop QA benchmarks even though the training data are multimodal VQA examples.

Load-bearing premise

The recipe assumes that paying a flat reward for any executable code block, while relying only on final-answer F1, is enough to make the model learn code that genuinely processes the image rather than learning to answer the distorted examples by other cues.

Editorial extensions

If this is right

  • A 3B open model can outperform a much larger proprietary model on MAT-Coding after Visual-ARFT training.
  • Twenty manually annotated multimodal search examples suffice to teach task decomposition that transfers to text-only multi-hop QA.
  • F1-based answer rewards train better than exact-match rewards for these open-ended agentic answers.
  • The learned tool-use behavior generalizes beyond the training distribution to unseen text-only benchmarks such as 2Wiki, HotpotQA, MuSiQue, and Bamboogle.
  • Open-source researchers can reproduce o3-style image manipulation, including rotation, cropping, and brightness adjustment, with open weights and verifiable rewards.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: Because code blocks are rewarded for executability rather than correctness, the MAT-Coding gains may partly reflect the model learning to exploit answer cues in distorted images; a code-content reward ablation would separate these effects.
  • Inference: The semantic-similarity reward for search queries could teach the model to phrase queries rather than to select retrieval results; comparing against a reward on retrieved evidence quality would test this.
  • Inference: With only 350 benchmark examples and 1,220 training examples, the reported margins are likely noisy, so the absolute rankings, not just the deltas, should be treated as preliminary.
  • Inference: If the recipe scales, the same modular rewards could extend to other tools such as APIs, calculators, and image editors with little redesign, since the rewards are environment-agnostic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Visual Agentic Reinforcement Fine-Tuning (Visual-ARFT), a GRPO-based reinforcement fine-tuning recipe that trains open-source LVLMs to act as multimodal agents with two tools: web search (via the Serper API) and Python/OpenCV code execution for image processing. The authors also construct the Multimodal Agentic Tool Bench (MAT), with 150 MAT-Search and 200 MAT-Coding test examples, and report that Visual-ARFT improves Qwen2.5-VL-7B by +18.56 F1 / +13.00 EM on MAT-Coding and +10.28 F1 / +8.66 EM on MAT-Search, outperforming a tool-less GPT-4o baseline. They further report large gains on text-only multi-hop QA benchmarks (2Wiki, HotpotQA, MuSiQue, Bamboogle), suggesting cross-modal generalization from only 20 search-training examples.

Significance. If the claims are validated, the contribution is timely and useful: it demonstrates that small amounts of RLVR data with modular verifiable rewards can teach open-source LVLMs structured tool use, and the out-of-domain multi-hop QA results provide a strong signal of generalization. The reward design is unusually simple, and the ablation switching the F1 reward to EM is informative and supports the design choice. However, the paper's central coding claim currently rests on a reward that never verifies the generated code, the GPT-4o comparison is not tool-for-tool, and the author-constructed MAT benchmark does not explicitly rule out train/test source overlap. These issues must be addressed before the headline conclusions can be accepted.

major comments (4)
  1. [Section 3.2.1, Eq. (4)] The coding reward assigns a constant reward of 1 to every executable <code> block and never verifies that the code performs the required image operation (rotation, crop, brightness, etc.). The paper reports no metric for generated-code correctness and no ablation that disables code execution or replaces the executed image with the original distorted image. Because the final-answer F1 reward is the only content-level signal, the reported +18.56 F1 / +13.00 EM gains on MAT-Coding could in principle come from learning the output format and answering strategies for the distorted test set rather than from genuine image manipulation. The authors should add a causal check, for example: (i) evaluate the trained policy with code execution disabled, (ii) measure whether the executed code actually changes the image in the intended way, and (iii) compare against a variant where code blocks are parsed but not executed. Without such evidence, the central "write and execute code to manipulate images" claim is not established.
  2. [Appendix B.2.1] The paper does not state whether the 1,200 MAT-Coding training examples and the 200 MAT-Coding test examples are disjoint in source image and question, nor whether the 20 MAT-Search training examples overlap with the 150 test examples. Since both splits are constructed from the same public VQA sources with synthetic distortions, an overlap in source instances could inflate the reported gains through instance-level memorization. Please explicitly describe the deduplication procedure and verify that no training example shares its source image or question with a test example.
  3. [Table 1 and Section 5.1] The headline claim that Visual-ARFT "surpasses GPT-4o" is not a tool-for-tool comparison: GPT-4o is evaluated without tools ("Reasoning with Tools ✗") while Visual-ARFT is evaluated with search and code execution. A fair comparison would allow GPT-4o the same tool interface, or would measure Visual-ARFT without tools. The current wording overstates the result and should be qualified or the experiment should be rerun.
  4. [Tables 1 and 2, Section 5.1] All reported numbers come from a single training run with no error bars, and the MAT test sets contain only 350 examples total (200 MAT-Coding, 150 MAT-Search). Given the small evaluation size and the stochasticity of GRPO, the absence of multiple seeds or confidence intervals makes it difficult to assess whether the headline gains (e.g., +18.56 F1 on MAT-Coding) are reliable. Please report mean and standard deviation over at least three seeds, or provide a clear justification for why single-run evaluation is sufficient.
minor comments (5)
  1. [Table 2] The average F1 and EM values for Qwen2.5-VL-7B-Instruct + RAG appear to be arithmetically inconsistent with the four per-dataset values: the per-dataset values average to 27.72 F1 and 18.16 EM, but the table reports 19.82 and 12.73. Please correct the averages or the per-dataset entries.
  2. [Figure 1] The top example in Figure 1 is difficult to parse: it mixes a text-recognition query, a <code> block, an answer "JUR TOWN HALL RD AYE", and then a separate search chain about Mount Everest with final answer "238". This appears to be a stitching of two different examples and should be cleaned up.
  3. [Section 5, Implementation Details] Several reproducibility-critical hyperparameters are missing: the KL regularization coefficient beta in Eq. (1), the learning rate and number of GRPO updates, the exact Sentence Transformer model used for the semantic similarity reward in Eq. (4), and the number of training epochs. Please provide these details.
  4. [Section 4, MAT-Search] The text says each MAT-Search test case includes "supporting reference material that may contain both relevant and distractor content," but the evaluation is described as using the Serper API for live web search. Please clarify whether this reference material is provided to the model during evaluation or is only used for constructing gold search queries.
  5. [Throughout] There are minor typos and naming inconsistencies, e.g., "Tool caling" in Figure 1, and the model name is sometimes given as "Qwen2.5-VL-7B-Instruct" and sometimes as "Qwen2.5-VL-7B". These should be harmonized.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Visual-ARFT is an empirical RLVR training recipe evaluated on held-out splits and external multi-hop QA benchmarks; the flat code reward is a supervision weakness, not a circular construction.

full rationale

Visual-ARFT does not claim a first-principles derivation; it is an empirical reinforcement fine-tuning recipe. The reward functions in Eq. (4)-(5) are defined over output format, final-answer F1, search-query semantic similarity, and code-block presence. None of these definitions is made in terms of the claimed outcomes, and no parameter is fitted to the evaluation results. MAT-Coding and MAT-Search results are reported on held-out test sets (200 and 150 examples) distinct from the training data (1,200 coding and 20 search examples), so the in-domain gains are not forced by construction. The generalization claims on 2Wiki, HotpotQA, MuSiQue, and Bamboogle (Table 2) are measured on external benchmarks, giving independent grounding. The flat reward of 1 for any executable code block is weak supervision and leaves open the possibility that gains come from format learning or direct answers rather than genuine tool use; that is a validity risk, not circularity. The author-constructed MAT benchmark is modest in scale, as the paper acknowledges in Appendix C, but benchmark authorship alone does not make the reported improvements equivalent to the training objective. Self-citations appear in the reference list (Visual-RFT, MIA-DPO) but are not invoked as the load-bearing justification for the empirical findings. No specific equation or fitted quantity reduces to the claimed result by construction.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the reward design, the quality of the MAT benchmark, and the external tool environments. The most fragile entries are the flat code-block reward and the unverified disjointness of train and test splits. No new physical or ontological entities are introduced.

free parameters (4)
  • KL regularization coefficient beta = not reported
    Equation (1) includes beta but the value is not given; it controls policy drift and can affect generalization.
  • GRPO sampled generations per update = 8
    Section 5 states 8 sampled generations are used; this is a hand-chosen hyperparameter that affects reward variance.
  • Sentence Transformer model for semantic similarity = not specified
    Section 3.2.1 uses semantic similarity between the model query and ground-truth query; the specific model and similarity threshold are not named, making the search reward irreproducible.
  • Code-block reward value = 1
    Equation (4) gives a fixed reward of 1 to any executable code block; this hand-chosen constant shapes exploration and is central to the coding reward design.
assumptions (6)
  • domain assumption MAT train and test examples are disjoint
    The paper says train and test sets are constructed separately (20 vs 150, 1,200 vs 200) but never explicitly states a deduplication check; the central gains could be inflated by leakage.
  • domain assumption F1 score on final answer is a valid proxy for task correctness
    Section 3.2.1 uses F1 as the accuracy reward; for yes/no and numeric VQA answers, F1 can reward partial lexical overlap that may not correspond to semantic correctness.
  • domain assumption Semantic similarity to ground-truth search queries captures retrieval intent
    Section 3.2.1 uses Sentence Transformer similarity; if the ground-truth queries are not representative or the embedding model is weak, the search reward is noisy.
  • domain assumption Serper API and the code execution environment return correct, useful feedback
    The agent's tool loop depends on external services; the paper does not audit failure cases or API nondeterminism.
  • ad hoc to paper A reward of 1 for any executable code block is sufficient to induce correct image processing
    Equation (4) removes content-level code supervision; the paper provides no causal evidence that the learned code, rather than answer memorization or other cues, drives the MAT-Coding gains.
  • domain assumption GRPO with verifiable rewards transfers to LVLMs
    The paper adopts GRPO from DeepSeekMath and assumes the same convergence behavior holds for LVLMs with tool-use outputs; this is supported mainly by self-cited Visual-RFT rather than by a formal proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Visual Agentic Reinforcement Fine-Tuning." pith.science (2026). https://pith.science/paper/Z3YQ75XK

@misc{pith2026250514246,
  author       = {Pith},
  title        = {Pith review of: Visual Agentic Reinforcement Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z3YQ75XK}},
  note         = {Machine review of arXiv:2505.14246}
}
read the original abstract

A key trend in Large Reasoning Models (e.g., OpenAI's o3) is the native agentic ability to use external tools such as web browsers for searching and writing/executing code for image manipulation to think with images. In the open-source research community, while significant progress has been made in language-only agentic abilities such as function calling and tool integration, the development of multi-modal agentic capabilities that involve truly thinking with images, and their corresponding benchmarks, are still less explored. This work highlights the effectiveness of Visual Agentic Reinforcement Fine-Tuning (Visual-ARFT) for enabling flexible and adaptive reasoning abilities for Large Vision-Language Models (LVLMs). With Visual-ARFT, open-source LVLMs gain the ability to browse websites for real-time information updates and write code to manipulate and analyze input images through cropping, rotation, and other image processing techniques. We also present a Multi-modal Agentic Tool Bench (MAT) with two settings (MAT-Search and MAT-Coding) designed to evaluate LVLMs' agentic search and coding abilities. Our experimental results demonstrate that Visual-ARFT outperforms its baseline by +18.6% F1 / +13.0% EM on MAT-Coding and +10.3% F1 / +8.7% EM on MAT-Search, ultimately surpassing GPT-4o. Visual-ARFT also achieves +29.3 F1% / +25.9% EM gains on existing multi-hop QA benchmarks such as 2Wiki and HotpotQA, demonstrating strong generalization capabilities. Our findings suggest that Visual-ARFT offers a promising path toward building robust and generalizable multimodal agents.

Figures

Figures reproduced from arXiv: 2505.14246 by the authors.

Figure 1
Figure 1. The benefits of our Visual Agentic Reinforcement Fine-Tuning (Visual-ARFT) to perform complex multi-modal reasoning tasks, such as (top) write and execute python code to accurately read text within a specified image region and (bottom) use internet search to answer a multi-hop question. Recent works such as Search-R1 [13] and ReTool [5] have successfully applied Reinforcement Fine￾Tuning (RFT) to agentic learning, e… view at source ↗
Figure 2
Figure 2. Overview of Visual-ARFT. We successfully empower LVLMs with multimodal agentic capabilities, including (a) agentic search and (b) agentic coding, enabling them to solve complex multimodal tasks through reasoning, decomposition, and tool interaction. Formally, the training objective in RLVR can be expressed as maximizing the expected reward, while simultaneously constraining the learned policy πθ to remain close to a… view at source ↗
Figure 3
Figure 3. Data Annotation Pipeline of our proposed Multimodal Agentic Tool Bench (MAT): (a) MAT-Search, a manually annotated and verified dataset for agentic search, and (b) MAT-Coding, an automatically generated dataset for agentic coding with a structured pipeline. difficulty and require different levels of reasoning depth—more complex queries involve more inference steps and factual knowledge, challenging the model’s abili… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Visualization Inference Cases of Visual-ARFT. Demonstrating Visual-ARFT’s multi-modal agentic capabilities: processing an image and answering a question via code generation and execution (left), and solving multi-hop VQA through query decomposition and search tool invo…
Figure 5
Figure 5. Figure 5: Prompt for Agentic Searching Tasks # Role You are a step-by-step image processing assistant. Your task is to solve an image-based task by applying OpenCV operations one step at a time, optionally using a reasoning chain. # Output Format At each step, output one of the …
Figure 6
Figure 6. Figure 6: Prompt for Agentic Coding Tasks decomposing the question and retrieving missing knowledge externally. This dataset forms the basis of both the training split and the MAT-Search benchmark, which we use to evaluate the model’s ability to plan, decompose tasks, and invoke…
Figure 7
Figure 7. Figure 7: MAT-Coding Data Examples We extract or design visual question-answering samples from these datasets and apply various distortions (e.g., rotation, blur, brightness variation) to the input images, thereby creating challenging scenarios that require the model to write an…
Figure 8
Figure 8. Figure 8: MAT-Search Data Examples typically involve factual knowledge, visual element, and indirect relationships, prompting the model to decompose the problem and retrieve intermediate facts via a search tool. MAT-Coding. To simulate real-world scenarios where visual input qua…
Figure 9
Figure 9. Figure 9: Examples of Visual-ARFT Inference on MAT. B.2.4 Difficulty Split: Simple vs. Hard To facilitate finer-grained analysis, we divide both MAT-Search and MAT-Coding benchmarks into Simple and Hard subsets based on the required reasoning complexity and tool invocation diffi…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Code-in-the-Loop Forensics: Agentic Tool Use for Image Forgery Detection

    cs.AI 2025-12 unverdicted novelty 7.0 of 10

    ForenAgent lets MLLMs create and iteratively improve low-level Python tools for image forgery detection via a two-stage training pipeline and a new 100k-image benchmark dataset.

  2. VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A multi-tool video agent trained with dual-branch RL (BiSPO) and a new VideoSearch-QA benchmark substantially beats open-source agentic baselines on video deep research and multimodal search tasks.

  3. POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Management

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    POINTS-Seeker-8B is an 8B multimodal model trained from scratch for agentic search that uses seeding and visual-space history folding to outperform prior models on six visual reasoning benchmarks.

  4. M2IO-R1: An Efficient RL-Enhanced Reasoning Framework for Multimodal Retrieval Augmented Multimodal Generation

    cs.IR 2025-08 conditional novelty 6.0 of 10

    M2IO-R1 uses GRPO reinforcement learning to train a 3B model that selects and places retrieved images into generated text answers, improving MRAMG quality on several benchmarks.

  5. Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback

    cs.CV 2025-07 conditional novelty 6.0 of 10

    RRVF trains an image-to-code MLLM using reinforcement learning with a render-and-compare visual feedback loop, and it outperforms supervised fine-tuning on chart and web benchmarks.

  6. Supermassive Black Hole Winds in X-rays: SUBWAYS IV. Tracing Radio Emission and Unveiling the Role of Winds

    astro-ph.GA 2026-04 unverdicted novelty 5.5 of 10

    In X-ray selected AGN, those hosting ultra-fast outflows exhibit larger radio extensions and steep spectra matching wind-driven shock models.

  7. Omnidirectional Spatial Modeling from Correlated Panoramas

    cs.CV 2025-09 conditional novelty 5.0 of 10

    The authors create a cross-frame panoramic VQA benchmark from 3D scene data and show that GRPO fine-tuning of Qwen2.5-VL raises its score on that benchmark.

Reference graph

Works this paper leans on

57 extracted references · 11 canonical work pages · cited by 7 Pith papers

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  2. [2]

    Learning to reason with search for llms via reinforcement learning

    Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, Fan Yang, et al. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470, 2025

    arXiv 2025

  3. [3]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  4. [4]

    Rico: A mobile app dataset for building data-driven design applications

    Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium on user interface software and technology, pages 845– 854, 2017

    2017

  5. [5]

    Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025

    arXiv 2025

  6. [6]

    Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning.arXiv preprint arXiv:2501.00321, 2024

    Ling Fu, Biao Yang, Zhebin Kuang, Jiajun Song, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Mingxin Huang, et al. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning.arXiv preprint arXiv:2501.00321, 2024

    arXiv 2024

  7. [7]

    Synthetic data generation & multi-step rl for reasoning & tool use.arXiv preprint arXiv:2504.04736, 2025

    Anna Goldie, Azalia Mirhoseini, Hao Zhou, Irene Cai, and Christopher D Manning. Synthetic data generation & multi-step rl for reasoning & tool use.arXiv preprint arXiv:2504.04736, 2025

    arXiv 2025

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 57 references
  1. [9]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps.arXiv preprint arXiv:2011.01060, 2020

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps.arXiv preprint arXiv:2011.01060, 2020

    2011 arXiv

  2. [10]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  3. [11]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  4. [12]

    Funsd: A dataset for form understand- ing in noisy scanned documents

    Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form understand- ing in noisy scanned documents. In2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1–6. IEEE, 2019

    2019

  5. [13]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    2025 arXiv

  6. [14]

    Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension

    Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Ha- jishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. InProceedings of the IEEE Conference on Computer Vision and Pattern re...

    2017

  7. [15]

    T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  8. [16]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  9. [17]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024. 10

    2024 arXiv

  10. [18]

    Search-o1: Agentic search-enhanced large reasoning models.arXiv preprint arXiv:2501.05366, 2025

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models.arXiv preprint arXiv:2501.05366, 2025

    2025 arXiv

  11. [19]

    Webthinker: Empowering large reasoning models with deep research capability.arXiv preprint arXiv:2504.21776, 2025

    Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability.arXiv preprint arXiv:2504.21776, 2025

    2025 arXiv

  12. [20]

    Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025

    2025 arXiv

  13. [21]

    Mario: Math reasoning with code interpreter output–a reproducible pipeline.arXiv preprint arXiv:2401.08190, 2024

    Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output–a reproducible pipeline.arXiv preprint arXiv:2401.08190, 2024

    2024 arXiv

  14. [22]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  15. [23]

    Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451, 2024

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451, 2024

    2024 arXiv

  16. [24]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  17. [25]

    Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  18. [26]

    Mia-dpo: Multi-image augmented direct preference optimization for large vision-language models.arXiv preprint arXiv:2410.17637, 2024

    Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Mia-dpo: Multi-image augmented direct preference optimization for large vision-language models.arXiv preprint arXiv:2410.17637, 2024

    arXiv

  19. [27]

    Towards end-to-end unified scene text detection and layout analysis

    Shangbang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, and Michalis Raptis. Towards end-to-end unified scene text detection and layout analysis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1049–1059, 2022

    2022

  20. [28]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021

    2021

  21. [29]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. https://openai.com/index/ o3-o4-mini-system-card/, 2025. Accessed: 2025-04-16

    2025

  22. [30]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  23. [31]

    Gorilla: Large language model connected with massive apis.Advances in Neural Information Processing Systems, 37:126544–126565, 2024

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis.Advances in Neural Information Processing Systems, 37:126544–126565, 2024

    2024

  24. [32]

    Measuring and narrowing the compositionality gap in language models.arXiv preprint arXiv:2210.03350, 2022

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models.arXiv preprint arXiv:2210.03350, 2022

    2022 arXiv

  25. [33]

    Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025

    2025 arXiv

  26. [34]

    Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  27. [35]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  28. [36]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 11

    2024 arXiv

  29. [37]

    Agentic reasoning and tool integration for llms via reinforcement learning.arXiv preprint arXiv:2505.01441, 2025

    Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for llms via reinforcement learning.arXiv preprint arXiv:2505.01441, 2025

    2025 arXiv

  30. [38]

    R1-searcher: Incentivizing the search capability in llms via reinforcement learning.arXiv preprint arXiv:2503.05592, 2025

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning.arXiv preprint arXiv:2503.05592, 2025

    2025 arXiv

  31. [39]

    Zerosearch: Incentivize the search capability of llms without searching.arXiv preprint arXiv:2505.04588, 2025

    Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, and Yan Zhang. Zerosearch: Incentivize the search capability of llms without searching.arXiv preprint arXiv:2505.04588, 2025

    2025 arXiv

  32. [41]

    Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  33. [42]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  34. [43]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  35. [44]

    Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    2022

  36. [45]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  37. [46]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering.arXiv preprint arXiv:1809.09600, 2018

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering.arXiv preprint arXiv:1809.09600, 2018

    2018 arXiv

  38. [47]

    Detecting texts of arbitrary orientations in natural images

    Cong Yao, Xiang Bai, Wenyu Liu, Yi Ma, and Zhuowen Tu. Detecting texts of arbitrary orientations in natural images. In2012 IEEE conference on computer vision and pattern recognition, pages 1083–1090. IEEE, 2012

    2012

  39. [48]

    RlHF-V: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. RlHF-V: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. InCVPR, 2024

    2024

  40. [49]

    RLAIF-V: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness.arXiv preprint arXiv:2405.17220, 2024

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. RLAIF-V: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness.arXiv preprint arXiv:2405.17220, 2024

    2024

  41. [50]

    Internlm-xcomposer2

    Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, et al. Internlm-xcomposer2. 5-reward: A simple yet effective multi-modal reward model.arXiv preprint arXiv:2501.12368, 2025

    2025 arXiv

  42. [51]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output.arXiv preprint arXiv:2407.03320, 2024

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output.arXiv preprint arXiv:2407.03320, 2024

    2024 arXiv

  43. [52]

    Nemotron-research-tool-n1: Tool-using language models with reinforced reasoning

    Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. Nemotron-research-tool-n1: Tool-using language models with reinforced reasoning. arXiv preprint arXiv:2505.00024, 2025

    2025 arXiv

  44. [53]

    Aligning modalities in vision large language models via preference fine-tuning.arXiv preprint arXiv:2402.11411, 2024

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning.arXiv preprint arXiv:2402.11411, 2024. 12 Appendix of Visual Agentic Reinforcement Fine-Tuning A Prompt Used A.1 Prompt for ...

    2024 arXiv

  45. [54]

    Issue **one specific, text-only search** enclosed in <search> </search> tags,

  46. [55]

    the image

    Or provide the **final answer** enclosed in <answer> </answer> tags. All outputs **must begin with a thought** enclosed in <think> </think> tags, explaining your current reasoning and what to do next. - Do not reference “the image” in your searches. - Do not repeat past querie...

  47. [56]

    <problem> Describe the image issue from {'rotation90', 'rotation180', 'dark', 'overexposure', 'blur', 'noise', 'crop', 'none'} </problem>

  48. [57]

    <code> OpenCV code to process and save the image </code>

  49. [58]

    Can You Find Vermeer's Milkmaid at the Rijksmusem?

    <answer> Final answer based on the processed image </answer> # Image Processing Rules - Always read from `'path_to_input_image.jpg'` and write to `'path_to_output_image.jpg'`. # Output Format (strict): Always begin with <think>. Then, depending on current reasoning chain, outp...

    1994

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.