Pith. sign in

REVIEW 3 major objections 6 minor 129 references

Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Tool-augmented vision models may not need the returned image: swapping it for a text placeholder matches or beats pixel feedback on current benchmarks, pointing to the tool-call text itself as the carrier of reasoning gains.

desk verdict Clean matched ablation with a real missing control: without a 65K no-scaffold baseline, the claim that the scaffold carries the gain is underdetermined. read the letter →

arxiv 2608.09682 v1 pith:6VOCLAC4 submitted 2026-08-10 cs.CV

Jiahao Shao , Yuanbo Yang , Yiyi Liao , Yujun Shen , Ceyuan Yang , Yinghao Xu This is my paper · ORCID
classification cs.CV
keywords tool-augmentedvision-languagemodelsthinking-with-imagescarrierswaptextualscaffoldtool-callreasoningreinforcementlearningcropandzoomtoolspixelredundancy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper targets the reasoning gains attributed to 'thinking-with-images' in tool-augmented vision-language models. It argues, and supports with a controlled carrier-swap intervention, that the active signal is the structured text the model emits before the tool returns: the tool name, crop coordinates, target description, and call intent. Replacing the returned image with the literal placeholder [Image output skipped] during both training and inference preserves accuracy across SFT and RL scales, reduces latency by 29-46%, and removes tool-execution API calls. The claim is scoped to current thinking-with-images benchmarks; the paper itself notes that constructing tasks where pixels are genuinely load-bearing remains open.

What carries the argument

The central mechanism is the training-aligned carrier swap: the tool-call scaffold (the emitted <think> reasoning block plus executable crop/zoom code with coordinates) is preserved exactly, while the post-call visual return is replaced by the fixed text sentinel [Image output skipped]. This isolates the pre-return textual scaffold from the returned pixels, avoiding the train-test mismatch confound of prior inference-time deletions. The scaffold is decomposed into reasoning text (why a region matters, what to look for) and spatial code (the concrete .crop() coordinates); a coordinate-precision ablation shows that the format of the code matters substantially while the numeric correctness of the coordinates does not detectably matter in the post-training regime.

What would settle it

Construct a benchmark task where the answer depends on visual detail that exists only in the region returned by the crop/zoom tool, and that cannot be anticipated from the scaffold text (e.g., reading a small label or distinguishing two near-identical objects only after cropping). If a TextCall-trained model, matched on scale and training data, significantly underperforms a thinking-with-images model on such a task, the Tool-Call Scaffold Hypothesis is refuted; if TextCall matches, the pixel return is redundant even there. The paper itself identifies visual-simulation tasks and fine-grained comparison across intermediate states as candidate settings for this test.

Watch

Extended reading notes

Core claim

The paper proposes the Tool-Call Scaffold Hypothesis: in current thinking-with-images training distributions, the load-bearing signal is the structured text emitted at tool-call time (tool name, coordinates, target description, intent), while the returned image is a redundant carrier. To test this, it introduces TextCall, which keeps the full tool-call scaffold but replaces every returned image with the fixed text sentinel [Image output skipped] during both training and inference. Across LoRA SFT, full fine-tuning, and GRPO reinforcement learning on a six-benchmark core suite, TextCall matches or exceeds full thinking-with-images accuracy (e.g., 76.96% vs 71.73% on V*Bench at 9.5K LoRA; +1.39pp mean on the 6-bench mean at 65K full FT). Under matched RL conditions, TextCall maintains active tool use while the thinking-with-images run collapses to direct answering. A factorial audit on 1,000 matched DeepEyes V2 trajectories shows that the scaffold alone recovers accuracy equivalent to the image-only condition (73.10% vs 73.50%, non-inferior at a 5pp margin), and a component decomposition shows that both reasoning text and spatial code contribute, with the dominant component varying by benchmark.

Load-bearing premise

The evaluated benchmarks and training corpus are such that all discriminative visual evidence needed to answer a question is present in the original input image and can be reflected in the pre-return tool-call scaffold, so that removing the post-call image loses no information needed for a correct answer.

Editorial extensions

If this is right

  • If the hypothesis holds, current tool-augmented VLM pipelines can drop returned-image tool execution entirely for perception-heavy tasks, cutting latency by roughly 30-50% and eliminating tool-execution API overhead while preserving accuracy.
  • The gain attributed to visual tool use is, on current benchmarks, largely carried by the structured text—so models can be trained to emit better scaffolds (more informative reasoning and code) rather than to consume pixel returns.
  • Under matched RL settings, the scaffold-only variant preserves active tool use where the pixel-return variant collapses to direct answering, suggesting the pixel carrier can destabilize tool-use behavior during policy optimization.
  • Future thinking-with-images systems should report a scaffold-only control so that the load-bearing carrier is measured rather than assumed.
  • The paper calls for new benchmark tasks where the returned pixels supply information the scaffold cannot verbalize or replace; until such tasks are constructed, TextCall is presented as the lower-cost default.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The scaffold's role may generalize beyond crop/zoom code tools: the paper's own Visual Jigsaw probe with text-to-image edits shows the pattern holds across a different tool family, but only at n=50, so this is suggestive rather than conclusive.
  • The coordinate-precision equivalence result suggests that in the post-training regime, the code body serves as a structured procedural cue—forcing region selection—rather than as an accurate spatial reference; this implies that scaffold quality is about the reasoning text and the act of specifying a region, not the coordinate values.
  • If pixels are redundant on current benchmarks, the bottleneck for tool-augmented visual reasoning shifts to the model's ability to verbalize where to look; progress may come from improving spatial verbalization and scaffold generation rather than from richer image-return mechanisms.
  • The paper's boundary conditions (representational bottleneck and visual prior knowledge gap) predict that tasks requiring visual simulation or fine-grained comparison across intermediate states may be where pixel returns become genuinely load-bearing—a testable prediction the authors leave open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes the Tool-Call Scaffold Hypothesis: in current thinking-with-images distributions, the structured text emitted before a tool returns (tool name, coordinates, target description, intent) is the load-bearing signal, while the returned image is a redundant carrier. To test this, the authors introduce TextCall, a training-aligned carrier swap that retains the tool-call scaffold but replaces the returned image with the fixed placeholder ‘[Image output skipped]’ during both training and inference. Using Qwen2.5-VL-7B on the DeepEyes V2 corpus, they report that TextCall matches or exceeds full thinking-with-images under 9.5K LoRA SFT, 65K full fine-tuning, and a matched GRPO run, with lower latency and no image-returning tool executions. They additionally present a factorial audit of 1,000 DeepEyes V2 trajectories showing that a scaffold-only input to a Gemini judge recovers image-only accuracy, a scaffold decomposition showing that reasoning text and code both matter on a 9.5K LoRA setup, and a coordinate-precision ablation suggesting that code format rather than numeric coordinate accuracy carries the effect. The authors explicitly scope their claims to one base model, one cold-start corpus, one seed, and one perception-heavy benchmark suite.

Significance. If the central claim is correct, the paper provides a valuable reframing of tool-augmented VLM reasoning: it offers a clean training-time intervention that prior inference-time ablations lacked, and it has practical implications for latency and architecture by eliminating post-call image processing. The carrier-swap design is a genuine methodological step forward, and the paper is unusually transparent, reporting compute, assets, licenses, RL dynamics, and a clear limitation statement. The central comparison is an external-benchmark intervention rather than a fitted-parameter prediction, so circularity is not a concern. However, the stronger claim that the scaffold itself is the load-bearing signal is underdetermined by the current controls, and the headline parity result lacks uncertainty quantification.

major comments (3)
  1. [Appendix C, Table 7; Section 4.2] The skeleton-only control reported in Appendix C (Table 7) materially weakens the claim that the scaffold content is load-bearing. At the same 9.5K LoRA scale, the skeleton-only condition (empty reasoning, pass code) reaches a mean of 71.63 on V*/HR-4K/CV-2D against 73.13 for the full scaffold, a difference of only 1.5pp. The paper dismisses this variant as ‘falling back to the base model’s direct-answering capability,’ but the base model in Agent Mode averages 62.70 on the same three benchmarks, so the skeleton-only result is 9pp above that baseline and is close to the direct-MC base mean of 75.78. This suggests the multi-turn SFT on the corpus itself, rather than the semantic content of the scaffold, may be responsible for much of the gain. More importantly, no analogous no-scaffold control is reported at the 65K full-FT scale where the headline TextCall-vs-TWI parity is established. Without a 65K direct-answer or skeleton-only control trained on the same data, the observed parity could arise from corpus SFT plus the model’s continued access to the original question image, not from the scaffold. Please add such a control and reinterpret the 9.5K decomposition in light of Table 7.
  2. [Table 1, Section 3.3] The central non-inferiority claim rests on point estimates without uncertainty quantification. V*Bench has only 191 samples; at the 65K scale the V*Bench gap is -0.52pp and the core six-benchmark mean gap is +1.39pp, while at 9.5K the V*Bench gap is +5.23pp. No confidence intervals, bootstrap intervals, or paired tests are provided, so it is impossible to assess which differences are within sampling noise. Since the paper explicitly claims that TextCall ‘matches or exceeds’ thinking-with-images, it should report paired bootstrap confidence intervals (as already done for the audit in Section 4.1) for the TextCall-versus-TWI comparisons on the core suite, at least for the 65K comparison.
  3. [Section 4.2, Table 3] The component decomposition is non-monotonic in a way that complicates the conclusion that both reasoning text and code contribute additively. The means are: full scaffold 73.13, reasoning-only 66.32, code-only 61.83, and skeleton-only 71.63. Removing both components hurts far less than removing either one, which suggests that the full-scaffold and skeleton-only models may be using different strategies (scaffold-guided reasoning versus direct answering) rather than exhibiting additive component contributions. The proposed explanation that code removal causes ‘hallucinated spatial attention’ is plausible, but the near-parity of skeleton-only indicates a strategy-switch alternative. Please provide per-sample agreement and error analysis between the full-scaffold and skeleton-only models, and report at least one no-content condition at the 65K scale, before concluding that the scaffold components are individually load-bearing.
minor comments (6)
  1. [Appendix I, Table 10] The RL comparison reports only tool-call rates for the thinking-with-images collapse; the text states that accuracy ‘remains reasonable’ without numbers. Since the claim is that the collapse is a failure mode, please report the benchmark accuracies of the collapsed thinking-with-images RL checkpoint alongside TextCall in Table 10, or explicitly state that the RL contrast is about tool-use behavior rather than accuracy.
  2. [Abstract, Section 1] The title and abstract phrase ‘Not With Pixels’ could be misread as claiming that no pixels at all are needed; in fact TextCall still processes the original question image and only removes the post-call crop/zoom returns. The paper does scope the claim correctly in the body, but the title is broader than the actual intervention.
  3. [Section 4.1] The 5pp non-inferiority margin for the scaffold-only versus image-only audit is stated without justification; please explain why 5pp is the appropriate equivalence margin for this comparison.
  4. [Appendix J] The coordinate-precision ablation uses Qwen3-VL-8B and a Direct MC think-mode evaluation protocol that differs from the main experimental setup, while the 9.5K replication uses a different base model; the main text should more prominently flag these as supporting evidence rather than as part of the main carrier-swap experiment.
  5. [Table 4] The latency-only no-execution row in Table 4 is explicitly not used for canonical accuracy comparisons, but its placement in the main table may confuse readers; consider moving it to an appendix or marking it more clearly as a reference-only row.
  6. [Table 1] The data-size label ‘77K’ for the TextCall SFT+RL row is unclear; it appears to denote the SFT corpus plus RL rollouts, but the paper should state the exact composition of that row relative to the 65K SFT data.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TextCall's training-time carrier swap is an external empirical intervention; missing no-scaffold control is a confound, not a circular derivation.

full rationale

The paper's central claim is tested by a controlled training-time intervention: identical SFT data, identical base model, identical evaluation runner, with the only manipulated variable being whether the post-call tool return is the actual image or the fixed sentinel '[Image output skipped]'. No parameter is fitted to the benchmark outcomes, no prior result by the authors is invoked as a uniqueness or forbidden-alternative argument, and the Tool-Call Scaffold Hypothesis is not defined in terms of the measured accuracy. The scaffold-only audit (Table 2) is an empirical probe on a held-out judge and is explicitly labeled a mechanism probe, not an independent causal claim. The component decomposition (Table 3) varies assistant text content and measures accuracy on external benchmarks. The most serious threat is the absence of a no-scaffold control at the 65K scale: the Appendix C skeleton-only baseline (71.63 vs. 73.13 at 9.5K) suggests that much of the gain might be attributable to the multi-turn wrapper or corpus SFT rather than scaffold content. That is an experimental under-determination or confound, not circularity: the numbers are not forced by construction, and the paper transparently reports the skeleton-only row alongside its interpretation. There are no self-citations, no fitted parameters renamed as predictions, and no imported uniqueness theorem. The derivation chain is therefore self-contained with respect to circularity, even though the causal inference is weaker than the prose suggests.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on domain assumptions about representativeness and image information, plus standard statistical procedures; no invented physical or mathematical entities are introduced.

free parameters (3)
  • GRPO reward weights = 0.8 accuracy + 0.2 format
    Hand-chosen reward; the RL tool-use stability contrast is conditional on this, and the paper notes a tool-use bonus might change the outcome.
  • Non-inferiority margin in data audit = 5 pp
    Hand-set threshold for declaring scaffold-only equivalent to image-only in the paired audit; the 95% CI [-3.20, +2.40] passes at 5 pp but would not pass at tighter margins.
  • Equivalence margin in coordinate-precision ablation = delta = 2 pp
    Hand-set TOST margin; the claim that coordinate values do not matter is only established at this margin, which is less than one-third of the code-removal effect.
assumptions (6)
  • domain assumption Representativeness of the DeepEyes V2 corpus and the six-benchmark core suite for 'current thinking-with-images distributions'.
    Section 3.2 defines the evaluation regime; the abstract and conclusion generalize from this single corpus and suite, so the claim inherits their representativeness.
  • domain assumption All task-relevant visual evidence in the evaluated benchmarks is present in the original input image and can be captured in the pre-return scaffold.
    TextCall removes only the post-call returned crop while keeping the original image; if a question needed crop-only detail, the swap would degrade. See Section 3.1 and Appendix G.
  • domain assumption Gemini-3-Flash at temperature 0 is a valid answerer for the factorial audit.
    The scaffold-only equivalence result (73.10% vs 73.50%) is measured by this judge, not by the trained Qwen model, so the audit's validity depends on this judge's reliability.
  • domain assumption The scaffold extraction excludes answer leakage.
    The audit restricts scaffold to the first think block and code blocks, and verifies final-answer literals appear in under 1% of code blocks; if leakage were material, the scaffold-only accuracy would be inflated.
  • standard math Paired bootstrap resampling and TOST equivalence tests are valid frequentist procedures.
    Used for the audit CIs, non-inferiority test, and coordinate-precision equivalence; no alternative error model is considered.
  • domain assumption Single-seed training outcomes represent the protocols.
    Main SFT and RL comparisons are single runs; the paper acknowledges one seed in Section 6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning." pith.science (2026). https://pith.science/paper/6VOCLAC4

@misc{pith2026260809682,
  author       = {Pith},
  title        = {Pith review of: Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6VOCLAC4}},
  note         = {Machine review of arXiv:2608.09682}
}
read the original abstract

Tool-augmented vision-language models increasingly "think with images": they call crop, zoom, or code tools and reason over the returned pixels. However, recent work using blind tests, gain decompositions, and attention analyses has shown that returned images contribute little, raising the question: if pixels do not carry the gain, what does? We hypothesize that the load-bearing signal is the structured text emitted before any returned pixel arrives: tool name, coordinates, target description, and intent. This textual scaffold encodes where to look and what to find. We introduce TextCall (call-but-no-return) to test this: it keeps the scaffold but replaces returned images with the text placeholder [Image output skipped]. Three studies support the hypothesis. (i) Non-necessity of returned pixels: across LoRA, full fine-tuning, and RL, TextCall matches or exceeds full thinking-with-images; under RL it preserves tool use at the reported checkpoint, avoiding the failure mode where, under matched settings, seeing the returned image causes the model to stop calling tools and answer directly. (ii) Sufficiency of the scaffold: on matched training queries, scaffold-only input yields equivalent accuracy to returned-image input. (iii) Component specificity: decomposing the scaffold into reasoning text and spatial code shows both components contribute, with the dominant one varying by task. Together these results support the Tool-Call Scaffold Hypothesis: in current thinking-with-images distributions, the active signal is the structured text emitted at tool-call time; the returned image is a redundant carrier. TextCall preserves accuracy while reducing latency by 29-46% and eliminating tool-execution API calls. Our claims hold for current thinking-with-images benchmarks; constructing tasks where pixels are genuinely load-bearing remains an open direction.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

129 extracted references · 4 canonical work pages

  1. [1]

    Latent reasoning with supervised thinking states.arXiv preprint arXiv:2602.08332, 2026

    Ido Amos, Avi Caciularu, Mor Geva, Amir Globerson, Jonathan Herzig, Lior Shani, and Idan Szpektor. Latent reasoning with supervised thinking states.arXiv preprint arXiv:2602.08332, 2026

    arXiv 2026

  2. [2]

    Acloserlookatbiasandchain-of-thought faithfulness of large (vision) language models.arXiv preprint arXiv:2505.23945, 2025

    SriramBalasubramanian, SamyadeepBasu, andSoheilFeizi. Acloserlookatbiasandchain-of-thought faithfulness of large (vision) language models.arXiv preprint arXiv:2505.23945, 2025

    arXiv 2025

  3. [3]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models.arXiv preprint arXiv:2403.06764, 2024

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models.arXiv preprint arXiv:2403.06764, 2024

    arXiv 2024

  4. [4]

    MMStar: An evaluator-centric benchmark for multi-modal large language models, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, 11 Thinking With Tools, Not With Pixels Yu Qiao, Dahua Lin, and Feng Zhao. MMStar: An evaluator-centric benchmark for multi-modal large language models, 2024

    2024

  5. [5]

    Perception before reasoning: Two-stage reinforcement learning for visual reasoning.arXiv preprint arXiv:2509.13031, 2025

    Yan Chen, Long Li, Teng Xi, Long Zeng, and Jingdong Wang. Perception before reasoning: Two-stage reinforcement learning for visual reasoning.arXiv preprint arXiv:2509.13031, 2025

    arXiv 2025

  6. [6]

    Bowman, Jan Leike, Jared Kaplan, and Ethan Perez

    Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don’t always say what they think.arXiv preprint arXiv:2505.05410, 2025

    arXiv 2025

  7. [7]

    v1: Learning to point visual tokens for multimodal grounded reasoning.arXiv preprint arXiv:2505.18842, 2025

    Jiwan Chung, Junhyeok Kim, Siyeol Kim, Jaeyoung Lee, Min Soo Kim, et al. v1: Learning to point visual tokens for multimodal grounded reasoning.arXiv preprint arXiv:2505.18842, 2025

    arXiv 2025

  8. [8]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning

    DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 129 references
  1. [9]

    Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi

    MattDeitke,ChristopherClark,SanghoLee,RohunTripathi,YueYang,JaeSungPark,Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head,...

    2024 arXiv

  2. [10]

    Virgo: A preliminary exploration on reproducing o1-like MLLM

    Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Virgo: A preliminary exploration on reproducing o1-like MLLM. arXiv preprint arXiv:2501.01904, 2025

    2025 arXiv

  3. [11]

    Revisiting the necessity of lengthy chain-of-thought in vision-centric reasoning.arXiv preprint arXiv:2511.22586, 2025

    Yifan Du, Kun Zhou, Yingqian Min, Yue Ling, Wayne Xin Zhao, and Youbin Wu. Revisiting the necessity of lengthy chain-of-thought in vision-centric reasoning.arXiv preprint arXiv:2511.22586, 2025

    2025

  4. [12]

    VLMEvalKit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. VLMEvalKit: An open-source toolkit for evaluating large multi-modality models. InProceedings of the 32nd ACM International Confere...

    2024

  5. [13]

    GRIT: Teaching MLLMs to think with images.arXiv preprint arXiv:2505.15879, 2025

    Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Naraya- naraju, Xinze Guan, and Xin Eric Wang. GRIT: Teaching MLLMs to think with images.arXiv preprint arXiv:2505.15879, 2025

    2025 arXiv

  6. [14]

    Reward shaping to mitigate reward hacking in RLHF.arXiv preprint arXiv:2502.18770, 2025

    Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in RLHF.arXiv preprint arXiv:2502.18770, 2025

    2025 arXiv

  7. [15]

    Smith, Wei-Chiu Ma, and Ranjay Krishna

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390, 2024. 12 Thinking With Tools, Not With Pixels

    2024 arXiv

  8. [16]

    Thinking with deltas: Incen- tivizingreinforcementlearningviadifferentialvisualreasoningpolicy.arXivpreprintarXiv:2601.06801, 2026

    Shujian Gao, Yuan Wang, Jiangtao Yan, Zuxuan Wu, and Yu-Gang Jiang. Thinking with deltas: Incen- tivizingreinforcementlearningviadifferentialvisualreasoningpolicy.arXivpreprintarXiv:2601.06801, 2026

    2026

  9. [17]

    Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Google. Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  10. [18]

    GLM-5V-Turbo: Toward a native foundation model for multimodal agents.arXiv preprint arXiv:2604.26752, 2026

    GLM-V Team. GLM-5V-Turbo: Toward a native foundation model for multimodal agents.arXiv preprint arXiv:2604.26752, 2026

    2026 arXiv

  11. [19]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    2023

  12. [20]

    Training large language models to reason in a continuous latent space

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. InarXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  13. [21]

    DeepEyesV2: Toward agentic multimodal model.arXiv preprint arXiv:2511.05271, 2025

    Jack Hong, Chenxiao Zhao, ChengLin Zhu, Weiheng Lu, Guohai Xu, and Xing Yu. DeepEyesV2: Toward agentic multimodal model.arXiv preprint arXiv:2511.05271, 2025

    2025 arXiv

  14. [22]

    Hollon, and Bryan Wang

    Xinhai Hou, Shaoyuan Xu, Manan Biyani, Moyan Li, Jia Liu, Todd C. Hollon, and Bryan Wang. CodeV: Code with images for faithful visual reasoning via tool-aware policy optimization.arXiv preprint arXiv:2511.19661, 2025

    2025

  15. [23]

    Visual sketchpad: Sketching as a visual chain of thought for multimodal language models.arXiv preprint arXiv:2406.09403, 2024

    Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, et al. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models.arXiv preprint arXiv:2406.09403, 2024

    2024 arXiv

  16. [24]

    VerlTool: Towards holistic agentic reinforcement learning with tool use.arXiv preprint arXiv:2509.01055, 2025

    Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, Tianyu Pang, and Wenhu Chen. VerlTool: Towards holistic agentic reinforcement learning with tool use.arXiv preprint arXiv:2509.01055, 2025

    2025

  17. [25]

    Kimi K2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    Kimi Team. Kimi K2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    2026 arXiv

  18. [26]

    Mini-o3: Scaling reasoning patterns and interaction turns for visual reasoning.arXiv preprint arXiv:2509.07969, 2025

    Xin Lai, Junyi Li, Wei Li, Tao Liu, Tianjian Li, and Hengshuang Zhao. Mini-o3: Scaling reasoning patterns and interaction turns for visual reasoning.arXiv preprint arXiv:2509.07969, 2025

    2025 arXiv

  19. [27]

    Bowman, and Ethan Perez

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamil˙e Lukoši¯ut˙e, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandl...

    2023 arXiv

  20. [28]

    Zebra-cot: A dataset for interleaved vision language reasoning.arXiv preprint arXiv:2507.16746, 2025

    Ang Li, Charles Wang, Deqing Fu, Kaiyu Yue, Zikui Cai, Wang Bill Zhu, Ollie Liu, Peng Guo, Willie Neiswanger, Furong Huang, Tom Goldstein, and Micah Goldblum. Zebra-cot: A dataset for interleaved vision language reasoning.arXiv preprint arXiv:2507.16746, 2025

    2025

  21. [29]

    Tir-bench: A comprehensive benchmark for agentic thinking-with-images reasoning.arXiv preprint arXiv:2511.01833, 2025

    Ming Li, Jike Zhong, Shitian Zhao, Haoquan Zhang, Shaoheng Lin, et al. Tir-bench: A comprehensive benchmark for agentic thinking-with-images reasoning.arXiv preprint arXiv:2511.01833, 2025. 13 Thinking With Tools, Not With Pixels

    2025

  22. [30]

    Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.arXiv preprint arXiv:2503.06520, 2025

    Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, et al. Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.arXiv preprint arXiv:2503.06520, 2025

    2025 arXiv

  23. [31]

    On the faithfulness of visual thinking: Measurement and enhancement.arXiv preprint arXiv:2510.23482, 2025

    Zujing Liu, Junwen Pan, Qi She, Yuan Gao, and Guisong Xia. On the faithfulness of visual thinking: Measurement and enhancement.arXiv preprint arXiv:2510.23482, 2025

    2025

  24. [32]

    Chain-of-spot: Interactive reasoning improves large vision-language models.arXiv preprint arXiv:2403.12966, 2024

    Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models.arXiv preprint arXiv:2403.12966, 2024

    2024 arXiv

  25. [33]

    Chameleon: Plug-and-play compositional reasoning with large language models

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. In Advances in Neural Information Processing Systems, 2023

    2023

  26. [34]

    Reasoning models can be effective without thinking.arXiv preprint arXiv:2504.09858, 2025

    Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking.arXiv preprint arXiv:2504.09858, 2025

    2025 arXiv

  27. [35]

    What does vision tool-use reinforcement learning really learn? disentangling tool-induced and intrinsic effects for crop-and- zoom.arXiv preprint arXiv:2602.01334, 2026

    Yan Ma, Weiyu Zhang, Tianle Li, Linge Du, Xuyang Shen, and Pengfei Liu. What does vision tool-use reinforcement learning really learn? disentangling tool-induced and intrinsic effects for crop-and- zoom.arXiv preprint arXiv:2602.01334, 2026

    2026 arXiv

  28. [36]

    ChartQA:Abenchmarkfor question answering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    AhmedMasry, DoXuanLong, JiaQingTan, ShafiqJoty, andEnamulHoque. ChartQA:Abenchmarkfor question answering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  29. [37]

    GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229, 2024

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229, 2024

    arXiv

  30. [38]

    Show your work: Scratchpads for intermediate computation with language models.arXiv preprint arXiv:2112.00114, 2021

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Biber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models.arXiv pr...

    2021 arXiv

  31. [39]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  32. [40]

    Thinking with images.https://openai.com/index/thinking-with-images/, 2025

    OpenAI. Thinking with images.https://openai.com/index/thinking-with-images/, 2025. Blog post, April 16, 2025

    2025

  33. [41]

    Do MLLMs really see it: Reinforcing visual attention in multimodal LLMs.arXiv preprint arXiv:2602.08241, 2026

    Siqu Ou, Tianrui Wan, Zhiyuan Zhao, Junyu Gao, and Xuelong Li. Do MLLMs really see it: Reinforcing visual attention in multimodal LLMs.arXiv preprint arXiv:2602.08241, 2026

    2026

  34. [42]

    Cogcom: A visual language model with chain-of-manipulations reasoning.arXiv preprint arXiv:2402.04236, 2024

    Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, et al. Cogcom: A visual language model with chain-of-manipulations reasoning.arXiv preprint arXiv:2402.04236, 2024

    2024 arXiv

  35. [43]

    Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution

    Qwen Team. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  36. [44]

    Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025

    Qwen Team. Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  37. [45]

    Qwen3-VL technical report.arXiv preprint arXiv:2511.21631, 2025

    Qwen Team. Qwen3-VL technical report.arXiv preprint arXiv:2511.21631, 2025

    2025 arXiv

  38. [46]

    Vision language models are blind.arXiv preprint arXiv:2407.06581, 2024

    Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind.arXiv preprint arXiv:2407.06581, 2024. 14 Thinking With Tools, Not With Pixels

    2024 arXiv

  39. [47]

    Tarr, Aviral Kumar, and Katerina Fragkiadaki

    Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael J. Tarr, Aviral Kumar, and Katerina Fragkiadaki. ViGoRL: Grounded reinforcement learning for visual reasoning.arXiv preprint arXiv:2505.23678, 2025

    2025 arXiv

  40. [48]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Lipkin, Fabio Petroni, Luke Zettlemoyer, and Mike Lewis. Toolformer: Language models can teach themselves to use tools. InAdvances in Neural Information Processing Systems, 2023

    2023

  41. [49]

    Visual CoT: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning.arXiv preprint arXiv:2403.16999, 2024

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Huang, Letian Gao, Shuai Li, Yu Liu, and Ping Luo. Visual CoT: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning.arXiv preprint arXiv:2403.16999, 2024

    2024 arXiv

  42. [50]

    HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. InAdvances in Neural Information Processing Systems, 2023

    2023

  43. [51]

    HybridFlow: A flexible and efficient RLHF framework.arXiv preprint arXiv:2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework.arXiv preprint arXiv:2409.19256, 2024

    2024 arXiv

  44. [52]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  45. [53]

    Breaking the chain: A causal analysis of LLM faithfulness to intermediate structures.arXiv preprint arXiv:2603.16475, 2026

    Oleg Somov, Mikhail Chaichuk, Mikhail Seleznyov, Alexander Panchenko, and Elena Tutubalina. Breaking the chain: A causal analysis of LLM faithfulness to intermediate structures.arXiv preprint arXiv:2603.16475, 2026

    2026 arXiv

  46. [54]

    OpenThinkIMG: Learning to think with images via visual tool reinforcement learning.arXiv preprint arXiv:2505.08617, 2025

    Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, and Yu Cheng. OpenThinkIMG: Learning to think with images via visual tool reinforcement learning.arXiv preprint arXiv:2505.08617, 2025

    2025 arXiv

  47. [55]

    Zhaochen Su, Peng Xia, Hangyu Guo, Zhenhua Liu, Yan Ma, Xiaoye Qu, Jiaqi Liu, Yanshu Li, Kaide Zeng, Zhengyuan Yang, Linjie Li, Yu Cheng, Heng Ji, Junxian He, and Yi R. Fung. Thinking with images for multimodal reasoning: Foundations, methods, and future frontiers.arXiv prepri...

    2025 arXiv

  48. [56]

    When thinking hurts: Mitigating visual forgetting via frame repetition.arXiv preprint arXiv:2603.16256, 2026

    Xiaokun Sun, Yubo Wang, Haoyu Cao, and Linli Xu. When thinking hurts: Mitigating visual forgetting via frame repetition.arXiv preprint arXiv:2603.16256, 2026

    2026

  49. [57]

    FACT-E: Causality-inspired evaluation for trustworthy chain-of-thought reasoning.arXiv preprint arXiv:2604.10693, 2026

    Yuxi Sun, Aoqi Zuo, Haotian Xie, Wei Gao, Mingming Gong, and Jing Ma. FACT-E: Causality-inspired evaluation for trustworthy chain-of-thought reasoning.arXiv preprint arXiv:2604.10693, 2026

    2026 arXiv

  50. [58]

    ViperGPT: Visual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual inference via python execution for reasoning. InInternational Conference on Computer Vision, 2023

    2023

  51. [59]

    CV-Bench: A computer vision benchmark for evaluating visual perception in multimodal language models

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yifan Ma, Yongming Zhang, Yann LeCun, and Saining Xie. CV-Bench: A computer vision benchmark for evaluating visual perception in multimodal language models. InNeurIPS 2024 Workshop on Position-Aware AI, 2024

    2024

  52. [60]

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. 2023. 15 Thinking With Tools, Not With Pixels

    2023

  53. [61]

    Journey before destination: Visual faithfulness in slow thinking.arXiv preprint arXiv:2512.12218, 2025

    Rheeya Uppaal, Phu Mon Htut, Min Bai, Nikolaos Pappas, Zheng Qi, and Sandesh Swamy. Journey before destination: Visual faithfulness in slow thinking.arXiv preprint arXiv:2512.12218, 2025

    2025

  54. [62]

    GeoEyes: On-demand visual focusing for ultra-high-resolution remote sensing.arXiv preprint arXiv:2602.14201, 2026

    Fengxiang Wang, Mingshuo Chen, Yueying Li, Yajie Yang, Yifan Zhang, Long Lan, Xue Yang, Hongda Sun, Yulin Wang, Di Wang, Jun Song, Jing Zhang, and Bo Du. GeoEyes: On-demand visual focusing for ultra-high-resolution remote sensing.arXiv preprint arXiv:2602.14201, 2026

    2026

  55. [63]

    Pixel-reasoner: Incentivizing pixel-scale reasoning with curiosity-driven reinforcement learning.arXiv preprint arXiv:2505.15966, 2025

    Haozhe Wang, Alex Su, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel-reasoner: Incentivizing pixel-scale reasoning with curiosity-driven reinforcement learning.arXiv preprint arXiv:2505.15966, 2025

    2025 arXiv

  56. [64]

    PLaT: Latent chain-of-thought as planning: Decoupling reasoning from verbalization.arXiv preprint arXiv:2601.21358, 2026

    Jiecong Wang, Hao Peng, and Chunyang Liu. PLaT: Latent chain-of-thought as planning: Decoupling reasoning from verbalization.arXiv preprint arXiv:2601.21358, 2026

    2026

  57. [65]

    VAGEN: Reinforcing world model reasoning for multi-turn VLM agents.arXiv preprint arXiv:2510.16907, 2025

    Kangrui Wang, Pingyue Zhang, Zihan Wang, Yaning Gao, Linjie Li, Qineng Wang, Hanyang Chen, Yiping Lu, Zhengyuan Yang, Lijuan Wang, Ranjay Krishna, Jiajun Wu, Li Fei-Fei, Yejin Choi, and Manling Li. VAGEN: Reinforcing world model reasoning for multi-turn VLM agents.arXiv prepri...

  58. [66]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistic...

    2023

  59. [67]

    A practitioner’s guide to multi-turn agentic reinforcement learning.arXiv preprint arXiv:2510.01132, 2025

    Ruiyi Wang and Prithviraj Ammanabrolu. A practitioner’s guide to multi-turn agentic reinforcement learning.arXiv preprint arXiv:2510.01132, 2025

    2025

  60. [68]

    Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models.arXiv preprint arXiv:2408.15556, 2024

    Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models.arXiv preprint arXiv:2408.15556, 2024

    2024 arXiv

  61. [69]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023

    2023

  62. [70]

    Simple o3: Towards interleaved vision-language reasoning.arXiv preprint arXiv:2508.12109, 2025

    Ye Wang, Qianglong Chen, Zejun Li, Siyuan Wang, Shijie Guo, Zhirui Zhang, and Zhongyu Wei. Simple o3: Towards interleaved vision-language reasoning.arXiv preprint arXiv:2508.12109, 2025

    2025 arXiv

  63. [71]

    RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning.arXiv preprint arXiv:2504.20073, 2025

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. RAGEN: Understanding self-evolution in LL...

    2025 arXiv

  64. [72]

    CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs.arXiv preprint arXiv:2406.18521, 2024

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs.arXiv preprint arXiv:2406.18...

    2024 arXiv

  65. [73]

    V-FAT: Benchmarking visual fidelity against text-bias.arXiv preprint arXiv:2601.04897, 2025

    Ziteng Wang, Yujie He, Guanliang Li, Siqi Yang, Jiaqi Xiong, and Songxiang Liu. V-FAT: Benchmarking visual fidelity against text-bias.arXiv preprint arXiv:2601.04897, 2025

    2025

  66. [74]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022. 16 Thinking With Tools, Not ...

    2022

  67. [75]

    Zooming without zooming: Region- to-image distillation for fine-grained multimodal perception.arXiv preprint arXiv:2602.11858, 2026

    Lai Wei, Liangbo He, Jun Lan, Lingzhong Dong, Yutong Cai, Siyuan Li, Huijia Zhu, Weiqiang Wang, Linghe Kong, Yue Wang, Zhuosheng Zhang, and Weiran Huang. Zooming without zooming: Region- to-image distillation for fine-grained multimodal perception.arXiv preprint arXiv:2602.11858, 2026

    2026

  68. [76]

    Visual generation unlocks human-like reasoning through multimodal world models.arXiv preprint arXiv:2601.19834, 2026

    Jialong Wu, Xiaoying Zhang, Hongyi Yuan, Xiangcheng Zhang, Tianhao Huang, Changjing He, Chaoyi Deng, Renrui Zhang, Youbin Wu, and Mingsheng Long. Visual generation unlocks human-like reasoning through multimodal world models.arXiv preprint arXiv:2601.19834, 2026

    2026

  69. [77]

    VTool-R1: VLMs learn to think with images via reinforcement learning on multimodal tool use.arXiv preprint arXiv:2505.19255, 2025

    Mingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengxiang Zhai, and Klara Nahrstedt. VTool-R1: VLMs learn to think with images via reinforcement learning on multimodal tool use.arXiv preprint arXiv:2505.19255, 2025

    2025

  70. [78]

    V*: Guided visual search as a core mechanism in multimodal LLMs

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal LLMs. arXiv preprint arXiv:2312.14135, 2023

    2023 arXiv

  71. [79]

    Tool-augmented policy optimization.arXiv preprint arXiv:2510.07038, 2025

    Wenxun Wu, Yuanyang Li, Guhan Chen, Linyue Wang, and Hongyang Chen. Tool-augmented policy optimization.arXiv preprint arXiv:2510.07038, 2025

    2025

  72. [80]

    Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks.arXiv preprint arXiv:2307.02477, 2023

    Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks.arXiv preprint arXiv:2307.02477, 2023

    2023 arXiv

  73. [81]

    LLaVA-CoT: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024

    Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. LLaVA-CoT: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024

    2024 arXiv

  74. [82]

    Act wisely: Cultivating meta-cognitive tool use in agentic multimodal models.arXiv preprint arXiv:2604.08545, 2026

    Shilin Yan, Jintao Tong, Hongwei Xue, Xiaojun Tang, Yangyang Wang, Kunyu Shi, Guannan Zhang, Ruixuan Li, and Yixiong Zou. Act wisely: Cultivating meta-cognitive tool use in agentic multimodal models.arXiv preprint arXiv:2604.08545, 2026

    2026 arXiv

  75. [83]

    Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V.arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  76. [84]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024

    2024 arXiv

  77. [85]

    VisionThink: Smart and efficient visual language model via reinforcement learning.arXiv preprint arXiv:2507.13348, 2025

    Senqiao Yang, Junyi Li, Xin Lai, Bei Yu, Hengshuang Zhao, and Jiaya Jia. VisionThink: Smart and efficient visual language model via reinforcement learning.arXiv preprint arXiv:2507.13348, 2025

    2025 arXiv

  78. [86]

    Look-back: Implicit visual re-focusing in MLLM reasoning.arXiv preprint arXiv:2507.03019, 2025

    Shuo Yang, Yuwei Niu, Yuyang Liu, Yang Ye, Bin Lin, and Li Yuan. Look-back: Implicit visual re-focusing in MLLM reasoning.arXiv preprint arXiv:2507.03019, 2025

    2025 arXiv

  79. [87]

    Walk the talk: Bridging the reasoning-action gap for thinking with images via multimodal agentic policy optimization.arXiv preprint arXiv:2604.06777, 2026

    Wenhao Yang, Yu Xia, Jinlong Huang, Shiyin Lu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Yuchen Zhou, Xiaobo Xia, Yuanyu Wan, Lijun Zhang, and Tat-Seng Chua. Walk the talk: Bridging the reasoning-action gap for thinking with images via multimodal agentic policy optimiza...

    2026 arXiv

  80. [88]

    Thinking with images via self-calling agent

    Wenxi Yang, Yuzhong Zhao, Fang Wan, and Qixiang Ye. Thinking with images via self-calling agent. arXiv preprint arXiv:2512.08511, 2025. 17 Thinking With Tools, Not With Pixels

    2025

  81. [89]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. InAdvances in Neural Information Processing Systems, 2023

    2023

  82. [90]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023

    2023

  83. [91]

    Richard J. Young. Lie to me: How faithful is chain-of-thought reasoning in reasoning models?arXiv preprint arXiv:2603.22582, 2026

    2026

  84. [92]

    ProRL agent: Rollout-as-a-service for reinforcement learning training of multi-turn LLM agents.arXiv preprint arXiv:2603.18815, 2026

    Hao Zhang, Mingjie Liu, Shaokun Zhang, Songyang Han, Jian Hu, Zhenghui Jin, Yuchi Zhang, Shizhe Diao, Ximing Lu, Binfeng Xu, Zhiding Yu, Jan Kautz, and Yi Dong. ProRL agent: Rollout-as-a-service for reinforcement learning training of multi-turn LLM agents.arXiv preprint arXiv:...

    2026

  85. [93]

    MM-CoT: A benchmark for probing visual chain-of-thought reasoning.arXiv preprint arXiv:2512.08228, 2025

    Jusheng Zhang, Kaitong Cai, Xiaoyang Guo, Sidi Liu, Qinhan Lv, Ruiqi Chen, Jing Yang, Yijia Fan, Xiaofei Sun, Jian Wang, Ziliang Chen, Liang Lin, and Keze Wang. MM-CoT: A benchmark for probing visual chain-of-thought reasoning.arXiv preprint arXiv:2512.08228, 2025

    2025

  86. [94]

    LLaVA-Mini: Efficient image and video large multimodal models with one vision token.arXiv preprint arXiv:2501.03895, 2025

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. LLaVA-Mini: Efficient image and video large multimodal models with one vision token.arXiv preprint arXiv:2501.03895, 2025

    2025 arXiv

  87. [95]

    MME-RealWorld: Could your multimodal LLM challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257, 2024

    Yi-FanZhang, HuanyuZhang, HaochenTian, ChaoyouFu, ShuangqingZhang, JunfeiWu, FengLi, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. MME-RealWorld: Could your multimodal LLM challenge high-resolution real-world scenarios that are difficult for humans...

    2024 arXiv

  88. [96]

    Thyme: Think beyond images

    Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, et al. Thyme: Think beyond images. arXiv preprint arXiv:2508.11630, 2025

    2025 arXiv

  89. [97]

    Skywork-r1v4: Toward agentic multimodal intelligence through interleaved thinking with images and deepresearch.arXiv preprint arXiv:2512.02395, 2025

    Yifan Zhang, Liang Hu, Haofeng Sun, Peiyu Wang, Yichen Wei, et al. Skywork-r1v4: Toward agentic multimodal intelligence through interleaved thinking with images and deepresearch.arXiv preprint arXiv:2512.02395, 2025

    2025

  90. [98]

    CM2: Reinforcement learning with checklist rewards for multi-turn and multi-step agentic tool use.arXiv preprint arXiv:2602.12268, 2026

    Zhen Zhang, Kaiqiang Song, Xun Wang, Yebowen Hu, Weixiang Yan, Chenyang Zhao, Henry Peng Zou, Haoyun Deng, Sathish Reddy Indurthi, Shujian Liu, Simin Ma, Xiaoyang Wang, Xin Eric Wang, and Song Wang. CM2: Reinforcement learning with checklist rewards for multi-turn and multi-st...

    2026

  91. [99]

    Multimodal chain-of-thought reasoning in language models.arXiv preprint arXiv:2302.00923, 2023

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models.arXiv preprint arXiv:2302.00923, 2023

    2023 arXiv

  92. [100]

    On robustness and chain-of-thought consistency of RL-finetuned VLMs.arXiv preprint arXiv:2602.12506, 2026

    Rosie Zhao, Anshul Shah, Xiaoyu Zhu, Xinke Deng, Zhongyu Jiang, Yang Yang, Joerg Liebelt, and Arnab Mondal. On robustness and chain-of-thought consistency of RL-finetuned VLMs.arXiv preprint arXiv:2602.12506, 2026

    2026 arXiv

  93. [101]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs.arXiv preprint arXiv:2312.07104, 20...

    2023 arXiv

  94. [102]

    LlamaFactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. LlamaFactory: Unified efficient fine-tuning of 100+ language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System D...

    2024

  95. [103]

    thinking with images

    Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing “thinking with images” via reinforcement learning.arXiv preprint arXiv:2505.14362, 2025

    2025 arXiv

  96. [104]

    Least-to-most prompting enables complex reasoning in large language models

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. InInternational Conference on Learning Representations, 2023

    2023

  97. [105]

    When visual- izing is the first step to reasoning: MIRA, a benchmark for visual chain-of-thought.arXiv preprint arXiv:2511.02779, 2025

    Yiyang Zhou, Haoqin Tu, Zijun Wang, Zeyu Wang, Niklas Muennighoff, Fan Nie, Yejin Choi, James Zou, Chaorui Deng, Shen Yan, Haoqi Fan, Cihang Xie, Huaxiu Yao, and Qinghao Ye. When visual- izing is the first step to reasoning: MIRA, a benchmark for visual chain-of-thought.arXiv ...

    2025

  98. [106]

    What is the color of the pink-haired woman’s handbag?

    Yujin Zhou, Pengcheng Wen, Jiale Chen, Boqin Yin, Han Zhu, Jiaming Ji, Juntao Dai, Chi-Min Chan, and Sirui Han. Process reward for thinking with images.arXiv preprint arXiv:2602.08346, 2026. A. Additional Experimental Setup Hardware.All training and evaluation jobs use NVIDIA ...

    2026

  99. [107]

    It interrupts a line of red ketchup and is immediately to the left of a green cucumber slice

    **Top Gap **: Located on the sandwich in the middle row. It interrupts a line of red ketchup and is immediately to the left of a green cucumber slice

  100. [108]

    name": "image_edit

    **Bottom Gap **: Located on the sandwich in the bottom row, towards the right end. It covers the bun and the end of the sausage. **Strategy**: I will test each option (A, B, C, D) by checking if its two pieces contain the specific visual features required to seamlessly fill th...

  101. [109]

    **SELECT the pieces from Option A **: Locate the two star-shaped image fragments under the label ’A’ at the bottom left

  102. [110]

    **IDENTIFY the target gaps **: Locate the two gray star-shaped holes in the ’PUZZLE INPUT’ image (one on the top-right sandwich, one on the bottom-center sandwich)

  103. [111]

    - Take the right piece from Option A (showing ham and bread) and place it over the other gap

    **PLACE the pieces **: - Take the left piece from Option A (showing a metal tray edge) and place it over one of the gaps. - Take the right piece from Option A (showing ham and bread) and place it over the other gap

  104. [112]

    The goal is to see if the ingredients (sausage, cucumber, bread, tray) match the surrounding context perfectly

    **BLEND**: Attempt to align the edges. The goal is to see if the ingredients (sausage, cucumber, bread, tray) match the surrounding context perfectly

  105. [113]

    name": "image_edit

    **OUTPUT**: Show the modified main image with the Option A pieces filled in. , "img_idx": 0}} </tool_call> <tool_response> Image edit successful. <image> </tool_response> assistant: Let me check the result: The user successfully placed the pieces from Option A into the designa...

  106. [114]

    - Right piece: Shows a segment of an orange sausage on a bun

    **SELECT the pieces from Option B ** at the bottom of the image: - Left piece: Shows golden fried dough texture with a dark metallic tray edge. - Right piece: Shows a segment of an orange sausage on a bun

  107. [115]

    - Align the piece so the texture fills the hole

    **PLACE the Left Piece ** into the **Top Gap **: - The top gap is the gray star shape on the sandwich in the middle-right. - Align the piece so the texture fills the hole

  108. [116]

    - Align the sausage segment in the piece with the sausage in the sandwich

    **PLACE the Right Piece ** into the **Bottom Gap **: - The bottom gap is the gray star shape on the sandwich in the bottom-left (cutting through the sausage). - Align the sausage segment in the piece with the sausage in the sandwich

  109. [117]

    name": "image_edit

    **PRESERVE** the rest of the image exactly as is. , "img_idx": 0}} </tool_call> <tool_response> Image edit successful. <image> </tool_response> Converted SFT chat trajectory (1/3) Figure7:Visual Jigsaw probe: converted SFT chat trajectory.The figure follows the post-conversion...

  110. [118]

    It is a star shape containing a metallic tray edge on the left and fried dough on the right

    **Source - Option B Left Piece **: Identify the left piece under ’B’. It is a star shape containing a metallic tray edge on the left and fried dough on the right

  111. [119]

    **Destination - Top Gap **: Place this Left Piece into the gray star-shaped gap located on the sandwich on the right side of the tray

  112. [120]

    It is a star shape containing an orange sausage segment at the top and fried dough at the bottom

    **Source - Option B Right Piece **: Identify the right piece under ’B’. It is a star shape containing an orange sausage segment at the top and fried dough at the bottom

  113. [121]

    Align the orange sausage segment in the piece with the sausage in the sandwich

    **Destination - Bottom Gap **: Place this Right Piece into the gray star-shaped gap located on the bottom-left sandwich. Align the orange sausage segment in the piece with the sausage in the sandwich

  114. [122]

    name": "image_edit

    **Refinement**: Resize and rotate the pieces slightly if necessary to fit the gray star masks exactly. Remove the gray color completely. Do not alter the surrounding sandwiches. , "img_idx": 0}} </tool_call> <tool_response> Image edit successful. <image> </tool_response> assis...

  115. [123]

    - Bottom Gap: Star-shaped hole on the left sandwich (bottom row)

    **Identify the Gaps **: - Top Gap: Star-shaped hole on the right sandwich (middle row). - Bottom Gap: Star-shaped hole on the left sandwich (bottom row)

  116. [124]

    Align the red ketchup line and green cucumber with the existing ingredients on the sandwich

    **Place Option C Pieces **: - Take the **left piece of Option C ** (containing the red ketchup line and green cucumber slice) and place it into the **Top Gap **. Align the red ketchup line and green cucumber with the existing ingredients on the sandwich. - Take the **right pie...

  117. [125]

    - Remove the gray placeholder color completely in the filled areas

    **Refine**: - Ensure the textures (bun, sausage, sauce) blend seamlessly with the surrounding image. - Remove the gray placeholder color completely in the filled areas. , "img_idx": 0}} </tool_call> <tool_response> Image edit successful. <image> </tool_response> Converted SFT ...

  118. [126]

    **Option A **: Eliminated

  119. [127]

    **Option B **: Eliminated

  120. [128]

    **Option C **: Success

  121. [129]

    Image edit successful. [no image returned]

    **Option D **: Not tested, but unnecessary. Based on the visual analysis above, I can conclude that: Option C Converted SFT chat trajectory (3/3) ✓ Figure7:Visual Jigsaw probe: converted SFT chat trajectory, continued.All four tool-call blocks preserve the rawimage_edit prompt...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.