{"id":"d991a597-3780-4b81-b305-f6c495d31822","arxiv_id":"2608.08907","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"A capability-aligned SFT and necessity-gated RL recipe lets an 8B multimodal model use visual tools effectively, outperforming larger models on high-resolution perception benchmarks.","lead":"ToolVision trains an 8B vision-language model to use visual tools such as OCR, detection, depth, and counting through a two-stage recipe: filter tool-use examples by evidence gain for student-scale models, then reward tool use only where the model itself benefits. It reports gains on seven benchmarks and beats larger models on high-resolution visual search tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"External comparisons in Table 1 mix decoding protocols: ToolVision reports avg@4 at temperature 0.7 while baseline numbers are taken from other reports, with margins as small as 1.1 points over Qwen3-VL-32B-Thinking; the headline superiority claim is not yet established under a common protocol.","rationale":"The reader's weakest assumption identifies the same general area: external baseline numbers may come from a different evaluation protocol. I agree this is the most load-bearing concern because the headline claims of surpassing Thyme, CodeVision, CodeDance, and especially Qwen3-VL-32B-Thinking rest entirely on Table 1 cross-paper comparisons. However, I would refine the mechanism: avg@4 is an average over independent scored samples, so it is not a best-of-N or majority-vote aggregation and does not inflate accuracy in expectation. The real risks are that temperature-0.7 sampling can behave differently from greedy decoding, that the 12-turn tool-enabled agent loop is a different evaluation condition from the baselines' likely single-turn protocol, and that the reference-conditioned judge may be more lenient than the matchers used in the original reports. These risks are concrete enough to matter: the two headline margins over the 32B model are only 1.1 points, comparable in scale to the reported sampling standard deviations. No internal inconsistency is apparent, and the method-level ablations (ToolVision SFT vs Distilled SFT; necessity-conditioned vs uniform reward) are internally controlled and support the central methodological story. I also considered the checkpoint-selection-by-training-reward disclosure in Appendix H.4 and the ArxivQA figure-overlap caveat in Appendix B.2, but those affect secondary claims or are explicitly disclosed rather than threatening the main external comparisons. The reader's CONDITIONAL verdict already reflects the need for verification, so my read does not move the verdict; it sharpens the specific test that should be run before the external superiority claims are accepted.","tokens_in":24143,"tokens_out":13666,"duration_ms":143847,"concrete_test":"Re-run Qwen3-VL-32B-Thinking (and, if feasible, Thyme-7B, CodeVision-8B, and CodeDance-7B) through ToolVision's exact evaluation harness on V* and HRBench 4K/8K: same prompts, same LMMs-Eval metric implementations, same reference-conditioned judge, same up-to-12-turn agent loop, and same avg@4 at temperature 0.7 over four samples per question. Recompute Table 1 from these numbers. If Qwen3-VL-32B-Thinking reaches or exceeds 85.9 on V* or 75.9 on HRBench 8K under this protocol, the 'outperforms Qwen3-VL-32B-Thinking' claim is attributable to evaluation protocol rather than to the training method. A complementary single-run check is to evaluate ToolVision-8B with greedy decoding under the original baseline protocols and compare whether the reported margins persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest empirical claim depends on Table 1, where ToolVision-8B's numbers are produced by ToolVision's own harness (Section 4.1: four sampled trajectories per question at temperature 0.7, reported as avg@4, up to 12 assistant turns, with a reference-conditioned LLM judge for semantic equivalence), while the baseline numbers for Thyme-7B, CodeVision-8B, CodeDance-7B, and Qwen3-VL-32B-Thinking are copied from other papers or reports. The paper does not establish that those baselines used the same decoding temperature, the same turn budget, the same tool harness, or the same semantic-equivalence judge. To be precise, avg@4 is a mean over four scored samples, not majority-vote or best-of-4, so it does not inflate accuracy in expectation; however, a temperature-0.7 sampling policy can score differently from a greedy-decoding policy, and a 12-turn tool-enabled agent loop is not the same evaluation condition as a single-turn answer extraction. The two specific 'outperforms Qwen3-VL-32B-Thinking' margins are both 1.1 points (85.9 vs 84.8 on V*; 75.9 vs 74.8 on HRBench 8K), while the reported per-run sampling standard deviations in Table 6 are 0.79 and 0.50 respectively. Without re-running the external baselines under a shared protocol, these margins are within plausible protocol-induced variation, so the headline claim that ToolVision surpasses a 32B reasoning model is not yet robustly supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ToolVision, a two-stage method for training a small multimodal LLM (Qwen3-VL-8B-Thinking) to use visual tools through code. During SFT, a Planner–Executor multi-agent search generates candidate trajectories, and an eight-model committee scores stepwise evidence gain to prune branches; only trajectories with successful execution, positive evidence gain, and correct final answers are kept. During RL, paired rollouts of the frozen SFT model with and without tools are used to assign per-question must-use-tool (MUT) weights (0.5, 0.2, 0), and the tool-success reward is gated by these weights, with an overuse penalty. The final model is evaluated on nine benchmarks, reporting improvements over its base on all seven main benchmarks, superiority over Thyme-7B, CodeVision-8B, and CodeDance-7B on three high-resolution benchmarks, and better scores than Qwen3-VL-32B-Thinking on V* and HRBench 8K. The appendix contains detailed algorithms, data sources, prompts, and reproducibility settings.","tokens_in":24564,"tokens_out":5556,"duration_ms":58307,"significance":"The manuscript makes a relevant contribution to the emerging 'thinking with images' area. The core conceptual claim — that SFT should filter tool trajectories by evidence usable at the student scale, and that RL should reward tool use only where the frozen learner itself benefits — is well motivated and supported by the internal ablations. The stage-wise comparisons (Distilled SFT vs ToolVision SFT; uniform bonus vs MUT) show large, consistent gaps, and the release of code, data manifests, and detailed settings strengthens reproducibility. If the headline comparisons to external baselines are reconfirmed under a common protocol, the method would be a useful step for teaching small MLLMs to leverage specialized visual tools. However, the external comparison issue (see major comment 1) and the checkpoint-selection weakness (major comment 2) currently block full confidence in the abstract's strongest claims.","major_comments":[{"comment":"The headline claim that ToolVision-8B outperforms Qwen3-VL-32B-Thinking on V* and HRBench 8K is not yet robustly supported because the baseline numbers are not matched to ToolVision's evaluation protocol. ToolVision-8B scores are avg@4 of four sampled trajectories at temperature 0.7 in a multi-turn agent harness with up to 12 turns and a reference-conditioned LLM judge, whereas the Thyme-7B, CodeVision-8B, CodeDance-7B, and Qwen3-VL-32B-Thinking numbers are cited from other reports without evidence that they used the same decoding scheme, turn budget, or semantic-equivalence judge. The reported margins over Qwen3-VL-32B-Thinking are 1.1 points on both benchmarks, while Table 6 gives per-run sampling standard deviations of 0.79 and 0.50 for ToolVision-8B; the additional uncertainty from protocol differences is unquantified and could easily be of the same magnitude. I request that the authors either re-run all external baselines under the common ToolVision harness (or provide evidence that the cited baseline numbers used an equivalent protocol) or explicitly restrict the superiority claims to comparisons made under the same evaluation conditions.","section":"Section 4.1–4.2, Table 1"},{"comment":"The reported checkpoint is selected as the one with the highest training reward, which occurs at step 180. Because this reward includes the MUT bonus and the overuse penalty, selecting on it can favor checkpoints that overfit the training mixture rather than generalize to benchmarks. The paper should report the training-reward curve and, more importantly, evaluation results on a held-out validation set for several checkpoints around step 180, and justify that the chosen checkpoint is not an artifact of reward maximization. This is particularly relevant because the final numbers in Table 1 and Table 3 all come from this single reward-selected checkpoint.","section":"Section 4.1, Appendix H.4"},{"comment":"The uniform-bonus ablation is used to support the claim that 'the benefit comes from conditioning the tool-success reward on measured necessity rather than from rewarding successful tool use indiscriminately.' While the comparison is internally consistent, only one constant weight (w=0.2) is studied. A sensitivity analysis over a few constant weights (e.g., 0.1, 0.2, 0.5) would strengthen the conclusion that the improvement is due to per-question conditioning rather than to the particular magnitude of the bonus. Without this, the reader cannot fully separate the effect of conditioning from the effect of the chosen weight scale.","section":"Section 4.4, Table 3"}],"minor_comments":[{"comment":"The parenthesized deltas in Table 1 are said to be computed before rounding the displayed scores, but the base-model scores themselves are rounded; a brief note on how these deltas are calculated from the unrounded values would improve transparency.","section":"Section 4.2, Table 1"},{"comment":"The ArxivQA-2k holdout is question-disjoint but not image-disjoint: the paper explicitly states that different questions may reuse the same source figure. This caveat should appear in the main text near Table 2, and ideally the authors should report the number of unique source figures or filter at the image level to rule out image leakage.","section":"Section 4.3, Table 2 and Appendix B.2"},{"comment":"The bar lengths are scaled within each benchmark, so the visual impression of ToolVision's advantage can be misleading across columns; the absolute labels help, but a common-scaled or grouped presentation would be easier to interpret.","section":"Figure 1"},{"comment":"The statement that 'the teacher's own perception rarely needs specialist localization' is used to explain why Distilled SFT contains almost no detector-guided cropping. This claim is plausible but could be supported with a simple statistic (e.g., the percentage of teacher trajectories that invoke the detector), which would make the behavioral comparison in Figure 4 more interpretable.","section":"Appendix A.2"},{"comment":"The term 'avg@4' is nonstandard; it is a mean over four independently scored samples rather than majority vote or best-of-k. Consider clarifying this in a footnote, since readers may otherwise assume a selection-based protocol.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The internal ablations are convincing and the reproducibility package is a strong asset. The main risk is the external comparison: the abstract's claim of beating Qwen3-VL-32B-Thinking rests on margins that are within plausible protocol-induced variation, and the checkpoint is selected on training reward. These issues are fixable by re-running baselines under a common harness and by reporting validation-based checkpoint selection. If the authors can do that, the paper would be a solid candidate for acceptance in this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution with a genuine methodological idea, and the internal evidence holds up. The external comparisons are the weak point, and the \"beats a 32B model\" headline is not yet robust.\n\nWhat's new: the cross-scale committee that scores stepwise evidence gain during SFT, and the offline paired-rollout necessity labels that gate a tool-success reward in RL. Both address a real failure mode — the paper's coordinate-crop example is a good diagnosis of why distilling tool calls from a stronger teacher can teach the wrong thing. The ablations are careful: Distilled SFT vs. ToolVision SFT, and uniform bonus vs. MUT, are run under the same harness, so the stage-wise claims are credible. The method builds supervision automatically from public data, which is a practical plus. The citation pattern is honest; the authors name the closest competitors and build on that line of work.\n\nSoft spots: Table 1 mixes evaluation protocols. ToolVision's numbers are avg@4 at temperature 0.7 under a multi-turn tool harness with a semantic-equivalence judge. The baseline numbers are copied from other papers, and the paper does not show those baselines used the same decoding temperature, turn budget, or judge. The two margins over Qwen3-VL-32B are both 1.1 points, which is inside the reported per-run standard deviation (0.79 and 0.50). So the claim that an 8B model surpasses a 32B reasoning model is not established under a common protocol. This is not a fatal flaw — avg@4 is a mean, so it doesn't inflate expectation — but it needs fixing. Also, the checkpoint is selected as the one with highest training reward (step 180), which is a minor selection concern, and no artifacts are released yet, so the central empirical claims are plausible but not fully verifiable from the preprint.\n\nThe internal comparisons suggest the method itself is sound; it's the external comparisons that need re-running or softening. The paper is worth a serious referee. I'd send it out, and ask for a common-protocol re-evaluation of at least the closest baselines, or a revised claim, plus code/data release.","headline":"Genuinely new training recipe with solid internal ablations; the 'beats a 32B model' claim rests on unmatched evaluation protocols and is not yet robust.","tokens_in":25035,"tokens_out":3306,"would_cite":true,"duration_ms":29511,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a small vision-language model to use visual tools well requires filtering teacher demonstrations and rewards by what the student itself can exploit.","keywords":["visual tool use","thinking with images","capability alignment","evidence gain","must-use-tool reward","multimodal large language models","supervised fine-tuning","reinforcement learning"],"falsifier":"Re-run the external baselines (Thyme-7B, CodeVision-8B, CodeDance-7B, and Qwen3-VL-32B-Thinking) under ToolVision's exact avg@4, temperature-0.7 evaluation harness. If, under that protocol, ToolVision-8B no longer beats all of them on the three high-resolution benchmarks, the headline superiority claim collapses, even if the internal ablations still support the method's value.","tokens_in":898,"feed_emoji":"🔍","tokens_out":6664,"duration_ms":83972,"temperature":0.7,"pith_summary":"This paper argues that the standard \"distill tool-using behavior from a strong teacher, then reinforce with outcome rewards\" recipe fails for small multimodal models in two distinct ways, and that both failures can be traced to a single cause: supervision is not aligned with the learner's own capabilities. During supervised fine-tuning, a student may imitate a teacher's tool calls without being able to reproduce the perception that made them useful; during reinforcement learning, outcome-only rewards suppress tool use while blanket tool bonuses reward ineffective operations. ToolVision fixes each stage with an automatic, capability-conditioned signal: SFT keeps only trajectories whose stepwise evidence gain helps student-scale models, and RL rewards successful tool use only on questions where paired rollouts show the frozen model actually benefits from tools. The resulting 8B model improves over its base on all seven main benchmarks, beats three closely related code-as-tool systems on high-resolution benchmarks, and surpasses a 32B reasoning model on two benchmarks. If these results hold, the recipe offers a general way to align teacher demonstrations and reward signals with any learner's abilities, without extra human annotations.","feed_headline":"8B model beats 32B rival by learning when tools help","feed_subtitle":"ToolVision filters teacher demo steps by evidence gain for the student, then rewards tool use only where measured necessary.","key_machinery":"The load-bearing object is a pair of capability-sensitive supervision signals built on the same toolbelt. First, the stepwise evidence gain $\\Delta_t = J(\\mathcal{E}_t;x) - J(\\mathcal{E}_{t-1};x)$, where $J$ is the average answerability score of an eight-member cross-scale committee—including student-scale models—over the accumulated visual evidence $\\mathcal{E}_t$; branches with negative or repeated zero gain are pruned, so only trajectories whose evidence a student can exploit enter SFT. Second, the must-use-tool (MUT) weight $w(x,\\pi_{\\text{SFT}})$, computed before RL by comparing the frozen SFT model's rollouts with tools disabled and enabled, and used to multiply a tool-success reward $R_{\\text{mut}}$. Both signals are constructed automatically from public task data, and together they make the RL reward depend on both the question and the learner.","core_discovery":"ToolVision-8B's central claim is that capability misalignment, not tool engineering, is what prevents small multimodal models from learning to use visual tools well. The paper proposes replacing unselective imitation and outcome-only reinforcement with two capability-aligned signals: trajectory filtering by stepwise evidence gain during SFT, and model-conditioned must-use-tool rewards during RL. On all seven main benchmarks, ToolVision-8B improves over its base (Qwen3-VL-8B-Thinking); it surpasses Thyme-7B, CodeVision-8B, and CodeDance-7B on all three high-resolution benchmarks (V*, HRBench 4K, HRBench 8K); and it outperforms Qwen3-VL-32B-Thinking on V* and HRBench 8K. The authors also report that the method improves tool-dependent tasks sharply: ArxivQA-2k accuracy rises from 58.3 to 74.5 and FSC-147 MAE drops from 44.7 to 11.6. The paper treats these numbers as evidence that routing perception through the right tool can partly offset model scale.","pith_inferences":["I would extend the paired-rollout necessity measurement to inference-time gating: run the no-tool arm first, and only spend tool compute on questions the model answers incorrectly, mirroring the training filter.","The stepwise evidence-gain committee could serve as a dense process reward for RL, replacing the binary tool-success term with a graded signal and possibly removing the need for the offline MUT measurement.","The capability-alignment principle—filter demonstrations by whether the student can exploit each step—should transfer to other agentic settings such as text-only tool use or code generation, where teacher trajectories also encode abilities the student lacks.","I would test the SFT filter's sensitivity to committee composition by dropping all strong members and keeping only student-scale judges; the paper reports a proposer-sensitivity check but not this particular ablation."],"forward_implications":["An 8B model can beat a 32B model on high-resolution perception by delegating localization to an external detector instead of predicting crop coordinates itself.","SFT data should be filtered by the student's own ability to use each step's evidence, not by whether the teacher's trajectory succeeds.","Reinforcement learning should reward tool use only where the frozen student model measurably benefits, avoiding both tool collapse and invalid overuse.","Because the must-use-tool labels are computed once and fixed, the 716-GPU-hour measurement cost is amortized across every subsequent RL run.","A policy trained this way specializes sharply after RL: counting invocations saturate on counting tasks, OCR on text-reading tasks, and detection-guided cropping on high-resolution search."],"supporting_citations":[{"why":"Documents the code-as-tool paradigm and the observation that unreliable code generation can push a policy toward direct answers, which motivates the MUT reward.","marker":"Zhang et al. 2026"},{"why":"Provides CodeVision's released evaluation procedure, which the paper adopts, and the seed rotate/flip examples used in SFT.","marker":"Guo et al. 2026"},{"why":"CodeDance, the main baseline whose dynamic group-based tool incentive the paper contrasts with its model-conditioned necessity measurement.","marker":"Song et al. 2026"},{"why":"Reports Qwen3-VL-32B-Thinking and 30B numbers used as large-model baselines, and defines the base model family.","marker":"Bai et al. 2025a"},{"why":"Supplies the Group Sequence Policy Optimization algorithm used for RL.","marker":"Zheng et al. 2025"},{"why":"Defines the V* benchmark and the guided-visual-search problem ToolVision targets.","marker":"Wu and Xie 2024"},{"why":"Defines HRBench 4K/8K, the high-resolution benchmarks where ToolVision reports large gains.","marker":"Wang et al. 2025"},{"why":"Defines OCRBench, a text-reading benchmark used in main results and task-tool specialization analysis.","marker":"Liu et al. 2024b"}],"fun_headline_variants":["8B model beats 32B by learning when tools help","ToolVision: filter teacher demos by evidence gain for small VLMs","Capability-aligned rewards teach small models when tools matter","Small multimodal model tops 32B rival on high-res benchmarks","ToolVision: 8B beats 32B by aligning tool supervision"],"cache_read_input_tokens":27136,"weakest_assumption_plain":"The headline comparisons assume that the external baseline numbers from Thyme, CodeVision, CodeDance, and the Qwen3-VL report were produced under the same evaluation protocol as ToolVision's avg@4 multi-sample decoding at temperature 0.7; if those baselines used greedy decoding or a different sampling scheme, part of the margin may be an artifact of evaluation rather than of the training method.","fun_headline_variants_meta":{"raw":{"variants":["8B model beats 32B by learning when tools help","ToolVision: filter teacher demos by evidence gain for small VLMs","Capability-aligned rewards teach small models when tools matter","Small multimodal model tops 32B rival on high-res benchmarks","ToolVision: 8B beats 32B by aligning tool supervision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000309,"raw_usage":{"total_tokens":1825,"prompt_tokens":1068,"completion_tokens":757,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":668}},"tokens_in":684,"tokens_out":757,"duration_ms":8856,"temperature":1.0,"reasoning_tokens":668,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:20:46.336126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the external baselines (Thyme-7B, CodeVision-8B, CodeDance-7B, and Qwen3-VL-32B-Thinking) under ToolVision's exact avg@4, temperature-0.7 evaluation harness. If, under that protocol, ToolVision-8B no longer beats all of them on the three high-resolution benchmarks, the headline superiority claim collapses, even if the internal ablations still support the method's value.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the code-as-tool paradigm and the observation that unreliable code generation can push a policy toward direct answers, which motivates the MUT reward."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides CodeVision's released evaluation procedure, which the paper adopts, and the seed rotate/flip examples used in SFT."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CodeDance, the main baseline whose dynamic group-based tool incentive the paper contrasts with its model-conditioned necessity measurement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines HRBench 4K/8K, the high-resolution benchmarks where ToolVision reports large gains."}],"review_version":1}