{"id":"c9c43923-9d4a-4425-ad1d-6799c965a832","arxiv_id":"2608.09682","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Replacing returned images with a fixed text placeholder in tool-augmented visual reasoning preserves benchmark accuracy, suggesting the tool-call text, not the returned pixels, carries the gain.","lead":"Tool-augmented vision-language models normally pass cropped images back to the model after each tool call. This paper shows that replacing those returned images with a fixed text placeholder during training and inference preserves accuracy on a six-benchmark suite while cutting latency by 29 to 46 percent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing no-scaffold control undercuts the carrier claim: Table 7's skeleton-only baseline (71.63) nearly matches full scaffold (73.13) at 9.5K, and no 65K direct-answer control is reported, so TextCall's parity with thinking-with-images could stem from corpus SFT alone.","rationale":"The reader's weakest assumption concerned external validity: whether the benchmark suite contained all necessary evidence in the original image, so the returned crop was redundant. My concern is different and more internal: the paper never tests whether the scaffold itself is necessary, even on the evaluated suite. The skeleton-only baseline in Table 7 is the closest control, and it nearly matches the full scaffold at 9.5K, suggesting that the semantic content of the scaffold (reasoning text and coordinates) contributes little. The coordinate-precision ablation in Appendix J reinforces this: shuffled coordinates are equivalent to correct ones, so the specific 'where to look' content is not used. If a 65K no-scaffold baseline also matches TextCall, then the Tool-Call Scaffold Hypothesis collapses: the apparent gain would come from SFT on a VQA trajectory corpus, not from the structured text. This is a concrete, addressable omission, and the paper already anticipates such controls in its skeleton-only discussion but does not run them at the main scale. The reader's conditional verdict is therefore appropriate; I recommend keeping it unchanged, but the missing control should be a required condition for acceptance.","tokens_in":26929,"tokens_out":11409,"duration_ms":101788,"concrete_test":"Train a 65K full-FT control on the same DeepEyes V2 trajectories with all assistant scaffold text removed (only the final answer as target; no <think>, no <code>), evaluated under the same protocol as TextCall. If this no-scaffold baseline's core 6-bench mean is within ~1pp of TextCall 65K (71.23), the scaffold is not load-bearing at the scale where the main claim is made. Additionally, run the skeleton-only ablation at 65K scale; if it also matches, the central hypothesis is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the structured tool-call scaffold is the load-bearing signal, not the returned pixels. The primary evidence is that TextCall (no pixel return) matches or exceeds thinking-with-images. However, this comparison only manipulates the pixel return; it never removes the scaffold. If a model trained on the same trajectories with the scaffold content erased performs equally, then the scaffold itself is not load-bearing. Appendix C (Table 7) provides a relevant control at 9.5K LoRA: 'Skeleton only' (empty reasoning + pass, preserving only the multi-turn wrapper) reaches 71.63 mean on V*/HR-4K/CV-2D, within 1.5pp of the full scaffold's 73.13. The paper dismisses this as falling back to base-model direct-answering capability, but that dismissal concedes that scaffold content contributes little at this scale. At the 65K full-FT scale where the headline TextCall-vs-TWI parity is established, no such no-scaffold control is reported. The observed parity could therefore arise not from the scaffold's semantic content but from SFT on the DeepEyes corpus itself (which may improve general VQA skill) combined with the model's continued access to the original input image. Without a direct-answer or CoT-only baseline trained on the same data, the inference that the scaffold carries the gain is underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Tool-Call Scaffold Hypothesis: in current thinking-with-images distributions, the structured text emitted before a tool returns (tool name, coordinates, target description, intent) is the load-bearing signal, while the returned image is a redundant carrier. To test this, the authors introduce TextCall, a training-aligned carrier swap that retains the tool-call scaffold but replaces the returned image with the fixed placeholder ‘[Image output skipped]’ during both training and inference. Using Qwen2.5-VL-7B on the DeepEyes V2 corpus, they report that TextCall matches or exceeds full thinking-with-images under 9.5K LoRA SFT, 65K full fine-tuning, and a matched GRPO run, with lower latency and no image-returning tool executions. They additionally present a factorial audit of 1,000 DeepEyes V2 trajectories showing that a scaffold-only input to a Gemini judge recovers image-only accuracy, a scaffold decomposition showing that reasoning text and code both matter on a 9.5K LoRA setup, and a coordinate-precision ablation suggesting that code format rather than numeric coordinate accuracy carries the effect. The authors explicitly scope their claims to one base model, one cold-start corpus, one seed, and one perception-heavy benchmark suite.","tokens_in":27239,"tokens_out":9139,"duration_ms":84761,"significance":"If the central claim is correct, the paper provides a valuable reframing of tool-augmented VLM reasoning: it offers a clean training-time intervention that prior inference-time ablations lacked, and it has practical implications for latency and architecture by eliminating post-call image processing. The carrier-swap design is a genuine methodological step forward, and the paper is unusually transparent, reporting compute, assets, licenses, RL dynamics, and a clear limitation statement. The central comparison is an external-benchmark intervention rather than a fitted-parameter prediction, so circularity is not a concern. However, the stronger claim that the scaffold itself is the load-bearing signal is underdetermined by the current controls, and the headline parity result lacks uncertainty quantification.","major_comments":[{"comment":"The skeleton-only control reported in Appendix C (Table 7) materially weakens the claim that the scaffold content is load-bearing. At the same 9.5K LoRA scale, the skeleton-only condition (empty reasoning, pass code) reaches a mean of 71.63 on V*/HR-4K/CV-2D against 73.13 for the full scaffold, a difference of only 1.5pp. The paper dismisses this variant as ‘falling back to the base model’s direct-answering capability,’ but the base model in Agent Mode averages 62.70 on the same three benchmarks, so the skeleton-only result is 9pp above that baseline and is close to the direct-MC base mean of 75.78. This suggests the multi-turn SFT on the corpus itself, rather than the semantic content of the scaffold, may be responsible for much of the gain. More importantly, no analogous no-scaffold control is reported at the 65K full-FT scale where the headline TextCall-vs-TWI parity is established. Without a 65K direct-answer or skeleton-only control trained on the same data, the observed parity could arise from corpus SFT plus the model’s continued access to the original question image, not from the scaffold. Please add such a control and reinterpret the 9.5K decomposition in light of Table 7.","section":"Appendix C, Table 7; Section 4.2"},{"comment":"The central non-inferiority claim rests on point estimates without uncertainty quantification. V*Bench has only 191 samples; at the 65K scale the V*Bench gap is -0.52pp and the core six-benchmark mean gap is +1.39pp, while at 9.5K the V*Bench gap is +5.23pp. No confidence intervals, bootstrap intervals, or paired tests are provided, so it is impossible to assess which differences are within sampling noise. Since the paper explicitly claims that TextCall ‘matches or exceeds’ thinking-with-images, it should report paired bootstrap confidence intervals (as already done for the audit in Section 4.1) for the TextCall-versus-TWI comparisons on the core suite, at least for the 65K comparison.","section":"Table 1, Section 3.3"},{"comment":"The component decomposition is non-monotonic in a way that complicates the conclusion that both reasoning text and code contribute additively. The means are: full scaffold 73.13, reasoning-only 66.32, code-only 61.83, and skeleton-only 71.63. Removing both components hurts far less than removing either one, which suggests that the full-scaffold and skeleton-only models may be using different strategies (scaffold-guided reasoning versus direct answering) rather than exhibiting additive component contributions. The proposed explanation that code removal causes ‘hallucinated spatial attention’ is plausible, but the near-parity of skeleton-only indicates a strategy-switch alternative. Please provide per-sample agreement and error analysis between the full-scaffold and skeleton-only models, and report at least one no-content condition at the 65K scale, before concluding that the scaffold components are individually load-bearing.","section":"Section 4.2, Table 3"}],"minor_comments":[{"comment":"The RL comparison reports only tool-call rates for the thinking-with-images collapse; the text states that accuracy ‘remains reasonable’ without numbers. Since the claim is that the collapse is a failure mode, please report the benchmark accuracies of the collapsed thinking-with-images RL checkpoint alongside TextCall in Table 10, or explicitly state that the RL contrast is about tool-use behavior rather than accuracy.","section":"Appendix I, Table 10"},{"comment":"The title and abstract phrase ‘Not With Pixels’ could be misread as claiming that no pixels at all are needed; in fact TextCall still processes the original question image and only removes the post-call crop/zoom returns. The paper does scope the claim correctly in the body, but the title is broader than the actual intervention.","section":"Abstract, Section 1"},{"comment":"The 5pp non-inferiority margin for the scaffold-only versus image-only audit is stated without justification; please explain why 5pp is the appropriate equivalence margin for this comparison.","section":"Section 4.1"},{"comment":"The coordinate-precision ablation uses Qwen3-VL-8B and a Direct MC think-mode evaluation protocol that differs from the main experimental setup, while the 9.5K replication uses a different base model; the main text should more prominently flag these as supporting evidence rather than as part of the main carrier-swap experiment.","section":"Appendix J"},{"comment":"The latency-only no-execution row in Table 4 is explicitly not used for canonical accuracy comparisons, but its placement in the main table may confuse readers; consider moving it to an appendix or marking it more clearly as a reference-only row.","section":"Table 4"},{"comment":"The data-size label ‘77K’ for the TextCall SFT+RL row is unclear; it appears to denote the SFT corpus plus RL rollouts, but the paper should state the exact composition of that row relative to the 65K SFT data.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is methodologically interesting and the authors are transparent about limitations, but the missing no-scaffold control at the scale where the headline claim is made is a substantive gap. I do not view this as a rejection: the non-necessity of the returned pixel carrier may survive the additional control, and the paper already contains many of the components needed to address the concern. However, the stronger Tool-Call Scaffold Hypothesis requires the control to be run. The lack of confidence intervals on the main parity table is also important for a non-inferiority claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely clean matched intervention — keep the tool-call text, drop the returned image — and the fact that TextCall matches thinking-with-images across SFT scales and one RL run is worth taking seriously. But the paper's headline inference, that the scaffold rather than the pixels is the load-bearing signal, is underdetermined, because there is no no-scaffold control at the 65K scale where the main parity is established. The one relevant control they do report, Appendix C's skeleton-only baseline at 9.5K, lands at 71.63 vs. 73.13 for the full scaffold, awkwardly close. They dismiss it as the base model falling back to direct answering, but that dismissal concedes what a 65K direct-answer control would test: whether corpus SFT alone, without any scaffold, gets you the same TextCall parity.\n\nWhat's genuinely new: the training-time carrier swap is a real improvement over inference-only image removal, and applying it across SFT and RL is a sensible design. The factorial data audit is transparent, with paired bootstrap uncertainty on the non-inferiority claim. The coordinate-precision equivalence — correct vs. shuffled coordinates being statistically equivalent — is a useful, non-obvious result: the code body acts as a structural cue, not a numerically accurate spatial reference. The appendices are honest about the RL reward caveat, the single-model scope, and the unreleased Long-CoT corpus.\n\nSoft spots, in proportion: the missing no-scaffold control is load-bearing. The skeleton-only result at 9.5K is close enough to the full scaffold that the decomposition story (full > reasoning-only > code-only) is muddier than they present; removing content entirely beats removing either component alone. That pattern needs explanation, not dismissal. Everything else is addressable: one seed, one model, one corpus, point estimates without confidence intervals in Table 1. The RL stability contrast is real but conditional on a reward that does not incentivize tool use; the authors say so themselves.\n\nWho this is for: anyone working on tool-augmented VLMs, or on whether CoT scaffolds versus image tokens carry reasoning. It's a useful paper even with the gap. My recommendation: send it to review, but the reviewers should insist on a 65K no-scaffold/direct-answer control, and a fuller discussion of the skeleton-only baseline, before the scaffold hypothesis is stated as established.","headline":"Clean matched ablation with a real missing control: without a 65K no-scaffold baseline, the claim that the scaffold carries the gain is underdetermined.","tokens_in":27757,"tokens_out":2490,"would_cite":true,"duration_ms":24782,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tool-augmented vision models may not need the returned image: swapping it for a text placeholder matches or beats pixel feedback on current benchmarks, pointing to the tool-call text itself as the carrier of reasoning gains.","keywords":["tool-augmented vision-language models","thinking-with-images","carrier swap","textual scaffold","tool-call reasoning","vision-language reinforcement learning","crop and zoom tools","pixel redundancy"],"falsifier":"Construct a benchmark task where the answer depends on visual detail that exists only in the region returned by the crop/zoom tool, and that cannot be anticipated from the scaffold text (e.g., reading a small label or distinguishing two near-identical objects only after cropping). If a TextCall-trained model, matched on scale and training data, significantly underperforms a thinking-with-images model on such a task, the Tool-Call Scaffold Hypothesis is refuted; if TextCall matches, the pixel return is redundant even there. The paper itself identifies visual-simulation tasks and fine-grained comparison across intermediate states as candidate settings for this test.","tokens_in":26754,"feed_emoji":"🖼️","tokens_out":3505,"duration_ms":25787,"temperature":0.7,"pith_summary":"The paper targets the reasoning gains attributed to 'thinking-with-images' in tool-augmented vision-language models. It argues, and supports with a controlled carrier-swap intervention, that the active signal is the structured text the model emits before the tool returns: the tool name, crop coordinates, target description, and call intent. Replacing the returned image with the literal placeholder [Image output skipped] during both training and inference preserves accuracy across SFT and RL scales, reduces latency by 29-46%, and removes tool-execution API calls. The claim is scoped to current thinking-with-images benchmarks; the paper itself notes that constructing tasks where pixels are genuinely load-bearing remains open.","feed_headline":"Tool-call text, not returned pixels, carries visual reasoning gains","feed_subtitle":"Swapping returned images for a text placeholder matches or beats pixel feedback on current benchmarks at 29-46% lower latency.","key_machinery":"The central mechanism is the training-aligned carrier swap: the tool-call scaffold (the emitted <think> reasoning block plus executable crop/zoom code with coordinates) is preserved exactly, while the post-call visual return is replaced by the fixed text sentinel [Image output skipped]. This isolates the pre-return textual scaffold from the returned pixels, avoiding the train-test mismatch confound of prior inference-time deletions. The scaffold is decomposed into reasoning text (why a region matters, what to look for) and spatial code (the concrete .crop() coordinates); a coordinate-precision ablation shows that the format of the code matters substantially while the numeric correctness of the coordinates does not detectably matter in the post-training regime.","core_discovery":"The paper proposes the Tool-Call Scaffold Hypothesis: in current thinking-with-images training distributions, the load-bearing signal is the structured text emitted at tool-call time (tool name, coordinates, target description, intent), while the returned image is a redundant carrier. To test this, it introduces TextCall, which keeps the full tool-call scaffold but replaces every returned image with the fixed text sentinel [Image output skipped] during both training and inference. Across LoRA SFT, full fine-tuning, and GRPO reinforcement learning on a six-benchmark core suite, TextCall matches or exceeds full thinking-with-images accuracy (e.g., 76.96% vs 71.73% on V*Bench at 9.5K LoRA; +1.39pp mean on the 6-bench mean at 65K full FT). Under matched RL conditions, TextCall maintains active tool use while the thinking-with-images run collapses to direct answering. A factorial audit on 1,000 matched DeepEyes V2 trajectories shows that the scaffold alone recovers accuracy equivalent to the image-only condition (73.10% vs 73.50%, non-inferior at a 5pp margin), and a component decomposition shows that both reasoning text and spatial code contribute, with the dominant component varying by benchmark.","pith_inferences":["The scaffold's role may generalize beyond crop/zoom code tools: the paper's own Visual Jigsaw probe with text-to-image edits shows the pattern holds across a different tool family, but only at n=50, so this is suggestive rather than conclusive.","The coordinate-precision equivalence result suggests that in the post-training regime, the code body serves as a structured procedural cue—forcing region selection—rather than as an accurate spatial reference; this implies that scaffold quality is about the reasoning text and the act of specifying a region, not the coordinate values.","If pixels are redundant on current benchmarks, the bottleneck for tool-augmented visual reasoning shifts to the model's ability to verbalize where to look; progress may come from improving spatial verbalization and scaffold generation rather than from richer image-return mechanisms.","The paper's boundary conditions (representational bottleneck and visual prior knowledge gap) predict that tasks requiring visual simulation or fine-grained comparison across intermediate states may be where pixel returns become genuinely load-bearing—a testable prediction the authors leave open."],"forward_implications":["If the hypothesis holds, current tool-augmented VLM pipelines can drop returned-image tool execution entirely for perception-heavy tasks, cutting latency by roughly 30-50% and eliminating tool-execution API overhead while preserving accuracy.","The gain attributed to visual tool use is, on current benchmarks, largely carried by the structured text—so models can be trained to emit better scaffolds (more informative reasoning and code) rather than to consume pixel returns.","Under matched RL settings, the scaffold-only variant preserves active tool use where the pixel-return variant collapses to direct answering, suggesting the pixel carrier can destabilize tool-use behavior during policy optimization.","Future thinking-with-images systems should report a scaffold-only control so that the load-bearing carrier is measured rather than assumed.","The paper calls for new benchmark tasks where the returned pixels supply information the scaffold cannot verbalize or replace; until such tasks are constructed, TextCall is presented as the lower-cost default."],"supporting_citations":[{"why":"DeepEyes V2 provides the 65K cold-start training corpus and RL setting that the TextCall carrier-swap experiments build on and compare against.","marker":"[21]"},{"why":"Its image-to-noise ablation (−0.52pp) is cited as prior inference-time evidence that returned pixels carry little causal weight, motivating the training-time swap.","marker":"[31]"},{"why":"The blind-test result that removing images raises MathVista accuracy by 3.5pp is cited as prior evidence against the pixel-return premise.","marker":"[16]"},{"why":"Its decomposition claiming over 70% of post-RL gains arise from intrinsic reasoning is cited as evidence that the pixel carrier is not the main driver.","marker":"[35]"},{"why":"Its finding that only 57% of returned crops contain the target object is cited as faithfulness evidence that returned images are often off-target, supporting scaffold-based reasoning.","marker":"[22]"},{"why":"Zooming Without Zooming is cited as an adjacent line that collapses multi-turn zoom into a single pass, indicating pixel returns may be distilled away.","marker":"[75]"},{"why":"Text-only self-calling CoT outperforming pixel-interleaved iMCoT is cited as evidence that the textual trajectory, not pixels, carries the reasoning gain.","marker":"[88]"}],"fun_headline_variants":["Tool-call text, not pixels, is the real signal for visual reasoning","TextCall: Skipping image returns matches or beats seeing them","Visual reasoning gains come from tool-call text, not returned images","Pixels are redundant: tool-call text scaffolds carry the reasoning","Why pixel returns don't drive visual reasoning: the text scaffold does"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluated benchmarks and training corpus are such that all discriminative visual evidence needed to answer a question is present in the original input image and can be reflected in the pre-return tool-call scaffold, so that removing the post-call image loses no information needed for a correct answer.","fun_headline_variants_meta":{"raw":{"variants":["Tool-call text, not pixels, is the real signal for visual reasoning","TextCall: Skipping image returns matches or beats seeing them","Visual reasoning gains come from tool-call text, not returned images","Pixels are redundant: tool-call text scaffolds carry the reasoning","Why pixel returns don't drive visual reasoning: the text scaffold does"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1554,"prompt_tokens":1143,"completion_tokens":411,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":759,"completion_tokens_details":{"reasoning_tokens":319}},"tokens_in":759,"tokens_out":411,"duration_ms":4667,"temperature":1.0,"reasoning_tokens":319,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:48:20.823431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a benchmark task where the answer depends on visual detail that exists only in the region returned by the crop/zoom tool, and that cannot be anticipated from the scaffold text (e.g., reading a small label or distinguishing two near-identical objects only after cropping). If a TextCall-trained model, matched on scale and training data, significantly underperforms a thinking-with-images model on such a task, the Tool-Call Scaffold Hypothesis is refuted; if TextCall matches, the pixel return is redundant even there. The paper itself identifies visual-simulation tasks and fine-grained comparison across intermediate states as candidate settings for this test.","supporting_citations":[],"review_version":1}