{"id":"635eddbe-69de-46dc-89e9-aa855e7d0a23","arxiv_id":"2608.11292","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A frozen vision-language model more than doubles its accuracy on precise visual-computation questions by writing Python code and refining plain-text skills from its own failed executions.","lead":"This paper introduces Code-with-Image, a benchmark where vision-language models must write and run Python programs to compute exact answers to visual puzzles. A training-free reflection loop lets a frozen model debug its own failed programs and turn the fixes into plain-text skills, roughly doubling accuracy on the new benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'beyond inspection' premise of CwI-Bench is load-bearing and unvalidated: if eye-visible targets leak into the test set, the Code-with-Image gap is overstated and the benchmark does not cleanly isolate computational reasoning. A human inspection baseline would settle this.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the benchmark's 'beyond inspection' premise is central and unvalidated. I agree that this is the most consequential issue because it underpins the interpretation of the headline numbers. If the premise fails, the paper's core conceptual contribution -- that CwI-Bench isolates a regime where language cannot carry the reasoning and only executing a visual algorithm can -- is weakened, even if the relative skill-evolution gains remain internally consistent. The reader's other concerns (missing confidence intervals, validation-gate regressions, absent code/commit) are real but secondary: the reported test gains are large and the per-task negative deltas do not overturn the overall pattern. I also note the paper has independent strengths: constructive ground-truth labels, disjoint COCO splits with a zero-intersection audit, three-seed ablations, and a clear validation-gate procedure. The recommended check is a human inspection baseline, not a re-derivation, because the issue is an empirical premise about human perception rather than an internal inconsistency. Since the reader already rendered a CONDITIONAL verdict, my analysis does not move that verdict; it sharpens the condition that would need to be satisfied for the paper's central claim to be fully supported.","tokens_in":30364,"tokens_out":6197,"duration_ms":68889,"concrete_test":"Draw a stratified sample of 5-10 items per task family (40 items total, across all four difficulty tiers) from the released CwI-Bench test set. Recruit 3-5 human participants who have not seen the paper's results, give them the images and questions with only zoom, pan, brightness, and contrast controls (no rulers, no pixel-readout tools, no calculator, no code), and score their answers with the paper's tolerance-based scorer from Appendix C. Compare human accuracy against the per-task chance baseline implied by the tolerances. If human accuracy is statistically indistinguishable from chance (e.g., overall <=2% and no family above 5%), the 'beyond inspection' premise is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 asserts that every CwI-Bench target lies 'beyond inspection' and that human annotation is 'not only unnecessary but impossible.' The only enforcement is generator-side screening: each target must beat its 'closest visually plausible look-alike' by a clear margin, and any instance an eyeball guess would solve within tolerance is discarded. No human-subject study, perceptual baseline, or quantitative definition of 'visually plausible' is provided. The reported evidence that inspection cannot solve the tasks comes from model agents (DeepEyes, Thyme) and from the low tool-free CoT scores of the same LVLMs being evaluated, not from humans. This matters because the central conceptual claim is that tool-free failure on CwI-Bench reflects missing computation rather than perception or instruction-following. If a meaningful share of test items were actually guessable by eye with zoom/pan/brightness tools, then (i) the low Instruct/Thinking numbers would overstate the gap between language reasoning and Code-with-Image, (ii) the benchmark would not cleanly isolate the program-native regime, and (iii) interpreting skill gains as 'debugging code becomes debugging reasoning' would be less secure. The skill-evolution results might still survive, but the benchmark's foundational claim would be weakened. This is the most load-bearing assumption because nearly every other conclusion -- the ceiling of thinking-with-images, the bare-interpreter gain, and the reflection-loop interpretation -- is built on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CwI-Bench, a 30-task benchmark for a regime the authors call Code-with-Image, in which a model is given only a Python interpreter and must author and execute a program that computes the answer from the pixels. It further proposes Self-Reflection over Executable Reasoning, a training-free loop in which a frozen model reflects on its own failed programs, repairs them in a sandbox against constructive ground truth, and accumulates portable plain-text skills. The reported results are large: on GPT-5.6-luna, test accuracy rises from 12.9% in tool-free Instruct mode to 25.0% in Thinking mode, 43.0% with a bare interpreter, and 66.6% with self-evolved skills; on Qwen3.5-27B the corresponding ladder is 8.6% to 14.4% to 32.6% to 55.9%. The paper also reports transfer of skills across model scales and families, and an ablation in which removing executable reflection reduces mean validation uplift by about 6 of 19.7 percentage points on Qwen3.5-27B.","tokens_in":30678,"tokens_out":8138,"duration_ms":68924,"significance":"If the claims hold, CwI-Bench is a valuable benchmark for program-native visual reasoning, and the self-reflection loop is a practical, weight-update-free mechanism for accumulating and transferring procedural knowledge across frozen LVLMs. The paper has notable strengths: constructive ground-truth generation with disjoint train/val/test splits, a sealed test set, verbatim reflection prompts, per-task worked examples, archived raw trajectories, and multi-executor consistency for both the benchmark evaluation and the ablation. The main risks are the unvalidated 'beyond inspection' premise that underlies the benchmark's interpretation, the absence of uncertainty quantification on 100-item test sets, and some over-strong claims about the safety of delivered skills.","major_comments":[{"comment":"Section 3.3 asserts that every CwI-Bench target lies 'beyond inspection' and that human annotation is 'not only unnecessary but impossible,' with enforcement only through generator-side screens against an undefined 'closest visually plausible look-alike.' The protocol notes in Appendix B, however, state that think-with-image agents 'concentrate on the few tasks where inspection suffices (counting colors, shapes, or stripes),' which directly contradicts the universal beyond-inspection claim. Because the benchmark's central interpretation—that tool-free failure isolates missing computation rather than perception—depends on this premise, please either provide a human-subject or perceptual baseline on a representative sample of test items, or explicitly identify and handle the tasks where inspection can suffice.","section":"§3.3, Appendix B"},{"comment":"All headline results are point estimates on 100-item test sets with no confidence intervals or significance tests. Per-task differences as small as +1 to +5 points and cross-condition comparisons such as the 9B transferred (34.5) versus self-evolved (30.6) overall accuracies may be within sampling error. Please report bootstrap confidence intervals or equivalent uncertainty measures, at least for the aggregate and transfer claims, before asserting that one regime outperforms another on every family.","section":"Tables 2, 3, 5–8"},{"comment":"The delivery gate in §4.3 and Eq. (6) is described as a validation-side safeguard that is 'not a guarantee against test regression,' yet Figure 2 labels the delivery rule as 'skills can never hurt,' and §5.3 emphasizes that no delivered luna skill regresses on test. In the open-model results, delivered skills do regress on specific test tasks (Table 5, film color: 39.0 to 29.0; Table 8, restore count: 18.0 to 14.0). Please remove the 'never hurt' framing, report the full distribution of per-task test deltas, or provide a statistical guarantee at the aggregate level.","section":"§4.3, Fig. 2, Table 5"}],"minor_comments":[{"comment":"The use of '⇐⇒' to equate classical zeroth-order ascent with the reflective update is misleading, since the right-hand side is not an equivalent formalization; I suggest replacing the equivalence symbol with a correspondence arrow.","section":"§4.2, Eq. (4)"},{"comment":"The paper says 'one shared recipe on every task with no per-task tuning' but luna uses a single restart and 8 rounds while the open models use 3 restarts and 12 rounds; this asymmetry should be acknowledged in the protocol comparison.","section":"§5.1, Appendix A.4"},{"comment":"The 'beyond human' checkmark for CwI-Bench repeats the unvalidated premise; consider marking it as asserted rather than validated, or providing the human baseline that would justify it.","section":"Table 1"},{"comment":"The phrase 'nothing to call' is overstated given that the Python interpreter includes rich libraries such as OpenCV and the delivered skill example uses SIFT and RANSAC; clarify that the distinction is the absence of a task-specific tool vocabulary, not the absence of any pre-existing functions.","section":"§3.1, Appendix E.1"},{"comment":"Figure 2 is dense and the small text and multiple arrows are hard to read in print; consider simplifying the diagram or enlarging the font.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically interesting and the empirical gains are large and consistent across executors. The main obstacle is the contradiction between the universal 'beyond inspection' claim and the appendix note that a few tasks are solvable by inspection; I would ask the authors to add a human-subject or perceptual baseline, or to revise the claim to match the evidence. I would also request confidence intervals for the key comparisons. If these points are addressed, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look. It introduces CwI-Bench, a 30-task benchmark where answers require executing a visual algorithm in code rather than inspecting pixels, and a training-free reflection loop that turns a frozen model's failed programs into portable text skills. The empirical story is coherent: bare Python lifts accuracy far above tool-free CoT, and self-evolved skills add roughly 20+ points across four executors, with transfer across scale and family. The ablations with three seeds give the main qualitative claims real support.\n\nWhat is genuinely new: CwI-Bench provides generated tasks with constructive labels and disjoint splits, making procedure learning measurable rather than just tool invocation. The two-mode reflection loop, especially executable reflection that re-runs failures in a sandbox and verifies repairs against ground truth, goes beyond earlier skill-evolution work in a useful way. The plain-text skill format is a nice practical touch.\n\nSoft spots, in proportion. The load-bearing claim that targets are beyond inspection rests on generator-side screens, not on human-subject validation. The paper argues convincingly that tolerances are tight and look-alike samples are discarded, but no human baseline is reported. That is a moderate concern, not a fatal one: even if some items were guessable, the skill gains on tasks like cyclic shift and crop origin would likely survive, but the benchmark would not cleanly isolate computation from perception. A small human study or a perceptual baseline would settle it.\n\nSecond, the main tables report point estimates on 100-item test sets with no confidence intervals, and the validation gate does not prevent test regression (e.g., Qwen film color drops 10 points). That is worth noting but not disqualifying given the consistent pattern. Third, the paper references a Hugging Face release but provides no explicit code or commit hash, which complicates reproducibility checks.\n\nThe citation pattern is honest; the method is clearly positioned against Reflexion, Voyager, XSkill, and Self-Debug. The novelty is incremental in the loop design but the benchmark is a real new artifact.\n\nWho this is for: anyone working on multimodal agents, code-augmented reasoning, or benchmark design for visual computation. It deserves peer review; a strong revision should add the human baseline, confidence intervals, and a code release.","headline":"CwI-Bench is a real contribution and the skill-evolution results are consistent, but the 'beyond inspection' premise needs a human baseline before the benchmark's central claim is airtight.","tokens_in":31233,"tokens_out":1399,"would_cite":true,"duration_ms":15458,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free reflection loop turns a frozen model's failed code into reusable plain-text skills that substantially improve Code-with-Image reasoning.","keywords":["Code-with-Image","visual reasoning","multimodal agents","skill evolution","training-free reflection","CwI-Bench","executable reasoning","program-aided reasoning"],"falsifier":"Show CwI-Bench images to expert humans with unlimited zoom and search but no computation, and ask for answers at the scored tolerances; if a meaningful fraction land within tolerance, the beyond-inspection premise is wrong and the gap between language reasoning and Code-with-Image would be overstated.","tokens_in":30158,"feed_emoji":"🧩","tokens_out":5982,"duration_ms":49092,"temperature":0.7,"pith_summary":"This paper tries to establish that some visual questions are not answerable by looking, no matter how many tools zoom, crop, or brighten, because the target is a precise quantity that only a multi-step computation over pixel arrays can recover. It argues that in this 'Code-with-Image' regime the model's program is the reasoning, and that a frozen model can improve itself by reflecting on why its own programs fail. The proposed loop, Self-Reflection over Executable Reasoning, studies failed trajectories, re-executes them in a sandbox to expose silent implementation faults, and saves verified repairs as short plain-text skills. On CwI-Bench, the paper reports bare Code-with-Image accuracy of 32.6% for Qwen3.5-27B and 43.0% for GPT-5.6-luna, rising to 55.9% and 66.6% with self-evolved skills, with the skills transferring across model scales and families. If right, this is a practical route to accumulating visual-reasoning procedures without weight updates.","feed_headline":"Frozen models learn visual algorithms by debugging their own code","feed_subtitle":"A training-free loop turns failed programs into portable text skills, lifting a frontier model from 43% to 67%.","key_machinery":"The load-bearing object is the Code-with-Image episode, in which the model writes a program, a sandboxed interpreter executes it, and the program's final output is the answer, so the model never carries the intermediate values in language. On top of that, the skill library is a capped set of plain-text skills with three fields—When, Procedure, and Verify—and the learning loop, Self-Reflection over Executable Reasoning, alternates between observational reflection (a single read-only call over failed trajectories) and executable reflection (a sandboxed debugging session that re-runs the failed code, renders intermediate arrays as images, and admits a repair only after it restores the gold answer on at least two failing cases). The library is the only mutable state: the model stays frozen, and edits are gated by a validation-side delivery margin so that a skill is deployed only if it beats bare Code-with-Image by at least three percentage points.","core_discovery":"The central claim is that the bottleneck in hard visual questions is not perception but computation, and that giving a frozen multimodal model a bare Python interpreter converts language-side description into executable reasoning. Once execution is available, the remaining failures are silent implementation faults that hide in intermediate state, and these can be found by re-running the failed computation, rendering intermediates, and testing repairs against constructive ground truth. The paper's evidence is CwI-Bench: 30 task families (transformations, perception, localization, color) whose targets are beyond inspection precision, with disjoint train/validation/test splits, and a training-free skill-evolution loop that edits a small library of When/Procedure/Verify skills. On this benchmark the loop raises bare Code-with-Image accuracy from 32.6% to 55.9% on Qwen3.5-27B and from 43.0% to 66.6% on GPT-5.6-luna, and the delivered text skills transfer: injecting the 27B model's skills into a 9B solver outperforms the 9B's own self-evolved skills (34.5% versus 30.6%), while cross-family transfer lifts gemma-4-26B from 27.7% to 45.0%.","pith_inferences":["If these skills are truly portable plain text, the same loop could be run once on a strong model and the resulting skill libraries shared as a lightweight curriculum, letting weaker models inherit algorithms they could not discover on their own; the paper's co-adaptation results suggest such transfer should be tuned to the recipient's execution ability.","CwI-Bench's 'beyond inspection' premise rests on generator screens rather than human-subject testing, so a direct human study at the scored tolerances would reveal whether any residual successes actually come from inspection rather than computation.","The executable-reflection mechanism—reload failed code, render intermediates, test repairs against gold—could apply to any program-native reasoning domain where intermediate state hides faults, such as numerical data analysis or scientific computing, not just images.","The paper evolves one library per task with bounded training splits; a natural next test is whether skills evolve further when the loop sees a broader distribution or multiple related families within a single library."],"forward_implications":["A frozen model can improve its own visual reasoning without any weight update, as long as it can run and debug programs; this applies even to closed models that expose only inference APIs.","Plain-text skills can be copied across model scales and families: a 27B model's procedures lift a 9B solver above the 9B's own self-evolved skills, and the same library improves a solver from a different model family.","Tool-free chain of thought, even with thinking mode enabled, stays below 30% on CwI-Bench, suggesting that language-only reasoning has a hard ceiling when the answer is a computed quantity rather than a described one.","The learning recipe is not tied to a strong reasoner: the same loop works on open 9B/27B models and on a frontier model, and removing executable reflection removes roughly a quarter to a third of the total gain, concentrated on tasks where faults hide in intermediate state."],"supporting_citations":[{"why":"Supplies the verbal-reinforcement pattern of reflecting on failures, which the loop adapts to multimodal executable trajectories.","marker":"[23]"},{"why":"Shows cross-task experience can be distilled into reusable text, the lineage the skill library generalizes.","marker":"[39]"},{"why":"Contributes the executable-skill library idea in an embodied setting, adapted here to visual computation.","marker":"[30]"},{"why":"Demonstrates execution feedback can repair a single program, which executable reflection lifts to persistent skills.","marker":"[5]"},{"why":"Establishes program-carried reasoning for numerical tasks, the foundation Code-with-Image extends to visual computation.","marker":"[4]"},{"why":"Provides the program-aided reasoning basis for letting a Python interpreter carry computation the model cannot hold in language.","marker":"[6]"},{"why":"The audit finding that most tool-use successes are also solvable without tools, motivating the beyond-inspection regime CwI-Bench isolates.","marker":"[8]"}],"fun_headline_variants":["Self-evolving code-with-image reasoning hits 67% on CwI-Bench","Frozen models teach themselves visual algorithms via code debugging","Code-with-image: a training-free loop turns failures into skills","Visual computation bottleneck: models debug code to reason"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's premise that every target is truly beyond human inspection at the scored precision, enforced by generator screens and examples rather than human-subject validation, is load-bearing; if many items were guessable by eye, the tool-free failure rates would not isolate computation.","fun_headline_variants_meta":{"raw":{"variants":["Self-evolving code-with-image reasoning hits 67% on CwI-Bench","Frozen models teach themselves visual algorithms via code debugging","Code-with-image: a training-free loop turns failures into skills","Visual computation bottleneck: models debug code to reason"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000411,"raw_usage":{"total_tokens":2205,"prompt_tokens":1101,"completion_tokens":1104,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":717,"completion_tokens_details":{"reasoning_tokens":1034}},"tokens_in":717,"tokens_out":1104,"duration_ms":10877,"temperature":1.0,"reasoning_tokens":1034,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:47.407583+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Show CwI-Bench images to expert humans with unlimited zoom and search but no computation, and ask for answers at the scored tolerances; if a meaningful fraction land within tolerance, the beyond-inspection premise is wrong and the gap between language reasoning and Code-with-Image would be overstated.","supporting_citations":[{"cited_title":"Re- flexion: language agents with verbal reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Supplies the verbal-reinforcement pattern of reflecting on failures, which the loop adapts to multimodal executable trajectories."},{"cited_title":"V oyager: An open-ended embodied agent with large language mod- els.Transactions on Machine Learning Research, 2024","cited_arxiv_id":null,"evidence_quote":"Contributes the executable-skill library idea in an embodied setting, adapted here to visual computation."},{"cited_title":"Teaching large language mod- els to self-debug","cited_arxiv_id":null,"evidence_quote":"Demonstrates execution feedback can repair a single program, which executable reflection lifts to persistent skills."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes program-carried reasoning for numerical tasks, the foundation Code-with-Image extends to visual computation."},{"cited_title":"PAL: Program-aided language models","cited_arxiv_id":null,"evidence_quote":"Provides the program-aided reasoning basis for letting a Python interpreter carry computation the model cannot hold in language."},{"cited_title":"Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains","cited_arxiv_id":"2606.02357","evidence_quote":"The audit finding that most tool-use successes are also solvable without tools, motivating the beyond-inspection regime CwI-Bench isolates."}],"review_version":1}