{"id":"ade1a74f-aab3-4579-a4c2-4fba54cf885e","arxiv_id":"2509.09286","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A vision-language model learns to dynamically switch between code-based and visual reasoning for chart questions, improving average accuracy by about one point over fixed strategies.","lead":"This paper trains a vision-language model to choose between solving chart questions by writing code or by looking at the image directly. The model learns when each approach works best, and the authors report small average accuracy gains over fixed reasoning strategies on four chart benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline claim of consistent outperformance is unsupported: Table 1 margins are ~1 point, no error bars or significance tests, and adaptive is numerically worse than a fixed baseline on ChartQA; 3B-scale anomalies suggest an unstable evaluation pipeline.","rationale":"The reader identifies the human annotation framework in Appendix D as the weakest assumption. I agree that unvalidated annotation labels are a real concern for the decision reward, but I see an even more direct threat to the central claim: the empirical comparison itself is statistically ungrounded. The strongest claim is a comparative performance claim, and the reported differences are small, not accompanied by error bars or significance tests, and are internally inconsistent with the phrase 'consistently outperforming' because adaptive loses on ChartQA. The 3B-scale anomalies reinforce that the evaluation pipeline is noisy. Since the central claim is about robust, consistent gains, the load-bearing condition is that the gains are real rather than noise. A concrete multi-seed rerun with significance testing would settle this. This does not change the reader's conditional verdict; it strengthens the same conclusion with a more fundamental reason for conditionality.","tokens_in":18766,"tokens_out":4787,"duration_ms":57368,"concrete_test":"Rerun the full Table 1 protocol (Qwen2.5-VL-7B, identical training data and hyperparameters) with at least 5 independent seeds. For each seed, evaluate on all four benchmarks; report per-benchmark and average accuracy as mean±std. Compute paired bootstrap or per-item sign tests comparing adaptive vs fixed Code-CoT and adaptive vs Standard CoT on each benchmark and on the average. If the adaptive-vs-Code-CoT difference on ChartQA is not significant or remains negative, the phrase 'consistently outperforming' must be weakened to 'comparable on average with benchmark-specific trade-offs.' Also report the variance across seeds for the 3B configuration to confirm the anomalies in Table 4 are not evaluation artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—stated in the abstract and Section 5.2—is that the adaptive model 'consistently outperforms' rigid strategies. The only direct evidence is Table 1, but the margins are tiny: adaptive averages 62.8% versus 61.7% for fixed Code-CoT and 61.2% for Standard CoT. Per-benchmark, adaptive is actually below Standard CoT on ChartQA (86.4 vs 86.6) and equal to it on CharXiv within rounding (44.3 vs 43.8). No error bars, number of seeds, or significance tests are reported anywhere in the paper, so a 1.1-point average gap on samples of size ~1,150–2,400 can easily arise from a single unlucky/lucky run. The 3B results in Table 4 compound this: the base 'Adaptive' model scores 1.0%, 0.7%, 0.3% on ChartX, ChartBench, and ChartQA, and RL-trained Standard CoT collapses from a 31.9% base average to 20.4%. These anomalies are unexplained and indicate the evaluation pipeline is noisy or unstable. Even if the Appendix D human annotations are perfectly reliable, the empirical evidence does not establish consistent, statistically meaningful superiority. Thus the load-bearing weakness is not primarily the annotation subjectivity identified by the reader, but the absence of any statistical grounding for the headline comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Visual Programmability, a per-task property indicating whether a chart–question pair is better solved by generating and executing code (Code-as-Thought) or by direct visual reasoning. The authors implement an adaptive framework on Qwen2.5-VL-7B that first emits a strategy token (<CODE> or <DIRECT>) and then follows the corresponding reasoning path. Training uses GRPO with a four-component reward (Eq. 3), including a data-accuracy reward that compares the DataFrame produced by the generated code with a ground-truth CSV, and a decision reward that encourages matching a human-annotated programmability label. Experiments on ChartX, ChartBench, ChartQA, and CharXiv report an average accuracy of 62.8% for the adaptive model versus 61.7% for fixed Code-CoT and 61.2% for fixed Standard CoT (Table 1). Ablations (Tables 5–6) attribute the gains to the decision and data-accuracy rewards. The paper also reports scaling experiments to 3B and 32B models (Table 4) and qualitative case studies.","tokens_in":19187,"tokens_out":3291,"duration_ms":35411,"significance":"If the empirical claims held up, the paper would make a useful contribution: the idea of learning when to use symbolic code vs. direct perception is intuitive, and the data-accuracy reward is a concrete mechanism for grounding code-based reasoning in chart facts. The authors ship substantial experimental infrastructure, detailed prompts, hyperparameters, and an open-source implementation, which are strengths. The central weakness is that the reported evidence does not support the headline claim of 'consistently outperforming' rigid strategies: the margins in Table 1 are about one point, there are no error bars, seeds, or significance tests, and the adaptive model is not the best on ChartQA. The selective code-usage behavior (e.g., 76% on ChartX vs. 10% on CharXiv, Table 2) is also, by construction, a direct consequence of the decision reward in Eq. (3), not an emergent discovery; the paper frames it as learned adaptive behavior without validating that human programmability labels correlate with the accuracy-optimal strategy. These issues are fixable with additional analysis and qualified claims, so the paper merits major revision rather than rejection.","major_comments":[{"comment":"The abstract and §5.2 claim that the adaptive model 'consistently outperforms' rigid strategies. The evidence in Table 1 is too weak to support this: the average advantage over fixed Code-CoT is 1.1 points (62.8 vs. 61.7), and on ChartQA the adaptive model (86.4) is lower than both fixed Code-CoT (86.7) and Standard CoT (86.6). No standard deviations, number of seeds, confidence intervals, or significance tests are reported anywhere. Given benchmark sizes of roughly 1,100–2,400 samples per dataset, a 1-point gap can easily arise from evaluation noise. Please report variance across at least 3–5 seeds (or a bootstrap CI) and use a paired test for the per-benchmark comparisons; alternatively, temper the claim to 'competitive or slightly better on average.'","section":"§5.2, Table 1"},{"comment":"Table 4 contains unexplained anomalies that suggest an unstable evaluation pipeline. On the 3B base model, the Adaptive strategy scores 1.0% on ChartX, 0.7% on ChartBench, and 0.3% on ChartQA, while Standard CoT scores 48.0%, 39.2%, and 13.8% respectively. A factor-of-~50 drop for merely adding a strategy-selection prefix is not a 'performance gap'; it indicates either a parsing failure (e.g., the base model did not emit the expected special token for most samples) or a broken evaluation harness for that condition. Similarly, RL-trained Standard CoT collapses from a 31.9% average to 20.4% after training, with ChartQA dropping from 13.8% to 41.8% (the reader should confirm this is not a typo: the table shows an increase at ChartQA but large decreases elsewhere). The paper acknowledges the 3B result is 'nuanced' but does not diagnose these pathologies. Please report the cause or remove the","section":"§5.4, Table 4"},{"comment":"The decision reward r_decision explicitly rewards the model for choosing the human-annotated strategy: 'It gives a full reward for a correct answer via the correct strategy, a partial reward for a wrong answer but using the correct strategy, and zero reward for using the wrong strategy.' Therefore the code-usage percentages in Table 2 (76% on ChartX, 10% on CharXiv) are a direct consequence of fitting the annotation labels, not an emergent property of the model learning to reason about programmability. The paper repeatedly frames these percentages as evidence of learned adaptivity. To support that framing, Appendix D must demonstrate that the human labels are reliable and correlate with actual optimal strategy. Currently it provides no inter-annotator agreement statistics, no second-annotator validation, and no evidence that the annotated strategy is the one that maximizes accuracy on th","section":"§4.3.2, Eq. (3); §5.2, Table 2"},{"comment":"The annotation framework is load-bearing for the decision reward, as the paper itself acknowledges in Appendix F ('A key challenge is reducing the reliance on annotated training data for programmability.') The criteria in Appendix D are subjective: annotators are asked whether 'a code-based representation preserves the essential information' and to perform a 'mental compilation,' with no quantitative measures of reliability. Since the central result depends on the decision reward steering the policy, the paper must at minimum report agreement (e.g., Cohen's kappa) on a held-out annotation set and, ideally, validate labels against accuracy-based oracle selection on a small subsample. Without that, the reader cannot distinguish 'the model learned human annotators' preferences' from 'the model learned the accuracy-optimal strategy.'","section":"Appendix D, Appendix F"}],"minor_comments":[{"comment":"The ablation rows use 'w/or decision' and 'w/or data' in the table header; this is a typo for 'w/o' (without). Please correct.","section":"Table 5 caption and header"},{"comment":"The note says 'All values are accuracy (%).' but the table also includes an 'Average' column; consider clarifying that the average is the unweighted mean across the four benchmarks, and state whether the benchmarks have equal weight despite different sample sizes.","section":"§5.2, Table 1 note"},{"comment":"For the 3B base model, the Adaptive row is visually remarkable but never discussed after the table. Either provide an explanation (e.g., failure to emit <CODE>/<DIRECT> tokens) or remove the row; as written, it raises more questions than it answers.","section":"§5.4, Table 4"},{"comment":"The Data Accuracy Reward formula in Algorithm 1 uses a row completeness score that is 1 only if the row counts exactly match (r_row = 1[len(DF_pred) == len(DF_gt)]). This is very brittle; consider using a soft or proportional row-match score, and state whether this strictness affected training.","section":"§4.3.2 and Algorithm 1"},{"comment":"The appendix says the specialist was trained on 'the ChartX validation set' and evaluated on 'a stratified sample from the official test set.' Please clarify the relationship between the training/validation split and the test split to rule out test-set contamination.","section":"Appendix A, Table 8"},{"comment":"Several appendix cross-references are incomplete, e.g., 'Prompt E' is referred to without distinguishing the three different prompts. Consider labeling them Prompt E.1, E.2, E.3.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within scope for the journal, and the code-as-thought idea plus data-accuracy reward are worth publishing if the empirical claims are statistically grounded. However, the headline 'consistently outperforms' claim currently rests on a 1-point average margin with no error bars, and the 3B results in Table 4 suggest a possibly broken evaluation condition. The decision-reward circularity is also more than a presentation issue: if the authors present the adaptive selection as an emergent capability, they need to validate the human annotation signal. I recommend inviting a revision that adds significance testing, explains or removes the 3B anomalies, and either validates or reframes the role of the decision reward."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper has a real idea: a VLM that learns, via RL, when to generate Python code versus direct visual reasoning for chart QA, with a decision reward that stops it from collapsing to one strategy. Second, the headline is not supported: the main comparison is a 1.1-point average gain with no error bars or seeds, and the 3B-scale results look like the evaluation pipeline broke rather than like a model scale effect.\n\nWhat's new: the dual-reward GRPO setup, with a data-accuracy reward that checks extracted DataFrames against ground truth, is a sensible way to ground code outputs. The negative result that fixed CaT excels on ChartX but collapses on CharXiv (18.4%) is a useful data point. The code-usage analysis across benchmarks is nice. They also ship an open implementation and include ablations.\n\nSoft spots, in order of severity.\n\n1. The central claim of consistent outperformance rests on Table 1: adaptive 62.8 vs 61.7/61.2. On ChartQA adaptive is below the fixed Code-CoT baseline and equal to Standard CoT. With per-benchmark sample sizes of ~1,150–2,400, a one-point gap is within run-to-run noise. No number of seeds, no significance tests, no error bars anywhere.\n\n2. The 3B results in Table 4 are a red flag. The base adaptive model scores 1.0%, 0.7%, 0.3% on ChartX/ChartBench/ChartQA—basically zero—and RL-trained Standard CoT averages 20.4 instead of its 31.9 base. The 'longer context' explanation doesn't cover a base model collapsing to 1%. If the evaluation/rollout was broken at 3B, the 7B results need the same scrutiny.\n\n3. The adaptive behavior is partly circular. The decision reward trains the model to match human annotations of programmability, so the code-usage percentages mirror the annotation distribution; they don't demonstrate emergent selection. Appendix D gives no inter-annotator agreement and no evidence labels correlate with the accuracy-optimal strategy. Appendix F does list 'self-supervised policy learning' as future work, which is an honest admission of the limitation.\n\nWho it's for: people working on multimodal RL or chart reasoning will find the framework thought-provoking, and the negative result on fixed strategies is worth knowing. It deserves a serious referee because the method is novel and clearly described, but as written the numbers don't establish the claims. I'd send it to review with the expectation of a major revision: add seeds/error bars, fix or explain the 3B pipeline, and validate the annotation framework.","headline":"Nice framework idea, but the headline result is a one-point average margin with no error bars and a 3B-scale evaluation that looks broken; worth a serious referee for the framing, not for the numbers as reported.","tokens_in":19616,"tokens_out":3247,"would_cite":true,"duration_ms":34942,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For chart questions, a vision-language model that first decides whether to reason in code or directly from the image outperforms any single fixed reasoning strategy across four benchmarks.","keywords":["chart understanding","vision-language models","code-as-thought","visual programmability","adaptive reasoning","reinforcement learning","GRPO","numerical hallucination"],"falsifier":"On a held-out set of chart-question pairs, compare each pair's human programmability label with the actual accuracies of the <CODE> and <DIRECT> paths. If the path the human label designates is not systematically the more accurate one, or if independent annotators disagree on the labels, the decision reward is training the model on labels that do not track performance, and removing the decision reward should not hurt accuracy.","tokens_in":18690,"feed_emoji":"📊","tokens_out":12561,"duration_ms":115754,"temperature":0.7,"pith_summary":"This paper claims that the best way to answer a chart question is not a single reasoning style but a learned choice between two: write code to extract and compute, or reason directly over the image. It introduces Visual Programmability, a property of a chart-question pair that says whether code can faithfully capture what the question needs, and trains a vision-language model to emit a strategy token before answering. The model is trained with reinforcement learning using rewards for answer correctness, for choosing the right strategy, and for extracting the chart's data faithfully. Across four benchmarks the adaptive model averages 62.8% accuracy, ahead of fixed Code-CoT at 61.7% and Standard CoT at 61.2%, and it adjusts code usage from above 60% on structured charts to roughly 10% on complex scientific charts. This matters because it moves chart understanding from building a better single reasoner to teaching a model when each way of reasoning applies.","feed_headline":"62.8%: chart AI that picks code or vision beats fixed styles","feed_subtitle":"A model that learns when to write code and when to look directly lifts accuracy across four chart benchmarks.","key_machinery":"The central object is Visual Programmability: a learnable, task-dependent property gauging whether a chart-question pair can be faithfully reasoned about in code, ranging from high (clean bar/line charts with explicit values) to low (scientific plots whose meaning lives in contours and distributions). The framework factorizes the policy as P(y|I,Q)=P(s|I,Q)·P(y|I,Q,s), where s is a strategy token in {<CODE>,<DIRECT>}, and trains the selection policy with Group Relative Policy Optimization (GRPO) under the weighted reward R = w_acc·r_acc + w_decision·r_decision + w_data·r_data + w_format·r_format. The decision reward gives positive signal for choosing the human-annotated correct strategy even","core_discovery":"The central claim: Code-as-Thought is powerful but not universal; a fixed code-only specialist reaches 71.6% on structured ChartX charts yet collapses to 18.4% on in-the-wild CharXiv charts. Visual Programmability is a task-dependent property of a chart-question pair: whether the chart's essential information can be faithfully translated into code. The adaptive model emits a strategy token (<CODE> or <DIRECT>) and then reasons along that path, trained with group-relative policy optimization (GRPO) under a reward combining answer accuracy, a decision reward for strategy selection, a data-accuracy reward comparing generated DataFrames to ground-truth tables, and a format reward. The model aver","pith_inferences":["The binary code/direct split is likely a simplification: hybrid reasoning that uses code for data extraction and direct visual reasoning for qualitative pattern interpretation may beat either pure pathway on mid-programmability charts, a direction the paper lists as future work.","The annotation bottleneck could be removed: a self-supervised variant that derives programmability labels from whether the code path answered correctly on each instance might learn the same decision boundary without human annotation.","The strategy-token plus dual-reward recipe likely transfers to other visual-analytic domains, such as diagrams, maps, and document figures, where symbolic extraction and holistic reading trade off; the paper does not test this extension."],"forward_implications":["Fixed code-only chart systems will keep failing on complex real-world charts no matter how well they code, because the gap is strategic, not skill-based: the model must know when not to code.","The decision reward is what prevents mode collapse: without it the model collapses to 0% or 100% code usage and loses performance on low-programmability charts; with it code usage tracks benchmark programmability.","The data-accuracy reward reduces numerical hallucination: on ChartX, high-fidelity data extraction (>0.8 reward score) correlates with 85.6% answer accuracy versus 48.4% for low-fidelity (<0.6), and the model unlearns guessing on low-fidelity tasks.","The adaptive framework scales to larger models: at 32B it posts the highest average accuracy (61.0%) and the best scores on ChartX and CharXiv; at 3B the adaptive prompt is too cognitively demanding and fixed Code-CoT wins.","The adaptive advantage comes where fixed strategies are weak: 44.3% versus 41.9% on CharXiv against fixed code, and 65.6% versus 61.5% on ChartX against standard CoT."],"fun_headline_variants":["Chart AI that learns when to code or look—adaptive wins","Fixed code reasoning flops on messy charts; adaptive picks path","Visual Programmability: model chooses code or vision per chart","Adaptive chart reasoning knows when to code, when to see","Teaching VLMs to pick reasoning path—code or direct—boosts accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The framework assumes that expert annotators' judgments about whether a chart's information survives translation into code are a reliable guide to which reasoning strategy actually maximizes accuracy; if those human labels are subjective or inconsistent, the decision reward trains the model toward a flawed strategy boundary.","fun_headline_variants_meta":{"raw":{"variants":["Chart AI that learns when to code or look—adaptive wins","Fixed code reasoning flops on messy charts; adaptive picks path","Visual Programmability: model chooses code or vision per chart","Adaptive chart reasoning knows when to code, when to see","Teaching VLMs to pick reasoning path—code or direct—boosts accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001114,"raw_usage":{"total_tokens":4518,"prompt_tokens":825,"completion_tokens":3693,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":3605}},"tokens_in":569,"tokens_out":3693,"duration_ms":30629,"temperature":1.0,"reasoning_tokens":3605,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:20:52.329841+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of chart-question pairs, compare each pair's human programmability label with the actual accuracies of the <CODE> and <DIRECT> paths. If the path the human label designates is not systematically the more accurate one, or if independent annotators disagree on the labels, the decision reward is training the model on labels that do not track performance, and removing the decision reward should not hurt accuracy.","supporting_citations":[],"review_version":1}