{"id":"27873aaa-b7ce-4681-ace3-caa8fd4da1f0","arxiv_id":"2508.11975","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Chart-CoCa makes a vision language model improve its own chart question answering by generating synthetic charts via code, extracting exact answers from that code, and training itself to synthesize a final answer from its own candidate responses.","lead":"An AI that reads charts can train itself to get better at answering questions about charts by drawing its own new charts with code and then practicing on them. The method needs no human-annotated answers and no stronger teacher model, and it reports gains of up to 15.5 accuracy points on a standard benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reasoning gains may reflect candidate selection rather than learned reasoning transfer, since training data are descriptive-only and direct FT degrades reasoning.","rationale":"The strongest support for the central claim is the consistent improvement across four different VLMs, the code-execution error analysis, and the ablations isolating code mediation; these are genuine empirical assets. The most load-bearing problem is not the use of GPT-4o as judge, which is evaluation-only and cited as reliable, nor the absence of error bars, which would weaken but not refute the results. It is the unexplained mechanism behind the reasoning gain. Table 2 is internally diagnostic: direct FT on the same synthetic triplets drops reasoning by 1.7 points, while Chart-CoCa gains 8.0 points over direct FT. Since the synthetic QA data are explicitly descriptive-only (Section 3.1), the answer model receives no supervised signal for reasoning questions. The only additional ingredient at inference is the candidate set, so the reasoning result depends entirely on whether descriptive candidate-conditioned training transfers to open-ended reasoning. The paper provides one encouraging case study but no aggregate measurement on all-wrong candidate sets. This is testable without retraining by subsetting queries on Pass@K failures and comparing MANS accuracy. If accuracy on that subset is near zero, the improvement over direct FT is a test-time selection effect, not self-improving reasoning; if accuracy is well above baseline, the transfer claim holds. The reader's weakest assumption identifies essentially the same concern, so this stress-test agrees. I do not see grounds to reject the paper, but the reasoning claim requires this verification before it can be accepted as stated. The verdict therefore remains CONDITIONAL.","tokens_in":16245,"tokens_out":7197,"duration_ms":86560,"concrete_test":"On the CharXiv validation reasoning split, identify the subset of queries where none of the five generated candidates matches the ground-truth answer (Pass@K failures; this is a large fraction given the reported Pass@K curves). Run the trained MANS on this subset and report its accuracy. If accuracy is near zero or not above the 21.90 direct-FT baseline, the candidate-conditioned model is selecting or paraphrasing from candidate coverage rather than synthesizing new reasoning, and the central transfer claim fails. If accuracy is materially above that baseline, the transfer and synthesis claim is supported. Optionally, have two human annotators rescore a stratified 100-sample of final answers using the same pairwise protocol to rule out GPT-4o judge bias toward more verbose answers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most striking result is the reasoning-accuracy gain on InternVL2-8B from 23.60 to 31.60 (Table 2). Because direct fine-tuning on the same synthetic data reduces reasoning accuracy to 21.90 (Section 4.2), this gain must come from the candidate-conditioned answering step. However, the synthetic training triplets are generated exclusively from descriptive CharXiv questions (Section 3.1), so MANS never trains on a reasoning question-answer pair. The load-bearing assumption is therefore that descriptive candidate-conditioned training transfers into a general ability to inspect a real chart, read several candidate answers, and synthesize a correct open-ended reasoning answer at test time. The paper does not verify this. It never reports accuracy on reasoning queries where all K candidates are wrong, and it never measures how often the final answer is simply a restatement of a correct candidate. If MANS mostly selects or paraphrases a correct candidate, the reasoning improvement is bounded by test-time candidate coverage rather than by learned reasoning synthesis, and the 'candidate-conditioned answering' contribution is closer to reward-model selection than to self-improving reasoning. The single case study in Table 5 is consistent with selecting a minority correct candidate and elaborating it, so it does not resolve the issue. The full self-improvement narrative depends on this transfer, so the reasoning claim is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Chart-CoCa, a self-improvement framework for chart understanding in vision language models (VLMs). The method has two main components: (1) a code-driven chart synthesis pipeline in which the VLM describes a real chart, generates Python/Matplotlib code, executes it to produce a new chart, and extracts chart information from plotting objects to obtain accurate chart-question-answer triplets; and (2) a candidate-conditioned answering strategy in which, at training time, the VLM is fine-tuned to produce the correct answer given the chart, question, and multiple sampled candidate answers, and at test time the fine-tuned model converts a fresh set of candidates into a final answer. Experiments on the CharXiv benchmark report consistent improvements over the initial VLM and several baselines, including up to a 15.50-point absolute gain on descriptive tasks and an 8.00-point gain on reasoning tasks for InternVL2-8B. The paper also includes ablations showing the importance of the description and code-mediation components, candidate-count scaling experiments, and an error analysis of code execution.","tokens_in":16425,"tokens_out":2594,"duration_ms":28206,"significance":"If the results hold, the paper offers a practical recipe for bootstrapping chart understanding without manually labeled training answers or an external teacher model for data generation. The code-driven synthesis idea is elegant in that labels are derived from executed code rather than from model predictions, which reduces label noise. The candidate-conditioned answering module is also interesting and shows consistent gains across four models in the ablation-oriented comparison. The paper provides machine-executable pipeline details and makes code public, which supports reproducibility. However, the paper's central claim—fully self-improving without human-labeled data or external models—is qualified by its reliance on CharXiv's human-curated questions as seeds and on a GPT-4o judge for evaluation, and the reasoning-task gain is not yet convincingly separated from candidate selection rather than learned reasoning synthesis.","major_comments":[{"comment":"The claim of a \"fully self-improving paradigm without either human-labeled data or external models\" is overstated. Section 3.1 states that the QA generation uses \"the descriptive questions in CharXiv as seeds,\" and these questions are human-curated (Appendix A). Furthermore, Section 4.1 states that evaluation uses GPT-4o as an automated judge. While no human-annotated answers are used for training and no external model is used to produce training labels, the method does depend on a human-annotated question bank and an external model for evaluation. The authors should either temper the claim or clearly scope it to \"no external model or human labels in the training-data generation loop\" and explicitly acknowledge the dependence on CharXiv and GPT-4o in the abstract and conclusion.","section":"Abstract / Section 3.1"},{"comment":"The reasoning-task improvement is not yet established as a learned reasoning ability. Synthetic training triplets are generated exclusively from descriptive CharXiv questions (Table 1), so the answer model MANS is never trained on a reasoning question-answer pair. Yet the main reasoning gain in Table 2 (from 23.60 to 31.60) is attributed to candidate-conditioned answering, while direct fine-tuning on the same synthetic data degrades reasoning to 21.90. The paper does not report whether the reasoning gains persist when all K test-time candidates are incorrect, nor does it measure how often the final answer is a verbatim selection or light paraphrase of a correct candidate. The case study in Table 5 shows the final answer elaborating the correct candidate (Candidate 4), which is consistent with selection-plus-paraphrase. To support the claim that the model learns to synthesize a correct answer from candidates, the authors should add an analysis that separates candidate selection from genuine answer synthesis, e.g., by reporting accuracy conditioned on the presence/absence of a correct candidate and by computing the fraction of final answers that contain substantial new content beyond the selected candidate.","section":"Section 4.2 / Section 3.1"},{"comment":"The absence of error bars, repeated seeds, or any statistical significance measure is a concern given the small margins between some baselines. For example, CoT (26.80) and Majority Voting (26.70) differ by only 0.10 points, and Self-Verification (28.30) is only 0.20 points above Majority Voting on reasoning. The claim that Chart-CoCa \"surpasses all baselines by significant margins\" is based on single runs. The authors should report variance across at least three seeds or provide a paired significance test, especially for the reasoning differences that are central to the paper's contribution.","section":"Section 4.2 / Table 2"},{"comment":"The Pass@K curves in Figure 4 provide a useful ceiling analysis, but the paper does not explain how Pass@K is computed for open-ended answers. Specifically, it is unclear whether a candidate is considered correct based on exact string match, GPT-4o judgment, or another criterion. Since the final evaluation uses GPT-4o pairwise comparison, the Pass@K values should be computed under the same protocol to be comparable to the Chart-CoCa and majority-voting curves. The authors should specify the correctness criterion used for Pass@K.","section":"Section 4.3 / Figure 4"}],"minor_comments":[{"comment":"The header \"Reasoning Quesitons\" contains a typo; it should read \"Reasoning Questions.\"","section":"Table 2"},{"comment":"The text contains a duplicated phrase: \"Pass@K also rises significantly, improving from from 23.6% (K=1) to 55.8%.\" Remove the second \"from.\"","section":"Section 4.3"},{"comment":"The caption states \"where the values in parentheses are the comparison with the initial model,\" but the table does not show any values in parentheses. Either add the relative gains or revise the caption.","section":"Table 3 caption"},{"comment":"Table 5's final answer text is described as showing a reasoning path, but the path consists of a single clause referencing \"the purple line in subplot e.\" The caption and surrounding text should more carefully characterize this as a minimal justification rather than an extensive chain of reasoning.","section":"Section 4.6 / Table 5"},{"comment":"The maximum number of code generation/execution attempts is informally described as \"we simply set a maximum number of attempts\" and later in Section 4.7 stated as 5 attempts. This value should be stated once, early in Section 3.1, rather than only appearing in the error analysis.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within scope for CIKM and addresses an important problem. The code-mediated synthesis idea is technically sound and the ablation results support the importance of code as an intermediate representation. The main risk is that the reasoning-task claim currently rests on an unverified transfer from descriptive-only training to reasoning synthesis at inference. I would recommend the revision focus on (a) adding a candidate-analysis experiment that separates selection from synthesis, (b) reporting variance across seeds, and (c) carefully rewriting the 'no human labels / no external models' claim to match the actual experimental design, which uses CharXiv question seeds and a GPT-4o judge. If the authors can provide the selection-vs-synthesis analysis, the paper would be considerably stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's real contribution is the code-as-intermediary synthesis pipeline: the VLM describes a chart, generates Python code, executes it, and extracts ground-truth chart properties from the plotting objects. That gives training triplets whose labels are exact by construction, which cleanly solves the noise-label problem that plagues self-training. The candidate-conditioned answering step is a second useful piece—train the same VLM to produce a final answer given its own sampled candidates. The evaluation is broader than usual: four VLMs, ablation, error-type analysis, and a case study, and the code is public. On the numbers, the gains are consistent. InternVL2-8B descriptive accuracy goes from 54.1 to 69.6, reasoning from 23.6 to 31.6, and the ablation shows code mediation matters (−Code loses 9.2 descriptive points). I trust the descriptive-task result.\n\nThe soft spots are real but manageable. The 'no human-labeled data or external models' claim is overstated. The question seeds are the human-curated descriptive questions from CharXiv, and GPT-4o is the evaluation judge. Using CharXiv test-set charts as synthesis seeds while reporting on the validation split is a mild contamination risk, though the synthetic items differ from the evaluation items. More important, the reasoning gain is probably not learned reasoning. The synthetic training data contains only descriptive QA pairs, and direct fine-tuning on it degrades reasoning accuracy from 23.6 to 21.9. So MANS is mostly learning to select or lightly paraphrase a correct candidate—the case study shows a minority correct candidate with added explanation. That is still a useful test-time scaling trick, but it is closer to reward-model selection than to self-improving reasoning. The paper should say so.\n\nMinor issues: no error bars or repeated seeds, and the synthetic dataset size and code success rate are not reported. These are easy to fix.\n\nVerdict: send to peer review. The descriptive-task recipe is solid, reproducible, and likely useful to practitioners. The reasoning-transfer question is testable and should be an explicit revision request. I would cite this if I worked on chart QA or synthetic data.\n\nRecommendation: accept with minor revisions, after the claims are aligned with what the experiment actually shows.","headline":"Code-executed synthetic labels are a genuinely useful fix for chart-QA self-training; the reasoning gains are real but likely candidate selection, not learned reasoning.","tokens_in":16992,"tokens_out":2662,"would_cite":true,"duration_ms":27238,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model improves its own chart understanding by up to 15.5 accuracy points, using generated code for clean training data and candidate-conditioned answering, with no human labels or external teacher model.","keywords":["chart understanding","vision language models","multimodal large language models","question answering","synthetic data generation","code-driven synthesis","candidate-conditioned answering","self-improvement"],"falsifier":"Isolate the CharXiv reasoning questions where all k sampled candidates are wrong, so Pass@K fails for that subset; if Chart-CoCa's accuracy on those questions is near zero or no better than picking a candidate at random, then the method is selecting from candidates rather than synthesizing new answers, and the reasoning gain would come from candidate coverage alone.","tokens_in":15976,"feed_emoji":"📊","tokens_out":10033,"duration_ms":98319,"temperature":0.7,"pith_summary":"Chart-CoCa tries to establish that a vision-language model can improve its own chart understanding without human-labeled answers or a stronger external model. It does this in two steps: first, the model turns an unlabeled chart into a description and then into executable Python code, and running that code produces both a new chart and the exact chart elements needed to label simple questions; second, the model is fine-tuned to produce a final answer when it is shown a chart, a question, and several candidate answers it has sampled. On the CharXiv benchmark with k=5 candidates, this lifts InternVL2-8B descriptive accuracy from 54.10 to 69.60 and reasoning accuracy from 23.60 to 31.60, and the same recipe improves other open vision-language models. If the claim holds, a vision-language model can bootstrap its own visual reasoning from code-verified synthetic data rather than relying on costly annotation or teacher models.","feed_headline":"Code-generated charts lift vision-model chart accuracy by 15.5 points","feed_subtitle":"Code-executed charts supply exact labels; a candidate-conditioned answer model beats all baselines.","key_machinery":"The mechanism that carries the argument is twofold. The first load-bearing object is executable code used as an intermediary: because the chart and its extracted information both come from executing the same code, the synthetic training labels are exact by construction rather than generated from noisy image-to-text predictions. The second is candidate-conditioned answering: the answer model receives the chart, the question, and several sampled responses, and is trained to emit the correct answer, so it learns to weigh and combine candidates instead of trusting majority consensus or a separately trained reward model. Code execution also provides the retry filter that discards failed synthesis attempts, and the candidate loop provides the test-time scaling that lets model accuracy grow with inference budget.","core_discovery":"The paper's central claim is that a single vision-language model can be its own data generator and its own answer synthesizer. Starting from an unlabeled real chart, the VLM writes a description, converts it to executable Matplotlib code, runs that code to draw a new chart, and reads chart elements such as titles, labels, ticks, legends, and lines back from the plotting objects; rule-based matching then turns those elements into accurate question-answer triplets without human annotation or a stronger teacher model. The same VLM is next fine-tuned on chart, question, and k sampled candidate answers to output the ground-truth answer, thereby learning to condition on candidates. At test time the initial model samples k answers and the fine-tuned answer model produces the final prediction. With k=5 on the CharXiv validation set, Chart-CoCa improves InternVL2-8B descriptive accuracy from 54.10 to 69.60 and reasoning accuracy from 23.60 to 31.60, and it outperforms majority voting, self-verification, reward-model selection, and direct fine-tuning.","pith_inferences":["The same code-as-intermediary design could generalize to any domain where executable code draws the visual input and exposes structured attributes, such as diagrams, maps, or synthetic scenes, giving a general route to noise-free self-training data.","The answer model may be learning to combine fragments of correct information across wrong candidates rather than simply selecting; this could be tested by feeding candidates that are all wrong but jointly contain the missing pieces and checking whether the final answer is assembled correctly.","Because direct fine-tuning on the synthetic triplets degrades reasoning, an iterative self-training loop should treat the code-execution filter, not the model's own confidence, as the guarantee that new training data stays clean."],"forward_implications":["On the CharXiv validation set, the full Chart-CoCa pipeline raises InternVL2-8B descriptive accuracy from 54.10 to 69.60 and reasoning accuracy from 23.60 to 31.60, surpassing majority voting, self-verification, reward-model selection, and direct fine-tuning.","Accuracy keeps rising as more candidate answers are sampled, with most of the practical gain reached by K=5–10 before the curve flattens.","The gains transfer across model families, including LLaVA-1.6-7B, Qwen2VL-7B, and InternVL2 in 7B, 8B, and 26B sizes, all improving over their initial models and over majority voting.","Smaller models improve the most in relative terms, so the method is most useful where the base vision-language model is weakest.","No human-labeled answers and no external teacher model are required, so the recipe can be rerun on any new vision-language model that can generate and execute code."],"supporting_citations":[{"why":"Provides the CharXiv benchmark, its descriptive/reasoning task split, and the evaluation protocol with an LLM-based answer checker.","marker":"[37]"},{"why":"Supplies the InternVL2 foundation models used throughout, including the initial-model baselines that Chart-CoCa improves.","marker":"[7]"},{"why":"Supplies the test-time scaling result that motivates spending more inference budget on candidate sampling.","marker":"[31]"},{"why":"Cited alongside [31] for the premise that compute-optimal sampling improves model reasoning at test time.","marker":"[4]"},{"why":"Provides the reward-model and best-of-N selection approach that Chart-CoCa is compared against and improves upon.","marker":"[19]"},{"why":"Provides the self-consistency and majority-voting baseline that serves as a main comparison for candidate aggregation.","marker":"[35]"},{"why":"Provides the self-verification baseline that Chart-CoCa is compared against in the candidate-based inference experiments.","marker":"[39]"}],"fun_headline_variants":["Self-improving chart model: code writes data, +15.5 pts","Chart VLM trains itself on its own code-generated charts","Candidate-conditioned answers lift chart VLM by 15.5","One model, no labels: code synthesis adds 15.5 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that candidate-conditioned answering trained on simple, rule-generated descriptive questions about synthetic charts transfers to open-ended reasoning questions about real charts, even though direct fine-tuning on the same synthetic data lowers reasoning accuracy in the paper's own ablation.","fun_headline_variants_meta":{"raw":{"variants":["Self-improving chart model: code writes data, +15.5 pts","Chart VLM trains itself on its own code-generated charts","Candidate-conditioned answers lift chart VLM by 15.5","One model, no labels: code synthesis adds 15.5 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1578,"prompt_tokens":911,"completion_tokens":667,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":603}},"tokens_in":527,"tokens_out":667,"duration_ms":6883,"temperature":1.0,"reasoning_tokens":603,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:26:58.407272+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Isolate the CharXiv reasoning questions where all k sampled candidates are wrong, so Pass@K fails for that subset; if Chart-CoCa's accuracy on those questions is near zero or no better than picking a candidate at random, then the method is selecting from candidates rather than synthesizing new answers, and the reasoning gain would come from candidate coverage alone.","supporting_citations":[{"cited_title":"Tran, and Mehran Kazemi","cited_arxiv_id":null,"evidence_quote":"Cited alongside [31] for the premise that compute-optimal sampling improves model reasoning at test time."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the reward-model and best-of-N selection approach that Chart-CoCa is compared against and improves upon."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the self-verification baseline that Chart-CoCa is compared against in the candidate-based inference experiments."}],"review_version":1}