Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replotting real-world charts so every training image carries its own exact data, then reinforcing numeric accuracy with GRPO, lifts 3B/7B vision-language models to state-of-the-art chart question answering.

desk verdict A genuinely useful replotting-and-RL recipe for chart reasoning, but the SOTA headline is undercut by an unaddressed potential train/eval overlap on the synthetic benchmarks. read the letter →

arxiv 2508.09804 v1 pith:QKPZTRU5 submitted 2025-08-13 cs.CL

Ahmed Masry , Abhay Puri , Masoud Hashemi , Juan A. Rodriguez , Megh Thakkar , Khyati Mahajan , Vikas Yadav , Sathwik Tejaswi Madhusudhan
show 8 more authors
Alexandre Piché Dzmitry Bahdanau Christopher Pal David Vazquez Enamul Hoque Perouz Taslakian Sai Rajeswar Spandana Gella
This is my paper
classification cs.CL
keywords chartquestionansweringreasoningvision-languagemodelsreplottingpipelinedatasetgenerationGroupRelativePolicyOptimizationverifiablerewardschain-of-thoughtfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that chart-reasoning models are held back by a data dilemma: synthetic charts are visually uniform, and real-world charts lack reliable underlying data. It proposes to break the dilemma with a replotting step — a teacher vision-language model writes plotting code for each collected real-world chart, the code is re-rendered into a fresh image, and the original is thrown away, so each image in the new BigCharts dataset comes with data that its own pixels actually show. On this resource the authors train 3B- and 7B-parameter open vision-language models with supervised fine-tuning followed by Group Relative Policy Optimization (GRPO), using a chart-specific reward that grades numeric answers by relative error. The paper reports that the resulting BigCharts-R1 models surpass previous chart-specific systems and several larger closed-source models on the averaged chart-QA benchmark suite, and that the reinforcement stage is what mainly buys out-of-distribution generalization — which, if right, makes this a reusable recipe for turning small open models into strong chart reasoners.

What carries the argument

Two mechanisms carry the argument. (1) The replotting pipeline: a teacher vision-language model (Gemini Flash 2.0) generates plotting code (matplotlib, plotly, or Chart.js) that reproduces each collected real-world chart; the code is executed to render a new image and the original is discarded. This yields a code-level data source consistent with the rendered pixels, so QA generation can trust the code for numbers and the image for visual detail. (2) The Chart Error Rate Reward (CERM): a dense reward mapping the relative error $\frac{|\hat{y}-y|}{|y|}$ to a value in $(0,1]$ via $\frac{1}{1+\mathrm{ER}}$, with exact match for non-numeric answers, summed with a response-format reward. CERM let

What would settle it

Sample a few hundred charts across log-scale, 3D, and dense-axis types; have two annotators read the key numeric values from the original images and compare them against the values embedded in the replotted code and QA ground truths. If systematic discrepancies appear — especially for log scales, where a misread tick is silently re-rendered as the 'truth' — the dataset's accuracy claim is falsified regardless of benchmark scores. Separately, ablating CERM to a binary exact-match reward while keeping everything else fixed would show whether the smooth relative-error reward, rather than RL in ge

Watch

Extended reading notes

Core claim

The paper zeroes in on a single trade-off: real chart images carry visual authenticity but no trustworthy data, while synthetic charts carry exact data but little visual variety. Its resolution is a replotting step — a teacher vision-language model writes code that reproduces each collected real-world chart, the code is re-rendered, and the original image is discarded, making image and underlying data consistent by construction. Trained on the resulting BigCharts dataset with supervised fine-tuning plus GRPO reinforcement learning and a chart-error reward (numeric answers graded by relative error), the 3B and 7B BigCharts-R1 models surpass prior chart-specific systems and several larger clos

Load-bearing premise

The dataset's accuracy claim rests on the teacher model's replotting code recovering the true data values of the original charts — a premise the paper does not validate, since discarding the originals makes image and code consistent with each other but not necessarily with the source chart.

Editorial extensions

If this is right

  • BigCharts-R1-3B and BigCharts-R1-7B exceed prior chart-specific models and several larger closed-source systems on the averaged chart-QA suite, so small open models can serve as practical chart reasoners without API calls.
  • The released BigCharts dataset (about 1.8 million questions with images, code, answers, and chains of thought) gives other groups a ready-made training resource for chart reasoning.
  • The RL stage is what the paper credits for out-of-distribution generalization: trained on ChartQA only, the GRPO model transfers better to PlotQA, DVQA, and FigureQA than the SFT model does, so RL can substitute for additional supervised data.
  • The paper's stated next targets — tables, geometric figures, chart summarization, and fact-checking — would inherit both the replotting pipeline and the reward-design approach.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the accuracy guarantee the pipeline advertises is self-consistency, not fidelity. Because the original images are discarded, any systematic error in the teacher's code — log-scale misreads, 3D charts, dense axes (Section 3.1.2) — becomes the ground truth for roughly 1.8M QA pairs, and the paper reports no validation of replotted values against the source charts. A small human audit c
  • My inference: the headline gains could partly come from QA+CoT supervision quality rather than from replotting per se, since the ablation in Table 2 compares original-chart QA (image only) against replotted-chart QA (image plus code), confounding the two.
  • My inference: if the reward design is the active ingredient, the recipe transfers to neighbouring problems — charts with ordinal or magnitude comparisons, tables, scientific figures — and a clean test is to swap CERM for a binary exact-match reward and measure where the RL gains disappear.
  • My inference: the stronger out-of-distribution result for RL over SFT (Section 5.4) implies that verification-driven RL may substitute for larger supervised datasets in chart reasoning; a direct test is training on a tenth of BigCharts with RL and comparing against full-data SFT.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes BigCharts, a 1.8M-question chart QA dataset built by collecting 245k real-world chart images (from existing datasets, Common Crawl, and Google Search) and 'replotting' them via Gemini Flash 2.0-generated code, then generating QA pairs with chain-of-thought from both the replotted image and the code. It also proposes BigCharts-R1, a training recipe combining SFT on BigCharts with GRPO-based RL using a chart-specific numeric error reward (CERM) and a format reward. Experiments on three synthetic benchmarks (FigureQA-Sub, DVQA-Sub, PlotQA-Sub) and two real-world benchmarks (ChartQA, CharXiv) show that BigCharts-R1-3B/7B outperform prior chart-specific models and several larger open/closed-source baselines. The paper additionally reports ablations (same-backbone comparison vs. TinyChart/ChartGemma data; original vs. replotted charts; RL vs. SFT for OOD generalization).

Significance. If the results hold, the contribution is substantial: a large, visually diverse chart reasoning dataset with (claimed) accurate underlying data, plus an SFT+RL recipe that improves chart reasoning and OOD generalization. The paper's methodological strengths are real: Table 2 conducts a same-backbone comparison against previous datasets, the replotted-vs-original ablation isolates the effect of the replotting pipeline, and Figure 4 provides a controlled RL-vs-SFT comparison on out-of-distribution benchmarks. The release of the dataset and code would be a useful resource for the chart QA community. However, two load-bearing issues—possible benchmark contamination from using FigureQA/DVQA/PlotQA images in the training data while evaluating on subsampled versions of those same benchmarks, and the unvalidated claim of 'accurate underlying data' in the replotting step—need to be resolved before the state-of-the-art claim can be accepted.

major comments (3)
  1. [§3.1.1 and §5.1 (Table 1)] The BigCharts training data aggregates images from FigureQA, DVQA, and PlotQA (§3.1.1), and the evaluation in Table 1 uses FigureQA-Sub, DVQA-Sub, and PlotQA-Sub created by sampling from 'the original splits' (§5.1). The paper never states that evaluation charts or their source images were excluded from the BigCharts training set. Moreover, the RL data explicitly includes 1K randomly sampled instances from the training subsets of PlotQA, DVQA, and FigureQA. If the same chart images (even after replotting) or the same question families appear in training and evaluation, the large gains on the three synthetic subsets reflect memorization rather than chart reasoning, and the averaged SOTA claim in Table 1 is directly undermined. Please report exactly which splits were used for FigureQA-Sub/DVQA-Sub/PlotQA-Sub (val/test), state explicitly whether any BigCharts source images or QA pairs overl
  2. [§3.1.2] The paper's central dataset-quality claim is that replotting 'ensures visual authenticity coupled with accurate underlying data.' But the replotting pipeline only guarantees consistency between the newly rendered image and the Gemini-generated code; it does not validate that the code recovers the true data values of the original real-world chart. The paper acknowledges that initial VLM code may be inaccurate but discards the original images, so any systematic misreading of log scales, 3D charts, dense axes, or color-coded values propagates into all ~1.8M QA pairs. No quantitative or human validation of replotted values against original source data is reported. This is load-bearing for the dataset's value and for the interpretation of Table 2's 'replotted vs. original' gain. Please report a validation study (e.g., human comparison of original vs. replotted values, or automated comparison
  3. [Eq. (3), §4.2] The CERM formula divides by |y| for numeric ground-truth values. If the ground-truth value is exactly zero (which can occur in real datasets, e.g., zero counts or zero growth), the error rate is undefined. This is a correctness issue in the reward function. Please specify how zero-valued answers are handled (e.g., absolute error threshold, or a separate case).
minor comments (6)
  1. [Figure 1 caption] The caption says 'a two-stage approach: (i) visual instruction tuning ... and (b) RL (GRPO)'—the second label should be '(ii)'.
  2. [§5.2] Typo: 'infact' should be 'in fact'.
  3. [§3.1.2] The mapping 1/(1+ER) is called 'homographic'; the intended term is probably 'homographic' as in a fractional linear transformation, but this is unusual. Consider calling it a 'hyperbolic' or 'reciprocal' mapping to avoid confusion.
  4. [Table 1] Several closed-source baselines have missing entries for ChartQA hum/aug and CharXiv reasoning/descriptive subsets. Please state why (e.g., not evaluated) so the reader can interpret the averaged column fairly.
  5. [A.7] The sample-output section repeatedly says 'Figure 14' for PlotQA, DVQA, FigureQA, and CharXiv samples; the figures are numbered 15–18 and should be cited correctly.
  6. [Abstract/§5.2] The phrase 'sets a new state-of-the-art standard' is stronger than the evidence warrants until the contamination question is resolved; consider qualifying the claim to 'state-of-the-art among models of comparable size on these benchmarks'.

Circularity Check

1 steps flagged · score 4.0 of 10

Replotting 'accuracy' claim is self-confirming by construction; benchmark SOTA remains largely independent.

  1. self definitional [Section 1 (Introduction) and Section 3.1.2 (Chart Re-Plotting)]
    "While initial VLM-generated code for the charts may have inaccuracies, we mitigate this by re-rendering these codes into new, improved chart images and discarding the original charts, ensuring visual authenticity coupled with accurate underlying data."

    The claimed 'accurate underlying data' is never validated against the original real-world charts. Because the original image is discarded and the new chart is rendered from the generated code, image-code consistency is true by construction: any VLM estimation error is baked into both the code and the re-rendered image. Thus 'accuracy' reduces to self-consistency with the same generated code, not fidelity to the source chart. This circularity supports the dataset-quality contribution, though it does not by itself force the benchmark results.

full rationale

The central benchmark claims are not circular in the strongest sense: ChartQA and CharXiv are external held-out benchmarks, the RL rewards are computed against provided ground-truth answers rather than fitted parameters, and the main SOTA comparison is not derived from the paper's own equations. The self-citations to ChartGemma and ChartQA are normal and not load-bearing: the evaluation would survive without them. The one genuine circular step is in the replotting pipeline: the paper defines 'accurate underlying data' as the code used to render the replacement chart, after discarding the original, so the claimed accuracy is guaranteed by construction rather than measured. A separate validity risk, not a demonstrated circular reduction, is that FigureQA/DVQA/PlotQA images are aggregated for training while evaluation subsets are sampled from 'original splits' of the same datasets; the paper does not state that evaluation charts were excluded, so those synthetic-benchmark gains could partly reflect overlap. This concern is real but conditional, so it is noted here rather than counted as a proven circular step.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the reliability of the teacher model and the representativeness of chosen data subsets rather than on fitted constants. The free parameters are hand-chosen pipeline choices (classifier thresholds, question-type mix, reward shape, RL data composition); none is fit to the evaluation benchmarks, which keeps the circularity burden low. The main axioms are domain assumptions about Gemini's code-generation fidelity and about subsample representativeness, both of which are testable but currently untested.

free parameters (4)
  • Chart classifier thresholds = 0.95 (stage 1), 0.98 (stage 2)
    Hand-chosen precision and recall trade-offs in Section 3.1.1 that determine the 57,196 Common Crawl charts.
  • Per-chart question type distribution = 16 questions (3 numeric-op, 3 visual-numeric, 3 retrieval, 2 yes/no, 2 counting, 1 unanswerable, 1 MCQ, 1 multi-turn)
    Hand-specified prompt structure in Figure 6; shapes the dataset and the model's learned behavior.
  • CERM reward mapping = 1/(1+ER)
    Hand-chosen homographic reward in Equation 3; affects RL optimization but not the benchmark definitions.
  • RL data composition = ChartQA train plus 1K each from PlotQA v1/v2, DVQA, and FigureQA training splits
    Hand-selected mix in Section 5.1; makes the template-based Sub evaluations partially in-distribution.
assumptions (5)
  • domain assumption Gemini Flash 2.0 code generation recovers chart data well enough that re-rendered charts plus generated QA are accurate training targets.
    Invoked in Sections 3.1.2 and 3.1.3; no validation against original chart data is reported.
  • domain assumption Replotted charts preserve the visual features that matter for transfer to real-world charts.
    Core to the visual authenticity claim in Sections 1 and 3.1.2; the replot-vs-original ablation in Table 2 provides indirect support.
  • standard math GRPO group-relative advantage estimates are unbiased in this RLVR setup.
    Adopted from Shao et al. 2024, Equation 2; standard result, not re-derived.
  • domain assumption 1K-sample subsets of FigureQA, DVQA, and PlotQA are representative of the test distributions.
    Section 5.1: subsets are self-created and representativeness is asserted without variance analysis.
  • domain assumption SFT teacher-generated chains-of-thought contain no systematic error that the RL stage cannot correct.
    Motivates RL in Section 4.2; RL uses only verifiable answers, not CoTs, so residual CoT errors remain in the SFT stage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning." pith.science (2026). https://pith.science/paper/QKPZTRU5

@misc{pith2026250809804,
  author       = {Pith},
  title        = {Pith review of: BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKPZTRU5}},
  note         = {Machine review of arXiv:2508.09804}
}
read the original abstract

Charts are essential to data analysis, transforming raw data into clear visual representations that support human decision-making. Although current vision-language models (VLMs) have made significant progress, they continue to struggle with chart comprehension due to training on datasets that lack diversity and real-world authenticity, or on automatically extracted underlying data tables of charts, which can contain numerous estimation errors. Furthermore, existing models only rely on supervised fine-tuning using these low-quality datasets, severely limiting their effectiveness. To address these issues, we first propose BigCharts, a dataset creation pipeline that generates visually diverse chart images by conditioning the rendering process on real-world charts sourced from multiple online platforms. Unlike purely synthetic datasets, BigCharts incorporates real-world data, ensuring authenticity and visual diversity, while still retaining accurate underlying data due to our proposed replotting process. Additionally, we introduce a comprehensive training framework that integrates supervised fine-tuning with Group Relative Policy Optimization (GRPO)-based reinforcement learning. By introducing novel reward signals specifically designed for chart reasoning, our approach enhances model robustness and generalization across diverse chart styles and domains, resulting in a state-of-the-art chart reasoning model, BigCharts-R1. Extensive experiments demonstrate that our models surpass existing methods on multiple chart question-answering benchmarks compared to even larger open-source and closed-source models.

Figures

Figures reproduced from arXiv: 2508.09804 by the authors.

Figure 1
Figure 1. BIGCHARTS construction and BIGCHARTS-R1 training pipeline. We begin by extracting a high-quality corpus from open chart datasets, Google Search, and Common Crawl(§3.1.1). We then generate and execute the code responsible for producing these charts to replot them (§3.1.2), and then derive question-answer pairs with chain-of-thought reasoning (§3.1.3). For training BIGCHARTS-R1, we use a two-stage approach: (i) visual… view at source ↗
Figure 2
Figure 2. Different Chart Generation Approaches. Generating QA pairs automatically can be achieved with three approaches: using only images (image-only), leveraging raw data values from the charts (data-only), or combining both (our proposed approach). Our method integrates semantic and visual features while ensuring accuracy in underlying data values. 3 BIGCHARTS Dataset 3.1 Generation Pipeline As illustrated in [PITH_FULL_… view at source ↗
Figure 3
Figure 3. BIGCHARTS Dataset Statistics. Type and topic distribution of the charts, chart sources, and type distribution of questions and answers. novel chart re-plotting strategy that recovers both visual styles and underlying data. We uti￾lize Gemini Flash 2.0 (Georgiev et al., 2024) to generate code replicating the design and con￾tent of each chart image. Recognizing potential inaccuracies in data estimation by vision￾langu… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Comparison between SFT and RL (GRPO) learning curves on in-distribution benchmark (ChartQA) and out-of-distribution benchmarks (PlotQA-Sub and DVQA-Sub). The RL model shows significant improvements over the SFT model in all cases. 5.4 Comparing RL vs SFT for Out-of-Dis…
Figure 5
Figure 5. Figure 5: Distribution of the number of tokens in Chain-of-Thoughts. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Prompt to SFT data using Gemini Flash-2.0. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Charts Underlying Python Code Generation Prompt. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Charts Underlying React Code Generation Prompt. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: The prompt used for identifying the chart type. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: The prompt used for identifying the chart topics. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: The prompt used for clustering the chart topics. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Examples of different rendered charts from our B [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Examples of different questions and their step-by-step solutions from our [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Examples of different outputs from BIGCHARTS-R1 on the ChartQA benchmark. Question: In how many years, is the enrolment ratio of male students in tertiary schools in Lao PDR greater than the average enrolment ratio of male students in tertiary schools in Lao PDR taken…
Figure 15
Figure 15. Figure 15: Examples of different outputs from BIGCHARTS-R1 on the PlotQA benchmark. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Examples of different outputs from BIGCHARTS-R1 on the DVQA benchmark. Question: Is Cyan less than Dark Blue? Question: Does Cyan have the highest value? Question: Is Dark Periwinkle the high median? Model Output: <thinking>The size of Cyan is 35 and the size of Dark …
Figure 17
Figure 17. Figure 17: Examples of different outputs from BIGCHARTS-R1 on the Figureqa benchmark. Question: What is the name of the subplot with a smoother transition on the surface from highest to lowest values? Question: When comparing the proportion of papers from top institutions with a…
Figure 18
Figure 18. Figure 18: Examples of different outputs from BIGCHARTS-R1 on the CharXiv benchmark. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Examples of original charts (the left) and their replotted variants (the right) in [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Chart Deception in Vision-Language Models: From Vulnerability to Mitigation

    cs.AI 2026-06 conditional novelty 6.0 of 10

    Misleading chart designs shift vision-language models' answers away from the true data interpretation; a paired benchmark measures this shift, and a model-extracted chart summary reduces it for most models.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [2023]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun

    URL https://arxiv.org/abs/2311.16483. Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv.org/abs/1512.03385. Enamul Hoque, Parsa Kavehzadeh and Ahmed Masry. Chart question answering: State of the art and future directions, 2022. URL https://arxiv.org/abs/2205.03966. Kevin Hu, Michiel...

    arXiv 2015

  2. [2025]

    chart_types

    URL https://arxiv.org/abs/2502.21321. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang et al. Tulu 3: Push- ing frontiers in open language model post-training, 2025. URL https://...

    arXiv 2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.