{"id":"46b2326f-0d52-4f2b-8a05-c7f1bc74389f","arxiv_id":"2507.01627","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 205-question chart QA benchmark from real analytical notebooks shows GPT-4.1 at 69% accuracy, exposing gaps in multimodal chart reasoning.","lead":"This paper introduces a new chart question answering benchmark built from real student visualization notebooks, with multiple charts, interactive views, and questions tied to written analysis. The best tested model, GPT-4.1, answers only 69% of questions correctly, showing current AI still struggles with realistic chart reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark validity rests on one annotator's answerability labels: Section 3.3's single-expert validation does not establish that all 205 questions are answerable from the supplied static screenshots, so the reported 69.27% accuracy partly reflects labeling decisions.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the single-expert validation in Section 3.3 determines whether questions are answerable from the visual inputs, and the benchmark's value depends on that determination being correct. I agree with this and find it the most consequential risk to the central claim. The paper's methodology is otherwise clearly described, the dataset is public, and the model-evaluation protocol is standard, but the core measurements rest on labels that have no demonstrated inter-annotator reliability. The concern is not that the authors behaved improperly; it is that a one-person pass, especially over LLM-generated questions extracted from narrative text, is a plausible source of systematic label noise. A second-annotator study would settle whether this risk is real. The existing CONDITIONAL verdict is appropriate; if the proposed test reveals low agreement, the verdict should move toward UNVERDICTED or REJECT until the dataset labels are revalidated.","tokens_in":7569,"tokens_out":3751,"duration_ms":48395,"concrete_test":"Have two independent annotators, blind to the published labels and to the narrative text, answer a random sample of at least 50 QA pairs using only the supplied images and answer choices, with 'cannot be determined' allowed. Compute agreement between the two annotators and between the annotators and the published labels. For each item, annotators should also indicate whether the correct answer is visually extractable from the images alone. If agreement with the published labels is below about 90%, or if a substantial fraction of items labeled 'answerable' are not visually determinable, then the headline accuracy and ecological-validity claim require revision. Report the annotation instructions, confusion patterns, and disagreements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the dataset measures chart QA in an ecologically valid setting, and the headline result is GPT-4.1 at 69.27% accuracy. That number is meaningful only if each of the 205 QA pairs has a determinate correct answer from the static screenshot(s) given to the model, or is correctly labeled 'cannot be determined' when no visual information suffices. The only gate for this is the manual validation described in Section 3.3, performed by a single 'data visualization expert' on all 429 generated pairs. The paper reports rejection criteria but gives no inter-annotator agreement, no second review pass, and no audit trail of per-item decisions. This matters because questions were generated from narrative text alone (Section 3.2, 'without parsing the chart images'), so the LLM had no access to the visual evidence. A chart can be 'reliably related to' a narrative claim without visually determining the answer, and the paper does not state that the validator independently re-derived each correct answer from the screenshot. The validator did label 33 questions as 'cannot be determined', showing that the distinction was applied, but the reliability of that application is unknown. With 2–68 questions per cell in Table 2, even a 5–10% labeling error rate could change per-task rankings and the overall gap. For example, Make Comparisons and Find Anomalies are near 50%, where a few mislabeled items would alter conclusions. Without quantified annotation reliability, the load-bearing assumption—answerability labels are correct—remains unverified, and the headline accuracy figures inherit that uncertainty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces RealWorld-ChartQA, a chart question answering dataset built from 22 student-authored litvis notebooks that combine analytical narratives with custom visualizations. The construction pipeline segments the narrative, uses an LLM to generate 429 multiple-choice QA pairs from text alone, and then filters them through a single-expert human validation step, yielding 205 QA pairs across 103 chart images; 36.6% of questions reference multiple views and 16.1% are labeled 'cannot be determined'. The authors benchmark GPT-4.1 and Qwen2.5-VL at 7B and 32B scales, reporting overall accuracies of 69.27%, 56.59%, and 31.71%, respectively, with task-level breakdowns. The central claims are that the dataset reflects ecologically valid analytical workflows, unlike prior CQA benchmarks, and that state-of-the-art MLLMs still show a significant performance gap on this more realistic setting.","tokens_in":7853,"tokens_out":4109,"duration_ms":48161,"significance":"The dataset addresses a real gap in CQA evaluation: most existing benchmarks use single static charts with questions derived from the image or caption, whereas this resource derives questions from surrounding analytical narratives and includes multi-view and interactively captured visualizations. The explicit 'cannot be determined' option is a useful design choice for probing over-confidence in MLLMs, and the public release of the dataset with a clearly documented pipeline is a strength. If the validity of the 205 answer labels can be established, the benchmark would complement ChartQA and CharXiv and give the community a more realistic testbed; the reported GPT-4.1 accuracy of 69.27% is a credible existence proof of an unsolved difficulty gap. However, the manuscript currently does not quantify the reliability of the single annotator's answerability decisions, and the small per-task sample sizes do not support the comparative task-level conclusions drawn in Section 5.","major_comments":[{"comment":"The benchmark's validity rests on a single 'data visualization expert' deciding that each retained QA pair is answerable from the available static screenshots, and this decision is not quantified. Because question generation in Section 3.2 explicitly operates on narrative text 'without parsing the chart images', the validator's answerability judgment is the only bridge between the LLM-generated questions and the visual evidence. The paper reports rejection criteria but gives no inter-annotator agreement, no second review pass, and no per-item audit trail. Since Table 2 contains task cells with as few as 2, 9, 12, and 15 items, even a 5-10% labeling error rate could change task-level rankings and the headline accuracy gap. Please add a second-annotator reliability study (e.g., Cohen's kappa on accept/reject decisions and on the final answer choice) or provide a complete item-level appendix documenting the specific visual evidence used for each retained and rejected pair.","section":"Section 3.3"},{"comment":"Accuracy is reported as point estimates without confidence intervals or significance tests on a 205-item set. The text claims that Qwen2.5-VL-32B outperforms GPT-4.1 on Make Comparisons (59.09% vs 50.00%, n=22) and Find Anomalies (55.56% vs 44.44%, n=9), but with these sample sizes the differences are within sampling error, and the Find Clusters row (n=2) cannot support any comparative conclusion. Please report binomial confidence intervals or exact significance tests and soften the task-level comparative claims accordingly; in particular, the conclusion that 'performance differences across task types are meaningful' is not supported by the reported statistics.","section":"Section 5, Table 2"},{"comment":"The evaluation protocol presents models only with the question and visualization image(s), not the analytical narrative from which the questions were derived. If the benchmark's contribution is that questions are grounded in analytical narratives, then the ecological-validity claim is stronger than what is measured: the reported performance gap reflects static visual QA, not narrative-grounded reasoning. Please either include a condition in which the narrative context is provided to the model, or explicitly scope the claims to 'questions derived from narratives' rather than to 'narrative-grounded reasoning workflows' as stated in the abstract and conclusion.","section":"Section 5, Appendix B"}],"minor_comments":[{"comment":"The description of interactive-view capture should quantify how many of the 103 images are interactive captures and how many interactive states could not be captured; Table 1's checkmark for 'Interactive' implies coverage that the dataset analysis does not currently substantiate.","section":"Section 3.1"},{"comment":"The model-evaluation prompt asks for only a letter, but the paper does not report how raw model outputs were post-processed (e.g., handling of refusals, extra text, or invalid letters) or what decoding parameters were used; this information is needed for reproducibility.","section":"Appendix B"},{"comment":"The description of CharXiv states that its questions 'still focus on one image'; since CharXiv images can contain multiple chart panels, the sentence should clarify that the restriction is to a single image rather than a single view.","section":"Section 2"},{"comment":"The Limitations section candidly notes the small dataset size and task imbalance, but it does not mention the unquantified annotator reliability; adding that caveat alongside the proposed second-annotation study would make the limitations statement more complete.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The dataset and question-generation pipeline are genuinely useful, and the single-annotator concern is addressable within the scope of a revision. If the revision does not add inter-annotator reliability evidence or an item-level audit, I would recommend rejection; the ecological-validity claim and all task-level comparisons depend on that evidence. The paper's framing in the abstract and conclusion should also be aligned with what the evaluation actually measures, i.e., static visual QA without narrative input."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper makes a real addition to the chart QA space. The dataset is small (205 QA pairs) but the design is new: questions generated from the narrative text of student-authored litvis notebooks, then validated against static captures that include multiple interactive views, plus a fifth 'cannot be determined' option. That combination is something CharXiv and MultiChartQA don't offer, and the ecological-validity argument is plausible. The method is clearly described, the data is public, and the benchmarking protocol is straightforward. Credit where it's due: the pipeline from narrative to QA via an LLM is adapted from Changpinyo et al., but applying it to visualization notebooks with a validation pass is a legitimate contribution.\n\nThe soft spots are in proportion to the paper's size. The stress-test concern about single-expert validation is correct. Section 3.3 relies on one 'data visualization expert' to decide whether each of 429 pairs is answerable from the supplied screenshots, and no inter-annotator reliability or audit trail is reported. With 205 final items, small per-task cells (e.g., Find Clusters n=2), and no confidence intervals or significance tests, the headline accuracy gap between GPT-4.1 and Qwen 32B (69% vs 57%) is not established as statistically meaningful. The 'cannot be determined' label is smart, but its correctness is exactly what isn't verified. I don't think this is fatal—the dataset doesn't need perfect labels to be useful as a pilot—but it does mean the strong claims in the abstract and conclusion outrun the evidence.\n\nI'd push back gently on the reader's conditional verdict: I think the paper is slightly stronger than the reader's moderate confidence suggests, because the qualitative examples in Appendix C show the questions do require multi-view reasoning and the 'cannot be determined' option is genuinely used. But the underlying concern is valid, and a referee would rightly ask for a second annotation pass, some measure of agreement, and error bars.\n\nWho is this for? People building or benchmarking CQA models will find it a fresh test bed, even if small. Visualization literacy researchers may also find the narrative-grounded generation method interesting. I'd bring it to a reading group, and I'd cite it if I were writing on CQA evaluation axes. It deserves a serious peer review, but the authors should be prepared to harden the validation and statistics.\n\nRecommendation: send to a venue that takes dataset papers seriously; ask for major revisions on annotation reliability and statistical grounding. If they add a second annotator and report uncertainty, this becomes a solid contribution.","headline":"A small but genuinely new CQA dataset with an ecological-validity angle; the single-annotator validation is the main weakness, not a fatal one.","tokens_in":8413,"tokens_out":2842,"would_cite":true,"duration_ms":33530,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a chart question answering dataset built from real literate-visualization notebooks, with questions generated from the students' analytical narratives rather than from chart images alone, and shows that…","keywords":["chart question answering","visualization notebooks","multimodal large language models","benchmark dataset","analytical narratives","interactive visualizations","multiple-choice evaluation","ecological validity"],"falsifier":"If an independent second annotation of the 205 pairs found that a meaningful fraction, say more than 10%, of questions labeled answerable cannot actually be answered from the supplied screenshots alone, the dataset's core claim of visual answerability would be undercut. A simpler check: run GPT-4.1 again with the relevant narrative text appended to the question; if accuracy jumps dramatically, the questions are not actually being answered from the charts.","tokens_in":7381,"feed_emoji":"📊","tokens_out":4904,"duration_ms":52023,"temperature":0.7,"pith_summary":"This paper introduces a chart question answering dataset built from real literate-visualization notebooks, with questions generated from the students' analytical narratives rather than from chart images alone. The resulting pairs involve multiple charts, multiple views, and some interactive states, and include a 'cannot be determined' option for genuinely unanswerable questions. The central claim is that this setup is more ecologically valid than existing CQA benchmarks, and that state-of-the-art multimodal language models still leave a clear performance gap: GPT-4.1 reaches 69.3% accuracy, with smaller models well below that. A sympathetic reading takes this as evidence that real-world chart reasoning is not yet solved by current MLLMs.","feed_headline":"Real chart questions stump GPT-4.1 nearly a third of the time","feed_subtitle":"A new benchmark from analyst notebooks pairs multiple views and narratives, exposing where multimodal models still stumble.","key_machinery":"The construction pipeline is the carrier: literate visualization notebooks provide narrative text plus custom charts; narrative is segmented, an LLM generates question-answer pairs grounded in quoted narrative and classified by the VLAT task taxonomy; a data-visualization expert manually validates each pair against actual screenshots, rejecting pairs that refer to unavailable visualizations or non-analytical narrative, and labels pairs as unanswerable when required information exists only in uncaptured interactive views. Interactive controls are captured statically by enumerating categorical options. This pipeline yields a benchmark whose questions cannot be answered from narrative alone, nor from a single static chart, and it is the mechanism that produces the claimed ecological validity.","core_discovery":"On its own terms, the paper's discovery is a dataset—205 expert-validated multiple-choice QA pairs paired with 103 chart images from 22 student notebooks—that re-anchors chart question answering in authentic analytical workflows. It shows that when questions derive from written narrative and target multi-view or interactive charts, prior benchmarks' single-image, single-view format misses an important part of real practice. Benchmarking GPT-4.1 and Qwen2.5-VL at two scales gives 69.3%, 56.6%, and 31.7% accuracy respectively, a substantial gap that varies by task type, with interpretive tasks like Make Comparisons and Find Anomalies proving harder than literal retrieval.","pith_inferences":["The interactive-view capture method generalizes: any benchmark built from notebooks or dashboards with discrete controls could adopt the same enumeration trick, pushing CQA beyond static images without needing full interaction logs.","A natural next experiment is to condition the same questions on narrative text plus charts, separating how much of the gap is visual versus contextual; the paper's design, with narrative removed from the vision-only evaluation, already sets up this control.","The unanswerable-pair subset could be used to test model abstention behavior separately, since over-answering on 'cannot be determined' items would indicate a tendency to hallucinate from ambiguous visual evidence.","With only 205 pairs and a single expert validator, the 69.3% figure is a point estimate; re-annotation studies or larger notebooks would tell whether the performance gap is stable or partly a small-sample artifact."],"forward_implications":["Current MLLMs answer only about 69% of these real-world chart questions, leaving a roughly 30-point gap for deployment in visual-assistant settings where multi-chart, narrative-grounded reasoning is common.","The dataset includes a 'cannot be determined' answer option, so models that cannot find evidence are expected to say so, steering CQA evaluation toward calibration and honesty rather than guessing.","Performance varies by task type: literal retrieval tasks are easier, while finding anomalies and making comparisons are harder, suggesting the VLAT taxonomy is a useful diagnostic for where multimodal reasoning breaks.","Because the data comes from student notebooks with explicit consent, the dataset offers a copyright-clean and ethically approved alternative to scraped chart corpora for future benchmark construction."],"supporting_citations":[{"why":"Supplies the eight-category task taxonomy used to structure question generation and to break down evaluation results.","marker":"Lee et al. (2017)"},{"why":"ChartQA is the prior real-world-chart benchmark whose evaluation protocol and single-chart limitation this work directly extends.","marker":"Masry et al. (2022)"},{"why":"CharXiv documents gaps in realistic chart understanding with multi-subplot charts, motivating the multi-view focus here.","marker":"Wang et al. (2024)"},{"why":"MultiChartQA is the closest prior benchmark allowing questions over multiple related charts, but it still lacks interactive views.","marker":"Zhu et al. (2025)"},{"why":"The idea of generating question-answer pairs from language context rather than image pixels is borrowed from this VQA work.","marker":"Changpinyo et al. (2022)"},{"why":"Literate visualization is the notebook format that provides the source narratives and charts for the dataset.","marker":"Wood et al. (2019)"},{"why":"Technical report for the Qwen2.5-VL models, one of the two model families benchmarked on the new dataset.","marker":"Bai et al. (2025)"},{"why":"GPT-4.1 is the strongest model evaluated; its 69.3% accuracy anchors the reported performance gap.","marker":"OpenAI (2025)"}],"fun_headline_variants":["Real-world chart benchmark trips GPT-4.1 at 69.3% accuracy","Notebook-sourced chart questions expose GPT-4.1's 30% gap","Multimodal models lag on narrative-driven chart QA, new dataset","Chart QA from analytical narratives: GPT-4.1 only 69.3% correct","New benchmark: analyst notebooks push GPT-4.1 to its limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's value rests on the assumption that one data-visualization expert correctly determined, from static screenshot captures, that each retained question is answerable from the provided visual inputs, and that no retained question implicitly depends on narrative-only information or on interactive dynamics that screenshots miss.","fun_headline_variants_meta":{"raw":{"variants":["Real-world chart benchmark trips GPT-4.1 at 69.3% accuracy","Notebook-sourced chart questions expose GPT-4.1's 30% gap","Multimodal models lag on narrative-driven chart QA, new dataset","Chart QA from analytical narratives: GPT-4.1 only 69.3% correct","New benchmark: analyst notebooks push GPT-4.1 to its limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000751,"raw_usage":{"total_tokens":3243,"prompt_tokens":747,"completion_tokens":2496,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":363,"completion_tokens_details":{"reasoning_tokens":2401}},"tokens_in":363,"tokens_out":2496,"duration_ms":20072,"temperature":1.0,"reasoning_tokens":2401,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:45:50.306738+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If an independent second annotation of the 205 pairs found that a meaningful fraction, say more than 10%, of questions labeled answerable cannot actually be answered from the supplied screenshots alone, the dataset's core claim of visual answerability would be undercut. A simpler check: run GPT-4.1 again with the relevant narrative text appended to the question; if accuracy jumps dramatically, the questions are not actually being answered from the charts.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CharXiv documents gaps in realistic chart understanding with multi-subplot charts, motivating the multi-view focus here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The idea of generating question-answer pairs from language context rather than image pixels is borrowed from this VQA work."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4.1 is the strongest model evaluated; its 69.3% accuracy anchors the reported performance gap."}],"review_version":1}