Pith. sign in

REVIEW 3 major objections 5 minor 91 references

CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-of-the-art vision-language models underperform human readers on data-visualization reasoning, even under lenient scoring, and their error patterns diverge from human performance.

desk verdict A useful human-centered benchmark and a mostly convincing demonstration of a VLM-human gap, but the unvalidated GPT-4 response extraction makes every accuracy number conditional on a pipeline that could be authoring answers. read the letter →

arxiv 2505.17202 v1 pith:BDNACV7M submitted 2025-05-22 cs.HC cs.CLcs.CV

Arnav Verma , Kushin Mukherjee , Christopher Potts , Elisa Kreiss , Judith E. Fan This is my paper
classification cs.HCcs.CLcs.CV
keywords chartunderstandingdatavisualizationliteracyvision-languagemodelshuman-AIcomparisoncognitivebenchmarkingerrorpatternsreasoningevaluationprotocol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether vision-language models can serve as computational models of how humans understand data visualizations, and its answer is no for the eight models tested. The authors assembled CHART-6, a suite of 851 items drawn from six assessments, five of which were originally built to measure visualization literacy in humans and one of which is a human-tested subset of ChartQA. Across all six tests, every model scored below human participants on average, and the gap persisted when invalid model responses were discarded to give the models the benefit of the doubt. The paper also shows that model error patterns are reliably different from human error patterns: no model's item-by-item response profile approached the human noise ceiling. The upshot is that current systems are not yet a strong foundation for cognitive models of visualization understanding, even though several approached human performance on individual assessments.

What carries the argument

The load-bearing object is CHART-6, a human-centered benchmark suite that combines five human-designed visualization-literacy tests—GGR, VLAT, CALVI, HOLF, and HOLF-Multi—with ChartQA-Human, a 125-item numerical-response subset of ChartQA. The evaluation protocol presents every one of the 851 items to each of eight vision-language models ten times, tunes decoding hyperparameters per model on VLAT, uses a separate GPT-4-based extraction step to convert raw outputs into scorable answers, and validates that an answer is formatted correctly before scoring. The comparison machinery includes two accuracy bounds (valid-only versus all-responses-with-invalid-as-wrong), mixed-effects regression for item-level variation, and a Spearman-Brown-corrected human noise ceiling used to judge whether model error patterns are within the range of human behavior.

What would settle it

Take a random sample of raw model outputs and have human annotators extract the answers using the same instructions given to GPT-4; if the human-extracted answers disagree systematically with the GPT-4-extracted answers, the reported accuracy gaps and error-pattern correlations are not stable. A decisive check would recompute the model-minus-human gap on GGR using only human-annotated extractions and see whether it remains negative.

Watch

Extended reading notes

Core claim

The central claim is that current state-of-the-art vision-language models consistently underperform humans on reasoning tasks involving data visualizations, and the gap is especially pronounced on assessments developed to measure these skills in humans. When only valid model responses were counted—a lenient condition that inflates model accuracy—GPT-4V, the best model, still scored reliably below humans on GGR, HOLF, and HOLF-Multi, and approached but did not clearly reach human level on VLAT and ChartQA-Human. Under the stricter condition where all responses, including invalid format failures, were marked wrong, GPT-4V fell below humans on five of the six tests. The paper further claims that relative difficulty across items was only weakly to moderately aligned between models and humans, and that all models produced error patterns that were statistically distinguishable from human error patterns, falling far short of the human noise ceiling.

Load-bearing premise

The conclusions rest on the unvalidated assumption that the GPT-4-based extraction pipeline faithfully converts raw model outputs into the answer formats used for scoring, without silently authoring or discarding valid answers.

Editorial extensions

If this is right

  • On human-designed visualization-literacy tests, every evaluated model scored below human participants on average, even when only valid responses were counted.
  • Benchmarking on ChartQA-style items alone would overstate progress: on the numerical subset, GPT-4V approached human performance with valid-only scoring but fell clearly behind once invalid responses were counted.
  • Relative item difficulty correlated with humans for GPT-4V (r = 0.40 across graph types; r = 0.94 across task categories), but the size of the performance gap varied reliably across graph and task types.
  • No model's error pattern approached the human noise ceiling; GPT-4V was closest on several tests, while different open models were closest on others, so no single architecture currently reproduces human visualization reasoning.
  • Gaps were largest on GGR, CALVI, and HOLF, suggesting that assessments designed to measure human literacy expose weaknesses that chart-specific AI benchmarks do not.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension (not claimed in the paper): replacing the GPT-4 response-extraction step with human annotation could shift validity rates and accuracy estimates; re-analyzing a sample of raw outputs with human annotators would test how much of the reported gap depends on the extraction pipeline.
  • Extension (not claimed in the paper): the reliable divergence in error patterns suggests a concrete alignment target: training models to reproduce human error distributions rather than only maximizing aggregate accuracy.
  • Extension (not claimed in the paper): the wide spread in valid-response rates across models means single-number accuracy claims for chart understanding are misleading unless accompanied by validity rates and both valid-only and all-response scoring.
  • Extension (not claimed in the paper): because human-designed literacy tests produced larger model-human gaps than ChartQA, future VLM evaluations should include such tests alongside AI-originated benchmarks to avoid overestimating human-like chart understanding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CHART-6, a suite of six data-visualization literacy assessments (GGR, VLAT, CALVI, HOLF, HOLF-Multi, and a constructed ChartQA-Human subset), and evaluates eight vision-language models on all 851 items, with each item presented ten times per model. Model responses are compared with human behavioral data from prior studies and a new Prolific sample. The central finding is that, even under a lenient 'valid responses only' scoring rule, the models perform worse than humans on average, and none of the models approach the human noise ceiling in item-level error-pattern correlations. The authors conclude that current VLMs do not yet provide a strong computational model of human visualization understanding, and they release code and data for reproduction.

Significance. If the central claim holds, this is a valuable contribution to cognitive-AI benchmarking: it connects human visualization-literacy assessments to VLM evaluation, spans six independently designed tests with different response formats, and includes eight models of varied architecture and scale. The study is methodologically careful in several respects: it reports both an upper bound (valid responses only) and a lower bound (invalid responses scored as incorrect), uses bootstrap confidence intervals for group comparisons, and the benchmarks and human baselines were not constructed by fitting the models, so the main inference is not circular. The open code and data are an additional strength. However, all model-accuracy and error-pattern results are conditional on an unvalidated GPT-4 response-extraction pipeline, which directly affects the 'valid-only' upper bound on which the paper's most important conclusion rests.

major comments (3)
  1. [Section 2.4 / Appendix A.2 / Figure 3] The response-extraction pipeline is load-bearing but is never validated. The GPT-4 prompt in Appendix A.2 embeds the original question, the multiple-choice alternatives, and instructs the model to 'answer the question and provide the correct option,' so GPT-4 is free to solve the item rather than transcribe the raw model output. No inter-annotator agreement, human-checked subset, or check that the extracted string appears in or is derivable from the raw response is reported. Because all accuracy estimates, including the 'valid only' upper bound in Section 3.2, and all error-pattern correlations in Section 3.4 are computed on processed outputs rather than raw outputs, the central claim that VLMs underperform humans even when only valid responses are counted is conditional on an untested transformation. I request a validation of the extraction pipeline on a random sample of raw outputs, with human annotation of the extracted answers, agreement statistics, and a breakdown of failure modes (e.g., GPT-4 discarding a correct raw answer, or hallucinating an option not present in the raw output).
  2. [Section 2.4 / Table 2] Decoding hyperparameters (temperature and top-p) were selected by a grid search that maximized each model's performance on VLAT, and the same values were then used for the remaining five assessments. This makes the VLAT results optimistically biased relative to the other tests and could distort cross-test comparisons such as the claim in Section 3.2 that GPT-4V 'did approach human performance on VLAT' while showing larger gaps on other tests. The paper should report a sensitivity analysis (for example, evaluating each model with multiple hyperparameter settings on a held-out test, or at least quantifying how much VLAT-tuned choices affect the other tests), or explicitly discuss this as a limitation of the cross-test comparisons. At present the reuse of tuned hyperparameters is a potential source of overfitting that is not addressed.
  3. [Section 3.4] The error-pattern correlations are computed on processed outputs with highly variable numbers of valid responses per item (at most 10, and far fewer for models such as LLaVA1.6-Yi-34B, which produced valid responses for only 32% of items). The bootstrap procedure resamples items but does not model response-level sampling noise within items, and the 'valid only' error-pattern vectors for different models are based on different item subsets. This makes the comparison to the human noise ceiling and the cross-model ranking of correlations difficult to interpret. The paper should report the number of items contributing to each model's valid-only error-pattern vector, consider weighting by the number of valid responses per item, and ideally provide a simulation or alternative estimator that accounts for item-level response noise.
minor comments (5)
  1. [Abstract / Introduction] The full-text abstract contains 'Y et' where 'Yet' is intended; this typo should be corrected.
  2. [Section 3.2] The sentence describing the upper and lower bounds contains a duplicated 'and and': 'when considering only valid responses (upper bound) and and when considering all responses.'
  3. [Section 3.2, lower-bound paragraph] The reported confidence interval for HOLF (95% CI = [-0.50, -0.23]) is identical to the CI reported for CALVI in the same sentence, whereas the valid-only HOLF CI given earlier is [-0.25, -0.15]; this appears to be a typographical error and should be verified.
  4. [Section 2.4] In the sentence 'Following prior work [85], we also used GPT-44', the '4' appears to be an erroneous superscript or duplicated character from the footnote marker; please fix.
  5. [Figure 7 / Section 3.4] The model name is inconsistent: Section 3.4 refers to 'LLaVA1.5-Vicuna-13B' while Figure 7 and Figure 5 labels use 'LLaVA1.6-Vicuna-13B'; the correct model name should be used consistently throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation is a measurement study whose benchmarks and human baselines are external empirical data; no fitted parameter or self-citation chain defines the reported model-human gap.

full rationale

The paper's central claim is that eight vision-language models underperform humans on six visualization-reasoning tests, even under a lenient 'valid responses only' criterion, and that model error patterns are reliably distinct from human error patterns. This claim is supported by an evaluation protocol, not by a derivation from fitted inputs. The six tests (GGR, VLAT, CALVI, HOLF, HOLF-Multi, ChartQA-Human) were developed independently of this paper, and the human baselines come either from prior published studies or from a newly recruited Prolific sample; none of the human data are derived from the models being evaluated. The only fitted quantities are per-model decoding hyperparameters (top-p and temperature), which are selected on VLAT and then applied to all tests. This is an evaluation decision that could inflate VLAT performance, but it does not make any result true by construction: the reported human-model gap is an empirical outcome, not an identity. The GPT-4-based response-extraction pipeline (Section 2.4, Appendix A.2) is an unvalidated measurement step that could in principle author or discard answers, but that is a correctness or reliability risk, not circularity: the extracted response is not defined as the target quantity, and the paper's conclusions are not logically forced by the extraction prompt. Self-citations such as [52] for GGR/VLAT human data and [41, 40] for HOLF/HOLF-Multi human data are normal uses of prior empirical datasets and are not load-bearing in a circular sense, because those data are external observations, not consequences of the present model evaluations. No uniqueness theorem, ansatz-via-citation, or renaming of a known result is used to justify the conclusions. The derivation chain is therefore self-contained with respect to circularity, and the appropriate score is 0.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities, forces, dimensions, or conserved quantities; CHART-6 is an assembly of existing tests. The main free parameters are per-model decoding settings tuned on VLAT, and the main assumptions concern pooling human data, the task-category mapping, the noise-ceiling interpretation, and the fidelity of GPT-4-based response extraction.

free parameters (8)
  • Decoding temperature and top-p for Blip2-FlanT5-4B = top-p 0.6, temperature 1.0 (Table 2)
    Selected by grid search maximizing VLAT accuracy; applied to all other tests.
  • Decoding temperature and top-p for Blip2-FlanT5-11B = top-p 1.0, temperature 1.0 (Table 2)
    Selected by grid search maximizing VLAT accuracy; applied to all other tests.
  • Decoding temperature and top-p for LLaVA1.5-Vicuna-7B = top-p 0.4, temperature 1.0 (Table 2)
    Selected by grid search maximizing VLAT accuracy; applied to all other tests.
  • Decoding temperature and top-p for LLaVA1.5-Vicuna-13B = top-p 1.0, temperature 0.4 (Table 2)
    Selected by grid search maximizing VLAT accuracy; applied to all other tests.
  • Decoding temperature and top-p for LLaVA1.6-Yi-34B = top-p 1.0, temperature 0.4 (Table 2)
    Selected by grid search maximizing VLAT accuracy; applied to all other tests.
  • Decoding temperature and top-p for Pix2Struct-0.3B = top-p 0.8, temperature 1.0 (Table 2)
    Selected by grid search maximizing VLAT accuracy; applied to all other tests.
  • Decoding temperature and top-p for MatCha-0.3B = top-p 0.4, temperature 1.0 (Table 2)
    Selected by grid search maximizing VLAT accuracy; applied to all other tests.
  • Decoding temperature and top-p for GPT-4V = top-p 1.0, temperature 0.2 (Table 2)
    Selected by grid search maximizing VLAT accuracy; applied to all other tests.
assumptions (4)
  • domain assumption Human data from different prior studies are comparable and can be pooled
    Sections 2.3 and 3.2 compare model accuracy to previously collected human data from GGR, VLAT, CALVI, HOLF, and HOLF-Multi, which used different participant samples, item counts, and response formats; pooling assumes these are exchangeable measures of human performance.
  • domain assumption Common task categories (value identification, arithmetic computation, statistical inference) are valid across all tests
    Section 2.2 and Table 1 assign these categories, including for ChartQA-Human which had none; the mapping is subjective and could affect graph and task analyses in Section 3.3.
  • domain assumption The human noise ceiling estimated by Spearman-Brown-corrected split-half correlation is an appropriate upper bound
    Section 3.4 uses this ceiling to conclude models are far from humans; this assumes the split-half procedure captures reliable human variance and that no other source of model-human agreement is missed.
  • domain assumption GPT-4 response extraction is lossless and unbiased
    Appendix A.2 provides no validation of the extractor against human annotation; all model performance and error-pattern analyses depend on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models." pith.science (2026). https://pith.science/paper/BDNACV7M

@misc{pith2026250517202,
  author       = {Pith},
  title        = {Pith review of: CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BDNACV7M}},
  note         = {Machine review of arXiv:2505.17202}
}
read the original abstract

Data visualizations are powerful tools for communicating patterns in quantitative data. Yet understanding any data visualization is no small feat -- succeeding requires jointly making sense of visual, numerical, and linguistic inputs arranged in a conventionalized format one has previously learned to parse. Recently developed vision-language models are, in principle, promising candidates for developing computational models of these cognitive operations. However, it is currently unclear to what degree these models emulate human behavior on tasks that involve reasoning about data visualizations. This gap reflects limitations in prior work that has evaluated data visualization understanding in artificial systems using measures that differ from those typically used to assess these abilities in humans. Here we evaluated eight vision-language models on six data visualization literacy assessments designed for humans and compared model responses to those of human participants. We found that these models performed worse than human participants on average, and this performance gap persisted even when using relatively lenient criteria to assess model performance. Moreover, while relative performance across items was somewhat correlated between models and humans, all models produced patterns of errors that were reliably distinct from those produced by human participants. Taken together, these findings suggest significant opportunities for further development of artificial systems that might serve as useful models of how humans reason about data visualizations. All code and data needed to reproduce these results are available at: https://osf.io/e25mu/?view_only=399daff5a14d4b16b09473cf19043f18.

Figures

Figures reproduced from arXiv: 2505.17202 by the authors.

Figure 1
Figure 1. Sample response from all evaluated models for a multiple-choice item. Responses after processing are shown in bold and are used for [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. We present CHART-6 (Comparative Human-AI Graphical Reasoning Tests), a human-centered suite of data visualization understanding benchmarks, to assess how close state-of-the-art vision-language models are to achieving both human-level performance and human-like behavior on reasoning tasks involving data visualizations. This test suite spans a wide array of different approaches to designing such assessments, ensuring … view at source ↗
Figure 3
Figure 3. Procedure for processing and validating model responses for comparison to human responses. All vision-language models were presented [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Proportion of valid responses produced by each model on each assessment. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Human and model performance on (A) the mean proportion correct in multiple-choice assessments (GGR, VLAT, and CALVI) and (B) the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Mean proportion of correct responses between [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Pearson correlation between error patterns produced by each model and [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

91 extracted references · 57 canonical work pages

  1. [1]

    Alayrac, J

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022. 1

    2022

  2. [2]

    Alper, N

    B. Alper, N. H. Riche, F. Chevalier, J. Boy, and M. Sezgin. Visualization literacy at elementary school. In Proceedings of the 2017 CHI conference on human factors in computing systems, pp. 5485–5497, 2017. 8

    2017

  3. [3]

    R. Amar, J. Eagan, and J. Stasko. Low-level components of analytic activity in information visualization. In IEEE Symposium on Information Visualization, 2005. INFOVIS 2005., pp. 111–117. IEEE, 2005. 1

    2005

  4. [4]

    D. Bear, E. Wang, D. Mrowca, F. Binder, H.-Y . Tung, P. RT, C. Holdaway, S. Tao, K. Smith, F.-Y . Sun, F.-F. Li, N. Kanwisher, J. Tenenbaum, D. Yamins, and J. Fan. Physion: Evaluating physical prediction from vision in humans and machines. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1, 2021. 2, 8

    2021

  5. [5]

    J. Bertin. Graphics and graphic information processing. Walter de Gruyter,

  6. [6]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models, 2022. 1

    2022

  7. [7]

    Bonnen, S

    T. Bonnen, S. Fu, Y . Bai, T. O’Connell, Y . Friedman, N. Kanwisher, J. Tenenbaum, and A. Efros. Evaluating multiview object consistency in humans and image models. Advances in Neural Information Processing Systems, 37:43533–43548, 2024. 2

    2024

  8. [8]

    Börner, A

    K. Börner, A. Bueckle, and M. Ginda. Data visualization literacy: Definitions, conceptual frameworks, exercises, and assessments. Proceedings of the National Academy of Sciences , 116(6):1857–1864,

Show all 91 references
  1. [9]

    Börner, A

    K. Börner, A. Maltese, R. N. Balliet, and J. Heimlich. Investigating aspects of data visualization literacy using 20 information visualizations and 273 science museum visitors. Information Visualization, 15(3):198–213, 2016. 8

    2016

  2. [10]

    J. Boy, R. A. Rensink, E. Bertini, and J.-D. Fekete. A principled way of assessing visualization literacy. IEEE Transactions on Visualization and Computer Graphics, 20(12):1963–1972, 2014. doi: 10.1109/TVCG.2014. 2346984 1, 8

    1963 doi

  3. [11]

    Brehmer and T

    M. Brehmer and T. Munzner. A multi-level typology of abstract visualization tasks. IEEE Transactions on Visualization and Computer Graphics, 19(12):2376–2385, 2013. doi: 10.1109/TVCG.2013.124 1

    2013 doi

  4. [12]

    Brockbank, A

    E. Brockbank, A. Verma, H. Lloyd, H. Huey, L. Padilla, and J. E. Fan. Evaluating convergence between two data visualization literacy assessments. Cognitive Research: Principles and Implications, 2025. 8

    2025

  5. [13]

    P. A. Carpenter and P. Shah. A model of the perceptual and conceptual processes in graph comprehension. Journal of Experimental Psychology: Applied, 4(2):75–100, 1998. doi: 10.1037/1076-898X.4.2.75 1

    1998 doi

  6. [14]

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V . Zhao, Y . Huang, A. Dai, H. Y...

    2024

  7. [15]

    W. S. Cleveland and R. McGill. Graphical perception: Theory, experimentation, and application to the development of graphical methods. Journal of the American Statistical Association, 79(387):531–554, 1984. doi: 10.1080/01621459.1984.10478080 1

    1984

  8. [16]

    N. R. Council. Developing assessments for the next generation science standards. National Academies Press, 2014. 1

    2014

  9. [17]

    Y . Cui, L. W. Ge, Y . Ding, L. Harrison, F. Yang, and M. Kay. Promises and Pitfalls: Using Large Language Models to Generate Visualization Items. IEEE Transactions on Visualization and Computer Graphics, 31(1):1094– 1104, Jan. 2025. Conference Name: IEEE Transactions on Visua...

    2025

  10. [18]

    Y . Cui, W. G. Lily, Y . Ding, L. Harrison, F. Yang, and M. Kay. Promises and pitfalls: Using large language models to generate visualization items. IEEE Transactions on Visualization and Computer Graphics, 2024. 8

    2024

  11. [19]

    Y . Cui, W. G. Lily, Y . Ding, F. Yang, L. Harrison, and M. Kay. Adaptive assessment of visualization literacy. IEEE Transactions on Visualization and Computer Graphics, 30(1):628–637, 2023. 8

    2023

  12. [20]

    Cumming and S

    G. Cumming and S. Finch. Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60(2):170–180, 2005. doi: 10.1037/0003-066X.60.2.170 1

    2005 doi

  13. [21]

    F. R. Curcio. Comprehension of mathematical relationships expressed in graphs. Journal for Research in Mathematics Education, 18(5):382–393,

  14. [22]

    J. E. Fan, W. A. Bainbridge, R. Chamberlain, and J. D. Wammes. Drawing as a versatile cognitive tool. Nature Reviews Psychology, 2(9):556–568,

  15. [23]

    T. Fel, I. Felipe, D. Linsley, and T. Serre. Harmonizing the object recognition strategies of deep neural networks with humans. Advances in neural information processing systems, 35:9432, 2022. 2, 8

    2022

  16. [24]

    E. E. Firat, A. Joshi, and R. S. Laramee. Interactive visualization literacy: The state-of-the-art. Information Visualization, 21(3):285–310, 2022. doi: 10.1177/14738716221081831 1

    2022 doi

  17. [25]

    A. R. Fox. Theories and models in graph comprehension. Visualization Psychology, pp. 39–64, 2023. 1

    2023

  18. [26]

    S. L. Franconeri, L. Padilla, P. Shah, J. M. Zacks, and J. Hullman. The science of visual data communication: What works.Psychological Science in the Public Interest, 22(3):110–161, 2021. 1

    2021

  19. [27]

    S. N. Friel, F. R. Curcio, and G. W. Bright. Making sense of graphs: Critical factors influencing comprehension and instructional implications. Journal for Research in Mathematics Education , 32(2):124–158, 2001. doi: 10.2307/749671 1, 2

    2001 doi

  20. [28]

    E. Fry. Graphical literacy. Journal of Reading, 24(5):383–389, 1981. 1

    1981

  21. [29]

    X. Fu, Y . Hu, B. Li, Y . Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pp. 148–166. Springer, 2024. 8

    2024

  22. [30]

    Galesic and R

    M. Galesic and R. Garcia-Retamero. Graph literacy: A cross-cultural comparison. Medical decision making , 31(3):444–457, 2011. doi: 10. 1177/0272989X10373805 2, 8

    2011

  23. [31]

    L. W. Ge, Y . Cui, and M. Kay. Calvi: Critical thinking assessment for literacy in visualizations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, article no. 815, 18 pages. Association for Computing Machinery, New York, NY , USA, 2023...

    2023

  24. [32]

    Geirhos, C

    R. Geirhos, C. R. Temme, J. Rauber, H. H. Schütt, M. Bethge, and F. A. Wichmann. Generalisation in humans and deep neural networks.Advances in neural information processing systems, 31, 2018. 2

    2018

  25. [33]

    D. J. Gillan and R. Lewis. A componential model of human interaction with graphs: 1. Linear regression modeling. Human Factors, 36(3):419– 440, 1994. doi: 10.1177/001872089403600303 1

    1994 doi

  26. [34]

    Gupta, V

    A. Gupta, V . Gupta, S. Zhang, Y . He, N. Zhang, and S. Shah. Enhancing question answering on charts through effective pre-training tasks. arXiv preprint arXiv:2406.10085, 2024. 8

    2024 arXiv

  27. [35]

    Gweon, J

    H. Gweon, J. Fan, and B. Kim. Socially intelligent machines that learn from humans and help humans learn. Philosophical Transactions of the Royal Society A, 381(2251):20220048, 2023. doi: 10.1098/rsta.2022.0048 8

    2023

  28. [36]

    M. Hegarty. Multimedia learning about physical systems. In R. E. Mayer, ed., The Cambridge Handbook of Multimedia Learning , pp. 447–466. Cambridge University Press, Cambridge, UK, 2005. 1

    2005

  29. [37]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 8

    2009 arXiv

  30. [38]

    Holtzman, J

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi. The curious case of neural text degeneration. International Conference on Learning Representations, 2020. 4

    2020

  31. [39]

    Huang, H

    K.-H. Huang, H. P. Chan, Y . R. Fung, H. Qiu, M. Zhou, S. Joty, S.-F. Chang, and H. Ji. From pixels to insights: A survey on automatic chart understanding in the era of large foundation models, 2024. 8

    2024

  32. [40]

    H. Huey. Adaptive Visualization Strategies Across Drawings, Diagrams, and Data Visualizations. University of California, San Diego, 2024. 2, 3

    2024

  33. [41]

    H. Huey, L. A. Oey, H. Lloyd, and J. E. Fan. How do communicative goals guide which data visualizations people think are effective? In Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 45, 2023. 2, 3, 8

    2023

  34. [42]

    D. M. Katz, M. J. Bommarito, S. Gao, and P. Arredondo. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270):20230254, 2024. doi: 10.1098/rsta.2023.0254 1

    2024

  35. [43]

    Kim and J

    Y . Kim and J. Heer. Assessing effects of task and data distribution on the effectiveness of visual encodings. In Computer Graphics Forum, vol. 37(3), pp. 157–167. Wiley Online Library, 2018. doi: 10.1111/cgf.13409 1

    2018 doi

  36. [44]

    Laurençon, L

    H. Laurençon, L. Tronchon, M. Cord, and V . Sanh. What matters when building vision-language models? Advances in Neural Information Processing Systems, 37:87874–87907, 2024. 1, 8

    2024

  37. [45]

    K. Lee, M. Joshi, I. R. Turc, H. Hu, F. Liu, J. M. Eisenschlos, U. Khandelwal, P. Shaw, M.-W. Chang, and K. Toutanova. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. Proceedings of the 40th International Conference on Machine Learning, 202:188...

    2023

  38. [46]

    Lee, S.-H

    S. Lee, S.-H. Kim, and B. C. Kwon. Vlat: Development of a visualization literacy assessment test.IEEE Transactions on Visualization and Computer Graphics, 23(1):551–560, 2017. doi: 10.1109/TVCG.2016.2598920 1, 2, 8

    2017

  39. [47]

    B. Li, Y . Ge, Y . Ge, G. Wang, R. Wang, R. Zhang, and Y . Shan. Seed- bench: Benchmarking multimodal large language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13299–13308, June 2024. 3

    2024

  40. [48]

    J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. Proceedings of the 40th International Conference on Machine Learning, 202:19730–19742, 23–29 Jul 2023. 3

    2023

  41. [49]

    Z. Li, H. Miao, V . Pascucci, and S. Liu. Visualization literacy of multimodal large language models: A comparative study, 2024. 8

    2024

  42. [50]

    F. Liu, F. Piccinno, S. Krichene, C. Pang, K. Lee, M. Joshi, Y . Altun, N. Collier, and J. Eisenschlos. MatCha: Enhancing visual language pretraining with math reasoning and chart derendering. Proceedings of the 61st Annual Meeting of the Association for Computational Linguist...

    2023 doi

  43. [51]

    H. Liu, C. Li, Y . Li, and Y . J. Lee. Improved baselines with visual instruction tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26296–26306, June 2024. 1, 3

    2024

  44. [52]

    Lloyd, H

    H. Lloyd, H. Huey, E. Brockbank, L. Padilla, and J. E. Fan. What is graph comprehension and how do you measure it? In Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 45, 2023. 3

    2023

  45. [53]

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. The Twelfth International Conference on Learning Representations, 2024. 1, 8

    2024

  46. [54]

    Lundgard and A

    A. Lundgard and A. Satyanarayan. Accessible visualization via natural language descriptions: A four-level model of semantic content. IEEE Transactions on Visualization and Computer Graphics, 28(1):1073–1083,

  47. [55]

    Marjieh, I

    R. Marjieh, I. Sucholutsky, P. van Rijn, N. Jacoby, and T. L. Griffiths. Large language models predict human sensory judgments across six modalities. Scientific Reports, 14(1):21445, 2024. 2, 8

    2024

  48. [56]

    Masry, X

    A. Masry, X. L. Do, J. Q. Tan, S. Joty, and E. Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263–2279, May 2022. doi: 10.18653/v1/2022.findings-acl .1...

    2022 doi

  49. [57]

    Methani, P

    N. Methani, P. Ganguly, M. M. Khapra, and P. Kumar. Plotqa: Reasoning over scientific plots. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1527–1536, 2020. 1, 5

    2020

  50. [58]

    Mukherjee, H

    K. Mukherjee, H. Huey, X. Lu, Y . Vinker, R. Aguina-Kang, A. Shamir, and J. Fan. Seva: Leveraging sketches to evaluate alignment between human and machine visual abstraction. Advances in Neural Information Processing Systems, 36:67138–67155, 2023. 2, 8

    2023

  51. [59]

    Mukherjee, D

    K. Mukherjee, D. Ren, D. Moritz, and Y . Assogba. Encqa: Benchmarking vision-language models on visual encodings for charts. arXiv preprint,

  52. [60]

    Mukherjee, T

    K. Mukherjee, T. T. Rogers, and K. B. Schloss. Large language models estimate fine-grained human color-concept associations. arXiv preprint arXiv:2406.17781, 2024. 2, 8

    2024 arXiv

  53. [61]

    T. Munzner. Visualization analysis and design. CRC press, 2014. 1

    2014

  54. [62]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 1, 4

    2023

  55. [63]

    L. M. Padilla, S. H. Creem-Regehr, M. Hegarty, and J. K. Stefanucci. Decision making with visualizations: a cognitive framework across disciplines. Cognitive Research: Principles and Implications, 3(1):29, Jul

  56. [64]

    Peppler, A

    K. Peppler, A. Keune, and A. Han. Cultivating data visualization literacy in museums. Information and Learning Sciences, 122(1/2):1–16, 2021. 8

    2021

  57. [65]

    S. Pinker. A theory of graph comprehension. , pp. 73–126. Artificial intelligence and the future of testing. Lawrence Erlbaum Associates, Inc, Hillsdale, NJ, US, 1990. 1

    1990

  58. [66]

    G. J. Quadri and P. Rosen. A Survey of Perception-Based Visualization Studies by Task. IEEE Transactions on Visualization and Computer Graphics, 28(12):5026–5048, Dec. 2022. Conference Name: IEEE Transactions on Visualization and Computer Graphics. doi: 10. 1109/TVCG.2021.3098240 1

    2022

  59. [67]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learni...

    2021

  60. [68]

    M. D. Rahman, B. Doppalapudi, G. J. Quadri, and P. Rosen. A survey on annotations in information visualization: Empirical insights, applications, and challenges. arXiv preprint arXiv:2410.05579, 2024. 8

    2024 arXiv

  61. [69]

    Rahmanzadehgervi, L

    P. Rahmanzadehgervi, L. Bolton, M. R. Taesiri, and A. T. Nguyen. Vision language models are blind. Proceedings of the Asian Conference on Computer Vision, pp. 18–34, 2024. 8

    2024

  62. [70]

    Renze and E

    M. Renze and E. Guven. The effect of sampling temperature on problem solving in large language models, 2024. 4

    2024

  63. [71]

    Shah and E

    P. Shah and E. G. Freedman. Bar and line graph comprehension: An interaction of top-down and bottom-up processes. Topics in cognitive science, 3(3):560–578, 2011. doi: 10.1111/j.1756-8765.2009.01066.x 1

    2011

  64. [72]

    P. Shah, E. G. Freedman, and I. Vekiri. The comprehension of quantitative information in graphical displays. The Cambridge handbook of visuospatial thinking, pp. 426–476, 2005. 1

    2005

  65. [73]

    Shah and J

    P. Shah and J. Hoeffner. Review of graph comprehension research: Implications for instruction. Educational Psychology Review, 14(1):47–69, Mar 2002. doi: 10.1023/A:1013180410169 1

    2002 doi

  66. [74]

    T. Shu, A. Bhandwaldar, C. Gan, K. Smith, S. Liu, D. Gutfreund, E. Spelke, J. Tenenbaum, and T. Ullman. Agent: A benchmark for core psychological reasoning. In International conference on machine learning, pp. 9614–

  67. [75]

    Simkin and R

    D. Simkin and R. Hastie. An information-processing analysis of graph perception. Journal of the American Statistical Association, 82(398):454– 465, 1987. 1

    1987

  68. [76]

    Stokes, C

    C. Stokes, C. X. Bearfield, and M. A. Hearst. The role of text in visualizations: How annotations shape perceptions of bias and influence predictions. IEEE Transactions on Visualization and Computer Graphics, 30(10):6787–6800, 2023. 8

    2023

  69. [77]

    E. R. Tufte. The visual display of quantitative information, vol. 2. Graphics press Cheshire, CT, 1983. 1

    1983

  70. [78]

    J. W. Tukey et al. Exploratory data analysis, vol. 2. Springer, 1977. 1

    1977

  71. [79]

    B. Tversky. Visualizing thought. Topics in Cognitive Science, 3(3):499– 535, 2011. 1

    2011

  72. [80]

    Z. Wang, M. Xia, L. He, H. Chen, Y . Liu, R. Zhu, K. Liang, X. Wu, H. Liu, S. Malladi, A. Chevalier, S. Arora, and D. Chen. Charxiv: Charting gaps in realistic chart understanding in multimodal llms, 2024. 1, 8

    2024

  73. [81]

    Wilkinson

    L. Wilkinson. The grammar of graphics. Springer, 2012. 1

    2012

  74. [82]

    Y . Wu, L. Yan, L. Shen, Y . Wang, N. Tang, and Y . Luo. Chartinsights: Evaluating multimodal large language models for low-level chart question answering. arXiv preprint arXiv:2405.07001, 2024. 1, 8

    2024 arXiv

  75. [83]

    Z. Xu, S. Du, Y . Qi, C. Xu, C. Yuan, and J. Guo. ChartBench: A Benchmark for Complex Visual Reasoning in Charts, June 2024. arXiv:2312.15915 [cs]. 8

    2024 arXiv

  76. [84]

    Young, B

    A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, K. Yu, P. Liu, Q. Liu, S. Yue, S. Yang, S. Yang, T. Yu, W. Xie, W. Huang, X. Hu, X. Ren, X. Niu, P. Nie, Y . Xu, Y . Liu, Y . Wang, Y . Cai, Z. Gu, Z. Liu, and Z. Dai. Yi: Open foundation...

  77. [85]

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2024

  78. [86]

    X. Zeng, H. Lin, Y . Ye, and W. Zeng. Advancing multimodal large language models in chart question answering with visualization-referenced instruction tuning, 2024. 8

    2024

  79. [87]

    Zhang, S

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. 1 A M ODEL ...

    2022

  80. [1987]

    doi: 10.2307/749086 1

    doi

  81. [2018]

    doi: 10.1186/s41235-018-0120-9 1

    doi

  82. [2019]

    doi: 10.1073/pnas.1807180116 1

    doi

  83. [2022]

    doi: 10.1109/TVCG.2021.3114770 1

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.