Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 7-billion-parameter multimodal model fine-tuned on 2,700 expert visualization critiques matches or beats much larger models at judging chart quality.

desk verdict A useful dataset and a clear pipeline, but the reference-relative critique labels undermine the claim that the trained critic works without a reference. read the letter →

arxiv 2506.13326 v1 pith:F5EACDXC submitted 2025-06-16 cs.CV cs.HC

Bo Pan , Yixiao Fu , Ke Wang , Junyu Lu , Lunke Pan , Ziyang Qian , Yuhan Chen , Guoliang Wang
show 10 more authors
Yitao Zhou Li Zheng Yinghao Tang Zhen Wen Yuchen Wu Junhua Lu Biao Zhu Minfeng Zhu Bo Zhang Wei Chen
This is my paper · ORCID
classification cs.CVcs.HC
keywords datavisualizationmultimodalLLMcritiquemodelfeedbackgenerationchartqualityevaluationfine-tuningLLM-as-judgeobservable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the bottleneck in LLM-based data visualization is not model size but the availability of high-quality feedback. It builds a dataset-construction pipeline that turns 1,700 curated human-made visualizations into 2,700 expert critiques of LLM-generated charts, then fine-tunes a small 7-billion-parameter multimodal model on them. The paper claims this small critic reaches the critique quality of models ten times larger and of proprietary frontier models. If true, automated visualization critique can be made cheap and open, enabling iterative improvement of chart generation without human intervention.

What carries the argument

The load-bearing artifact is the critique dataset construction framework that yields (instruction, dataset, human visualization, LLM visualization, critique) quintuplets, together with the four-category defect taxonomy used by annotators. Fine-tuning on these pairs teaches the base model to look for the same defects an expert would name (truncated legends, unclear encodings, missed instruction requirements) rather than produce fluent but generic praise or vague suggestions. The ablation showing monotonic improvement with data quantity is what links the dataset to the performance claim.

What would settle it

Take 200 charts generated by a modern LLM from a distribution not covered by Observable (e.g., news infographics or Excel exports), have visualization experts mark ground-truth defects, and run VIS-Shepherd in a blind head-to-head against the 7B base model. If its win rate is not above chance, the critique skill does not transfer beyond the training distribution.

Watch

Extended reading notes

Core claim

The discovery claim is that a specialized visualization critic does not require a frontier-scale model: a 7-billion-parameter multimodal language model, fine-tuned for one epoch on 2.5k expert-written critiques, produces feedback that human evaluators rate on par with or above much larger open models and close to a leading proprietary model. The route to that result is a four-stage dataset pipeline: curating 1.7k human-made visualizations, synthesizing user instructions and exporting their datasets, generating flawed LLM versions with two strong generators, and having visualization experts annotate the LLM versions against the human reference using a fixed defect taxonomy (instruction compliance, visual clarity, semantic readability, or no-defect suggestions). The same data supports an ablation: critique quality rises monotonically with the amount of high-quality critique examples used for training.

Load-bearing premise

The curated 1.7k human visualizations and the 2.7k expert critiques are representative of the real-world patterns and defect types that visualization generators will encounter.

Editorial extensions

If this is right

  • VIS-Shepherd can slot into LLM visualization pipelines as an automated critic, giving concrete fixable feedback without human intervention.
  • Small open models can substitute for expensive proprietary judges in visualization quality evaluation, lowering cost and allowing private deployment.
  • The monotonic data-ablation result implies that further scaling the critique corpus should yield further gains, at least within this task.
  • Because the model is trained to spot defects it can name, its feedback is actionable for downstream code repair rather than a scalar score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same four-stage recipe could be applied to other visually grounded generation tasks (maps, data comics, slide design) where human quality judgments are scarce.
  • An implied but untested use is to feed VIS-Shepherd's critiques as a reward signal for reinforcement learning on visualization generators, closing the loop the paper only evaluates in open-loop mode.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces VIS-Shepherd, a 7B-parameter multimodal LLM (Qwen-2.5-VL-7B) fine-tuned to critique LLM-generated data visualizations. The authors build a dataset by curating 1.7K high-quality human-created visualizations from Observable, synthesizing instructions and datasets, generating corresponding LLM visualizations with GPT-4o and Claude-Sonnet-3.5, and collecting 2.7K critiques from visualization experts who compare the LLM output against the human reference. The model is evaluated with a GPT-4o-based automatic judge on a 0.16K test set and with a human preference study against GPT-4o, Llama-4-Maverick, Qwen-2.5-VL-72B, and the untuned base model. The central claim is that the fine-tuned 7B model reaches critique quality comparable to much larger open-source and proprietary models.

Significance. If the result holds, the paper makes a useful data-centric contribution: it shows that a relatively small, specialized MLLM trained on a modest set of expert critiques can approach or match much larger generalist models for visualization critique. The dataset-construction pipeline is concrete and the authors provide both an automatic evaluation and a human preference study, which is stronger than relying on a single metric. The qualitative example in Figure 5 is compelling and illustrates a real failure mode of general-purpose MLLMs. The paper does not contain mathematical derivations, but it does ship a reproducible pipeline and a public repository link. The main reservation is that the critique labels, and therefore the learned model, are reference-relative while the deployment scenario is reference-free; this mismatch needs to be addressed before the generalization claim can be accepted.

major comments (3)
  1. [Sec. 3.4 and Fig. 1] The training and test critiques are reference-relative, while deployment is reference-free. In Stage 4, annotators are explicitly asked to compare V_LLM against the human-created V_h, and GPT-4o is used to generate reference suggestions by comparing the two; the 0.16K test set is produced by the same protocol. At inference (Fig. 1), VIS-Shepherd receives only the instruction, dataset, and rendered candidate, with no V_h. This means that a defect present in both V_h and V_LLM is systematically labeled as non-defective, and a valid alternative visualization that differs from the idiosyncratic human design can be labeled defective. The reported automatic scores and human-preference parity therefore measure performance on the reference-relative training task, not on the reference-free task the deployed model is asked to perform. Please provide a reference-free evaluation, for example human ratings of critiques where evaluators see only I, D, and the rendered candidate, or a re-annotation of the test set without V_h; without this, the abstract's generalization claim is not supported.
  2. [Sec. 5.2 and Fig. 4(a)] The automatic judge is GPT-4o, which is also one of the compared baselines. The judge receives a human ground-truth critique that was annotated in the reference-relative manner, and VIS-Shepherd is trained to imitate exactly that annotation style, so the judge may systematically favor VIS-Shepherd. The human preference study is the intended mitigation, but Section 5.3 and Figure 4(b) do not report the number of annotators, the number of comparisons per model pair, inter-annotator agreement, or whether the human evaluators had access to the human reference visualization. Please report these details and, if possible, use an independent judge or human adjudication of the automatic scores to remove the baseline-judge overlap.
  3. [Sec. 6.1 and Fig. 4(a)] The reported results contain an internal inconsistency and lack statistical grounding. The text states that VIS-Shepherd performs 'better than [Llama-4-Maverick] but slightly below [GPT-4o]' in the automated evaluation, but Figure 4(a) reports a mean Likert score of 3.41 for VIS-Shepherd versus 2.98 for GPT-4o and high-score rates of 72.0% versus 67.1%, which imply VIS-Shepherd outperforms GPT-4o. Moreover, the automatic evaluation is based on only 160 critiques, and the human preference win rates in Figure 4(b) (e.g., 50.6% vs. 28.1% ties against Qwen-2.5-VL-72B) are reported without confidence intervals or significance tests. Please correct the text/figure inconsistency and report score distributions, standard errors, and paired significance tests (e.g., Wilcoxon signed-rank for the Likert scores and a binomial or chi-square test for the preference counts).
minor comments (5)
  1. [Fig. 4 caption] The caption contains a duplicated prefix: 'Figure 4: Figure 4: Comparison...' should be 'Figure 4: Comparison...'.
  2. [Sec. 6.3 and Fig. 6] The x-axis labels in Figure 6 are ambiguous: '7B' could be the untuned base model or a model trained with the full dataset; clarify the notation (e.g., 'Qwen-2.5-VL-7B (untuned)' versus 'VIS-Shepherd (2.5k)').
  3. [Sec. 7] The limitations section does not mention the reference-relative nature of the annotation protocol or the domain gap between the curated Observable D3 examples and typical production visualization tools; both are important scope restrictions and should be acknowledged explicitly.
  4. [Sec. 5.1] The description of Qwen-2.5-VL-7B says it is 'pre-trained with 1 million synthetic chart-type samples,' which refers to the base model's own pretraining rather than the fine-tuning introduced in this paper; rephrase to avoid implying that this pretraining is part of the proposed method.
  5. [Sec. 3.4 and Sec. 6.1] The notation '0.16k' should be written as 160 for consistency and clarity, since the paper otherwise uses absolute counts such as 1.7K and 2.7K.

Circularity Check

2 steps flagged · score 3.0 of 10

Critique labels are reference-relative by construction (Sec. 3.4) while deployment is reference-free (Fig. 1), so automated and human-preference scores measure the constructed distribution; GPT-4o also serves as data-constructor, judge, and baseline, yielding a partial, not total, circularity.

  1. other [Section 3.4 (Stage 4) and Section 5.2 (model-based evaluation); cf. Figure 1 deployment]
    "we recruited annotators with visualization expertise to carefully identify defects and provide critiques of the LLM-generated visualizations by comparing them against their human-created counterparts, resulting in 2.7k high-quality visualization critiques."

    By the construction protocol in Section 3.4, every critique label is the annotator-reported difference between V_LLM and the human-created reference Vh. The same protocol produced the 0.16k test critiques, and Figure 2's rubric scores candidate feedback only against these reference-relative annotations. The reported gains ('reaching levels comparable to much larger open-source or even proprietary models') thus measure agreement with the label-construction task. At deployment (Figure 1) the critic receives only instruction, dataset, and rendered candidate—Vh is absent—so defects shared by Vh and V_LLM are never labeled and the skill need not transfer to the reference-free task.

  2. other [Section 5.1-5.2 versus Sections 3.1, 3.3, and 3.4]
    "We employed a similar approach by prompting GPT-4o to automatically evaluate the quality of feedback provided by different models against human critique annotations in our test set, using a 5-point Likert scale."

    GPT-4o appears on both sides of the evaluation pipeline: it filters training instances (Sec. 3.1), generates the V_LLM candidates (Sec. 3.3), supplies the reference suggestions that seed the human critiques (Sec. 3.4: 'we used GPT-4o to automatically compare the LLM-generated visualization ... generating potential preferential suggestions for annotators to reference'), and then judges the same-style critiques while also being a headline baseline (Secs. 5.1-5.2). Because the judge scores against human ground truth rather than its own outputs, and because the human preference study independently reproduces the ranking, this overlap skews but does not force the result; it is evaluator bias, not a reduction of the derivation to its inputs.

full rationale

VIS-Shepherd contains no mathematical derivation and no fitted parameter that is then renamed as a prediction: the 2.5k training critiques and 0.16k test critiques are disjoint, Likert scores compare candidate feedback to human ground truth, and a separate human preference study (Fig. 4b) independently confirms the ranking over Qwen-2.5-VL-72B, Llama-4-Maverick, and GPT-4o. Measured against the strict circularity patterns—self-definitional equations, fitted inputs called predictions, load-bearing self-citations, imported uniqueness theorems, ansatz smuggling, renaming known results—the paper is largely clean; the only self-citation ([16]) is an incidental related-work reference. Two concerns keep the score above zero. First, the critique labels are reference-relative by design: annotators 'identify defects and provide critiques of the LLM-generated visualizations by comparing them against their human-created counterparts' (Sec. 1/3.4), and the automated evaluation re-uses exactly those labels, so the headline parity claim is measured on the label-construction protocol while deployment (Fig. 1) supplies no human reference; this is a partial circularity of evaluation task, not a fabrication of the held-out scores. Second, GPT-4o straddles data construction, judging, and baselining, though scoring against human annotations and the human-preference confirmation bound its influence. Under proportionality, these are construct-validity and evaluator-overlap issues rather than a derivation that reduces to its own inputs, supporting a moderate score of 3.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality and representativeness of the curated dataset and on the reliability of human and LLM-based evaluation. These are domain assumptions, not mathematical axioms.

assumptions (4)
  • domain assumption The 1,700 human-created visualization instances curated from Observable after MLLM and human filtering are of sufficiently high quality and diversity to serve as ground-truth references.
    Section 3.1 relies on this to build the dataset; if wrong, critiques are anchored to biased references.
  • domain assumption The synthesized user instructions (Stage 2) plausibly represent real user queries for visualization generation.
    Section 3.2 uses Claude-3.5-Sonnet role-playing to generate instructions; if these are unrepresentative, downstream LLM-generated visualizations will not reflect real defects.
  • domain assumption Visualization experts can reliably identify defects and write critiques consistent with the taxonomy in Table 1.
    Section 3.4 relies on 10 annotators; if their judgments are inconsistent, the ground-truth critiques are noisy.
  • domain assumption GPT-4o's automatic scoring (Section 5.2) correlates with human judgment of critique quality.
    The paper manually inspects outputs to tune the prompt, but no quantitative inter-rater agreement is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generation." pith.science (2026). https://pith.science/paper/F5EACDXC

@misc{pith2026250613326,
  author       = {Pith},
  title        = {Pith review of: VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F5EACDXC}},
  note         = {Machine review of arXiv:2506.13326}
}
read the original abstract

Data visualization generation using Large Language Models (LLMs) has shown promising results but often produces suboptimal visualizations that require human intervention for improvement. In this work, we introduce VIS-Shepherd, a specialized Multimodal Large Language Model (MLLM)-based critic to evaluate and provide feedback for LLM-generated data visualizations. At the core of our approach is a framework to construct a high-quality visualization critique dataset, where we collect human-created visualization instances, synthesize corresponding LLM-generated instances, and construct high-quality critiques. We conduct both model-based automatic evaluation and human preference studies to evaluate the effectiveness of our approach. Our experiments show that even small (7B parameters) open-source MLLM models achieve substantial performance gains by leveraging our high-quality visualization critique dataset, reaching levels comparable to much larger open-source or even proprietary models. Our work demonstrates significant potential for MLLM-based automated visualization critique and indicates promising directions for enhancing LLM-based data visualization generation. Our project page: https://github.com/bopan3/VIS-Shepherd.

Figures

Figures reproduced from arXiv: 2506.13326 by the authors.

Figure 1
Figure 1. Overview of the VIS-Shepherd framework. The diagram illustrates the end-to-end visual [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Instruction and scoring rubric for model-based automatic evaluation of visualization [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Instruction for human preference evaluation of visualization feedback. Evaluators are [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Figure 4: Comparison of VIS-Shepherd against baseline models. (a) Automated evaluation [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison of model critiques for an MA plot. GPT-4o and Qwen-2.5-7B (our [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Ablation study on training data quantity. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Screenshots of the annotation system for manual high-quality visualization selection. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Examples of rule-based dataset previews used to help the LLM understand the data structure. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Screenshot of the annotation system for critique collection. The interface displays the [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Spatial Visual Analytics for Multi-Document Summary Verification

    cs.HC 2026-07 conditional novelty 6.0 of 10

    Placing source documents by how they align with summary sentences improves people's ability to verify multi-document AI summaries, outperforming a linear list.

  2. AI for Auto-Research: Roadmap & User Guide

    cs.AI 2026-05 unverdicted novelty 4.0 of 10

    The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.

Reference graph

Works this paper leans on

52 extracted references · 35 canonical work pages · cited by 2 Pith papers

  1. [1]

    Semiology of graphics

    Jacques Bertin. Semiology of graphics. University of Wisconsin Press, 1983

    work page 1983

  2. [2]

    The grammar of graphics

    Leland Wilkinson. The grammar of graphics. InHandbook of computational statistics: Concepts and methods, pages 375–414. Springer, 2011

    work page 2011

  3. [3]

    Visualization analysis and design

    Tamara Munzner. Visualization analysis and design. CRC press, 2014

    work page 2014

  4. [4]

    Matplotagent: Method and evaluation for llm-based agentic scientific data visualization

    Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, et al. Matplotagent: Method and evaluation for llm-based agentic scientific data visualization. In Findings of the Association for Computational Linguistics ACL 2024, pages 11789–11804, 2024

    work page 2024

  5. [5]

    LIDA: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models

    Victor Dibia. LIDA: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models. In Danushka Bollegala, Ruihong Huang, and Alan Ritter, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 113–126, Toronto, Canada, July ...

    work page 2023

  6. [6]

    Chartllama: A multimodal llm for chart understanding and generation

    Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A multimodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483, 2023

    arXiv 2023

  7. [7]

    Visualization generation with large language models: An evaluation, 2024

    Guozheng Li, Xinyu Wang, Gerile Aodeng, Shunyuan Zheng, Yu Zhang, Chuangxin Ou, Song Wang, and Chi Harold Liu. Visualization generation with large language models: An evaluation, 2024. 10

    work page 2024

  8. [8]

    Viseval: A benchmark for data visualization in the era of large language models

    Nan Chen, Yuge Zhang, Jiahang Xu, Kan Ren, and Yuqing Yang. Viseval: A benchmark for data visualization in the era of large language models. IEEE Transactions on Visualization and Computer Graphics, 31(1):1301–1311, January 2025

    work page 2025

Show all 52 references
  1. [9]

    Shep- herd: A critic for language model generation, 2023

    Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O’Brien, Ramakanth Pasunuru, Jane Dwivedi- Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Shep- herd: A critic for language model generation, 2023

    2023

  2. [10]

    Ultrafeedback: boosting language models with scaled ai feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: boosting language models with scaled ai feedback. In Proceedings of the 41st International Conference on Machine L...

    2024

  3. [11]

    CritiqueLLM: Towards an informative critique generation model for evaluation of large language model generation

    Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. CritiqueLLM: Towards an informative critique generation model for evaluation of large language model generation. In Lun-Wei Ku,...

    2024

  4. [12]

    Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeti...

    2024

  5. [13]

    Sahraoui

    Martin Weyssow, Aton Kamanda, and Houari A. Sahraoui. Codeultrafeedback: An llm-as-a- judge dataset for aligning large language models to coding preferences. ArXiv, abs/2403.09032, 2024

    2024 arXiv

  6. [14]

    Med-code: Medical critique based disagree- ment evaluation framework, 2025

    Mohit Gupta, Akiko Aizawa, and Rajiv Ratn Shah. Med-code: Medical critique based disagree- ment evaluation framework, 2025

    2025

  7. [15]

    Chat2vis: Generating data visualizations via natural language using chatgpt, codex and gpt-3 large language models

    Paula Maddigan and Teo Susnjak. Chat2vis: Generating data visualizations via natural language using chatgpt, codex and gpt-3 large language models. Ieee Access, 11:45181–45193, 2023

    2023

  8. [16]

    Exploring multimodal prompt for visualization authoring with large language models

    Zhen Wen, Luoxuan Weng, Yinghao Tang, Runjin Zhang, Yuxin Liu, Bo Pan, Minfeng Zhu, and Wei Chen. Exploring multimodal prompt for visualization authoring with large language models. arXiv preprint arXiv:2504.13700, 2025

    2025 arXiv

  9. [17]

    Chartgpt: Leveraging llms to generate charts from abstract natural language

    Yuan Tian, Weiwei Cui, Dazhen Deng, Xinjing Yi, Yurun Yang, Haidong Zhang, and Yingcai Wu. Chartgpt: Leveraging llms to generate charts from abstract natural language. IEEE Transactions on Visualization and Computer Graphics, 2024

    2024

  10. [18]

    Vispath: Auto- mated visualization code synthesis via multi-path reasoning and feedback-driven optimization

    Wonduk Seo, Seungyong Lee, Daye Kang, Zonghao Yuan, and Seunghyun Lee. Vispath: Auto- mated visualization code synthesis via multi-path reasoning and feedback-driven optimization. arXiv preprint arXiv:2502.11140, 2025

    2025 arXiv

  11. [19]

    nvagent: Automated data visualization from natural language via collaborative agent workflow

    Geliang Ouyang, Jingyao Chen, Zhihe Nie, Yi Gui, Yao Wan, Hongyu Zhang, and Dongping Chen. nvagent: Automated data visualization from natural language via collaborative agent workflow. arXiv preprint arXiv:2502.05036, 2025

    2025 arXiv

  12. [20]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems , 36:46534–46594, 2023

    2023

  13. [21]

    G-eval: Nlg evaluation using gpt-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023. 11

    2023 arXiv

  14. [22]

    Roscoe: A suite of metrics for scoring step-by-step reasoning

    Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Roscoe: A suite of metrics for scoring step-by-step reasoning. arXiv preprint arXiv:2212.07919, 2022

    2022 arXiv

  15. [23]

    Generating sequences by learning to self-correct

    Welleck Sean, Ximing Lu, West Peter, Brahman Faeze, Shen Tianxiao, Khashabi Daniel, and Choi Yejin. Generating sequences by learning to self-correct. In Proceedings of The 11th International Conference on Learning Representations (ICLR), 2023

    2023

  16. [24]

    Criticbench: Benchmarking llms for critique-correct reasoning

    Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. Criticbench: Benchmarking llms for critique-correct reasoning. arXiv preprint arXiv:2402.14809, 2024

    2024 arXiv

  17. [25]

    Text-to-viz: Automatic generation of infographics from proportion-related natural language statements

    Weiwei Cui, Xiaoyu Zhang, Yun Wang, He Huang, Bei Chen, Lei Fang, Haidong Zhang, Jian-Guan Lou, and Dongmei Zhang. Text-to-viz: Automatic generation of infographics from proportion-related natural language statements. IEEE transactions on visualization and computer graphics, 2...

    2019

  18. [26]

    Collecting and characterizing natural language utterances for specifying data visualizations

    Arjun Srinivasan, Nikhila Nyapathy, Bongshin Lee, Steven M Drucker, and John Stasko. Collecting and characterizing natural language utterances for specifying data visualizations. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–10, 2021

    2021

  19. [27]

    nvbench: A large-scale synthesized dataset for cross-domain natural language to visualization task

    Yuyu Luo, Jiawei Tang, and Guoliang Li. nvbench: A large-scale synthesized dataset for cross-domain natural language to visualization task. arXiv preprint arXiv:2112.12926, 2021

    2021 arXiv

  20. [28]

    nvbench 2.0: A benchmark for natural language to visualization under ambiguity

    Tianqi Luo, Chuhan Huang, Leixian Shen, Boyan Li, Shuyu Shen, Wei Zeng, Nan Tang, and Yuyu Luo. nvbench 2.0: A benchmark for natural language to visualization under ambiguity. arXiv preprint arXiv:2503.12880, 2025

    2025

  21. [29]

    Gemini: A family of highly capable multimodal models

    Gemini Team. Gemini: A family of highly capable multimodal models. 2023

    2023

  22. [30]

    Simhash: Hash-based similarity detection, 2007

    Caitlin Sadowski and Greg Levin. Simhash: Hash-based similarity detection, 2007

    2007

  23. [31]

    Scaling synthetic data creation with 1,000,000,000 personas

    Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094, 2024

    2024 arXiv

  24. [32]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Lu...

    2025

  25. [33]

    Tidy data

    Hadley Wickham. Tidy data. Journal of statistical software, 59:1–23, 2014

    2014

  26. [34]

    Gpt-4o system card, 2024

    OpenAI Team. Gpt-4o system card, 2024

    2024

  27. [35]

    Introducing claude 3.5 sonnet, 2024

    Anthropic. Introducing claude 3.5 sonnet, 2024

    2024

  28. [36]

    Declarative language design for interactive visualization

    Jeffrey Heer and Michael Bostock. Declarative language design for interactive visualization. IEEE Transactions on Visualization and Computer Graphics , 16(6):1149–1156, November 2010

    2010

  29. [37]

    D 3 data-driven documents

    Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. D 3 data-driven documents. IEEE transactions on visualization and computer graphics, 17(12):2301–2309, 2011

    2011

  30. [38]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  31. [39]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations

  32. [40]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024. 12

    2024 arXiv

  33. [41]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  34. [42]

    Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-yeol Ahn. Can we trust the evaluation on ChatGPT? In Anaelia Ovalle, Kai-Wei Chang, Ninareh Mehrabi, Yada Pruksachatkun, Aram Galystan, Jwala Dhamala, Apurv Verma, Trista Cao, Anoop Kumar, and Rahul Gupta, editors, Proceedings ...

    2023

  35. [43]

    the rendered content is not a visualization,

    Jian Chen, Petra Isenberg, Robert S. Laramee, Tobias Isenberg, Michael Sedlmair, Torsten Moeller, and Rui Li. An image-based typology for visualization, 2025. 13 A Appendix A.1 Source Code and Dataset Please check the dataset and the source code for training and evaluating VIS...

    2025

  36. [44]

    Examples: bar charts, stacked bar charts, box plots, or sunburst diagrams

    Generalized Bar Representations Label: Bar Description: Graphs that represent data with straight bars that can be arranged on a straight or curved baseline and whose heights or lengths are proportional to the values they represent. Examples: bar charts, stacked bar charts, box...

  37. [45]

    These locations are often shown using dots or circles, but also other shapes such as 3D spheres, triangles, stars, etc

    Point-based Representations Label: Point Description: Representations that use point locations. These locations are often shown using dots or circles, but also other shapes such as 3D spheres, triangles, stars, etc. Examples: scatterplots, point clouds, dot plots, or bubble charts

  38. [46]

    point" type and (b) candidate visualizations of the

    Line-based Representations Label: Line 15 (a) (b) Figure 7: Screenshots of the annotation system for manual high-quality visualization selection. (a) Candidate visualizations of the "point" type and (b) candidate visualizations of the "grid" type. Visualizations selected by an...

  39. [47]

    Examples: node-link diagrams, node-link trees, node-link graphs, meshes, arc diagrams, or Sankey diagrams

    Node-link Trees/Graphs, Networks, Meshes Label: Node-link Description: Representations using points for and explicit connections between these points to convey relationships between data values. Examples: node-link diagrams, node-link trees, node-link graphs, meshes, arc diagr...

  40. [48]

    Areas can be geographical regions or polygons whose size or shape represents abstract data

    Area-based Representations Label: Area Description: Representations with a focus on areas of 2D space or 2D surfaces including sub-sets of these surfaces. Areas can be geographical regions or polygons whose size or shape represents abstract data. Examples: (stacked) area chart...

  41. [49]

    The grid often has rectangular cells but may also use other shapes such as hexagons or cubes

    Generalized Matrix / Grid Label: Grid Description: Representations that separate data into a discrete spatial grid structure. The grid often has rectangular cells but may also use other shapes such as hexagons or cubes. Elements such as glyphs or a color encoding can appear in...

  42. [50]

    These patterns can be evoked by changes in intensity, changes in hue, brightness, and/or saturation

    Continuous Color and Grey-scale, and Textures Label: Continuous-ColorPattern Description: Representations of structured patterns across an image or atop a geometric 3D object. These patterns can be evoked by changes in intensity, changes in hue, brightness, and/or saturation. ...

  43. [51]

    Placement is usually meaningful and typically multiple glyphs are displayed for comparison

    Glyph-based Representations Label: Glyph Description: Multiple small independent visual representations (often encoded by position and additional dimensions using color, shape, or other geometric primitives) that depict multiple attributes (dimensions) of a data record. Placem...

  44. [52]

    Label":

    Text-based Representations Label: Text Description: Representations of data (often text itself) that use varying properties of letters/words such as font size, color, width, style, or type to encode data. Examples: Tag clouds, word trees, parallel tag clouds, typomaps. ## Outp...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.