Pith. sign in

REVIEW 2 major objections 5 minor 85 references

How Do Researchers Manage Visualization Experiment Stimuli?

T0 review · 2 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Visualization experiment stimulus pain reduces to two unmet objectives: thoroughness and controllability.

desk verdict A well-executed interview study that gives the visualization-experiments subfield its first real map of stimulus-management practices; the sample skew is real but acknowledged and does not sink the central claim. read the letter →

arxiv 2607.26443 v1 pith:IUXKQTIP submitted 2026-07-29 cs.HC

Hyeok Kim , Jeffrey Heer This is my paper · ORCID
classification cs.HC
keywords visualizationexperimentsexperimentstimuliinterviewstudythematicanalysishuman-centeredcomputingAI-assisteddesignthoroughnessandcontrollabilitycrowd-sourcingstudies
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the everyday pain of visualization experiments—finding, creating, checking, and shipping the charts and visual materials participants see—is not a random pile of complaints but a structured problem. Through 45-minute interviews with 19 visualization researchers of different seniority, it argues that nearly all reported challenges reduce to two objectives researchers are trying to satisfy: thoroughness (stimuli that cover the breadth and depth of the design space) and controllability (full, fine-grained command over what participants see). Four kinds of constraints—labor, skill, uncertainty, and what simply cannot be fixed—get in the way. A sympathetic reader would care because naming the constraints turns an invisible craft problem into an engineering agenda: automated stimuli creation, guaranteed shipment verification, simulated inspection responses, and cross-device support. If the taxonomy holds, it changes what tool builders should prioritize and gives the field a shared vocabulary for an understudied stage of empirical work.

What carries the argument

The load-bearing object is the life-cycle model of stimuli (seed idea, exploration, selection, shipment, deployment, analysis) combined with a two-axis objective space: thoroughness and controllability. Each axis is frustrated by four named constraint classes—labor, skill, uncertainty, and unfixable constraints—and the paper uses this matrix to organize findings, tactics, and future research opportunities. The mechanism that carries the argument is the mapping from interview quotes to cells of this matrix; it provides the vocabulary for why an AI feature like automated shipment checking is needed.

What would settle it

A direct check would be a broader survey or diary study of visualization experimenters, independent of this interview protocol, testing whether shipment inspection and condition-assignment checking actually rank among their top bottlenecks; if most researchers report no such labor in their workflows, the controllability constraint story is not general. A stronger falsifier is a field deployment of an automated shipment-verification tool that still lets real condition-assignment bugs reach participants at the same rate as manual practice—showing that the perceived bottleneck is not the binding

Watch

Extended reading notes

Core claim

The central claim is that visualization researchers' challenges with experiment stimuli, from the first seed idea through exploration, selection, shipment, deployment, and analysis, can be characterized as constraints that prevent them from fulfilling two objectives: thoroughness (ensuring stimuli cover the design space in breadth and depth) and controllability (having full and fine control over stimuli and the study apparatus). The authors derive this taxonomy from thematic analysis of 19 semi-structured interviews, then map each challenge to one of four constraint types—labor, skill, uncertainty, or un-fixability—and show how researchers cope through tactics like running pilots early, fixi

Load-bearing premise

The load-bearing premise is that 19 self-selected interviewees, mostly experienced with online crowd-sourcing studies and skewed toward junior and early-career researchers, are representative enough that their self-reported practices and challenge taxonomy generalize to visualization experimenters at large.

Editorial extensions

If this is right

  • Stimuli-creation tools should support the whole pipeline—explore, select, verify, ship—rather than one-shot generation of charts from a hypothesis.
  • Systems need version control, design-change propagation, and overview dashboards, e.g., for adding or removing conditions without starting from scratch.
  • Automated inspection could be hybrid: ML-based checks of full stimulus sequences plus a minimum guaranteed manually inspected subset.
  • Response simulation is useful for pre- or post-study sanity checks (demographic-specific or normative scenarios), not as a replacement for human participants.
  • Cross-device support, preferably using perceptual units like visual angle, is a concrete way to address controllability over participants' varied apparatuses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the taxonomy generalizes, tool builders could prioritize shipment verification first, since it is the most labor-scalable and concrete failure point; this is my inference, not a ranking stated in the paper.
  • The recurring strategies 'analysis as guidelines' and 'close the loop ASAP' suggest a testable training intervention: checklist-based scaffolding that makes the lifecycle visible to novice experimenters might reduce the apprenticeship bottleneck.
  • The paper's boundary condition for response simulation—accepted for parameterization and planning, rejected for formal inference—could be probed as models improve; a follow-up study could see whether that boundary is stable or shifts.
  • The authors flag their sample skews toward online crowd-sourcing and junior/early-career researchers; a complementary interview set emphasizing in-person, deployment-heavy, or senior-led studies might surface different bottleneck distributions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper reports a qualitative interview study of 19 visualization researchers about their practices, challenges, and strategies in creating, managing, shipping, and deploying experiment stimuli. It contributes (i) a life-cycle model of stimuli work (seed idea, exploration, selection, shipment, deployment, analysis; Fig. 1A), (ii) a taxonomy characterizing challenges as constraints (labor, skill, uncertainty, unfixability) that prevent researchers from achieving two objectives (thoroughness and controllability; Fig. 1B), (iii) a set of recurring strategies and low-level tactics (Table 2), and (iv) an account of researchers' concerns and hopes regarding AI-assisted experiment design. On this basis, the discussion proposes research opportunities in stimuli creation and management, automated inspection, response simulation, cross-apparatus support, and scaffolding. The study used pilot-tested, 45-minute semi-structured interviews and thematic analysis following Braun and Clarke, with direct quotes tied to participant IDs.

Significance. Structured, evidence-based accounts of the day-to-day work of constructing visualization experiment stimuli are scarce; existing resources cover reporting checklists, platform support, and methodology frameworks, but not the lived practices of stimuli management. If the taxonomy holds, it gives the community a useful vocabulary and a grounded, falsifiable agenda for tool building. The paper's strengths include transparent recruitment and demographic reporting (Table 1), a pilot-tested protocol, quotes tied to participant IDs throughout, an explicit scope statement in Section 1, an honest Limitations paragraph, and supplementary documentation of the coding evolution. The sample of 19 is appropriate for thematic analysis, and the second-order reorganization of codes (Section 3.3) is disclosed rather than hidden. The main risk is scope: the abstract's phrasing generalizes beyond the sample's acknowledged skew toward online crowd-sourcing, junior researchers, and a positivist paradigm, but this is fixable with qualifiers rather than being a fundamental flaw.

major comments (2)
  1. [Abstract; Section 1; Section 6; Section 3.1] The abstract and title describe visualization researchers' challenges and practices broadly, but Section 1 scopes the study to online, crowd-sourcing experiments, and the Limitations paragraph (Section 6) concedes that participants tended to have experience in online crowd-sourcing, that a large portion were junior and early-career researchers, and that most designed experiments based on a positivist paradigm. Recruitment via Slack, LinkedIn, Bluesky, and direct contact (Section 3.1) adds a further selection channel toward professionally networked, tooling-aware researchers that the Limitations does not discuss. This mismatch between the stated scope and the sampled population affects the central characterization and the Section 5 agenda. Please add the sample qualifier to the abstract and temper the Section 5 claims accordingly, or provide evidence that the sample covers other segments.
  2. [Sections 4.4.1, 4.4.2] The text uses universal quantifiers in two findings: 'Our interviewees unanimously wanted to take the control' (Section 4.4.1) and 'all of our interviewees noted that they would never replace human responses with synthetic ones' (Section 4.4.2). For a thematic analysis, these are strong claims that need counts or a clear data basis; otherwise, they should be revised to 'nearly all' or accompanied by supporting counts from the codebook.
minor comments (5)
  1. [Figure 1] Typo in panel B: 'Particpants' should be 'Participants' cognitive capacity. The constraint labels are also quite small; consider increasing the figure's font size.
  2. [Table 1 caption] Typo: 'Y ears.' should be 'Years.'
  3. [Table 2] The column semantics are unclear: the lifecycle-stage columns (Overall, Exp., Sel., Ship., Depl., Anal.) appear to map to participant-ID numbers, but the numbers are shown only under 'Overall.' Please clarify the mapping, e.g., with a separate 'Participants' column and explicit stage indicators, so the table is readable without guessing.
  4. [Section 2.1; reference [16]] Spelling inconsistency: the text refers to 'Elliot et al.' while the reference list has 'Elliott, M. A.' Please standardize.
  5. [Section 3.3; Supplementary Materials] The final taxonomy is the product of a second-order reorganization of initial codes. The main text discloses this and the supplementary documents the evolution, but a compact code-to-theme mapping table would let readers audit whether the taxonomy captures or distorts the codes. Consider adding one to the supplementary material.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: interview-based findings are self-contained; background self-citations are not load-bearing.

full rationale

This paper is a qualitative interview study, not a mathematical derivation or model-fitting exercise. The central findings—challenges and practices across the stimuli life cycle—are generated from thematic analysis of 19 semi-structured interviews (Sections 3.1, 3.3), not from a fitted parameter or a prior result by the same authors. The thoroughness/controllability and labor/skill/uncertainty/unfixable taxonomy is presented as an interpretive characterization of the interview data, not as a prediction derived from an external formal system. Self-citations such as Kim & Heer [34], ReVISit [14, 51], Vega-Lite [65], and related systems appear only in related work, background, or discussion of future opportunities; they are not used to justify the empirical findings, and interviewees explicitly criticize some of these tools (e.g., P10's comments on ReVISit). No uniqueness theorem, ansatz, or fitted quantity is imported or renamed. The limitations paragraph candidly acknowledges sample skew, which is a validity limitation rather than circularity. There is no step where an output is equivalent to an input by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 2 invented entities

No free parameters or fitted values. The paper's contribution is interpretive, so the central claim rests on the representativeness of 19 interviewees and on the validity of two paper-introduced conceptual structures: the stimulus lifecycle and the thoroughness/controllability taxonomy. Both are plausible and grounded in quotes, but they lack external validation.

assumptions (3)
  • domain assumption Interviewee self-reports are a valid proxy for actual practices
    All findings come from 45-minute semi-structured interviews (Section 3.2); there is no observation of real workflows. This is standard for qualitative studies but unverified.
  • ad hoc to paper The thematic coding frame (thoroughness/controllability objectives; labor/skill/uncertainty/unfixability constraints) is a meaningful organization of the data
    The framework is introduced by this paper (Section 4.2, Figure 1B); it is grounded in interviews but not externally validated.
  • domain assumption The sample is sufficiently representative of visualization experimenters
    Recruitment via professional networks and direct contact (Section 3.1) is purposive; limitations are acknowledged in Section 6.
invented entities (2)
  • Lifecycle of visualization experiment stimuli (seed idea → exploration → selection → shipment → deployment → analysis)
    purpose: Provides the temporal structure for reporting challenges and strategies
    An organizational model synthesized from interviews; useful but not independently validated.
  • Thoroughness-controllability objective framework
    purpose: Organizes reported challenges into two researcher objectives and four constraint types
    A paper-introduced interpretive taxonomy; no external benchmark or independent evidence is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Do Researchers Manage Visualization Experiment Stimuli?." pith.science (2026). https://pith.science/paper/IUXKQTIP

@misc{pith2026260726443,
  author       = {Pith},
  title        = {Pith review of: How Do Researchers Manage Visualization Experiment Stimuli?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IUXKQTIP}},
  note         = {Machine review of arXiv:2607.26443}
}
read the original abstract

Visualization experiments need a set of "good" stimuli that effectively address research questions and hypotheses. Creating, managing, and deploying stimuli are often challenging, as these tasks require tremendous care. Inappropriate stimuli can make the outcome invalid or uninteresting, wasting both researchers' and participants' resources. As the speed of science increases, better support for stimuli-related tasks is essential, yet we lack a closer look at how visualization researchers deal with them. To understand the experiences of visualization experimenters and guide future improvements, we interviewed 19 visualization researchers with diverse backgrounds and experiences. Our findings describe practices and challenges across the life cycle of stimuli, from exploration and selection through shipment, deployment, and analysis. For example, stimuli management and deployment require tedious manual effort, which does not scale for experiments with many levels and complex conditioning. We also discuss both concerns and optimism around AI-assisted visualization experiment design. We conclude with future research opportunities in supporting stimuli creation, automated stimuli inspection, and experimental apparatus concerns.

Figures

Figures reproduced from arXiv: 2607.26443 by the authors.

Figure 1
Figure 1. (A) Lifecycle of visualization experiment stimuli. Rectangles indicate current status, while circles indicate decisions made [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. First, thoroughness means researchers’ objectives to ensure the quality of an experiment in breadth and depth. Thoroughness objec￾tives include exploring a wide variety of possible stimuli ideas, having a generalizable set of stimuli, running technically sound experiment A Thoroughness B Controllability Space to cover Space covered Stimuli set Researcher’s control [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

85 extracted references · 2 linked inside Pith

  1. [1]

    Agarwal, G

    S. Agarwal, G. Sahu, A. Puri, I. H. Laradji, K. D. Dvijotham, J. Stanley et al. LitLLMs, LLMs for literature review: Are we there yet?Transactions on Machine Learning Research, 2025. 2

    2025

  2. [2]

    Al-Marzouki, S

    S. Al-Marzouki, S. Evans, T. Marshall, and I. Roberts. Are these data real? statistical methods for the detection of data fabrication in clinical trials. BMJ, 331(7511):267–270, 2005. doi: 10/btnnrv 9

    2005

  3. [3]

    H. An, A. Narechania, E. Wall, and K. Xu. vitaLITy 2: Reviewing academic literature using large language models, 2024. NLVIZ Workshop at IEEE VIS ’24.https://arxiv.org/abs/2408.13450. 2

    arXiv 2024

  4. [4]

    Bianchini, M

    S. Bianchini, M. Müller, and P. Pelletier. Artificial intelligence in sci- ence: An emerging general method of invention.Research Policy, 51(10):104604:1–15, 2022. doi: 10/gqm63c 2

    2022

  5. [5]

    Borgo, L

    R. Borgo, L. Micallef, B. Bach, F. McGee, and B. Lee. Information visualization evaluation using crowdsourcing.Comput. Graphics Forum, 37(3):573–595, 2018. doi: 10/gdxdk8 1, 2

    2018

  6. [6]

    Brand, A

    J. Brand, A. Israeli, and D. Ngwe. Using LLMs for market research. Harvard Business School Marketing Unit Working Paper No. 23-062, pp. 61–71, 2023. doi: 10/qw6j 3

    2023

  7. [7]

    Braun and V

    V . Braun and V . Clarke. Using thematic analysis in psychology.Qualitative Research in Psychology, 3(2):77–101, 2006. doi: 10/fswdcx 3

    2006

  8. [8]

    Bujack, T

    R. Bujack, T. L. Turton, F. Samsel, C. Ware, D. H. Rogers, and J. Ahrens. The good, the bad, and the ugly: A theoretical framework for the as- sessment of continuous colormaps.IEEE Trans. Vis. Comput. Graphics, 24(1):923–933, 2018. doi: 10/gcqb9w 1

    2018

Show all 85 references
  1. [9]

    Burns, C

    A. Burns, C. Xiong, S. Franconeri, A. Cairo, and N. Mahyar. How to evaluate data visualizations across different levels of understanding. In2020 IEEE Workshop on Evaluation and Beyond - Methodological Approaches to Visualization (BELIV), pp. 19–28, 2020. doi: 10/kz34 2

    2020

  2. [10]

    Carpendale.Evaluating Information Visualizations, pp

    S. Carpendale.Evaluating Information Visualizations, pp. 19–45. Springer- Verlag, Berlin, Heidelberg, 2008. 2

    2008

  3. [11]

    K. Choe, S. Park, S. Jung, H. Kim, J. W. Yang, H. Hong et al. Supporting novice researchers to write literature review using language models. In ACM Human Factors in Computing Systems, CHI EA ’24, pp. 307:1–9. ACM, 2024. doi: 10/qw5n 2

    2024

  4. [12]

    W. S. Cleveland and R. McGill. Graphical perception: Theory, experimen- tation, and application to the development of graphical methods.J. Am. Stat. Assoc, 79(387):531–554, 1984. doi: 10/gdvmwd 5

    1984

  5. [13]

    W. S. Cleveland and R. McGill. An experiment in graphical perception. International Journal of Man-Machine Studies, 25(5):491–500, 1986. doi: 10/b6vt7v 5

    1986

  6. [14]

    Cutler, J

    Z. Cutler, J. Wilburn, H. Shrestha, Y . Ding, B. Bollen, K. A. Nadib et al. ReVISit 2: A full experiment life cycle user study framework.IEEE Trans. Vis. Comput. Graphics, 32(1):13–23, 2026. doi: 10/hbkxwp 1, 2, 3, 4, 6, 9

    2026

  7. [15]

    J. R. de Leeuw, R. A. Gilbert, and B. Luchterhandt. jsPsych: Enabling an open-source collaborative ecosystem of behavioral experiments.Journal of Open Source Software, 8(85):5351:1–4, 2023. doi: 10/gtzcxz 1, 2

    2023

  8. [16]

    M. A. Elliott, C. Nothelfer, C. Xiong, and D. A. Szafir. A design space of vision science methods for visualization research.IEEE Trans. Vis. Comput. Graphics, 27(2):1117–1127, 2021. doi: 10/ghgt5q 2

    2021

  9. [17]

    M. H. Everts, H. Bekker, J. B. T. M. Roerdink, and T. Isenberg. Interactive illustrative line styles and line style transfer functions for flow visualization,

  10. [18]

    D. Fanelli. How many scientists fabricate and falsify research? a system- atic review and meta-analysis of survey data.PLOS ONE, 4(5):1–11, 05

  11. [19]

    Gelman, J

    A. Gelman, J. Hullman, and L. Kennedy. Causal quartets: Different ways to attain the same average treatment effect.Am. Stat., 78(3):267–272,

  12. [20]

    Grossmann, M

    I. Grossmann, M. Feinberg, D. C. Parker, N. A. Christakis, P. E. Tetlock, and W. A. Cunningham. AI and the transformation of social science research.Science, 380(6650):1108–1109, 2023. doi: 10/gscnp5 3, 9

    2023

  13. [21]

    Z. Guo, A. Kale, M. Kay, and J. Hullman. VMC: A grammar for vi- sualizing statistical model checks.IEEE Trans. Vis. Comput. Graphics, 31(1):798–808, 2025. doi: 10/qw57 9

    2025

  14. [22]

    Haehn, J

    D. Haehn, J. Tompkin, and H. Pfister. Evaluating ‘graphical perception’ with CNNs.IEEE Trans. Vis. Comput. Graphics, 25(1):641–650, 2019. doi: 10/gd52dt 3, 9

    2019

  15. [23]

    Q. Hao, F. Xu, Y . Li, and J. Evans. Artificial intelligence tools expand scientists’impact but contract science’s focus.Nature, 649(8099):1237– 1243, 2026. doi: 10/hbj27r 3, 8

    2026

  16. [24]

    Harper and M

    J. Harper and M. Agrawala. Deconstructing and restyling D3 visualiza- tions. InACM Symposium on User Interface Software and Technology, UIST ’14, pp. 253–262. ACM, 2014. doi: 10/ghvkfk 8

    2014

  17. [25]

    Harper and M

    J. Harper and M. Agrawala. Converting basic D3 charts into reusable style templates.IEEE Trans. Vis. Comput. Graphics, 24(1):1274–1286, 2017. doi: 10/gcxtgt 8

    2017

  18. [26]

    Hoffswell, W

    J. Hoffswell, W. Li, and Z. Liu. Techniques for flexible responsive visual- ization design. InACM Human Factors in Computing Systems, CHI ’20, pp. 1–13. ACM, 2020. doi: 10/gk9mqp 8, 9

    2020

  19. [27]

    K. Hu, M. A. Bakker, S. Li, T. Kraska, and C. Hidalgo. VizML: A machine learning approach to visualization recommendation. InACM Human Factors in Computing Systems, CHI ’19, pp. 128:1–12. ACM,

  20. [28]

    Hullman, D

    J. Hullman, D. Broska, S. Huaman, and S. Aaron. This human study did not involve human subjects: Validating LLM simulations as behavioral evidence, 2025. Working manuscript. doi: 10/qw5p 3, 9

    2025

  21. [29]

    Jianu, A

    R. Jianu, A. Slingsby, D. Laksono, and M. Okoe. VisUnit: Literate visualisation studies assembled from reusable test-suites. InACM Human Factors in Computing Systems, CHI ’25, pp. 917:1–15. ACM, 2025. doi: 10/qw5j 2

    2025

  22. [30]

    E. Jun, M. Daum, J. Roesch, S. Chasins, E. Berger, R. Just et al. Tea: A high-level language and runtime system for automating statistical analysis. InACM Symposium on User Interface Software and Technology, UIST ’19, pp. 591–603. ACM, 2019. doi: 10/gmbdfj 2, 9

    2019

  23. [31]

    E. Jun, E. Misback, J. Heer, and R. Just. rTisane: Externalizing conceptual models for data analysis prompts reconsideration of domain assumptions and facilitates statistical modeling. InACM Human Factors in Computing Systems, CHI ’24, pp. 1037:1–16. ACM, 2024. doi: 10/qw5k 2, 9

    2024

  24. [32]

    E. Jun, A. Seo, J. Heer, and R. Just. Tisane: Authoring statistical models via formal reasoning from conceptual and data relationships. InACM Human Factors in Computing Systems, CHI ’22, pp. 490:1–16. ACM,

  25. [33]

    Kafle, B

    K. Kafle, B. Price, S. Cohen, and C. Kanan. DVQA: Understanding data visualizations via question answering. InIEEE CVPR, June 2018. 3

    2018

  26. [34]

    Kim and J

    H. Kim and J. Heer. Data augmentation for visualization design knowledge bases.IEEE Trans. Vis. Comput. Graphics, 32(1):538–548, 2026. doi: 10/qw5x 3, 8

    2026

  27. [35]

    H. Kim, R. Rossi, J. Hullman, and J. Hoffswell. Dupo: A mixed-initiative authoring tool for responsive visualization.IEEE Trans. Vis. Comput. Graph., 30(1):934–943, 2024. doi: 10/qw5s 3, 8, 9

    2024

  28. [36]

    Y . Kim, K. Wongsuphasawat, J. Hullman, and J. Heer. GraphScape: A model for automated reasoning about visualization similarity and se- quencing. InACM Human Factors in Computing Systems, CHI ’17, pp. 2628–2638, 2017. doi: 10/gd7hg5 1

    2017

  29. [37]

    Kunin, J

    D. Kunin, J. Guo, T. Dae Devlin, and D. Xiang. Seeing theory, 2016. Last accessed Mar. 18, 2026. https://seeing-theory.brown.edu/ probability-distributions/index.html. 9

    2016

  30. [38]

    B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people.Behav. Brain Sci., 40:e253:1–72,

  31. [39]

    H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale. Empirical studies in information visualization: Seven scenarios.IEEE Trans. Vis. Comput. Graphics, 18(9):1520–1536, 2012. doi: 10/drrh6j 1, 2

    2012

  32. [40]

    M. S. Lam, J. Teoh, J. A. Landay, J. Heer, and M. S. Bernstein. Concept induction: Analyzing unstructured text with high-level concepts using LLooM. InACM Human Factors in Computing Systems, CHI ’24, p. 766. ACM, 2024. doi: 10/qw5m 2

    2024

  33. [41]

    H. Li, Y . Wang, S. Zhang, Y . Song, and H. Qu. KG4Vis: A knowledge graph-based approach for visualization recommendation.IEEE Trans. Vis. Comput. Graph., 28(1):195–205, 2022. doi: 10/gndq6h 1

    2022

  34. [42]

    yeah, this graph doesn’t show that

    M. Lisnic, A. Lex, and M. Kogan. "yeah, this graph doesn’t show that": Analysis of online engagement with misleading data visualizations. In ACM Human Factors in Computing Systems, CHI ’24, pp. 199:1–14. ACM,

  35. [43]

    Liu and J

    Y . Liu and J. Heer. Somewhere over the rainbow: An empirical assessment of quantitative colormaps. InACM Human Factors in Computing Systems, 10 © 2026 IEEE. This is the author’s version of the article that has been published in IEEE Transactions on Visualization and Computer ...

    2026

  36. [44]

    Y . Liu, A. Kale, T. Althoff, and J. Heer. Boba: Authoring and visualizing multiverse analyses.IEEE Trans. Vis. Comput. Graphics, 27(2):1753– 1763, 2021. doi: 10/ghgt7j 9

    2021

  37. [45]

    E. J. Livesey, M. B. Goldwater, and B. Colagiuri. Will human-like ma- chines make human-like mistakes?Behav. Brain Sci., 40:e270:41, 2017. doi: 10/g537kf 9

    2017

  38. [46]

    Merchant, S

    A. Merchant, S. Batzner, S. S. Schoenholz, M. Aykol, G. Cheon, and E. D. Cubuk. Scaling deep learning for materials discovery.Nature, 624(7990):80–85, 2023. doi: 10/gs7sbc 3

    2023

  39. [47]

    Moritz, C

    D. Moritz, C. Wang, G. L. Nelson, H. Lin, A. M. Smith, B. Howe et al. Formalizing visualization design knowledge as constraints: Actionable and extensible models in Draco.IEEE Trans. Vis. Comput. Graph., 25(1):438– 448, 2018. doi: 10/cs68 1, 3, 8

    2018

  40. [48]

    Musslick, L

    S. Musslick, L. K. Bartlett, S. H. Chandramouli, M. Dubova, F. Gobet, T. L. Griffiths et al. Automating the practice of science: Opportunities, challenges, and implications.Proceedings of the National Academy of Sciences, 122(5):e2401238121:1–11, 2025. doi: 10/g84s7d 2

    2025

  41. [49]

    Narechania, A

    A. Narechania, A. Karduni, R. Wesslen, and E. Wall. VITALITY: Pro- moting serendipitous discovery of academic literature with transformers & visual analytics.IEEE Trans. Vis. Comput. Graphics, 28(1):486–496,

  42. [50]

    Narechania, A

    A. Narechania, A. Srinivasan, and J. Stasko. NL4DV: A toolkit for gener- ating analytic specifications for data visualization from natural language queries.IEEE Trans. Vis. Comput. Graphics, 27(2):369–379, 2021. doi: 10/ghgthp 8

    2021

  43. [51]

    Nobre, D

    C. Nobre, D. Wootton, Z. Cutler, L. Harrison, H. Pfister, and A. Lex. ReVISit: Looking under the hood of interactive visualization studies. In ACM Human Factors in Computing Systems, CHI ’21, pp. 25:1–13. ACM,

  44. [52]

    Nobre, K

    C. Nobre, K. Zhu, E. Mörth, H. Pfister, and J. Beyer. Reading between the pixels: Investigating the barriers to visualization literacy. InACM Human Factors in Computing Systems, CHI ’24, pp. 197:1–17. ACM, 2024. doi: 10/n98g 1, 2

    2024

  45. [53]

    Okoe and R

    M. Okoe and R. Jianu. GraphUnit: Evaluating interactive graph visual- izations using crowdsourcing.Comput. Graphics Forum, 34(3):451–460,

  46. [54]

    L. Pan, A. Liu, Z. He, Z. Gao, X. Zhao, Y . Lu et al. MarkLLM: An open-source toolkit for LLM watermarking. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Sys- tem Demonstrations, pp. 61–71. ACL, 2024. doi: 10/qw6f 9

    2024

  47. [55]

    Pandey, S

    A. Pandey, S. L’Yi, Q. Wang, M. A. Borkin, and N. Gehlenborg. GenoREC: A recommendation system for interactive genomics data visualization. IEEE Trans. Vis. Comput. Graph., 29(1):570–580, 2023. doi: 10/qw5t 3

    2023

  48. [56]

    Pandey, A

    A. Pandey, A. Srinivasan, and V . Setlur. MEDLEY: Intent-based recom- mendations to support dashboard composition.IEEE Trans. Vis. Comput. Graph., 29(1):1135–1145, 2023. doi: 10/g9g7qk 3

    2023

  49. [57]

    Peirce, J

    J. Peirce, J. R. Gray, S. Simpson, M. MacAskill, R. Höchenberger, H. Sogo et al. PsychoPy2: Experiments in behavior made easy.Behavior Research Methods, 51(1):195–203, 2019. doi: 10/gft89w 2

    2019

  50. [58]

    J. W. Peirce. PsychoPy—psychophysics software in python.J. Neurosci. Methods, 162(1):8–13, 2007. doi: 10/b5dqq8 2

    2007

  51. [59]

    Plaisant

    C. Plaisant. The challenge of information visualization evaluation. In Proceedings of the Working Conference on Advanced Visual Interfaces, A VI ’04, pp. 109–116. ACM, 2004. doi: 10/c7g3x4 2

    2004

  52. [60]

    Your first data collection, 2026

    Prolific. Your first data collection, 2026. Last visited Mar. 18, 2026. https://docs.prolific.com/documentation/ get-started/your-first-data-collection. 9

    2026

  53. [61]

    Survey API Introduction, 2026

    Qualtrics. Survey API Introduction, 2026. Last vis- ited Mar. 18, 2026. https://api.qualtrics.com/ f8fdcc4586e1f-survey-api-introduction. 9

    2026

  54. [62]

    S. Sah, R. Mitra, A. Narechania, A. Endert, J. Stasko, and W. Dou. Gener- ating analytic specifications for data visualization from natural language queries using large language models, 2024. NLVIZ Workshop at IEEE VIS ’24.https://arxiv.org/abs/2408.13391. 3, 8

    2024 arXiv

  55. [63]

    Sarma, K

    A. Sarma, K. Hwang, J. Hullman, and M. Kay. Milliways: Taming multiverses through principled evaluation of data analysis paths. InACM Human Factors in Computing Systems, CHI ’24, pp. 607:1–15. ACM,

  56. [64]

    Sarma, A

    A. Sarma, A. Kale, M. J. Moon, N. Taback, F. Chevalier, J. Hullman et al. multiverse: Multiplexing alternative data analyses in r notebooks. InACM Human Factors in Computing Systems, CHI ’23, pp. 148:1–15. ACM,

  57. [65]

    Satyanarayan, D

    A. Satyanarayan, D. Moritz, K. Wongsuphasawat, and J. Heer. Vega-Lite: A grammar of interactive graphics.IEEE Trans. Visualization & Comp. Graphics, 23(1):341–350, 2017. doi: 10/f92f32 6

    2017

  58. [66]

    Savva, N

    M. Savva, N. Kong, A. Chhajta, L. Fei-Fei, M. Agrawala, and J. Heer. ReVision: automated classification, analysis and redesign of chart images. InACM Symposium on User Interface Software and Technology, UIST ’11, pp. 393–402. ACM, 2011. doi: 10/bcbnfw 8

    2011

  59. [67]

    Schöttler, J

    S. Schöttler, J. Dykes, J. Wood, U. Hinrichs, and B. Bach. Constraint- based breakpoints for responsive visualization design and development. IEEE Trans. Vis. Comput. Graphics, pp. 4593–4604, 2024. doi: 10/m3f3 8

    2024

  60. [68]

    L. S. Snyder and J. Heer. DIVI: Dynamically interactive visualization. IEEE Trans. Vis. Comput. Graphics, 30(1):403–413, 2024. doi: 10/qw54 8

    2024

  61. [69]

    G. Stoet. PsyToolkit: A novel web-based method for running online questionnaires and reaction-time experiments.Teaching of Psychology, 44(1):24–31, 2017. doi: 10/gftft5 2

    2017

  62. [70]

    Van Noorden and J

    R. Van Noorden and J. M. Perkel. AI and science: what 1,600 researchers think.Nature, 621:672–675, 2023. doi: 10/gtw8f2 3

    2023

  63. [71]

    H. W. Wang, M. Gordon, L. Battle, and J. Heer. DracoGPT: Extracting visualization design preferences from large language models.IEEE Trans. Vis. Comput. Graph., 31(1):710–720, 2025. doi: 10/qw5w 3, 9

    2025

  64. [72]

    H. W. Wang, J. Hoffswell, S. M. Thazin Thane, V . S. Bursztyn, and C. X. Bearfield. How aligned are human chart takeaways and llm predictions? a case study on bar charts with varying layouts.IEEE Trans. Vis. Comput. Graphics, 31(1):536–546, 2025. doi: 10/n82w 3, 9

    2025

  65. [73]

    H. Wickham. A layered grammar of graphics.J. Comput. Graphical Stat., 19(1):3–28, 2010. doi: 10/cttwkf 6

    2010

  66. [74]

    Wongsuphasawat, D

    K. Wongsuphasawat, D. Moritz, A. Anand, J. Mackinlay, B. Howe, and J. Heer. V oyager: Exploratory analysis via faceted browsing of visualiza- tion recommendations.IEEE Trans. Vis. Comput. Graph., 22(1):649–658,

  67. [75]

    Wongsuphasawat, Z

    K. Wongsuphasawat, Z. Qu, D. Moritz, R. Chang, F. Ouk, A. Anand et al. V oyager 2: Augmenting visual analysis with partial view specifications. InProc. CHI, CHI ’17, pp. 2648–2659. ACM, 2017. doi: 10/b7jv 3

    2017

  68. [76]

    A. Wu, Y . Wang, X. Shu, D. Moritz, W. Cui, H. Zhang et al. AI4VIS: Survey on artificial intelligence approaches for data visualization.IEEE Trans. Vis. Comput. Graphics, 28(12):5049–5070, 2022. doi: 10/grxn3d 2

    2022

  69. [77]

    A. Wu, Y . Wang, M. Zhou, X. He, H. Zhang, H. Qu et al. MultiVision: De- signing analytical dashboards with deep learning based recommendation. IEEE Trans. Vis. Comput. Graph., 28(1):162–172, 2022. doi: 10/gm393p 3

    2022

  70. [78]

    Y . Wu, Z. Guo, M. Mamakos, J. Hartline, and J. Hullman. The ratio- nal agent benchmark for data visualization.IEEE Trans. Vis. Comput. Graphics, 30(1):338–347, 2024. doi: 10/hbjqpb 9

    2024

  71. [79]

    J. Yang, P. F. Gyarmati, Z. Zeng, and D. Moritz. Draco 2: An extensible platform to model visualization design. InIEEE Visualization and Visual Analytics, VIS ’23, pp. 166–170. IEEE, 2023. doi: 10/qw5r 3

    2023

  72. [80]

    Zeng and L

    Z. Zeng and L. Battle. A review and collation of graphical perception knowledge for visualization recommendation. InACM Human Factors in Computing Systems, CHI ’23, pp. 820:1–16. ACM, 2023. doi: 10/kz36 3

    2023

  73. [81]

    Z. Zeng, J. Yang, D. Moritz, J. Heer, and L. Battle. Too many cooks: Exploring how graphical perception studies influence visualization recom- mendations in draco.IEEE Trans. Vis. Comput. Graph., 30(1):1063–1073,

  74. [82]

    Zhang, H

    S. Zhang, H. Li, H. Qu, and Y . Wang. AdaVis: Adaptive and explainable visualization recommendation for tabular data.IEEE Trans. Vis. Comput. Graph., 30(9):5923–5938, 2024. doi: 10/g94bsg 3

    2024

  75. [83]

    J. Zhu, J. Zhang, Q. Zhu, W. Li, J. Wu, and Y . Guo. A knowledge-guided visualization framework of disaster scenes for helping the public cognize risk information.International Journal of Geographical Information Science, 38(4):626–653, 2024. doi: 10/g4jdj9 2

    2024

  76. [84]

    Ziemkiewicz, M

    C. Ziemkiewicz, M. Chen, D. H. Laidlaw, B. Preim, and D. Weiskopf. Open Challenges in Empirical Visualization Research, pp. 243–252. Springer International Publishing, Cham, 2020. doi: 10/qw5h 2 11

    2020

  77. [2021]

    doi: 10/gksk5b 1, 2, 4, 6

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.