Pith. sign in

REVIEW 3 major objections 4 minor 78 references

The Role of Visualization in LLM-Assisted Knowledge Graph Systems: Effects on User Trust, Exploration, and Workflows

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Visual transparency made users trust wrong LLM answers.

desk verdict Solid design-study paper; the overtrust observation is real but the causal role of visualizations is asserted more strongly in the abstract than the design can support. read the letter →

arxiv 2505.21512 v1 pith:5DJ3PYZE submitted 2025-05-20 cs.LG cs.HC

Harry Li , Gabriel Appleby , Kenneth Alperin , Steven R Gomez , Ashley Suh This is my paper
classification cs.LGcs.HC
keywords knowledgegraphslargelanguagemodelsnaturalinterfacesSPARQLquerygenerationusertrustovertrustvisualizationdesignqualitativestudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Knowledge graphs are powerful but hard to query, and LLM interfaces promise to lower that barrier. This paper asks whether the transparency visualizations added to such an interface help users judge the LLM's work or simply make them trust it more. The authors built LinkQ, a system that turns natural-language questions into SPARQL queries over knowledge graphs, and wrapped it in five visual mechanisms meant to expose where the LLM could go wrong. In a think-aloud study with 14 practitioners, users—including knowledge-graph experts—often accepted LinkQ's incorrect answers and cited the visualizations as evidence that the system was working. The paper argues that 'helpful' visual transparency can reduce critical scrutiny, and that these systems cannot be one-size-fits-all because workflows and trust vary with expertise and prior skepticism.

What carries the argument

The load-bearing mechanism is LinkQ's prompting protocol together with its five visualization components: the LLM-KG state diagram, which shows which stage of the pipeline the system is in; the query editor, which pairs generated SPARQL with an LLM explanation; the entity-relation ID table, which gives human-readable labels for the entities and relations the LLM found; the query structure graph, which draws the graph pattern the query will traverse; and the results graph visualization, which embeds result tables in the query graph. These components are designed to surface the points where an LLM could hallucinate or misidentify data. The study's finding is that the same mechanisms supply the visual plausibility that lets users trust incorrect outputs.

What would settle it

Present matched groups of participants with the same deliberately wrong LinkQ answers, one group using the full visual interface and one using a chat-only version with identical LLM text but no state diagram, query graph, or entity-relation table; if wrong-answer acceptance rates are equal, the visualizations are not the cause of overtrust.

Watch

Extended reading notes

Core claim

LinkQ is posed as a testbed: an LLM-assisted knowledge-graph exploration system whose five visual mechanisms are intended to make the query-generation pipeline inspectable. The central discovery is that this transparency may backfire. The paper reports that users, even KG experts, tended to overtrust LinkQ's outputs when the LLM was wrong, citing the state diagram and the query structure graph as reasons for confidence. Incorrect answers were sometimes rationalized as plausible rather than rejected. The authors attribute this overconfidence to the added visual transparency of the natural-language interface, while also observing that users adopted distinct workflows depending on their knowledge-graph and LLM experience, so a single interface design will not serve all users.

Load-bearing premise

The claim that the visualizations caused overtrust rests on the assumption that this effect can be separated from the general authority of the LLM and from users' pre-existing trust dispositions; the study had no condition without or with altered visualizations to test that separation.

Editorial extensions

If this is right

  • If the finding holds, transparency visualizations in LLM-assisted tools are trust-shaping features, not neutral explainability aids, and should be designed with the possibility of false confidence in mind.
  • Users with different knowledge-graph and LLM experience will need different supports: query-graph inspection for those who can read queries, chat and entity-relation context for those who cannot, and source-linking for skeptical users.
  • The results graph view went unused while tabular results dominated, so graph-style output needs a concrete exploratory purpose rather than being included for visual appeal.
  • Because participants tackled open-ended tasks by asking narrow, deliberate sub-questions, LLM-assisted exploration may reduce overall data coverage compared to unassisted exploration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference — The study had no no-visualization baseline, so the overtrust could instead come from the LLM's textual authority; a chat-only control condition would separate these causes.
  • Editorial inference — A quantitative follow-up that toggles each visualization off and measures acceptance of seeded wrong answers could rank the components by their trust effect; participant quotes suggest the LLM-KG state diagram is the most likely driver.
  • Editorial inference — If visual plausibility drives false confidence, then visualizations that actively challenge the answer—for example, by showing alternative queries or explaining empty results—may reduce overtrust more than neutral transparency panels; the authors gesture at this direction but do not test it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents LinkQ, an LLM-assisted visual knowledge-graph exploration system with five visualization components: an LLM-KG state diagram, a query editor with explanations, an entity-relation table, a query structure graph, and a results graph. The authors report a quantitative benchmark on the Mintaka dataset showing that LinkQ's chained-prompting protocol achieves higher query-generation accuracy than a GPT-4 baseline, and a think-aloud qualitative study with 14 KG/LLM practitioners. The study's main findings are that users, including KG experts, tended to overtrust LinkQ's outputs and sometimes rationalized incorrect answers; that users with different KG and LLM expertise adopted distinct verification and workflow strategies; and that the results graph was unused in favor of the tabular results view. Based on these observations, the paper offers design implications and preliminary guidelines for LLM-based KG systems.

Significance. If the causal link between visual transparency and overtrust were established, the paper would make an important contribution to visualization and explainable-AI research, challenging the common assumption that transparency mechanisms uniformly increase critical scrutiny. The descriptive findings—that workflow strategies vary systematically with KG/LLM expertise and that the most popular visualization (the state diagram) fosters trust—are timely and relevant for designers of LLM-assisted visual analytics systems. The system itself is open-source, and the quantitative benchmark, while limited, supports the claim that the chained-prompting protocol improves query accuracy over a plain LLM. The qualitative data are presented with direct participant quotes, which is a strength and provides useful grounding for future work. However, the central causal claim that visualizations caused overtrust is not supported by the current study design, so the paper's main contribution is better characterized as a hypothesis-generating qualitative study than as an established causal finding.

major comments (3)
  1. [Abstract; §5.1; §5.2; §6.2] The central claim that LinkQ's visualizations caused user overtrust is not supported by the study design. All participants saw the full visualization suite, with no condition that omitted or altered the visualizations, so the observed overtrust cannot be separated from the general authority of an LLM or from participants' prior dispositions. The two concrete incorrect-answer rationalizations in §6.2—the Gladiator and Google Chrome examples—make no reference to any visualization; they are plausibility-based rationalizations that would plausibly occur with a text-only LLM assistant. The quotes in §5.2 praising the state diagram and entity-relation table concern comprehension and perceived transparency, not acceptance of incorrect outputs, and the paper does not report how often overtrust occurred ('often' is not quantified). The abstract's claim that users 'overtrust LinkQ's outputs due to its helpful visualizations' should be softened to a hypothesis, or a controlled comparison (e.g., an LLM chat-only baseline) should be added to substantiate the causal role.
  2. [§5.1 (Participants)] Three of the fourteen participants were involved in the initial stages of developing LinkQ. Their prior involvement creates a risk of demand characteristics and pro-system bias, yet the paper does not report whether their data were analyzed separately or whether any themes, including overtrust and visualization preferences, were robust to their exclusion. The authors should disclose this treatment or justify why these participants do not bias the reported findings.
  3. [§4.4; §5.1 (Targeted questions)] The selection of 'incorrect' questions for the qualitative study depends entirely on the quantitative evaluation reported in Table 1, but that table reports only aggregate percentages with no per-question results, confidence intervals, or significance tests. Moreover, the qualitative section does not specify which targeted questions each participant received or whether the pre-determined 'incorrect' label was verified in the actual session (e.g., by comparing with a ground-truth answer from the KG). Without this, the overtrust episodes cannot be interpreted as responses to genuinely incorrect outputs. Please provide the question set and the error-verification procedure.
minor comments (4)
  1. [§2.2] Typo: 'aethestetically-pleasing' should be 'aesthetically pleasing.'
  2. [§6.2] The Gladiator example says 'a second one came out recently, so maybe that's when it won?'; the film Gladiator II was released in 2024, not 2023, so the participant's rationalization is even more implausible than the text implies. Please verify the details of the example.
  3. [§5.2] The statement that the graph visualization 'was never used' is strong; please clarify whether it was displayed by default or required an explicit user action, and whether 'not used' means not selected as the primary view rather than not seen.
  4. [General] The paper would benefit from a short appendix listing the targeted questions used in the qualitative study, along with their ground-truth answers and the authors' classification (correct vs. incorrect). This would aid reproducibility and the interpretation of the overtrust episodes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the overtrust finding is an empirical result from participant behavior, not an equation fitted to or derived from its own inputs.

full rationale

This paper contains no formal derivation whose conclusion is equivalent to its premises. The central claim—that users, including KG experts, overtrusted LinkQ's outputs and cited the visualizations as reassurance—is grounded in think-aloud observations and interview quotes reported in Sections 5.2 and 6.2, not in any fitted parameter renamed as a prediction. The only quantitative benchmark (Section 4.4, Table 1) compares LinkQ's query accuracy to GPT-4 on Mintaka; it is used to select test questions and to argue that prompt-engineered LLMs still need human oversight, but it does not generate the trust finding. The self-citations [40,41,42] support background claims and the provenance of the LinkQ system; the current manuscript fully describes the protocol and prompts, and the qualitative result is independent of those citations. The paper explicitly hedges the causal claim in Section 6.2 ('we believe ... may have resulted') and acknowledges design limitations in Section 7.2 (no non-LLM control, no longitudinal comparison), which is the opposite of presenting a forced derivation. The two concrete overtrust rationalizations (Gladiator, Google Chrome) are plausibility-based and do not mention visual features, which weakens the causal interpretation but does not make it circular. No Eq. X = Eq. Y by construction, no fitted input called a prediction, and no uniqueness theorem imported from the authors' own prior work.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper makes no parametric fits. Its central claims rest on qualitative interpretation of self-reported and observed user behavior, plus a small quantitative benchmark. The axioms listed are the loading-bearing assumptions about the validity of the measurement instruments and the fairness of the baseline comparison. No new physical or conceptual entities are introduced beyond the LinkQ system itself.

assumptions (5)
  • domain assumption Self-reported Likert expertise accurately reflects participants' true KG and LLM expertise.
    Participants are grouped into experts and non-experts based on self-reported scores in Section 5.1 and Table 2, and these groups are then used to explain differing workflows in Section 5.4.
  • domain assumption Think-aloud verbalizations and post-study interviews accurately reveal participants' trust and reasoning processes.
    The study relies on think-aloud protocol and follow-up questions to infer trust and overtrust, as stated in Section 5.1.
  • domain assumption Mintaka question accuracy is a valid operationalization of query correctness.
    Section 4.4 uses accuracy on 120 Mintaka questions to claim LinkQ outperforms GPT-4, assuming the dataset's labels are ground truth for what the query should retrieve.
  • domain assumption Direct GPT-4 answering is a fair baseline for LinkQ's query-generation approach.
    Table 1 compares LinkQ's accuracy to 'GPT-4 Accuracy' without specifying the baseline prompting or output format, assuming the comparison isolates the benefit of LinkQ's protocol.
  • domain assumption Thematic saturation is reached at 14 participants.
    Section 5.1 states participants were recruited until no new themes emerged, following a saturation approach, but no formal saturation analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Role of Visualization in LLM-Assisted Knowledge Graph Systems: Effects on User Trust, Exploration, and Workflows." pith.science (2026). https://pith.science/paper/5DJ3PYZE

@misc{pith2026250521512,
  author       = {Pith},
  title        = {Pith review of: The Role of Visualization in LLM-Assisted Knowledge Graph Systems: Effects on User Trust, Exploration, and Workflows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5DJ3PYZE}},
  note         = {Machine review of arXiv:2505.21512}
}
read the original abstract

Knowledge graphs (KGs) are powerful data structures, but exploring them effectively remains difficult for even expert users. Large language models (LLMs) are increasingly used to address this gap, yet little is known empirically about how their usage with KGs shapes user trust, exploration strategies, or downstream decision-making - raising key design challenges for LLM-based KG visual analysis systems. To study these effects, we developed LinkQ, a KG exploration system that converts natural language questions into structured queries with an LLM. We collaborated with KG experts to design five visual mechanisms that help users assess the accuracy of both KG queries and LLM responses: an LLM-KG state diagram that illustrates which stage of the exploration pipeline LinkQ is in, a query editor displaying the generated query paired with an LLM explanation, an entity-relation ID table showing extracted KG entities and relations with semantic descriptions, a query structure graph that depicts the path traversed in the KG, and an interactive graph visualization of query results. From a qualitative evaluation with 14 practitioners, we found that users - even KG experts - tended to overtrust LinkQ's outputs due to its "helpful" visualizations, even when the LLM was incorrect. Users exhibited distinct workflows depending on their prior familiarity with KGs and LLMs, challenging the assumption that these systems are one-size-fits-all - despite often being designed as if they are. Our findings highlight the risks of false trust in LLM-assisted data analysis tools and the need for further investigation into the role of visualization as a mitigation technique.

Figures

Figures reproduced from arXiv: 2505.21512 by the authors.

Figure 1
Figure 1. The LinkQ system, an LLM-assisted KG visual exploration system developed in collaboration with KG practitioners. Following the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A high-level overview of LinkQ’s [41] prompting protocol, described in Section 4.2. For this pipeline, the LLM, System, and User have precise responsibilities for completing the question-to-query translation. question; or (3) the user wants to ask a different question. Each button produces a templated prompt that can be edited by the user. LLM-KG State Diagram: The top panel of LinkQ’s NLI is a state diagram visuali… view at source ↗
Figure 4
Figure 4. Example of LinkQ’s Results Graph Visualization, where nodes [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: An example of the LinkQ prompting protocol described in Section [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 5
Figure 5. Figure 5: Process of our qualitative user study (Section [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 56 canonical work pages

  1. [1]

    https://www.w3.org/TR/PR-rdf-syntax/Overview

    Resource Description Framework (RDF) Model and Syntax Specification — w3.org. https://www.w3.org/TR/PR-rdf-syntax/Overview. html. [Accessed 29-07-2024]. 2, 3

    work page 2024

  2. [2]

    Agrawal, T

    G. Agrawal, T. Kumarage, Z. Alghamdi, and H. Liu. Can knowledge graphs reduce hallucinations in LLMs? : A survey. In K. Duh, H. Gomez, and S. Bethard, eds.,Proc. NAACL: Hum. Lang. Tech. (Vol. 1: Long Pa- pers), pp. 3947–3960. Association for Computational Linguistics, Mexico City, Mexico, June 2024. doi: 10.18653/v1/2024.naacl-long.219 2

    doi:10.18653/v1/2024.naacl-long.219 2024

  3. [3]

    Alexander, P

    J. Alexander, P. Nanda, K.-C. Yang, and A. Sarvghad. Can gpt-4 models detect misleading visualizations? In2024 IEEE Visualization and Visual Analytics (VIS), pp. 106–110. IEEE, 2024. 2

    work page 2024

  4. [4]

    AlKhamissi, M

    B. AlKhamissi, M. Li, A. Celikyilmaz, M. Diab, and M. Ghazvinine- jad. A review on language models as knowledge bases.arXiv preprint arXiv:2204.06031, 2022. doi: 10.48550/arXiv.2204.06031 2

    arXiv doi:10.48550/arxiv.2204.06031 2022

  5. [5]

    Arawjo, C

    I. Arawjo, C. Swoopes, P. Vaithilingam, M. Wattenberg, and E. L. Glass- man. Chainforge: A visual toolkit for prompt engineering and llm hypoth- esis testing. InProceedings of the CHI Conference on Human Factors in Computing Systems, CHI ’24, article no. 304, 18 pages. Association for Computing Machinery, New York, NY , USA, 2024. doi: 10.1145/3613904 .36...

    doi:10.1145/3613904 2024

  6. [6]

    Arazzi, D

    M. Arazzi, D. Ligari, S. Nicolazzo, and A. Nocera. Augmented knowledge graph querying leveraging llms.arXiv preprint arXiv:2502.01298, 2025. 1, 2

    arXiv 2025

  7. [7]

    Aurisano, A

    J. Aurisano, A. Kumar, A. Gonzales, J. Leigh, B. DiEugenio, and A. John- son. Articulate 2 : Toward a conversational interface for visual data exploration. InProc. VIS, 2016. 2

    work page 2016

  8. [8]

    R. C. Basole and T. Major. Generative ai for visualization: Opportunities and challenges.IEEE Computer Graphics and Applications, 44(2):55–64,

    work page

Show all 78 references
  1. [9]

    Bendeck and J

    A. Bendeck and J. Stasko. An empirical evaluation of the gpt-4 multimodal language model on visualization literacy tasks.IEEE Transactions on Visualization and Computer Graphics, 2024. 2

    2024

  2. [10]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. C...

    1901

  3. [11]

    Buçinca, M

    Z. Buçinca, M. B. Malaya, and K. Z. Gajos. To trust or to think: cognitive forcing functions can reduce overreliance on ai in ai-assisted decision- making.Proceedings of the ACM on Human-computer Interaction, 5(CSCW1):1–21, 2021. 2, 8

    2021

  4. [12]

    Crisan, N

    A. Crisan, N. Butters, and Zoe. Exploring subjective notions of explain- ability through counterfactual visualization of sentiment analysis. In2024 IEEE Evaluation and Beyond-Methodological Approaches for Visualiza- tion (BELIV), pp. 15–24. IEEE, 2024. 2

    2024

  5. [13]

    R. J. Crouser, S. Matoussi, L. Kung, S. Pandey, O. G. McKinley, and A. Ottley. Building and eroding: Exogenous and endogenous factors that influence subjective trust in visualization. In2024 IEEE Visualization and Visual Analytics (VIS), pp. 306–310. IEEE, 2024. 3, 9

  6. [14]

    L. A. Curry, I. M. Nembhard, and E. H. Bradley. Qualitative and mixed methods provide unique contributions to outcomes research.Circulation, 119(10):1442–1452, 2009. 6

    2009

  7. [15]

    Ehrlinger and W

    L. Ehrlinger and W. Wöß. Towards a definition of knowledge graphs.Proc. ESWC Posters and Demos Track, 48(1-4):2, 2016. 2

    2016

  8. [16]

    B. Ell, D. Vrandeˇci´c, and E. Simperl. Spartiqulation: Verbalizing sparql queries. InProc. ESWC, pp. 117–131. Springer, 2013. 3

    2013

  9. [17]

    A. G. et. al. The llama 3 herd of models, 2024. doi: 10.48550/arXiv.2407. 21783 3

    2024 doi

  10. [18]

    C. Feng, X. Zhang, and Z. Fei. Knowledge solver: Teaching llms to search for domain knowledge from knowledge graphs, 2023. 3

    2023

  11. [19]

    R. C. Fernandez, A. J. Elmore, M. J. Franklin, S. Krishnan, and C. Tan. How large language models will disrupt data management.Proc. VLDB, 16(11):3302–3309, 2023. 2

    2023

  12. [20]

    S. Ferré. Sparklis: An expressive query builder for sparql endpoints with guidance in natural language.Semant. Web, 8(3):405–418, 2017. doi: 10. 3233/SW-150208 2, 3

    2017

  13. [21]

    Grafkin, M

    P. Grafkin, M. Mironov, M. Fellmann, B. Lantow, K. Sandkuhl, and A. V . Smirnov. Sparql query builders: Overview and comparison. InBIR Workshops, pp. 255–274, 2016. 2, 3

    2016

  14. [22]

    Guest, A

    G. Guest, A. Bunce, and L. Johnson. How many interviews are enough? an experiment with data saturation and variability.Field methods, 18(1):59– 82, 2006. 6

    2006

  15. [23]

    J. Ha, H. Jeon, D. Han, J. Seo, and C. Oh. Clochat: Understanding how people customize, interact, and experience personas in large language models. InProceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–24, 2024. 2, 4

    2024

  16. [24]

    S. Ha, S. Monadjemi, and A. Ottley. Guided by ai: Navigating trust, bias, and data exploration in ai-guided visual analytics. InComputer Graphics Forum, vol. 43, p. e15108. Wiley Online Library, 2024. 2, 8

    2024

  17. [25]

    Harte, L

    R. Harte, L. Glynn, A. Rodríguez-Molinero, P. M. Baker, T. Scharf, L. R. Quinlan, G. ÓLaighin, et al. A human-centered design methodology to en- hance the usability, human factors, and user experience of connected health systems: a three-phase methodology.JMIR human factors, 4...

  18. [26]

    G. He, N. Aishwarya, and U. Gadiraju. Is conversational xai all you need? human-ai decision making with a conversational xai assistant.arXiv preprint arXiv:2501.17546, 2025. 2, 8

    2025 arXiv

  19. [27]

    Hemberg, J

    E. Hemberg, J. Kelly, M. Shlapentokh-Rothman, B. Reinstadler, K. Xu, N. Rutar, and U.-M. O’Reilly. Linking threat tactics, techniques, and patterns with defensive weaknesses, vulnerabilities and affected platform configurations for cyber hunting.arXiv preprint arXiv:2010.00533...

    2010 doi arXiv

  20. [28]

    Hogan, E

    A. Hogan, E. Blomqvist, M. Cochez, C. D’amato, G. D. Melo, C. Gutier- rez, S. Kirrane, J. E. L. Gayo, R. Navigli, S. Neumaier, A.-C. N. Ngomo, A. Polleres, S. M. Rashid, A. Rula, L. Schmelzeisen, J. Sequeda, S. Staab, and A. Zimmermann. Knowledge graphs.ACM Comput. Surv., 54(4...

    2021 doi

  21. [29]

    Z. Hong, Z. Yuan, Q. Zhang, H. Chen, J. Dong, F. Huang, and X. Huang. Next-generation database interfaces: A survey of llm-based text-to-sql,

  22. [30]

    Huang, Y

    J. Huang, Y . Xi, J. Hu, and J. Tao. Flownl: Asking the flow data in natural languages.IEEE Trans. Vis. Comput. Graph., 29(1):1200–1210, 2023. doi: 10.1109/TVCG.2022.3209453 2, 3

    2023

  23. [31]

    Jacovi, J

    A. Jacovi, J. Bastings, S. Gehrmann, Y . Goldberg, and K. Filippova. Diagnosing ai explanation methods with folk concepts of behavior.Journal of Artificial Intelligence Research, 78:459–489, 2023. 2

    2023

  24. [32]

    Karanikolas, E

    N. Karanikolas, E. Manga, N. Samaridi, E. Tousidou, and M. Vassilakopou- los. Large language models versus natural language understanding and generation. InIn Proc. PCI, pp. 278–290, 2023. 2

    2023

  25. [33]

    J. F. Kelley. An iterative design methodology for user-friendly natural lan- guage office information applications.ACM Transactions on Information Systems (TOIS), 2(1):26–41, 1984. 9

    1984

  26. [34]

    Koonchanok, P

    R. Koonchanok, P. Baser, A. Sikharam, N. K. Raveendranath, and K. Reda. Data prophecy: Exploring the effects of belief elicitation in visual analyt- ics. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–12, 2021. 8

    2021

  27. [35]

    Koonchanok, G

    R. Koonchanok, G. Y . Tawde, G. R. Narayanasamy, S. Walimbe, and K. Reda. Visual belief elicitation reduces the incidence of false discovery. InProceedings of the 2023 CHI conference on human factors in computing systems, pp. 1–17, 2023. 8

    2023

  28. [36]

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 3

    2023

  29. [37]

    C. Lee, T. Yang, G. D. Inchoco, G. M. Jones, and A. Satyanarayan. Viral visualizations: How coronavirus skeptics use orthodox data practices to promote unorthodox science online. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, article n...

    2021

  30. [38]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küt- tler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval- augmented generation for knowledge-intensive nlp tasks. InProc. NIPS, article no. 793, 16 pages. Red Hook, NY , USA, 2020. 9

    2020

  31. [39]

    Li and H

    F. Li and H. V . Jagadish. Constructing an interactive natural language interface for relational databases.In Proc. VLDB, 8(1):73–84, 2014. 2

    2014

  32. [40]

    H. Li, G. Appleby, C. D. Brumar, R. Chang, and A. Suh. Knowledge graphs in practice: Characterizing their users, challenges, and visualization opportunities.IEEE Trans. Vis. Comput. Graph., 30(1):584–594, 2024. doi: 10.1109/TVCG.2023.3326904 2, 4, 6

    2024

  33. [41]

    H. Li, G. Appleby, and A. Suh. Linkq: An llm-assisted visual interface for knowledge graph question-answering. In2024 IEEE Visualization and Visual Analytics (VIS), pp. 116–120, 2024. doi: 10.1109/VIS55277.2024. 00031 4

    2024

  34. [42]

    H. Li, G. Appleby, and A. Suh. A preliminary roadmap for llms as assistants in exploring, analyzing, and visualizing knowledge graphs. In IEEE VIS NLVIZ Workshop: Exploring Research Opportunities for Natural Language, Text, and Data Visualization, 2024. 2, 3

    2024

  35. [43]

    H. Li, Y . Wang, S. Zhang, Y . Song, and H. Qu. Kg4vis: A knowledge graph-based approach for visualization recommendation.IEEE Trans. Vis. Comput. Graph., 28(01):195–205, 2022. doi: 10.1109/TVCG.2021. 3114863 1

    2022 doi

  36. [44]

    Z. C. Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.Queue, 16(3):31–57, 2018. 2

    2018

  37. [45]

    Lisnic, C

    M. Lisnic, C. Polychronis, A. Lex, and M. Kogan. Misleading beyond visual tricks: How people actually lie with charts. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, article no. 817, 21 pages. Association for Computing Machinery, New Y...

    2023

  38. [46]

    Lissandrini, D

    M. Lissandrini, D. Mottin, K. Hose, and T. B. Pedersen. Knowledge graph exploration systems: are we lost? InCIDR, vol. 22, pp. 10–13, 2022. 1, 2, 3

    2022

  39. [47]

    Lissandrini, D

    M. Lissandrini, D. Mottin, T. Palpanas, and Y . Velegrakis. Graph-query suggestions for knowledge graph exploration. InProc. ACM WWW, pp. 2549–2555, 2020. 2

    2020

  40. [48]

    Martino, M

    A. Martino, M. Iannelli, and C. Truong. Knowledge injection to counter large language model (llm) hallucination. In C. Pesquita, H. Skaf- Molli, V . Efthymiou, S. Kirrane, A. Ngonga, D. Collarana, R. Cerqueira, M. Alam, C. Trojahn, and S. Hertling, eds.,The Semantic Web: ESWC ...

    2023

  41. [49]

    McKinley, S

    O. McKinley, S. Pandey, and A. Ottley. Trustworthy by design: The viewer’s perspective on trust in data visualization.arXiv preprint arXiv:2503.10892, 2025. 3, 8

    2025 arXiv

  42. [50]

    J. J. Miller. Graph database applications and concepts with Neo4J.Proc. SAIS, 2324(36), 2013. 3

    2013

  43. [51]

    Mitra, A

    R. Mitra, A. Narechania, A. Endert, and J. Stasko. Facilitating conversa- tional interaction in natural language interfaces for visualization. InProc. VIS, pp. 6–10, 2022. doi: 10.1109/VIS54862.2022.00010 2, 3

    2022

  44. [52]

    Narechania, A

    A. Narechania, A. Srinivasan, and J. Stasko. NL4DV: A Toolkit for generating Analytic Specifications for Data Visualization from Natural Language queries.IEEE Trans. Vis. Comput. Graph., 2020. doi: 10. 1109/TVCG.2020.3030378 2

    2020

  45. [53]

    Ngonga Ngomo, L

    A.-C. Ngonga Ngomo, L. Bühmann, C. Unger, J. Lehmann, and D. Gerber. Sorry, i don’t speak sparql: translating sparql queries into natural language. InProc. ACM WWW, 12 pages, p. 977–988. ACM, New York, NY , USA,

  46. [54]

    C. North. Toward measuring visualization insight.IEEE computer graph- ics and applications, 26(3):6–9, 2006. 4

    2006

  47. [55]

    S. Pan, L. Luo, Y . Wang, C. Chen, J. Wang, and X. Wu. Unifying large language models and knowledge graphs: A roadmap.IEEE Trans. Knowl. Data Eng., pp. 1–20, 2024. doi: 10.1109/TKDE.2024.3352100 1, 2, 3

    2024

  48. [56]

    Petroni, T

    F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y . Wu, and A. Miller. Language models as knowledge bases? InProc. EMNLP/IJCNLP, pp. 2463–2473. ACL, Hong Kong, 2019. doi: 10.18653/ v1/D19-1250 2

    2019

  49. [57]

    J. C. Rangel, T. M. de Farias, A. C. Sima, and N. Kobayashi. Sparql generation: an analysis on fine-tuning openllama for question answering over a life science knowledge graph.arXiv preprint arXiv:2402.04627,

    arXiv

  50. [58]

    Rawte, A

    V . Rawte, A. Sheth, and A. Das. A survey of hallucination in large foundation models.arXiv preprint arXiv:2309.05922, 2023. 1, 2

    2023 arXiv

  51. [59]

    M. C. Rillig, M. Ågerstrand, M. Bi, K. A. Gould, and U. Sauerland. Risks and benefits of large language models for the environment.Environmental Science & Technology, 57(9):3464–3466, 2023. 9

    2023

  52. [60]

    Rogers.HCI theory: classical, modern, and contemporary, vol

    Y . Rogers.HCI theory: classical, modern, and contemporary, vol. 14. Morgan & Claypool Publishers, 2012. 9

    2012

  53. [61]

    Sedlmair, M

    M. Sedlmair, M. Meyer, and T. Munzner. Design study methodology: Reflections from the trenches and the stacks.IEEE Trans. Vis. Comput. Graph., 18(12):2431–2440, 2012. doi: 10.1109/TVCG.2012.213 3

    2012 doi

  54. [62]

    P. Sen, A. F. Aji, and A. Saffari. Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering.arXiv preprint arXiv:2210.01613, 2022. 4, 6

    2022 arXiv

  55. [63]

    Setlur, S

    V . Setlur, S. E. Battersby, M. Tory, R. Gossweiler, and A. X. Chang. Eviza: A natural language interface for visual analysis. InIn Proc. ACM UIST, pp. 365–377, 2016. 8

    2016

  56. [64]

    Srinivasan and V

    A. Srinivasan and V . Setlur. Snowy: Recommending utterances for con- versational visual analysis. InACM UIST, pp. 864–880, 2021. 3, 8

    2021

  57. [65]

    S. Suh, M. Chen, B. Min, T. J.-J. Li, and H. Xia. Luminate: Structured generation and exploration of design space with large language models for human-ai co-creation. InProceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–26, 2024. 2

    2024

  58. [66]

    Sultanum, D

    N. Sultanum, D. Bromley, and M. Correll. Data guards: Challenges and solutions for fostering trust in data. In2024 IEEE Visualization and Visual Analytics (VIS), pp. 56–60. IEEE, 2024. 3, 9

    2024

  59. [67]

    Sultanum and A

    N. Sultanum and A. Srinivasan. Datatales: Investigating the use of large language models for authoring data-driven articles. InProc. VIS, pp. 231–235, 2023. 2, 3, 4

    2023

  60. [68]

    Y . Tian, W. Cui, D. Deng, X. Yi, Y . Yang, H. Zhang, and Y . Wu. Chartgpt: Leveraging llms to generate charts from abstract natural language.IEEE Transactions on Visualization and Computer Graphics, 2024. 2

    2024

  61. [69]

    Vargas, C

    H. Vargas, C. Buil-Aranda, A. Hogan, and C. López. Rdf explorer: A visual sparql query builder. InThe Semantic Web–ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part I 18, pp. 647–663. Springer, 2019. 2, 3, 4

    2019

  62. [70]

    W. W. W. C. (W3C). RDF Primer. https://www.w3.org/TR/ rdf11-primer/. Accessed: 2024-08-26. 2

    2024

  63. [71]

    Z. Wang, Y . Huang, D. Song, L. Ma, and T. Zhang. Promptcharm: Text- to-image generation through multi-modal prompting and refinement. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–21, 2024. 2, 4

    2024

  64. [72]

    H. Wei. Combining knowledge graphs, quickly and accurately.Amazon Science, 2020. 2

    2020

  65. [73]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022. 3

    2022

  66. [74]

    Y . Wen, Z. Wang, and J. Sun. Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models. InProc. Association for Computational Linguistics, 2024. 2, 3

    2024

  67. [75]

    Wikidata Statistics

    Wikipedia. Wikidata Statistics. https://www.wikidata.org/wiki/ Wikidata:Statistics. Accessed: 2025-03-18. 2, 6

    2025

  68. [76]

    S. Yang, M. Teng, X. Dong, and F. Bo. Llm-based sparql generation with selected schema from large scale knowledge base. InChina Conference on Knowledge Graph and Semantic Computing, pp. 304–316. Springer,

  69. [77]

    Zhang, X

    L. Zhang, X. Liu, Z. Li, X. Pan, P. Dong, R. Fan, R. Guo, X. Wang, Q. Luo, S. Shi, et al. Dissecting the runtime performance of the train- ing, fine-tuning, and inference of large language models.arXiv preprint arXiv:2311.03687, 2023. 2

    2023 arXiv

  70. [2013]

    doi: 10.1145/2488388.2488473 2

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.