Pith. sign in

REVIEW 4 major objections 4 minor 54 references

From Queries to Criteria: Understanding How Astronomers Evaluate LLMs

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that real astronomy users' queries and interviews reveal evaluation criteria that should guide LLM benchmarks, and that a 40-item gold benchmark enables automated scoring correlated with user satisfaction at $r = 0.8239$.

desk verdict A genuinely useful user-grounded taxonomy and a released astronomy benchmark, with a validation correlation that needs its N, judge, and baseline reported before the benchmark's headline claim can be trusted. read the letter →

arxiv 2507.15715 v2 pith:BXDLFMW6 submitted 2025-07-21 cs.CL astro-ph.IM

Alina Hyk , Kiera McCormick , Mian Zhong , Ioana Ciucă , Sanjib Sharma , John F Wu , J. E. G. Peek , Kartheik G. Iyer
show 2 more authors
Ziang Xiao Anjalie Field
This is my paper · ORCID
classification cs.CLastro-ph.IM
keywords LLMevaluationbenchmarkconstructionastronomyretrieval-augmentedgenerationuserstudyinductivecodingRAGscientificliterature
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Astronomers evaluating an AI literature assistant care about far more than factual accuracy: they ask questions whose answers they already know, stress-test the model's limits, request synthesis and speculation, and judge responses on citation quality, hedging, specificity, correct interpretation of intent, and broader impact. The paper claims that these behaviors, recovered by coding 368 real queries to a Slack-based retrieval-augmented bot and interviewing 11 astronomers, should define what LLM benchmarks measure. To show this is practical, the authors build a 40-question gold benchmark from real queries with expert-written answers and report that automated LLM-judge scores on it correlate with users' thumbs-up/down ratings at Pearson's $r = 0.8239$. If the claim holds, benchmark design for scientific AI can stop leaning on closed-form questions and ground itself in documented user evaluation strategies.

What carries the argument

The machinery is a three-part pipeline. First, an inductive coding scheme, built by four annotators with astronomy, NLP, and psychology backgrounds, labels each query by type, yielding nine categories from specific factual to unresolved topic. Second, a mapping table connects the query types to the five user-reported evaluation criteria and proposes automated metrics for each, such as epistemic markers ('maybe', 'definitely') for hedging and natural-language-inference style checks for citation support. Third, a sample benchmark of 40 real queries with expert-written gold answers lets a separate LLM judge score any system response by closeness to the gold answer, and that single score is the instrument whose correlation with user ratings validates the approach.

What would settle it

A direct test: recruit a new set of astronomers who did not see the original deployment, have them rate the same 40 bot responses on the five reported criteria, and compare their multi-criterion ratings to the single LLM-judge score and to the thumbs-up/down counts; if the single score misses, say, hedging or citation-quality judgments that users care about, the benchmark is not capturing the full reported criteria. A simpler check is to see whether the $0.8239$ correlation survives on a held-out set of newly authored gold answers rather than the same 40 items.

Watch

Extended reading notes

Core claim

The central discovery is that real astronomers evaluate an LLM literature assistant through at least nine distinct question types and five evaluation criteria, and that those criteria can be operationalized into benchmark items. The most common queries were specific factual questions (33%), followed by deep-knowledge questions requiring summary, synthesis, or speculation (21%), and users kept asking bibliometric questions even when the bot performed worst on them. Across interviews, users consistently reported judging general correctness, including whether citations are relevant, recent, and accurately summarized; hedging and caveating; specificity and clarity; correct interpretation of the query; and broader impact on their workflow and research community. The paper demonstrates the criteria's practicality with a 40-item gold benchmark whose automated LLM-judge scoring is highly indicative of user satisfaction ($r = 0.8239$).

Load-bearing premise

Everything rests on the assumption that the 40 gold answers written by 7 astronomers and checked by 1, plus a single LLM judge's similarity score, faithfully represent the five evaluation criteria users actually reported, and that the self-selected users at one institute speak for astronomy users generally.

Editorial extensions

If this is right

  • Benchmarks for astronomy LLMs should include deep-knowledge, bibliometric, stress-testing, bot-capability, and unresolved-topic queries, not only closed-form factual ones.
  • Several user criteria can be proxied by automated signals: epistemic markers for hedging, entailment-based checks that a claim is supported by its citation, and closeness to a human-written gold answer for specificity and interpretation.
  • The released 40-item benchmark can compare system versions automatically; the authors report an upgraded retrieval system scoring 0.2–0.3 higher than the original.
  • Model development can iterate without running a new user study for every change, since the gold benchmark gives a fast proxy for satisfaction.
  • The user-grounded benchmark-building recipe is repeatable in other fields, especially observational sciences where evaluation risk is low.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single LLM-judge score likely folds the five reported criteria into one number; a testable extension is to score each criterion separately and compare them against user ratings dimension by dimension.
  • With 40 items and one self-selected research institute, the $0.8239$ correlation is suggestive rather than definitive; replicating on a second cohort or with more gold answers would show how stable the relationship is.
  • The finding that recent graduates stress-tested more and rated responses more positively implies that benchmarks may need career-stage stratification if they are meant to serve the whole community.
  • Because the correlation is computed on the same deployment that produced the queries, part of the signal could come from gold answers echoing the bot's own phrasing; independent human preference judgments on fresh queries would separate benchmark quality from circularity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents a user-centered study of an LLM-powered retrieval-augmented generation bot for astronomy literature, deployed on Slack at STScI. The authors log 368 real user queries over four weeks and conduct 11 semi-structured interviews. Through inductive coding they identify nine question types (e.g., Knowledge Seeking: specific factual, Deep Knowledge, Stress Testing) and five evaluation criteria that users apply (correctness, hedging/caveating, specificity/clarity, correct interpretation of the query, and broader impact). They translate these into a table mapping query types to measurable criteria, and construct a 40-question benchmark with expert-written gold answers and citations. The benchmark is validated by having gpt-4o-mini score similarity between original bot responses and gold answers, then correlating these scores with user thumbs-up/down counts; the paper reports Pearson's r=0.8239 and claims this shows automated evaluation is highly indicative of user satisfaction.

Significance. The qualitative contribution is substantial: the paper directly addresses a gap in LLM evaluation by grounding evaluation criteria in actual user behavior rather than assuming standard metrics. The data collection, coding procedure with reported inter-coder agreement, and public release of de-identified queries and the benchmark are strengths, and the mapping in Table 2 is a useful practical resource for benchmark designers. The main quantitative validation of the benchmark, however, rests on an under-reported correlation analysis whose load-bearing assumptions are not yet tested. If these issues are addressed, the paper would make a valuable contribution to human-centered evaluation of LLMs in scientific domains.

major comments (4)
  1. [§5.1 and Appendix F] The validation reports Pearson's r=0.8239 but never states the sample size N, the number of benchmark queries that actually had thumbs-up/down ratings, or a confidence interval. Since Table 1 indicates that many responses had no ratings, the effective N could be considerably smaller than 40, and r=0.8239 may not be distinguishable from chance at that N. Please report N, a two-sided 95% confidence interval or p-value, and a null baseline (e.g., correlation under permuted ratings, a constant judge score, or a length-based score). Without these details, the claim that automated evaluation is 'highly indicative of user satisfaction' is not supported by the evidence as presented.
  2. [§5.1 and Appendix F] The LLM judge gpt-4o-mini is from the same OpenAI GPT-4 model family as the gpt-4o bot that generated the evaluated responses. This invites a systematic bias where the judge rewards stylistic similarity to its own generation family rather than the substantive criteria users articulated in interviews. Moreover, the judge prompt in Appendix F uses only generic dimensions (accuracy, completeness, relevance, coherence, conciseness) and does not include user-identified criteria such as hedging/caveating, correct interpretation of query intent, or broader impact. Please run the correlation with a judge from a different model family (e.g., Anthropic Claude or an open-weight model), and include an ablation where the judge scores the response without seeing the gold answer, to test whether r=0.8239 is driven by gold-answer similarity or by generic response quality.
  3. [§5.1 and Table 2] The benchmark's query composition does not match the paper's own recommendation or the stated intention to 'trace the frequency distribution of our primary dataset.' The paper reports the benchmark contains approximately 55% Knowledge Seeking: specific factual, 22% Deep Knowledge, and 17% Knowledge Seeking: broad description, whereas the primary dataset (Table 1) has 33%, 21%, and 13% for these categories, and also contains Bibliometric Search (16%), Stress Testing (7%), Bot Capabilities (5%), Knowledge Seeking: procedure (2%), and Unresolved Topic (2%). The benchmark omits all of these latter categories, including Stress Testing and Bot Capabilities that Table 2 itself recommends for evaluating hedging/denial explanations and correct interpretation. Please either expand the benchmark to proportionally represent all query types or explicitly restrict the validation and generalization claims to the three included types.
  4. [§5.1] The validation is in-sample: the 40 benchmark queries are drawn from the same 368-query dataset whose thumbs-up/down ratings are used as the criterion, and the ratings come from the same self-selected group of STScI users who interacted with this specific bot. The r=0.8239 therefore reflects a within-corpus correlation and may not generalize to new queries, other users, or other RAG systems. To support the claim that automated evaluation using the benchmark is highly indicative of user satisfaction more generally, please provide an out-of-sample check, such as leave-one-user-out correlation, a held-out subset of queries not used in selecting gold answers, or an evaluation on responses from a different deployed system.
minor comments (4)
  1. [§3.2] The paper reports Krippendorff's alpha = 0.51 for the independent coding but does not state whether this is a pooled alpha across all nine categories or a per-category value. Given the taxonomy underlies the later benchmark recommendations, please clarify the computation and report whether agreement varied substantially across categories (e.g., stress testing vs. knowledge seeking).
  2. [§4.1.1] The claim that 'in 73% of cases the rating was thumbs-down' for re-asked questions is based on a small absolute number of re-asks (22 total, with ratings on only a subset). Please report the raw counts (e.g., 8 of 11 rated re-asks) so readers can judge the stability of this percentage.
  3. [Table 2] The layout of Table 2 is hard to parse because the 'Metric' and 'Query Type' columns are separated by a column break that may be ambiguous in print. Consider reformatting the table, for example by listing query types as explicit sub-bullets under each evaluation criterion, to improve readability.
  4. [§5.1] The sentence 'This strong correlation suggests that automated evaluation using our sample benchmark is highly indicative of user satisfaction' overstates the strength of the evidence given the unstated N and lack of baselines; consider rephrasing to 'correlates with' or 'is associated with' until the additional analyses are provided.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the query taxonomy and evaluation criteria come from user data and interviews, and the benchmark is validated empirically rather than by construction.

full rationale

The paper's central contributions are the inductively derived query-type taxonomy and the interview-based evaluation criteria, both of which are grounded in external user data (368 logged queries and 11 follow-up interviews) rather than in the benchmark itself. The 40-item benchmark is built by selecting real queries and having astronomers write gold answers, and the r=0.8239 validation is an empirical correlation between LLM-judge similarity scores and thumbs-up/down counts, not an identity or a parameter fit. The benchmark queries are drawn from the same primary dataset used for the correlation, which is a limitation for external validation but not a definitional reduction: the correlation could have been low, and the user ratings are not an input to the gold answers or to the judge score. The self-citations present (Wu et al. 2024 for the RAG framework and Iyer et al. 2024 for the upgraded bot) are system-description references and are not used to justify the taxonomy, the criteria, or the benchmark's validity. No uniqueness theorem is imported, no prior result is used to forbid alternatives, and no fitted parameter is renamed as a prediction. The reported limitations about generalizability and small sample size are acknowledged by the authors and affect external validity, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The study rests on qualitative inference from a small, self-selected, single-institution sample, and on the validity of thumbs-up/down and LLM-judge scores as proxies for user satisfaction. These are domain assumptions rather than mathematical axioms, and they are partially acknowledged in the Limitations section.

assumptions (4)
  • domain assumption Thumbs up/down emoji reactions on bot responses are a valid and sufficiently informative proxy for user satisfaction.
    Used to validate the benchmark in Section 5.1 (Pearson r=0.8239), but the paper itself notes in Section 4.2 that thumbs up/down ratings are insufficiently detailed to understand evaluation criteria.
  • domain assumption The 11 interviewed users and 35 active users are representative of the broader astronomy community's evaluation practices.
    The paper generalizes to astronomy and other observational sciences in Sections 1 and 6; the Limitations section acknowledges single-institute, opted-in, US-based, PhD-holding users.
  • domain assumption The inductive coding scheme with Krippendorff's alpha 0.51 yields categories reliable enough to support the conclusions.
    Section 3.2 reports alpha = 0.51, which is moderate agreement; disagreements were resolved by majority vote and discussion.
  • domain assumption The LLM judge (gpt-4o-mini) similarity scores to a single gold answer capture the multi-dimensional criteria users care about.
    Section 5.1 and Appendix F use this to claim the benchmark is indicative of user satisfaction; the gold answer is one expert's response, checked by one additional astronomer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Queries to Criteria: Understanding How Astronomers Evaluate LLMs." pith.science (2026). https://pith.science/paper/BXDLFMW6

@misc{pith2026250715715,
  author       = {Pith},
  title        = {Pith review of: From Queries to Criteria: Understanding How Astronomers Evaluate LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BXDLFMW6}},
  note         = {Machine review of arXiv:2507.15715}
}
read the original abstract

There is growing interest in leveraging LLMs to aid in astronomy and other scientific research, but benchmarks for LLM evaluation in general have not kept pace with the increasingly diverse ways that real people evaluate and use these models. In this study, we seek to improve evaluation procedures by building an understanding of how users evaluate LLMs. We focus on a particular use case: an LLM-powered retrieval-augmented generation bot for engaging with astronomical literature, which we deployed via Slack. Our inductive coding of 368 queries to the bot over four weeks and our follow-up interviews with 11 astronomers reveal how humans evaluated this system, including the types of questions asked and the criteria for judging responses. We synthesize our findings into concrete recommendations for building better benchmarks, which we then employ in constructing a sample benchmark for evaluating LLMs for astronomy. Overall, our work offers ways to improve LLM evaluation and ultimately usability, particularly for use in scientific research.

Figures

Figures reproduced from arXiv: 2507.15715 by the authors.

Figure 1
Figure 1. We deploy @Ask-astro-ph as a Slack bot and analyze resulting usage through inductive coding of user queries and follow-up interviews. We ultimately synthesize results into user-desired evaluation criteria, which are operationalizable in benchmark datasets. as an ideal test domain for investigating users queries to LLMs for scientific research for several reasons. First, the field contains rich open-access literature… view at source ↗
Figure 2
Figure 2. Question type and rating of bot re￾sponse for each user. Query bot about things they know 9 out of 11 users referenced querying the bot about things they know, primarily their spe￾cific area of expertise. An example is prob￾ing the bot for a question they knew was answerable from one of their recent publi￾cations: “So I knew[. . . ] I had a paper that was published during this period[. . . ] That it did have data av… view at source ↗
Figure 3
Figure 3. Frequency of question types by gender for men (24) and women (10), as self [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Frequency of question types by the year of PhD completion, as self-reported on an [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: Distribution of how users rated the chat bot separated by year of PhD completion [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Frequency of question type over time for each user. Each user asks a variety [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 16 canonical work pages

  1. [1]

    Kurtz, Edwin A

    Alberto Accomazzi, Michael J. Kurtz, Edwin A. Henneken, Roman Chyla, James Luker, Carolyn S. Grant, Donna M. Thompson, Alexandra Holachek, Rahul Dave, and Stephen S. Murray. ADS : The next generation search platform. Open Science at the Frontiers of Librarianship, 492: 0 189, 2015

    work page 2015

  2. [2]

    Henneken, Carolyn S

    Alberto Accomazzi, Edwin A. Henneken, Carolyn S. Grant, Donna M. Thompson, Matthew R. Templeton, Jennifer Koch, Sergi Blanco-Cuaresma, Roman Chyla, Stephen McDonald, Golnaz Shapurian, Timothy W. Hostetler, Kelly E. Lockhart, Nemanja Martinovi \' c , Shinyi Chen, Felix Grezes, and Michael J. Kurtz. Content of the future in the ADS . Bulletin of the AAS, 54...

    work page 2022

  3. [3]

    Openscholar: Synthesizing scientific literature with retrieval-augmented lms

    Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D'arcy, et al. Openscholar: Synthesizing scientific literature with retrieval-augmented lms. arXiv preprint arXiv:2411.14199, 2024

    arXiv 2024

  4. [4]

    Search B ots: User engagement with ChatBots during collaborative search

    Sandeep Avula, Gordon Chadwick, Jaime Arguello, and Robert Capra. Search B ots: User engagement with ChatBots during collaborative search. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval, CHIIR '18, pp.\ 52–61, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450349253. doi:10.1145/3176349.3176380. ...

    arXiv 2018

  5. [5]

    Embedding search into a conversational platform to support collaborative search

    Sandeep Avula, Jaime Arguello, Robert Capra, Jordan Dodson, Yuhui Huang, and Filip Radlinski. Embedding search into a conversational platform to support collaborative search. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, CHIIR '19, pp.\ 15–23, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 978145...

    arXiv 2019

  6. [6]

    The effects of system initiative during conversational collaborative search

    Sandeep Avula, Bogeum Choi, and Jaime Arguello. The effects of system initiative during conversational collaborative search. Proc. ACM Hum.-Comput. Interact., 6 0 (CSCW1), April 2022. doi:10.1145/3512913. URL https://doi.org/10.1145/3512913

    work page doi:10.1145/3512913 2022

  7. [7]

    Blackwell, Jon Barry, and Anthony G

    Robert E. Blackwell, Jon Barry, and Anthony G. Cohn. Towards reproducible llm evaluation: Quantifying uncertainty in llm benchmark scores, 2024. URL https://arxiv.org/abs/2410.03492

    arXiv 2024

  8. [8]

    Bowman and George Dahl

    Samuel R. Bowman and George Dahl. What will it take to fix benchmarking in natural language understanding? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association fo...

    2021

Show all 54 references
  1. [9]

    Toward algorithmic accountability in public services: A qualitative study of affected community perspectives on algorithmic decision-making in child welfare services

    Anna Brown, Alexandra Chouldechova, Emily Putnam-Hornstein, Andrew Tobin, and Rhema Vaithianathan. Toward algorithmic accountability in public services: A qualitative study of affected community perspectives on algorithmic decision-making in child welfare services. In Proceedi...

    2019

  2. [10]

    Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B. Chilton. How Novelists Use Generative Language Models : An Exploratory User Study . In Werner Geyer, Yasaman Khazaeni, and Michal Shmueli-Scheuer (eds.), Joint Proceedings of the Workshops on Human - AI Co - Creation ...

    2020

  3. [11]

    Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

    Alexander R. Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. S umm E val: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9: 0 391--409, 2021. doi:10.1162/tacl_a_00373....

    2021 doi

  4. [12]

    K. J. Kevin Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy X. Zhang, and David W. McDonald. Canvil: Designerly adaptation for llm-powered user experiences. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI '25, New York, NY, US...

    2025

  5. [13]

    Enabling large language models to generate text with citations

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 6465--6488, Sin...

    2023 doi

  6. [14]

    What can large language models do in chemistry? A comprehensive benchmark on eight tasks

    Taicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. What can large language models do in chemistry? A comprehensive benchmark on eight tasks. In Thirty-seventh Conference on Neural Information Processing Systems D...

    2023

  7. [15]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Tra...

    2021

  8. [16]

    Iyer, Mikaeel Yunus, Charles O’Neill, Christine Ye, Alina Hyk, Kiera McCormick, Ioana Ciucă, John F

    Kartheik G. Iyer, Mikaeel Yunus, Charles O’Neill, Christine Ye, Alina Hyk, Kiera McCormick, Ioana Ciucă, John F. Wu, Alberto Accomazzi, Simone Astarita, Rishabh Chakrabarty, Jesse Cranney, Anjalie Field, Tirthankar Ghosal, Michele Ginolfi, Marc Huertas-Company, Maja Jabłońska,...

    2024

  9. [17]

    Michael Katell, Meg Young, Dharma Dailey, Bernease Herman, Vivian Guetler, Aaron Tam, Corinne Bintz, Daniella Raz, and P. M. Krafft. Toward situated interventions for algorithmic equity: lessons from the field. In Proceedings of the 2020 Conference on Fairness, Accountability,...

    2020

  10. [18]

    On scientific understanding with artificial intelligence

    Mario Krenn, Robert Pollice, Si Yue Guo, Matteo Aldeghi, Alba Cervera-Lierta, Pascal Friederich, Gabriel dos Passos Gomes, Florian Häse, Adrian Jinich, AkshatKumar Nigam, Zhenpeng Yao, and Alán Aspuru-Guzik. On scientific understanding with artificial intelligence. Nature Revi...

    2022 doi

  11. [19]

    Wikibench: Community-driven data curation for AI evaluation on W ikipedia

    Tzu-Sheng Kuo, Aaron Lee Halfaker, Zirui Cheng, Jiwoo Kim, Meng-Hsin Wu, Tongshuang Wu, Kenneth Holstein, and Haiyi Zhu. Wikibench: Community-driven data curation for AI evaluation on W ikipedia. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI '...

    2024

  12. [20]

    Chapter 8 - interviews and focus groups

    Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. Chapter 8 - interviews and focus groups. In Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser (eds.), Research Methods in Human Computer Interaction (Second Edition), pp.\ 187--228. Morgan Kaufmann, Boston, secon...

    2017 doi

  13. [21]

    H alu E val: A large-scale hallucination evaluation benchmark for large language models

    Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. H alu E val: A large-scale hallucination evaluation benchmark for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural La...

    2023 doi

  14. [22]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Aco...

    2023

  15. [23]

    Rethinking model evaluation as narrowing the socio-technical gap

    Q Vera Liao and Ziang Xiao. Rethinking model evaluation as narrowing the socio-technical gap. arXiv preprint arXiv:2306.03100, 2023

    2023 arXiv

  16. [24]

    Vera Liao, Daniel Gruen, and Sarah Miller

    Q. Vera Liao, Daniel Gruen, and Sarah Miller. Questioning the ai: Informing design practices for explainable ai user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20, pp.\ 1–15, New York, NY, USA, 2020. Association for Compu...

    2020

  17. [25]

    S elf C heck GPT : Zero-resource black-box hallucination detection for generative large language models

    Potsawee Manakul, Adian Liusie, and Mark Gales. S elf C heck GPT : Zero-resource black-box hallucination detection for generative large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lang...

    2023 doi

  18. [26]

    Mathewson, Jaylen Pittman, and Richard Evans

    Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, New York, NY, ...

    2023

  19. [27]

    Wolf, Josh Andres, Michael Desmond, Narendra Nath Joshi, Zahra Ashktorab, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Evelyn Duesterwald, and Casey Dugan

    Michael Muller, Christine T. Wolf, Josh Andres, Michael Desmond, Narendra Nath Joshi, Zahra Ashktorab, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Evelyn Duesterwald, and Casey Dugan. Designing ground truth and the social life of labels. In Proceedings of the 2021 CHI Confere...

    2021

  20. [28]

    Nagaraj Mulukunte Narayana and M.K. Bhandi. Arxiv in the Open Access Era : its usage and impact on physics researchers. Bulletin of the AAS, 54 0 (2), apr 27 2022. https://baas.aas.org/pub/2022n2i016

    2022

  21. [29]

    Rodriguez Mendez, Thang Bui, Alyssa Goodman, Alberto Accomazzi, Jill Naiman, Jesse Cranney, Kevin Schawinski, and Roberta Raileanu

    Tuan Dung Nguyen, Yuan-Sen Ting, Ioana Ciuca, Charles O ' Neill, Ze-Chang Sun, Maja Jab o \'n ska, Sandor Kruk, Ernest Perkowski, Jack Miller, Jason Jason Jingsh Li, Josh Peek, Kartheik Iyer, Tomasz Rozanski, Pranav Khetarpal, Sharaf Zaman, David Brodrick, Sergio J. Rodriguez ...

    2023

  22. [30]

    emr QA : A large corpus for question answering on electronic medical records

    Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. emr QA : A large corpus for question answering on electronic medical records. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ' ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Method...

    2018 doi

  23. [31]

    B io R ead: A new dataset for biomedical reading comprehension

    Dimitris Pappas, Ion Androutsopoulos, and Haris Papageorgiou. B io R ead: A new dataset for biomedical reading comprehension. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani,...

    2018

  24. [32]

    B io MRC : A dataset for biomedical machine reading comprehension

    Dimitris Pappas, Petros Stavropoulos, Ion Androutsopoulos, and Ryan McDonald. B io MRC : A dataset for biomedical machine reading comprehension. In Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (eds.), Proceedings of the 19th SIGBioMed Worksh...

    2020 doi

  25. [33]

    Exploring temperature effects on large language models across various clinical tasks

    Dhavalkumar Patel, Prem Timsina, Ganesh Raut, Robert Freeman, Matthew Levin, Girish Nadkarni, Benjamin S Glicksberg, and Eyal Klang. Exploring temperature effects on large language models across various clinical tasks. medRxiv, pp.\ 2024--07, 2024

    2024

  26. [34]

    Smith, Huiling Liu, Kevin Schawinski, Kartheik Iyer, Ioana Ciucă, and UniverseTBD

    Ernest Perkowski, Rui Pan, Tuan Dung Nguyen, Yuan-Sen Ting, Sandor Kruk, Tong Zhang, Charlie O’Neill, Maja Jablonska, Zechang Sun, Michael J. Smith, Huiling Liu, Kevin Schawinski, Kartheik Iyer, Ioana Ciucă, and UniverseTBD. Astrollama-chat: Scaling astrollama with conversatio...

    2024 doi

  27. [35]

    Bender, Alex Hanna, and Amandalynne Paullada

    Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. Ai and the everything in the whole wide world benchmark. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1...

    2021

  28. [36]

    The effect of sampling temperature on problem solving in large language models

    Matthew Renze. The effect of sampling temperature on problem solving in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 7346--7356, Miami, Florida, USA, November 202...

    2024 doi

  29. [37]

    Improving evidence retrieval for automated explainable fact-checking

    Chris Samarinas, Wynne Hsu, and Mong Li Lee. Improving evidence retrieval for automated explainable fact-checking. In Avi Sil and Xi Victoria Lin (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

    2021 doi

  30. [38]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansf...

    2023

  31. [39]

    The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism, 2024

    Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism, 2024. URL https://arxiv.org/abs/2407.10457

    2024 arXiv

  32. [40]

    Scieval: a multi-level large language model evaluation benchmark for scientific research

    Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. Scieval: a multi-level large language model evaluation benchmark for scientific research. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth ...

    2024 doi

  33. [41]

    Nestor, Ali Soroush, Pierre A

    Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G. Nestor, Ali Soroush, Pierre A. Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F. Rousseau, Chunhua Weng, and Yifan Peng. Evaluating large language models on medical evidence summarization. npj Digital Medicine, 6 0 (1): 0 1–8,...

    2023 doi

  34. [42]

    FEVER : a large-scale dataset for fact extraction and VER ification

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER : a large-scale dataset for fact extraction and VER ification. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North A merican Chapter of the Asso...

    2018 doi

  35. [43]

    Group chat ecology in enterprise instant messaging: How employees collaborate through multi-user chat channels on slack

    Dakuo Wang, Haoyu Wang, Mo Yu, Zahra Ashktorab, and Ming Tan. Group chat ecology in enterprise instant messaging: How employees collaborate through multi-user chat channels on slack. Proc. ACM Hum.-Comput. Interact., 6 0 (CSCW1), April 2022. doi:10.1145/3512941. URL https://do...

    2022 doi

  36. [44]

    Evaluating large language models on academic literature understanding and review: An empirical study among early-stage scholars

    Jiyao Wang, Haolong Hu, Zuyuan Wang, Song Yan, Youyu Sheng, and Dengbo He. Evaluating large language models on academic literature understanding and review: An empirical study among early-stage scholars. In Proceedings of the CHI Conference on Human Factors in Computing System...

    2024

  37. [45]

    as an ai language model, i cannot

    Joel Wester, Tim Schrills, Henning Pohl, and Niels van Berkel. “as an ai language model, i cannot”: Investigating llm denials of user requests. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI '24, New York, NY, USA, 2024. Association for Computi...

    2024

  38. [46]

    John F. Wu, Alina Hyk, Kiera McCormick, Christine Ye, Simone Astarita, Elina Baral, Jo Ciuca, Jesse Cranney, Anjalie Field, Kartheik Iyer, Philipp Koehn, Jenn Kotler, Sandor Kruk, Michelle Ntampaka, Charles O'Neill, Joshua E. G. Peek, Sanjib Sharma, and Mikaeel Yunus. Designin...

    2024 arXiv

  39. [47]

    C-pack: Packed resources for general chinese embeddings

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, pp.\ ...

    2024

  40. [48]

    AI as an active writer: Interaction strategies with generated text in human- AI collaborative fiction writing

    Daijin Yang, Yanpeng Zhou, Zhiyuan Zhang, Toby Jia-Jun Li, and Ray LC. AI as an active writer: Interaction strategies with generated text in human- AI collaborative fiction writing. In Alison Smith-Renner and Ofra Amir (eds.), Joint Proceedings of the IUI 2022 Workshops: APEx-...

    2022

  41. [49]

    User-controlled knowledge fusion in large language models: Balancing creativity and hallucination, 2023

    Chen Zhang. User-controlled knowledge fusion in large language models: Balancing creativity and hallucination, 2023. URL https://arxiv.org/abs/2307.16139

    2023 arXiv

  42. [50]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. BERTS core: Evaluating text generation with BERT . In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SkeHuCVFDr

    2020

  43. [51]

    WildChat : 1 M C hat GPT interaction logs in the wild

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. WildChat : 1 M C hat GPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Bl8u7ZRlbM

    2024

  44. [52]

    Deconstructing NLG evaluation: Evaluation practices, assumptions, and their implications

    Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daum \'e III, Kaheer Suleman, and Alexandra Olteanu. Deconstructing NLG evaluation: Evaluation practices, assumptions, and their implications. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.),...

    2022

  45. [53]

    Navigating the grey area: How expressions of uncertainty and overconfidence affect language models

    Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lang...

    2023 doi

  46. [54]

    Relying on the unreliable: The impact of language models ' reluctance to express uncertainty

    Kaitlyn Zhou, Jena Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models ' reluctance to express uncertainty. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computatio...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.