Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants

Jonan Richards; Mairieli Wessel

REVIEW 3 major objections 6 minor 2 cited by

Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Combining simulated users with LLM-as-a-Judge can provide automatic human-centered evaluation of conversational software engineering assistants.

desk verdict A clearly written position paper that sensibly puts simulated users and LLM-as-a-Judge on a shared requirements table, but the central 'can fulfill' claim is an untested expectation, not a demonstrated result. read the letter →

arxiv 2502.07956 v1 pith:QGOWELVA submitted 2025-02-11 cs.SE

Jonan Richards , Mairieli Wessel This is my paper · ORCID

classification cs.SE

keywords conversationalagentssoftwareengineeringLLM-as-a-Judgesimulatedusershuman-centeredevaluationpersonasusabilityLLM-basedassistants

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that evaluating LLM-based conversational assistants for software engineering requires an automatic method that still centers human needs. It identifies four requirements — realistic multi-turn interactions, support for diverse users, quantitative metrics for rapid iteration, and qualitative insights for finding usability issues — and claims that neither simulated users nor LLM-as-a-Judge alone satisfies all of them. The proposed answer is a combined pipeline: persona-driven simulated users interact with the assistant and produce qualitative feedback, while a persona-conditioned LLM judge supplies quantitative scores. If this works, designers could test prompt changes and catch inclusivity bugs quickly during prototyping, complementing rather than replacing user studies.

What carries the argument

The load-bearing mechanism is a three-step evaluation loop run per persona. First, an LLM-simulated user, instructed with a persona description, converses with the target assistant over multiple turns to generate realistic interaction data. Second, a simulated user with the same persona produces qualitative feedback on the interaction, surfacing potential usability issues. Third, an LLM-as-a-Judge — also conditioned on the persona — scores the interaction against quantitative evaluation criteria. The paper defines personas as descriptions of users, grounded in existing methods such as GenderMag, and expects the persona set and criteria to evolve as designers learn from the results.

What would settle it

Run the proposed pipeline and a conventional user study on the same conversational SE assistant with the same personas and tasks; if the synthetic evaluations fail to surface usability issues the human study reveals, or if judge scores systematically disagree with human ratings, the central claim that the combined method provides human-centered evaluation is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that bridging HCI and AI evaluation practice — combining simulated users with LLM-as-a-Judge — can fulfill the requirements of realistic multi-turn exploration, user diversity, quantitative measurement, and qualitative feedback, delivering what the paper calls automatic human-centered evaluation of LLM-based conversational SE assistants. The paper states it is the first to propose this combination, with the novelty lying in the framing that allows practical insights to transfer between the HCI and AI fields. The proposal is explicitly meant to complement, not replace, expert analysis and user studies.

Load-bearing premise

The whole pipeline assumes that LLM-simulated users and persona-conditioned LLM judges behave and judge like real, diverse software developers, so that synthetic interactions and scores reflect genuine usability and inclusivity.

Editorial extensions

If this is right

Designers of conversational SE assistants could test prompt and feature changes rapidly during prototyping, without the cost of a full user study for every iteration.
Evaluation could cover a wider range of developer backgrounds, experience levels, and interaction styles, surfacing inclusivity bugs that single-perspective evaluations miss.
Because both simulated interactions and judge scores can be regenerated for many paraphrase variations, quantitative metrics would reflect changes in prompt structure rather than wording artifacts.
The method would still require periodic validation against real user studies to keep personas, interactions, and judgments aligned with actual developers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the combined method proves reliable, it could extend beyond software engineering to any domain where conversational agents are prototyped, such as educational tutors or customer support, because nothing in the pipeline is SE-specific.
The paper's own admission that simulated users hallucinate suggests a testable extension: measuring how often simulated interactions produce non-prototypical developer behavior, and whether such hallucinations degrade judge scores.
A natural next step the paper leaves implicit is comparing coverage: the set of usability issues found by the synthetic pipeline versus a real user study, weighted by severity, to decide whether the synthetic method can safely gate releases.
Since persona-conditioned judges are shown to approximate individual preferences, the pipeline might eventually be used to predict not just average quality but per-persona satisfaction, informing adaptive assistant behavior.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a quick read if you work on evaluation of AI coding assistants. The paper is a position piece: it pulls together two evaluation paradigms that usually live in separate communities, states four requirements for automatic human-centered evaluation of conversational SE assistants, and argues that combining simulated users with LLM-as-a-Judge can meet all four. The synthesis is competent and the writing is clear. The Table 1 comparison of where each approach currently falls short (R1 for judges, R4 for sim users, question marks on the other two) is a useful framing, and the proposed three-step pipeline is concrete enough to implement. Credit goes to the authors for explicitly acknowledging the biggest threat: there is no data, no implementation, and no comparison against real user studies. They state plainly that repeated comparison with pilot user studies is necessary, and they list hallucination and bias in both techniques as open challenges. That honesty keeps the paper from overclaiming.

The soft spot is the load-bearing assertion in Section IV.A: 'combining the two methods can fulfill the discussed requirements.' That is an expectation, not a finding. The paper also concedes that a persona-conditioned judge is arguably equivalent to a simulated user giving quantitative assessments, which somewhat weakens the novelty claim. A second concern, which the paper does not fully confront, is internal consistency: the same underlying LLM generates the simulated interactions, the qualitative feedback, and the judge scores. Without an external anchor, the pipeline could be coherent with itself while diverging from real developers. The authors point at this indirectly through the bias discussion, but they do not address the risk that the combined method could systematically certify an assistant that real users would reject.

For what it is, the paper is fine. It does not present a falsifiable research result, so I would not call it a strong empirical contribution. But the requirements list and the comparison table are useful scaffolding for the subfield, and the paper's own limitations are stated rather than hidden. The citation pattern looks reasonable; the only self-citation is an example of an inclusivity bug and is not load-bearing.

Who is this for? Researchers building or evaluating LLM-based SE instruments who want a checklist and a proposed pipeline before running their own studies. It deserves serious peer review as a vision or position paper, with the expectation that the stated claim will be tested later. I would not cite it as evidence for an effect, but I might cite it as a proposed methodology. Send it to review.

Referee Report

3 major / 6 minor

Summary. This paper is a position/vision paper arguing that automatic, human-centered evaluation of LLM-based conversational software-engineering assistants can be achieved by combining two currently separate lines of work: LLM-based simulated users (from HCI) and LLM-as-a-Judge (from AI). The authors identify four requirements for such evaluation (R1 realistic multi-turn conversations, R2 diversity of users, R3 quantitative metrics, R4 qualitative insights), review each technique against these requirements, and propose a three-step pipeline in which a persona-driven simulated user interacts with the assistant, produces qualitative feedback, and then an LLM-as-a-Judge scores the interaction. They also list three challenges and position the method as complementing, not replacing, human evaluation.

Significance. If the central claim were validated, the paper would make a useful contribution by connecting HCI and AI evaluation practices and by offering a scalable approach to formative evaluation of conversational SE assistants. The stated requirements are clearly motivated, and the literature synthesis is broad and relevant, covering simulated users, LLM-as-a-Judge, inclusivity bugs, and persona-based methods. However, the paper provides no empirical evidence for its central claim, and the pipeline as described leaves important methodological threats unaddressed. Its main value at this stage is the requirement framework and the cross-community synthesis, not a demonstrated evaluation method.

major comments (3)

[Section IV.A (first paragraph)] The load-bearing claim that 'combining the two methods can fulfill the discussed requirements' is asserted rather than demonstrated. Table I assigns '?' to R3 for simulated users and R4 for LLM-as-a-Judge, and Section V concedes simulated-user hallucination and task-dependent judge performance. Because the paper does not show that the two methods' limitations are exactly complementary, the three-step pipeline should be presented as a hypothesis with falsifiable success criteria, and a concrete validation protocol (for example, comparing synthetic interactions, feedback, and scores against a pilot user study) should be part of the proposal, not only future work.
[Section IV.B, C1/C2] The method's claim to support diversity (R2) rests on the representativeness of personas and simulated users, but the paper cites evidence of biased LLM simulations [37], [38], an inability to portray identity groups [39], and occasional hallucination [16]. Without an anchoring procedure that checks the synthetic data against real users, the pipeline could certify an assistant against synthetic users while missing actual usability failures. The paper mentions repeated comparison with pilot user studies in Section IV.A, but this step is not integrated into the proposed method or operationalized; the revision should specify how persona representativeness and inclusivity-bug coverage are measured.
[Section IV.A, Steps 1-3] The pipeline uses the same underlying LLM for the simulated user, the qualitative feedback, and the judge. This creates a common-source bias: agreement between the simulated user and the LLM-as-a-Judge may reflect shared model priors rather than alignment with human users. The paper does not address this threat, which is central to the phrase 'automatic human-centered evaluation.' The method needs an explicit external anchor, such as real interaction logs, human-rated examples, or a calibration corpus, to distinguish internally consistent synthetic evaluation from human-centered evaluation.

minor comments (6)

[Section III.A, first paragraph] The phrase 'LLM-simulated simulated user interactions' contains a duplicated word and should be corrected to 'LLM-simulated user interactions'.
[Section III.B, final paragraph] The sentence ending 'qualitative insights R4)' is missing the opening parenthesis before 'R4'.
[Section V, final paragraph] 'In should be noted' should read 'It should be noted'.
[Section IV.A, paragraph 1] The claim 'We are the first to propose combining simulated users and LLM-as-a-Judge' is a priority claim that the paper does not substantiate with a search protocol; consider softening it to 'To our knowledge' or providing the search details.
[Reference [9]] The journal name contains a formatting artifact, 'F oundations'; this should be corrected to 'Foundations'.
[Figure 2] The caption states that reference-free datasets 'miss conversations where responses refer to earlier messages,' but the left panel shows a flat dataset and the right panel shows a single interaction; a more explicit callout of the coreference (e.g., 'What does in-place mean?') would make the point clearer.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the proposed method is a literature-grounded synthesis, not a self-referential derivation; the only self-citations are non-load-bearing background evidence.

full rationale

The paper contains no equations, no fitted parameters, and no quantitative predictions. Its load-bearing assertion (Section IV.A) is that combining simulated users and LLM-as-a-Judge can satisfy requirements R1–R4; this is supported by external empirical studies of each method ([16], [17], [19], [20], [29]–[31], [35]) rather than by the authors' own results. The requirements themselves are derived from HCI and AI evaluation literature, not defined in terms of the proposed pipeline, so there is no self-definitional reduction. The only author self-citations ([3], [25]) provide background examples of multi-task bots and inclusivity bugs; neither is used to justify the central claim, and the inclusivity-bug point is corroborated by independent citations ([8], [24]). The acknowledged risks—hallucination in simulated users and bias in LLM judges (Sections III.A, IV.B, C2)—are stated as open challenges requiring external validation, not as grounds for claiming success. The paper even plans to compare synthesized results against a real user study (Section V), confirming that the central assertion is an empirical proposal rather than a conclusion forced by construction. The strongest criticism is an unvalidated representativeness assumption, which is a correctness/validity concern, not circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No mathematical model, fit parameters, or new entities are introduced. The proposal rests entirely on empirical assumptions imported from cited studies concerning realism of LLM-simulated users and reliability of LLM-as-a-Judge, plus the unvalidated assumption that the two methods compose cleanly.

assumptions (5)

domain assumption LLM-based simulated users can produce realistic, believable dialogue and qualitative usability feedback in HCI settings.
Imported from cited empirical studies (Hamalainen et al. [29], Xiang et al. [16], de Wit [17]); used in Section III.A to support R1 and R4 for simulated users.
domain assumption LLM-as-a-Judge assessments align well with human judgments, including for software engineering artifacts.
Imported from Zheng et al. [19] and Ahmed et al. [20]; used in Section III.B to support R3 for judge models.
domain assumption Persona-conditioned LLMs can predict individual preferences, making judge models effective for diverse users.
Imported from Hu and Collier [34] and Dong et al. [35]; used in Section III.B to support R2.
ad hoc to paper The two methods can be combined in sequence without introducing interaction effects that invalidate the evaluation, and their respective gaps cover each other.
This is the core unvalidated premise of the proposed method in Section IV.A; no experiment compares the combined pipeline against real user studies.
domain assumption Existing persona frameworks (GenderMag, Microsoft's knowledge worker personas) can be extended to conversational SE assistants.
Assumed in Section IV.B, C1; the authors themselves note that more research is needed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants." pith.science (2026). https://pith.science/paper/QGOWELVA

@misc{pith2026250207956,
  author       = {Pith},
  title        = {Pith review of: Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGOWELVA}},
  note         = {Machine review of arXiv:2502.07956}
}

read the original abstract

As Large Language Models (LLMs) are increasingly adopted in software engineering, recently in the form of conversational assistants, ensuring these technologies align with developers' needs is essential. The limitations of traditional human-centered methods for evaluating LLM-based tools at scale raise the need for automatic evaluation. In this paper, we advocate combining insights from human-computer interaction (HCI) and artificial intelligence (AI) research to enable human-centered automatic evaluation of LLM-based conversational SE assistants. We identify requirements for such evaluation and challenges down the road, working towards a framework that ensures these assistants are designed and deployed in line with user needs.

Figures

Figures reproduced from arXiv: 2502.07956 by the authors.

**Figure 2.** Reference-free datasets miss conversations where responses refer to [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues
cs.CL 2025-06 conditional novelty 7.0 of 10

A new benchmark and metric show that large language models still struggle to call tools when the needed details are scattered across multi-party, multi-round group dialogues.
Evaluating the Effectiveness of Large Language Models in Solving Simple Programming Tasks: A User-Centered Study
cs.HC 2025-07 reject novelty 5.0 of 10

A within-subjects experiment with 15 high school students reports faster task completion with a collaborative ChatGPT-4o style than with a passive style, but the result is not significant versus proactive style and is...

Reference graph

Works this paper leans on

41 extracted references · 29 canonical work pages · cited by 2 Pith papers

[37]

Out of One, Many: Using Language Models to Simulate Human Samples,

L. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate, “Out of One, Many: Using Language Models to Simulate Human Samples,” Political Analysis , vol. 31, no. 3, pp. 337–351, Jul. 2023

work page 2023
[38]

Systematic Biases in LLM Simulations of Debates,

A. Taubenfeld, Y . Dover, R. Reichart, and A. Goldstein, “Systematic Biases in LLM Simulations of Debates,” Feb. 2024, arXiv:2402.04049 [cs]

arXiv 2024
[39]

Large language models cannot replace human participants because they cannot portray identity groups,

A. Wang, J. Morgenstern, and J. P. Dickerson, “Large language models cannot replace human participants because they cannot portray identity groups,” Feb. 2024, arXiv:2402.01908 [cs]

arXiv 2024
[16]

SimUser: Generating Usability Feedback by Simulating Various Users Interacting with Mobile Applications,

W. Xiang, H. Zhu, S. Lou, X. Chen, Z. Pan, Y . Jin, S. Chen, and L. Sun, “SimUser: Generating Usability Feedback by Simulating Various Users Interacting with Mobile Applications,” in Proceedings of the CHI Conference on Human Factors in Computing Systems . Honolulu HI USA: ACM, May 2024, pp. 1–17

work page 2024
[1]

Current and Future Bots in Software Development,

L. Erlenhov, F. Gomes de Oliveira Neto, R. Scandariato, and P. Leitner, “Current and Future Bots in Software Development,” in2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE) , May 2019, pp. 7–11

work page 2019
[2]

Large Language Models for Software Engineering: A Systematic Literature Review,

X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large Language Models for Software Engineering: A Systematic Literature Review,” ACM Transactions on Software Engineering and Methodology , p. 3695988, Sep. 2024

work page 2024
[3]

Bots for pull requests: the good, the bad, and the promising,

M. Wessel, A. Abdellatif, I. Wiese, T. Conte, E. Shihab, M. A. Gerosa, and I. Steinmacher, “Bots for pull requests: the good, the bad, and the promising,” in Proceedings of the 44th International Conference on Software Engineering . Pittsburgh Pennsylvania: ACM, May 2022, pp. 274–286

work page 2022
[4]

Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models,

H. Li, C.-P. Bezemer, and A. E. Hassan, “Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models,” Oct. 2024, arXiv:2410.09012

arXiv 2024

Show all 41 references

[5]

Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes,

G. Pinto, C. De Souza, T. Rocha, I. Steinmacher, A. Souza, and E. Monteiro, “Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes,” in Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engine...

2024
[6]

The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development,

S. I. Ross, F. Martinez, S. Houde, M. Muller, and J. D. Weisz, “The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development,” in Proceedings of the 28th International Conference on Intelligent User Interfaces . Sydney NSW Austral...

2023
[7]

Program Synthesis with Large Language Models,

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program Synthesis with Large Language Models,” Aug. 2021, arXiv:2108.07732 [cs]

2021 arXiv
[8]

How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering,

R. Choudhuri, D. Liu, I. Steinmacher, M. Gerosa, and A. Sarma, “How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . Lisbon Portugal: ACM, Apr. 2024, pp. 1–13

2024
[9]

How practitioners perceive the relevance of software engineering research,

D. Lo, N. Nagappan, and T. Zimmermann, “How practitioners perceive the relevance of software engineering research,” in Proceedings of the 2015 10th Joint Meeting on F oundations of Software Engineering , ser. ESEC/FSE 2015. New York, NY , USA: Association for Computing Machine...

2015
[10]

Programmers Are Users Too: Human-Centered Methods for Improving Programming Tools,

B. A. Myers, A. J. Ko, T. D. LaToza, and Y . Yoon, “Programmers Are Users Too: Human-Centered Methods for Improving Programming Tools,” Computer, vol. 49, no. 7, pp. 44–52, Jul. 2016

2016
[11]

What’s (Not) Working in Programmer User Studies?

M. C. Davis, E. Aghayi, T. D. Latoza, X. Wang, B. A. Myers, and J. Sunshine, “What’s (Not) Working in Programmer User Studies?”ACM Transactions on Software Engineering and Methodology , vol. 32, no. 5, pp. 120:1–120:32, Jul. 2023

2023
[12]

Conversational Agents: Goals, Technologies, Vision and Challenges,

M. Allouch, A. Azaria, and R. Azoulay, “Conversational Agents: Goals, Technologies, Vision and Challenges,” Sensors, vol. 21, no. 24, p. 8448, Jan. 2021

2021
[13]

Human-Centered Design Recommen- dations for LLM-as-a-judge,

Q. Pan, Z. Ashktorab, M. Desmond, M. Santillán Cooper, J. Johnson, R. Nair, E. Daly, and W. Geyer, “Human-Centered Design Recommen- dations for LLM-as-a-judge,” in Proceedings of the 1st Human-Centered Large Language Modeling Workshop , N. Soni, L. Flek, A. Sharma, D. Yang, S....

2024
[14]

PromptMaker: Prompt-based Prototyping with Large Language Models,

E. Jiang, K. Olson, E. Toh, A. Molina, A. Donsbach, M. Terry, and C. J. Cai, “PromptMaker: Prompt-based Prototyping with Large Language Models,” in CHI Conference on Human Factors in Computing Systems Extended Abstracts. New Orleans LA USA: ACM, Apr. 2022, pp. 1–8

2022
[15]

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation,

C.-W. Liu, R. Lowe, I. Serban, M. Noseworthy, L. Charlin, and J. Pineau, “How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation,” in Proceedings of the 2016 Conference on Empirical Methods in Natural La...

2016
[17]

Leveraging Large Language Models as Simulated Users for Initial, Low-Cost Evaluations of Designed Conversations,

J. de Wit, “Leveraging Large Language Models as Simulated Users for Initial, Low-Cost Evaluations of Designed Conversations,” in Chat- bot Research and Design , A. Følstad, T. Araujo, S. Papadopoulos, E. L.- C. Law, E. Luger, M. Goodwin, S. Hobert, and P. B. Brandtzaeg, Eds. C...

2024
[18]

Can AI serve as a substitute for human subjects in software engineering research?

M. Gerosa, B. Trinkenreich, I. Steinmacher, and A. Sarma, “Can AI serve as a substitute for human subjects in software engineering research?” Automated Software Engineering , vol. 31, no. 1, p. 13, Jan. 2024

2024
[19]

Judging LLM-as-a-judge with MT-bench and Chatbot Arena,

L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-bench and Chatbot Arena,” in Proceedings of the 37th International Conference on Neural Information Proce...

2024
[20]

Can LLMs Replace Manual Annotation of Software Engineering Artifacts?

T. Ahmed, P. Devanbu, C. Treude, and M. Pradel, “Can LLMs Replace Manual Annotation of Software Engineering Artifacts?” Aug. 2024, arXiv:2408.05534

2024 arXiv
[21]

GenderMag: A Method for Evaluating Software’s Gender Inclusiveness,

M. Burnett, S. Stumpf, J. Macbeth, S. Makri, L. Beckwith, I. Kwan, A. Peters, and W. Jernigan, “GenderMag: A Method for Evaluating Software’s Gender Inclusiveness,” Interacting with Computers , vol. 28, no. 6, pp. 760–787, Nov. 2016

2016
[22]

How to debug inclusivity bugs?: a debugging process with information architecture,

M. Guizani, I. Steinmacher, J. Emard, A. Fallatah, M. Burnett, and A. Sarma, “How to debug inclusivity bugs?: a debugging process with information architecture,” in Proceedings of the 2022 ACM/IEEE 44th International Conference on Software Engineering: Software Engineer- ing i...

2022
[23]

Guidelines for Human-AI Interaction,

S. Amershi, D. Weld, M. V orvoreanu, A. Fourney, B. Nushi, P. Collisson, J. Suh, S. Iqbal, P. N. Bennett, K. Inkpen, J. Teevan, R. Kikin-Gil, and E. Horvitz, “Guidelines for Human-AI Interaction,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems ...

2019
[24]

Using an LLM to Help With Code Understanding,

D. Nam, A. Macvean, V . Hellendoorn, B. Vasilescu, and B. Myers, “Using an LLM to Help With Code Understanding,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . Lisbon Portugal: ACM, Apr. 2024, pp. 1–13

2024
[25]

What You Need is What You Get: Theory of Mind for an LLM-Based Code Understanding Assistant,

J. Richards and M. Wessel, “What You Need is What You Get: Theory of Mind for an LLM-Based Code Understanding Assistant,” in 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2024, pp. 666–671

2024
[26]

Safeguarding Large Language Models: A Survey,

Y . Dong, R. Mu, Y . Zhang, S. Sun, T. Zhang, C. Wu, G. Jin, Y . Qi, J. Hu, J. Meng, S. Bensalem, and X. Huang, “Safeguarding Large Language Models: A Survey,” Jun. 2024, arXiv:2406.02622

2024 arXiv
[27]

Large Language Models can Accurately Predict Searcher Preferences,

P. Thomas, S. Spielman, N. Craswell, and B. Mitra, “Large Language Models can Accurately Predict Searcher Preferences,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . Washington DC USA: ACM, Jul. 2024, pp. 1930–1940

2024
[28]

Trends, Challenges and Processes in Conversational Agent Design: Exploring Practitioners’ Views through Semi-Structured Interviews,

M. Sadek, R. A. Calvo, and C. Mougenot, “Trends, Challenges and Processes in Conversational Agent Design: Exploring Practitioners’ Views through Semi-Structured Interviews,” in Proceedings of the 5th International Conference on Conversational User Interfaces . Eind- hoven Neth...

2023
[29]

Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study,

P. Hämäläinen, M. Tavast, and A. Kunnari, “Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . Hamburg Germany: ACM, Apr. 2023, pp. 1–19

2023
[30]

Simulating Social Media Using Large Language Models to Evaluate Alternative News Feed Algorithms,

P. Törnberg, D. Valeeva, J. Uitermark, and C. Bail, “Simulating Social Media Using Large Language Models to Evaluate Alternative News Feed Algorithms,” Oct. 2023, arXiv:2310.05984 [cs]

2023 arXiv
[31]

TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Students,

H. Jin, M. Yoo, J. Park, Y . Lee, X. Wang, and J. Kim, “TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Students,” Oct. 2024, arXiv:2410.04078

2024 arXiv
[32]

Can ChatGPT emulate humans in software engineering sur- veys?

I. Steinmacher, J. M. Penney, K. R. Felizardo, A. F. Garcia, and M. A. Gerosa, “Can ChatGPT emulate humans in software engineering sur- veys?” in Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , ser. ESEM ’24. New Yor...

2024
[33]

Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards,

H. Wang, Y . Lin, W. Xiong, R. Yang, S. Diao, S. Qiu, H. Zhao, and T. Zhang, “Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards,” Mar. 2024, arXiv:2402.18571 [cs, stat]

2024 arXiv
[34]

Quantifying the Persona Effect in LLM Simula- tions,

T. Hu and N. Collier, “Quantifying the Persona Effect in LLM Simula- tions,” Jun. 2024, arXiv:2402.10811 [cs]

2024 arXiv
[35]

Can LLM be a Personalized Judge?

Y . R. Dong, T. Hu, and N. Collier, “Can LLM be a Personalized Judge?” Jun. 2024, arXiv:2406.11657 [cs]

2024 arXiv
[36]

Characterizing Software Engineering Work with Personas Based on Knowledge Worker Actions,

D. Ford, T. Zimmermann, C. Bird, and N. Nagappan, “Characterizing Software Engineering Work with Personas Based on Knowledge Worker Actions,” in 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) , Nov. 2017, pp. 394– 403

2017
[40]

Supporting Con- textual Conversational Agent-Based Software Development,

G. Melo, L. F. Lins, P. Alencar, and D. Cowan, “Supporting Con- textual Conversational Agent-Based Software Development,” in 2023 IEEE/ACM 5th International Workshop on Bots in Software Engineering (BotSE), May 2023, pp. 9–13

2023
[41]

Retrieval-Augmented Generation for Large Language Models: A Survey,

Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, M. Wang, and H. Wang, “Retrieval-Augmented Generation for Large Language Models: A Survey,” Mar. 2024, arXiv:2312.10997 [cs]

2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [37]

Out of One, Many: Using Language Models to Simulate Human Samples,

L. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate, “Out of One, Many: Using Language Models to Simulate Human Samples,” Political Analysis , vol. 31, no. 3, pp. 337–351, Jul. 2023

work page 2023

[2] [38]

Systematic Biases in LLM Simulations of Debates,

A. Taubenfeld, Y . Dover, R. Reichart, and A. Goldstein, “Systematic Biases in LLM Simulations of Debates,” Feb. 2024, arXiv:2402.04049 [cs]

arXiv 2024

[3] [39]

Large language models cannot replace human participants because they cannot portray identity groups,

A. Wang, J. Morgenstern, and J. P. Dickerson, “Large language models cannot replace human participants because they cannot portray identity groups,” Feb. 2024, arXiv:2402.01908 [cs]

arXiv 2024

[4] [16]

SimUser: Generating Usability Feedback by Simulating Various Users Interacting with Mobile Applications,

W. Xiang, H. Zhu, S. Lou, X. Chen, Z. Pan, Y . Jin, S. Chen, and L. Sun, “SimUser: Generating Usability Feedback by Simulating Various Users Interacting with Mobile Applications,” in Proceedings of the CHI Conference on Human Factors in Computing Systems . Honolulu HI USA: ACM, May 2024, pp. 1–17

work page 2024

[5] [1]

Current and Future Bots in Software Development,

L. Erlenhov, F. Gomes de Oliveira Neto, R. Scandariato, and P. Leitner, “Current and Future Bots in Software Development,” in2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE) , May 2019, pp. 7–11

work page 2019

[6] [2]

Large Language Models for Software Engineering: A Systematic Literature Review,

X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large Language Models for Software Engineering: A Systematic Literature Review,” ACM Transactions on Software Engineering and Methodology , p. 3695988, Sep. 2024

work page 2024

[7] [3]

Bots for pull requests: the good, the bad, and the promising,

M. Wessel, A. Abdellatif, I. Wiese, T. Conte, E. Shihab, M. A. Gerosa, and I. Steinmacher, “Bots for pull requests: the good, the bad, and the promising,” in Proceedings of the 44th International Conference on Software Engineering . Pittsburgh Pennsylvania: ACM, May 2022, pp. 274–286

work page 2022

[8] [4]

Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models,

H. Li, C.-P. Bezemer, and A. E. Hassan, “Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models,” Oct. 2024, arXiv:2410.09012

arXiv 2024