pith. machine review for the scientific record. sign in

arxiv: 2605.02624 · v1 · submitted 2026-05-04 · 💻 cs.CL

Recognition: 2 theorem links

· Lean Theorem

Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversations

Hyokun Yun, Tanya Roosta, Yu Lu Liu, Ziang Xiao

Authors on Pith no claims yet

Pith reviewed 2026-05-08 19:22 UTC · model grok-4.3

classification 💻 cs.CL
keywords user simulationdialogue evaluationchatbot realismmulti-turn conversationscommunication frictionssynthetic usersevaluation frameworkdomain variability
0
0 comments X

The pith

Simulated users miss communication frictions that real users introduce, making chatbot evaluations overly optimistic.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes realsim, a framework that compares real and simulated multi-turn user-chatbot dialogues along eight dimensions spanning communicative functions, user states, and message surface forms. It applies this to a curated set of one thousand real dialogues covering sixteen domains and finds that current simulators consistently under-represent the frictions and variability real users create. This gap matters because many chatbot evaluations now rely on simulation in place of live user testing. The analysis also reveals performance differences across domains, indicating that generic simulators may not generalize well.

Core claim

The authors establish that simulated users struggle to capture the communication frictions real users introduce during multi-turn chatbot interactions, which risks producing overly optimistic evaluations of chatbot performance, and that simulator quality varies substantially across the sixteen application domains examined.

What carries the argument

The realsim framework, which produces distributional comparisons of real versus simulated dialogues across eight dimensions of communicative functions, user states, and surface form.

If this is right

  • Evaluations that rely on current simulated users may overestimate how well chatbots handle real interactions.
  • User simulators require specific improvements in modeling communication frictions rather than only task completion.
  • Domain-specific simulators are likely needed, given the observed performance variability.
  • The framework supplies a reusable benchmark for testing future simulators against real distributional patterns.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Teams developing production chatbots may need to retain some real-user testing in domains where friction modeling is critical.
  • Hybrid evaluation pipelines that alternate between simulation and targeted real-user sampling could reduce optimism bias.
  • The eight-dimension lens could be extended to measure whether newer simulation techniques close the friction gap over time.

Load-bearing premise

The eight chosen dimensions are sufficient to reveal the main differences between real and simulated dialogues and the one-thousand-dialogue dataset is representative of interactions in the sixteen domains.

What would settle it

A new simulation method that produces dialogues statistically indistinguishable from real ones on all eight dimensions when measured on an independent collection of multi-turn conversations would falsify the claim that simulators inherently struggle with frictions.

Figures

Figures reproduced from arXiv: 2605.02624 by Hyokun Yun, Tanya Roosta, Yu Lu Liu, Ziang Xiao.

Figure 1
Figure 1. Figure 1: Illustration of the realsim framework. instantiate the framework with a curated dataset of 1K multi-turn task-focused real user￾chatbot dialogues that cover 16 domains of chatbot applications, allowing us to gather domain-specific insights about the behaviors of simulated users. As illustrated in view at source ↗
Figure 2
Figure 2. Figure 2: Category Frequency for Intent, Identity Feedback, and Emotion. view at source ↗
Figure 3
Figure 3. Figure 3: Heatmap of Knowledge Statements Semantic Similarity Scores Between Real vs. view at source ↗
Figure 4
Figure 4. Figure 4: Intent and Identity Distributions for Job Application Domain. view at source ↗
Figure 5
Figure 5. Figure 5: Frequency Variation Across Domains (domains ordered according to Real fre view at source ↗
Figure 6
Figure 6. Figure 6: Intent Heatmap for i) TVD Scores Across Domains and ii) Correlation Scores view at source ↗
Figure 7
Figure 7. Figure 7: Identity Heatmap for i) TVD Scores Across Domains and ii) Correlation Scores view at source ↗
read the original abstract

There is growing interest in exploring user simulation as an alternative to gathering and scoring real user-chatbot interactions for AI chatbot evaluation. For this purpose, it is important to ensure the realism of the simulation, i.e., the extent to which simulated dialogues reflect real dialogues users have with chatbots. Most existing methods evaluating simulation realism produce coarse quality signal and remain solely at the level of individual dialogues. To support more rigorous evaluation in this area, we propose realsim, an evaluation framework that enables practitioners to take a distributional view of real vs. simulated dialogues along 8 dimensions, covering attributes related to the communicative functions of the interaction, user states, and the surface form of user messages. We then instantiate the framework with a curated dataset of 1K multi-turn task-focused real user-chatbot dialogues that cover 16 domains of chatbot applications. Overall, we find that simulated users tend to struggle at capturing communication frictions that real users introduce to interactions, which could make evaluations based on such simulations overly optimistic. We also observe variability in performance across different domains, which may indicate a need for domain-specific user simulators.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The manuscript introduces realsim, a framework for evaluating the realism of user simulators for multi-turn conversations by comparing distributions of real and simulated dialogues across eight dimensions encompassing communicative functions, user states, and surface forms. Using a curated dataset of 1K real dialogues from 16 domains, the authors find that simulated users under-capture communication frictions, potentially leading to overly optimistic chatbot evaluations, and observe domain variability.

Significance. This framework offers a more granular, distributional approach to assessing simulation quality than existing coarse methods, which could enhance the validity of simulation-based evaluations in chatbot development. The multi-domain dataset is a notable resource. If the dimensions prove predictive of evaluation biases, the findings would encourage development of better simulators that handle real-user frictions. The work highlights important limitations in current simulation techniques.

major comments (3)
  1. Abstract, final paragraph: The claim that simulated users struggle to capture communication frictions 'which could make evaluations based on such simulations overly optimistic' is not backed by any direct evidence linking the observed differences in the 8 dimensions to actual discrepancies in chatbot evaluation outcomes, such as human preference scores or task completion rates.
  2. Dataset section (likely §4): The 1K dataset curation process does not specify how it ensures balanced representation of friction-inducing interactions (e.g., multi-turn clarifications or error recoveries) across the 16 domains, and no pre-specified analysis plan for the observed domain variability is mentioned, weakening the generalizability of the findings.
  3. Framework and Experiments sections (likely §3 and §5): No details are provided on the operationalization and measurement of the eight dimensions, including any statistical tests, inter-annotator agreement, or validation against external criteria like human realism ratings, making it difficult to assess the robustness of the reported differences.
minor comments (1)
  1. Clarify the exact operational definitions and annotation procedures for each of the eight dimensions to improve reproducibility.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for their constructive feedback, which highlights important areas for clarification and strengthening. We address each major comment below and will incorporate revisions to improve the manuscript's rigor and transparency.

read point-by-point responses

  1. Referee: Abstract, final paragraph: The claim that simulated users struggle to capture communication frictions 'which could make evaluations based on such simulations overly optimistic' is not backed by any direct evidence linking the observed differences in the 8 dimensions to actual discrepancies in chatbot evaluation outcomes, such as human preference scores or task completion rates.

    Authors: We agree that the connection is inferential rather than directly demonstrated through new experiments on evaluation outcomes. The dimensions were chosen because they capture known friction points (e.g., clarifications, error recoveries) that prior literature links to reduced user satisfaction and task success. We will revise the abstract to use more cautious language ('potentially leading to overly optimistic evaluations') and add a dedicated discussion paragraph citing supporting studies on how these frictions affect real chatbot assessments. A full empirical linkage would require additional human evaluation experiments beyond the current scope. revision: partial

  2. Referee: Dataset section (likely §4): The 1K dataset curation process does not specify how it ensures balanced representation of friction-inducing interactions (e.g., multi-turn clarifications or error recoveries) across the 16 domains, and no pre-specified analysis plan for the observed domain variability is mentioned, weakening the generalizability of the findings.

    Authors: The curation drew from public multi-turn dialogue corpora to achieve domain coverage, with selection criteria focused on task-oriented interactions of at least three turns; however, explicit stratification by friction type was not applied. We will expand the dataset section with a detailed description of the filtering and sampling procedure, including statistics on multi-turn length and friction indicators per domain. The domain variability analysis was exploratory, so we will explicitly label it as such and outline a pre-specified analysis plan for any follow-up studies. revision: yes

  3. Referee: Framework and Experiments sections (likely §3 and §5): No details are provided on the operationalization and measurement of the eight dimensions, including any statistical tests, inter-annotator agreement, or validation against external criteria like human realism ratings, making it difficult to assess the robustness of the reported differences.

    Authors: We apologize for the insufficient detail in the submitted version. The eight dimensions combine rule-based and LLM-assisted extraction for surface features with human annotation for communicative functions and user states. We will substantially expand §3 with precise operational definitions, annotation guidelines, inter-annotator agreement (Cohen's kappa), statistical tests (e.g., two-sample KS tests for distributional comparisons), and any available validation against external realism judgments. An appendix will include examples and full annotation protocols. revision: yes

standing simulated objections not resolved
  • Direct empirical evidence linking the eight dimensions to specific downstream chatbot evaluation discrepancies (human preference scores or task completion rates) cannot be provided without new experiments outside the current manuscript scope.

Circularity Check

0 steps flagged

No circularity: empirical comparison on external dataset

full rationale

The paper defines realsim as a framework with eight dimensions and applies it to a separately curated 1K real-dialogue dataset spanning 16 domains. Reported differences between real and simulated dialogues are direct distributional observations under those dimensions, with no equations, fitted parameters, or self-citations that reduce the central claims to inputs by construction. The analysis is therefore self-contained against external benchmarks rather than self-referential.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities; the framework implicitly assumes the eight dimensions are exhaustive and the dataset representative.

pith-pipeline@v0.9.0 · 5503 in / 1085 out tokens · 36199 ms · 2026-05-08T19:22:43.157350+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

Reference graph

Works this paper leans on

40 extracted references · 12 canonical work pages · 1 internal anchor

  1. [1]

    Attention is All you Need , url =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =

  2. [2]

    2025 , eprint=

    Taxonomy of User Needs and Actions , author=. 2025 , eprint=

    2025

  3. [3]

    Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents

    Chen, Chaoran and Yao, Bingsheng and Zou, Ruishi and Hua, Wenyue and Lyu, Weimin and Li, Toby Jia-Jun and Wang, Dakuo. Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.938

    work page doi:10.18653/v1/2025.findings-acl.938 2025

  4. [4]

    ``What ' s Up, Doc?'': Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasets

    Paruchuri, Akshay and Aziz, Maryam and Vartak, Rohit and Ali, Ayman and Uchehara, Best and Liu, Xin and Chatterjee, Ishan and Agrawal, Monica. ``What ' s Up, Doc?'': Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasets. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/2025.findin...

    work page doi:10.18653/v1/2025.findings-emnlp.125 2025

  5. [5]

    2025 , eprint=

    Flipping the Dialogue: Training and Evaluating User Language Models , author=. 2025 , eprint=

    2025

  6. [6]

    2024 , eprint=

    WildChat: 1M ChatGPT Interaction Logs in the Wild , author=. 2024 , eprint=

    2024

  7. [7]

    2024 , eprint=

    LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset , author=. 2024 , eprint=

    2024

  8. [8]

    , title =

    Park, Joon Sung and O'Brien, Joseph and Cai, Carrie Jun and Morris, Meredith Ringel and Liang, Percy and Bernstein, Michael S. , title =. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , articleno =. 2023 , isbn =. doi:10.1145/3586183.3606763 , abstract =

    work page doi:10.1145/3586183.3606763 2023

  9. [9]

    2023 , eprint=

    Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations , author=. 2023 , eprint=

    2023

  10. [10]

    Character- LLM : A Trainable Agent for Role-Playing

    Shao, Yunfan and Li, Linyang and Dai, Junqi and Qiu, Xipeng. Character- LLM : A Trainable Agent for Role-Playing. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023

    2023

  11. [11]

    C o MP os T : Characterizing and Evaluating Caricature in LLM Simulations

    Cheng, Myra and Piccardi, Tiziano and Yang, Diyi. C o MP os T : Characterizing and Evaluating Caricature in LLM Simulations. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.669

    work page doi:10.18653/v1/2023.emnlp-main.669 2023

  12. [12]

    2024 , eprint=

    How Far Are LLMs from Believable AI? A Benchmark for Evaluating the Believability of Human Behavior Simulation , author=. 2024 , eprint=

    2024

  13. [13]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, H ´erve J ´egou, and Tomas Mikolov

    Kim, Eunsu and Suk, Juyoung and Kim, Seungone and Muennighoff, Niklas and Kim, Dongkwan and Oh, Alice. LLM -as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.1357

    work page doi:10.18653/v1/2025.findings-acl.1357 2025

  14. [14]

    InCharacter: Evaluating personality fidelity in role-playing agents through psychological interviews

    Wang, Xintao and Xiao, Yunze and Huang, Jen-tse and Yuan, Siyu and Xu, Rui and Guo, Haoran and Tu, Quan and Fei, Yaying and Leng, Ziang and Wang, Wei and Chen, Jiangjie and Li, Cheng and Xiao, Yanghua. I n C haracter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews. Proceedings of the 62nd Annual Meeting of the Asso...

    work page doi:10.18653/v1/2024.acl-long.102 2024

  15. [15]

    Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =

    Liu, Jiaheng and Ni, Zehao and Que, Haoran and Sun, Tao and Wang, Zekun and Yang, Jian and Wang, Jiakai and Guo, Hongcheng and Peng, Zhongyuan and Zhang, Ge and Tian, Jiayi and Bu, Xingyuan and Xu, Ke and Rong, Wenge and Peng, Junran and Zhang, Zhaoxiang , title =. Proceedings of the 38th International Conference on Neural Information Processing Systems ,...

    2024

  16. [16]

    2024 , eprint=

    Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue , author=. 2024 , eprint=

    2024

  17. [17]

    2019 , eprint=

    ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons , author=. 2019 , eprint=

    2019

  18. [18]

    MT -Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models

    Kwan, Wai-Chung and Zeng, Xingshan and Jiang, Yuxin and Wang, Yufei and Li, Liangyou and Shang, Lifeng and Jiang, Xin and Liu, Qun and Wong, Kam-Fai. MT -Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.1124

    work page doi:10.18653/v1/2024.emnlp-main.1124 2024

  19. [19]

    MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback , url =

    Wang, Xingyao and Wang, Zihan and Liu, Jiateng and Chen, Yangyi and Yuan, Lifan and Peng, Hao and Ji, Heng , booktitle =. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback , url =

  20. [20]

    Chang, Serina and Anderson, Ashton and Hofman, Jake M. , year=. ChatBench: From Static Benchmarks to Human-AI Evaluation , url=. doi:10.18653/v1/2025.acl-long.1262 , booktitle=

    work page doi:10.18653/v1/2025.acl-long.1262 2025

  21. [21]

    BASES : Large-scale Web Search User Simulation with Large Language Model based Agents

    Ren, Ruiyang and Qiu, Peng and Qu, Yingqi and Liu, Jing and Zhao, Wayne Xin and Wu, Hua and Wen, Ji-Rong and Wang, Haifeng. BASES : Large-scale Web Search User Simulation with Large Language Model based Agents. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.50

    work page doi:10.18653/v1/2024.findings-emnlp.50 2024

  22. [22]

    Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =

    Li, Ruosen and Li, Ruochen and Wang, Barry and Du, Xinya , title =. Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =. 2024 , isbn =

    2024

  23. [23]

    Approximating Online Human Evaluation of Social Chatbots with Prompting

    Svikhnushina, Ekaterina and Pu, Pearl. Approximating Online Human Evaluation of Social Chatbots with Prompting. Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2023. doi:10.18653/v1/2023.sigdial-1.25

    work page doi:10.18653/v1/2023.sigdial-1.25 2023

  24. [24]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , url =

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and Zhang, Hao and Gonzalez, Joseph E and Stoica, Ion , booktitle =. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , url =

  25. [25]

    2025 , eprint=

    Scaling Synthetic Data Creation with 1,000,000,000 Personas , author=. 2025 , eprint=

    2025

  26. [26]

    BERTopic: Neural topic modeling with a class-based TF-IDF procedure

    BERTopic: Neural topic modeling with a class-based TF-IDF procedure , author=. arXiv preprint arXiv:2203.05794 , year=

    work page internal anchor Pith review arXiv

  27. [27]

    2026 , eprint=

    SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation , author=. 2026 , eprint=

    2026

  28. [28]

    2024 , eprint=

    MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models , author=. 2024 , eprint=

    2024

  29. [29]

    The knowledge engineering review , volume=

    A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies , author=. The knowledge engineering review , volume=. 2006 , publisher=

    2006

  30. [30]

    User modeling and user-adapted interaction , volume=

    Natural language processing and user modeling: Synergies and limitations , author=. User modeling and user-adapted interaction , volume=. 2001 , publisher=

    2001

  31. [31]

    Quality and User Experience , volume=

    Users' experiences with chatbots: findings from a questionnaire study , author=. Quality and User Experience , volume=. 2020 , publisher=

    2020

  32. [32]

    International Journal of Human--Computer Interaction , volume=

    How should my chatbot interact? A survey on social characteristics in human--chatbot interaction design , author=. International Journal of Human--Computer Interaction , volume=. 2021 , publisher=

    2021

  33. [33]

    Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , pages=

    If I hear you correctly: Building and evaluating interview chatbots with active listening skills , author=. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , pages=

    2020

  34. [34]

    arXiv preprint arXiv:2602.01405 , year=

    Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents , author=. arXiv preprint arXiv:2602.01405 , year=

    work page arXiv

  35. [35]

    2024 , eprint=

    -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. 2024 , eprint=

    2024

  36. [36]

    2026 , eprint=

    HumanLM: Simulating Users with State Alignment Beats Response Imitation , author=. 2026 , eprint=

    2026

  37. [37]

    Large language models that replace human participants can harmfully misportray and flatten identity groups

    Wang, Angelina and Morgenstern, Jamie and Dickerson, John P. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence

  38. [38]

    Not Yet: Large Language Models Cannot Replace Human Respondents for Psychometric Research , doi =

    Wang, Pengda and Zou, Huiqi and Yan, Zihan and Guo, Feng and Sun, Tianjun and Xiao, Ziang and Zhang, Bo , year =. Not Yet: Large Language Models Cannot Replace Human Respondents for Psychometric Research , doi =

  39. [39]

    1975 , abstract =

    Kincaid, J P and Fishburne, Jr , Robert P and Rogers, Richard L and Chissom, Brad S , copyright =. 1975 , abstract =

    1975

  40. [40]

    MTLD , vocd-D, and HD-D : A validation study of sophisticated approaches to lexical diversity assessment

    McCarthy, Philip M and Jarvis, Scott. MTLD , vocd-D, and HD-D : A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods