The reviewed record of science sign in
Pith

arxiv: 2402.13374 · v1 · pith:SGL6ZXEB · submitted 2024-02-20 · cs.CL

Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems

Ivan Sekuli\'c , Silvia Terragni , Victor Guimar\~aes , Nghia Khau , Bruna Guedes , Modestas Filipavicius , Andr\'e Ferreira Manso , Roland Mathis This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:SGL6ZXEBrecord.jsonopen to challenge →

classification cs.CL
keywords usersimulatordialoguesystemstask-orienteddatadausevaluation
0
0 comments X
read the original abstract

In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based methods or on annotated data. This paper introduces DAUS, a Domain-Aware User Simulator. Leveraging large language models, we fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment. Notably, we have observed that fine-tuning enhances the simulator's coherence with user goals, effectively mitigating hallucinations -- a major source of inconsistencies in simulator responses.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science

    cs.AI 2026-05 unverdicted novelty 7.0

    SCICONVBENCH is a new benchmark evaluating LLMs on multi-turn disambiguation and inconsistency resolution for task formulation in computational science, with frontier models reaching only 52.7% success on fluid mechan...

  2. PERCEIVE: A Benchmark for Personalized Emotion and Communication Behavior Understanding on Social Media

    cs.SI 2026-04 unverdicted novelty 7.0

    PERCEIVE is the first bilingual benchmark integrating author content, reader emotions from comments, communication behavior, user attributes, and social graphs for personalized social media emotion understanding.

  3. Reinforcing Human Behavior Simulation via Verbal Feedback

    cs.LG 2026-05 unverdicted novelty 6.0

    DITTO uses RL with verbal feedback to train LLMs for human behavior simulation, reporting 36% average gains over base models and outperforming GPT-5.4 on 6 of 10 SOUL benchmark tasks.