REVIEW 4 cited by
Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based methods or on annotated data. This paper introduces DAUS, a Domain-Aware User Simulator. Leveraging large language models, we fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment. Notably, we have observed that fine-tuning enhances the simulator's coherence with user goals, effectively mitigating hallucinations -- a major source of inconsistencies in simulator responses.
Forward citations
Cited by 4 Pith papers
-
Mind the Sim2Real Gap in User Simulation for Agentic Tasks
On τ-bench, LLM user simulators are more cooperative, more verbose, and more lenient than real human users, so agent benchmarks that rely on them overstate real-world performance.
-
Simulating Before Planning: Constructing Intrinsic User World Model for User-Tailored Dialogue Policy Planning
UDP, a user-tailored dialogue policy planner with a diffusion-based persona portrayer and a Brownian Bridge feedback anticipator, outperforms existing planners on simulated persuasion and emotional-support tasks.
-
CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions
A new 109-conversation, 86-API benchmark for LLM function-calling in multi-turn dialogue shows top models at about 40% accuracy and near-zero performance on chains of 4+ calls.
-
VicSim: Enhancing Victim Simulation with Emotional and Linguistic Fidelity
VicSim, a fine-tuned Llama-2 victim simulator with GAN-style training and keyword prompting, produced messages that human raters found indistinguishable from real victim reports and more human-like than GPT-4.
Discussion (0). Continue with ORCID to comment.