REVIEW 3 major objections 5 minor 30 references
A Literature Review on Simulation in Conversational Recommender Systems
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This review organizes all simulation work in conversational recommender systems into four categories and argues simulation is becoming the field's main path forward.
desk verdict A useful but methodologically under-specified simulation-in-CRS survey whose 11-paper claim does not match its own 13-row table. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the four-category taxonomy itself, built from a title-keyword search for 'conversation* recommend*' in a computer science bibliography, followed by iterative relevance screening. The taxonomy is the mechanism that does the review's work: it maps each of 11 simulation-focused publications to one of four research objectives and thereby turns scattered papers into a claim about how simulation functions across the field. Within that map, the recurring technical engine is user simulation, ranging from agenda-based simulators with separate natural-language, preference, and interaction modules to LLM-based simulators that generate open-ended user utterances and feedback.
What would settle it
A comprehensive search for CRS simulation papers using additional terms such as 'user simulator', 'interactive recommendation simulation', and 'LLM agents for recommendation' across multiple bibliographic databases, followed by the same screening, would settle the completeness question: if it returns substantial simulation work that the four categories cannot absorb, the taxonomy's coverage claim fails.
Extended reading notes
Core claim
On its own terms, the paper's discovery is an organizing framework: every use of simulation in CRS research can be placed in one of four categories, and mapping the 11 selected publications onto them shows a coherent research trajectory. The trajectory starts with crowdsourced datasets like REDIAL, whose per-dialogue cost and limited scale motivate LLM-generated synthetic data such as LLM-REDIAL; continues with simulation-based data augmentation and user simulators for algorithm training; and culminates in user simulation as an evaluation and empirical-research instrument, with LLM-based simulators offering flexible alternatives to human evaluation. The paper also asserts that no existing review covered simulation in CRSs after large language models emerged, making the taxonomy a first attempt at a structured summary of this phase.
Load-bearing premise
The whole taxonomy rests on the assumption that the keyword search and relevance screening captured every important simulation-in-CRS paper; if important simulation work falls outside the retrieved set, the four categories and the field-level conclusions drawn from them would be incomplete.
Editorial extensions
If this is right
- LLM-generated dialogue data can replace or augment crowdsourced CRS datasets at far lower cost; the review cites a 47.6k-dialogue, 482.6k-utterance dataset generated for about $750, versus roughly $1 per dialogue for crowdsourcing.
- User simulators provide an offline evaluation path that avoids the cost, risk, and ethical problems of live or laboratory experiments, at the price of validity that must be checked.
- Simulation-based counterfactual data augmentation can improve CRS recommendation algorithms by expanding user preferences beyond observed interactions.
- LLM-based user simulators enable empirical studies of CRS phenomena, such as emotional transmission between user and system, that would be infeasible with human subjects.
- The four-category taxonomy gives later researchers a coordinate system for placing new simulation work and for spotting which functions remain underdeveloped.
Reading between the lines
- A title-keyword search restricted to 'conversation* recommend*' may miss simulation work published under adjacent terms such as 'dialog policy', 'user modeling', or 'interactive recommendation', so the true population of simulation-in-CRS papers could be larger than 11 and the taxonomy's completeness is an open empirical question.
- The reported 'cognitive supermen' problem suggests a concrete calibration test: comparing LLM-simulated user preference distributions with human preference distributions on the same recommendation tasks could show whether simulated evaluations overstate system quality.
- The gap the review names between text-semantic space and behavioral semantics points toward hybrid evaluation designs in which LLM simulators handle dialogue while a small human panel anchors behavior, rather than full replacement of human users.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a literature review of simulation methods in conversational recommender systems (CRSs). The authors report a title keyword search on dblp that yielded 447 CRS publications, from which 11 were selected through iterative relevance screening as directly addressing simulation. The selected works are organized into a four-category taxonomy: dataset construction, algorithm design, system evaluation, and empirical studies. The review summarizes each category, emphasizes the growing role of LLM-based simulation, and closes with challenges and opportunities. The central claim is that this is the first systematic and comprehensive analysis of simulation methods in CRSs.
Significance. If the methodological foundation were fully transparent, this review would provide a timely and useful organizing framework for a rapidly evolving area. The paper's strengths include a concise synthesis of key simulation works, a specific and cited cost comparison between REDIAL and LLM-REDIAL, and a clear articulation of important open problems such as the 'cognitive superman' issue and the gap between text semantic space and behavioral semantics. The taxonomy itself is intuitive and could serve as a useful starting point for future work. However, the significance is currently limited by the lack of an auditable search and screening protocol and by an internal inconsistency in the reported corpus size, both of which bear directly on the paper's claim to comprehensiveness.
major comments (3)
- [Method; Table 1] The text states that '11 publications directly addressing simulation in CRSs were identified' and were mapped onto the taxonomy, but Table 1 lists 13 distinct entries: 2 in Datasets Construction, 2 in Algorithm Design, 8 in System Evaluation, and 1 in Empirical Study. This is an internal inconsistency in the central evidence base of the review. The authors should either correct the count, explain that some table rows are grouped sub-items, or adjust the claimed number of included publications.
- [Method] The search and screening protocol is not reproducible as reported. The paper gives only a title keyword search for 'conversation* recommend*' across dblp, followed by 'iterative screening based on relevance criteria,' with no dblp snapshot date, no inclusion/exclusion definitions, no inter-rater reliability procedure, and no flow diagram showing how 447 publications were reduced to 11. In addition, the paper says preprints were removed, yet the reference list includes arXiv-only items ([29] and [30]). As written, the Abstract's claim of a 'comprehensive analysis' is not supported by the described method. Please provide a complete protocol or substantially qualify the comprehensiveness claim.
- [Method] The title-based search strategy with roots 'conversation* recommend*' risks missing highly relevant simulation papers whose titles do not contain both roots, such as works phrased as 'user simulation for conversational recommendation' or as evaluation-methodology studies without the exact stem combination. Since the review aims to systematically taxonomize simulation methods in CRSs, the absence of a validated search strategy (e.g., full-text search, citation snowballing, or a second complementary query) leaves the corpus potentially incomplete and the taxonomy's coverage unverified. The authors should either strengthen the search methodology or temper the claim of completeness.
minor comments (5)
- [Abstract] The sentence 'Despite several challenges, such as dataset bias, the limited output flexibility of LLM-based simulations, and the gap between text semantic space and behavioral semantics, persist due to the complexity in Human-Computer Interaction (HCI) of CRSs' is grammatically incomplete; 'persist' should be 'persisting' or the sentence should be restructured.
- [Table 1] In the System Evaluation block, the phrase 'Human-Involves' should be 'Human-Involvement'.
- [References] Reference [9] contains a garbled author string: 'Kim M, Kim M, Kim Bw Hana nd Kwak et al.' should be corrected to the proper author list.
- [Figure 1] The figure caption states that publication statistics are 'as of July 1, 2025,' while the manuscript is dated June 25, 2025; please reconcile these dates.
- [Method] The four taxonomy categories are introduced only as 'aligned with core research objectives'; adding one or two sentences defining the inclusion criterion for each category would help readers understand and reproduce the mapping.
Circularity Check
No circularity: the review's taxonomy and findings summarize externally published work; the only self-citation ([28]) is an illustrative example and is not load-bearing.
full rationale
This paper is a literature review, not a derivation or prediction chain. Its central claim is that it systematically categorizes simulation methods in conversational recommender systems into four groups, and that claim rests on a described search and screening process plus a table of external publications. The four categories (datasets construction, algorithm design, system evaluation, empirical studies) are presented as an organizing framework for the reviewed papers, not as a conclusion forced by an equation, a fitted parameter, or an unverified uniqueness theorem. The mapped entries in Table 1 are all external works with independent venues and dates; the review's findings are summaries of those works. The only self-citation is reference [28] (Zhang, Kang, and Guo, WWW '25), which appears in a general sentence about LLM-based simulations across economic, information-access, and gaming domains; it is used as one example among several and does not support the taxonomy or any load-bearing claim. I note for completeness that the manuscript contains an internal counting inconsistency: the Method section says 11 publications were identified and mapped, while Table 1 lists 13 distinct entries, and the screening procedure is described only as an iterative relevance-based screening without a full protocol. These are reproducibility and completeness concerns, not circularity: they do not make the taxonomy equivalent to its inputs by construction, and they do not involve a fitted input being renamed as a prediction. The review is therefore self-contained relative to the circularity standard, and the only reason the score is not zero is the presence of one minor, non-load-bearing self-citation.
Assumptions & free parameters
assumptions (3)
- domain assumption Search query 'conversation* recommend*' on dblp is sufficient to capture the relevant CRS literature.
- ad hoc to paper The 'relevance criteria' used in iterative screening are well-defined and identify all 11 truly relevant papers.
- domain assumption The four-category taxonomy is a valid, mutually exclusive partition of the selected works.
Cite this review
Pith. "Pith review of A Literature Review on Simulation in Conversational Recommender Systems." pith.science (2026). https://pith.science/paper/ABXAOJLC
@misc{pith2026250620291,
author = {Pith},
title = {Pith review of: A Literature Review on Simulation in Conversational Recommender Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/ABXAOJLC}},
note = {Machine review of arXiv:2506.20291}
}
read the original abstract
Conversational Recommender Systems (CRSs) have garnered attention as a novel approach to delivering personalized recommendations through multi-turn dialogues. This review developed a taxonomy framework to systematically categorize relevant publications into four groups: dataset construction, algorithm design, system evaluation, and empirical studies, providing a comprehensive analysis of simulation methods in CRSs research. Our analysis reveals that simulation methods play a key role in tackling CRSs' main challenges. For example, LLM-based simulation methods have been used to create conversational recommendation data, enhance CRSs algorithms, and evaluate CRSs. Despite several challenges, such as dataset bias, the limited output flexibility of LLM-based simulations, and the gap between text semantic space and behavioral semantics, persist due to the complexity in Human-Computer Interaction (HCI) of CRSs, simulation methods hold significant potential for advancing CRS research. This review offers a thorough summary of the current research landscape in this domain and identifies promising directions for future inquiry.
Figures
Reference graph
Works this paper leans on
-
[28]
Zhang H, Kang X and Guo J. How does search affect personalized recommendations and user behavior? evidence from llm-based synthetic data. In Companion Proceedings of the ACM on Web Conference 2025 . WWW ’25, New York, NY , USA: Association for Computing Machinery. ISBN 9798400713316, p. 2434–2443. DOI:10.1145/3701716.3717530
arXiv 2025
-
[29]
Muse: A Multimodal Conversational Recommendation Dataset with Scenario-Grounded User Profiles
Wang Z, Yang X, Liu Y et al. Muse: A multimodal conversational recommendation dataset with scenario- grounded user profiles, 2025. URL https:// arxiv.org/abs/2412.18416
work page Pith review arXiv 2025
-
[30]
Vlachou M and Macdonald C. What else would i like? a user simulator using alternatives for improved evaluation of fashion conversational recommendation systems, 2024. URL https://arxiv.org/abs/ 2401.05783. Prepared using sagej.cls
work page Pith review arXiv 2024
-
[1]
Evaluating large language models as generative user simulators for conversational recommendation
Yoon Se, He Z, Echterhoff J et al. Evaluating large language models as generative user simulators for conversational recommendation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) . Mexico City, Mexico: Association for Computational L...
doi:10.18653/v1/2024 2024
-
[2]
Rethinking the evaluation for conversational recommendation in the era of large language models
Wang X, Tang X, Zhao X et al. Rethinking the evaluation for conversational recommendation in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, pp. 10052–10065. DOI: 10.18653/v1/2023.emnlp-main.621. Prepared using sagej.cls Zhang...
-
[3]
Liang T, Jin C, Wang L et al. LLM-REDIAL: A large- scale dataset for conversational recommender systems created from user behaviors with LLMs. In Findings of the Association for Computational Linguistics: ACL 2024 . Bangkok, Thailand: Association for Computational Linguistics, pp. 8926–8939. DOI:10. 18653/v1/2024.findings-acl.529
work page 2024
-
[4]
Advances and challenges in conversational recommender systems: A survey
Gao C, Lei W, He X et al. Advances and challenges in conversational recommender systems: A survey. AI Open 2021; 2: 100–126. DOI:https://doi.org/10.1016/j. aiopen.2021.06.002
doi:10.1016/j 2021
-
[5]
Evaluating conversational recommender systems
Jannach D. Evaluating conversational recommender systems. Artificial Intelligence Review 2023; 56(3): 2365–2400. DOI:10.1007/s10462-022-10229-x
Show all 30 references
-
[6]
Conversational recommender systems techniques, tools, acceptance, and adoption: A state of the art review.Expert Systems with Applications 2022; 203: 117539
Pramod D and Bafna P. Conversational recommender systems techniques, tools, acceptance, and adoption: A state of the art review.Expert Systems with Applications 2022; 203: 117539. DOI:https://doi.org/10.1016/j. eswa.2022.117539
2022
-
[7]
Conversational recommenda- tion: A grand ai challenge
Jannach D and Chen L. Conversational recommenda- tion: A grand ai challenge. AI Magazine 2022; 43(2): 151–163. DOI:https://doi.org/10.1002/aaai.12059
2022 doi
-
[8]
User simulation for evaluating information access systems
Balog K and Zhai C. User simulation for evaluating information access systems. Foundations and Trends® in Information Retrieval 2024; 18(1-2): 1–261. DOI: 10.1561/1500000098
2024 doi
-
[9]
Pearl: A review-driven persona-knowledge grounded conversational recommendation dataset
Kim M, Kim M, Kim Bw Hana nd Kwak et al. Pearl: A review-driven persona-knowledge grounded conversational recommendation dataset. In Findings of the Association for Computational Linguistics: ACL 2024 . Bangkok, Thailand: Association for Computational Linguistics, pp. 1105–112...
2024
-
[10]
Improving con- versational recommendation systems via counterfactual data simulation
Wang X, Zhou K, Tang X et al. Improving con- versational recommendation systems via counterfactual data simulation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. KDD ’23, New York, NY , USA: Associa- tion for Computing Machinery. ISBN...
-
[11]
Multi-interest multi- round conversational recommendation system with fuzzy feedback based user simulator
Shen Q, Wu L, Zhang Y et al. Multi-interest multi- round conversational recommendation system with fuzzy feedback based user simulator. ACM Trans Recomm Syst 2024; 2(4). DOI:10.1145/3616379
2024 doi
-
[12]
Evaluating conversational recommender systems via user simulation
Zhang S and Balog K. Evaluating conversational recommender systems via user simulation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . KDD ’20, New York, NY , USA: Association for Computing Machinery. ISBN 9781450379984, p...
-
[13]
Analyzing and sim- ulating user utterance reformulation in conversational recommender systems
Zhang S, Wang MC and Balog K. Analyzing and sim- ulating user utterance reformulation in conversational recommender systems. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ’22, New York, NY , USA: Assoc...
-
[14]
Usersimcrs: A user simulation toolkit for evaluating conversational recommender systems
Afzali J, Drzewiecki AM, Balog K et al. Usersimcrs: A user simulation toolkit for evaluating conversational recommender systems. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining . WSDM ’23, New York, NY , USA: Association for Computing...
-
[15]
Identifying breakdowns in conversational recommender systems using user simulation
Bernard N and Balog K. Identifying breakdowns in conversational recommender systems using user simulation. In Proceedings of the 6th ACM Conference on Conversational User Interfaces . CUI ’24, New York, NY , USA: Association for Computing Machinery. ISBN 9798400705113, pp. 1–1...
-
[16]
Linguistics- based dialogue simulations to evaluate argumentative conversational recommender systems
Di Bratto M, Origlia A, Di Maro M et al. Linguistics- based dialogue simulations to evaluate argumentative conversational recommender systems. User Modeling and User-Adapted Interaction2024; 34(5): 1581–1611. DOI:10.1007/s11257-024-09403-3
-
[17]
A llm-based controllable, scalable, human-involved user simulator framework for conversational recommender systems
Zhu L, Huang X and Sang J. A llm-based controllable, scalable, human-involved user simulator framework for conversational recommender systems. In Proceedings of the ACM on Web Conference 2025 . WWW ’25, New York, NY , USA: Association for Computing Machinery. ISBN 979840071274...
2025
-
[18]
How reliable is your simulator? analysis on the limitations of current llm-based user simulators for conversational recommendation
Zhu L, Huang X and Sang J. How reliable is your simulator? analysis on the limitations of current llm-based user simulators for conversational recommendation. In Companion Proceedings of the ACM Web Conference 2024 . WWW ’24, New York, NY , USA: Association for Computing Machi...
2024
-
[19]
Towards deep conversational recommendations
Li R, Kahou S, Schulz H et al. Towards deep conversational recommendations. InProceedings of the 32nd International Conference on Neural Information Processing Systems. NIPS ’18, Red Hook, NY , USA: Curran Associates Inc., p. 9748–9758
-
[20]
Towards conversational search and recommendation: System ask, user respond
Zhang Y , Chen X, Ai Q et al. Towards conversational search and recommendation: System ask, user respond. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. CIKM ’18, New York, NY , USA: Association for Computing Machinery. ISBN 97...
-
[21]
An automatic procedure for generating datasets for conversational recommender systems
Suglia A, Greco C, Basile P et al. An automatic procedure for generating datasets for conversational recommender systems. CEUR Workshop Proceedings 2017; 1866. Prepared using sagej.cls 6 Journal Title XX(X)
2017
-
[22]
Role play with large language models
Shanahan M, McDonell K and Reynolds L. Role play with large language models. Nature 2023; 623(7987): 493–498. DOI:10.1038/s41586-023-06647-8
2023 doi
-
[23]
Crs-que: A user-centric evaluation framework for conversational recommender systems
Jin Y , Chen L, Cai W et al. Crs-que: A user-centric evaluation framework for conversational recommender systems. ACM Trans Recomm Syst 2024; 2(1). DOI: 10.1145/3631534
2024 doi
-
[24]
Consumption and performance: Understanding longitudinal dynam- ics of recommender systems via an agent-based simu- lation framework
Zhang J, Adomavicius G, Gupta A et al. Consumption and performance: Understanding longitudinal dynam- ics of recommender systems via an agent-based simu- lation framework. Information Systems Research 2020; 31(1): 76–101. DOI:10.1287/isre.2019.0876
2020
-
[25]
Generative agents: Interactive simulacra of human behavior
Park JS, O’Brien J, Cai CJ et al. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology . UIST ’23, New York, NY , USA: Association for Computing Machinery. ISBN 9798400701320, pp. ...
-
[26]
User behavior simulation with large language model-based agents
Wang L, Zhang J, Yang H et al. User behavior simulation with large language model-based agents. ACM Trans Inf Syst 2025; 43(2). DOI:10.1145/ 3708985
2025
-
[27]
Usimagent: Large language models for simulating search users
Zhang E, Wang X, Gong P et al. Usimagent: Large language models for simulating search users. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . SIGIR ’24, New York, NY , USA: Association for Computing Machinery....
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.