REVIEW 4 major objections 6 minor 26 references
AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Data
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A Patient Simulator built from real EHR vignettes produces conversations clinicians judge consistent with their source vignettes in 97.7% of 519 cases, and a multi-agent AI Triage system keeps the likely diagnosis in its top three about…
desk verdict Genuinely useful simulation framework, but the headline consistency claim is circular and the paper overstates what it has shown. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the patient vignette: a structured abstract of a real EHR encounter, containing chief complaint, history of present illness, medications, and related history, which the Patient Simulator is instructed to embody in everyday language. The simulator's behavioral rules do the heavy lifting: describe symptoms one or two at a time, volunteer nothing until asked, avoid jargon, make 'common sense' inferences when the vignette is silent, and never reveal that it is following a script. On the other side, AI Triage is an eight-agent workflow with a controller setting tone, a SymptomCollector gathering symptoms over RAG-retrieved EHR data, planners and retrievers pulling relevant labs and medications, a Summary agent synthesizing the case, a Differential Diagnosis agent reasoning step-by-step, a Next Steps agent issuing recommendations and escalation guidance, and an optional Guideline Verifier that retrieves clinical guidelines and adjusts urgency. The evaluation itself is a 14-question clinician rubric plus free text, independently completed by two specialists.
What would settle it
Take a sample of the 519 vignettes and obtain the full original encounter notes or audio transcripts behind them, run the Patient Simulator in the same protocol, and compare its answers about symptom onset, duration, timing, and negatives against the real patient's statements; if material contradictions appear in more than a small fraction, the 97.7% vignette-consistency score does not establish real-encounter fidelity.
Extended reading notes
Core claim
On the paper's own terms, the central result is that EHR-grounded simulation can stand in for live patient interaction well enough for preliminary clinical evaluation. Two internal-medicine specialists independently reviewed 519 simulated encounters spanning nine symptom categories and found the Patient Simulator consistent with its source vignette 97.7% of the time; they rated the case summary produced from the conversation as relevant in 99% of cases. Against that benchmark, the AI Triage system asked precise, non-redundant questions in 81.7% of encounters, used appropriate tone in 99.6%, and had the most likely diagnosis within its top three differentials in 95.4% and 94.8% of cases as judged by the two reviewers, with model-reviewer agreement slightly exceeding reviewer-reviewer agreement. The paper frames these as preliminary, simulation-based numbers, not proof of real-world safety.
Load-bearing premise
The load-bearing premise is that clinician agreement that a simulated conversation matches its short written summary is a valid stand-in for the simulator being faithful to the real patient's actual encounter, even though the summary is condensed and the simulator is allowed to invent plausible details when the summary is silent.
Editorial extensions
If this is right
- If the 97.7% consistency score is representative, a single EHR-derived dataset can yield hundreds of varied simulated conversations suitable for benchmarking triage agents before any live-patient trial.
- If the AI Triage results are representative, a modular multi-agent design can transparently collect symptoms, retrieve relevant patient data, and put the likely diagnosis in its top three roughly 19 times out of 20.
- The Guideline Verifier path implies urgency decisions can be checked against retrieved clinical guidelines rather than relying only on the model's internal knowledge, which should make the reasoning easier to audit.
- Because the dataset deliberately balances nine symptom categories, the reported performance covers a spread of primary-care presentations, though the paper excludes psychological complaints and follow-up or routine visits.
Reading between the lines
- A testable extension is to vary the interviewing agent's persistence and style; if consistency scores shift with the questioner, the 97.7% figure describes the simulator-plus-triage pair rather than the simulator alone.
- The paper does not report re-identification risk analysis for the simulated utterances, so its privacy claim is best read as 'no direct patient data is shown' rather than 'no patient could be recognized'; a formal risk assessment would strengthen that inference.
- The same vignette-plus-clinician-rubric loop could serve as a cheap regression gate whenever the triage prompts are updated, catching degradations before deployment.
- Extending the method to follow-up visits and chronic-condition monitoring, which the authors note as future work, would test whether the initial-encounter results transfer to patients with established histories.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a Patient Simulator that converts EHR-derived vignettes (based on history of present illness abstracts) into simulated patient conversations, and an AI Triage multi-agent system that conducts symptom collection, EHR retrieval, differential diagnosis, and triage. The authors evaluate 519 simulated encounters with two clinician reviewers using a 14-item questionnaire, reporting 97.7% consistency between simulated patient responses and the source vignette, 99.2% summary relevance, and 95.4%/94.8% top-3 differential diagnosis accuracy. They conclude that the Patient Simulator demonstrates consistency with the real patient encounter, enabling safe testing of conversational triage agents.
Significance. The proposed pipeline—using real EHR data to generate vignettes that drive a patient simulator—addresses an important need for safe, scalable evaluation of conversational AI in healthcare. The multi-agent architecture is modular and includes a guideline verifier, which is a commendable safety mechanism. The dataset of 519 vignettes across nine symptom categories is a useful resource. However, the central validation of the Patient Simulator is circular: consistency is judged against the very vignette used to prompt the simulator, not against the actual patient encounter. Consequently, the headline claim of realism is unsupported, and the downstream metrics inherit this limitation.
major comments (4)
- [Appendix A.1 and Section 3.3] The 97.7% consistency score is circular. The Patient Simulator is prompted with the vignette (Appendix A.1: 'Your background and symptoms are based on the following context: <patient vignette>'), and rule 4 instructs it to answer 'no' for symptoms not mentioned. Clinicians then rate consistency with that same vignette. This evaluates instruction-following, not fidelity to the real patient. The statement in Section 4 that the simulator 'demonstrates consistency with the real patient encounter' is not supported. I request either an external validation against full encounter notes or transcripts or a reframing of all claims as vignette-level consistency.
- [Section 3.3] The evaluation of AI Triage lacks any baseline or ablation. The 95.4%/94.8% top-3 diagnosis rates, the triage approval, and the kappa statistics are reported without comparison to a single-prompt LLM, a frequency-based classifier, or a conventional triage protocol. As a result, the reader cannot attribute the observed performance to the multi-agent architecture or the guideline verifier. Please add at least one strong baseline (e.g., GPT-4o with the same case summary prompt) and an ablation without the multi-agent orchestration.
- [Section 3.1] The case selection pipeline relies on GPT-4o-mini for two classifications: encounter type (Initial Encounter vs. others) and symptom category (nine categories). No validation of these labels is reported. Since the final 519 encounters are selected by these labels, any systematic misclassification biases the distribution of conditions and the evaluation. Please validate a random sample (e.g., 50-100 cases) against clinician labels and report agreement, and disclose the prompt used for classification.
- [Section 3.3] The reporting of clinician agreement is inconsistent. The section opens with 'we report numbers where both physicians independently confirmed the model output,' but the top-3 diagnosis rates are given for each reviewer separately, and triage approval is reported as 'at least one reviewer approved.' These are different standards. Please report, for each outcome, the per-reviewer rates, the joint-confirmation rate, and the definition of 'confirmed' (e.g., both selected the same top-3 Dx, or both answered 'Yes').
minor comments (6)
- [Abstract] The phrase 'consistent with the patient vignettes in those same 97.7% of cases' is unclear; 'same' has no antecedent. Also, 'We demonstrate the Patient Simulator's consistency with the real patient encounter' overstates the vignette-level measurement.
- [Section 3.3] The kappa values (0.79, 0.74, 0.72) are reported without specifying the outcome variable (e.g., urgency status, diagnosis). Please clarify.
- [References] Schmidgall et al. (2024a) and (2024b) are duplicate citations of the same arXiv preprint (2405.07960).
- [Figure 2] The caption contains a typo: 'replying purely on knowledge' should read 'relying purely on knowledge.'
- [Section 3.1] The filtering from 21,779 records to 519 encounters is not documented step by step; a flow diagram would help the reader understand the inclusion criteria.
- [Table 1] The description of HealthDataRetriever appears truncated in the table; the row is incomplete.
Circularity Check
The 97.7% Patient Simulator consistency score is self-referential: the vignette is both the prompt source and the clinician reference standard, so it measures instruction-following rather than fidelity to the real encounter.
-
self definitional
[Appendix A.1 (Patient Simulator Prompt, rule 4), with Section 2.1, Section 3.2, Section 3.3, and Section 4]
"Your background and symptoms are based on the following context: <patient vignette> ... Answer questions about symptoms not explicitly mentioned in your context with logical consistency (e.g., if not mentioned, assume you don’t have them)."
The Patient Simulator is generated from each vignette and prompted to answer from that same vignette; clinician reviewers then judge consistency against the same vignette (Section 3.2: 'Reviewers had access to case vignettes...'). Rule 4 instructs the simulator to deny any symptom absent from the vignette, so a high consistency score is largely forced by construction unless the LLM fails to follow instructions. Section 3.3 reports 'consistent with the patient vignettes in those same 97.7% of cases,' and Section 4 converts this into 'consistency with the real patient encounter.' The real encounter is never used as a comparator; the vignette is an abstraction of the HPI, not a transcript.
full rationale
The paper's central validation loop is genuinely self-referential: the same patient vignette is the generative input for the Patient Simulator, the reference standard for the clinicians' 97.7% consistency judgment, and the basis for the Section 4 conclusion that the simulator is 'consistent with the real patient encounter.' The prompt's default-deny rule makes agreement with the vignette nearly inevitable, so that headline metric is closer to an instruction-following check than to an external validation. The AI Triage differential-diagnosis result (95.4% top-three) is not itself circular, because the triage agents never see the vignette and the clinician diagnosis is an independent judgment; however, it inherits the same limitation because the conversations are grounded only in the vignette-derived abstraction. There is no fitted parameter renamed as a prediction, no load-bearing self-citation, and no imported uniqueness theorem. The partial circularity is confined to the Patient Simulator fidelity claim, which is the paper's stated primary contribution, hence a score of 6 rather than 0 or 2.
Assumptions & free parameters
free parameters (2)
- Cases per symptom category =
44 to 68 per category across nine categories, 519 total
- Top-k differential diagnosis window =
3 of 5 proposed differentials
assumptions (5)
- domain assumption The HealthVerity EHR records, specifically the chief complaint and history of present illness fields, faithfully represent the patient's presenting encounter.
- domain assumption Clinician agreement on the 14-question rubric is a valid and sufficient ground truth for triage appropriateness.
- ad hoc to paper An LLM instructed to make plausible inferences and to answer no for unmentioned symptoms produces responses representative of real patient behavior.
- ad hoc to paper GPT-4o-mini classification of encounter type and symptom category is accurate enough for sample selection.
- domain assumption Excluding non-initial encounters and Psychological cases does not bias the triage evaluation.
Cite this review
Pith. "Pith review of AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Data." pith.science (2026). https://pith.science/paper/HN7BBZDG
@misc{pith2026250604032,
author = {Pith},
title = {Pith review of: AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/HN7BBZDG}},
note = {Machine review of arXiv:2506.04032}
}
read the original abstract
Background: We present a Patient Simulator that leverages real world patient encounters which cover a broad range of conditions and symptoms to provide synthetic test subjects for development and testing of healthcare agentic models. The simulator provides a realistic approach to patient presentation and multi-turn conversation with a symptom-checking agent. Objectives: (1) To construct and instantiate a Patient Simulator to train and test an AI health agent, based on patient vignettes derived from real EHR data. (2) To test the validity and alignment of the simulated encounters provided by the Patient Simulator to expert human clinical providers. (3) To illustrate the evaluation framework of such an LLM system on the generated realistic, data-driven simulations -- yielding a preliminary assessment of our proposed system. Methods: We first constructed realistic clinical scenarios by deriving patient vignettes from real-world EHR encounters. These vignettes cover a variety of presenting symptoms and underlying conditions. We then evaluate the performance of the Patient Simulator as a simulacrum of a real patient encounter across over 500 different patient vignettes. We leveraged a separate AI agent to provide multi-turn questions to obtain a history of present illness. The resulting multiturn conversations were evaluated by two expert clinicians. Results: Clinicians scored the Patient Simulator as consistent with the patient vignettes in those same 97.7% of cases. The extracted case summary based on the conversation history was 99% relevant. Conclusions: We developed a methodology to incorporate vignettes derived from real healthcare patient data to build a simulation of patient responses to symptom checking agents. The performance and alignment of this Patient Simulator could be used to train and test a multi-turn conversational AI agent at scale.
Figures
Reference graph
Works this paper leans on
-
[1]
Bonow, R. O., Grant, A. O., and Jacobs, A. K. (2005). The cardiovascular state of the union: confronting the challenges of the new millennium . Circulation , 111(10):1205--1207
work page 2005
-
[2]
Capps, L. (2003). Health care disparities . JAMA , 290:2487--2488
work page 2003
-
[3]
Chandran, M. and Schulman, K. A. (2022). Racial disparities in healthcare and health . Health Services Research , 57:218
work page 2022
-
[4]
Chowdhary, K. and Chowdhary, K. (2020). Natural language processing . In Fundamentals of Artificial Intelligence , pages 603--649. Springer
work page 2020
-
[5]
Feder, G., Eccles, M., Grol, R., Griffiths, C., and Grimshaw, J. (1999). Using clinical guidelines . BMJ , 318:728--730
work page 1999
-
[6]
Goldstein, B. A., Navar, A. M., Pencina, M. J., and Ioannidis, J. P. (2017). Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review . Journal of the American Medical Informatics Association , 24(1):198--208. Epub 2016 May 17
work page 2017
-
[7]
Huang, X. et al. (2024). Understanding the planning of LLM agents: A survey . arXiv preprint arXiv:2402.02716
arXiv 2024
-
[8]
Iserson, K. V. and Moskop, J. C. (2007). Triage in medicine, part I: concept, history, and types . Annals of Emergency Medicine , 49:275--281
work page 2007
Show all 26 references
-
[9]
A., Schlessinger, D
Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., et al. (2025). An evaluation framework for clinical use of large language models in patient interaction tasks. Nature Medicine , pages 1--10
2025
-
[10]
Li, H., Ai, Q., Chen, J., Dong, Q., Wu, Z., Liu, Y., Chen, C., and Tian, Q. (2024). Blade: Enhancing black-box large language models with small domain-specific models . arXiv preprint arXiv:2403.18365
2024 arXiv
-
[11]
Li, Y., Rao, S., Solares, J. R. A., Hassaine, A., Ramakrishnan, R., Canoy, D., Zhu, Y., Rahimi, K., and Salimi-Khorshidi, G. (2020). BEHRT: Transformer for Electronic Health Records . Scientific Reports , 10(1):7155
2020
-
[12]
Liz \'e e, A., Beaucot \'e , P.-A., Whitbeck, J., Doumeingts, M., Beaugnon, A., and Feldhaus, I. (2024). Conversational medical ai: Ready for practice. arXiv preprint arXiv:2411.12808
2024 arXiv
-
[13]
Mielke, S. J. et al. (2021). Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP . arXiv preprint arXiv:2112.10508
2021 arXiv
-
[14]
Moritz, M., Topol, E., and Rajpurkar, P. (2025). Coordinated ai agents for advancing healthcare. Nature Biomedical Engineering , pages 1--7
2025
-
[15]
S., Kant, N., Aggarwal, K., Manjunath, N., Datta, D., Liu, Z., Ding, J., Busacca, S., and Bianco, C
Mukherjee, S., Gamble, P., Ausin, M. S., Kant, N., Aggarwal, K., Manjunath, N., Datta, D., Liu, Z., Ding, J., Busacca, S., and Bianco, C. (2024). Polaris: A safety-focused llm constellation architecture for healthcare . arXiv preprint arXiv:2403.13313
2024 arXiv
-
[16]
Murray, C. J. (2024). Findings from the global burden of disease study 2021 . The Lancet , 403:2259--2262
2024
-
[17]
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., and Jiang, X. (2021). Webgpt: Browser-assisted question-answering with human feedback . arXiv preprint arXiv:2112.09332
2021 arXiv
-
[18]
U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2023). A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435
2023 arXiv
-
[19]
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., and Zhi, D. (2021). Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction . NPJ Digital Medicine , 4(1):86
2021
-
[21]
Schmidgall, S., Ziaei, R., Harris, C., Reis, E., Jopling, J., and Moor, M. (2024b). AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments . arXiv preprint arXiv:2405.07960
2024 arXiv
-
[22]
and Nadiri, A
Talebirad, Y. and Nadiri, A. (2023). Multi-agent collaboration: Harnessing the power of intelligent llm agents . arXiv preprint arXiv:2306.03314
2023 arXiv
-
[23]
Thirunavukarasu, A. J. et al. (2023). Large language models in medicine . Nature Medicine , 29:1930--1940
2023
-
[24]
Tu, T. et al. (2024). Towards conversational diagnostic AI . arXiv preprint arXiv:2401.05654
2024 arXiv
-
[25]
Vaid, A., Landi, I., Nadkarni, G., and Nabeel, I. (2023). Using fine-tuned large language models to parse clinical notes in musculoskeletal pain disorders . The Lancet Digital Health , 5:e855--e858
2023
-
[26]
C., Smith, K
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., Zhang, Y., Magoc, T., Harle, C. A., Lipori, G., Mitchell, D. A., Hogan, W. R., Shenkman, E. A., Bian, J., and Wu, Y. (2022). A large language mod...
2022
-
[27]
R., Kim, J
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024). Almanac—retrieval-augmented language models for clinical medicine. Nejm ai , 1(2):AIoa2300068
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.