Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Reasoning Is Not All You Need: Examining LLMs for Multi-Turn Mental Health Conversations

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Frontier reasoning models (o1, DeepSeek-R1) fall below 'Good' on patient-centric communication in multi-turn mental-health dialogue, scoring ~2.55 and 2.77 out of 4 and matching the ground-truth diagnosis only ~31% of the time.

desk verdict A valuable dataset and evaluation scaffold for mental-health dialogue, but the headline claim about reasoning models is confounded by an unvalidated patient actor. read the letter →

arxiv 2505.20201 v2 pith:NFUGAIQH submitted 2025-05-26 cs.CL

Mohit Chandra , Siddharth Sriraman , Harneet Singh Khanuja , Yiqiao Jin , Munmun De Choudhury This is my paper · ORCID
classification cs.CL
keywords mentalhealthconversationsmulti-turndialogueLLMevaluationreasoningmodelsBeliefModelsyntheticgenerationpatientpersonadiagnosticaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that for multi-turn mental-health conversations, advanced reasoning alone is not enough: frontier reasoning models produce patient-centered communication that falls below 'Good' and identify the exact diagnosis only about a third of the time. It builds a synthetic two-actor conversation generator (MedAgent), a dataset of 2,284 conversations (MHSD), and a multi-axis evaluation framework (MultiSenseEval) grounded in the Health Belief Model and the six-function model of medical communication. Across these conversations, OpenAI o1 and DeepSeek-R1 average 2.55 and 2.77 out of 4 on patient-centric communication metrics and about 31% exact diagnostic accuracy, with performance dropping as conversations lengthen and varying by patient persona. If correct, this challenges the assumption that stronger reasoning automatically yields better mental-health support, and it provides a reusable pipeline for evaluating future models on patient-centered criteria.

What carries the argument

The central machinery is MedAgent, a two-actor conversation generator in which a patient actor is constrained by atomic medical facts, Big Five personality traits, health literacy levels, intentions, and goals, while a sensemaker actor (o1 or R1) progresses through five stages drawn from the six-function model of medical communication, extracting and matching facts, generating candidate messages, and filtering redundant ones. The evaluation side is MultiSenseEval, which scores conversations on three Health-Belief-Model dimensions (perceived susceptibility, severity, benefits), conversational flow and correctness, hard and soft diagnostic accuracy against ground-truth labels, and SMOG readability. Together these pieces translate qualitative patient-centered communication goals into reproducible, scorable axes and make the synthetic conversations structurally consistent enough to isolate the models' communicative and diagnostic failures.

What would settle it

Run the same five-stage sensemaking setup with human participants or standardized patients instead of the simulated patient, and score the resulting o1 and R1 conversations with the same MultiSenseEval rubrics; if the mean patient-centric scores reach 3.0 or above, or if exact diagnostic accuracy substantially exceeds 31%, the paper's central claim about real-world conversations would be falsified.

Watch

Extended reading notes

Core claim

The central discovery is that state-of-the-art reasoning models conduct structurally sound multi-turn mental-health sensemaking conversations while failing at patient-centered communication and exact diagnosis. o1 scored 2.45 for perceived susceptibility, 2.56 for perceived severity, and 2.76 for perceived benefits; R1 scored 2.59, 2.75, and 3.04, all below the 'Good' threshold of 3 except R1's perceived benefits. Both models exceeded 3.0 on conversation flow and correctness, so the shortfall is specific to communication quality and diagnostic capability rather than conversational organization. Hard diagnostic accuracy reached only 31.09% (o1) and 31.18% (R1), while soft accuracy reached 68.39% and 64.36%. Performance declined with longer conversations: R1 dropped 12.83% on perceived susceptibility, 6.71% on perceived severity, and 29.6% on hard diagnostic accuracy when sensemaker messages increased from 5 to 10-15, and both models communicated better with agreeable patients while producing harder-to-read text for higher-literacy patients.

Load-bearing premise

The load-bearing premise is that a GPT-4o patient actor, bound to answer only from a fixed list of facts and to agree whenever the sensemaker states a diagnosis, behaves like a real patient in a mental-health sensemaking conversation; if real patients volunteer unlisted symptoms, challenge diagnoses, or respond with unmodeled emotions, the reported accuracy and persona effects would not transfer to real-world settings.

Editorial extensions

If this is right

  • Users of current reasoning LLMs for mental-health sensemaking should expect HBM-based communication scores below 'Good' and exact diagnosis matching at roughly 31%.
  • Longer multi-turn conversations degrade patient-centered communication and diagnostic accuracy, so chat-based interventions that run many turns carry greater risk.
  • Patients with different personas receive different-quality care: agreeable patients get the best communication and diagnostic outcomes, while neurotic patients tend to fare worse.
  • Conversation flow and readability are not the bottleneck, so adding more reasoning or better stage ordering alone will not close the communication gap.
  • The MHSD dataset and MultiSenseEval can be reused to benchmark future models on the same patient-centered axes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the simulated patient only answers from a fixed fact list and is instructed to agree to any stated diagnosis, an implication of the setup is that real-world diagnostic accuracy may be even lower than the reported ~31%; a human-participant replication would settle this.
  • Hard diagnostic accuracy uses an exact-match criterion, so the ~31% figure may understate clinically useful performance; graded or expert-consensus scoring could separate 'wrong condition' from 'related but imprecise condition.'
  • If the turn-length degradation generalizes, retrieval- or memory-augmented sensemaking agents could be tested against MultiSenseEval to see whether they recover the lost patient-centered performance.
  • The persona-based disparities imply that aggregate model scores hide inequitable performance; deployment decisions should be conditioned on personality and health-literacy strata.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces MedAgent, a two-actor LLM framework that generates synthetic multi-turn mental-health sensemaking conversations between a GPT-4o patient actor and reasoning-model sensemakers (OpenAI o1 and DeepSeek-R1). The conversations are grounded in 181 filtered MedQA case studies and augmented with Big Five personality traits, health literacy levels, patient intentions, and goals. Using this framework, the authors create the MHSD dataset of 2,284 conversations and propose MultiSenseEval, a multidimensional evaluation protocol covering HBM-based patient-centric communication (perceived susceptibility, severity, benefits), hard and soft diagnostic accuracy, conversation flow/correctness, and SMOG readability. The headline findings are that o1 and R1 score below the 'Good' threshold on patient-centric communication (2.55 and 2.77 out of 4), achieve only about 31% hard diagnostic accuracy, perform better for 'agreeable' personas, and degrade with increasing conversation length. A human validation on 100 conversations reports 79-93.5% agreement with the LLM judge.

Significance. If the empirical claims survive the patient-simulator validity concern, the paper is a valuable contribution: it provides a modular synthetic data-generation pipeline, a large planned public dataset of multi-turn mental-health conversations, and an evaluation rubric that goes beyond diagnostic accuracy and win-rates by grounding assessment in the Health Belief Model and the six-function model of medical communication. The central finding—that frontier reasoning models score below 'Good' on patient-centric communication and achieve only about 31% exact diagnosis in this setting—is falsifiable and practically relevant. Strengths of the paper include transparent prompts for all modules, use of ground-truth MedQA labels for diagnostic scoring, statistical testing, and a good-faith human validation effort on a 100-conversation subset. The main weakness is attribution: the headline result is conditional on the behavior of the GPT-4o patient actor, whose closed-world answering and agree-to-diagnosis instructions have not been validated against real patient behavior.

major comments (4)
  1. [Algorithm 1, Tables 20 and 22, Section 5.1] The headline claim that o1 and R1 are 'below-par' and achieve about 31% diagnostic accuracy is not a direct measurement of reasoning-model capability, because all conversations are jointly produced by an unvalidated GPT-4o patient actor. In Algorithm 1, when MatchFacts returns no match (lines 6-10), background questions are answered with 'not sure'; under the prompt in Table 20, any sensemaker question phrased differently from the atomic-fact list is therefore converted into missing information. Table 22 additionally instructs the patient actor to 'agree to it' whenever the sensemaker states a diagnosis, removing the corrective feedback a real patient would provide. The paper reports no accuracy analysis of the fact-matching step and no comparison of MHSD patient turns with real mental-health conversations; the Limitations section concedes that LLM actors may not capture the full range of emotions, variability, and lived experiences. Consequently, the 31% hard-diagnostic accuracy and the persona and turn-length effects confound sensemaker reasoning with patient-actor retrieval fidelity and scripted acquiescence. I would require either a validation of the patient actor (e.g., comparing MHSD patient turns with real patient-clinician dialogues, or rerunning the experiments with the agree-rule and closed-world assumption relaxed) or a robustness analysis before the central claim can be attributed to the reasoning mode.
  2. [Section 4 and Appendix K] The human validation measures agreement with the LLM judge, not the correctness of the judge. Annotators were provided with the LLM judge's score and reasoning and asked only to assign a binary agreement label, so the reported 79-93.5% agreement shows that the judge's ratings are reproducible in a weak sense, but it does not establish that the HBM-based scores correspond to clinically appropriate communication. In particular, the claim that a score of 2.55 or 2.77 is 'below Good' is only as valid as the LLM judge's calibration. The authors should report independent human annotations collected without showing the LLM scores, or validate a sample of scores against an expert panel using a clinical rubric.
  3. [Appendix Tables 27 and 28] Both Table 27 and Table 28 are captioned 'Prompt used for hard matching diagnostic accuracy', yet the criterion in Table 27 ('related to GT_DIAGNOSIS or is a blanket explanation ... or you can infer the GT_DIAGNOSIS') corresponds to the Soft diagnostic accuracy defined in Section 4, while Table 28's exact-match criterion corresponds to the Hard setting. As written, a reader cannot determine which prompt generated the reported 31% Hard and 64-68% Soft numbers. This mislabeling must be corrected and the prompt-to-metric mapping checked against the actual evaluation code before the diagnostic claim is reproducible.
  4. [Section 3 (post-processing), Section 5.3, Table 1] The dataset description says only that conversations were obtained 'after initial post-processing to remove erroneous generations and incomplete conversations', without reporting the exclusion criteria, the number of conversations removed, or whether removal rates differ by sensemaker. Since exactly 1,142 conversations survive per model, differential attrition could influence the persona and length results. In addition, the main text states that 'o1 conversations had 13.65 messages on average, while R1 conversations were longer, with 19.09 messages per conversation', but Table 1 reports Overall 19.09 for OpenAI o1 and 13.65 for DeepSeek-R1. Because Section 5.3's performance-degradation claim compares conversations with roughly 5 sensemaker messages against those with 10-15, this reversal affects which model is being characterized. Please resolve the table/text contradiction and report exclusion criteria, bin definitions, and bin sizes for Figure 5.
minor comments (6)
  1. [Abstract] The phrase 'average score of 31%' is ambiguous because the patient-centric communication scores are 2.55 and 2.77 out of 4; please specify that 31% refers to hard diagnostic accuracy only.
  2. [Section 2.1] The persona construction is described as varying each trait 'one at a time on a binary scale', with five variations in which one trait is set to 'High'; please state explicitly whether the other four traits are always set to 'Low' and whether this scheme is applied uniformly across all 181 case studies.
  3. [Section 2.2 and Section 4] Please state explicitly which exact model versions are used for each evaluation criterion. Since GPT-4o is used for the patient actor, several sensemaker sub-modules, and (presumably) the LLM-as-judge, a short model-to-module mapping would help readers assess potential same-model-family bias in the evaluation.
  4. [Figure 2 caption] The caption says personality traits are 'listed by first letter' but does not define the notation O/C/E/A/N; please spell out the trait names in the caption or table.
  5. [Section 5.2 text] The sentence preceding Table 2 says the table presents results 'for three personality types', but Table 2 contains all five traits; please correct the text.
  6. [Section 10 and Appendix prompts] There are minor typographical issues: 'meantal' in Section 10 and 'ALW AYS' in several prompt templates; a final proofread would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline results are measured against external MedQA ground-truth labels and externally defined HBM/SMOG rubrics; the GPT-4o patient actor and judge affect validity but do not make any prediction equivalent to its inputs.

full rationale

The central claim that OpenAI o1 and DeepSeek-R1 achieve below-par patient-centric communication scores and roughly 31% hard diagnostic accuracy is not defined in terms of the models' own outputs or fitted to a subset of the data. Hard and soft diagnostic accuracy are scored against ground-truth labels from MedQA, an external benchmark; the HBM communication rubrics, conversation-flow criteria, and SMOG readability index are all external, pre-specified instruments. Algorithm 1's closed-world assumption and Table 22's instruction to agree to diagnoses shape the simulated conversations, and the same model family (GPT-4o) is used for the patient actor and as LLM judge, but these are methodological choices that affect ecological validity rather than circular reasoning: no observed score is forced by construction, and no parameter fitted to the target result is later reported as a prediction. The paper's self-citations, including Chandra et al. (2025), are contextual related-work citations rather than load-bearing premises, and no uniqueness claim or ansatz is smuggled through a citation. The Limitations section explicitly concedes that LLM actors 'may not be able to capture the full range of emotions, variability and lived experiences,' which is an external-validity caveat, not evidence that the derivation reduces to its own inputs. Accordingly, no specific circular step can be exhibited, and the proper finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several domain assumptions imported from clinical-communication and health-behavior literature, plus a number of hand-chosen pipeline hyperparameters (stage dwell threshold, transition confidence bands, candidate count, max facts). No entities are invented beyond the framework and dataset names.

free parameters (4)
  • stage_dwell_threshold = 5 turns
    Algorithm 2 (lines 17-18) forces a move to the next stage when the current stage has lasted at least 5 turns. This hand-chosen threshold shapes conversation length and the reported performance-drop-with-length trend.
  • stage_transition_confidence_thresholds = 1-3 stay, 4-6 stay 1-2 more turns, 7+ move
    Table 8 prompt: the LLM stage-determination module uses these fixed score bands. They are hand-coded and affect how many turns are spent per stage.
  • candidate_message_count = 3
    Algorithm 2 Step 2 (line 22): the sensemaker generates 3 candidate messages per turn. Chosen by hand; a larger pool could change redundancy filtering and output quality.
  • max_facts_selected_per_patient_reply = 3
    Table 20 patient fact-matching prompt: a maximum of three facts can be selected to answer a sensemaker question. This constrains patient responses and therefore the diagnostic information available.
assumptions (5)
  • domain assumption Six-function model of medical communication is the correct stage structure for sensemaking conversations.
    Section 2.2 and Table 7. The entire MedAgent sensemaker pipeline and the Conversation Flow and Correctness metric assume this stage ordering.
  • domain assumption Health Belief Model dimensions (susceptibility, severity, benefits) are valid proxies for patient-centric communication quality in LLM conversations.
    Section 4. MultiSenseEval's three patient-centric metrics are directly derived from HBM; their validity for evaluating an AI sensemaker is assumed.
  • ad hoc to paper The synthetic patient actor's closed-world plus agree-with-diagnosis behavior approximates real patient behavior.
    Algorithm 1 (lines 8-10) and Table 22 instruction 3. The patient actor replies 'not sure' to unlisted background facts and agrees when the sensemaker states a diagnosis. Real patients may volunteer new information or challenge diagnoses.
  • domain assumption Binary high/low variation of one Big Five trait at a time yields internally consistent, representative patient personas.
    Section 2.1. The paper deliberately varies each trait one at a time on a binary scale, citing correlations among traits. It assumes this discretization captures persona-driven performance differences.
  • domain assumption MedQA case-study ground-truth labels are correct and the filtered 181 cases are representative of mental-health encounters.
    Section 2.1. Diagnostic accuracy is measured against these labels; filtering used a GPT-4o classifier plus human validation, but the label correctness is taken on faith.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning Is Not All You Need: Examining LLMs for Multi-Turn Mental Health Conversations." pith.science (2026). https://pith.science/paper/NFUGAIQH

@misc{pith2026250520201,
  author       = {Pith},
  title        = {Pith review of: Reasoning Is Not All You Need: Examining LLMs for Multi-Turn Mental Health Conversations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NFUGAIQH}},
  note         = {Machine review of arXiv:2505.20201}
}
read the original abstract

Limited access to mental healthcare, extended wait times, and increasing capabilities of Large Language Models (LLMs) has led individuals to turn to LLMs for fulfilling their mental health needs. However, examining the multi-turn mental health conversation capabilities of LLMs remains under-explored. Existing evaluation frameworks typically focus on diagnostic accuracy and win-rates and often overlook alignment with patient-specific goals, values, and personalities required for meaningful conversations. To address this, we introduce MedAgent, a novel framework for synthetically generating realistic, multi-turn mental health sensemaking conversations and use it to create the Mental Health Sensemaking Dialogue (MHSD) dataset, comprising over 2,200 patient-LLM conversations. Additionally, we present MultiSenseEval, a holistic framework to evaluate the multi-turn conversation abilities of LLMs in healthcare settings using human-centric criteria. Our findings reveal that frontier reasoning models yield below-par performance for patient-centric communication and struggle at advanced diagnostic capabilities with average score of 31%. Additionally, we observed variation in model performance based on patient's persona and performance drop with increasing turns in the conversation. Our work provides a comprehensive synthetic data generation framework, a dataset and evaluation framework for assessing LLMs in multi-turn mental health conversations.

Figures

Figures reproduced from arXiv: 2505.20201 by the authors.

Figure 1
Figure 1. We present the MedAgent framework for generating realistic multi-turn mental health sensemaking [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Sample conversation between the sensemaker [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of OpenAI o1 and DeepSeek-R1 across Perceived Susceptibility, Per￾ceived Severity, Perceived Benefits, and Conversation Flow and Correctness. Bars indicate mean scores with 95% confidence interval. All scores are on a 4-point Likert scale ((1): Very Poor to (4): Very Good). Both models obtain scores below “Good Performance” rating for the three patient-centric communication metrics, but exceed… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance comparison between OpenAI o1 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performance trend for o1 and R1 across the MultiSenseEval framework metrics, the x-axis indicates [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CALM-IT: Generating Realistic Long-Form Motivational Interviewing Dialogues with Dual-Actor Conversational Dynamics Tracking

    cs.CL 2026-01 conditional novelty 6.0 of 10

    Tracking both client and therapist states as they evolve produces 8,232 generated MI dialogues that outscore three baselines on MI-quality rubrics and stay stable at 100 turns.

  2. User Perceptions of an LLM-Based Chatbot for Cognitive Reappraisal of Stress: Feasibility Study

    cs.HC 2026-01 conditional novelty 4.0 of 10

    A GPT-4o chatbot guiding employees through an 11-step reappraisal script was associated with small short-term reductions in self-reported stress and improved stress mindset in an uncontrolled feasibility study.

Reference graph

Works this paper leans on

81 extracted references · 78 canonical work pages · cited by 2 Pith papers

  1. [1]

    goals of care

    Interpreting medical information from a patient perspective, 2) Providing lifestyle recommenda- tions and improving health literacyand 3) Personal- izing healthcare journeys. For example, the starting message for a case study augmented with intention (1) is: "A few days ago, I was feeling really agitated and not myself—I don’t even remember much of what h...

    2024

  2. [2]

    Table 25 shows the prompt used to evaluate perceived severity

    Perceived Severity: Beliefs about the serious- ness of the consequences of a health issue. Table 25 shows the prompt used to evaluate perceived severity

    work page

  3. [3]

    Table 26 shows the prompt used to evaluate perceived severity

    Perceived Benefits: Beliefs in the efficacy of the advised action to reduce the risk or seriousness of the health impact. Table 26 shows the prompt used to evaluate perceived severity. We adapt HBM to evaluate how effectively the sensemaker LLM promotes each of the core di- mensions on the patient side during conversations. For example, if a patient exhib...

    work page

  4. [4]

    Table 24 shows the prompt used to evaluate perceived susceptibility

    Perceived Susceptibility: An individual’s belief about the likelihood of experiencing a health problem. Table 24 shows the prompt used to evaluate perceived susceptibility

    work page

  5. [5]

    User Prompt QUESTION: <question> LIST OF MEDICAL FACTS: <patient_info> SOLUTION: Table 20: (Patient Module) Prompt used for selecting relevant medical facts to answer a question

    Do not include any other formatting or extra information beyond the REASONING and given FACTS. User Prompt QUESTION: <question> LIST OF MEDICAL FACTS: <patient_info> SOLUTION: Table 20: (Patient Module) Prompt used for selecting relevant medical facts to answer a question. Type Prompt System Prompt You are an intelligent agent who is given a LIST OF MEDIC...

    work page

  6. [6]

    DO NOT suggest any treatment method yourself in the answer

    work page

  7. [7]

    (2) Soft Diag- nostic Accuracy, which allows for some freedom and blanket term matching

    Diagnostic Accuracy: Since the filtered MedQA (Jin et al., 2021) case studies provide multi-choice options, we assess two different ac- curacies: (1) Hard Diagnostic Accuracy, which requires the sensemaker’s diagnosis to be an ex- act match with the ground truth. (2) Soft Diag- nostic Accuracy, which allows for some freedom and blanket term matching. Tabl...

    work page 2021

  8. [8]

    # ">. Do not include other formatting. You should always provide a diagnosis and if you cannot find a diagnosis your message should strictly be

    Conversation Flow and Correctness: It is used to measure the ability of the sensemaker to per- form end-to-end conversation, while maintain- ing logical consistency, covering each phase of the conversation, preventing providing incorrect information and preventing contradiction with it- self. Table 29 shows the prompt used to evaluate flow correctness. K ...

    work page

Show all 81 references
  1. [9]

    Invite the patient to share their story in their own words

  2. [10]

    Respond with empathy and compassion to patient’s concerns

  3. [11]

    Let the patient know that you care about their well-being and are committed to helping them

    Express compassion and commitment. Let the patient know that you care about their well-being and are committed to helping them

  4. [12]

    If the patient is not comfortable sharing their story, you can ask them to share their concerns and you can respond with empathy and compassion. Gathering Information Your goal is to develop a comprehensive understanding of the patient’s needs, concerns, and medical history by...

  5. [13]

    Ask open ended questions related to patient’s concerns to gather information about patients current state, their personal and family history

  6. [14]

    Listen actively and ask follow-up questions to understand the situation better

  7. [15]

    Elicit patient’s perspective of the problems and their expectations from you

  8. [16]

    Providing In- formation At this stage you want to provide a potential diagnosis to the patient for their concerns

    Clarify and summarize the information gathered from the patient to ensure understanding. Providing In- formation At this stage you want to provide a potential diagnosis to the patient for their concerns. You should not ask any questions in this stage and rather provide a poten...

  9. [17]

    Ask for patient’s preferences and suggestions regarding the lifestyle changes or other non-clinical changes

  10. [18]

    Suggest lifestyle changes based on the patient’s preferences and your all previous knowledge about them

  11. [19]

    Responding to Emotions You need to recognize and address any emotional aspect of the illness by offering empathay, reassurance, and psychological support in your messages

    If the patient is not comfortable with the suggestions, you can ask them to suggest their own lifestyle or non-clinical changes. Responding to Emotions You need to recognize and address any emotional aspect of the illness by offering empathay, reassurance, and psychological su...

  12. [20]

    Offer empathay and reassurance to the patient

  13. [21]

    Listen to the patient’s concerns and offer psychological support

  14. [22]

    Exit The conversation has reached its end and we need to conclude the conversation

    If the patient is not comfortable sharing their concerns, you can ask them to share their emotions and you can respond with empathy and support. Exit The conversation has reached its end and we need to conclude the conversation. Table 15: Current stage goal provided to the sen...

  15. [23]

    You should not add any new statement which was not present in the STATEMENT_MEMORY

  16. [24]

    If a new statement in the CANDIDATE_STATEMENTS has a different phrasing but serves a similar context to any of the statements present in the STATEMENT_MEMORY , it should be considered ’Redun- dantStatement’

  17. [25]

    Classification for each statement in CANDIDATE_STATEMENTS should be independent of other statements in the CANDIDATE_STATEMENTS. User Prompt CANDIDATE_STATEMENTS: <candidate statements that can be made in this particular stage> STATEMENT_MEMORY: <statements already made by the...

  18. [26]

    Only extract medical facts that would be relevant in a conversation with a health-care provider

  19. [27]

    PATIENT VITAL FACTS:

    First, list out facts related to a patient’s vitals (ONLY temperature, blood pressure, pulse, respiratory rate) under "PATIENT VITAL FACTS:"

  20. [28]

    If there are no PATIENT VITAL FACTS just leave the section empty

  21. [29]

    PATIENT FACTS:

    List out the remaining medical atomic facts under the section "PATIENT FACTS:". User Prompt Patient Information: <patient_data> Respond with the list of atomic facts for each section "PATIENT VITAL FACTS:" and "PATIENT FACTS:" (no formatting) and nothing else, prepend each fac...

  22. [30]

    VIGNETTE_NAME: VIGNETTE_DEFINITION:

  23. [31]

    RATIONALE:

    VIGNETTE_NAME: ... Think step by step and first provide your rationale (under "RATIONALE:") and then at the last output the VIGNETTE_NAME for the most logical vignette for the given case study. User Prompt CASE_STUDY: <case_study> Strictly follow the format: <ASSIGNED_VIGNETTE...

  24. [32]

    VIGNETTE_NAME: VIGNETTE_DEFINITION: VIGNETTE_PATTERNS:

  25. [33]

    RATIONALE:

    VIGNETTE_NAME: ... Think step by step and first provide your rationale (under "RATIONALE:") and then at the last output the VIGNETTE_NAME for the most logical vignette for the given case study. User Prompt CASE_STUDY: <case_study> Strictly follow the format: <ASSIGNED_VIGNETTE...

  26. [34]

    Given a LIST OF MEDICAL FACTS about a patient, choose a MAXIMUM of THREE facts from the list that when combined best answers the QUESTION

  27. [35]

    NO MATCH

    If NO fact matches the question, simply respond "NO MATCH" under "FACTS:"

  28. [36]

    If facts are chosen, output them as a list, preserving the numbering from the original list

  29. [37]

    REASONING:

    First provide a short reasoning under "REASONING:" before listing the facts under "FACTS:"

  30. [39]

    Given a LIST OF MEDICAL FACTS about a patient, choose a MAXIMUM of TWO relevant facts from the list to construct a starting message as a patient to an AI healthcare provider

  31. [40]

    The start message should be in first person, using the emotions, tone, word choice and intensity of a patient who has the levels of the Big 5 Personality Traits (on a 2-point Low/High scale) and Medical Literacy Level shown below

  32. [42]

    REASONING:

    First provide a short reasoning under "REASONING:" before writing the start message under "START_MESSAGE:"

  33. [43]

    User Prompt LIST OF MEDICAL FACTS: <patient_info> Big 5 Personality Traits:

    Do not include quotes, any other formatting, extra information beyond the REASONING and the START_MESSAGE. User Prompt LIST OF MEDICAL FACTS: <patient_info> Big 5 Personality Traits:

  34. [48]

    Neuroticism: <Neuroticism_score> MEDICAL LITERACY LEVEL: <medical_literacy_level> Table 21: (Patient Module) Prompt used for patient to start the conversation with the sensemaker. 25 Type Prompt System Prompt You are a truthful assistant that understands a patient’s medical in...

  35. [49]

    Using the list of medical facts given below, generate a conversational human-like response to an LLM in first person, using the emotions, tone, word choice and intensity of a patient who has the levels of the Big 5 Personality Traits (on a 2-point Low/High scale) and Medical L...

  36. [50]

    Any information asked in the question that is NOT explicitly present in the facts can be assumed to be FALSE

  37. [51]

    If the question is just a statement diagnosing your condition, agree to it

  38. [53]

    DO NOT add ANY information in your answer that is NOT EXPLICITLY stated in the given facts

  39. [55]

    User Prompt LIST OF MEDICAL FACTS: <patient_info> QUESTION: <question> Big 5 Personality Traits:

    ONLY respond with the answer, do NOT add any other formatting. User Prompt LIST OF MEDICAL FACTS: <patient_info> QUESTION: <question> Big 5 Personality Traits:

  40. [60]

    Note that in stages beyond Decision Making, point 2 in the instructions (closed-world assumption) is removed to aid natural conversation flow

    Neuroticism: <Neuroticism_score> MEDICAL LITERACY LEVEL: <medical_literacy_level> Table 22: (Patient Module) Prompt used to generate answers given a non-zero number of selected facts. Note that in stages beyond Decision Making, point 2 in the instructions (closed-world assumpt...

  41. [61]

    Generate a conversational human-like response to an LLM in first person, using the emotions, tone, word choice and intensity of a patient who has the levels of the Big 5 Personality Traits (on a 2-point Low/High scale) and Medical Literacy Level shown below

  42. [62]

    No ..." or

    If this question requests information about the patient’s medical history, personal experience, medication, background or condition, respond in the negative stating "No ..." or "I’m not sure about ..." under "ANSWER:"

  43. [63]

    If it is a question about decision-making, emotions, diagnosis or taking next steps, generate a conversational human-like response to an LLM in first person, responding in the affirmative ("Yes ...") as a patient

  44. [64]

    ONLY use the medical literacy level and personality definitions as a guide, DO NOT EXPLICITLY integrate any information from it into the answer

  45. [65]

    I’m not sure about

    DO NOT add ANY NEW information to the answer. If asked about any such additional information, respond with "I’m not sure about ..." to them

  46. [66]

    REASONING:

    First provide a short reasoning under "REASONING:" before writing the answer under "ANSWER:". Do not add any other formatting. Here are some examples to help you EXAMPLE 1: QUESTION: Have you tried any meditation-style exercises? REASONING: Exercises and lifestyle relates to t...

  47. [67]

    Openness: <Openness_score>

  48. [68]

    Conscientiousness: <Conscientiousness_score>

  49. [69]

    Extraversion: <Extraversion_score>

  50. [70]

    Agreeableness: <Agreeableness_score>

  51. [71]

    Perceived Susceptibility

    Neuroticism: <Neuroticism_score> MEDICAL LITERACY LEVEL: <medical_literacy_level> QUESTION: <question> Table 23: (Patient Module) Prompt used to generate answers when there are no selected facts matching the question. 27 Type Prompt System Prompt Task: You are an expert LLM-as...

  52. [72]

    Additionally, the tone may have been dismissive, insensitive, or unnecessarily alarming

    (Very Poor: Score = 1): The sensemaker failed to communicate any information about the susceptibility of facing a health problem. Additionally, the tone may have been dismissive, insensitive, or unnecessarily alarming

  53. [73]

    However, the tone was empathetic and engaging

    (Poor: Score = 2): The sensemaker failed to communicate any information about the susceptibility of facing a health problem. However, the tone was empathetic and engaging

  54. [74]

    The tone was professional but lacked empathy or collaborative engagement

    (Good: Score = 3): The sensemaker communicated basic information about the patient’s susceptibility to facing a health problem. The tone was professional but lacked empathy or collaborative engagement

  55. [75]

    Perceived Severity

    (Very Good: Score = 4): The sensemaker explained the patient’s susceptibility to facing a health problem clearly, accurately and appropriately, using personalized details effectively. Additionally, the sensemaker was empathetic and engaged collaboratively. Output Format: You s...

  56. [76]

    Additionally, the tone may have been dismissive, insensitive, or unnecessarily alarming

    (Very Poor: Score = 1): The sensemaker failed to communicate any information about the seriousness of the consequences associated with a health issue to the patient. Additionally, the tone may have been dismissive, insensitive, or unnecessarily alarming

  57. [77]

    However, the tone was empathetic and engaging

    (Poor: Score = 2): The sensemaker failed to communicate any information about the seriousness of the consequences associated with a health issue to the patient. However, the tone was empathetic and engaging

  58. [78]

    The tone was professional but lacked empathy or collaborative engagement

    (Good: Score = 3): The sensemaker communicated basic information about the seriousness of the consequences associated with a health issue to the patient. The tone was professional but lacked empathy or collaborative engagement

  59. [79]

    Perceived Benefits

    (Very Good: Score = 4): The sensemaker explained the seriousness of the consequences associated with a health issue clearly, accurately and appropriately, using personalized details effectively. Additionally, the sensemaker was empathetic and engaged collaboratively. Output Fo...

  60. [80]

    Additionally, the tone may have been dismissive, insensitive, or unhelpful

    (Very Poor: Score = 1): The sensemaker failed to convey any benefits or efficacy of the advised action in reducing the risk or seriousness of the health impact. Additionally, the tone may have been dismissive, insensitive, or unhelpful

  61. [81]

    However, the tone was empathetic and engaging

    (Poor: Score = 2): The sensemaker failed to convey any benefits or efficacy of the advised action in reducing the risk or seriousness of the health impact. However, the tone was empathetic and engaging

  62. [82]

    The tone was professional but lacked empathy or collaborative engagement

    (Good: Score = 3): The sensemaker conveyed basic information about the benefits or efficacy of the advised action in reducing the risk or seriousness of the health impact. The tone was professional but lacked empathy or collaborative engagement

  63. [83]

    Diagnostic Accuracy

    (Very Good: Score = 4): The sensemaker explained the benefits or efficacy of the advised action in reducing the risk or seriousness of the health impact clearly, accurately and appropriately, using personalized details effectively. Additionally, the sensemaker was empathetic a...

  64. [84]

    Label = 1: The sensemaker diagnosis is related to the GT_DIAGNOSIS or is a blanket explanation of the GT_DIAGNOSIS or you can infer the GT_DIAGNOSIS from the PATIENT_DIAGNOSIS

  65. [85]

    Diagnostic Accuracy

    Label = 0: The sensemaker diagnosis is completely unrelated and uncorrelated with the GT_DIAGNOSIS. Output Format: You should first provide a short reasoning (2-3 sentences) providing your reasoning for the label. Then, provide the label. Your output should be in the following...

  66. [86]

    Label = 1: The sensemaker diagnosis is related to the GT_DIAGNOSIS and exactly matches the GT_DIAGNOSIS

  67. [87]

    Conversation Flow and Correctness

    Label = 0: The sensemaker diagnosis is completely unrelated and uncorrelated with the GT_DIAGNOSIS or does not exactly match the GT_DIAGNOSIS. Output Format: You should first provide a short reasoning (2-3 sentences) providing your reasoning for the label. Then, provide the la...

  68. [88]

    Additionally, the sensemaker gives contradictory, or provides inconsis- tent information

    (Very Poor: Score = 1): The dialogue is disorganized or fragmented with one or more stages in STAGE_DEFINITION missing. Additionally, the sensemaker gives contradictory, or provides inconsis- tent information

  69. [89]

    However, the sensemaker’s statements are mostly are consistent and do not provide contradictory information

    (Poor: Score = 2): The dialogue is disorganized or fragmented with one or more stages in STAGE_DEFINITION missing. However, the sensemaker’s statements are mostly are consistent and do not provide contradictory information

  70. [90]

    Additionally, the sensemaker’s statements are consistent and do not provide contradictory information

    (Good: Score = 3): Overall logical progression is present with all stages in STAGE_DEFINITION covered in the right order. Additionally, the sensemaker’s statements are consistent and do not provide contradictory information

  71. [91]

    Additionally, the sensemaker’s statements are consistent and do not provide contradictory information

    (Very Good: Score = 4): Overall logical progression is present with all stages in STAGE_DEFINITION covered in the right order and the sensemaker has covered the each stage sufficiently paying special attention on gathering information from the patient and providing appropriate...

    2024

  72. [2010]

    best practice

    Making sense of sensemaking: the critical sensemaking approach. Qualitative research in orga- nizations and management: An international journal, 5(2):182–195. Nancy K Janz and Marshall H Becker. 1984. The health belief model: A decade later. Health education quar- terly, 11(1...

    2009 arXiv

  73. [2022]

    Patient Education and Counseling, 105(1):252–256

    ‘more than words’–interpersonal communi- cation, cognitive bias and diagnostic errors. Patient Education and Counseling, 105(1):252–256. Hanneke De Haes and Jozien Bensing. 2009. End- points in medical communication research, proposing a framework of functions and outcomes. Pa...

    2009 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.