Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

Clinical knowledge in LLMs does not translate to human interactions

T0 review · 2 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper reports that proficient LLMs provide no measurable benefit, and even some harm, when the public uses them for medical self-assessment.

desk verdict Large RCT shows LLM assistance doesn't help laypeople; before taking the 'worse than control' condition result at face value, ask for a blinded semantic re-score. read the letter →

arxiv 2504.18919 v1 pith:KDKEFTE5 submitted 2025-04-26 cs.HC cs.AIcs.CL

Andrew M. Bean , Rebecca Payne , Guy Parsons , Hannah Rose Kirk , Juan Ciro , Rafael Mosquera , Sara Hincapié Monsalve , Aruna S. Ekanayaka
show 3 more authors
Lionel Tarassenko Luc Rocher Adam Mahdi
This is my paper · ORCID
classification cs.HCcs.AIcs.CL
keywords largelanguagemodelsmedicaladvicehuman-computerinteractionrandomizedcontrolledtrialbenchmarkspatientsafetyAI-assisteddecisionmakinghealthinformation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a randomized controlled trial in which 1,298 members of the public used one of three large language models (GPT-4o, Llama 3, Command R+) to decide what to do about ten everyday medical scenarios. Tested alone, the models named a relevant condition in 94.9% of cases and chose the correct disposition in 56.3%. But the participants who used the models identified a relevant condition in at most 34.5% of cases, worse than the control group's 47.0%, and were no better than control at choosing the right disposition. The authors argue that the bottleneck is the human-model interaction, not the model's medical knowledge, and that neither medical licensing-style benchmarks nor simulated-patient interactions predict these failures. This matters because health services are considering deploying LLMs for direct public consultations.

What carries the argument

The study's central object is a randomized, between-subjects trial with four arms: three LLM-assisted arms and a control arm, all scored against physician-constructed gold standards. Ten scenarios were drafted by three doctors and a gold-standard list of relevant conditions was built from the union of differentials given by four additional doctors; the correct disposition was set by unanimous agreement. Two further comparison devices carry the argument: direct prompting of each LLM with the full scenario text (model-only performance), and a simulated-user condition where an LLM instance played the patient while another LLM provided assistance. These three layers isolate the effect of inserting a human user between a competent model and the task.

What would settle it

Take the same ten scenarios and re-score a fresh set of participants using a gold standard that includes only the single condition each scenario was designed to test, rather than the full union of differentials. If LLM-assisted participants then outperform the control group at naming that condition, the paper's 'no benefit' conclusion for condition identification would collapse; if they still do not, the interaction-failure explanation would be confirmed.

Watch

Extended reading notes

Core claim

The central discovery is that expert-level performance by an LLM in isolation does not translate into improved performance by a non-expert using that LLM, and can even reduce it. In the trial, LLM users were significantly less likely than the unassisted control group to name at least one condition from the physicians' gold-standard list (all three models, p<0.001), and their disposition accuracy was statistically indistinguishable from control. Analysis of the chat transcripts shows two points of breakdown: users often gave the model incomplete or misleading information, and even when the model suggested a correct condition, users frequently did not carry it into their final answer. These failures persist even though the same models score well above passing on a scenario-matched subset of MedQA, showing that knowledge benchmarks do not capture human-LLM interaction performance.

Load-bearing premise

The condition-identification results rest on the gold-standard lists of relevant conditions being the right target; these lists are the union of four physicians' free-text differentials, so if they are too broad, the low absolute scores could partly reflect a scoring artifact, although the comparison with the control group remains internally valid.

Editorial extensions

If this is right

  • If this pattern holds, current medical licensing-style benchmarks overstate the readiness of LLMs for direct public use.
  • Simulated-patient evaluations that replace humans with LLMs can give misleadingly optimistic or falsely stable results.
  • Improving the base model's accuracy alone will not close the gap; effort must go into the interaction design, including information elicitation and conveying recommendations.
  • LLM users may underestimate the acuity of serious conditions, which carries asymmetric risk compared with overestimation.
  • Public-facing medical LLMs will need to be proactive in requesting missing information, rather than passively responding to whatever the user says.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A concrete extension would be to vary the interface, for example by requiring structured questionnaires before the LLM responds, and test whether the condition-identification gap narrows; if it does, the failure is at information elicitation rather than comprehension or trust.
  • The condition-identification result may be sensitive to how the gold-standard list is scored; a strict scoring that only counts the single intended condition could change absolute numbers while preserving the control comparison, clarifying how much of the low rate is artifact.
  • The interaction-level finding suggests a latent failure mode for other high-stakes advisory domains, such as legal or financial advice: knowledge stored in the model is not the same as knowledge delivered through a chat exchange.
  • The weakness of simulated users as predictors points to a need for small but real human samples in safety evaluations, which regulators might mandate at far lower cost than is often imagined.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper reports a randomized controlled trial with 1,298 UK participants who were asked to assess one of ten physician-written medical scenarios. Participants were randomized to one of three LLM assistance arms (GPT-4o, Llama 3, Command R+) or a control arm that used usual home resources (mainly internet search). The study measured whether participants identified at least one condition from a physician-generated gold-standard list and whether they selected the correct disposition on a five-point acuity scale. The LLMs alone answered the same scenario questions correctly in 94.9% of cases for conditions and 56.3% for dispositions, but participants using the LLMs identified relevant conditions in at most 34.5% of responses (control: 47.0%) and showed no significant improvement in disposition accuracy relative to control. The paper further argues that performance on MedQA and on simulated-patient interactions does not predict these human-LLM interaction failures, and recommends human user testing before deployment.

Significance. If the findings are robust, the paper provides important evidence that high standalone performance of LLMs on medical knowledge tasks does not guarantee effective assistance for lay users. Strengths include a preregistration-style randomized design, stratified sampling, physician-generated scenarios and gold standards, a control group with realistic home resources, attrition analysis, public code and data, and multiple robustness checks. The finding that participants using LLMs were no better than control on disposition and worse on condition identification, despite the underlying LLMs performing well alone, is a practically consequential result for regulators and healthcare providers. However, the strength of the condition-identification claim depends on the validity of the fuzzy-matching scoring procedure, and the simulated-patient comparison rests on a very small number of simulation repetitions. These issues do not undermine the overall direction of the work but require attention before the headline claims can be accepted.

major comments (2)
  1. [Methods, Scoring; Supplementary 'Relevant condition ablations'] The central comparison for relevant conditions (Fig. 2b: 47.0% control vs at most 34.5% for each LLM arm, and Fig. 2: 94.9% LLM-alone vs below 34.5% with users) rests on an orthographic fuzzy-match rule. A response is scored as correct if it has at least 80% character similarity to any term on the physician-generated gold-standard list. This rule cannot recognize semantically equivalent lay paraphrases such as 'blood clot in the lung' for pulmonary embolism or 'kidney infection' for pyelonephritis. The control arm, instructed to use usual home resources, can copy exact disease names from internet search results, whereas LLM users receive conversational explanations and must restate them in their own words. The two arms are therefore not symmetric under this scoring rule. The threshold was tuned on 200 manually scored cases, but that sample was used to select the threshold and is not an independent validation of arm-specific lexical behavior; the reported 95.8% precision and recall do not rule out a systematic undercount of LLM-arm paraphrases. I ask the authors to re-score a random sample of responses from each arm with blinded clinician semantic judgment and to report whether the condition-identification difference between LLM users and control, and the LLM-alone versus with-user gap, persist under that scoring.
  2. [Methods, Simulated participants baseline; Fig. 4b] The conclusion that simulated users do not predict human performance is based on only 10 simulation repetitions per model-scenario cell (300 simulated conversations in total). With 10 Bernoulli trials per cell, observing 0% or 100% accuracy in 26 of 30 scenarios is expected under substantial sampling variation, and the per-scenario regression coefficients have wide interval estimates (e.g., 0.20 ± 0.38 for Command R+ on disposition; -0.01 ± 0.51 for Command R+ on relevant conditions). The point estimates near zero are suggestive, but the evidence is too weak to support the strong claim that simulated interactions 'do not predict' human-LLM failures. A larger number of simulation repetitions, or reporting interval estimates for the correlations and a power analysis, would be needed to make this claim convincing.
minor comments (4)
  1. [Methods, User interactions; Fig. 3] The analysis of conditions mentioned during user-LLM conversations uses GPT-4o to extract medical conditions from the transcripts, but this extraction is not validated. Since this extraction underlies the claim that LLMs suggested relevant conditions in 65.7-73.2% of conversations and that only 34.0% of suggestions were correct, a small validation study (e.g., comparing extraction against human annotation on a sample of transcripts) would strengthen the interaction-failure interpretation.
  2. [Methods, Question-answering baseline] The MedQA subset contains only 236 questions, and the per-scenario samples are very small (e.g., tinnitus n=6, allergic rhinitis n=30). The comparison of benchmark accuracy to human experimental accuracy across scenarios would benefit from reporting uncertainty intervals for the per-scenario benchmark estimates, especially for scenarios with fewer than ten questions.
  3. [Methods, Scenarios and Scoring; Supplementary Table 14] The gold-standard condition lists are the union of four physicians' differentials and include some very broad items (e.g., 'cold' for allergic rhinitis; 'depression' for anaemia). Because participants are scored as correct if any fuzzy-matched term appears, the absolute condition-identification rates may be inflated by broad lists. I recommend reporting a sensitivity analysis that restricts scoring to red-flag conditions or to conditions that at least two physicians listed.
  4. [Results, Experimental performance] The sentence 'The overall correct response rate of 43.0%±2.0% exceeds a random guessing baseline of 20%' is reported without a clear denominator; please clarify whether this is the average across all arms or the control arm, and state the corresponding test statistic.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; the study is an empirical RCT with external control and independent gold-standard scoring.

full rationale

The paper makes no derivation claim that reduces to its inputs. The central comparison is between randomized arms (LLM-assisted vs. control) using a physician-generated gold standard, and the headline claim (LLM assistance does not improve, and for conditions worsens, layperson performance) rests on direct experimental measurement rather than on an equation or fitted parameter. The 'LLM alone' baseline is an independent prompt-evaluation on the same scenarios, not a prediction derived from the human data. The MedQA subset is constructed by filtering external exam questions using the scenario gold-standard conditions; this is a benchmark-construction choice, and the subsequent claim that benchmarks do not predict interactive performance is an empirical negative result, not a tautology. The simulated-patient baseline is a design choice (GPT-4o simulated patients) borrowed from prior work and used for comparability; the finding that simulated users do not match human users does not presuppose that result. The fuzzy-matching threshold was calibrated on a manually scored sample, but this calibrates outcome measurement and does not fit any parameter that is then renamed as a prediction. No self-citation is load-bearing: cited prior work provides prompts, platform, and context, not the paper's conclusions. The paper's own limitations (e.g., common-condition focus, LLM-alone as a minimum estimate) are acknowledged scope notes, not admissions of circularity. Overall, the derivation chain is self-contained and empirically grounded, so no circular step can be quoted or exhibited.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The central claim rests on measurement assumptions about the scenarios and gold standards, plus the unvalidated GPT-4o transcription of conversation content.

free parameters (2)
  • Fuzzy matching character difference threshold = 20% (80% match)
    Selected by grid search on 200 manually annotated responses to balance precision and recall for scoring whether a participant named a gold-standard condition. Affects the condition-identification rate but not the treatment comparison.
  • MedQA question filtering rules = Exclude keywords matching more than 10 or less than 1 questions; one scenario per keyword
    Researcher-chosen rules to build scenario-specific benchmark subsets. Affects the benchmark accuracy values but not the main RCT result.
assumptions (4)
  • domain assumption Gold-standard dispositions are the correct answer for each scenario.
    Three physicians revised scenarios until unanimous agreement on the best disposition; this unanimity is treated as ground truth (Methods, Scenarios).
  • domain assumption Union of four physicians' differential diagnoses forms a valid gold-standard list of relevant conditions.
    Methods, Scoring: participant free-text responses were matched against this union; its completeness and breadth determine the condition-identification metric.
  • domain assumption LLM-simulated patients using GPT-4o are a representative proxy for real users in the simulation baseline.
    Methods, Simulated participants baseline: GPT-4o is used for all simulated patients, and the authors compare this to human behavior; the validity of that proxy is assumed.
  • domain assumption GPT-4o accurately extracts medical conditions mentioned in conversation transcripts.
    Results, Performance in user interactions: no validation of the GPT-4o extraction is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Clinical knowledge in LLMs does not translate to human interactions." pith.science (2026). https://pith.science/paper/KDKEFTE5

@misc{pith2026250418919,
  author       = {Pith},
  title        = {Pith review of: Clinical knowledge in LLMs does not translate to human interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDKEFTE5}},
  note         = {Machine review of arXiv:2504.18919}
}
read the original abstract

Global healthcare providers are exploring use of large language models (LLMs) to provide medical advice to the public. LLMs now achieve nearly perfect scores on medical licensing exams, but this does not necessarily translate to accurate performance in real-world settings. We tested if LLMs can assist members of the public in identifying underlying conditions and choosing a course of action (disposition) in ten medical scenarios in a controlled study with 1,298 participants. Participants were randomly assigned to receive assistance from an LLM (GPT-4o, Llama 3, Command R+) or a source of their choice (control). Tested alone, LLMs complete the scenarios accurately, correctly identifying conditions in 94.9% of cases and disposition in 56.3% on average. However, participants using the same LLMs identified relevant conditions in less than 34.5% of cases and disposition in less than 44.2%, both no better than the control group. We identify user interactions as a challenge to the deployment of LLMs for medical advice. Standard benchmarks for medical knowledge and simulated patient interactions do not predict the failures we find with human participants. Moving forward, we recommend systematic human user testing to evaluate interactive capabilities prior to public deployments in healthcare.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MedRedFlag: Investigating how LLMs Redirect Misconceptions in Real-World Health Communication

    cs.CL 2026-01 unverdicted novelty 7.0 of 10

    LLMs often fail to redirect health questions containing misconceptions, unlike clinicians, exposing safety gaps in patient-facing medical AI.

Reference graph

Works this paper leans on

34 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    ChatGPT diagnoses cause of child’s chronic pain after 17 doctors failed.The Independent(2023)

    O’Neill, M. ChatGPT diagnoses cause of child’s chronic pain after 17 doctors failed.The Independent(2023)

    work page 2023

  2. [2]

    & Published, L

    Presiado, M., Montero, A., Lopes, L. & Published, L. H. KFF Health Misinfor- mation Tracking Poll: Artificial Intelligence and Health Information (2024)

    work page 2024

  3. [3]

    & Choudhury, A

    Shahsavar, Y. & Choudhury, A. User Intentions to Use ChatGPT for Self- Diagnosis and Health-Related Purposes: Cross-sectional Survey Study.JMIR Human Factors10, e47564 (2023)

    work page 2023

  4. [4]

    Jin, D.et al.What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams.Applied Sciences11, 6421 (2021)

    work page 2021

  5. [5]

    W.et al.Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum.JAMA Internal Medicine183, 589–596 (2023)

    Ayers, J. W.et al.Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum.JAMA Internal Medicine183, 589–596 (2023). 22

    work page 2023

  6. [6]

    Van Veen, D.et al.Adapted large language models can outperform medical experts in clinical text summarization.Nature Medicine30, 1134–1142 (2024)

    work page 2024

  7. [7]

    Williams, C. Y. K.et al.Use of a Large Language Model to Assess Clinical Acuity of Adults in the Emergency Department.JAMA Network Open7, e248895 (2024)

    work page 2024

  8. [8]

    & Salz, T

    Agarwal, N., Moehring, A., Rajpurkar, P. & Salz, T. Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology (2023). 31422

    work page 2023

Show all 34 references
  1. [9]

    G.et al.Superhuman performance of a large language model on the reasoning tasks of a physician (2024)

    Brodeur, P. G.et al.Superhuman performance of a large language model on the reasoning tasks of a physician (2024). 2412.10849

    2024 arXiv

  2. [10]

    & Topol, E

    Rajpurkar, P. & Topol, E. J. Opinion|The Robot Doctor Will See You Now. The New York Times(2025)

    2025

  3. [11]

    J., Simmons, J

    Dietvorst, B. J., Simmons, J. P. & Massey, C. Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology. General144, 114–126 (2015)

    2015

  4. [12]

    & Cohen, I

    Babic, B., Gerke, S., Evgeniou, T. & Cohen, I. G. Beware explanations from AI in health care.Science373, 284–286 (2021)

    2021

  5. [13]

    Doshi, A. R. & Hauser, O. P. Generative artificial intelligence enhances creativity but reduces the diversity of novel content (2024). arXiv:2312.00506

    2024 arXiv

  6. [14]

    Digital Access – a ‘front door to the NHS’

    NHS England-South East. Digital Access – a ‘front door to the NHS’

  7. [15]

    AI Opportunities Action Plan (2025)

    Kyle, P. AI Opportunities Action Plan (2025)

    2025

  8. [16]

    Grand Challenges Africa: Catalyzing Equitable Artificial Intelligence (AI) Use to Improve Global Health (2023)

    Global Grand Challenges. Grand Challenges Africa: Catalyzing Equitable Artificial Intelligence (AI) Use to Improve Global Health (2023). 23

    2023

  9. [17]

    HHS Releases Strategic Plan for the Use of Artificial Intelligence to Enhance and Protect the Health and Well-Being of Americans (2025)

    US Department of Health and Human Services. HHS Releases Strategic Plan for the Use of Artificial Intelligence to Enhance and Protect the Health and Well-Being of Americans (2025)

    2025

  10. [18]

    Haupt, C. E. & Marks, M. AI-Generated Medical Advice—GPT and Beyond. JAMA329, 1349–1350 (2023)

    2023

  11. [19]

    & Petro, J

    Lee, P., Bubeck, S. & Petro, J. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine.New England Journal of Medicine388, 1233–1239 (2023)

    2023

  12. [20]

    Nature616, 259–265 (2023)

    Moor, M.et al.Foundation models for generalist medical artificial intelligence. Nature616, 259–265 (2023)

    2023

  13. [21]

    Singhal, K.et al.Large language models encode clinical knowledge.Nature1–9 (2023)

    2023

  14. [22]

    Tu, T.et al.Towards Generalist Biomedical AI.NEJM AI1(2024)

    2024

  15. [23]

    M., Carignan, D

    Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on Medical Challenge Problems (2023). arXiv:2303.13375

    2023 arXiv

  16. [24]

    arXiv:2405.07960

    Schmidgall, S.et al.AgentClinic: A multimodal agent benchmark to evaluate AI in simulated clinical environments (2024). arXiv:2405.07960

    2024 arXiv

  17. [25]

    Johri, S.et al.An evaluation framework for clinical use of large language models in patient interaction tasks.Nature Medicine1–10 (2025)

    2025

  18. [26]

    arXiv:2404

    Saab, K.et al.Capabilities of Gemini Models in Medicine (2024). arXiv:2404. 18416

    2024

  19. [27]

    Goh, E.et al.Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial.JAMA Network Open7, e2440969 (2024). 24

    2024

  20. [28]

    & Kamar, E

    Wilder, B., Horvitz, E. & Kamar, E. Learning to Complement Humans (2020)

    2020

  21. [29]

    Bansal, G.et al.Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2021)

    2021

  22. [30]

    Kiela, D.et al.Dynabench: Rethinking Benchmarking in NLP (2021)

    2021

  23. [31]

    Rahnev, D.et al.The Confidence Database.Nature Human Behaviour4, 317–325 (2020)

    2020

  24. [32]

    NICE guidelines

    National Institute for Health and Care Excellence. NICE guidelines

  25. [33]

    C., Ostermann, T

    Vollmar, H. C., Ostermann, T. & Reda` elli, M. Using the scenario method in the context of health and health care – a scoping review.BMC Medical Research Methodology15, 89 (2015)

    2015

  26. [34]

    Z., Shah, S., Stone, J

    Xiong, K. Z., Shah, S., Stone, J. A., Jacobson, N. & Chui, M. A. Using a Scenario- Based Hybrid Approach to Understand Participant Health Behavior.Research in social & administrative pharmacy : RSAP17, 2070 (2021). Supplementary information.This paper has Supplementary Materia...

    2021

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.