REVIEW 10 cited by
Towards Conversational Diagnostic AI
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
At the heart of medicine lies the physician-patient dialogue, where skillful history-taking paves the way for accurate diagnosis, effective management, and enduring trust. Artificial Intelligence (AI) systems capable of diagnostic dialogue could increase accessibility, consistency, and quality of care. However, approximating clinicians' expertise is an outstanding grand challenge. Here, we introduce AMIE (Articulate Medical Intelligence Explorer), a Large Language Model (LLM) based AI system optimized for diagnostic dialogue. AMIE uses a novel self-play based simulated environment with automated feedback mechanisms for scaling learning across diverse disease conditions, specialties, and contexts. We designed a framework for evaluating clinically-meaningful axes of performance including history-taking, diagnostic accuracy, management reasoning, communication skills, and empathy. We compared AMIE's performance to that of primary care physicians (PCPs) in a randomized, double-blind crossover study of text-based consultations with validated patient actors in the style of an Objective Structured Clinical Examination (OSCE). The study included 149 case scenarios from clinical providers in Canada, the UK, and India, 20 PCPs for comparison with AMIE, and evaluations by specialist physicians and patient actors. AMIE demonstrated greater diagnostic accuracy and superior performance on 28 of 32 axes according to specialist physicians and 24 of 26 axes according to patient actors. Our research has several limitations and should be interpreted with appropriate caution. Clinicians were limited to unfamiliar synchronous text-chat which permits large-scale LLM-patient interactions but is not representative of usual clinical practice. While further research is required before AMIE could be translated to real-world settings, the results represent a milestone towards conversational diagnostic AI.
Forward citations
Cited by 10 Pith papers
-
Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent
A 70-row table compiled from ophthalmology guidelines can label 3,000 synthetic dialogues, and fine-tuning a 9B model on them improves agreement with an author-defined reference from 61.7% to 74.1% and emergent recall...
-
Active Learning for Neurosymbolic Program Synthesis
The abstract claims a new active learning technique, constrained conformal evaluation (tool SmartLabel), that finds the ground-truth program in 98% of benchmarks, but the delivered full text is a different paper, leav...
-
BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning
BELO is a new ophthalmology benchmark of 900 expert-checked multiple-choice questions with reasoning, used to evaluate six LLMs on accuracy and explanation quality.
-
BehaviorSFT: Behavioral Token Conditioning for Clinical Agents Across the Proactivity Spectrum
BehaviorSFT uses `reactive` and `proactive` control tokens to fine-tune clinical LLM agents, improving their scores on the authors' new BehaviorBench dataset, though the benchmark is AI-generated and minimally clinici...
-
MedBrowseComp: Benchmarking Medical Deep Research and Computer Use
A new benchmark of more than 1,000 multi-hop medical browsing questions shows that even the best deep-research and computer-use AI agents answer fewer than half correctly.
-
MM-Agent: LLM as Agents for Real-world Mathematical Modeling Problem
MM-Agent, a multi-stage LLM pipeline with a hierarchical modeling method library, is claimed to outperform prior agents and award-winning human solutions on a new 111-problem MCM/ICM-based mathematical modeling benchmark.
-
MATRIX: Multi-Agent simulaTion fRamework for safe Interactions and conteXtual clinical conversational evaluation
MATRIX combines a structured safety taxonomy, an LLM hazard judge, and a patient simulator to benchmark clinical dialogue agents, claiming expert-level hazard detection and revealing weak emergency handling in current LLMs.
-
A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domains
A new 2,069-item, 30-criterion benchmark of clinical scenarios shows six LLMs average 57.2%, with safety (54.7%) below effectiveness (62.3%) and a 13.3% drop in high-risk cases.
-
AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Data
A patient simulator built from EHR vignettes was rated consistent with those vignettes in 97.7% of 519 conversations by two clinicians, while the AI triage system's top three diagnoses contained the most likely diagno...
-
The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy
Medical agents should be scaled mainly by richer clinical environments and self-evolution loops, not parameter growth alone, under a three-level autonomy taxonomy.
Discussion (0). Sign in to comment.