REVIEW 16 cited by
Large Language Models Encode Clinical Knowledge
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, but the quality bar for medical and clinical applications is high. Today, attempts to assess models' clinical knowledge typically rely on automated evaluations on limited benchmarks. There is no standard to evaluate model predictions and reasoning across a breadth of tasks. To address this, we present MultiMedQA, a benchmark combining six existing open question answering datasets spanning professional medical exams, research, and consumer queries; and HealthSearchQA, a new free-response dataset of medical questions searched online. We propose a framework for human evaluation of model answers along multiple axes including factuality, precision, possible harm, and bias. In addition, we evaluate PaLM (a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM, on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA, MedMCQA, PubMedQA, MMLU clinical topics), including 67.6% accuracy on MedQA (US Medical License Exam questions), surpassing prior state-of-the-art by over 17%. However, human evaluation reveals key gaps in Flan-PaLM responses. To resolve this we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, recall of knowledge, and medical reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal important limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLM models for clinical applications.
Forward citations
Cited by 16 Pith papers
-
MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
On a new benchmark of 5,620 real multimodal online consultations, top LLMs trail the original physicians mainly because they trigger more unsafe or unsupported negative criteria.
-
EpiBench: Can LLMs Understand Epitopes for Antibody Drug Discovery?
A new closed-book, sequence-only benchmark, EpiBench, measures epitope reasoning in LLMs and finds them near chance on residue-level localization and escape assessment, with only coarse region-level signal.
-
CardioBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios
MyoCardBench, a 13-task real-world cardiology benchmark, shows GPT-5.4 leading seven LLMs while ECG reading and clinical ethics remain near-random.
-
PertReason: A Knowledge-Grounded Benchmark and Framework for Cell-State-Conditioned Mechanistic Reasoning of Perturbation Effects
PertReasonQA scores AI models on cell-state-conditioned mechanistic reasoning about perturbation effects, and PertReasonLM, trained with reasoning supervision, reaches 0.736 balanced accuracy and 0.976 edge recall ver...
-
CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching
A new benchmark and training strategy show LLMs trained to internalize causal reasoning steps are less fooled by semantically similar, label-flipped questions than models using explicit chain-of-thought.
-
Essential-Web v1.0: 24T tokens of organized web data
A 24T-token web corpus with automatic document-level taxonomy labels enables competitive domain-specific datasets via simple filters.
-
Optimization-Inspired Few-Shot Adaptation for Large Language Models
OFA tunes LayerNorm parameters as optimization preconditioners and adds step-ratio and sharpness penalties, reporting consistent few-shot accuracy gains over baselines on Llama and GPT-2 models.
-
TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
Trident-Bench provides 2,652 professionally validated harmful prompts across finance, law, and medicine, and shows that domain-specialized LLMs often comply with unethical requests more than generalist models.
-
CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
CLONE combines generative pruning, LoRA adapters, a parameter-free Mixture-of-Experts router, and learning-based DVFS to make LLM inference on edge devices faster and more energy-efficient, claiming up to 11.92x speed...
-
Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection
Using hallucinated benchmark answers as rejected DPO pairs, ordered by an external fact-checker's grounding score, improves hallucination detection in 1B-3B Llama models.
-
Towards medical AI misalignment: a preliminary study
A custom role-playing prompt called the Goofy Game made four major LLMs produce plausible but incorrect medical recommendations.
-
RADAR: Enhancing Radiology Report Generation with Supplementary Knowledge Injection
RADAR filters an LLM's radiology findings by agreement with an expert classifier and retrieves only the missing observations, reporting improved clinical accuracy on three datasets.
-
Open-Source Agentic Hybrid RAG Framework for Scientific Literature Review
A DPO-tuned agentic hybrid RAG system that routes queries between a knowledge graph and a vector store beat a static baseline on a self-generated benchmark.
-
Prompt Mechanisms in Medical Imaging: A Comprehensive Survey
A broad survey that organizes prompt mechanisms for medical image generation, segmentation, and classification into a two-dimensional taxonomy of core technologies and clinical applications.
-
Balancing Knowledge Delivery and Emotional Comfort in Healthcare Conversational Systems
Fine-tuning a 1B medical chatbot on LLM-rewritten emotional dialogues improves its emotion scores with only small changes in n-gram overlap with the original medical responses.
-
BioPars: A Pretrained Biomedical Large Language Model for Persian Biomedical Text Mining
A proposed Persian biomedical LLM, BioPars, is evaluated on medical QA datasets and reported to beat GPT-4 on a self-built Persian QA benchmark, but the training setup is not described.
Discussion (0). Continue with ORCID to comment.