Pith. sign in

REVIEW 2 cited by

VeriFact: Verifying Facts in LLM-Generated Clinical Text with Electronic Health Records

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.16672 v1 pith:UG7I4OOP submitted 2025-01-28 cs.AI cs.CLcs.IRcs.LO

Philip Chung , Akshay Swaminathan , Alex J. Goodell , Yeasul Kim , S. Momsen Reincke , Lichy Han , Ben Deverett , Mohammad Amin Sadeghi
show 21 more authors
Abdel-Badih Ariss Marc Ghanem David Seong Andrew A. Lee Caitlin E. Coombes Brad Bradshaw Mahir A. Sufian Hyo Jung Hong Teresa P. Nguyen Mohammad R. Rasouli Komal Kamra Mark A. Burbridge James C. McAvoy Roya Saffary Stephen P. Ma Dev Dash James Xie Ellen Y. Wang Clifford A. Schmiesing Nigam Shah Nima Aghaeepour
This is my paper
classification cs.AIcs.CLcs.IRcs.LO
keywords verifacttextclinicalpatientagreementaverageclinicianelectronic
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Methods to ensure factual accuracy of text generated by large language models (LLM) in clinical medicine are lacking. VeriFact is an artificial intelligence system that combines retrieval-augmented generation and LLM-as-a-Judge to verify whether LLM-generated text is factually supported by a patient's medical history based on their electronic health record (EHR). To evaluate this system, we introduce VeriFact-BHC, a new dataset that decomposes Brief Hospital Course narratives from discharge summaries into a set of simple statements with clinician annotations for whether each statement is supported by the patient's EHR clinical notes. Whereas highest agreement between clinicians was 88.5%, VeriFact achieves up to 92.7% agreement when compared to a denoised and adjudicated average human clinican ground truth, suggesting that VeriFact exceeds the average clinician's ability to fact-check text against a patient's medical record. VeriFact may accelerate the development of LLM-based EHR applications by removing current evaluation bottlenecks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MedFact: Benchmarking the Fact-Checking Capabilities of Large Language Models on Chinese Medical Texts

    cs.CL 2025-09 conditional novelty 6.0 of 10

    MedFact, a new Chinese medical fact-checking benchmark, shows LLMs often detect errors but localize them poorly, and more reasoning time triggers over-criticism.

  2. MedReadCtrl: Personalizing medical text generation with readability-controlled instruction learning

    cs.CL 2025-07 conditional novelty 4.0 of 10

    MedReadCtrl instruction-tunes LLaMA3 to control readability at 12 grade levels, reporting lower readability errors than GPT-4 and higher content scores on unseen clinical simplification.

Pith tools