REVIEW 2 cited by
MedGPTEval: A Dataset and Benchmark to Evaluate Responses of Large Language Models in Medicine
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
METHODS: First, a set of evaluation criteria is designed based on a comprehensive literature review. Second, existing candidate criteria are optimized for using a Delphi method by five experts in medicine and engineering. Third, three clinical experts design a set of medical datasets to interact with LLMs. Finally, benchmarking experiments are conducted on the datasets. The responses generated by chatbots based on LLMs are recorded for blind evaluations by five licensed medical experts. RESULTS: The obtained evaluation criteria cover medical professional capabilities, social comprehensive capabilities, contextual capabilities, and computational robustness, with sixteen detailed indicators. The medical datasets include twenty-seven medical dialogues and seven case reports in Chinese. Three chatbots are evaluated, ChatGPT by OpenAI, ERNIE Bot by Baidu Inc., and Doctor PuJiang (Dr. PJ) by Shanghai Artificial Intelligence Laboratory. Experimental results show that Dr. PJ outperforms ChatGPT and ERNIE Bot in both multiple-turn medical dialogue and case report scenarios.
Forward citations
Cited by 2 Pith papers
-
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
LLMEval-Med is a physician-validated Chinese medical QA benchmark with 2,996 open and closed questions; tested LLMs score below 70% usability, weakest at medical text generation.
-
ACE-$M^3$: Automatic Capability Evaluator for Multimodal Medical Models
A branch-merge and reward-token DPO-based multimodal evaluator that surpasses GPT-4-Turbo on image-text medical QA, but relies on LLM-generated labels.
Discussion (0). Continue with ORCID to comment.