REVIEW 6 cited by
Benchmarking large language models for biomedical natural language processing applications and recommendations
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs, GPT and LLaMA representatives on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here we show that traditional fine-tuning outperforms zero or few shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.
Forward citations
Cited by 6 Pith papers
-
Automatic Evaluation of Healthcare LLMs Beyond Question-Answering
In healthcare LLM evaluation, multiple-choice accuracy and open-ended task scores correlate only weakly, and the paper's proposed Relaxed Perplexity metric aims to improve open-ended factuality scoring but rests on an...
-
Extracting Information in a Low-resource Setting: Case Study on Bioinformatics Workflows
A new 52-article annotated corpus and baseline NER experiments show that bioinformatics workflow entities can be extracted with 70.4 F1 using SciBERT, supporting the feasibility of automated workflow documentation.
-
Development of the user-friendly decision aid Rule-based Evaluation and Support Tool (REST) for optimizing the resources of an information extraction task
REST provides per-entity criteria and visualizations to judge rule feasibility before an extraction pipeline is built, validated only by inter-expert agreement, not by actual extraction performance.
-
TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification
Without any parameter updates, a retrieval-augmented generalist-specialist agent pair with consistency-based verification raises accuracy on 862 hard medical QA questions for GPT-4o, DeepSeek-R1, and Qwen2.5-7B.
-
Can OpenAI o1 Reason Well in Ophthalmology? A 6,990-Question Head-to-Head Evaluation Study
On 6,990 ophthalmology MCQs, OpenAI o1 beat five other LLMs on answer accuracy yet lagged GPT-4o and GPT-4 on text-similarity metrics used to gauge reasoning.
-
Diversity Over Quantity: A Lesson From Few Shot Relation Classification
Holding training set size fixed, increasing the number of relation types improves few-shot relation classification performance on unseen relations, demonstrated in the new REBEL-FS benchmark.
Discussion (0). Continue with ORCID to comment.