BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved pith:VQCHQL7Wrecord.jsonopen to challenge →
read the original abstract
Large Language Models (LLMs) have demonstrated remarkable versatility in recent years, offering potential applications across specialized domains such as healthcare and medicine. Despite the availability of various open-source LLMs tailored for health contexts, adapting general-purpose LLMs to the medical domain presents significant challenges. In this paper, we introduce BioMistral, an open-source LLM tailored for the biomedical domain, utilizing Mistral as its foundation model and further pre-trained on PubMed Central. We conduct a comprehensive evaluation of BioMistral on a benchmark comprising 10 established medical question-answering (QA) tasks in English. We also explore lightweight models obtained through quantization and model merging approaches. Our results demonstrate BioMistral's superior performance compared to existing open-source medical models and its competitive edge against proprietary counterparts. Finally, to address the limited availability of data beyond English and to assess the multilingual generalization of medical LLMs, we automatically translated and evaluated this benchmark into 7 other languages. This marks the first large-scale multilingual evaluation of LLMs in the medical domain. Datasets, multilingual evaluation benchmarks, scripts, and all the models obtained during our experiments are freely released.
This paper has not been read by Pith yet.
Forward citations
Cited by 26 Pith papers
-
From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents
A dataset-agnostic framework converts text tool-calling benchmarks to paired audio versions via TTS and noise, showing model-dependent performance with small text-to-voice gaps of 1.8-4.8 points on Confetti and When2Call.
-
Serialisation Strategy Matters: How FHIR Data Format Affects LLM Medication Reconciliation
Clinical narrative format beats raw JSON for LLMs up to 8B parameters on medication reconciliation but raw JSON wins at 70B scale, with omissions as the main error type.
-
Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?
Evaluation of 22 LLMs shows they are more susceptible to spin in medical abstracts than humans but can recognize and mitigate it when prompted.
-
Dep-LLM: Training-Free Depression Diagnosis via Evidence-Guided Structured Multi-factor with Reliable LLM Reasoning
Dep-LLM is a training-free three-stage LLM framework that decomposes clinical interviews into clinical themes, modulates signals by token entropy, and outperforms zero-shot and supervised baselines on DAIC-WOZ and E-D...
-
Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models
Empirical benchmarks show distribution similarity between adaptation and pretraining data increases practical privacy leakage in DP-adapted LLMs at fixed theoretical guarantees, with LoRA providing strongest protectio...
-
From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents
A dataset-agnostic framework converts text tool-calling benchmarks to paired audio evaluations via TTS, speaker variation and noise, then evaluates seven omni-modal models showing model- and task-dependent performance...
-
A Scientific Human-Agent Reproduction Pipeline
SHARP is a human-AI collaboration pipeline for reproducing scientific analyses, demonstrated by recreating a jet classification task from a particle physics paper.
-
Large Language Models Meet Biomedical Knowledge Graphs for Mechanistically Grounded Therapeutic Prioritization
DrugKLM integrates knowledge graphs and LLMs to prioritize mechanistically plausible drug repurposing candidates, outperforming baselines and aligning scores with improved survival signatures in cancer data.
-
Representation geometry shapes task performance in vision-language modeling for CT enterography
Mean pooling and multi-window RGB encoding optimize vision-language performance on CT enterography, with retrieval-augmented generation substantially improving automated report severity accuracy over fine-tuning alone.
-
VeriSim: A Configurable Framework for Evaluating Medical AI Under Realistic Patient Noise
VeriSim adds six literature-derived noise types to patient simulations and shows medical LLMs lose 15-25% diagnostic accuracy under realistic conditions.
-
To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models
Domain-adapted clinical LLMs provide only marginal and unstable gains over general models on English clinical MCQA benchmarks, while new Spanish Marmoka models perform better.
-
CURE-Med: Curriculum-Informed Reinforcement Learning for Multilingual Medical Reasoning
CURE-MED pairs a new 13-language medical reasoning benchmark with curriculum RL to raise logical correctness to 70% and language consistency to 95% at 32B scale while outperforming baselines.
-
Real-World Doctor Agent with Proactive Consultation through Multi-Agent Reinforcement Learning
DoctorAgent-RL trains a Qwen2.5-7B doctor agent via multi-agent RL on the new MTMedDialog dataset to conduct dynamic, question-driven consultations, reaching 70% exact diagnostic match in real-patient trials.
-
HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs
HuatuoGPT-o1 achieves superior medical complex reasoning by using a verifier to curate reasoning trajectories for fine-tuning and then applying RL with verifier-based rewards.
-
Reinforcement learning to improve large language model-based automated code compliance systems
P4IR applies supervised fine-tuning followed by GRPO reinforcement learning to reduce tree edit distance by up to 23.8% and Levenshtein distance by up to 38.6% versus SFT baselines while outperforming several frontier...
-
ChatHealthAI: Aligning Electronic Health Record Representations with Large Language Models for Grounded Clinical Reasoning
ChatHealthAI is a multimodal framework that aligns EHR foundation model representations with LLMs via a task-aware resampler for grounded clinical reasoning on longitudinal patient data while preserving predictive per...
-
MedMIX: Modality-Internal Expert Fusion for Multimodal Medical Diagnosis
MedMIX combines intra-modality expert fusion, learned inter-modality fusion, and training-only large-small collaboration to deliver robust multimodal medical prediction under incomplete modalities across three benchmarks.
-
RadLite: Multi-Task LoRA Fine-Tuning of Small Language Models for CPU-Deployable Radiology AI
LoRA fine-tuning of 3-4B SLMs on 162K multi-task radiology data yields strong performance deployable on consumer CPUs at 4-8 tokens/second.
-
Handling and Interpreting Missing Modalities in Patient Clinical Trajectories via Autoregressive Sequence Modeling
Autoregressive transformer modeling with missingness-aware contrastive pre-training outperforms baselines on MIMIC-IV and eICU benchmarks and mitigates divergent behavior from removed modalities in clinical trajectories.
-
A Scientific Human-Agent Reproduction Pipeline
Autoregressive LLM decoders with a missingness-aware contrastive pre-training objective outperform static baselines on MIMIC-IV/eICU and reveal demographic-bias failure modes under modality ablation.
-
Diamonds in the rough: Transforming SPARCs of imagination into a game concept by leveraging medium sized LLMs
Medium-sized LLMs can supply useful feedback on game concepts in early design stages, as demonstrated by model comparisons and a positive student pilot study.
-
Cross-Domain Feature Expansion for Tabular Medical Data via Knowledge Graphs Injection
MedKGTab integrates data-driven statistical priors with the SPOKE biomedical knowledge graph via dual-attention to expand cross-domain features in tabular medical data and claims to outperform existing models.
-
Why Specialist Models Still Matter: A Heterogeneous Multi-Agent Paradigm for Medical Artificial Intelligence
HetMedAgent is a heterogeneous multi-agent framework that fuses generalist LLMs and specialist models via conflict-aware fusion and uncertainty triggers, outperforming either alone on three clinical tasks.
-
ECG Foundation Models and Medical LLMs for Agentic Cardiovascular Intelligence at the Edge: A Review and Outlook
ECG foundation models for signal interpretation and medical LLMs for reasoning can be integrated into agentic systems for real-time cardiovascular intelligence on edge devices.
-
Development and Preliminary Evaluation of a Domain-Specific Large Language Model for Tuberculosis Care in South Africa
A domain-specific LLM for TB care in South Africa, created by fine-tuning BioMistral-7B with QLoRA and GraphRAG on local guidelines, shows improved contextual alignment over the base model.
-
Enhancing LLMs for Identifying and Prioritizing Important Medical Jargons from Electronic Health Record Notes Utilizing Data Augmentation
Fine-tuning and data augmentation improve LLM performance on medical jargon extraction and prioritization from EHR notes, with augmented open-source models sometimes outperforming closed-source ones on 106 annotated notes.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.