Self-Alignment Pretraining for Biomedical Entity Representations
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:LFVQOMEQrecord.jsonopen to challenge →
read the original abstract
Despite the widespread success of self-supervised learning via masked language models (MLM), accurately capturing fine-grained semantic relationships in the biomedical domain remains a challenge. This is of paramount importance for entity-level tasks such as entity linking where the ability to model entity relations (especially synonymy) is pivotal. To address this challenge, we propose SapBERT, a pretraining scheme that self-aligns the representation space of biomedical entities. We design a scalable metric learning framework that can leverage UMLS, a massive collection of biomedical ontologies with 4M+ concepts. In contrast with previous pipeline-based hybrid systems, SapBERT offers an elegant one-model-for-all solution to the problem of medical entity linking (MEL), achieving a new state-of-the-art (SOTA) on six MEL benchmarking datasets. In the scientific domain, we achieve SOTA even without task-specific supervision. With substantial improvement over various domain-specific pretrained MLMs such as BioBERT, SciBERTand and PubMedBERT, our pretraining scheme proves to be both effective and robust.
This paper has not been read by Pith yet.
Forward citations
Cited by 8 Pith papers
-
MedGuards: Multi-Agent System for Reliable Medical Error Detection and Correction
MedGuards introduces a multi-agent in-context learning framework for medical error detection and correction plus the KPCS metric, reporting improvements on four multilingual clinical note datasets.
-
MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings
A text-to-FHIR pipeline produces a synthetic dataset with 82.5% valid bundles and demonstrates reduced LLM diagnostic accuracy on structured EHR data versus plain text.
-
MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings
A staged LLM pipeline with terminology validation produces FHIR bundles from text cases at 82.5% validity rate, revealing reduced LLM diagnostic accuracy on structured EHR-like inputs versus plain text.
-
MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings
A pipeline generates 1,732 valid FHIR bundles from clinician cases, revealing lower LLM diagnostic accuracy on structured inputs than on plain text.
-
Spherical Mixture Integration for Latent Embedding Alignment across Multi-Source Feature Spaces
SMILE aligns latent embeddings across multi-source EHR feature spaces via spherical mixtures of von Mises-Fisher distributions, provides non-asymptotic error bounds, and enables consistent synonym cluster recovery.
-
Spherical Mixture Integration for Latent Embedding Alignment across Multi-Source Feature Spaces
SMILE models synonymy in multi-EHR codes via spherical mixtures of von Mises-Fisher distributions and develops a composite quasi-likelihood estimator with non-asymptotic error bounds and consistent cluster recovery.
-
The Provenance Gap in Clinical AI: Evidence-Traceable Temporal Knowledge Graphs for Rare Disease Reasoning
HEG-TKG grounds LLM clinical reasoning in hierarchical evidence-based temporal knowledge graphs from 4,512 PubMed records, delivering 100% citation verifiability and error detectability where standard RAG and unprompt...
-
MedGuards: Multi-Agent System for Reliable Medical Error Detection and Correction
MedGuards proposes a multi-agent system for medical error detection and correction plus the KPCS metric, reporting gains on four multilingual clinical-note datasets.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.