REVIEW 10 cited by
Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Transformers-based models, such as BERT, have dramatically improved the performance for various natural language processing tasks. The clinical knowledge enriched model, namely ClinicalBERT, also achieved state-of-the-art results when performed on clinical named entity recognition and natural language inference tasks. One of the core limitations of these transformers is the substantial memory consumption due to their full self-attention mechanism. To overcome this, long sequence transformer models, e.g. Longformer and BigBird, were proposed with the idea of sparse attention mechanism to reduce the memory usage from quadratic to the sequence length to a linear scale. These models extended the maximum input sequence length from 512 to 4096, which enhanced the ability of modeling long-term dependency and consequently achieved optimal results in a variety of tasks. Inspired by the success of these long sequence transformer models, we introduce two domain enriched language models, namely Clinical-Longformer and Clinical-BigBird, which are pre-trained from large-scale clinical corpora. We evaluate both pre-trained models using 10 baseline tasks including named entity recognition, question answering, and document classification tasks. The results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT as well as other short-sequence transformers in all downstream tasks. We have made our source code available at [https://github.com/luoyuanlab/Clinical-Longformer] the pre-trained models available for public download at: [https://huggingface.co/yikuan8/Clinical-Longformer].
Forward citations
Cited by 10 Pith papers
-
LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models
A pre-training method that aligns ICU time-series windows with LLM-encoded event summaries via a regularised InfoNCE loss improves downstream predictions and cross-dataset transfer.
-
ImmunoFOMO: Are Language Models missing what oncologists see?
Small domain-specific language models identify fine-grained immunotherapy hallmarks in breast cancer abstracts more accurately than large language models do, while large models handle coarser categories better.
-
BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP
A continued-pretrained ModernBERT encoder for biomedical and clinical text claims SOTA on several clinical NLP tasks, with caveats about data overlap between pretraining and evaluation.
-
Temporal Entailment Pretraining for Clinical Language Models over EHR Data
A temporal entailment pretraining objective, which classifies later EHR segments as entailed, contradictory, or neutral relative to earlier segments, is claimed to improve clinical language models on temporal QA, earl...
-
NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text
Contrastive pre-training that aligns medical note text with ICD-10 code sequences outperforms prior models on MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full, with the clearest gains on rare codes.
-
Multimodal Semantic-Aware Contrastive Learning For False Negative Mitigation in 3D Medical Imaging
MseaCL, a semantic-aware contrastive pretraining method for 3D brain MRI and reports, reports a 0.226 external AUC gain over instance-based contrastive learning for pediatric brain tumor molecular classification.
-
Can human clinical rationales improve the performance and explainability of clinical text classification models?
Adding 96,679 human rationale highlights improves cancer-site classification less than adding the same number of full pathology reports, and the explainability gain is small.
-
A Data-Centric Approach to Detecting and Mitigating Demographic Bias in Pediatric Mental Health Text: A Case Study in Anxiety Detection
Fine-tuned clinical language models under-diagnose anxiety in female adolescents, partly because their notes are shorter and worded differently; TF-IDF-based sentence filtering plus gender-neutral substitution reduces...
-
Bridging Electronic Health Records and Clinical Texts: Contrastive Learning for Enhanced Clinical Tasks
A contrastive pretraining step that aligns static EHR tables with discharge notes improves downstream TabNet predictions on MIMIC-IV readmission and critical outcome tasks.
-
Multimodal Prescriptive Deep Learning
A softmax policy network trained on estimated counterfactual rewards reports large estimated clinical improvements on two multimodal datasets.
Discussion (0). Continue with ORCID to comment.