Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BioClinical ModernBERT achieves state-of-the-art results on four of five clinical and biomedical benchmarks by continuing to pretrain ModernBERT on 53.5 billion tokens of biomedical and clinical text.

desk verdict Useful model release with a serious evaluation hygiene problem: at least three of the five benchmarks overlap with the pretraining corpus, so the SOTA claim is not yet established. read the letter →

arxiv 2506.10896 v1 pith:LB5BVT4K submitted 2025-06-12 cs.CL cs.AI

Thomas Sounack , Joshua Davis , Brigitte Durieux , Antoine Chaffin , Tom J. Pollard , Eric Lehman , Alistair E. W. Johnson , Matthew McDermott
show 2 more authors
Tristan Naumann Charlotta Lindvall
This is my paper
classification cs.CLcs.AI
keywords clinicalNLPbiomedicalencoder-onlytransformercontinuedpretraininglong-contextmodelingModernBERTnamedentityrecognitionde-identification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BioClinical ModernBERT is a domain-adapted encoder built by continuing to pretrain ModernBERT, a recently released general-purpose encoder, on 53.5 billion tokens of PubMed, PMC, and clinical notes drawn from 20 datasets. The paper claims that this two-phase training recipe yields a long-context model, supporting inputs up to 8,192 tokens, that outperforms existing clinical and biomedical encoders on four of five tested benchmarks, covering chemical-protein interaction classification, phenotype detection, social-history extraction, and de-identification. The authors argue that using a diverse multi-institution clinical corpus rather than MIMIC alone improves generalization, and that training on texts with realistic surrogate identifiers instead of PHI masking tokens improves de-identification performance. A careful reader would care because clinical NLP still depends heavily on encoders for classification and entity extraction, and this is a direct upgrade path for models that need to process entire clinical notes in one pass.

What carries the argument

The load-bearing mechanism is the ModernBERT architecture's training schedule and efficiency features, reused for domain adaptation. Warmup-Stable-Decay is a three-stage learning-rate schedule that lets the authors resume from the pre-decay checkpoint without a cold restart; the alternating attention pattern, with two-thirds of layers using local sliding windows and one-third using global attention, keeps computation feasible at 8,192-token inputs; and unpadding keeps inference speed nearly constant across variable-length sequences. On top of that, the two-phase recipe mixes biomedical and clinical text in Phase 1 to prevent catastrophic forgetting, then masks at 15% during Phase 2 clinical specialization, with learning-rate schedules tuned separately for the base and large variants.

What would settle it

Retrain or fine-tune with evaluation notes held out: remove every MIMIC-III note used in the Phenotype split and every MTSamples note used in the Social History split from the pretraining corpus, then rerun the same fine-tuning protocol; if the F1 gains on those two tasks drop toward the ModernBERT baseline, the reported state-of-the-art results reflect memorization of training texts rather than improved clinical understanding.

Watch

Extended reading notes

Core claim

The central claim is that a two-phase continued-pretraining strategy, resuming ModernBERT from its pre-decay Warmup-Stable-Decay checkpoint first on a joint biomedical and clinical mixture and then on the clinical subset alone, produces an encoder with state-of-the-art downstream performance without sacrificing ModernBERT's long-context or efficiency advantages. In the paper's evaluation, the large model reaches 90.8% F1 on ChemProt and 60.8% on Phenotype, the base model reaches 58.5% on Social History, and the large model reaches 83.8% on DEID, with both models at 95.1% on COS. The paper also reports that clinical specialization barely moves biomedical performance on ChemProt, indicating that joint pretraining prevents catastrophic forgetting.

Load-bearing premise

The evaluation is not inflated by pretraining overlap: the Phenotype test notes from MIMIC-III and the Social History notes from MTSamples also appear in the pretraining corpus, and the paper does not test whether excluding them changes the reported scores.

Editorial extensions

If this is right

  • Clinical NLP pipelines can process entire notes, up to 8,192 tokens, in a single pass, avoiding the fragmentation that short-context encoders like BERT require.
  • The model is the first clinical encoder to combine long-context support with ModernBERT-level inference throughput, staying near 71 to 76 thousand tokens per second across a wide range of input lengths and distributions.
  • De-identification performance improves when pretraining includes realistic surrogate identifiers rather than generic PHI masking tokens, a result the paper attributes to the diversity of its clinical corpus.
  • The two-phase training recipe preserves biomedical knowledge: after clinical specialization, ChemProt performance drops only 0.3 points for the base model and rises slightly for the large model.
  • Releasing base and large versions alongside training checkpoints gives practitioners a concrete speed and accuracy trade-off for downstream fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: ablate CheXpert Plus and the other surrogate-PHI datasets from Phase 2 and remeasure DEID; if the gain disappears, the de-identification improvement is caused by surrogate identifiers rather than corpus scale or diversity.
  • If the contamination concern is controlled, the same resume-from-pre-decay WSD recipe could be applied to other specialized domains, such as legal, financial, or scientific text, where a strong general encoder already exists and a curated in-domain corpus is available.
  • The paper's own numbers suggest a scale hypothesis: the largest gains over prior encoders appear on Phenotype, the only task whose notes average over 3,000 tokens, implying that long-context inputs, not just domain adaptation, drive part of the improvement; a controlled comparison with a 512-token prefix of the same notes would isolate this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript reports BioClinical ModernBERT, base (150M) and large (396M) encoders obtained by continued pretraining of ModernBERT on a 53.5B-token biomedical and clinical corpus assembled from PubMed, PMC, and 20 clinical datasets. Training follows a two-phase WSD schedule: Phase 1 jointly trains on biomedical and clinical data (160.5B tokens), and Phase 2 refines on the clinical datasets alone (three epochs). The authors evaluate the models on five tasks (ChemProt, Phenotype, COS, Social History, DEID) by fine-tuning with five seeds and reporting median F1, and they measure inference throughput on synthetic datasets. They claim state-of-the-art performance on four of the five tasks, with efficiency inherited from ModernBERT, and they release models, checkpoints, and code. The abstract states that the model outperforms existing encoders on four downstream tasks, although the evaluation section describes five.

Significance. If the reported results were established on clean held-out benchmarks, the contribution would be valuable: a large-scale, long-context clinical encoder with released checkpoints, a principled WSD-based continual-pretraining recipe, and an unusually broad clinical corpus. The speed measurements confirm that the architecture inherits ModernBERT's unpadding and sliding-window efficiency, and the release of code, models, and training checkpoints is a concrete reproducibility strength. However, the state-of-the-art claim is currently undercut by pretraining/evaluation overlap for two of the tasks, and the absence of variance or significance reporting makes several reported gains difficult to distinguish from noise. The core modeling contribution is plausible, but the evidence as presented does not yet support the claimed state-of-the-art status across the evaluated benchmark set.

major comments (3)
  1. [§3.1.2 (Table 1) and §3.2.2] The pretraining corpus listed in Table 1 includes MTSamples and MIMIC-III, while Section 3.2.2 states that the Social History task uses 364 MTSamples notes and the Phenotype task uses MIMIC-III notes. The exact Social History test documents were almost certainly seen during MLM pretraining, and the Phenotype test notes overlap with a corpus seen by the proposed model but not by ModernBERT-base. No held-out control is provided, so the reported gains on Social History (58.5 vs. 56.8) and Phenotype (58.1 vs. 48.4 against ModernBERT-base) cannot be attributed to generalizable clinical adaptation. The 'state-of-the-art' claim in the abstract and Section 5 is not established for these tasks without a re-evaluation on text excluded from pretraining.
  2. [§4.1 (Table 2)] Table 2 reports median F1 over five seeds without standard deviations, confidence intervals, or significance tests. Several headline differences are small: ChemProt base is 89.9 vs. 89.5 for BioBERT, Social History large is 57.1 vs. 56.8 for Clinical-Longformer, and DEID large is 83.8 vs. 82.1 for Clinical-Longformer. Without variance estimates or paired tests, these differences are not demonstrably significant, and the paper's claim of outperforming existing encoders rests on numerically marginal gaps for several tasks.
  3. [§3.1.2, Table 1 and §4.1] The evaluation of Phenotype and Social History is further complicated by the fact that MIMIC-III and MTSamples together account for only a small fraction of the pretraining corpus but constitute the exact source of the test text. The paper neither states this overlap explicitly nor provides a control, such as evaluating on notes from a held-out institution or on a split that was excluded from pretraining. This is a load-bearing issue for the central state-of-the-art claim, and it should be addressed directly rather than implicitly assumed away.
minor comments (5)
  1. [Abstract and §4.1] The abstract says the model outperforms existing encoders on 'four downstream tasks,' but Table 2 and Section 3.2.2 describe five tasks; on COS the proposed model attains 95.1, which does not exceed the 95.2 of Clinical-Longformer, so the number and the set of claimed wins should be stated consistently.
  2. [§5, Limitations] The sentence 'Finally, With respect to benchmarking datasets' has an incorrect capitalization of 'With'; it should read 'with.'
  3. [§3.1.3] The phrase '1−sqrt decay' is used without defining the precise schedule; please give the formula or a citation so that the training procedure is reproducible.
  4. [Table 4] The cell entry 'BioClinical ModernBERT (ours)20 clinical datasets2.8' is missing spacing and is hard to read; the same issue appears in other cells where source and token counts are concatenated.
  5. [§4.2, Table 3] Table 3 shows that BioClinical ModernBERT has exactly the same throughput as ModernBERT in all configurations; this is expected since the architecture is unchanged, but the discussion should state explicitly that the efficiency results are inherited rather than contributed by the clinical adaptation.

Circularity Check

2 steps flagged · score 6.0 of 10

Social History 'state-of-the-art' is not a held-out result: MTSamples is in the pretraining corpus, so the reported 58.5 F1 partly measures exposure to the test text; the Phenotype gain over ModernBERT-base is likewise confounded by MIMIC-III pretraining.

  1. fitted input called prediction [Section 3.2.2 (Social History task definition) and Table 1 (MTSamples row); test split in Appendix C]
    "Table 1 lists "MTSamples (MTSamples, 2018) Not Reported Clinical Notes Internal Medicine 2,358 1.7" in the pretraining corpus; Section 3.2.2: "Social History or Lifestyle and Environmental Factors in Social History Sections (Yetisgen and Vanderwende, 2017) is a corpus of 364 clinical notes' social history sections from MTSamples.""

    The Social History test documents are MTSamples notes, and Table 1 places the entire MTSamples collection (2,358 samples; 1.7M tokens) inside BioClinical ModernBERT's MLM pretraining corpus. The benchmark is "a corpus of 364 clinical notes' social history sections from MTSamples" (Section 3.2.2) with a 55-note test split (Appendix C). The paper reports no exclusion of benchmark text from pretraining and its stated policy is to "not seek to exclude overlapping samples" (Section 3.1.2), so the test text was almost certainly an MLM training input. None of the baselines (best: Clinical-Longformer, 56.8) trained on MTSamples.

  2. other [Section 3.2.2 (Phenotype task) and Table 1 (MIMIC-III row); Table 2 Phenotype column]
    "Table 1 lists "MIMIC-III (Johnson et al., 2016) US Clinical Notes Internal Medicine 2,021,411 1,047.7" in the pretraining corpus; Section 3.2.2: "Phenotype (Moseley et al., 2020) provides annotations for 2,270 patient notes of MIMIC-III"; Section 3.1.2: "Finally, while MIMIC-III and MIMIC-IV have a small degree of overlap, we do not anticipate this to adversely affect pretraining. We therefore do not expect the training to be affected by this redundancy, and do not seek to exclude overlapping samples.""

    Section 3.2.1 states the ModernBERT comparison is included "to demonstrate the added value of the continued pre-training," and Table 2 shows a Phenotype jump from 48.4 (ModernBERT-base) to 58.1 (ours). But Phenotype is annotated on 2,270 MIMIC-III notes, and MIMIC-III is the largest pretraining component (1,047.7M tokens, Table 1); the paper's stated policy is not to exclude overlapping samples (Section 3.1.2). ModernBERT-base never trained on MIMIC-III, so this gain conflates domain adaptation with direct exposure to the benchmark text, making the "added value" comparison not a clean test.

full rationale

The paper's central claim, outperforming existing clinical and biomedical encoders (Section 5), rests on Table 2. On Social History, the derivation chain reduces by construction: the pretraining corpus (Table 1) contains MTSamples (2,358 samples, 1.7M tokens), and the benchmark is "364 clinical notes' social history sections from MTSamples" (Section 3.2.2). The paper reports no decontamination and states that it does not seek to exclude overlapping samples (Section 3.1.2), so the test text was an MLM training input while no baseline saw MTSamples; the 58.5 versus 56.8 F1 "state-of-the-art" is therefore partly memorization, not generalization. The Phenotype claim is partly confounded: MIMIC-III appears in the corpus, so the 58.1 versus 48.4 gain over ModernBERT-base conflates continued pretraining with benchmark-text exposure; comparisons to Clinical BERT, Clinical-Longformer, and Clinical-BigBird, which also trained on MIMIC-III, remain informative. COS (UW Harborview notes) and DEID (MIMIC-II, absent from Table 1) are clean external benchmarks, and ChemProt is PubMed text shared by all baselines, so it does not differentially favor the authors. The paper's own Limitations section concedes that "MIMIC-III and MIMIC-IV still constituted over 95% of the clinical training corpus" yet never discloses that two of the five evaluation sets derive from that corpus, leaving the state-of-the-art claim unverified against held-out clinical notes. Self-citations to Warner et al. (ModernBERT) are not load-bearing: ModernBERT is a released, externally verifiable artifact, and the inference-speed results in Table 3 simply inherit its architecture. Overall, this is partial circularity: one benchmark prediction reduces to training input by construction, and one is confounded; the remaining tasks provide independent support, so the score is 6 rather than higher.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the architectural assumptions of ModernBERT, the representativeness of the selected corpora, and the cleanliness of the evaluation protocol. The most fragile input is the unstated assumption that pretraining on the same text later used for evaluation does not inflate results; this is not controlled for anywhere in the paper.

free parameters (4)
  • Phase 2 MLM masking probability = 15%
    Chosen after the authors' own experiments reported improved downstream performance relative to 30% (Section 3.1.3).
  • Fine-tuning learning rates = per-task values in Table 5
    Selected by grid search for each downstream task and model size (Section 3.2.2).
  • Pretraining dataset inclusion (MIMIC-III and MTSamples) = MIMIC-III 1,047.7M tokens; MTSamples 1.7M tokens
    Dataset selection decision that directly overlaps with Phenotype and Social History evaluation sets (Section 3.1.2, Table 1).
  • Number of epochs = Phase 1: 3 epochs over combined corpus (160.5B tokens implied); Phase 2: 3 epochs over clinical data
    Hand-set; Phase 2 three epochs stated in Section 3.1.3.
assumptions (5)
  • standard math Transformer architecture and masked language modeling objective provide useful representations for clinical NLP
    Assumed from ModernBERT (Warner et al., 2024) and prior pretraining literature; not re-derived in this paper.
  • domain assumption Continued pretraining on domain text improves downstream task performance
    Core hypothesis of the paper; supported only by a few downstream tasks, not proven generally.
  • domain assumption The five evaluation tasks are representative of clinical NLP
    The paper selects ChemProt, Phenotype, COS, Social History, and DEID without justifying coverage of the broader clinical task space.
  • domain assumption MIMIC-III/IV text is representative of clinical notes and suitable for pretraining
    The paper relies on MIMIC for over 95% of the clinical corpus and acknowledges this limitation in Section 5.
  • ad hoc to paper Including the evaluation text in the pretraining corpus does not inflate downstream scores
    This assumption is required for the SOTA claims on Phenotype and Social History, yet the paper neither states nor tests it (Section 3.1.2 vs Section 3.2.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP." pith.science (2026). https://pith.science/paper/LB5BVT4K

@misc{pith2026250610896,
  author       = {Pith},
  title        = {Pith review of: BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LB5BVT4K}},
  note         = {Machine review of arXiv:2506.10896}
}
read the original abstract

Encoder-based transformer models are central to biomedical and clinical Natural Language Processing (NLP), as their bidirectional self-attention makes them well-suited for efficiently extracting structured information from unstructured text through discriminative tasks. However, encoders have seen slower development compared to decoder models, leading to limited domain adaptation in biomedical and clinical settings. We introduce BioClinical ModernBERT, a domain-adapted encoder that builds on the recent ModernBERT release, incorporating long-context processing and substantial improvements in speed and performance for biomedical and clinical NLP. BioClinical ModernBERT is developed through continued pretraining on the largest biomedical and clinical corpus to date, with over 53.5 billion tokens, and addresses a key limitation of prior clinical encoders by leveraging 20 datasets from diverse institutions, domains, and geographic regions, rather than relying on data from a single source. It outperforms existing biomedical and clinical encoders on four downstream tasks spanning a broad range of use cases. We release both base (150M parameters) and large (396M parameters) versions of BioClinical ModernBERT, along with training checkpoints to support further research.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A pre-training method that aligns ICU time-series windows with LLM-encoded event summaries via a regularised InfoNCE loss improves downstream predictions and cross-dataset transfer.

  2. DLT-Corpus: A Large-Scale Text Collection for the Distributed Ledger Technology Domain

    cs.CL 2026-02 conditional novelty 6.0 of 10

    DLT-Corpus is a 2.98B-token scientific/patent/Twitter corpus for blockchain NLP, plus LedgerBERT (+23% NER vs BERT), a sentiment dataset, and cross-domain innovation-diffusion analyses.

  3. Generalised Medical Phrase Grounding

    cs.CV 2025-11 conditional novelty 6.0 of 10

    MedGrounder grounds radiology sentences to zero, one, or multiple scored image regions, outperforming single-box and grounded-report baselines on multi-box and non-groundable phrases.

Reference graph

Works this paper leans on

70 extracted references · 26 canonical work pages · cited by 3 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hussein Abdel-Jaber , Disha Devassy, Azhar Al Salam, Lamya Hidaytallah, and Malak EL-Amir . 2022. https://doi.org/10.3390/a15020071 A Review of Deep Learning Algorithms and Their Applications in Healthcare . Algorithms, 15(2):71

    work page doi:10.3390/a15020071 2022

  4. [4]

    Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. https://doi.org/10.18653/v1/W19-1909 Publicly Available Clinical BERT Embeddings . In Proceedings of the 2nd Clinical Natural Language Processing Workshop , pages 72--78. Association for Computational Linguistics

    doi:10.18653/v1/w19-1909 2019

  5. [5]

    Asi, Saud Alsahli, Amal Alhashem, Hanan E

    Shams Anazi, Sateesh Maddirevula, Vincenzo Salpietro, Yasmine T. Asi, Saud Alsahli, Amal Alhashem, Hanan E. Shamseldin, Fatema AlZahrani, Nisha Patel, Niema Ibrahim, Firdous M. Abdulwahab, Mais Hashem, Nadia Alhashmi, Fathiya Al Murshedi, Adila Al Kindy, Ahmad Alshaer, Ahmed Rumayyan, Saeed Al Tala, Wesam Kurdi, Abdulaziz Alsaman, Ali Alasmari, Selina Ban...

    work page 2017

  6. [6]

    Zachary Ankner, Naomi Saphra, Davis Blalock, Jonathan Frankle, and Matthew Leavitt. 2024. Dynamic Masking Rate Schedules for MLM Pretraining . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics ( Volume 2: Short Papers ) , pages 477--487, St. Julian's, Malta. Association for Computational Linguistics

    work page 2024

  7. [7]

    Jordan Ash and Ryan P Adams. 2020. On Warm-Starting Neural Network Training . In Advances in Neural Information Processing Systems , volume 33, pages 3884--3894. Curran Associates, Inc

    work page 2020

  8. [8]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. https://doi.org/10.48550/arXiv.2004.05150 Longformer: The Long-Document Transformer

    arXiv doi:10.48550/arxiv.2004.05150 2020

Show all 70 references
  1. [9]

    Auffermann, Jessica Chan, Phuong-Anh T

    Ricardo Bigolin Lanfredi, Mingyuan Zhang, William F. Auffermann, Jessica Chan, Phuong-Anh T. Duong, Vivek Srikumar, Trafton Drew, Joyce D. Schroeder, and Tolga Tasdizen. 2022. https://doi.org/10.1038/s41597-022-01441-z REFLACX , a dataset of reports and eye-tracking data for l...

    2022 doi

  2. [10]

    Langlotz

    Pierre Chambon, Jean-Benoit Delbrouck, Thomas Sounack, Shih-Cheng Huang, Zhihong Chen, Maya Varma, Steven QH Truong, Chu The Chuong, and Curtis P. Langlotz. 2024. https://doi.org/10.48550/arXiv.2405.19538 CheXpert Plus : Augmenting a Large Chest X-ray Dataset with Text Radiolo...

    2024 doi arXiv

  3. [11]

    Chapman, Dieter Hillert, Sumithra Velupillai, Maria Kvist, Maria Skeppstedt, Brian E

    Wendy W. Chapman, Dieter Hillert, Sumithra Velupillai, Maria Kvist, Maria Skeppstedt, Brian E. Chapman, Mike Conway, Melissa Tharp, Danielle L. Mowery, and Louise Deleger. 2013. https://doi.org/10.3233/978-1-61499-289-9-677 Extending the NegEx Lexicon for Multiple Languages . ...

    2013 doi

  4. [12]

    Ha Na Cho, Tae Joon Jun, Young-Hak Kim, Heejun Kang, Imjin Ahn, Hansle Gwon, Yunha Kim, Jiahn Seo, Heejung Choi, Minkyoung Kim, Jiye Han, Gaeun Kee, Seohyun Park, and Soyoung Ko. 2024. https://doi.org/10.2196/49724 Task- Specific Transformer-Based Language Models in Health Car...

    2024 doi

  5. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Associati...

    2019 doi

  6. [14]

    Faiha Fareez, Tishya Parikh, Christopher Wavell, Saba Shahab, Meghan Chevalier, Scott Good, Isabella De Blasi, Rafik Rhouma, Christopher McMahon, Jean-Paul Lam, Thomas Lo, and Christopher W. Smith. 2022. https://doi.org/10.1038/s41597-022-01423-1 A dataset of simulated patient...

    2022 doi

  7. [15]

    Jonas Geiping and Tom Goldstein. 2022. https://doi.org/10.48550/arXiv.2212.14034 Cramming: Training a Language Model on a Single GPU in One Day

    2022 doi arXiv

  8. [16]

    Saibo Geng, Hudson Cooper, Micha Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, and Harsha Nori. 2025. https://doi.org/10.48550/arXiv.2501.10868 JSONSchemaBench : A Rigorous Benchmark of Structured Outputs for Language Models

    2025 doi arXiv

  9. [17]

    Goldberger, Luis A

    Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Planem Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and H. Eugene Stanley. 2000. https://doi.org/10.1161/01.cir.101.23.e215 PhysioBank , PhysioToolkit , and PhysioNet : Co...

    doi

  10. [18]

    Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo. 2012. https://doi.org/10.1016/j.jbi.2012.04.008 Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical ca...

    2012 doi

  11. [19]

    Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. https://doi.org/10.48550/arXiv.2004.10964 Don't Stop Pretraining : Adapt Language Models to Domains and Tasks

    2020 doi arXiv

  12. [20]

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai L...

    2024 doi arXiv

  13. [21]

    Lungren, Curtis P

    Shih-Cheng Huang, Zepeng Huo, Ethan Steinberg, Chia-Chun Chiang, Matthew P. Lungren, Curtis P. Langlotz, Serena Yeung, Nigam H. Shah, and Jason A. Fries. 2023. https://doi.org/10.48550/arXiv.2311.10798 INSPECT : A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis

    2023 doi arXiv

  14. [22]

    Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. 2024. http://arxiv.org/abs/2405.18392 Scaling laws and compute-optimal training beyond fixed training durations

    2024 arXiv

  15. [23]

    Richter, Quentin Anthony, Timoth \'e e Lesort, Eugene Belilovsky, and Irina Rish

    Adam Ibrahim, Benjamin Th \'e rien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timoth \'e e Lesort, Eugene Belilovsky, and Irina Rish. 2024. https://doi.org/10.48550/arXiv.2403.08763 Simple and Scalable Strategies to Continually Pre-train Large Language Models

    2024 doi arXiv

  16. [24]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. https://doi.org/10.48550/arXiv.2009.13081 What Disease does this Patient Have ? A Large-scale Open Domain Question Answering Dataset from Medical Exams

    2020 doi arXiv

  17. [25]

    Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. 2023. https://doi.org/10.1038/s41597-022-01899-x MIMIC-IV , a freely accessib...

    2023 doi

  18. [26]

    Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. https://doi.org/10.1038/sdata.2016.35 MIMIC-III , a freely accessible critical care database . 3(1):160035

    2016 doi

  19. [27]

    Wu, Arjun Sharma, Matthew Tong, Shafiq Abedin, David Beymer, Vandana Mukherjee, Elizabeth A

    Alexandros Karargyris, Satyananda Kashyap, Ismini Lourentzou, Joy T. Wu, Arjun Sharma, Matthew Tong, Shafiq Abedin, David Beymer, Vandana Mukherjee, Elizabeth A. Krupinski, and Mehdi Moradi. 2021. https://doi.org/10.1038/s41597-021-00863-5 Creation and validation of a chest X-...

    2021 doi

  20. [28]

    Chulho Kim, Vivienne Zhu, Jihad Obeid, and Leslie Lenert. 2019. https://doi.org/10.1371/journal.pone.0212778 Natural language processing and machine learning algorithm to identify brain MRI reports with acute ischemic stroke . 14(2):e0212778

    2019 doi

  21. [29]

    Wade, Linda Wagner, Lucy Lu Wang, Chris Wilhelm, Caroline Wu, Jiangjiang Yang, Angele Zamarron, Madeleine Van Zuylen, and Daniel S

    Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, Da...

    2025 doi arXiv

  22. [30]

    Prescott Klassen, Fei Xia, Lucy Vanderwende, and Meliha Yetisgen. 2014. https://aclanthology.org/L14-1334/ Annotating Clinical Events in Text Snippets for Phenotype Detection . In Proceedings of the Ninth International Conference on Language Resources and Evaluation ( LREC '14...

    2014

  23. [31]

    Alex Papadopoulos Korfiatis, Francesco Moramarco, Radmila Sarac, and Aleksandar Savkov. 2022. https://doi.org/10.48550/arXiv.2204.00333 PriMock57 : A Dataset Of Primary Care Mock Consultations

    2022 doi arXiv

  24. [32]

    Martin Krallinger, Obdulia Rabal, and Anália Lourenço. 2017. https://biocreative.bioinformatics.udel.edu/tasks/biocreative-vi/track-5/ Overview of the biocreative vi chemical-protein interaction track . Proceedings of the BioCreative VI Workshop,, 141-146

    2017

  25. [33]

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. https://doi.org/10.1093/bioinformatics/btz682 Biobert: a pre-trained biomedical language representation model for biomedical text mining . Bioinformatics, 36(4):1234--1240

    2019 doi

  26. [34]

    Lee, Anthony Wu, and Jeffrey N

    Simon A. Lee, Anthony Wu, and Jeffrey N. Chiang. 2025. https://doi.org/10.48550/arXiv.2504.03964 Clinical ModernBERT : An efficient and long context encoder for biomedical text

    2025 doi arXiv

  27. [35]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks . In Ad...

    2020

  28. [36]

    Wehbe, Faraz S

    Yikuan Li, Ramsey M. Wehbe, Faraz S. Ahmad, Hanyin Wang, and Yuan Luo. 2022. https://doi.org/10.48550/arXiv.2201.11838 Clinical- Longformer and Clinical-BigBird : Transformers for long clinical sequences

    2022 doi arXiv

  29. [37]

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards General Text Embeddings with Multi-stage Contrastive Learning . https://arxiv.org/abs/2308.03281v1

    2023 arXiv

  30. [38]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://doi.org/10.48550/arXiv.1907.11692 RoBERTa : A Robustly Optimized BERT Pretraining Approach

    2019 doi arXiv

  31. [39]

    Ng, and Pranav Rajpurkar

    Cécile Logé, Emily Ross, David Yaw Amoah Dadey, Saahil Jain, Adriel Saporta, Andrew Y. Ng, and Pranav Rajpurkar. 2021. https://doi.org/10.48550/arXiv.2108.01764 Q- Pain : A Question Answering Dataset to Measure Social Bias in Pain Management

    2021 doi arXiv

  32. [40]

    Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2024. https://doi.org/10.1145/3605943 Recent Advances in Natural Language Processing via Large Pre-trained Language Models : A Survey . ACM Com...

    2024 doi

  33. [41]

    Moseley, Joy T

    Edward T. Moseley, Joy T. Wu, Jonathan Welt, John Foote, Patrick D. Tyler, David W. Grant, Eric T. Carlson, Sebastian Gehrmann, Franck Dernoncourt, and Leo Anthony Celi. 2020. https://aclanthology.org/2020.lrec-1.170/ A Corpus for Detecting High-Context Medical Conditions in I...

    2020

  34. [42]

    MTSamples. 2018. https://www.kaggle.com/datasets/tboyle10/medicaltranscriptions Medical Transcriptions

    2018

  35. [43]

    Hiroki Nakayama. 2018. https://github.com/chakki-works/seqeval seqeval : A python framework for sequence labeling evaluation . Software available from https://github.com/chakki-works/seqeval

    2018

  36. [44]

    Douglass, Li-wei H

    Ishna Neamatullah, Margaret M. Douglass, Li-wei H. Lehman, Andrew Reisner, Mauricio Villarroel, William J. Long, Peter Szolovits, George B. Moody, Roger G. Mark, and Gari D. Clifford. 2008. https://doi.org/10.1186/1472-6947-8-32 Automated de-identification of free-text medical...

    2008 doi

  37. [45]

    Dan Saattrup Nielsen, Kenneth Enevoldsen, and Peter Schneider-Kamp . 2025. https://doi.org/10.48550/arXiv.2406.13469 Encoder vs Decoder : Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks

    2025 doi arXiv

  38. [46]

    John David Osborne, Tobias O'Leary, Amy Mudano, James Booth, Giovanna Rosas, Gurusai Sujitha Peramsetty, Anthony Knighton, Jeff Foster, Ken Saag, and Maria Ioana Danila. 2020. https://doi.org/10.13026/96V3-DW72 Gout Emergency Department Chief Complaint Corpora

    2020 doi

  39. [47]

    Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019)

    2019

  40. [48]

    Jacob Portes, Alex Trott, Sam Havens, Daniel King, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, and Jonathan Frankle. 2024. https://doi.org/10.48550/arXiv.2312.17482 MosaicBERT : A Bidirectional Encoder Optimized for Fast Pretraining

    2024 doi arXiv

  41. [49]

    Lance Ramshaw and Mitch Marcus. 1995. Text Chunking using Transformation-Based Learning . In Third Workshop on Very Large Corpora

    1995

  42. [50]

    Alexey Romanov and Chaitanya Shivade. 2018. https://doi.org/10.48550/arXiv.1808.06752 Lessons from Natural Language Inference in the Clinical Domain

    2018 doi arXiv

  43. [51]

    Teng, Joanne S

    Vlada Rozova, Anna Khanina, Jasmine C. Teng, Joanne S. K. Teh, Leon J. Worth, Monica A. Slavin, Karin A. Thursky, and Karin Verspoor. 2023. https://doi.org/10.1016/j.jbi.2023.104293 Detecting evidence of invasive fungal infections in cytology and histopathology reports enriche...

    2023

  44. [52]

    Chiang, and Michelle R

    Adam Rule, Steven Bedrick, Michael F. Chiang, and Michelle R. Hribar. 2021. https://doi.org/10.1001/jamanetworkopen.2021.15334 Length and Redundancy of Outpatient Progress Notes Across a Decade at an Academic Medical Center . JAMA Network Open, 4(7):e2115334

    2021

  45. [53]

    Reisner, Gari Clifford, Li-Wei Lehman, George Moody, Thomas Heldt, Tin H

    Mohammed Saeed, Mauricio Villarroel, Andrew T. Reisner, Gari Clifford, Li-Wei Lehman, George Moody, Thomas Heldt, Tin H. Kyaw, Benjamin Moody, and Roger G. Mark. 2011. https://doi.org/10.1097/CCM.0b013e31820a92c6 Multiparameter Intelligent Monitoring in Intensive Care II ( MIM...

    2011 doi

  46. [54]

    Sina Shool, Sara Adimi, Reza Saboori Amleshi, Ehsan Bitaraf, Reza Golpira, and Mahmood Tara. 2025. https://doi.org/10.1186/s12911-025-02954-4 A systematic review of large language model ( LLM ) evaluations in clinical medicine . BMC Medical Informatics and Decision Making, 25(1):117

    2025 doi

  47. [55]

    Amber Stubbs, Christopher Kotfila, and \"O zlem Uzuner. 2015. https://doi.org/10.1016/j.jbi.2015.06.007 Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/ UTHealth shared task Track 1 . Journal of Biomedical Informatics, 58 ...

    2015 doi

  48. [56]

    Amber Stubbs and \"O zlem Uzuner. 2015. https://doi.org/10.1016/j.jbi.2015.07.020 Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/ UTHealth corpus . Journal of Biomedical Informatics, 58 Suppl(Suppl):S20--S29

    2015 doi

  49. [57]

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. https://doi.org/10.48550/arXiv.2104.09864 RoFormer : Enhanced Transformer with Rotary Position Embedding

    2023 doi arXiv

  50. [58]

    Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013. https://doi.org/10.1016/j.jbi.2013.07.004 Annotating temporal information in clinical narratives . Journal of Biomedical Informatics, 46 Suppl(0):S5--S12

    2013 doi

  51. [59]

    Kennedy, Divneet Mandair, Brenda Y

    Madhumita Sushil, Vanessa E. Kennedy, Divneet Mandair, Brenda Y. Miao, Travis Zack, and Atul J. Butte. 2024. https://doi.org/10.1056/AIdbp2300110 CORAL : Expert-Curated medical Oncology Reports to Advance Language Model Inference . 1(4)

    2024 doi

  52. [60]

    Ozlem Uzuner, Yuan Luo, and Peter Szolovits. 2007. https://doi.org/10.1197/jamia.M2444 Evaluating the state-of-the-art in automatic de-identification . Journal of the American Medical Informatics Association: JAMIA, 14(5):550--563

    2007 doi

  53. [61]

    \"O zlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. https://doi.org/10.1136/amiajnl-2011-000203 2010 i2b2/ VA challenge on concepts, assertions, and relations in clinical text . Journal of the American Medical Informatics Association : JAMIA, 18(5):552--556

    2011 doi

  54. [62]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding . In Proceedings of the 2018 EMNLP Workshop BlackboxNLP : Ana...

    2018 doi

  55. [63]

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. http://arxiv.org/abs/2412.13663 Smarter, better, fa...

    2024 arXiv

  56. [64]

    Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. 2024. http://arxiv.org/abs/2410.05192 Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective

    2024 arXiv

  57. [65]

    Jiageng Wu, Xiaocong Liu, Minghui Li, Wanxin Li, Zichang Su, Shixu Lin, Lucas Garay, Zhiyun Zhang, Yujie Zhang, Qingcheng Zeng, Jie Shen, Changzheng Yuan, and Jie Yang. 2024. https://doi.org/10.1056/AIra2400012 Clinical text datasets for medical artificial intelligence and lar...

    2024 doi

  58. [66]

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. https://doi.org/10.48550/arXiv.2309.07597 C- Pack : Packed Resources For General Chinese Embeddings

    2024 doi arXiv

  59. [67]

    Meliha Yetisgen and Lucy Vanderwende. 2017. https://doi.org/10.1007/978-3-319-59758-4_18 Automatic Identification of Substance Abuse from Social History in Clinical Text . In Artificial Intelligence in Medicine , volume 10259, pages 171--181. Springer International Publishing

    2017 doi

  60. [68]

    Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. https://doi.org/10.1038/s41597-023-02487-3 Aci-bench: A Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation . 10(1):586

    2023 doi

  61. [69]

    Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2021. https://doi.org/10.48550/arXiv.2007.14062 Big Bird : Transformers for Longer Sequences

    2021 doi arXiv

  62. [70]

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2022. https://doi.org/10.1109/CVPR52688.2022.01179 Scaling Vision Transformers . In 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) , pages 1204--1213

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.