Pith. sign in

REVIEW 3 major objections 5 minor 53 references

Diagnosing our datasets: How does my language model learn clinical information?

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that how well open-weights LLMs understand clinical shorthand tracks how often that shorthand co-occurs with its expansion in pretraining corpora, and that real EHR notes are full of shorthand those corpora rarely contain.

desk verdict Worth a serious referee: the MedLingo dataset is a real contribution, the frequency-accuracy correlation holds up even with raw counts, and the GPT-4o filtering concern is soft but not fatal. read the letter →

arxiv 2505.15024 v2 pith:C6JOTVGL submitted 2025-05-21 cs.CL

Furong Jia , David Sontag , Monica Agrawal This is my paper
classification cs.CL
keywords clinicaljargonLLMpretrainingdataMedLingocorpusfrequencyacronymsmedicalmisinformationEHRnotesopen-sourceLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models answer clinical questions without being trained on electronic health records, and this paper asks where that ability comes from. Its answer is that much of it comes from the frequency of clinical jargon in ordinary web pretraining corpora. The authors build MedLingo, a 100-item benchmark of abbreviation–expansion pairs taken from real MIMIC-IV discharge notes, and measure how often each pair occurs in Dolma, C4, and RedPajama using a GPT-4o-filtered co-occurrence count. On the existing CASI acronym dataset, those estimated counts correlate strongly with model accuracy (Spearman 0.56–0.72 across models and corpora), while the same counts correlate only weakly (0.15) with how often the jargon actually appears in clinical notes—so models fail on shorthand that is routine in hospitals but rare online. Applying the same dataset-side diagnostic to disputed medical claims shows that the share of supportive documents in pretraining corpora predicts, better than raw counts do, whether models will reproduce the claim when asked a presuppositional question.

What carries the argument

The load-bearing object is the estimated co-occurrence frequency $\hat{N}_{\text{final}}(A, E) = \max(\hat{N}_{\text{cooc}}(A, E), \hat{N}_{\text{context}}(A))$, computed over WIMBD's indexes of Dolma, C4, and RedPajama. Raw document counts are converted to estimates by sampling up to 500 documents per pair, asking GPT-4o whether the abbreviation is used in its clinical sense, and scaling the judged fraction to the total count; the context-based variant applies the same filter to all documents containing the abbreviation, and the maximum of the two is the final predictor. This filtered count is what turns page-level co-occurrence into a proxy for learnable clinical knowledge and is the variable that correlates with accuracy. The paper also introduces MedLingo, a 100-example benchmark of abbreviation–expansion pairs extracted from MIMIC-IV notes with a regex and minimal context, so that jargon knowledge can be tested without context leaking the answer.

What would settle it

Take a stratified random sample of the GPT-4o relevance labels from Section 4.1.2 (say, 200 documents spanning rare and common abbreviations) and have two human annotators relabel them; if human–GPT-4o concordance on the rare, ambiguous items is at or near chance instead of the 98% seen for answer grading, the estimated co-occurrence frequencies are not measuring what the paper claims, and the Spearman correlations would need to be recomputed on human-validated counts.

Watch

Extended reading notes

Core claim

The paper's central discovery is that an LLM's clinical jargon ability is largely a function of pretraining corpus composition, measurable through estimated co-occurrence frequencies. For each abbreviation–expansion pair, the authors count documents containing both terms in a corpus, sample up to 500 of those documents, and have GPT-4o judge which ones actually use the abbreviation in its clinical sense; scaling this judged fraction to the total count gives an estimated frequency that correlates with per-pair accuracy at Spearman 0.56–0.72 on CASI. The relationship is monotone: rarer pairs are less likely to be answered correctly, and larger LLaMA models climb the accuracy curve for rare pairs without any change in the data. A second result is the mismatch between training data and clinical reality: estimated frequencies in Dolma correlate only 0.15 with occurrences in MIMIC-IV discharge notes, so common hospital shorthand like 'A VSS' (afebrile, vital signs stable), which appears 10,766 times in MIMIC-IV but only 12 times in Dolma, is missed by all evaluated open-weights models. On the safety side, documents supporting debunked claims are not rare—several claims have support ratios above 50% in Dolma—and the support ratio, not the raw count, predicts whether instruction-tuned models echo a claim after a presuppositional prompt, with supportive responses rising from 10% on direct queries to 31% on 'How does X cause Y?' prompts.

Load-bearing premise

The frequency estimates that drive every correlation depend on GPT-4o correctly deciding whether a sampled web document uses an abbreviation in its clinical sense, and unlike the answer grading, that relevance filter has no reported human validation, so systematic labeling errors on rare ambiguous shorthand would make the counts—and the correlations built on them—artifacts.

Editorial extensions

If this is right

  • A corpus-frequency scan becomes a cheap diagnostic: a practitioner could predict which clinical abbreviations an open model will fail on by estimating filtered co-occurrence counts, without running the model.
  • Models trained only on general web text will keep missing common hospital shorthand ('A VSS', 'HKS', 'MMM') until real EHR-note text is added to pretraining or fine-tuning data; the MeLLaMA comparison shows such adaptation can narrow the gap.
  • Medical LLM benchmarks should include real-note-style jargon tests such as MedLingo rather than synthetic vignettes or licensing exams, because the latter do not expose the distribution shift documented here.
  • Deployment pipelines should filter or reweight documents that support disputed claims, since the supportive-document ratio predicts model behavior better than raw counts, and presuppositional prompts turn a 10% support rate into 31% on the same claims.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same GPT-4o-filtered co-occurrence diagnostic could transfer to other jargon-heavy specialties (law, finance, software), giving a dataset-side predictor of model competence before running any evaluation.
  • The MIMIC-to-web mismatch suggests a testable 'clinical text distribution shift' law: if confirmed on other hospitals and note types, adding even a modest set of authentic ICU notes to fine-tuning should yield outsized gains on real-note benchmarks.
  • A data-cleaning rule the paper does not state follows from its support-ratio result: remove or downweight documents that co-locate known debunked keyword pairs with a high supportive-document ratio, which may reduce presuppositional-prompt failures without sacrificing factual content.
  • One could validate the mechanism directly by constructing MedLingo-style questions split by MIMIC frequency deciles and showing that accuracy on each decile tracks $\hat{N}_{\text{final}}$; a tight fit would let dataset curators budget exactly how much note text to add.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies how open-source LLMs acquire clinical knowledge from large mined pretraining corpora, through two lenses: interpretation of clinical jargon and generation of unsupported medical claims. For jargon, the authors introduce a new benchmark MedLingo, evaluate models on MedLingo and CASI, estimate the frequency of jargon-expansion co-occurrences in RedPajama, Dolma, and C4 using WIMBD and GPT-4o relevance filtering, and correlate these estimated counts with model accuracy. They report Spearman correlations between 0.56 and 0.72 on CASI and claim that filtered counts correlate more strongly than raw counts. They also find a weak, non-significant Spearman correlation of 0.15 (p=0.13) between jargon frequency in MIMIC-IV notes and in Dolma, which they interpret as a mismatch between clinical notes and pretraining data. For disputed claims, they construct 12 keyword pairs, estimate the prevalence of supporting documents in the corpora, manually classify model responses as denial/neutral/support, and analyze the types of online sources where clinical jargon and unsupported claims appear.

Significance. If the central empirical claims hold, the paper provides a valuable and actionable account of how pretraining-data composition shapes clinical NLP behavior: the availability of a new clinical jargon benchmark (MedLingo), the systematic use of WIMBD for terabyte-scale corpus analysis, the release of code and data, and the source-type breakdowns are all useful contributions that fit the CHIL scope. The paper's strongest asset is that the frequency-accuracy association is not solely an artifact of the GPT-4o-based filtering: raw co-occurrence counts also correlate with accuracy (Table 10, rho 0.44-0.66), giving independent support to the broad claim. The paper also makes falsifiable predictions (e.g., models fail on jargon rare in pretraining corpora such as "A VSS") that could be tested on future models. However, the quantitative headline claims—that GPT-4o relevance filtering improves correlations, that there is a mismatch between MIMIC and pretraining frequencies, and that the ratio of supportive documents is a better predictor of claim propagation—rest on steps that currently lack sufficient validation or statistical support, which tempers the significance of the specific numbers.

major comments (3)
  1. [Section 4.1.2, Equations (3) and (4); Tables 9 and 10] The estimated counts bN_final(A,E) rely on GPT-4o relevance filtering with no reported human validation, sample sizes capped at 500, and a max-of-two-estimates rule (Nfinal = max(Ncooc, Ncontext)) that can systematically overestimate counts, especially for noisy context-based estimates. Because GPT-4o also grades model answers (Section 4.1.1, validated on only 50 CASI examples), the reported improvement from raw counts (Table 10, rho 0.44-0.66) to filtered counts (Table 9, rho 0.56-0.72) could partly reflect a shared judge bias rather than a genuine gain from clinical-relevance filtering. The authors should provide human validation of the relevance filter (e.g., stratified by abbreviation rarity), bootstrap confidence intervals for the Spearman correlations, and a sensitivity analysis that recomputes correlations using raw counts or alternative filters; this is necessary to support the Section 7 statement that "our additional post-processing on the raw counts yields higher correlations."
  2. [Section 5.1 and Appendix E] The classification of model responses into denial/neutral/support was performed by a single author with no inter-annotator agreement reported, and the GPT-4o classification of documents as support/against/unknown is similarly unvalidated. The claim that the ratio metric correlates better than the count metric (Appendix E) is based on only 12 keyword pairs; the reported Spearman values (0.28 vs -0.20) lack p-values, confidence intervals, and any acknowledgment that n=12 makes these estimates highly unstable. Please add inter-annotator agreement for the response classification, human validation of the document-claim classification, and appropriate inferential statistics for the small sample, or explicitly frame the ratio-versus-count finding as exploratory.
  3. [Section 4.2.3 and Figure 4] The paper states that the Spearman correlation of 0.15 (p=0.13) between MIMIC-IV counts and Dolma counts "indicat[es] a mismatch" between clinical notes and pretraining data. This is not supported by the significance test: p=0.13 means the data do not provide strong evidence against the null hypothesis of no association, so the finding should be described as "we do not find a significant correlation" rather than as evidence of a mismatch. Because the abstract and introduction present this mismatch as a central motivation, the authors should either report a confidence interval for rho and explicitly acknowledge the limited statistical evidence, or soften the claim accordingly.
minor comments (5)
  1. [Section 4.1.2, footnote 1] The footnote states that RedPajama's index became inaccessible and that various analyses may not include RedPajama; please clarify in which tables and figures RedPajama is excluded (e.g., Table 5 marks two entries with "-"), and add a note where missing.
  2. [Throughout] There are typos and spacing errors: "pretraining copora" (Section 4.2.1), "fluroide" (Appendix E), "Comparsion" (Figure 8 caption), and "F uture W ork" (Section 7 heading).
  3. [Section 4.1.2] The sentence "further define the estimated co-occurrence frequency counts:" appears incomplete and disrupts the flow from Equation (3) to the definition of bN_final; also fix "as . ˆfcontext is often 0" in the CASI paragraph.
  4. [Figures 3, 7, 9, and 10] The Spearman correlation labels in several figures render as "= 0.72" or " = 0.56" without the rho symbol; please ensure the Greek letter appears consistently.
  5. [Section 3.2] The contamination check for CASI reports that a random selection of 10 sentences was searched via WIMBD; given that the dataset contains 5887 examples, this is a very small check and should be reported with the number of acronyms covered or expanded to a more systematic substring search.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the frequency-accuracy analysis is self-contained, and the shared-GPT-4o concern is a measurement-validity caveat, not a definitional reduction.

full rationale

The paper's derivation chain is self-contained. Model accuracy is measured on CASI and MedLingo against human-annotated ground truth, with GPT-4o as judge validated to 98% concordance on 50 CASI examples and with manual adjudication of MedLingo judge disagreements. Corpus frequency is estimated independently via WIMBD raw co-occurrence counts scaled by GPT-4o relevance fractions (fcooc, fcontext), yielding bN_final(A,E)=max(bN_cooc,bN_context); this estimate is a function of corpus samples and relevance judgments only, not of model accuracy. The central correlation claims (Spearman 0.56-0.72 for filtered counts, 0.44-0.66 for raw counts, and 0.15 for MIMIC-IV vs. Dolma) are empirical comparisons between independently measured quantities, not identities, and no fitted parameter is renamed as a prediction. The raw-count correlations provide independent support for the broad frequency-accuracy relationship, so the estimated-count correlations are not the sole evidence. The only mild concern is that GPT-4o is used both to filter relevant documents and to grade answers, which is a potential shared-measurement bias rather than circularity, because the two variables are not equated by construction and no equation in Section 4 defines one in terms of the other. The paper also contains no load-bearing self-citation or imported uniqueness theorem; the sole author self-citation (Agrawal et al., 2022) is motivational. Therefore no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces a new benchmark (MedLingo) and uses GPT-4o as a filter and judge in several places. The main uncharged assumptions are the accuracy of WIMBD counting, the fidelity of RedPajama to LLaMA's true training data, and the validity of GPT-4o relevance filtering.

free parameters (3)
  • close co-occurrence window = 200 characters
    In Section 5.1, documents are counted only if the two claim keywords appear within 200 characters; this hand-chosen threshold directly affects Nsupport.
  • sample size cap for relevance estimation = 500 (co-occurrence), 100 (claims)
    In Sections 4.1.2 and 5.1, frequencies are estimated from samples capped at 500 or 100 documents; sampling error is not propagated into the reported estimates.
  • final frequency estimator = max(co-occurrence, context)
    In Section 4.1.2, bNfinal is defined as the maximum of the co-occurrence and context-based estimates; this hand-chosen combination affects the correlations in Table 9.
assumptions (4)
  • domain assumption WIMBD indexes accurately count document frequencies for the three corpora.
    Section 4.1.2 relies on WIMBD counts as ground truth; index errors would bias the estimated counts.
  • domain assumption RedPajama v1 is a faithful reproduction of LLaMA's pretraining data.
    Section 3.1 maps LLaMA models to RedPajama; if the actual training data differs, the observed correlations may be miscounted.
  • ad hoc to paper GPT-4o's relevance filtering of co-occurrences matches human judgment.
    Section 4.1.2 uses GPT-4o to decide whether an abbreviation's co-occurrence is clinically relevant, but unlike the answer grading (98% human concordance), this filter is not validated against human annotations.
  • domain assumption LLM-as-a-judge grading is a valid measure of clinical jargon accuracy.
    Section 4.1.1 reports 98% concordance with human annotators on a 50-example CASI sample, so this is partially supported; for MedLingo, disagreements were manually adjudicated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diagnosing our datasets: How does my language model learn clinical information?." pith.science (2026). https://pith.science/paper/C6JOTVGL

@misc{pith2026250515024,
  author       = {Pith},
  title        = {Pith review of: Diagnosing our datasets: How does my language model learn clinical information?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C6JOTVGL}},
  note         = {Machine review of arXiv:2505.15024}
}
read the original abstract

Large language models (LLMs) have performed well across various clinical natural language processing tasks, despite not being directly trained on electronic health record (EHR) data. In this work, we examine how popular open-source LLMs learn clinical information from large mined corpora through two crucial but understudied lenses: (1) their interpretation of clinical jargon, a foundational ability for understanding real-world clinical notes, and (2) their responses to unsupported medical claims. For both use cases, we investigate the frequency of relevant clinical information in their corresponding pretraining corpora, the relationship between pretraining data composition and model outputs, and the sources underlying this data. To isolate clinical jargon understanding, we evaluate LLMs on a new dataset MedLingo. Unsurprisingly, we find that the frequency of clinical jargon mentions across major pretraining corpora correlates with model performance. However, jargon frequently appearing in clinical notes often rarely appears in pretraining corpora, revealing a mismatch between available data and real-world usage. Similarly, we find that a non-negligible portion of documents support disputed claims that can then be parroted by models. Finally, we classified and analyzed the types of online sources in which clinical jargon and unsupported medical claims appear, with implications for future dataset composition.

Figures

Figures reproduced from arXiv: 2505.15024 by the authors.

Figure 1
Figure 1. An overview of our analysis: 1) Benchmarking models on their knowledge of the clinical jargon and debunked medical claims. 2) Estimating the prevalence of clinical keywords in the pretraining corpora and examining its correlation with model performance, and 3) Investigating the sources of clinical data in pretraining corpora, both for jargon and unsupported medical claims. narrow well-defined tasks that enable us to… view at source ↗
Figure 2
Figure 2. Example of the difference between lan￾guage in clinical notes vs. benchmarks. On the flip side, it is also important to understand how LLMs may be acquiring potentially dangerous information from these mined online corpora. Gener￾ation of unsupported medical claims poses risks when models are used in patient-facing applications, and there are existing concerns around model fragility and safety for high-stakes medica… view at source ↗
Figure 3
Figure 3. OLMo accuracy vs. Dolma estimated co-occurrence frequency on CASI dataset. Each dot shows a jargon-expansion pair. For each jargon-expansion pair in CASI, our esti￾mated occurrence counts in the training corpora cor￾relate strongly (0.56 ≤ ρ ≤ 0.72) with performance across all models ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Estimated frequency of jargon in the Dolma dataset vs. in MIMIC-IV Notes the prevalence of unsupported medical claims online, we further investigate how this propagates into the generations of LLMs. 5.1. Methods Evaluating Misleading Model Response We evaluated the ins…
Figure 5
Figure 5. Figure 5: Source classification for CASI, MedLingo, and the documents supporting disputed medical claims. port disputed medical claims in Dolma’s corpus. The content originates primarily from commercial health 10 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Steps to Construct MedLingo Dataset. The extraction of potential jargon follows the regex selec￾tion criterion described in Appendix C Additional filtering, including lemmatization and exclusion of common English words from the NLTK corpus, is applied in the pipeline. …
Figure 7
Figure 7. Figure 7: Accuracy on CASI dataset across LLaMA models of different sizes. ρ means Spearman correlation score. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Comparsion across LLaMA 7B, LLaMA 13B, LLaMA 33B, LLaMA 65B on [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Accuracy on CASI dataset across models pretrained on the different corpus. ρ means Spearman correlation score. 10 0 10 1 10 2 10 3 10 4 10 5 10 6 Estimated Frequency for c4 (log scale) 0 2 4 6 8 10 Number of Questions Distribution of Flan_T5 Performance Across Estimate…
Figure 10
Figure 10. Figure 10: Instruction-Tuned Model Correctness vs. Estimated frequency in Pretraining Corpora on MedLingo 19 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: OLMo accuracy vs. Dolma esti￾mated co-occurrence frequency on CASI dataset. Each dot shows a jargon-expansion pair. Green points indicate high-accuracy low-frequency instances (top 5 rows in adjacent table); red points represent low￾accuracy despite high-frequency cas…
Figure 12
Figure 12. Figure 12: Comparsion of MeLLaMA2 and LLaMA2 chat on [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Level of Disputed Medical Claims in Alpaca’s response across ratio of supportive documents and estimated counts of supportive documents in its pretraining corpora RedPajama 22 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 23 canonical work pages

  1. [1]

    Zero-shot clinical acronym expansion via latent meaning cells

    Griffin Adams, Mert Ketenci, Shreyas Bhave, Adler Perotte, and No \'e mie Elhadad. Zero-shot clinical acronym expansion via latent meaning cells. In Machine Learning for Health, pages 12--40. PMLR, 2020

    work page 2020

  2. [2]

    Large language models are few-shot clinical information extractors

    Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. Large language models are few-shot clinical information extractors. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1998--2022, Abu Dhabi, United Arab Emirates, December 2022...

    doi:10.18653/v1/2022.emnlp-main.130 2022

  3. [3]

    Medical large language models are vulnerable to data-poisoning attacks

    Daniel Alexander Alber, Zihao Yang, Anton Alyakin, Eunice Yang, Sumedha Rai, Aly A Valliani, Jeff Zhang, Gabriel R Rosenbaum, Ashley K Amend-Thomas, David B Kurland, et al. Medical large language models are vulnerable to data-poisoning attacks. Nature Medicine, pages 1--9, 2025

    work page 2025

  4. [4]

    Openbiollms: Advancing open-source large language models for healthcare and life sciences

    Malaikannan Sankarasubbu Ankit Pal. Openbiollms: Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B, 2024

    2024

  5. [5]

    Give me Some Hard Questions: Synthetic Data Generation for Clinical QA

    Fan Bai, Keith Harrigian, Joel Stremmel, Hamid Hassanzadeh, Ardavan Saeedi, and Mark Dredze. Give me some hard questions: Synthetic data generation for clinical qa. arXiv preprint arXiv:2412.04573, 2024

    work page Pith review arXiv 2024

  6. [6]

    Testing and evaluation of health care applications of large language models: a systematic review

    Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason A Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA, 2024

    2024

  7. [7]

    Cross-care: Assessing the healthcare implications of pre-training data on language model bias

    Shan Chen, Jack Gallifant, Mingye Gao, Pedro Moreira, Nikolaj Munch, Ajay Muthukkumar, Arvind Rajan, Jaya Kolluri, Amelia Fiske, Janna Hastings, et al. Cross-care: Assessing the healthcare implications of pre-training data on language model bias. arXiv preprint arXiv:2405.05506, 2024

    arXiv 2024

  8. [8]

    Meditron-70b: Scaling medical pretraining for large language models

    Zeming Chen, Alejandro Hern \'a ndez Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas K \"o pf, Amirkeivan Mohtashami, et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079, 2023

    arXiv 2023

Show all 53 references
  1. [9]

    Med42-v2: A suite of clinical llms

    Cl \'e ment Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. Med42-v2: A suite of clinical llms. arXiv preprint arXiv:2408.06142, 2024

    2024 arXiv

  2. [10]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024

    2024

  3. [11]

    Documenting large webtext corpora: A case study on the colossal clean crawled corpus

    Jesse Dodge, Maarten Sap, Ana Marasovi \'c , William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758, 2021

    2021 arXiv

  4. [12]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    What's in my big data? arXiv preprint arXiv:2310.20707, 2023

    Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What's in my big data? arXiv preprint arXiv:2310.20707, 2023

    2023 arXiv

  6. [14]

    Language models are surprisingly fragile to drug names in biomedical benchmarks

    Jack Gallifant, Shan Chen, Pedro Jos \'e Ferreira Moreira, Nikolaj Munch, Mingye Gao, Jackson Pond, Leo Anthony Celi, Hugo Aerts, Thomas Hartvigsen, and Danielle Bitterman. Language models are surprisingly fragile to drug names in biomedical benchmarks. In Yaser Al-Onaizan, Mo...

    2024

  7. [15]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  8. [16]

    Olmo: Accelerating the science of language models

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024

    2024 arXiv

  9. [17]

    Studying large language model generalization with influence functions

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023

    2023 arXiv

  10. [18]

    Medalpaca--an open-source collection of medical conversational ai models and training data

    Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander L \"o ser, Daniel Truhn, and Keno K Bressem. Medalpaca--an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247, 2023

    2023 arXiv

  11. [19]

    Mednli is not immune: Natural language inference artifacts in the clinical domain

    Christine Herlihy and Rachel Rudinger. Mednli is not immune: Natural language inference artifacts in the clinical domain. arXiv preprint arXiv:2106.01491, 2021

    2021 arXiv

  12. [20]

    Medical adaptation of large language and vision-language models: Are we making progress? arXiv preprint arXiv:2411.04118, 2024 a

    Daniel P Jeong, Saurabh Garg, Zachary C Lipton, and Michael Oberst. Medical adaptation of large language and vision-language models: Are we making progress? arXiv preprint arXiv:2411.04118, 2024 a

    2024 arXiv

  13. [21]

    The limited impact of medical adaptation of large language and vision-language models

    Daniel P Jeong, Pranav Mani, Saurabh Garg, Zachary C Lipton, and Michael Oberst. The limited impact of medical adaptation of large language and vision-language models. arXiv preprint arXiv:2411.08870, 2024 b

    2024 arXiv

  14. [22]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11 0 (14): 0 6421, 2021

    2021

  15. [23]

    Matching patients to clinical trials with large language models

    Qiao Jin, Zifeng Wang, Charalampos S Floudas, Fangyuan Chen, Changlin Gong, Dara Bracken-Clarke, Elisabetta Xue, Yifan Yang, Jimeng Sun, and Zhiyong Lu. Matching patients to clinical trials with large language models. Nature communications, 15 0 (1): 0 9074, 2024

    2024

  16. [24]

    Mimic-iii

    Alistair Johnson, Pollard Tom, and Roger Mark. Mimic-iii. PhysioNet. Available online at: https://physionet.org/content/mimiciii/1.4/, 2016 a

    2016

  17. [25]

    Mimic-iv

    Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021), pages 49--55, 2020

    2021

  18. [26]

    Mimic-iii, a freely accessible critical care database

    Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3 0 (1): 0 1--9, 2016 b

    2016

  19. [27]

    Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports

    Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6 0 (1): 0 317, 2019

    2019

  20. [28]

    Mimic-iv, a freely accessible electronic health record dataset

    Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10 0 (1): 0 1, 2023

    2023

  21. [29]

    Large language models struggle to learn long-tail knowledge

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696--15707. PMLR, 2023

    2023

  22. [30]

    Debunking health fake news with domain specific pre-trained model

    Santoshi Kumari, Harshitha K Reddy, Chandan S Kulkarni, and Vanukuri Gowthami. Debunking health fake news with domain specific pre-trained model. Global Transitions Proceedings, 2 0 (2): 0 267--272, 2021

    2021

  23. [31]

    Can large language models abstract medical coded language? arXiv preprint arXiv:2403.10822, 2024

    Simon A Lee and Timothy Lindsey. Can large language models abstract medical coded language? arXiv preprint arXiv:2403.10822, 2024

    2024 arXiv

  24. [32]

    A scoping review of using large language models (llms) to investigate electronic health records (ehrs)

    Lingyao Li, Jiayan Zhou, Zhenxiang Gao, Wenyue Hua, Lizhou Fan, Huizi Yu, Loni Hagen, Yongfeng Zhang, Themistocles L Assimes, Libby Hemphill, et al. A scoping review of using large language models (llms) to investigate electronic health records (ehrs). arXiv preprint arXiv:240...

    2024 arXiv

  25. [33]

    Are clinical t5 models better for clinical text?, 2024 b

    Yahan Li, Keith Harrigian, Ayah Zirikly, and Mark Dredze. Are clinical t5 models better for clinical text?, 2024 b . URL https://arxiv.org/abs/2412.05845

    2024 arXiv

  26. [34]

    Infini-gram: Scaling unbounded n-gram language models to a trillion tokens

    Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377, 2024

    2024 arXiv

  27. [35]

    S2orc: The semantic scholar open research corpus

    Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Dan S Weld. S2orc: The semantic scholar open research corpus. arXiv preprint arXiv:1911.02782, 2019

    1911 arXiv

  28. [36]

    Fake or real news about covid-19? pretrained transformer model to detect potential misleading news

    SreeJagadeesh Malla and PJA Alphonse. Fake or real news about covid-19? pretrained transformer model to detect potential misleading news. The European Physical Journal Special Topics, 231 0 (18): 0 3347--3356, 2022

    2022

  29. [37]

    Evaluating base and retrieval augmented llms with document or online support for evidence based neurology

    Lars Masanneck, Sven G Meuth, and Marc Pawlitzki. Evaluating base and retrieval augmented llms with document or online support for evidence based neurology. npj Digital Medicine, 8 0 (1): 0 137, 2025

    2025

  30. [38]

    A sense inventory for clinical abbreviations and acronyms created using clinical notes and medical dictionary resources

    Sungrim Moon, Serguei Pakhomov, Nathan Liu, James O Ryan, and Genevieve B Melton. A sense inventory for clinical abbreviations and acronyms created using clinical notes and medical dictionary resources. Journal of the American Medical Informatics Association, 21 0 (2): 0 299--...

    2014

  31. [39]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  32. [40]

    It’s time to bench the medical exam benchmark, 2025

    Inioluwa Deborah Raji, Roxana Daneshjou, and Emily Alsentzer. It’s time to bench the medical exam benchmark, 2025

    2025

  33. [41]

    Dolma: An open corpus of three trillion tokens for language model pretraining research

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159, 2024

    2024 arXiv

  34. [42]

    Large language models are poor medical coders—benchmarking of medical code querying

    Ali Soroush, Benjamin S Glicksberg, Eyal Zimlichman, Yiftach Barash, Robert Freeman, Alexander W Charney, Girish N Nadkarni, and Eyal Klang. Large language models are poor medical coders—benchmarking of medical code querying. NEJM AI, 1 0 (5): 0 AIdbp2300040, 2024

    2024

  35. [43]

    Prevalence of health misinformation on social media: systematic review

    Victor Suarez-Lledo and Javier Alvarez-Galvez. Prevalence of health misinformation on social media: systematic review. Journal of medical Internet research, 23 0 (1): 0 e17187, 2021

    2021

  36. [44]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  37. [45]

    RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset , April 2023

    Together Computer . RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset , April 2023. URL https://github.com/togethercomputer/RedPajama-Data

    2023

  38. [46]

    Clinical camel: An open-source expert-level medical language model with dialogue-based knowledge encoding

    Augustin Toma, Patrick R Lawler, Jimmy Ba, Rahul G Krishnan, Barry B Rubin, and Bo Wang. Clinical camel: An open-source expert-level medical language model with dialogue-based knowledge encoding. CoRR, 2023

    2023

  39. [47]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  40. [48]

    Adapted large language models can outperform medical experts in clinical text summarization

    Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerov \'a , et al. Adapted large language models can outperform medical experts in clinical text summarizati...

    2024

  41. [49]

    Redpajama: an open dataset for training large language models

    Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. arXiv preprint arXiv:2411.12372, 2024

    2024 arXiv

  42. [50]

    Me-llama: Foundation large language models for medical applications

    Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, et al. Me-llama: Foundation large language models for medical applications. Research square, pages rs--3, 2024

    2024

  43. [51]

    Almanac—retrieval-augmented language models for clinical medicine

    Cyril Zakka, Rohan Shad, Akash Chaurasia, Alex R Dalal, Jennifer L Kim, Michael Moor, Robyn Fong, Curran Phillips, Kevin Alexander, Euan Ashley, et al. Almanac—retrieval-augmented language models for clinical medicine. NEJM AI, 1 0 (2): 0 AIoa2300068, 2024

    2024

  44. [52]

    A dataset for evaluating clinical research claims in large language models

    Boya Zhang, Alban Bornet, Anthony Yazdani, Philipp Khlebnikov, Marija Milutinovic, Hossein Rouhizadeh, Poorya Amini, and Douglas Teodoro. A dataset for evaluating clinical research claims in large language models. Scientific Data, 12 0 (1): 0 86, 2025

    2025

  45. [53]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.