Pith. sign in

REVIEW 6 major objections 5 minor 51 references

Distilling Large Language Models for Efficient Clinical Information Extraction

T0 review · 6 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Distilled BERT models rival LLM teachers on clinical NER at 101x lower cost.

desk verdict Solid empirical distillation study for clinical NER, but the headline cost savings rest on an inconsistent GPU rate and the Gemini Flash comparison inverts under the paper's own stated price. read the letter →

arxiv 2501.00031 v1 pith:INES2YPZ submitted 2024-12-21 cs.CL

Karthik S. Vedula , Annika Gupta , Akshay Swaminathan , Ivan Lopez , Suhana Bedi , Nigam H. Shah This is my paper · ORCID
classification cs.CL
keywords knowledgedistillationclinicalNERBERTlargelanguagemodelsnamedentityrecognitionelectronichealthrecordsBiomedicationextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the accuracy of large language models on clinical named-entity recognition can be transferred to a model roughly 1,000 times smaller, making the task cheap enough for routine use. The authors generate training labels for medications, diseases, and symptoms from API-based LLMs (GPT-4o, o1-mini, Gemini Flash) and medical ontologies (RxNorm, SNOMED), then fine-tune BERT-family models on those labels. On three public clinical datasets, the distilled BioBERT model reaches F1 of 0.84 for diseases and 0.87 for medications, close to the 0.89 and 0.91 of BERT trained on human labels, and above the LLM teachers' own direct F1 of 0.82 and 0.84. The distilled model runs up to 12 times faster and, at the paper's chosen pricing, up to 101 times cheaper than the teacher LLMs. The paper argues that distillation is a practical route to scalable clinical information extraction without large expert-annotated corpora.

What carries the argument

The load-bearing mechanism is teacher labeler ensembling followed by supervised fine-tuning. Five teachers are used: four LLMs (GPT-4o, GPT-4o-mini, o1-mini, Gemini 1.5 Flash) and one ontology matcher (RxNorm for medications, SNOMED CT for diseases and symptoms). All 31 non-empty subsets of these teachers are evaluated on a development set, and the union of entities from the best-scoring combination is converted into Inside-Outside token labels for fine-tuning a BERT variant (BERT-base, BioBERT, or BioClinBERT). This union-of-entities operation is what transfers the teachers' knowledge into the small model, and the per-task selection of the best teacher subset is what makes the transfer work across different entity types.

What would settle it

Recompute the cost per note using the $28/hour for 4xH100 stated in the Methods; if the cost advantage over o1-mini falls from 101x to roughly 15x, the headline economy claim as stated does not hold.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that knowledge distillation from multiple teacher labelers—commercial LLMs plus ontology matchers—produces small BERT models that perform essentially as well as those same LLMs do directly, and nearly as well as BERT fine-tuned on human annotations. For diseases, o1-mini alone was the best teacher (direct F1 0.787), yet the distilled BioBERT scored 0.84; for medications, Gemini Flash plus GPT-4o was the best teacher (0.881) and distilled BioBERT scored 0.87; for symptoms, the best teacher combination scored 0.801 and the distilled model 0.68. In external validation on notes from a different health system, the distilled model's F1 was 0.883 for medications, 0.726 for diseases, and 0.699 for symptoms. An error analysis found that 63–82% of the models' false positives were entities correctly identified but missing from the human test labels, so the paper argues the reported precision and F1 are lower bounds. The paper concludes that distillation offers an efficient and scalable alternative to LLM API calls for clinical NER.

Load-bearing premise

The cost and latency comparison assumes that running BERT on a rented GPU at a single hourly rate is directly comparable to calling commercial LLM APIs at list token prices, so the claimed savings multiplier depends on which hardware price and which API pricing are current.

Editorial extensions

If this is right

  • A healthcare system can run clinical NER on local hardware with a roughly 110-million-parameter BERT model, avoiding per-query API costs and the need to send protected health information to external endpoints.
  • The distilled model's F1 on medications and diseases is within 0.02–0.05 of BERT trained on fully human-labeled data, so the practical trade-off for most downstream uses such as cohort selection and phenotyping is small.
  • Because teacher labels come from APIs and ontologies rather than manual annotation, the pipeline can be re-run at scale on new note types or new entity types at a fraction of the cost of building new gold-standard corpora.
  • External validation on MedAlign notes suggests the distilled model generalizes across health systems for medication and disease extraction, although symptom extraction remains below 0.70 F1.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Recomputing the cost comparison with the paper's stated $28/hour 4xH100 rate instead of the $4.74/hour 1xA100 rate used in Table 4 drops the headline 101x advantage over o1-mini to roughly 15x; the qualitative claim of large savings holds, but the exact multiplier depends on hardware pricing.
  • The union-of-entities rule for combining teachers may need per-task tuning for other entity types, since the paper finds that adding ontology labels sometimes hurts F1 by increasing false positives and that no single teacher wins across all tasks.
  • If test-set labels are as noisy as the error analysis suggests, published benchmark scores across the field may systematically understate real-world performance; cleaning and re-annotating the n2c2, NCBI, and CORAL test sets could change the ranking of many clinical NER models.
  • A self-improving loop is possible: as LLM teachers improve, regenerating teacher labels and re-fine-tuning BERT would improve the distilled models without any new human annotation, provided the teacher combination is re-selected on a stable development set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper proposes a knowledge-distillation pipeline for clinical named entity recognition (NER) in which large language models (GPT-4o, GPT-4o-mini, o1-mini, Gemini 1.5 Flash) and medical ontologies (RxNorm, SNOMED) serve as teacher labelers for medication, disease, and symptom extraction. The authors evaluate all 31 teacher combinations on development sets, select the best per task, fine-tune BERT variants on the resulting hard labels, and compare performance against the teachers and against BERT fine-tuned on human labels across five datasets, with external validation on 10 notes from MedAlign. The headline claims are that distilled BioBERT achieves F1 comparable to its teachers (0.84 vs 0.82 for disease; 0.87 vs 0.84 for medication; 0.68 vs 0.73 for symptoms) and is up to 101x cheaper and 12x faster in inference.

Significance. If the efficiency claims hold, distillation from LLM teacher labelers is a practically useful strategy for clinical NER, and the paper's direct comparison of teacher-only versus distilled-BERT performance, combined with error analysis across multiple datasets, is a valuable contribution. The authors also release code and report inter-annotator agreement, which supports reproducibility. However, the central cost and speed claims rest on an internally inconsistent computational-cost model, and the external validation is based on a very small sample, so the strength of the practical-deployment conclusion is currently not commensurate with the evidence presented.

major comments (6)
  1. [Methods (Inference Time and Cost Analysis) and Results (Inference Time and Cost), Table 4, Table 6] The cost model is internally inconsistent. Methods states that BERT inference cost is estimated at $28/hour for a 4xH100 virtual machine, while Table 4 and the Results text use $4.74/hour for a 1xA100 80GB, and Table 6 lists only 1xA100 vendor rates. Under the paper's stated $28/hour rate, the distilled BioBERT cost per note becomes about $0.00109 rather than $0.000187, which changes the headline ratios from 85x/101x/2x to roughly 14.6x/17.3x/0.42x for GPT-4o/o1-mini/Gemini Flash. The Gemini Flash comparison would reverse, with BERT being more expensive. The abstract and conclusions do not hedge on this dependency. The authors must reconcile the stated hardware cost with the numbers used in the cost tables and re-evaluate the economic claims accordingly.
  2. [Methods (External Validation), Table 11, Table 13] The external validation is based on only 10 manually annotated notes from MedAlign, with just 2 notes doubly annotated and a reported Cohen's kappa of 0.61. A kappa of 0.61 indicates moderate agreement and casts doubt on the reliability of the gold standard for this small sample. The paper nonetheless claims 'strong performance' on the external dataset. Confidence intervals or other uncertainty estimates are not provided, and the Discussion does not list this as a limitation. The generalizability claim is therefore not well supported by the presented evidence.
  3. [Discussion, paragraph 1; Table 2] The Discussion states that 'distilled BERT models outperformed teacher labelers,' but this is contradicted by the symptom extraction results in Table 2, where the teacher-only F1 is 0.73 and the distilled BioBERT F1 is 0.68. The blanket claim is therefore not accurate as written, and the symptom result is a negative result relative to the teacher. The Discussion should be revised to acknowledge that distillation underperformed the teacher for symptoms and to discuss possible reasons.
  4. [Abstract and Discussion, speed multipliers] The reported speed-up factors are inconsistent: the Abstract says distilled BERT was '12x, 4x, 8x faster' than GPT-4o, o1-mini, and Gemini Flash, respectively, while the Discussion says '2x, 4x, 8x'. Table 4's timing data (1.66 s vs 0.14 s) support 12x, but the Discussion's 2x appears to be an error. This inconsistency in a headline quantitative claim must be corrected.
  5. [Results (Inference Time and Cost), Table 4] The inference-cost comparison omits the one-time cost of generating teacher labels via LLM API calls and the cost of fine-tuning the BERT models. For a fair 'cheaper' claim to be sustained in a deployment scenario, these fixed costs should be reported and amortized over a plausible volume of notes, or the authors should explicitly state that the comparison is inference-only and justify that the fixed costs are negligible for the intended use case.
  6. [Results (Inference Time and Cost), Table 4] The cost calculation for the LLM teachers is not fully transparent. The paper says tiktoken was used to compute token counts, but it does not report the number of input and output tokens per note, the pricing per million tokens used, or whether the long system prompts (which are included in the Supplementary) were counted. Providing these details is necessary for reproducibility and for assessing whether the reported per-note costs are accurate.
minor comments (5)
  1. [Results (Inference Time and Cost), Table 6 and accompanying text] The Results text says the cost estimate is for 'a 4xH100 virtual machine,' but Table 6 reports '1xA100 80GB' vendor prices. Unify the hardware description and the cost source.
  2. [Methods (Model Distillation Implementation)] The learning rate is written as '2x10e-5'; this should be '2e-5' or '2 × 10⁻⁵' to avoid ambiguity.
  3. [Table 12] The table caption states development splits were used for 'prompt engineering,' but the Methods describe using them for selecting the optimal teacher labeler combination. Align the terminology.
  4. [Abstract] The abstract says 'over 3,300 clinical notes,' but the teacher-labeling set is described as 2,096 documents; clarify whether the 3,300 figure includes test and development notes from all datasets, and make the arithmetic explicit.
  5. [Supplementary Material, Medication Prompt] The output format instructions say 'Output a string delimited by //', but the example shows a JSON object with 'entities' and 'rationale' keys. This mismatch could confuse readers trying to replicate the teacher-labeling pipeline.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all student-model F1 claims are benchmarked against held-out human annotations, and teacher selection on a development set is standard model selection rather than a fitted prediction.

full rationale

The derivation chain is self-contained against external benchmarks. Teacher labels are produced by LLMs and ontologies independently of the student BERT models; the students are then fine-tuned on those labels and evaluated on held-out human-annotated test sets (NCBI, n2c2, CORAL, and the MedAlign external set), so the reported F1 values are not forced by construction. Selecting the best teacher combination by F1 on the development set is a standard model-selection step, and the test-set F1s could have differed materially from the dev-set rankings; there is no fitted parameter that is later renamed as a prediction. Human-labeled BERT models provide an additional external comparator, and the MedAlign validation uses freshly annotated notes with reported inter-rater agreement. Citations to prior work from the same research group (e.g., the TROVE weak-supervision paper and observational-EHR references) are background or dataset citations and are not load-bearing for the central claim. The one notable weakness is an internal inconsistency in the cost model: the Methods section states a $28/hour 4xH100 rate while Tables 4 and 6 use $4.74/hour for a 1xA100, which materially changes the claimed savings and can invert the Gemini Flash comparison. That is a correctness and accounting risk in an economic calculation, not a circular derivation: the cost ratio is arithmetic from chosen rates, not an empirical result that reduces to its own inputs. No circular step meeting the quoted-evidence standard is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or forces. The key free parameters are the selected teacher subset per task, the fixed BERT hyperparameters, and the hourly compute rate used in cost calculations. The axioms reflect that the entire study depends on the quality of teacher labels, the validity of noisy gold-standard test sets, and the representativeness of the small external annotation sample.

free parameters (4)
  • Optimal teacher subset per task = Gemini+GPT-4o for medications and symptoms; o1-mini for diseases
    Selected by maximizing F1 on the development set among 31 subsets; affects all downstream distilled model results.
  • BERT fine-tuning hyperparameters = lr=2e-5, batch=8, 10 epochs, weight decay 0.01
    Chosen by hand, standard values; not tuned per task.
  • Hourly compute cost for BERT inference = $4.74/hour (1xA100) in Table 4; $28/hour (4xH100) in Methods
    Directly determines the claimed 85x-101x cost savings; the two rates are inconsistent, changing the savings to roughly 14x-17x at $28/hour.
  • LLM sampling parameters = temperature=0.01, top-p=0.9
    Set for consistency; not varied.
assumptions (4)
  • domain assumption The IO token labeling scheme and token-level F1 are appropriate for measuring NER performance.
    The paper uses Inside-Outside labels and token-level metrics; entity-level F1 might differ, and the error analysis suggests token-level labels are noisy.
  • domain assumption The test sets (NCBI, n2c2, CORAL) are valid gold standards despite acknowledged label errors.
    The paper's error analysis shows 63-82% of model false positives are actually 'correct', implying the gold labels are unreliable; this directly affects all reported F1s.
  • domain assumption Teacher labels from LLMs and ontologies are suitable training signal for distillation.
    The whole approach rests on using teacher labels in place of human labels; the paper shows human-labeled BERT is better, so this assumption is only partially satisfied.
  • ad hoc to paper The 10 annotated MedAlign notes are a representative gold standard for external validation.
    Only 10 notes were annotated (2 doubly), with kappa 0.61; the F1 on this tiny sample is treated as the external generalization result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distilling Large Language Models for Efficient Clinical Information Extraction." pith.science (2026). https://pith.science/paper/INES2YPZ

@misc{pith2026250100031,
  author       = {Pith},
  title        = {Pith review of: Distilling Large Language Models for Efficient Clinical Information Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/INES2YPZ}},
  note         = {Machine review of arXiv:2501.00031}
}
read the original abstract

Large language models (LLMs) excel at clinical information extraction but their computational demands limit practical deployment. Knowledge distillation--the process of transferring knowledge from larger to smaller models--offers a potential solution. We evaluate the performance of distilled BERT models, which are approximately 1,000 times smaller than modern LLMs, for clinical named entity recognition (NER) tasks. We leveraged state-of-the-art LLMs (Gemini and OpenAI models) and medical ontologies (RxNorm and SNOMED) as teacher labelers for medication, disease, and symptom extraction. We applied our approach to over 3,300 clinical notes spanning five publicly available datasets, comparing distilled BERT models against both their teacher labelers and BERT models fine-tuned on human labels. External validation was conducted using clinical notes from the MedAlign dataset. For disease extraction, F1 scores were 0.82 (teacher model), 0.89 (BioBERT trained on human labels), and 0.84 (BioBERT-distilled). For medication, F1 scores were 0.84 (teacher model), 0.91 (BioBERT-human), and 0.87 (BioBERT-distilled). For symptoms: F1 score of 0.73 (teacher model) and 0.68 (BioBERT-distilled). Distilled BERT models had faster inference (12x, 4x, 8x faster than GPT-4o, o1-mini, and Gemini Flash respectively) and lower costs (85x, 101x, 2x cheaper than GPT-4o, o1-mini, and Gemini Flash respectively). On the external validation dataset, the distilled BERT model achieved F1 scores of 0.883 (medication), 0.726 (disease), and 0.699 (symptom). Distilled BERT models were up to 101x cheaper and 12x faster than state-of-the-art LLMs while achieving similar performance on NER tasks. Distillation offers a computationally efficient and scalable alternative to large LLMs for clinical information extraction.

Figures

Figures reproduced from arXiv: 2501.00031 by the authors.

Figure 1
Figure 1. Clinical documents were passed to teacher labelers—LLMs and ontologies—for medication, symptom, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 39 canonical work pages

  1. [1]

    K., Wei, W

    Ross, M. K., Wei, W. & Ohno-Machado, L. ”Big data” and the electronic health record. Yearbook of Medical Informatics 9, 97–104 (2014)

    work page 2014

  2. [2]

    Wornow, M. et al. Zero-Shot Clinical Trial Patient Matching with LLMs (2024). URL http://arxiv.org/ abs/2402.05125. ArXiv:2402.05125 [cs]

    arXiv 2024

  3. [3]

    Callahan, A., Shah, N. H. & Chen, J. H. Research and Reporting Considerations for Observational Studies Using Electronic Health Record Data. Annals of Internal Medicine 172, S79–S84 (2020)

    work page 2020

  4. [4]

    & Couto, F

    Lamurias, A. & Couto, F. M. LasigeBioTM at MEDIQA 2019: Biomedical Question Answering using Bidirectional Transformers and Named Entity Recognition. In Proceedings of the 18th BioNLP Workshop and Shared Task , 523–527 (Association for Computational Linguistics, Florence, Italy, 2019). URL https://www.aclweb.org/anthology/W19-5057

    work page 2019

  5. [5]

    & Cohen, K

    Zweigenbaum, P., Demner-Fushman, D., Yu, H. & Cohen, K. B. Frontiers of biomedical text mining: current progress. Briefings in bioinformatics 8, 358–375 (2007). URL https://www.ncbi.nlm.nih.gov/ pmc/articles/PMC2516302/

    work page 2007

  6. [6]

    Wang, Y. et al. Clinical information extraction applications: A literature review. Journal of Biomedical Informatics 77, 34–49 (2018)

    2018

  7. [7]

    & Dyer, C

    Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K. & Dyer, C. Neural Architectures for Named Entity Recognition (2016). URL http://arxiv.org/abs/1603.01360. ArXiv:1603.01360 [cs]

    arXiv 2016

  8. [8]

    The Unified Medical Language System (UMLS): integrating biomedical terminology

    Bodenreider, O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Research 32, D267–D270 (2004). URL https://www.ncbi.nlm.nih.gov/pmc/articles/ PMC308795/

    work page 2004

Show all 51 references
  1. [9]

    Liao, K. P. et al. Development of phenotype algorithms using electronic medical records and incorporating natural language processing. The BMJ 350, h1885 (2015). URL https://www.ncbi.nlm.nih.gov/pmc/ articles/PMC4707569/

    2015

  2. [10]

    & Moreno-Sandoval, A

    Campillos-Llanos, L., Valverde-Mateos, A., Capllonch-Carri´ on, A. & Moreno-Sandoval, A. A clinical trials corpus annotated with UMLS entities to enhance the access to evidence-based medicine. BMC Medical Informatics and Decision Making 21, 69 (2021). URL https://doi.org/10.11...

    2021 doi

  3. [11]

    & Toutanova, K

    Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Burstein, J., Doran, C. & Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computati...

    2019

  4. [12]

    Fries, J. A. et al. Ontology-driven weak supervision for clinical entity classification in electronic health records. Nature Communications 12, 2017 (2021). URL https://www.nature.com/articles/ s41467-021-22328-4 . Publisher: Nature Publishing Group

    2021

  5. [13]

    Lee, J. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234–1240 (2020). URL https://doi.org/10.1093/bioinformatics/btz682

    2020 doi

  6. [14]

    & Ranganath, R

    Huang, K., Altosaar, J. & Ranganath, R. ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission (2020). URL http://arxiv.org/abs/1904.05342. ArXiv:1904.05342 [cs]. 9

    2020 arXiv

  7. [15]

    Chaves, J. M. Z. et al. RaLEs: a Benchmark for Radiology Language Evaluations. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023). URL https://openreview.net/forum?id=PWLGrvoqiR

    2023

  8. [16]

    Monajatipoor, M. et al. LLMs in Biomedicine: A study on clinical Named Entity Recognition (2024). URL http://arxiv.org/abs/2404.07376. ArXiv:2404.07376 [cs]

    2024 arXiv

  9. [17]

    URL https://openai.com/api/pricing/

    Pricing. URL https://openai.com/api/pricing/

  10. [18]

    & Dean, J

    Hinton, G., Vinyals, O. & Dean, J. Distilling the Knowledge in a Neural Network (2015). URL http: //arxiv.org/abs/1503.02531. ArXiv:1503.02531 [stat]

    2015 arXiv

  11. [19]

    & Poon, H

    Zhou, W., Zhang, S., Gu, Y., Chen, M. & Poon, H. UniversalNER: Targeted distillation from large language models for open named entity recognition. In The Twelfth International Conference on Learning Representations (2024). URL https://openreview.net/forum?id=r65xfUb76p

    2024

  12. [20]

    Rhouma, R. et al. Leveraging mobile NER for real-time capture of symptoms, diagnoses, and treatments from clinical dialogues. Informatics in Medicine Unlocked 48, 101519 (2024). URL https://www.sciencedirect. com/science/article/pii/S2352914824000753

    2024

  13. [21]

    Gu, Y. et al. Distilling Large Language Models for Biomedical Knowledge Extraction: A Case Study on Adverse Drug Events (2023). URL http://arxiv.org/abs/2307.06439. ArXiv:2307.06439 [cs]

    2023 arXiv

  14. [22]

    Gallifant, J. et al. The TRIPOD-LLM Statement: A Targeted Guideline For Reporting Large Language Models Use. medRxiv: The Preprint Server for Health Sciences 2024.07.24.24310930 (2024)

    2024

  15. [23]

    & Uzuner, O

    Henry, S., Buchan, K., Filannino, M., Stubbs, A. & Uzuner, O. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. Journal of the American Medical Informatics Association: JAMIA 27, 3–12 (2020)

    2020

  16. [24]

    Johnson, A. E. et al. MIMIC-III, a freely accessible critical care database. Scientific Data 3, 160035 (2016). URL https://www.nature.com/articles/sdata201635

    2016

  17. [25]

    I., Leaman, R

    Do˘ gan, R. I., Leaman, R. & Lu, Z. NCBI disease corpus: a resource for disease name recognition and concept normalization. Journal of Biomedical Informatics 47, 1–10 (2014)

    2014

  18. [26]

    Sushil, M. et al. CORAL: expert-Curated medical Oncology Reports to Advance Language model inference. URL https://physionet.org/content/curated-oncology-reports/1.0/

  19. [27]

    Fleming, S. L. et al. MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (2023). URL http://arxiv.org/abs/2308.14089. ArXiv:2308.14089 [cs]

    2023 arXiv

  20. [28]

    URL https://openai.com/index/hello-gpt-4o/

    Hello GPT-4o. URL https://openai.com/index/hello-gpt-4o/

  21. [29]

    URL https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/

    GPT-4o mini: advancing cost-efficient intelligence. URL https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/

  22. [30]

    URL https://openai.com/index/openai-o1-mini-advancing-cost-efficient-reasoning/

    OpenAI o1-mini. URL https://openai.com/index/openai-o1-mini-advancing-cost-efficient-reasoning/

  23. [31]

    Georgiev, P. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context (2024). URL http://arxiv.org/abs/2403.05530. ArXiv:2403.05530 [cs]

    2024 arXiv

  24. [32]

    Whetzel, P. L. et al. BioPortal: enhanced functionality via new Web services from the National Center for Biomedical Ontology to access and use ontologies in software applications. Nucleic Acids Research 39, W541–545 (2011)

    2011

  25. [33]

    & Nelson, S

    Liu, S., Wei Ma, Moore, R., Ganesan, V. & Nelson, S. RxNorm: prescription for electronic drug information exchange. IT Professional 7, 17–23 (2005). URL http://ieeexplore.ieee.org/document/1516084/

    2005

  26. [34]

    Q., Price, C., Spackman, K

    Stearns, M. Q., Price, C., Spackman, K. A. & Wang, A. Y. SNOMED clinical terms: overview of the development process and project status. Proceedings of the AMIA Symposium 662–666 (2001). URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2243297/. 10

    2001

  27. [35]

    Alsentzer, E. et al. Publicly Available Clinical BERT Embeddings. In Rumshisky, A., Roberts, K., Bethard, S. & Naumann, T. (eds.) Proceedings of the 2nd Clinical Natural Language Processing Workshop , 72–78 (Association for Computational Linguistics, Minneapolis, Minnesota, US...

    2019

  28. [36]

    URL https://github.com/openai/tiktoken

    openai/tiktoken (2024). URL https://github.com/openai/tiktoken. Original-date: 2022-12- 01T23:22:11Z

    2024

  29. [37]

    URL https://datacrunch.io/blog/ cloud-gpu-pricing-comparison-in-2024

    Cloud GPU Pricing Comparison in 2024 (2024). URL https://datacrunch.io/blog/ cloud-gpu-pricing-comparison-in-2024 . Section: GPUs

    2024

  30. [38]

    P., Garg, S., Lipton, Z

    Jeong, D. P., Garg, S., Lipton, Z. C. & Oberst, M. Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? (2024). URL http://arxiv.org/abs/2411.04118. ArXiv:2411.04118 [cs]

    2024 arXiv

  31. [39]

    remarkable propensity to bacterial infections

    TUI Semantic Type List. URL https://lhncbc.nlm.nih.gov/ii/tools/MetaMap/Docs/ SemanticTypes_2018AB.txt. Author Contributions The study was conceptualized by KSV and AS. Coding and data analysis were performed by KSV, AS, and AG, while data annotation was carried out by AS and ...

    2024

  32. [40]

    • If present in the note, include the full name, like tiotropium bromide or albuterol sulfate, instead of just tiotropium or albuterol

    Make sure to include: (a) Specific medication names (both brand and generic). • If present in the note, include the full name, like tiotropium bromide or albuterol sulfate, instead of just tiotropium or albuterol. (b) Drug class names, both singular and plural, including (but ...

  33. [41]

    (b) Modes of administration or formulations, such as: • IV, drip, gtt (drops), liquid form, transfused, supplement, etc

    Do not include: (a) Medical devices or equipment, such as: • Inhaler, nebulizer, BiPAP machine, etc. (b) Modes of administration or formulations, such as: • IV, drip, gtt (drops), liquid form, transfused, supplement, etc. (c) Methods of delivery or routes of administration, un...

  34. [42]

    • Include only pharmacologically relevant terms (e.g., antibiotic, anticoagulant, steroid)

    Additional Notes: • Avoid listing terms like increased, solution (Soln), isotonic, or daycare, which are not medications or drug classes. • Include only pharmacologically relevant terms (e.g., antibiotic, anticoagulant, steroid)

  35. [43]

    • Do not list FIO2 unless explicitly described as oxygen therapy

    Examples of Challenging Cases: (a) False Positives to Avoid: • Oxygen as a standalone word unless explicitly used as a therapy or treatment. • Do not list FIO2 unless explicitly described as oxygen therapy. • Avoid terms like isotonic or Sodium unless part of a drug name (e.g....

  36. [44]

    • Minimal entity spans: Use the simplest terms that convey the symptom information (e.g., ”nausea” instead of ”the patient presents with nausea”)

    Include: • All mentions of symptoms or complaints, such as ”fatigue,” ”nausea,” ”vomiting,” or ”pain.” • Negated symptoms (e.g., ”denies nausea” should still include ”nausea”). • Minimal entity spans: Use the simplest terms that convey the symptom information (e.g., ”nausea” i...

  37. [45]

    • Information about the location of the symptom (e.g., ”low back pain” → Include only ”pain”)

    Do not include: • Signs or clinical findings that are not symptoms (e.g., ”hyperbilirubinemia,” ”ascites”). • Information about the location of the symptom (e.g., ”low back pain” → Include only ”pain”). • Adjectives or descriptors unrelated to the symptom itself (e.g., ”low,” ...

  38. [46]

    • Avoid listing any context or causes of the symptom

    Edge Cases: • For combinations of symptoms (e.g., ”nausea and vomiting”), list each symptom separately (e.g., ”nausea // vomiting”). • Avoid listing any context or causes of the symptom. For example: – ”pain secondary to surgery” → Include only ”pain.” – ”headache from dehydra...

  39. [47]

    Write the symptoms exactly as they appear in the note

    ”entities”: a single string of symptoms, separated by ‘//‘. Write the symptoms exactly as they appear in the note

  40. [48]

    Patient denies nausea and vomiting

    ”negated symptoms included”: a string, affirming that all negated symptoms were included Example: Note: ”Patient is experiencing muscle pain, lower back pain, and fatigue, secondary to statin therapy for coronary artery disease. Patient denies nausea and vomiting. The patient ...

  41. [49]

    Include: • Specific diseases and disorders – Examples: ”ataxia-telangiectasia”, ”Phenylketonuria”, ”Aniridia” • Disease categories or classes – Examples: ”Inherited human disease”, ”Chromosome abnormalities”, ”Cancer” • Composite mentions indicating diseases or conditions – Ex...

  42. [50]

    Exclude: • Genetic mutations or hypotheses without explicit disease mention – Examples: ”Disease-causing mutations”, ”Hypothesis of a defective gene” • Hypothetical or unconfirmed conditions – Examples: ”Hypothesis of a defective C9”, ”Compound heterozygote for uncharacterized...

  43. [51]

    Ensure no acronyms are omitted from the output

    Additional Instructions for Acronyms: • Focus on identifying acronyms that represent diseases, disorders, and findings. Ensure no acronyms are omitted from the output. Output Format Output a string delimited by ‘//‘ to separate the diseases. Write them exactly as they were wri...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.