Pith. sign in

REVIEW 3 major objections 5 minor 45 references

NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that contrastive pretraining aligning medical notes with ICD-10 diagnosis sequences improves automated diagnostic coding, with the largest gains on rare codes.

desk verdict Genuinely new contrastive pretraining recipe for medical coding, with solid benchmark work and honest limitations; the rare-code attribution needs an in/out-of-vocabulary split before I'd fully trust it. read the letter →

arxiv 2412.11477 v1 pith:VJEMPQWZ submitted 2024-12-16 cs.LG cs.CL

Prajwal Kailas , Max Homilius , Rahul C. Deo , Calum A. MacRae This is my paper · ORCID
classification cs.LGcs.CL
keywords automaticdiagnosticcodingmedicaltextcontrastivelearningICD-10ICD-9MIMIC-IIIrepresentationraredisease
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that pre-training a medical-text model together with a model of ICD-10 diagnosis sequences, using a contrastive objective, improves automated diagnostic coding over prior state-of-the-art models. The authors evaluate on the MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full benchmarks and report gains across macro and micro AUC and F1, with the largest improvements on rare codes. They argue that data-driven representations of diagnosis codes, learned from a large real-world corpus of hospital encounters, capture structure that static ontology-based code distances miss. If true, this means that the routine pairing of clinical notes with their billed diagnostic codes is a strong pretraining signal for medical language understanding, and that contrastive alignment is a practical route to better code assignment.

What carries the argument

The central mechanism is the contrastive alignment of two encoders: a medical-text encoder for long notes and a diagnosis-code encoder for sequences of ICD-10 codes from the same clinical encounters. The diagnosis encoder is a transformer pretrained with a masked-language objective on millions of real-world hospital encounters, using relative position encodings to capture temporal proximity to the encounter of interest. During contrastive pre-training, the InfoNCE loss maximizes cosine similarity between the text embedding of a note and the code-sequence embedding of its associated codes, while minimizing similarity to other notes in the batch; the model also keeps a masked-language loss on the text side, with the two losses combined by uncertainty weighting. Prompt-based fine-tuning with textual descriptions of ICD-9 codes then adapts the aligned representations to the target coding tasks.

What would settle it

Run the same text encoder and prompt-based fine-tuning but replace the contrastively pretrained ICD-10 encoder with a static embedding derived from the ICD-10 hierarchy, and compare macro-F1 on MIMIC-III-rare50; if the rare-code advantage disappears, the claim that contrastive alignment of data-driven code sequences is the driver is falsified. A second check: replace the random ICD-9-to-ICD-10 mapping with a deterministic, frequency-weighted mapping and see whether the rare-code numbers move.

Watch

Extended reading notes

Core claim

The best NoteContrast model, which combines a long-document text encoder, a data-driven ICD-10 sequence encoder, and contrastive fine-tuning, reaches macro-AUC 93.8 and macro-F1 69.2 on MIMIC-III-50, and on MIMIC-III-rare50 it reaches macro-AUC 88.92 and macro-F1 40.26 when initialized from MIMIC-50 fine-tuning, versus 82.30 and 28.94 for the previous best prompt-based model under the same initialization. The authors attribute the gains to the contrastive pre-training step: aligning note representations with ICD-10 code-sequence representations pulls matching note-code pairs together and separates non-matching pairs, and this signal, combined with an auxiliary masked-language objective, produces embeddings that transfer to ICD-9 coding at fine-tuning time.

Load-bearing premise

The load-bearing premise is that the data-driven ICD-10 sequence encoder, pretrained with a vocabulary that omits some rare codes and connected to ICD-9 through a random mapping for ambiguous cases, transfers accurately to the MIMIC-III coding tasks; if those codes are missing or mistranslated, the reported rare-code improvements may come from encoder coverage rather than from the contrastive objective.

Editorial extensions

If this is right

  • Automated coding systems can be built from routinely available billing data: the note-code pairs used for contrastive pretraining require no manual annotation beyond the codes already assigned for billing.
  • Rare-disease coding, where ontologies are sparse and static distances mislead, is the regime that benefits most: macro-F1 on MIMIC-III-rare50 jumps by more than eleven points over the prior best model.
  • Longer document contexts matter: scaling the text encoder from 4096 to 8192 tokens further improves performance, suggesting that full discharge summaries carry coding-relevant information beyond the first few thousand tokens.
  • The pretraining recipe is portable: the same contrastive objective can be applied to any structured label with noisy correspondences to clinical text, without ontology engineering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The random selection among ambiguous ICD-9-to-ICD-10 mappings is a potential confound: a systematic mapping might change which codes the encoder sees and could either inflate or deflate rare-code scores; re-running with a deterministic mapping would isolate this effect.
  • Because the contrastive batch size is small (64) relative to large-scale multimodal pretraining, the InfoNCE objective may be under-sampled; larger batches could sharpen the rare-code alignment and further improve the tail.
  • The same architecture could be turned into a retrieval system: given a note, retrieve likely codes directly from the shared embedding space, which would make the model's coding decisions more inspectable than a flat classifier.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes NoteContrast, a contrastive pretraining method that aligns medical-note text with ICD-10 diagnostic-code sequences. The diagnostic-code encoder is a RoBERTa-style transformer trained on 7.5 million real-world sequences from MassGeneral Brigham, and the text encoder is a BioLM-based BigBird model pretrained on MIMIC-III notes; the two are jointly trained with an InfoNCE loss plus masked language modeling, followed by a short contrastive fine-tuning step using ICD-10 code descriptions. The authors evaluate on MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full diagnostic coding, reporting that NoteContrast outperforms prior state-of-the-art models such as KEPT, MSMN, ISD, TreeMAN, and DiscNet+RE on most metrics. The central claim is that aligning note text with a data-driven code-sequence encoder rather than ontology-based code distances is what drives the improvement.

Significance. If the results hold, the paper makes a useful contribution: it provides a clean recipe for contrastive language-diagnostic pretraining, with five-seed averages, dev-set threshold selection, explicit exclusion of MIMIC-III test patients from pretraining, and publicly available code. The NoteLM versus NoteContrast ablation does isolate the contrastive objective from the text-encoder architecture, and the gains on MIMIC-III-50 and especially MIMIC-III-rare50 are large. However, the mechanistic attribution to the data-driven ICD-10 encoder is partially confounded by encoder vocabulary coverage and by the random ICD-9-to-ICD-10 mapping, and the abstract overstates performance on the MIMIC-III-full task. The contribution is therefore significant but needs qualification and additional analysis before the paper's main attribution can be accepted.

major comments (3)
  1. [§3.5, §5.1, Table 2] The paper's central mechanistic claim—that aligning notes with a data-driven ICD-10 sequence encoder, rather than ontology-based code distances, drives the rare-code gains—is not fully supported by the current evidence. Section 5.1 states that 'certain rare codes are likely missing from our diagnostic code model' due to vocabulary cutoff, and Section 3.5 maps ICD-9 codes to ICD-10 with 'selected an ICD10 code at random' for ambiguous mappings. Since the rare50 labels are rare by construction, the macro-AUC/F1 improvements over KEPT (88.92 vs 82.30 and 40.26 vs 28.94 in Table 2) could reflect which rare labels happen to have a code-side token in the MassGeneral Brigham encoder vocabulary, rather than the contrastive alignment itself. The NoteLM versus NoteContrast ablation controls for the training objective but not for encoder pretraining data, vocabulary, or institution. Please report Table 2 results split by whether each label's mapped ICD-10 code is in the encoder vocabulary, and ideally compare with a code encoder trained without the real-world MGB sequences or with a vocabulary-based coverage control.
  2. [Abstract; §4.3, Table 3] The abstract's claim that the contrastive approach 'improves performance over prior state-of-the-art models for the MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full diagnostic coding tasks' is contradicted by the paper's own Table 3 for the full task: NoteContrast 8k ICD achieves macro-F1 11.9 ± 0.3, while DiscNet+RE achieves 14 and ISD achieves 11.90 ± 0.2. The body correctly says the model 'improved over the previous state-of-the-art method for most metrics,' but the abstract and conclusion are not so qualified. The claim should be restricted to micro-averaged metrics and precision/recall@K, or the macro-F1 deficit should be discussed explicitly.
  3. [Table 2, §4.3] The rare50 headline result is obtained from a NoteContrast 8k ICD model that was first fine-tuned on MIMIC-III-50, which the authors report 'performed better than fine-tuning the NoteContrast 8k ICD model on the MIMIC-III-rare50 dataset alone.' The paper does not state whether this initialization choice was made using the development set or the test set. If the choice was made after inspecting test performance, the reported rare50 gains are inflated by test-set model selection. Please clarify the selection rule and report both variants as pre-specified or with a dev-based selection criterion. Given the 142-note test set, per-label breakdowns and bootstrap confidence intervals would also help gauge stability.
minor comments (5)
  1. [Appendix A.2.2] '2,083,180 million de-identified notes' should read '2,083,180 de-identified notes'; the word 'million' is erroneous.
  2. [§3.5] Footnote 1's URL is run into the text after 'ICD-9 codes to ICD-10 codes 1.'; please add a space or restructure the sentence.
  3. [Throughout] The task names 'MIMIC-50', 'MIMIC-III-50', and 'MIMIC-50 rare' are used inconsistently; please define the three benchmarks once and use the same names throughout.
  4. [Tables 5–7] Table 5 groups four model variants under a single set of hyperparameters, but 'Maximum Sequence Length' differs by model; please state clearly that all other listed hyperparameters are identical across variants, or indicate exceptions in a footnote.
  5. [Figure 2] The UMAP axis labels appear as 'UMAP1' and 'UMAP2' without a separating space; also, the caption should state whether the 5000-code sample is fixed across the four panels so the Procrustes alignment is well defined.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NoteContrast's gains are measured on external MIMIC-III benchmarks, not derived from its own inputs.

full rationale

The paper's central claim is an empirical comparison: contrastive pretraining improves ICD coding performance on MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full. These benchmarks are external held-out splits created by Yang et al. (2022), and Appendix A.2.2 states that all patients appearing in any test split were removed from both MLM and contrastive pretraining sets. The reported metrics (macro/micro AUC, F1, P@K) are computed from model predictions against ground-truth labels; none is defined in terms of a fitted parameter or a self-cited construction. The contrastive objective (InfoNCE over text-code pairs) is a standard loss, not a theorem whose conclusion is the benchmark result. The prompt-based fine-tuning schema is adopted from Yang et al. (2022), a prior work with no identifiable overlapping authorship, and is used only as an evaluation scaffold. There are no self-citations in the reference list, and no uniqueness theorem is imported to force a particular model choice. Section 5.1's admitted vocabulary cutoff for rare ICD-10 codes is a real attribution confound for the rare50 improvement, and the random ICD-9-to-ICD-10 mapping (Section 3.5) could affect code coverage, but these are correctness or robustness concerns rather than circularity: the rare50 numbers are not equal to an input by construction. No step in the paper reduces, by equation or by self-reference, to its own input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on domain assumptions about label reliability, code-system mapping, and cross-institution transfer, all of which are acknowledged in the paper's own limitations section. There are no invented entities. The fitted quantities are standard training and threshold selections, all anchored to dev sets.

free parameters (3)
  • Uncertainty weighting for contrastive and MLM losses = learned during training (not reported)
    Section 3.3: 'The contrastive and masked language losses were combined and weighted using uncertainty weighting (Kendall et al., 2018)'; the learned weights are not disclosed.
  • Classification thresholds per task = chosen on dev set per task
    Section 4.2: 'The best thresholds for classification and computing precision, recall, and F1 were selected using the dev set for each task.'
  • Fine-tuning hyperparameters (learning rate, weight decay, max steps, batch size) = Tables 5, 6, 7
    Appendix A.3: 'we tuned the learning rate and weight decay using the dev set' for each downstream task.
assumptions (5)
  • standard math InfoNCE contrastive loss approximates maximization of mutual information between matched notes and code sequences.
    Section 3.3 and Section 6 rely on the Oord et al. (2018) interpretation; this is a standard, externally established result.
  • domain assumption MIMIC-III discharge summaries with their billing-derived ICD codes are reliable enough ground truth for coding evaluation.
    Section 4.1; the introduction itself notes coding labels have low sensitivity and specificity, so evaluation inherits label noise.
  • domain assumption Random resolution of ambiguous ICD-9 to ICD-10 mappings preserves enough diagnostic structure for pretraining.
    Section 3.5: 'To resolve ambiguous mappings, we selected an ICD10 code at random.' If this mapping is systematically wrong for rare codes, the encoder's rare-code representations are misaligned.
  • domain assumption Representations learned from MassGeneral Brigham ICD-10 encounter sequences transfer across institutions and across code systems (ICD-10 to ICD-9).
    Section 3.1 and Section 4: the encoder is trained on MGB data and evaluated on MIMIC-III ICD-9 tasks with no domain adaptation other than fine-tuning.
  • domain assumption BigBird length extension by copying BioLM position embeddings does not degrade the text representation.
    Section 3.2: 'we repeatedly copied over 512-position embeddings and pre-trained the model on longer text'; treated as standard practice following Beltagy et al. (2020).

how reviews work

0 comments
Cite this review

Pith. "Pith review of NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text." pith.science (2026). https://pith.science/paper/VJEMPQWZ

@misc{pith2026241211477,
  author       = {Pith},
  title        = {Pith review of: NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VJEMPQWZ}},
  note         = {Machine review of arXiv:2412.11477}
}
read the original abstract

Accurate diagnostic coding of medical notes is crucial for enhancing patient care, medical research, and error-free billing in healthcare organizations. Manual coding is a time-consuming task for providers, and diagnostic codes often exhibit low sensitivity and specificity, whereas the free text in medical notes can be a more precise description of a patients status. Thus, accurate automated diagnostic coding of medical notes has become critical for a learning healthcare system. Recent developments in long-document transformer architectures have enabled attention-based deep-learning models to adjudicate medical notes. In addition, contrastive loss functions have been used to jointly pre-train large language and image models with noisy labels. To further improve the automated adjudication of medical notes, we developed an approach based on i) models for ICD-10 diagnostic code sequences using a large real-world data set, ii) large language models for medical notes, and iii) contrastive pre-training to build an integrated model of both ICD-10 diagnostic codes and corresponding medical text. We demonstrate that a contrastive approach for pre-training improves performance over prior state-of-the-art models for the MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full diagnostic coding tasks.

Figures

Figures reproduced from arXiv: 2412.11477 by the authors.

Figure 1
Figure 1. A) The input to the diagnosis model is a sequence of ICD-10 codes, where positions and token type ids are relative to a “current encounter” (shown in red). B) The text encoder is a language model for long documents pre-trained on medical notes. C) Contrastive training of corresponding ICD-10 code sequences and medical note pairs. D) Pairs of ICD-10 codes are matched to textual descriptions of the codes as a fine-tun… view at source ↗
Figure 2
Figure 2. Comparing outputs of the ICD-10 diagnosis model and text model before and after contrastive pre-training (left and right panels, respectively). We used a sample of 5000 ICD-10 codes as input to the ICD-10 encoder model, and textual descriptions of each code as input to the text model. For visualization, 768-dimensional outputs were projected using UMAP into two dimen￾sions, scaled, and rotated using the Procrustes t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 14 canonical work pages

  1. [1]

    Publicly available clinical BERT embeddings

    Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72--78, Minneapolis, Minnesota, USA, 2019. Association for Computational Linguistics. doi:10.18653/v1/W19-1909. URL https://...

    doi:10.18653/v1/w19-1909 2019

  2. [2]

    S ci BERT : A pretrained language model for scientific text

    Iz Beltagy, Kyle Lo, and Arman Cohan. S ci BERT : A pretrained language model for scientific text. In Proc. of EMNLP, pages 3615--3620, Hong Kong, China, 2019. Association for Computational Linguistics. doi:10.18653/v1/D19-1371. URL https://aclanthology.org/D19-1371

    doi:10.18653/v1/d19-1371 2019

  3. [3]

    Longformer: The Long-Document Transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The Long-Document Transformer . arXiv, 2020. doi:10.48550/arxiv.2004.05150

    arXiv doi:10.48550/arxiv.2004.05150 2020

  4. [4]

    H yper C ore: Hyperbolic and co-graph representation for automatic ICD coding

    Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Shengping Liu, and Weifeng Chong. H yper C ore: Hyperbolic and co-graph representation for automatic ICD coding. In Proc. of ACL, pages 3105--3114, Online, 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.282. URL https://aclanthology.org/2020.acl-main.282

    doi:10.18653/v1/2020.acl-main.282 2020

  5. [5]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proc. of ICML, volume 119 of Proceedings of Machine Learning Research, pages 1597--1607. PMLR , 2020. URL http://proceedings.mlr.press/v119/chen20j.html

    work page 2020

  6. [6]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL-HLT, pages 4171--4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi:10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423

    doi:10.18653/v1/n19-1423 2019

  7. [7]

    PLM - ICD : Automatic ICD coding with pretrained language models

    Chao-Wei Huang, Shang-Chi Tsai, and Yun-Nung Chen. PLM - ICD : Automatic ICD coding with pretrained language models. In Proceedings of the 4th Clinical Natural Language Processing Workshop, pages 10--20, Seattle, WA, 2022. Association for Computational Linguistics. doi:10.18653/v1/2022.clinicalnlp-1.2. URL https://aclanthology.org/2022.clinicalnlp-1.2

    doi:10.18653/v1/2022.clinicalnlp-1.2 2022

  8. [8]

    Lungren, and Serena Yeung

    Shih - Cheng Huang, Liyue Shen, Matthew P. Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 3922--3931. IEEE , 2021. doi:10.1109/ICCV48922.20...

    arXiv 2021

Show all 45 references
  1. [9]

    Unsupervised Dense Information Retrieval with Contrastive Learning

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised Dense Information Retrieval with Contrastive Learning . arXiv, 2021. doi:10.48550/arxiv.2112.09118

    2021 doi arXiv

  2. [10]

    Dilated convolutional attention network for medical code assignment from clinical text

    Shaoxiong Ji, Erik Cambria, and Pekka Marttinen. Dilated convolutional attention network for medical code assignment from clinical text. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 73--78, Online, 2020. Association for Computational Linguisti...

    2020 doi

  3. [11]

    Le, Yun - Hsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi - Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun - Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang, editors, Proc. of ICML, vo...

    2021

  4. [12]

    Johnson, Tom J

    Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. MIMIC-III, a freely accessible critical care database . Scientific Data, 3: 0 sdata201635, 2016. ISSN 2052-...

    2016 doi

  5. [13]

    Multi-task learning using uncertainty to weigh losses for scene geometry and semantics

    Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 7482--7491. IE...

    2018

  6. [14]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, Proc. of ICLR, 2015. URL http://arxiv.org/abs/1412.6980

    2015 arXiv

  7. [15]

    BioBERT: a pre-trained biomedical language representation model for biomedical text mining

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: a pre-trained biomedical language representation model for biomedical text mining . Bioinformatics, 36 0 (4): 0 1234--1240, 2019. ISSN 1367-4803. doi:10.1093/bioinformatics/btz682

    2019 doi

  8. [16]

    Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art

    Patrick Lewis, Myle Ott, Jingfei Du, and Veselin Stoyanov. Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 146--157, Online, 2020. Ass...

    2020 doi

  9. [17]

    ICD coding from clinical text using multi-filter residual convolutional neural network

    Fei Li and Hong Yu. ICD coding from clinical text using multi-filter residual convolutional neural network. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, T...

    2020

  10. [18]

    BEHRT: Transformer for Electronic Health Records

    Yikuan Li, Shishir Rao, Jos \'e Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi-Khorshidi. BEHRT: Transformer for Electronic Health Records . Scientific Reports, 10 0 (1): 0 7155, 2020. doi:10.1038/s415...

    2020 doi

  11. [19]

    Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences

    Yikuan Li, Ramsey M Wehbe, Faraz S Ahmad, Hanyin Wang, and Yuan Luo. Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences . arXiv, 2022. doi:10.48550/arxiv.2201.11838

    2022 doi arXiv

  12. [20]

    RoBERTa: A Robustly Optimized BERT Pretraining Approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach . arXiv, 2019. doi:10.48550/arxiv.1907.11692

    2019 doi arXiv

  13. [21]

    T ree MAN : Tree-enhanced multimodal attention network for ICD coding

    Zichen Liu, Xuyuan Liu, Yanlong Wen, Guoqing Zhao, Fen Xia, and Xiaojie Yuan. T ree MAN : Tree-enhanced multimodal attention network for ICD coding. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3054--3063, Gyeongju, Republic of Korea,...

    2022

  14. [22]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proc. of ICLR. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  15. [23]

    ICDB ig B ird: A contextual embedding model for ICD code classification

    George Michalopoulos, Michal Malyska, Nicola Sahar, Alexander Wong, and Helen Chen. ICDB ig B ird: A contextual embedding model for ICD code classification. In Proceedings of the 21st Workshop on Biomedical Language Processing, pages 330--336, Dublin, Ireland, 2022. Associatio...

    2022 doi

  16. [24]

    Explainable prediction of medical codes from clinical text

    James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. Explainable prediction of medical codes from clinical text. In Proc. of NAACL-HLT, pages 1101--1111, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi:10.18653/v1/N18-1100...

    2018 doi

  17. [25]

    Joint Learning of Localized Representations from Medical Images and Reports

    Philip M \"u ller, Georgios Kaissis, Congyu Zou, and Daniel Rueckert. Joint Learning of Localized Representations from Medical Images and Reports . arXiv, 2021. doi:10.48550/arxiv.2112.02889

    2021 doi arXiv

  18. [26]

    Text and Code Embeddings by Contrastive Pre-Training

    Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski S...

    2022 doi arXiv

  19. [27]

    Representation Learning with Contrastive Predictive Coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding . arXiv, 2018. doi:10.48550/arxiv.1807.03748

    2018 doi arXiv

  20. [28]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K \" o pf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner...

    2019

  21. [29]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and ...

    2021

  22. [30]

    Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction

    Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction . npj Digital Medicine, 4 0 (1): 0 86, 2021. doi:10.1038/s41746-021-00455-y

    2021 doi

  23. [31]

    Pre-training of graph augmented transformers for medication recommendation

    Junyuan Shang, Tengfei Ma, Cao Xiao, and Jimeng Sun. Pre-training of graph augmented transformers for medication recommendation. In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August...

    2019 doi

  24. [32]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, ...

    2017

  25. [33]

    A label attention model for ICD coding from clinical text

    Thanh Vu, Dat Quoc Nguyen, and Anthony Nguyen. A label attention model for ICD coding from clinical text. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020 , pages 3335--3341. ijcai.org, 2020. d...

    2020 doi

  26. [34]

    Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays

    Xiaosong Wang, Ziyue Xu, Leo Tam, Dong Yang, and Daguang Xu. Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays . arXiv, 2021. doi:10.48550/arxiv.2103.16022

    2021 doi arXiv

  27. [35]

    M ed CLIP : Contrastive learning from unpaired medical images and text

    Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. M ed CLIP : Contrastive learning from unpaired medical images and text. In Proc. of EMNLP, pages 3876--3887, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. URL https://aclanthology.org...

    2022

  28. [36]

    Should you mask 15 \ In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2985--3000, Dubrovnik, Croatia, 2023

    Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15 \ In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2985--3000, Dubrovnik, Croatia, 2023. Association for Computational Linguistic...

    2023

  29. [37]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  30. [38]

    Yu, and Yangyong Zhu

    Xiancheng Xie, Yun Xiong, Philip S. Yu, and Yangyong Zhu. EHR coding with multi-scale feature attention and structured knowledge graph propagation. In Wenwu Zhu, Dacheng Tao, Xueqi Cheng, Peng Cui, Elke A. Rundensteiner, David Carmel, Qi He, and Jeffrey Xu Yu, editors, Proceed...

    2019

  31. [39]

    Knowledge injected prompt based fine-tuning for multi-label few-shot ICD coding

    Zhichao Yang, Shufan Wang, Bhanu Pratap Singh Rawat, Avijit Mitra, and Hong Yu. Knowledge injected prompt based fine-tuning for multi-label few-shot ICD coding. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1767--1781, Abu Dhabi, United Arab E...

    2022

  32. [40]

    Code synonyms do matter: Multiple synonyms matching network for automatic ICD coding

    Zheng Yuan, Chuanqi Tan, and Songfang Huang. Code synonyms do matter: Multiple synonyms matching network for automatic ICD coding. In Proc. of ACL, pages 808--814, Dublin, Ireland, 2022. Association for Computational Linguistics. doi:10.18653/v1/2022.acl-short.91. URL https://...

    2022 doi

  33. [41]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Onta \ n \' o n, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, M...

    2020

  34. [42]

    SCEHR: Supervised Contrastive Learning for Clinical Risk Prediction using Electronic Health Records

    Chengxi Zang and Fei Wang. SCEHR: Supervised Contrastive Learning for Clinical Risk Prediction using Electronic Health Records . 2021 IEEE International Conference on Data Mining (ICDM), 00: 0 857--866, 2021. ISSN 1550-4786. doi:10.1109/icdm51629.2021.00097

    2021

  35. [43]

    Automatic ICD coding exploiting discourse structure and reconciled code embeddings

    Shurui Zhang, Bozheng Zhang, Fuxin Zhang, Bo Sang, and Wanchun Yang. Automatic ICD coding exploiting discourse structure and reconciled code embeddings. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2883--2891, Gyeongju, Republic of Ko...

    2022

  36. [44]

    Contrastive Learning of Medical Visual Representations from Paired Images and Text

    Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive Learning of Medical Visual Representations from Paired Images and Text . Proceedings of the 7th Machine Learning for Healthcare Conference, 182: 0 2--25, 2020. URL https://procee...

    2020

  37. [45]

    Automatic ICD coding via interactive shared representation networks with self-distillation mechanism

    Tong Zhou, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Kun Niu, Weifeng Chong, and Shengping Liu. Automatic ICD coding via interactive shared representation networks with self-distillation mechanism. In Proc. of ACL, pages 5948--5957, Online, 2021. Association for Computationa...

    2021 doi

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.