Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

Multimodal Medical Code Tokenizer

T0 review · 5 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MedTok shows that tokenizing medical codes through both their textual definitions and their relational context improves EHR transformer models, with reported AUPRC gains of 4.10% on MIMIC-III, 4.78% on MIMIC-IV, and 11.32% on EHRShot.

desk verdict A genuinely useful engineering contribution with a new multimodal tokenizer for medical codes, but the abstract oversells the results and the evaluation leaves a capacity confound unresolved. read the letter →

arxiv 2502.04397 v3 pith:HMUPI2EQ submitted 2025-02-06 cs.CL cs.AIcs.LG

Xiaorui Su , Shvat Messica , Yepeng Huang , Ruth Johnson , Lukas Fesser , Shanghua Gao , Faryad Sahneh , Marinka Zitnik This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords multimodaltokenizationmedicalcodeselectronichealthrecordsvectorquantizationknowledgegraphclinicalpredictionquestionansweringtransformermodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that treating medical codes as isolated tokens is a bottleneck for electronic health record (EHR) foundation models. It introduces MedTok, a tokenizer that represents each code by its official text description and its neighborhood in a biomedical knowledge graph, then quantizes both views into a shared discrete codebook. Across five EHR transformer models and three datasets, replacing the standard tokenizer with MedTok improves AUPRC on every model and task, with reported gains of 4.10% on MIMIC-III, 4.78% on MIMIC-IV, and 11.32% on EHRShot, and the largest gains on drug recommendation. MedTok tokens also improve few-shot medical question answering when prepended to LLM prompts.

What carries the argument

The load-bearing object is a vector-quantized multimodal tokenizer: a codebook of N discrete token vectors divided into text-specific, graph-specific, and shared regions, fed by a frozen text encoder and a trainable graph encoder connected by cross-attention. Quantization selects the top-K nearest codebook vectors for each embedding and aggregates them with distance-weighted softmax, while three loss families (codebook loss, KL alignment of cross-modal distance distributions, and token-packing losses for shared and specific information) train the tokenizer. This machinery is what lets one tokenizer serve multiple coding systems and plug into any transformer-based model.

What would settle it

Retrain MedTok with the same losses but randomly permute the graph subgraphs and text descriptions among codes; if the AUPRC gains over the standard tokenizer persist, the semantic text-and-graph signal is not what carries the improvement.

Watch

Extended reading notes

Core claim

The central claim is that tokenization itself can be a learnable, multimodal step for structured EHR data. For each of 617,490 medical codes drawn from eight coding systems, MedTok encodes the code's textual definition with a language model encoder and its local subgraph from a biomedical knowledge graph with a graph encoder, then selects top-K entries from a codebook divided into text-specific, graph-specific, and shared regions. The resulting discrete tokens carry both modality-specific and cross-modality information, which the paper argues is why downstream models improve without any change to their architectures. The empirical claim is that a drop-in swap of the tokenizer improves AUPRC across five EHR models, with especially large gains on longitudinal outpatient data and on drug recommendation.

Load-bearing premise

The load-bearing premise is that the automatic mappings from medical codes to knowledge-graph nodes and the GPT-4-expanded text descriptions are accurate enough to be meaningful, because the paper does not report validation of mapping precision or description quality.

Editorial extensions

If this is right

  • A drop-in tokenizer swap, with no other change to a model's architecture or training, should transfer to transformer-based EHR models beyond the five tested.
  • Drug recommendation should see the largest benefit, because the relational graph encodes drug-disease associations that text-only tokens omit.
  • Longitudinal outpatient records should benefit at least as much as inpatient records, matching the EHRShot improvement and the sensitivity to larger codebooks.
  • Medical QA systems can consume MedTok tokens as structured prefix context to improve few-shot accuracy, a distinct use from EHR prediction.
  • A single tokenizer can span eight coding systems, reducing cross-system redundancy that standard tokenizers preserve as separate tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A random-permutation control would separate the semantic contribution of text and graph data from the extra capacity of MedTok's encoders; the paper does not report one.
  • Rare codes are the natural test case: if MedTok's value comes from shared semantics, its gains should be largest for low-frequency codes, which standard tokenizers represent poorly.
  • MedTok could become a shared clinical vocabulary across institutions that use different coding systems, letting pretrained models transfer without re-tokenizing.
  • Because EHRShot's longer visit histories show the largest gains, MedTok may matter most for longitudinal and few-shot settings rather than single-visit prediction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. MedTok is a vector-quantized tokenizer that maps each medical code to a sequence of tokens derived from two modalities: the code's textual description, encoded by a frozen language model, and a local subgraph from PrimeKG, encoded by a trainable graph encoder. The tokenizer uses modality-specific and shared codebook regions together with cross-attention and packing losses, and can be swapped into existing transformer-based EHR models. The paper evaluates MedTok by replacing standard code tokens in five EHR backbones (ETHOS, GT-BEHRT, MulT-EHR, TransformEHR, BEHRT) on MIMIC-III, MIMIC-IV, and EHRShot, reporting AUPRC improvements in Tables 3 and 4, and also shows accuracy gains when MedTok tokens are prepended to three LLMs for medical question answering. The central claim is that MedTok is a general-purpose multimodal medical code tokenizer whose text-and-graph tokens improve downstream EHR and QA performance without changing the downstream architecture.

Significance. The paper addresses a real bottleneck in EHR foundation models: medical vocabularies contain hundreds of thousands of codes whose relational and textual context is ignored by standard tokenizers. If the reported gains were attributable to the multimodal semantic information, MedTok would be a useful plug-in component for EHR models. The authors ship code, evaluate with five distinct backbones in both inpatient and outpatient settings, include modality ablations, and attempt an interpretability analysis. However, the current evidence does not separate the contribution of the multimodal tokenizer from added token count, added embedding capacity, and per-dataset hyperparameter tuning. The headline numbers are also per-task gains rather than dataset-level averages. The empirical claim is therefore plausible but not established at the level claimed in the abstract.

major comments (5)
  1. [Abstract; §4.1; Tables 3-4] The headline numbers in the Abstract and Contributions (4.10%, 4.78%, 11.32%) are not dataset-level average improvements. Section 4.1 reports average improvements of 3.29% on MIMIC-III and 2.67% on MIMIC-IV; the 4.10% and 4.78% figures are the DrugRec rows of Table 3. The 11.32% is the mortality row under Operational Outcomes in Table 4, while the average across the seven EHRShot columns is about 5.0%. Please report dataset-level averages explicitly and clearly label per-task gains.
  2. [§3.1; Appendix B.3] MedTok replaces a single code token with K tokens for each of four embeddings (text-specific, graph-specific, text-cross, graph-cross), so the downstream sequence is longer. Appendix B.3 states that for ETHOS the original vocabulary is extended with MedTok tokens, and for the other models MedTok embeddings initialize the original tokens, changing the embedding-table size. The value of K, the number of tokens per code, is never reported, and no baseline is matched for sequence length, token count, or embedding parameter count. Because larger token counts and embedding tables add capacity, the observed AUPRC gains cannot currently be attributed to the multimodal text-and-graph semantics; a capacity-matched baseline (e.g., K random or independently quantized tokens of the same dimension per code) is needed.
  3. [Tables 3-4] Many individual improvements are within one standard deviation of the baseline. Examples include TransformEHR LOS on MIMIC-IV (0.119 ± 0.001 vs. 0.121 ± 0.002), BEHRT MT on MIMIC-IV (0.028 ± 0.003 vs. 0.032 ± 0.006), and GT-BEHRT MT on MIMIC-III (0.160 ± 0.037 vs. 0.193 ± 0.046). No significance tests, confidence intervals, or effect sizes are reported. The statement that MedTok 'consistently improves performance across all five tasks' is stronger than the table supports.
  4. [Appendix A.1] The code-to-graph and code-to-text data are constructed automatically: UMLS/MONDO mapping, a custom entity linker, SciSpacy for semantic similarity, and GPT-4-expanded descriptions. No precision, recall, or manual validation of these mappings is reported, and GPT-4 expansion of sparse drug descriptions can introduce hallucinated clinical content. Because these are the sole inputs to MedTok's text and graph encoders, noisy or incorrect mappings are an alternative explanation for any performance differences and should be quantified, for example with a human-validated subset or a corruption/replacement ablation.
  5. [§3.2; §4.4] Equations (6)-(9) define objectives in terms of mutual information, but the implemented losses are InfoNCE and orthogonal-packing terms; the claimed equivalence is asserted ('Following Wang et al.') without proof or citation. In addition, the hyperparameters λ=β and codebook size N are tuned per dataset on the same benchmarks used for the main results: Figure 5 recommends λ=β=0.1 for inpatient and 0.01 for outpatient settings, and N=12,000 for MIMIC-III/IV but 24,000 for EHRShot. The main tables do not state which values were used, and no validation split is described to show that the reported numbers are not selected by this tuning.
minor comments (6)
  1. [Equation (4)] The weighting in Equation (4) uses -softmax(dist(...)), while §3.2's KL term uses softmax(-dist(...)); the sign convention should be made consistent.
  2. [§3.1; §4.4] The value of K and the sizes of the text-specific, graph-specific, and shared codebook regions are never reported; without them, the actual token counts per code and codebook utilization are unclear.
  3. [Table 3; Appendix C.5] The LOS task is multi-class (10 classes) but the table reports AUPRC, while Appendix C.5 says phenotype and drug recommendation use macro-averaged AUC-ROC; please specify how AUPRC is computed for multi-class and multi-label settings.
  4. [References; §3.2] The VQGraph reference appears multiple times with different years and venues (2023a, 2024a, 2024b), and the sentence 'Following Wang et al.' in §3.2 lacks a citation; please disambiguate.
  5. [Figure 5] The x-axis labels in Figure 5A run together ('0.010.1 0.200.300.4 0.5'), making the tick values illegible; please fix the figure.
  6. [Contributions] The Introduction states an EHRShot gain of 11.30% while the Abstract states 11.32%; please make these numbers consistent.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-by-construction circularity; the AUPRC gains are external empirical measurements, though per-dataset hyperparameter selection on the same benchmarks weakens the strength of the reported improvements.

full rationale

MedTok's central claim is an empirical comparison: swapping standard EHR tokenizers for MedTok changes AUPRC on MIMIC-III, MIMIC-IV, and EHRShot. The tokenizer is pre-trained with L = Lvq + LKL + Ltoken (Sections 3.1-3.3), and none of those loss terms contains a downstream label or task objective; the downstream EHR models are trained and evaluated on external benchmarks. There is no equation in the paper that makes the reported AUPRC equal to a tokenizer input or a fitted parameter by construction. The self-cited resources (PrimeKG, Chandak et al. 2023; MedDDx, Su et al. 2024) are data or benchmark artifacts rather than load-bearing theorems or fitted calibrations, so they do not create circularity. The main methodological caveat is that Section 4.4 selects lambda = beta and codebook size N by inspecting average AUPRC on the same three datasets and tasks that Tables 3-4 later report, with no validation split documented; this is test-set selection that can optimistically bias the gains, but it is not a circular reduction of the tokenizer's output into the evaluation metric. The unstated number of tokens per code (K) and the larger embedding tables used in MedTok+ configurations are a capacity confound for causal attribution of the improvements, again a validity concern rather than circularity. Overall, the derivation chain is not self-referential enough to score above the 0-2 range.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method is an empirical engineering contribution; it does not derive new constants or entities. The main costs are the per-dataset hyperparameters (codebook size, loss weights) and the unvalidated data-generation steps (code mapping, GPT-4 descriptions). The information-theoretic packing losses are borrowed from prior work without proof of equivalence.

free parameters (4)
  • Codebook size N = 12,000 (MIMIC-III/IV), 24,000 (EHRShot)
    Selected by the sweep in Figure 5B to maximize average AUPRC on each dataset; headline results use these per-dataset values.
  • Loss weight coefficients lambda = beta = 0.1 (in-patient), 0.01 (out-patient)
    Recommended in Section 4.4 after tuning on the same evaluation datasets; authors constrain lambda = beta to halve the search.
  • Number of tokens per code K = Not stated in main text
    The top-K quantization in Eq. 3 is a design choice that is not reported in the paper, blocking exact re-implementation.
  • Quantized embedding dimension d = 64
    Given in Appendix B.2 as a hand-chosen setting; no sensitivity analysis is reported for this value.
assumptions (4)
  • domain assumption PrimeKG, UMLS, MONDO and the custom entity/NLP linkers align the 617,490 codes to meaningful knowledge-graph nodes.
    Appendix A.1.1; if alignment fails for rare codes, the graph modality injects spurious structure into the tokens.
  • domain assumption GPT-4 enriched text descriptions are clinically accurate and faithful to the official definitions.
    Appendix A.1.2; no human or automated validation of the enriched descriptions is reported.
  • ad hoc to paper The information-theoretic objectives in Equations 6-9 are equivalent to the implemented InfoNCE and orthogonal packing losses.
    Section 3.2 states this 'Following Wang et al.' but does not derive the equivalence or prove the optimal solutions.
  • domain assumption Freezing the text encoder while training the graph encoder is a sufficient training scheme.
    Appendix B.2; no ablation tests this choice, which could hide a failure mode where the text encoder needs adaptation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Medical Code Tokenizer." pith.science (2026). https://pith.science/paper/HMUPI2EQ

@misc{pith2026250204397,
  author       = {Pith},
  title        = {Pith review of: Multimodal Medical Code Tokenizer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HMUPI2EQ}},
  note         = {Machine review of arXiv:2502.04397}
}
read the original abstract

Foundation models trained on patient electronic health records (EHRs) require tokenizing medical data into sequences of discrete vocabulary items. Existing tokenizers treat medical codes from EHRs as isolated textual tokens. However, each medical code is defined by its textual description, its position in ontological hierarchies, and its relationships to other codes, such as disease co-occurrences and drug-treatment associations. Medical vocabularies contain more than 600,000 codes with critical information for clinical reasoning. We introduce MedTok, a multimodal medical code tokenizer that uses the text descriptions and relational context of codes. MedTok processes text using a language model encoder and encodes the relational structure with a graph encoder. It then quantizes both modalities into a unified token space, preserving modality-specific and cross-modality information. We integrate MedTok into five EHR models and evaluate it on operational and clinical tasks across in-patient and out-patient datasets, including outcome prediction, diagnosis classification, drug recommendation, and risk stratification. Swapping standard EHR tokenizers with MedTok improves AUPRC across all EHR models, by 4.10% on MIMIC-III, 4.78% on MIMIC-IV, and 11.32% on EHRShot, with the largest gains in drug recommendation. Beyond EHR modeling, we demonstrate using MedTok tokenizer with medical QA systems. Our results demonstrate the potential of MedTok as a unified tokenizer for medical codes, improving tokenization for medical foundation models.

Figures

Figures reproduced from arXiv: 2502.04397 by the authors.

Figure 1
Figure 1. MEDTOK is a multimodal tokenizer of medical codes that combines text descriptions of codes with relational represen￾tation of dependencies between codes in clinical ontologies and medical terminologies. MEDTOK is a general-purpose tokenizer that can be integrated into any transformer-based model or system that requires tokenization. 1. Introduction Electronic health records (EHRs) are the backbone of mod￾ern healthc… view at source ↗
Figure 2
Figure 2. MEDTOK is a general multimodal tokenizer of medical codes that can be integrated into any transformer-based model or a system that requires tokenization. ‘X-attn’ denotes a cross-attention module. 3.1. Multimodal tokenization Given a medical code m, paired with its description t and its biological subgraph G, MEDTOK first adopts the text en￾coder, denoted as Et and the graph encoder, denoted as Eg, to generate two e… view at source ↗
Figure 3
Figure 3. The AUPRC values of three types of tokenizers on in￾patient and out-patient datasets, where OO means Operational Outcomes and ND means assignment of new diagnoses. 4.2. MEDTOK tokenizer with out-patient EHR models [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The AUPRC values obtained by removing the text and graph modalities across all tasks on two in-patient datasets and one out-patient dataset. Optimization MIMIC III MIMIC IV EHRShot Lvq 0.373 0.387 0.287 Lvq + L c token + LKL 0.379 0.409 0.314 Lvq + L s token 0.382 0.40…
Figure 5
Figure 5. Figure 5: B presents the results for various codebook sizes across all tasks on the three datasets. The performance trends observed on MIMIC-III and MIMIC-IV are quite con￾sistent, demonstrating a clear pattern where increasing the codebook size enhances the model’s performance.…
Figure 6
Figure 6. Figure 6: The accuracy of LLMs vs. MEDTOK+LLMs on three medical QA datasets. kens, which provide structured medical context before the main input, allowing the LLM to incorporate medical codes. For this evaluation, we use three medical QA datasets, in￾cluding MMLU (Hendrycks et …
Figure 7
Figure 7. Figure 7: Top 100 frequent token IDs appearing in patients at high risk of Hyperlipidemia. 5. Conclusion Tokenizing medical codes is a critical yet challenging step in developing foundation models for EHRs. Existing tok￾enizers treat medical codes as isolated textual units, fail…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FoMoH benchmarks six structured EHR foundation models on 14 tasks and finds they do not consistently outperform supervised baselines, particularly for rare diseases and low-data regimes.

  2. The Latent Space Hypothesis: Toward Universal Medical Representation Learning

    q-bio.QM 2025-06 conditional novelty 4.0 of 10

    The paper argues that all medical data modalities encode projections of a single latent physiological state, so a universal learned geometry could unify diagnosis, monitoring, and treatment.

Reference graph

Works this paper leans on

77 extracted references · 31 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Cosmos world foundation model platform for physical ai

    Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E., Cai, T., Chattopadhyay, P., Chen, Y., Cui, Y., Ding, Y., et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025

    arXiv 2025

  3. [3]

    F., Mullin, R

    Averill, R. F., Mullin, R. L., Steinbeck, B. A., Goldfield, N. I., and Grant, T. M. Development of the icd-10 procedure coding system (icd-10-pcs). Topics in health information management, 21 0 (3): 0 54--88, 2001

    work page 2001

  4. [4]

    vq-wav2vec: Self-supervised learning of discrete speech representations

    Baevski, A., Schneider, S., and Auli, M. vq-wav2vec: Self-supervised learning of discrete speech representations. arXiv preprint arXiv:1910.05453, 2019

    arXiv 1910

  5. [5]

    H., Carbon, S., et al

    Balsa-Canto, E., Brush, M. H., Carbon, S., et al. Mondo: Unifying diseases for the world, by the world. Nucleic Acids Research, 51 0 (D1): 0 D1182--D1190, 2023. doi:10.1093/nar/gkac1062

    doi:10.1093/nar/gkac1062 2023

  6. [6]

    The unified medical language system (umls): integrating biomedical terminology

    Bodenreider, O. The unified medical language system (umls): integrating biomedical terminology. Nucleic Acids Research, 32 0 (Database issue): 0 D267--D270, 2004. doi:10.1093/nar/gkh061

    doi:10.1093/nar/gkh061 2004

  7. [7]

    H., Yin, G., Bae, K., and Yu, L

    Chan, T. H., Yin, G., Bae, K., and Yu, L. Multi-task heterogeneous graph learning on electronic health records. Neural Networks, 180: 0 106644, 2024

    work page 2024

  8. [8]

    Building a knowledge graph to enable precision medicine

    Chandak, P., Huang, K., and Zitnik, M. Building a knowledge graph to enable precision medicine. Scientific Data, 10 0 (1): 0 67, 2023

    work page 2023

Show all 77 references
  1. [9]

    M., and Jeni, L

    Choudhury, R., Zhu, G., Liu, S., Niinuma, K., Kitani, K. M., and Jeni, L. Don’t look twice: Faster video transformers with run-length tokenization. 0 (arXiv:2411.05222), November 2024. doi:10.48550/arXiv.2411.05222. URL http://arxiv.org/abs/2411.05222. arXiv:2411.05222

    2024 doi arXiv

  2. [10]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805

    2018 arXiv

  3. [11]

    Donnelly, K. et al. Snomed-ct: The advanced terminology and coding system for ehealth. Studies in health technology and informatics, 121: 0 279, 2006

    2006

  4. [12]

    Cpt codes: what are they, why are they necessary, and how are they developed?, 2013

    Dotson, P. Cpt codes: what are they, why are they necessary, and how are they developed?, 2013

    2013

  5. [13]

    On the role of discrete tokenization in visual representation learning

    Du, T., Wang, Y., and Wang, Y. On the role of discrete tokenization in visual representation learning. arXiv preprint arXiv:2407.09087, 2024

    2024 arXiv

  6. [14]

    The llama 3 herd of models, 2024

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., and Fan, A. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  7. [15]

    M., Daley, J., Hughes, J., Fisher, E

    Foley, S. M., Daley, J., Hughes, J., Fisher, E. S., Heeren, T., et al. Comorbidities, complications, and coding bias: does the number of diagnosis codes matter in predicting in-hospital mortality? Jama, 267 0 (16): 0 2197--2203, 1992

    1992

  8. [16]

    Fragment and geometry aware tokenization of molecules for structure-based drug design using language models

    Fu, C., Li, X., Olson, B., Ji, H., and Ji, S. Fragment and geometry aware tokenization of molecules for structure-based drug design using language models. 0 (arXiv:2408.09730), August 2024. doi:10.48550/arXiv.2408.09730. URL http://arxiv.org/abs/2408.09730. arXiv:2408.09730

    2024 doi arXiv

  9. [17]

    W., Xu, J., and Bodenreider, O

    Fung, K. W., Xu, J., and Bodenreider, O. The new international classification of diseases 11th edition: a comparative analysis with icd-10 and icd-10-cm. Journal of the American Medical Informatics Association, 27 0 (5): 0 738--746, 2020

    2020

  10. [18]

    A., Navar, A

    Goldstein, B. A., Navar, A. M., and Pencina, M. J. Risk prediction with electronic health records: The importance of model validation and clinical context. JAMA Cardiology, 1 0 (9): 0 976, December 2016. ISSN 2380-6583. doi:10.1001/jamacardio.2016.3826

    2016

  11. [19]

    Gu, Y., Wang, X., Ge, Y., Shan, Y., and Shou, M. Z. Rethinking the objectives of vector-quantized tokenizers for image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7631--7640, 2024

    2024

  12. [20]

    C., Ver Steeg, G., and Galstyan, A

    Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., and Galstyan, A. Multitask learning and benchmarking with clinical time series data. Scientific data, 6 0 (1): 0 96, 2019

    2019

  13. [21]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  14. [22]

    A., Zhang, X., Zappia, L., Knoll, R., Lang, N

    Heumos, L., Ehmele, P., Treis, T., Upmeier Zu Belzen, J., Roellin, E., May, L., Namsaraeva, A., Horlava, N., Shitov, V. A., Zhang, X., Zappia, L., Knoll, R., Lang, N. J., Hetzel, L., Virshup, I., Sikkema, L., Curion, F., Eils, R., Schiller, H. B., Hilgendorff, A., and Theis, F...

    2024 doi

  15. [23]

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  16. [24]

    Jensen, K., Soguero-Ruiz, C., Oyvind Mikalsen, K., Lindsetmo, R.-O., Kouskoumvekaki, I., Girolami, M., Olav Skrovseth, S., and Augestad, K. M. Analysis of free text in electronic health records for identification of cancer patient trajectories. Scientific Reports, 7 0 (1): 0 4...

    2017 doi

  17. [25]

    Y., Liu, X

    Jiang, L. Y., Liu, X. C., Nejatian, N. P., Nasir-Moin, M., Wang, D., Abidin, A., Eaton, K., Riina, H. A., Laufer, I., Punjabi, P., et al. Health system-scale language models are all-purpose prediction engines. Nature, 619 0 (7969): 0 357--362, 2023 a

    2023

  18. [26]

    Graphcare: Enhancing healthcare predictions with personalized knowledge graphs

    Jiang, P., Xiao, C., Cross, A., and Sun, J. Graphcare: Enhancing healthcare predictions with personalized knowledge graphs. arXiv preprint arXiv:2305.12788, 2023 b

    2023 arXiv

  19. [27]

    MIMIC-III Clinical Database (version 1.4), 2016

    Johnson, A., Pollard, T., and Mark, R. MIMIC-III Clinical Database (version 1.4), 2016. URL https://doi.org/10.13026/C2XW26. Available at: https://doi.org/10.13026/C2XW26

    2016 doi

  20. [28]

    A., and Mark, R

    Johnson, A., Bulgarelli, L., Pollard, T., Gow, B., Moody, B., Horng, S., Celi, L. A., and Mark, R. MIMIC-IV (version 3.1), 2024. URL https://doi.org/10.13026/kpb9-mt58

    2024 doi

  21. [29]

    Digital twins for health: a scoping review

    Katsoulakis, E., Wang, Q., Wu, H., Shahriyari, L., Fletcher, R., Liu, J., Achenie, L., Liu, H., Jackson, P., Xiao, Y., Syeda-Mahmood, T., Tuli, R., and Deng, J. Digital twins for health: a scoping review. npj Digital Medicine, 7 0 (1): 0 77, March 2024. ISSN 2398-6352. doi:10....

    2024 doi

  22. [30]

    A., Deng, A., Balston, A., Ross, J., Idowu, E., Teo, J

    Kraljevic, Z., Bean, D., Shek, A., Bendayan, R., Hemingway, H., Yeung, J. A., Deng, A., Balston, A., Ross, J., Idowu, E., Teo, J. T., and Dobson, R. J. B. Foresight—a generative pretrained transformer for modelling of patient timelines using electronic health records: a retros...

    2024 doi

  23. [31]

    and Richardson, J

    Kudo, T. and Richardson, J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. 0 (arXiv:1808.06226), August 2018. doi:10.48550/arXiv.1808.06226. URL http://arxiv.org/abs/1808.06226. arXiv:1808.06226

    2018 doi arXiv

  24. [32]

    Li, Y., Rao, S., Solares, J. R. A., Hassaine, A., Ramakrishnan, R., Canoy, D., Zhu, Y., Rahimi, K., and Salimi-Khorshidi, G. Behrt: transformer for electronic health records. Scientific reports, 10 0 (1): 0 7155, 2020

    2020

  25. [33]

    and Britt, H

    Miller, G. and Britt, H. A new drug classification for computer systems: the atc extension code. International journal of bio-medical computing, 40 0 (2): 0 121--124, 1995

    1995

  26. [34]

    M., and Vulić, I

    Minixhofer, B., Ponti, E. M., and Vulić, I. Zero-shot tokenizer transfer. 0 (arXiv:2405.07883), May 2024. doi:10.48550/arXiv.2405.07883. URL http://arxiv.org/abs/2405.07883. arXiv:2405.07883

    2024 doi

  27. [35]

    J., Zeng, K., Kilbourne, J., Powell, T., and Moore, R

    Nelson, S. J., Zeng, K., Kilbourne, J., Powell, T., and Moore, R. Normalized names for clinical drugs: Rxnorm at 6 years. Journal of the American Medical Informatics Association, 18 0 (4): 0 441--448, 2011

    2011

  28. [36]

    Afrimed-qa: A pan-african, multi-specialty, medical question-answering benchmark dataset

    Olatunji, T., Nimo, C., Owodunni, A., Abdullahi, T., Ayodele, E., Sanni, M., Aka, C., Omofoye, F., Yuehgoh, F., Faniran, T., et al. Afrimed-qa: A pan-african, multi-specialty, medical question-answering benchmark dataset. arXiv preprint arXiv:2411.15640, 2024

    2024

  29. [37]

    Organization, W. H. International Statistical Classification of Diseases and related health problems: Alphabetical index, volume 3. World Health Organization, 2004

    2004

  30. [38]

    Organization, W. H. et al. International classification of diseases—ninth revision (icd-9). Weekly Epidemiological Record= Relev \'e \'e pid \'e miologique hebdomadaire , 63 0 (45): 0 343--344, 1988

    1988

  31. [39]

    K., and Sankarasubbu, M

    Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pp.\ 248--260. PMLR, 2022

    2022

  32. [40]

    What is ndc? Pension reform: Issues and prospects for non-financial defined contribution (NDC) schemes, pp.\ 17--34, 2006

    Palmer, E. What is ndc? Pension reform: Issues and prospects for non-financial defined contribution (NDC) schemes, pp.\ 17--34, 2006

    2006

  33. [41]

    Let your graph do the talking: Encoding structured data for llms

    Perozzi, B., Fatemi, B., Zelle, D., Tsitsulin, A., Kazemi, M., Al-Rfou, R., and Halcrow, J. Let your graph do the talking: Encoding structured data for llms. 0 (arXiv:2402.05862), February 2024. doi:10.48550/arXiv.2402.05862. URL http://arxiv.org/abs/2402.05862. arXiv:2402.05862

    2024 doi arXiv

  34. [42]

    and Beheshti, R

    Poulain, R. and Beheshti, R. Graph transformers on ehrs: Better representation improves downstream performance. In The Twelfth International Conference on Learning Representations, 2024

    2024

  35. [43]

    Model decides how to tokenize: Adaptive dna sequence tokenization with mxdna

    Qiao, L., Ye, P., Ren, Y., Bai, W., Liang, C., Ma, X., Dong, N., and Ouyang, W. Model decides how to tokenize: Adaptive dna sequence tokenization with mxdna. 0 (arXiv:2412.13716), December 2024. doi:10.48550/arXiv.2412.13716. URL http://arxiv.org/abs/2412.13716. arXiv:2412.13716

    2024 doi arXiv

  36. [44]

    Towards building multilingual language model for medicine

    Qiu, P., Wu, C., Zhang, X., Lin, W., Wang, H., Zhang, Y., Wang, Y., and Xie, W. Towards building multilingual language model for medicine. Nature Communications, 15 0 (1): 0 8384, 2024

    2024

  37. [45]

    K., Yuan, Z., and Wu, X

    Qu, L., Zhang, H., Liu, Y., Wang, X., Jiang, Y., Gao, Y., Ye, H., Du, D. K., Yuan, Z., and Wu, X. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069, 2024

    2024 arXiv

  38. [46]

    Recommender systems with generative retrieval

    Rajput, S., Mehta, N., Singh, A., Hulikal Keshavan, R., Vu, T., Heldt, L., Hong, L., Tay, Y., Tran, V., Samost, J., et al. Recommender systems with generative retrieval. Advances in Neural Information Processing Systems, 36: 0 10299--10315, 2023

    2023

  39. [47]

    E., Was, J., Li, Q., Bates, D

    Renc, P., Jia, Y., Samir, A. E., Was, J., Li, Q., Bates, D. W., and Sitek, A. Zero shot health trajectory prediction using transformer. NPJ Digital Medicine, 7 0 (1): 0 256, 2024

    2024

  40. [48]

    Neural machine translation of rare words with subword units

    Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. June 2016. doi:10.48550/arXiv.1508.07909. URL http://arxiv.org/abs/1508.07909

    2016 doi arXiv

  41. [49]

    S., Wei, J., Chung, H

    Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. Large language models encode clinical knowledge. Nature, 620 0 (7972): 0 172--180, 2023

    2023

  42. [50]

    R., Cole-Lewis, H., et al

    Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. Toward expert-level medical question answering with large language models. Nature Medicine, pp.\ 1--8, 2025

    2025

  43. [51]

    Fast wordpiece tokenization

    Song, X., Salcianu, A., Song, Y., Dopson, D., and Zhou, D. Fast wordpiece tokenization. 0 (arXiv:2012.15524), October 2021. doi:10.48550/arXiv.2012.15524. URL http://arxiv.org/abs/2012.15524. arXiv:2012.15524

    2012 doi arXiv

  44. [52]

    Knowledge graph based agent for complex, knowledge-intensive qa in medicine

    Su, X., Wang, Y., Gao, S., Liu, X., Giunchiglia, V., Clevert, D.-A., and Zitnik, M. Knowledge graph based agent for complex, knowledge-intensive qa in medicine. arXiv preprint arXiv:2410.04660, 2024

    2024 arXiv

  45. [53]

    Learning to tokenize for generative retrieval

    Sun, W., Yan, L., Chen, Z., Wang, S., Zhu, H., Ren, P., Chen, Z., Yin, D., Rijke, M., and Ren, Z. Learning to tokenize for generative retrieval. Advances in Neural Information Processing Systems, 36, 2024

    2024

  46. [54]

    T., Shahgir, H

    Tahmid, M. T., Shahgir, H. S., Mahbub, S., Dong, Y., and Bayzid, M. S. Birna-bert allows efficient rna language modeling with adaptive tokenization. November 2024. doi:10.1101/2024.07.02.601703. URL https://www.biorxiv.org/content/10.1101/2024.07.02.601703v3

    2024 doi

  47. [55]

    Towards generalist biomedical ai

    Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. Towards generalist biomedical ai. NEJM AI, 1 0 (3): 0 AIoa2300138, 2024

    2024

  48. [56]

    Neural discrete representation learning

    Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017

    2017

  49. [57]

    An information criterion for controlled disentanglement of multimodal data

    Wang, C., Gupta, S., Zhang, X., Tonekaboni, S., Jegelka, S., Jaakkola, T., and Uhler, C. An information criterion for controlled disentanglement of multimodal data. arXiv preprint arXiv:2410.23996, 2024 a

    2024 arXiv

  50. [58]

    Tokenization matters! degrading large language models through challenging their tokenization

    Wang, D., Li, Y., Jiang, J., Ding, Z., Jiang, G., Liang, J., and Yang, D. Tokenization matters! degrading large language models through challenging their tokenization. 0 (arXiv:2405.17067), May 2024 b . doi:10.48550/arXiv.2405.17067. URL http://arxiv.org/abs/2405.17067. arXiv:...

    2024 doi arXiv

  51. [59]

    Learning graph quantized tokenizers for transformers

    Wang, L., Hassani, K., Zhang, S., Fu, D., Yuan, B., Cong, W., Hua, Z., Wu, H., Yao, N., and Long, B. Learning graph quantized tokenizers for transformers. arXiv preprint arXiv:2410.13798, 2024 c

    2024 arXiv

  52. [60]

    Learnable item tokenization for generative recommendation

    Wang, W., Bao, H., Lin, X., Zhang, J., Li, Y., Feng, F., Ng, S.-K., and Chua, T.-S. Learnable item tokenization for generative recommendation. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp.\ 2400--2409, 2024 d

    2024

  53. [61]

    S., Feunang, Y

    Wishart, D. S., Feunang, Y. D., Guo, A. C., Lo, E. J., Marcu, A., Grant, J. R., Sajed, T., Johnson, D., Li, C., Sayeeda, Z., et al. Drugbank 5.0: a major update to the drugbank database for 2018. Nucleic Acids Research, 46 0 (D1): 0 D1074--D1082, 2018. doi:10.1093/nar/gkx1037

    2018 doi

  54. [62]

    A., and Shah, N

    Wornow, M., Thapa, R., Steinberg, E., Fries, J. A., and Shah, N. EHRSHOT : An EHR benchmark for few-shot evaluation of foundation models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    2023

  55. [63]

    D., Ho, J., and Yang, C

    Xu, R., Shi, W., Yu, Y., Zhuang, Y., Jin, B., Wang, M. D., Ho, J., and Yang, C. RAM - EHR : Retrieval augmentation meets clinical predictions on electronic health records. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Associa...

    2024 doi

  56. [64]

    Vqgraph: Rethinking graph representation space for bridging gnns and mlps

    Yang, L., Tian, Y., Xu, M., Liu, Z., Hong, S., Qu, W., Zhang, W., Cui, B., Zhang, M., and Leskovec, J. Vqgraph: Rethinking graph representation space for bridging gnns and mlps. arXiv preprint arXiv:2308.02117, 2023 a

    2023 arXiv

  57. [65]

    Vqgraph: Rethinking graph representation space for bridging gnns and mlps

    Yang, L., Tian, Y., Xu, M., Liu, Z., Hong, S., Qu, W., Zhang, W., Cui, B., Zhang, M., and Leskovec, J. Vqgraph: Rethinking graph representation space for bridging gnns and mlps. 0 (arXiv:2308.02117), March 2024 a . doi:10.48550/arXiv.2308.02117. URL http://arxiv.org/abs/2308.0...

    2024 doi arXiv

  58. [66]

    Vqgraph: Rethinking graph representation space for bridging gnns and mlps

    Yang, L., Tian, Y., Xu, M., Liu, Z., Hong, S., Qu, W., Zhang, W., CUI, B., Zhang, M., and Leskovec, J. Vqgraph: Rethinking graph representation space for bridging gnns and mlps. In International Conference on Learning Representations, 2024 b

    2024

  59. [67]

    Transformehr: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records

    Yang, Z., Mitra, A., Liu, W., Berlowitz, D., and Yu, H. Transformehr: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records. Nature communications, 14 0 (1): 0 7857, 2023 b

    2023

  60. [68]

    Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y

    Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021

    2021 arXiv

  61. [69]

    Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y

    Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. Vector-quantized image modeling with improved vqgan. 0 (arXiv:2110.04627), June 2022. doi:10.48550/arXiv.2110.04627. URL http://arxiv.org/abs/2110.04627. arXiv:2110.04627

    2022 doi arXiv

  62. [70]

    B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Birodkar, V., Gupta, A., Gu, X., et al

    Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Birodkar, V., Gupta, A., Gu, X., et al. Language model beats diffusion--tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023

    arXiv

  63. [71]

    B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Birodkar, V., Gupta, A., Gu, X., Hauptmann, A

    Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Birodkar, V., Gupta, A., Gu, X., Hauptmann, A. G., Gong, B., Yang, M.-H., Essa, I., Ross, D. A., and Jiang, L. Language model beats diffusion -- tokenizer is key to visual generation. 0 (arXi...

    doi arXiv

  64. [72]

    Predict and interpret health risk using ehr through typical patients

    Yu, Z., Zhang, C., Wang, Y., Tang, W., Wang, J., and Ma, L. Predict and interpret health risk using ehr through typical patients. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1506--1510, 2024 b . doi:10.1109/ICAS...

    2024

  65. [73]

    Soundstream: An end-to-end neural audio codec

    Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 0 495--507, 2021

    2021

  66. [74]

    A., Schmid, C., Katabi, D., and Gu, X

    Zha, K., Yu, L., Fathi, A., Ross, D. A., Schmid, C., Katabi, D., and Gu, X. Language-guided image tokenization for generation. 0 (arXiv:2412.05796), December 2024. doi:10.48550/arXiv.2412.05796. URL http://arxiv.org/abs/2412.05796. arXiv:2412.05796

    2024 doi arXiv

  67. [75]

    Regularized vector quantization for tokenized image synthesis

    Zhang, J., Zhan, F., Theobalt, C., and Lu, S. Regularized vector quantization for tokenized image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 18467--18476, 2023

    2023

  68. [76]

    ibot: Image bert pre-training with online tokenizer

    Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. ibot: Image bert pre-training with online tokenizer. January 2022. doi:10.48550/arXiv.2111.07832. URL http://arxiv.org/abs/2111.07832

    2022 doi arXiv

  69. [77]

    Emerge: Enhancing multimodal electronic health records predictive modeling with retrieval-augmented generation

    Zhu, Y., Ren, C., Wang, Z., Zheng, X., Xie, S., Feng, J., Zhu, X., Li, Z., Ma, L., and Pan, C. Emerge: Enhancing multimodal electronic health records predictive modeling with retrieval-augmented generation. In Proceedings of the 33rd ACM International Conference on Information...

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.