{"id":"02ab6650-3d2a-4d34-ad8f-c9917c686f94","arxiv_id":"2502.04397","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MedTok encodes medical codes with text and graph information into a shared vector-quantized token space, improving downstream EHR prediction and medical QA when swapped in for standard tokenizers.","lead":"MedTok is a new tokenizer that turns medical codes from electronic health records into tokens using both the code's text description and its relationships in medical knowledge graphs. It reports consistent improvements in EHR prediction tasks when plugged into five existing models, though the headline gains are the best single-task numbers rather than dataset averages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported AUPRC gains may reflect MedTok's added token count and embedding capacity rather than its multimodal semantics; the abstract also presents per-task improvements as dataset-level averages.","rationale":"The reader's weakest assumption concerns the quality of the automatically built text and graph data (UMLS/MONDO/PrimeKG mappings and GPT-4 enriched descriptions). That is a genuine concern: if the input modalities are noisy, the tokenizer's semantic premise is weakened. However, an even more load-bearing threat is that the headline comparison itself does not control for token count or model capacity. MedTok replaces one code token with multiple quantized tokens (Eq. 3 and token packing in Section 3.2), yet no baseline is matched for sequence length, embedding dimension, or parameter count. This confound threatens the empirical claim directly, independent of whether the text and graph data are accurate. The paper does provide some independent support: consistent positive results across all five EHR models and released code, and the ablation in Figure 4 suggests both modalities contribute when the tokenizer architecture is otherwise fixed. But those ablations do not control for the baseline token-count mismatch, so they do not resolve the capacity issue. In addition, the abstract presents the per-task DrugRec improvements (4.10% and 4.78%) and the EHRShot mortality improvement (11.32%) as dataset-level averages, which misstates the magnitude of the general claim. Because the concern is substantial but not yet disproven, the appropriate verdict remains CONDITIONAL, matching the reader's verdict; the condition should explicitly require a capacity-matched control and corrected reporting of aggregate numbers.","tokens_in":21963,"tokens_out":6096,"duration_ms":62934,"concrete_test":"Run a capacity-matched control on MIMIC-III: keep MedTok's tokenizer output format (same K tokens per code, same sequence length, same codebook and embedding dimensions), but train it on randomly permuted text/description and graph/subgraph inputs so any semantic alignment is destroyed; fine-tune TransformEHR with this control and compare AUPRC against MedTok and against the original single-token baseline. If the random-input control retains most of MedTok's gains, the AUPRC improvement is explained by added tokens/parameters rather than by multimodal medical semantics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that swapping standard tokenizers with MedTok improves AUPRC because MedTok generates 'more informative tokens' (Section 4.1)—is not isolated from a capacity confound. In Section 3.1, each code is quantized into K tokens for each of four embeddings (text-specific, graph-specific, text-cross, graph-cross), so a code that previously occupied one token now occupies multiple tokens. The value of K is never reported, and no baseline is matched for token count, sequence length, or embedding parameter count. For ETHOS the vocabulary is explicitly extended with MedTok tokens (Appendix B.3); for other models MedTok embeddings initialize the original tokens (Appendix B.3), again changing the embedding table size. A model with more tokens per code and a larger embedding table has more capacity to fit the training signal, which can inflate AUPRC, especially in multi-label drug recommendation where the largest per-task gains (4.10% and 4.78% in Table 3) appear. The abstract reports these two numbers and the EHRShot mortality gain (11.32%, Table 4) as if they were dataset-level averages, although Section 4.1 gives averages of 3.29% and 2.67% for MIMIC-III/IV. Consequently, even if every row in Tables 3-4 is positive, the evidence does not yet establish that the multimodal text-and-graph token representation—rather than extra tokens and parameters—causes the improvements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MedTok is a vector-quantized tokenizer that maps each medical code to a sequence of tokens derived from two modalities: the code's textual description, encoded by a frozen language model, and a local subgraph from PrimeKG, encoded by a trainable graph encoder. The tokenizer uses modality-specific and shared codebook regions together with cross-attention and packing losses, and can be swapped into existing transformer-based EHR models. The paper evaluates MedTok by replacing standard code tokens in five EHR backbones (ETHOS, GT-BEHRT, MulT-EHR, TransformEHR, BEHRT) on MIMIC-III, MIMIC-IV, and EHRShot, reporting AUPRC improvements in Tables 3 and 4, and also shows accuracy gains when MedTok tokens are prepended to three LLMs for medical question answering. The central claim is that MedTok is a general-purpose multimodal medical code tokenizer whose text-and-graph tokens improve downstream EHR and QA performance without changing the downstream architecture.","tokens_in":22256,"tokens_out":5899,"duration_ms":50901,"significance":"The paper addresses a real bottleneck in EHR foundation models: medical vocabularies contain hundreds of thousands of codes whose relational and textual context is ignored by standard tokenizers. If the reported gains were attributable to the multimodal semantic information, MedTok would be a useful plug-in component for EHR models. The authors ship code, evaluate with five distinct backbones in both inpatient and outpatient settings, include modality ablations, and attempt an interpretability analysis. However, the current evidence does not separate the contribution of the multimodal tokenizer from added token count, added embedding capacity, and per-dataset hyperparameter tuning. The headline numbers are also per-task gains rather than dataset-level averages. The empirical claim is therefore plausible but not established at the level claimed in the abstract.","major_comments":[{"comment":"The headline numbers in the Abstract and Contributions (4.10%, 4.78%, 11.32%) are not dataset-level average improvements. Section 4.1 reports average improvements of 3.29% on MIMIC-III and 2.67% on MIMIC-IV; the 4.10% and 4.78% figures are the DrugRec rows of Table 3. The 11.32% is the mortality row under Operational Outcomes in Table 4, while the average across the seven EHRShot columns is about 5.0%. Please report dataset-level averages explicitly and clearly label per-task gains.","section":"Abstract; §4.1; Tables 3-4"},{"comment":"MedTok replaces a single code token with K tokens for each of four embeddings (text-specific, graph-specific, text-cross, graph-cross), so the downstream sequence is longer. Appendix B.3 states that for ETHOS the original vocabulary is extended with MedTok tokens, and for the other models MedTok embeddings initialize the original tokens, changing the embedding-table size. The value of K, the number of tokens per code, is never reported, and no baseline is matched for sequence length, token count, or embedding parameter count. Because larger token counts and embedding tables add capacity, the observed AUPRC gains cannot currently be attributed to the multimodal text-and-graph semantics; a capacity-matched baseline (e.g., K random or independently quantized tokens of the same dimension per code) is needed.","section":"§3.1; Appendix B.3"},{"comment":"Many individual improvements are within one standard deviation of the baseline. Examples include TransformEHR LOS on MIMIC-IV (0.119 ± 0.001 vs. 0.121 ± 0.002), BEHRT MT on MIMIC-IV (0.028 ± 0.003 vs. 0.032 ± 0.006), and GT-BEHRT MT on MIMIC-III (0.160 ± 0.037 vs. 0.193 ± 0.046). No significance tests, confidence intervals, or effect sizes are reported. The statement that MedTok 'consistently improves performance across all five tasks' is stronger than the table supports.","section":"Tables 3-4"},{"comment":"The code-to-graph and code-to-text data are constructed automatically: UMLS/MONDO mapping, a custom entity linker, SciSpacy for semantic similarity, and GPT-4-expanded descriptions. No precision, recall, or manual validation of these mappings is reported, and GPT-4 expansion of sparse drug descriptions can introduce hallucinated clinical content. Because these are the sole inputs to MedTok's text and graph encoders, noisy or incorrect mappings are an alternative explanation for any performance differences and should be quantified, for example with a human-validated subset or a corruption/replacement ablation.","section":"Appendix A.1"},{"comment":"Equations (6)-(9) define objectives in terms of mutual information, but the implemented losses are InfoNCE and orthogonal-packing terms; the claimed equivalence is asserted ('Following Wang et al.') without proof or citation. In addition, the hyperparameters λ=β and codebook size N are tuned per dataset on the same benchmarks used for the main results: Figure 5 recommends λ=β=0.1 for inpatient and 0.01 for outpatient settings, and N=12,000 for MIMIC-III/IV but 24,000 for EHRShot. The main tables do not state which values were used, and no validation split is described to show that the reported numbers are not selected by this tuning.","section":"§3.2; §4.4"}],"minor_comments":[{"comment":"The weighting in Equation (4) uses -softmax(dist(...)), while §3.2's KL term uses softmax(-dist(...)); the sign convention should be made consistent.","section":"Equation (4)"},{"comment":"The value of K and the sizes of the text-specific, graph-specific, and shared codebook regions are never reported; without them, the actual token counts per code and codebook utilization are unclear.","section":"§3.1; §4.4"},{"comment":"The LOS task is multi-class (10 classes) but the table reports AUPRC, while Appendix C.5 says phenotype and drug recommendation use macro-averaged AUC-ROC; please specify how AUPRC is computed for multi-class and multi-label settings.","section":"Table 3; Appendix C.5"},{"comment":"The VQGraph reference appears multiple times with different years and venues (2023a, 2024a, 2024b), and the sentence 'Following Wang et al.' in §3.2 lacks a citation; please disambiguate.","section":"References; §3.2"},{"comment":"The x-axis labels in Figure 5A run together ('0.010.1 0.200.300.4 0.5'), making the tick values illegible; please fix the figure.","section":"Figure 5"},{"comment":"The Introduction states an EHRShot gain of 11.30% while the Abstract states 11.32%; please make these numbers consistent.","section":"Contributions"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a clinical machine-learning venue, and the code release is a genuine asset. My main concern is that the central empirical claim needs an honest reporting of dataset-level averages plus a capacity-matched comparison before the multimodal mechanism can be accepted. I would not reject the paper; I would condition acceptance on the major revisions above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MedTok is a real contribution: a unified tokenizer for 617k medical codes that combines frozen text embeddings, ontology subgraphs, and vector quantization into a shared codebook. That combination is new in the cited literature, and the paper evaluates it across five EHR models, three datasets, and multiple tasks. The integration story is clean — you swap the tokenizer and keep the downstream architecture — and the ablations showing both modalities matter are a nice touch. The token-packing losses are principled, even if each component is standard. The GitHub link also suggests the code will be available, which is a mark in its favor.\n\nThe soft spots are real, though. The headline numbers in the abstract are single-task gains, not dataset averages. The 4.10%, 4.78%, and 11.32% are drug recommendation on MIMIC-III/IV and mortality on EHRShot; Section 4.1 gives dataset averages of 3.29% and 2.67% for the in-patient sets, and the EHRShot average is around 5%. Those are still positive, but they are not what the abstract implies.\n\nMore concerning is the capacity confound. Each code is quantized into K tokens for four different embeddings, yet K is never reported, and no baseline matches the resulting token count, sequence length, or embedding-table size. Some improvements are within one standard deviation — ETHOS mortality on MIMIC-IV goes from 0.282 to 0.412, but other rows, like BEHRT mortality on MIMIC-IV from 0.028 to 0.032, are tiny relative to the error bars. Without significance tests or token-matched controls, the causal claim that the multimodal text-and-graph signal drives the gains is not established. The hyperparameters λ, β, and codebook size N are also tuned on the same benchmarks used for the reported numbers, with no separate validation set.\n\nThe data-generation pipeline is another soft spot. The mapping from codes to PrimeKG nodes relies on UMLS, MONDO, a custom entity linker, SciSpacy, and GPT-4 description enrichment, but no validation of mapping precision or GPT-4 output quality is reported. If those mappings are noisy, the \"multimodal\" signal is partly random. The QA experiment also lacks a clean control — no alternative prefix tokenizer is compared.\n\nNone of these issues are fatal. The method is plausible, well-engineered, and broadly applicable. But the paper needs revision: report dataset-level averages honestly, match token counts and parameters, add significance tests, validate the graph/text mappings, and include a token-matched QA control. I would send it to peer review — a serious referee will have clear asks — but with the expectation of heavy revision. It is worth a reading-group discussion, mainly for the evaluation pitfalls.","headline":"A genuinely useful engineering contribution with a new multimodal tokenizer for medical codes, but the abstract oversells the results and the evaluation leaves a capacity confound unresolved.","tokens_in":22841,"tokens_out":3458,"would_cite":true,"duration_ms":34038,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MedTok shows that tokenizing medical codes through both their textual definitions and their relational context improves EHR transformer models, with reported AUPRC gains of 4.10% on MIMIC-III, 4.78% on MIMIC-IV, and 11.32% on EHRShot.","keywords":["multimodal tokenization","medical codes","electronic health records","vector quantization","knowledge graph","clinical prediction","medical question answering","transformer models"],"falsifier":"Retrain MedTok with the same losses but randomly permute the graph subgraphs and text descriptions among codes; if the AUPRC gains over the standard tokenizer persist, the semantic text-and-graph signal is not what carries the improvement.","tokens_in":21716,"feed_emoji":"🩺","tokens_out":6764,"duration_ms":61626,"temperature":0.7,"pith_summary":"The paper claims that treating medical codes as isolated tokens is a bottleneck for electronic health record (EHR) foundation models. It introduces MedTok, a tokenizer that represents each code by its official text description and its neighborhood in a biomedical knowledge graph, then quantizes both views into a shared discrete codebook. Across five EHR transformer models and three datasets, replacing the standard tokenizer with MedTok improves AUPRC on every model and task, with reported gains of 4.10% on MIMIC-III, 4.78% on MIMIC-IV, and 11.32% on EHRShot, and the largest gains on drug recommendation. MedTok tokens also improve few-shot medical question answering when prepended to LLM prompts.","feed_headline":"Swap in MedTok to lift EHR model AUPRC by up to 11.32%","feed_subtitle":"A drop-in tokenizer that uses code descriptions plus ontology graphs boosts five clinical prediction models and medical QA.","key_machinery":"The load-bearing object is a vector-quantized multimodal tokenizer: a codebook of N discrete token vectors divided into text-specific, graph-specific, and shared regions, fed by a frozen text encoder and a trainable graph encoder connected by cross-attention. Quantization selects the top-K nearest codebook vectors for each embedding and aggregates them with distance-weighted softmax, while three loss families (codebook loss, KL alignment of cross-modal distance distributions, and token-packing losses for shared and specific information) train the tokenizer. This machinery is what lets one tokenizer serve multiple coding systems and plug into any transformer-based model.","core_discovery":"The central claim is that tokenization itself can be a learnable, multimodal step for structured EHR data. For each of 617,490 medical codes drawn from eight coding systems, MedTok encodes the code's textual definition with a language model encoder and its local subgraph from a biomedical knowledge graph with a graph encoder, then selects top-K entries from a codebook divided into text-specific, graph-specific, and shared regions. The resulting discrete tokens carry both modality-specific and cross-modality information, which the paper argues is why downstream models improve without any change to their architectures. The empirical claim is that a drop-in swap of the tokenizer improves AUPRC across five EHR models, with especially large gains on longitudinal outpatient data and on drug recommendation.","pith_inferences":["A random-permutation control would separate the semantic contribution of text and graph data from the extra capacity of MedTok's encoders; the paper does not report one.","Rare codes are the natural test case: if MedTok's value comes from shared semantics, its gains should be largest for low-frequency codes, which standard tokenizers represent poorly.","MedTok could become a shared clinical vocabulary across institutions that use different coding systems, letting pretrained models transfer without re-tokenizing.","Because EHRShot's longer visit histories show the largest gains, MedTok may matter most for longitudinal and few-shot settings rather than single-visit prediction."],"forward_implications":["A drop-in tokenizer swap, with no other change to a model's architecture or training, should transfer to transformer-based EHR models beyond the five tested.","Drug recommendation should see the largest benefit, because the relational graph encodes drug-disease associations that text-only tokens omit.","Longitudinal outpatient records should benefit at least as much as inpatient records, matching the EHRShot improvement and the sensitivity to larger codebooks.","Medical QA systems can consume MedTok tokens as structured prefix context to improve few-shot accuracy, a distinct use from EHR prediction.","A single tokenizer can span eight coding systems, reducing cross-system redundancy that standard tokenizers preserve as separate tokens."],"supporting_citations":[{"why":"Supplies the vector-quantization mechanism that MedTok adapts to quantize text and graph embeddings into a codebook.","marker":"(Van Den Oord et al., 2017)"},{"why":"Provides PrimeKG, the knowledge graph from which each code's relational subgraph is extracted.","marker":"(Chandak et al., 2023)"},{"why":"Supplies MIMIC-III, the inpatient EHR dataset where MedTok reports a 4.10% AUPRC gain.","marker":"(Johnson et al., 2016)"},{"why":"Supplies MIMIC-IV, the second inpatient benchmark where MedTok reports a 4.78% AUPRC gain.","marker":"(Johnson et al., 2024)"},{"why":"Provides EHRShot, the outpatient benchmark where MedTok reports its largest gain of 11.32%.","marker":"(Wornow et al., 2023)"},{"why":"Defines TransformEHR, the transformer backbone used in the tokenizer comparisons and ablations.","marker":"(Yang et al., 2023b)"},{"why":"Defines BEHRT, one of the five EHR models whose tokenizer MedTok replaces in the evaluation.","marker":"(Li et al., 2020)"},{"why":"Supplies the 24 phenotype definitions and length-of-stay task setup used in the inpatient evaluation.","marker":"(Harutyunyan et al., 2019)"}],"fun_headline_variants":["MedTok: multimodal tokenizer lifts EHR AUPRC by up to 11.32%","Swap in MedTok to boost EHR model AUPRC 11%","Medical code tokenizer uses text and graphs to improve EHR predictions","Multimodal code tokens outperform text-only EHR tokenizers","MedTok: drop-in tokenizer with 11% AUPRC boost on EHRShot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatic mappings from medical codes to knowledge-graph nodes and the GPT-4-expanded text descriptions are accurate enough to be meaningful, because the paper does not report validation of mapping precision or description quality.","fun_headline_variants_meta":{"raw":{"variants":["MedTok: multimodal tokenizer lifts EHR AUPRC by up to 11.32%","Swap in MedTok to boost EHR model AUPRC 11%","Medical code tokenizer uses text and graphs to improve EHR predictions","Multimodal code tokens outperform text-only EHR tokenizers","MedTok: drop-in tokenizer with 11% AUPRC boost on EHRShot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000426,"raw_usage":{"total_tokens":2188,"prompt_tokens":954,"completion_tokens":1234,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":1134}},"tokens_in":570,"tokens_out":1234,"duration_ms":8924,"temperature":1.0,"reasoning_tokens":1134,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T00:37:33.675350+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain MedTok with the same losses but randomly permute the graph subgraphs and text descriptions among codes; if the AUPRC gains over the standard tokenizer persist, the semantic text-and-graph signal is not what carries the improvement.","supporting_citations":[{"cited_title":"Building a knowledge graph to enable precision medicine","cited_arxiv_id":null,"evidence_quote":"Provides PrimeKG, the knowledge graph from which each code's relational subgraph is extracted."},{"cited_title":"A., and Shah, N","cited_arxiv_id":null,"evidence_quote":"Provides EHRShot, the outpatient benchmark where MedTok reports its largest gain of 11.32%."},{"cited_title":"C., Ver Steeg, G., and Galstyan, A","cited_arxiv_id":null,"evidence_quote":"Supplies the 24 phenotype definitions and length-of-stay task setup used in the inpatient evaluation."}],"review_version":1}