Large Language Models are Powerful Electronic Health Record Encoders

Benjamin Wild; David Sontag; Georg von Arnim; Gerhard Hindricks; Noel Kronenberg; Roland Eils; Stefan Hegselmann; Tillmann Rheude

arxiv: 2502.17403 · v5 · submitted 2025-02-24 · 💻 cs.LG · cs.AI· cs.CL

Large Language Models are Powerful Electronic Health Record Encoders

Stefan Hegselmann , Georg von Arnim , Tillmann Rheude , Noel Kronenberg , David Sontag , Gerhard Hindricks , Roland Eils , Benjamin Wild This is my paper

Pith reviewed 2026-05-23 01:26 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL

keywords large language modelselectronic health recordsembeddingsclinical predictionEHRSHOT benchmarkfoundation modelsEHR encoders

0 comments

The pith

General-purpose LLMs match specialized EHR models by converting medical codes to plain text for embeddings.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that electronic health records can be converted into plain text by replacing medical codes with natural language descriptions, allowing general-purpose large language models to generate embeddings for clinical prediction tasks. These embeddings perform on par with those from a specialized EHR foundation model across 15 tasks in the EHRSHOT benchmark. In external validation on UK Biobank data, the LLM approach yields statistically significant gains on some tasks, credited to broader vocabulary coverage and improved generalization. The method requires no access to private medical training data or site-specific pretraining, unlike traditional EHR models. This setup illustrates a trade-off between the efficiency of custom models and the portability of LLM-based encodings.

Core claim

By converting EHR data into plain text by replacing medical codes with natural-language descriptions, general-purpose Large Language Models produce high-dimensional embeddings for downstream prediction tasks without access to private medical training data. LLM-based embeddings perform on par with a specialized EHR foundation model, CLMBR-T-Base, across 15 clinical tasks from the EHRSHOT benchmark. In an external validation using the UK Biobank, an LLM-based model shows statistically significant improvements for some tasks, which we attribute to higher vocabulary coverage and slightly better generalization.

What carries the argument

Conversion of EHR medical codes to natural-language text descriptions, enabling general LLMs to extract embeddings without domain-specific pretraining or private data.

If this is right

LLM embeddings achieve performance parity with specialized EHR models on 15 clinical prediction tasks.
External validation reveals advantages for some tasks due to wider vocabulary coverage.
The approach removes the requirement for private medical data in developing predictive models.
A trade-off exists between computational efficiency of specialized models and the data independence of LLM embeddings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Hospitals lacking large private datasets could adopt this method to build prediction tools more readily.
The technique may extend to other code-heavy domains like insurance claims or lab results for similar encoding benefits.
Combining these embeddings with additional data types such as notes or images could further boost task performance.

Load-bearing premise

Converting EHR data into plain text by replacing medical codes with natural-language descriptions preserves sufficient clinical signal for general-purpose LLMs to produce embeddings that support accurate downstream prediction without any domain-specific pretraining or private medical data.

What would settle it

A new set of clinical tasks where LLM embeddings after text conversion underperform CLMBR-T-Base by a large margin on average would falsify the on-par performance claim.

read the original abstract

Electronic Health Records (EHRs) offer considerable potential for clinical prediction, but their complexity and heterogeneity challenge traditional machine learning. Domain-specific EHR foundation models trained on unlabeled EHR data have shown improved predictive accuracy and generalization. However, their development is constrained by limited data access and site-specific vocabularies. We convert EHR data into plain text by replacing medical codes with natural-language descriptions, enabling general-purpose Large Language Models (LLMs) to produce high-dimensional embeddings for downstream prediction tasks without access to private medical training data. LLM-based embeddings perform on par with a specialized EHR foundation model, CLMBR-T-Base, across 15 clinical tasks from the EHRSHOT benchmark. In an external validation using the UK Biobank, an LLM-based model shows statistically significant improvements for some tasks, which we attribute to higher vocabulary coverage and slightly better generalization. Overall, we reveal a trade-off between the computational efficiency of specialized EHR models and the portability and data independence of LLM-based embeddings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Converting EHR codes to plain text lets general LLMs match a specialized model on EHRSHOT tasks and beat it on some UK Biobank ones, but the abstract alone gives no way to verify the numbers or setup.

read the letter

The paper's main claim is that turning medical codes into natural-language descriptions and running them through an off-the-shelf LLM produces embeddings that match CLMBR-T-Base across 15 EHRSHOT tasks and show statistically significant gains on some external UK Biobank predictions. They tie the edge to broader vocabulary coverage and better generalization without any site-specific pretraining or private data access. That directly tackles the portability problem in clinical ML where vocabularies and data stay locked inside institutions. The abstract also flags the efficiency trade-off against specialized models, which is a fair point to surface. The approach itself is simple and avoids the usual barriers around training data and custom tokenizers. If the full paper backs the numbers with proper controls, it could be a practical option for groups that need models to run across hospitals without retraining from scratch. The soft spot is obvious: only the abstract exists here. No methods section, no tables, no error bars, no description of the exact conversion process or which LLM was used, and no raw stats to inspect. That leaves the parity claim and the external improvement uncheckable, so the central empirical result stays provisional. The assumption that code-to-text conversion keeps the necessary clinical signal is plausible but untested in what we can see. This work sits squarely in applied clinical prediction. Readers who build or deploy models across sites would get the most out of it once the experiments are visible. It deserves a serious referee because the portability angle is concrete and the benchmark results are stated clearly enough to be worth checking in full, even if heavy revision is likely.

Referee Report

2 major / 0 minor

Summary. The manuscript claims that converting EHR data into plain text by replacing medical codes with natural-language descriptions enables general-purpose LLMs to produce embeddings that achieve performance parity with the specialized EHR foundation model CLMBR-T-Base across 15 clinical tasks from the EHRSHOT benchmark. It further claims statistically significant improvements for some tasks in external validation on the UK Biobank, attributed to higher vocabulary coverage and better generalization, while noting a trade-off between computational efficiency of domain-specific models and the portability/data independence of LLM-based embeddings.

Significance. If the empirical results hold after full verification, the work would be significant for demonstrating that general-purpose LLMs can serve as effective EHR encoders without domain-specific pretraining or access to private medical data, potentially increasing accessibility and reducing barriers posed by limited data access and site-specific vocabularies. The direct comparison to an established baseline (CLMBR-T-Base) and the external validation add potential value, though the absence of the full manuscript prevents assessment of robustness or reproducibility.

major comments (2)

[Abstract] Abstract: The central claims of performance parity with CLMBR-T-Base on 15 EHRSHOT tasks and statistically significant improvements on UK Biobank are stated without any methods details, error bars, statistical tests, data-handling descriptions, or quantitative results, making it impossible to verify whether the data support the claims.
[Abstract] Abstract: The key assumption that replacing medical codes with natural-language descriptions preserves sufficient clinical signal for accurate downstream prediction is presented without any supporting evidence, ablation, or discussion, and this assumption is load-bearing for the portability claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their comments. We address each major comment below. The full manuscript (arXiv:2502.17403) contains the detailed methods, results, and statistical analyses referenced in the abstract.

read point-by-point responses

Referee: [Abstract] Abstract: The central claims of performance parity with CLMBR-T-Base on 15 EHRSHOT tasks and statistically significant improvements on UK Biobank are stated without any methods details, error bars, statistical tests, data-handling descriptions, or quantitative results, making it impossible to verify whether the data support the claims.

Authors: Abstracts are intentionally concise and omit detailed methods, error bars, statistical tests, and quantitative results due to length constraints. The full manuscript provides all of these: the specific statistical tests used to establish significance on UK Biobank tasks, error bars from repeated evaluations, data-handling procedures for the 15 EHRSHOT tasks, and the exact performance numbers supporting parity with CLMBR-T-Base. The abstract serves only as a high-level summary of those findings. revision: no
Referee: [Abstract] Abstract: The key assumption that replacing medical codes with natural-language descriptions preserves sufficient clinical signal for accurate downstream prediction is presented without any supporting evidence, ablation, or discussion, and this assumption is load-bearing for the portability claim.

Authors: The reported performance parity with CLMBR-T-Base across all 15 EHRSHOT tasks constitutes direct empirical evidence that the conversion preserves clinical signal. The statistically significant gains on UK Biobank tasks, attributed to superior vocabulary coverage, provide additional support for the portability claim. The full manuscript contains the relevant ablations and discussion; we can expand the introduction to explicitly link these results to the assumption if the referee recommends it. revision: partial

Circularity Check

0 steps flagged

No significant circularity

full rationale

The provided abstract contains no equations, parameters, or derivation chain of any kind. The central claims are framed as direct empirical comparisons (LLM embeddings vs. CLMBR-T-Base on EHRSHOT tasks, plus UK Biobank external validation) with no fitted inputs renamed as predictions, no self-citations invoked as uniqueness theorems, and no ansatz or renaming steps. The argument is therefore self-contained against external benchmarks and receives the default non-finding.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities; the central claim rests on the unstated premise that text conversion preserves clinical signal, which is not formalized here.

pith-pipeline@v0.9.0 · 5692 in / 1066 out tokens · 31116 ms · 2026-05-23T01:26:35.328861+00:00 · methodology

discussion (0)

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning
cs.LG 2026-04 unverdicted novelty 7.0

Multimodal contrastive learning using multilinear products is fragile to single bad modalities, and a gated version improves top-1 retrieval accuracy on synthetic and real trimodal data.
Fusion or Confusion? Multimodal Complexity Is Not All You Need
cs.LG 2025-12 unverdicted novelty 6.0

Complex multimodal architectures do not reliably outperform unimodal baselines or a simple multimodal baseline under standardized evaluation.
Representation Before Training: A Fixed-Budget Benchmark for Generative Medical Event Models
cs.LG 2026-04 unverdicted novelty 5.0

Fused code-value tokenization improves mortality AUROC from 0.891 to 0.915 and other clinical outcome predictions, while certain temporal encodings like event order match or exceed time tokens with shorter sequences.