REVIEW 4 major objections 4 minor 27 references
Foundation Models for Clinical Records at Health System Scale
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A decoder-only Transformer pretrained on next-visit prediction can forecast dementia and knee osteoarthritis 2 and 5 years out without fine-tuning.
desk verdict Useful evaluation commentary on repeated tokens, but the zero-shot long-horizon claim hinges on an unstated inference-time mechanism. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a decoder-only GPT-2-style Transformer modified for visit-structured clinical sequences. Tokens within the same visit share a rotary positional embedding corresponding to elapsed time since the first visit, and the separator token ending visit i receives the positional embedding of visit i+1, so the representation the model conditions on explicitly encodes the inter-visit interval. Attention is causal across visits but full within a visit, reflecting the lack of fine-grained temporal order inside a single encounter. The training objective is weighted binary cross-entropy over the multi-hot token set of the next visit; the weight for each positive token decays as a power of its frequency in the patient's history, so repeated codes are down-weighted and new events are emphasized. This design is what allows zero-shot inference: the same separator representation plus a chosen horizon stands in for the next-visit time.
What would settle it
Hold a fixed test patient's history constant and vary only the separator positional offset across horizons from one month to five years; a true long-horizon forecast should produce smoothly increasing predicted incidence of the new-onset code. A flat, non-monotonic, or discontinuous curve would show that the positional mechanism does not extrapolate, undermining the zero-shot long-horizon scores.
Extended reading notes
Core claim
The central claim is that next-visit multi-label prediction is a sufficient pretraining task for clinically useful zero-shot forecasting from structured electronic health records. Because every token in the next visit is predicted jointly, the model learns relationships across diagnoses, medications, and labs without task-specific labels; because the separator token carries the positional embedding of the next visit's timestamp, the model can in principle be asked to forecast over a chosen horizon. The pretraining regularization down-weights tokens that have appeared many times in the patient's history, shifting the model toward new onsets. Tested on dementia and knee osteoarthritis, the zero-shot model reaches AUROC 0.814 at two years and 0.781 at five years for dementia, beating fine-tuned BERT (0.731 and 0.721); for knee OA it reaches 0.724 and 0.693 versus 0.743 and 0.717. The authors read this as showing that the generative objective captures complex clinical dependencies without task-specific fine-tuning.
Load-bearing premise
The zero-shot forecasts depend on the separator token's positional embedding, learned from real gaps between visits, to stand in for 2- and 5-year horizons at inference; if that embedding does not generalize to those longer gaps, the zero-shot scores do not measure long-horizon forecasting.
Editorial extensions
If this is right
- Zero-shot forecasting works without task-specific fine-tuning for the two tested conditions, so the same checkpoint can be pointed at a new disease by supplying its code set.
- Repeated-event regularization trades off overall precision and recall against earlier detection, so downstream users can choose an operating point by setting the decay factor.
- The on-time rate gives a way to measure whether a model is forecasting new onsets or merely repeating chronic diagnoses.
- Because the model conditions on the time of the next visit, the same pretrained weights can address multiple prediction horizons without retraining.
Reading between the lines
- If the horizon mechanism extrapolates, this could collapse disease-specific model building into one pretraining run; any outcome with a code set becomes a zero-shot query.
- A direct test of the mechanism would be to compare model probabilities for the same patient at interpolated horizons against actual two- and five-year outcomes, checking for monotonic increase with the horizon.
- The on-time rate metric could generalize beyond this paper as a standard reporting practice, since it penalizes models that inflate scores by repeating known chronic codes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents GPT-EHR, a decoder-only Transformer pretrained on a large institutional EHR dataset (NYU Langone, ~1.29M patients) using next-visit multi-label prediction with a power-decay regularization that down-weights repeated clinical tokens. The authors claim that this generative pretraining enables zero-shot forecasting of dementia and knee osteoarthritis incidence within 2- and 5-year horizons, matching or exceeding a fully fine-tuned masked-pretrained BERT baseline. They also introduce an 'on-time rate' metric to distinguish prediction of new onsets from repetition of known chronic conditions, and they report a comparison against a zero-shot LLaMA-3.3-70B model in an appendix. The central empirical claim rests on Table 1, which compares AUROC and AUPRC for the two conditions and horizons.
Significance. If the central claim holds, the paper demonstrates a valuable result: a generative pretraining objective for structured EHR data can produce clinically useful risk forecasts without task-specific fine-tuning, potentially lowering the cost of deploying predictive models across many conditions. The paper also contributes a practical warning about evaluation pitfalls in EHR foundation models (repeated events inflating performance) and proposes a concrete regularization and metric to address it. The scale of pretraining (1.6B parameters on 1.29M patients, 10 years of data) and the inclusion of a zero-shot LLM baseline are strengths. However, the significance is currently tempered by missing evaluation details that are load-bearing for the main claim, as detailed below.
major comments (4)
- [§2.1, §3.2, Appendix B.2] The inference-time mechanism for the 2-year and 5-year prediction horizons is never specified. In pretraining, the model conditions on the time of the next visit through the positional embedding of the <sep> token (RoPE). At zero-shot evaluation, it is not stated whether the <sep> embedding is set to the patient's actual next recorded visit time (which would be future information unavailable at prediction time), or to prediction time plus 730/1825 days. If the former, the model is receiving oracle visit timing and the '2/5-year' label applies only to the label window, not the model input. If the latter, the model is asked to extrapolate RoPE to relative gaps far beyond typical training inter-visit intervals (median 21 visits over 10 years implies gaps on the order of months). The paper reports no ablation, calibration curve, or analysis showing that predictions actually vary with the encoded horizon. Without this, Table 1 cannot be interpreted as evidence of genuine long-horizon zero-shot forecasting. This is a central, load-bearing point for the paper's main claim.
- [§3.2, Table 1] The fine-tuned BERT baseline is never described. The paper does not state the BERT architecture (e.g., number of layers/parameters), its pretraining data and masking strategy, the tokenization and visit-boundary representation used for fine-tuning, the fine-tuning procedure (including task-specific heads, loss, hyperparameters, and validation-based threshold selection), or whether it uses the same exclusion criteria as the GPT model. Since the central claim is that zero-shot GPT-EHR 'matches, and in some cases exceeds' a fully fine-tuned masked pretrained Transformer, the baseline must be specified in enough detail to assess whether the comparison is fair and reproducible. As written, the comparison is not verifiable.
- [Table 1, §4 Conclusion] The reported metrics do not fully support the claim of matching or exceeding the fine-tuned baseline. For knee OA, the BERT baseline has higher AUROC and AUPRC at both horizons. For dementia, GPT-EHR has higher AUROC but substantially lower AUPRC at both horizons (e.g., 5-year AUPRC 0.106 vs. 0.146, with non-overlapping confidence intervals). Given the low prevalence of these conditions, AUPRC is the more informative metric for clinical risk prediction, and the text 'maintained superior predictive performance' (for dementia) and 'matches, and in some cases exceeds' (Conclusion) overstates the evidence. The paper should either report a balanced interpretation or adjust the claim; at minimum, the discrepancy between AUROC and AUPRC should be explicitly discussed.
- [§3.1, Figure 2, §4] The contribution of the repeat-token regularization to the downstream zero-shot results is not demonstrated. The regularization is evaluated only on the pretraining next-visit prediction task (on-time rate for dementia and pancreatic cancer), and the downstream experiments use only δ=0.5 with no comparison to δ=1 (no regularization) or other values. The central 'key pitfall' claim about repeated events inflating performance is supported for pretraining metrics, but the fact that the proposed model's zero-shot advantage (or lack thereof) depends on the regularization is not established. Additionally, the definition of the on-time rate and the precision/recall calculation appears inconsistent: true positives are defined at the trajectory level ('a patient is considered a true positive if they eventually develop the condition and the model predicts it at any point across the rolling evaluation windows'), yet the text also states 'we only count the prediction as correct if it appears in the next visit.' These two definitions cannot both hold; the metric needs to be clarified.
minor comments (4)
- [Appendix B.4, Table 3] The hyperparameter table lists 'temporal decay 0.5' but the main text refers to 'decay factor δ'; please unify the nomenclature to avoid confusion.
- [Appendix B.3, Table 2] Several medication names contain typographical errors, e.g., 'RIV ASTIGMINE' should be 'RIVASTIGMINE'. These should be corrected for reproducibility since the codes are used for label construction.
- [Appendix C.2] The LLaMA comparison reports only precision and recall for dementia at 5 years, with no AUROC/AUPRC, no number of test patients, and no threshold details; a fuller description would strengthen the comparison.
- [§3.2] The exclusion criterion 'the disease onset occurred within one year following the prediction time point' is described, but it is not stated whether this exclusion is applied to both GPT and BERT evaluations identically; please clarify.
Circularity Check
No circular dependency: the zero-shot claims are tested against an external fine-tuned baseline and labels not used to train the model.
full rationale
The central derivation is self-contained. The pretraining objective (Section 2.3, Appendix B.2) predicts tokens of the next visit from history and the next-visit time encoded in the <sep> positional embedding. The zero-shot evaluations (Section 3.2, Appendix B.3) use externally defined onset labels and compare against a fully fine-tuned BERT baseline and LLaMA-3.3-70B, so the reported AUROC/AUPRC numbers are not constructed from the model's own training targets. The only overlapping-author citation (Zhu et al. 2024) is used for the general clinical motivation of early detection and is not load-bearing for any equation or evaluation. The repeat-token regularization hyperparameter δ is selected on the validation split and then applied to the test split, which is standard model selection, not a fit of the reported quantities. The paper does leave the inference-time horizon encoding (how t_{i+1} is set for 2- and 5-year windows) underspecified; that is a reproducibility and extrapolation concern, not a circularity, because no step defines the evaluation labels as the model's training objective or makes the test result true by construction. No circular step can be quoted from the paper.
Assumptions & free parameters
free parameters (2)
- decay factor delta =
0.5
- minimum weight w_min =
not reported
assumptions (3)
- ad hoc to paper The positional embedding of the <sep> token can be set to an arbitrary future time to condition zero-shot prediction at 2- and 5-year horizons.
- domain assumption The curated sets of ICD-10 and medication codes in Appendix B.3 accurately identify first incidence of dementia and knee OA.
- ad hoc to paper The fine-tuned BERT baseline is a strong and fair comparator, despite its configuration being unspecified.
Cite this review
Pith. "Pith review of Foundation Models for Clinical Records at Health System Scale." pith.science (2026). https://pith.science/paper/KAXOUSY4
@misc{pith2026250700574,
author = {Pith},
title = {Pith review of: Foundation Models for Clinical Records at Health System Scale},
year = {2026},
howpublished = {\url{https://pith.science/paper/KAXOUSY4}},
note = {Machine review of arXiv:2507.00574}
}
read the original abstract
Large-scale pretraining has transformed modeling of language and other data types, but its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present a novel generative pretraining strategy for sequential EHR data using next-visit event prediction. Our model learns to autoregressively generate various tokenized clinical events for the next visit based on patient history and inherently handles the joint prediction of heterogeneous data types. Additionally, we introduce regularization on predicting repeated events and highlight a key pitfall in EHR-based foundation model evaluations: repeated event tokens can inflate performance metrics when new onsets are not distinguished from subsequent occurrences. Our model is evaluated via zero-shot prediction for forecasting dementia and knee osteoarthritis incidence within 2 and 5 years, and the model performance rivals a fully fine-tuned masked pretrained Transformer baseline, demonstrating that our approach captures complex clinical dependencies without requiring costly task-specific fine-tuning.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
R., Cardoso, F., Siesling, S., et al
Arnold, M., Morgan, E., Rumgay, H., Mafra, A., Singh, D., Laversanne, M., Vignat, J., Gralow, J. R., Cardoso, F., Siesling, S., et al. Current and future burden of breast cancer: Global statistics for 2020 and 2040. The Breast, 66: 0 15--23, 2022
work page 2020
-
[3]
Choi, E., Schuetz, A., Stewart, W. F., and Sun, J. Using recurrent neural network models for early detection of heart failure onset. Journal of the American Medical Informatics Association, 24 0 (2): 0 361--370, 2017
work page 2017
-
[4]
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp.\ 4171--4186, 2019
2019
-
[5]
Timely diagnosis for alzheimer’s disease: a literature review on benefits and challenges
Dubois, B., Padovani, A., Scheltens, P., Rossi, A., and Dell’Agnello, G. Timely diagnosis for alzheimer’s disease: a literature review on benefits and challenges. Journal of Alzheimer’s disease, 49 0 (3): 0 617--631, 2015
work page 2015
-
[6]
Ehrmamba: Towards generalizable and scalable foundation models for electronic health records
Fallahpour, A., Alinoori, M., Ye, W., Cao, X., Afkanpour, A., and Krishnan, A. Ehrmamba: Towards generalizable and scalable foundation models for electronic health records. arXiv preprint arXiv:2405.14567, 2024
arXiv 2024
-
[7]
Guo, L. L., Steinberg, E., Fleming, S. L., Posada, J., Lemmon, J., Pfohl, S. R., Shah, N., Fries, J., and Sung, L. Ehr foundation models improve robustness in the presence of temporal distribution shift. Scientific Reports, 13 0 (1): 0 3767, 2023
work page 2023
-
[8]
L., Fries, J., Steinberg, E., Fleming, S
Guo, L. L., Fries, J., Steinberg, E., Fleming, S. L., Morse, K., Aftandilian, C., Posada, J., Shah, N., and Sung, L. A multi-center study on the adaptability of a shared foundation model for electronic health records. NPJ Digital Medicine, 7 0 (1): 0 171, 2024
work page 2024
Show all 27 references
-
[9]
A., and Mark, R
Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., and Mark, R. Mimic-iv. PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021), pp.\ 49--55, 2020
2021
-
[10]
Disease-modifying treatments for osteoarthritis (dmoads) of the knee and hip: lessons learned from failures and opportunities for the future
Karsdal, M., Michaelis, M., Ladel, C., Siebuhr, A., Bihlet, A., Andersen, J., Guehring, H., Christiansen, C., Bay-Jensen, A., and Kraus, V. Disease-modifying treatments for osteoarthritis (dmoads) of the knee and hip: lessons learned from failures and opportunities for the fut...
2013
-
[11]
A., Deng, A., Baston, A., Ross, J., Idowu, E., et al
Kraljevic, Z., Bean, D., Shek, A., Bendayan, R., Hemingway, H., Yeung, J. A., Deng, A., Baston, A., Ross, J., Idowu, E., et al. Foresight--generative pretrained transformer (gpt) for modelling of patient timelines using ehrs. arXiv preprint arXiv:2212.08072, 2022
2022 arXiv
-
[12]
Li, Y., Rao, S., Solares, J. R. A., Hassaine, A., Ramakrishnan, R., Canoy, D., Zhu, Y., Rahimi, K., and Salimi-Khorshidi, G. Behrt: transformer for electronic health records. Scientific reports, 10 0 (1): 0 7155, 2020
2020
-
[13]
Hi-behrt: hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records
Li, Y., Mamouei, M., Salimi-Khorshidi, G., Rao, S., Hassaine, A., Canoy, D., Lukasiewicz, T., and Rahimi, K. Hi-behrt: hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records. IEEE journal of biome...
2022
-
[14]
McDermott, M., Nestor, B., Argaw, P., and Kohane, I. S. Event stream gpt: a data pre-processing and modeling library for generative, pre-trained transformers over continuous-time sequences of complex events. Advances in Neural Information Processing Systems, 36: 0 24322--24334, 2023
2023
-
[15]
H., Levaux, H
Nordo, A. H., Levaux, H. P., Becnel, L. B., Galvez, J., Rao, P., Stem, K., Prakash, E., and Kush, R. D. Use of ehrs data for clinical research: historical progress and current applications. Learning health systems, 3 0 (1): 0 e10076, 2019
2019
-
[16]
S., Spotnitz, M., Chen, R., Perotte, A., and Natarajan, K
Pang, C., Jiang, X., Kalluri, K. S., Spotnitz, M., Chen, R., Perotte, A., and Natarajan, K. Cehr-bert: Incorporating temporal information from structured ehr data to improve prediction tasks. In Machine Learning for Health, pp.\ 239--260. PMLR, 2021
2021
-
[17]
Improving language understanding by generative pre-training
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018
2018
-
[18]
Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., and Zhi, D. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine, 4 0 (1): 0 86, 2021
2021
-
[19]
E., Was, J., Li, Q., Bates, D
Renc, P., Jia, Y., Samir, A. E., Was, J., Li, Q., Bates, D. W., and Sitek, A. Zero shot health trajectory prediction using transformer. NPJ Digital Medicine, 7 0 (1): 0 256, 2024
2024
-
[20]
Pre-training of graph augmented transformers for medication recommendation
Shang, J., Ma, T., Xiao, C., and Sun, J. Pre-training of graph augmented transformers for medication recommendation. arXiv preprint arXiv:1906.00346, 2019
1906 arXiv
-
[21]
J., Bihorac, A., and Rashidi, P
Shickel, B., Tighe, P. J., Bihorac, A., and Rashidi, P. Deep ehr: a survey of recent advances in deep learning techniques for electronic health record (ehr) analysis. IEEE journal of biomedical and health informatics, 22 0 (5): 0 1589--1604, 2017
2017
-
[22]
A., Corbin, C
Steinberg, E., Jung, K., Fries, J. A., Corbin, C. K., Pfohl, S. R., and Shah, N. H. Language models are an effective representation learning technique for electronic health record data. Journal of biomedical informatics, 113: 0 103637, 2021
2021
-
[23]
Roformer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024
2024
-
[24]
Ehrshot: An ehr benchmark for few-shot evaluation of foundation models
Wornow, M., Thapa, R., Steinberg, E., Fries, J., and Shah, N. Ehrshot: An ehr benchmark for few-shot evaluation of foundation models. Advances in Neural Information Processing Systems, 36: 0 67125--67137, 2023
2023
-
[25]
Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review
Xiao, C., Choi, E., and Sun, J. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. Journal of the American Medical Informatics Association, 25 0 (10): 0 1419--1428, 2018
2018
-
[26]
Transformehr: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records
Yang, Z., Mitra, A., Liu, W., Berlowitz, D., and Yu, H. Transformehr: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records. Nature communications, 14 0 (1): 0 7857, 2023
2023
-
[27]
R., Huang, S.-L., Ma, X., Chaudhari, A., Madaan, D., Almahmoud, E., Chopra, S., et al
Zhu, W., Tang, H., Zhang, H., Rajamohan, H. R., Huang, S.-L., Ma, X., Chaudhari, A., Madaan, D., Almahmoud, E., Chopra, S., et al. Predicting risk of alzheimer’s diseases and related dementias with ai foundation model on electronic health records. medRxiv, 2024
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.