Pith. sign in

REVIEW 4 major objections 4 minor 27 references

Foundation Models for Clinical Records at Health System Scale

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A decoder-only Transformer pretrained on next-visit prediction can forecast dementia and knee osteoarthritis 2 and 5 years out without fine-tuning.

desk verdict Useful evaluation commentary on repeated tokens, but the zero-shot long-horizon claim hinges on an unstated inference-time mechanism. read the letter →

arxiv 2507.00574 v1 pith:KAXOUSY4 submitted 2025-07-01 cs.LG

classification cs.LG
keywords electronichealthrecordsfoundationmodelszero-shotpredictionnext-visitgenerativepretrainingdementiaforecastingkneeosteoarthritisrepeatedeventregularization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a decoder-only Transformer trained on one objective—predict every clinical event in the next visit from the full patient history plus the time elapsed until that visit—can carry out useful long-horizon disease forecasting with no task-specific fine-tuning. The authors pretrain a roughly 1.6-billion-parameter model on ten years of records from about 1.29 million patients at a large health system, then test it zero-shot on first incidence of dementia and knee osteoarthritis within 2 and 5 years. On dementia their model's AUROC exceeds that of a fully fine-tuned masked-pretrained BERT baseline; on knee OA it is close behind despite seeing no task labels. The paper also identifies an evaluation pitfall: repeated chronic-condition codes can inflate apparent performance, and it introduces a decay-based regularization plus an 'on-time rate' metric to separate new onsets from repeated events.

What carries the argument

The central object is a decoder-only GPT-2-style Transformer modified for visit-structured clinical sequences. Tokens within the same visit share a rotary positional embedding corresponding to elapsed time since the first visit, and the separator token ending visit i receives the positional embedding of visit i+1, so the representation the model conditions on explicitly encodes the inter-visit interval. Attention is causal across visits but full within a visit, reflecting the lack of fine-grained temporal order inside a single encounter. The training objective is weighted binary cross-entropy over the multi-hot token set of the next visit; the weight for each positive token decays as a power of its frequency in the patient's history, so repeated codes are down-weighted and new events are emphasized. This design is what allows zero-shot inference: the same separator representation plus a chosen horizon stands in for the next-visit time.

What would settle it

Hold a fixed test patient's history constant and vary only the separator positional offset across horizons from one month to five years; a true long-horizon forecast should produce smoothly increasing predicted incidence of the new-onset code. A flat, non-monotonic, or discontinuous curve would show that the positional mechanism does not extrapolate, undermining the zero-shot long-horizon scores.

Watch

Extended reading notes

Core claim

The central claim is that next-visit multi-label prediction is a sufficient pretraining task for clinically useful zero-shot forecasting from structured electronic health records. Because every token in the next visit is predicted jointly, the model learns relationships across diagnoses, medications, and labs without task-specific labels; because the separator token carries the positional embedding of the next visit's timestamp, the model can in principle be asked to forecast over a chosen horizon. The pretraining regularization down-weights tokens that have appeared many times in the patient's history, shifting the model toward new onsets. Tested on dementia and knee osteoarthritis, the zero-shot model reaches AUROC 0.814 at two years and 0.781 at five years for dementia, beating fine-tuned BERT (0.731 and 0.721); for knee OA it reaches 0.724 and 0.693 versus 0.743 and 0.717. The authors read this as showing that the generative objective captures complex clinical dependencies without task-specific fine-tuning.

Load-bearing premise

The zero-shot forecasts depend on the separator token's positional embedding, learned from real gaps between visits, to stand in for 2- and 5-year horizons at inference; if that embedding does not generalize to those longer gaps, the zero-shot scores do not measure long-horizon forecasting.

Editorial extensions

If this is right

  • Zero-shot forecasting works without task-specific fine-tuning for the two tested conditions, so the same checkpoint can be pointed at a new disease by supplying its code set.
  • Repeated-event regularization trades off overall precision and recall against earlier detection, so downstream users can choose an operating point by setting the decay factor.
  • The on-time rate gives a way to measure whether a model is forecasting new onsets or merely repeating chronic diagnoses.
  • Because the model conditions on the time of the next visit, the same pretrained weights can address multiple prediction horizons without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the horizon mechanism extrapolates, this could collapse disease-specific model building into one pretraining run; any outcome with a code set becomes a zero-shot query.
  • A direct test of the mechanism would be to compare model probabilities for the same patient at interpolated horizons against actual two- and five-year outcomes, checking for monotonic increase with the horizon.
  • The on-time rate metric could generalize beyond this paper as a standard reporting practice, since it penalizes models that inflate scores by repeating known chronic codes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents GPT-EHR, a decoder-only Transformer pretrained on a large institutional EHR dataset (NYU Langone, ~1.29M patients) using next-visit multi-label prediction with a power-decay regularization that down-weights repeated clinical tokens. The authors claim that this generative pretraining enables zero-shot forecasting of dementia and knee osteoarthritis incidence within 2- and 5-year horizons, matching or exceeding a fully fine-tuned masked-pretrained BERT baseline. They also introduce an 'on-time rate' metric to distinguish prediction of new onsets from repetition of known chronic conditions, and they report a comparison against a zero-shot LLaMA-3.3-70B model in an appendix. The central empirical claim rests on Table 1, which compares AUROC and AUPRC for the two conditions and horizons.

Significance. If the central claim holds, the paper demonstrates a valuable result: a generative pretraining objective for structured EHR data can produce clinically useful risk forecasts without task-specific fine-tuning, potentially lowering the cost of deploying predictive models across many conditions. The paper also contributes a practical warning about evaluation pitfalls in EHR foundation models (repeated events inflating performance) and proposes a concrete regularization and metric to address it. The scale of pretraining (1.6B parameters on 1.29M patients, 10 years of data) and the inclusion of a zero-shot LLM baseline are strengths. However, the significance is currently tempered by missing evaluation details that are load-bearing for the main claim, as detailed below.

major comments (4)
  1. [§2.1, §3.2, Appendix B.2] The inference-time mechanism for the 2-year and 5-year prediction horizons is never specified. In pretraining, the model conditions on the time of the next visit through the positional embedding of the <sep> token (RoPE). At zero-shot evaluation, it is not stated whether the <sep> embedding is set to the patient's actual next recorded visit time (which would be future information unavailable at prediction time), or to prediction time plus 730/1825 days. If the former, the model is receiving oracle visit timing and the '2/5-year' label applies only to the label window, not the model input. If the latter, the model is asked to extrapolate RoPE to relative gaps far beyond typical training inter-visit intervals (median 21 visits over 10 years implies gaps on the order of months). The paper reports no ablation, calibration curve, or analysis showing that predictions actually vary with the encoded horizon. Without this, Table 1 cannot be interpreted as evidence of genuine long-horizon zero-shot forecasting. This is a central, load-bearing point for the paper's main claim.
  2. [§3.2, Table 1] The fine-tuned BERT baseline is never described. The paper does not state the BERT architecture (e.g., number of layers/parameters), its pretraining data and masking strategy, the tokenization and visit-boundary representation used for fine-tuning, the fine-tuning procedure (including task-specific heads, loss, hyperparameters, and validation-based threshold selection), or whether it uses the same exclusion criteria as the GPT model. Since the central claim is that zero-shot GPT-EHR 'matches, and in some cases exceeds' a fully fine-tuned masked pretrained Transformer, the baseline must be specified in enough detail to assess whether the comparison is fair and reproducible. As written, the comparison is not verifiable.
  3. [Table 1, §4 Conclusion] The reported metrics do not fully support the claim of matching or exceeding the fine-tuned baseline. For knee OA, the BERT baseline has higher AUROC and AUPRC at both horizons. For dementia, GPT-EHR has higher AUROC but substantially lower AUPRC at both horizons (e.g., 5-year AUPRC 0.106 vs. 0.146, with non-overlapping confidence intervals). Given the low prevalence of these conditions, AUPRC is the more informative metric for clinical risk prediction, and the text 'maintained superior predictive performance' (for dementia) and 'matches, and in some cases exceeds' (Conclusion) overstates the evidence. The paper should either report a balanced interpretation or adjust the claim; at minimum, the discrepancy between AUROC and AUPRC should be explicitly discussed.
  4. [§3.1, Figure 2, §4] The contribution of the repeat-token regularization to the downstream zero-shot results is not demonstrated. The regularization is evaluated only on the pretraining next-visit prediction task (on-time rate for dementia and pancreatic cancer), and the downstream experiments use only δ=0.5 with no comparison to δ=1 (no regularization) or other values. The central 'key pitfall' claim about repeated events inflating performance is supported for pretraining metrics, but the fact that the proposed model's zero-shot advantage (or lack thereof) depends on the regularization is not established. Additionally, the definition of the on-time rate and the precision/recall calculation appears inconsistent: true positives are defined at the trajectory level ('a patient is considered a true positive if they eventually develop the condition and the model predicts it at any point across the rolling evaluation windows'), yet the text also states 'we only count the prediction as correct if it appears in the next visit.' These two definitions cannot both hold; the metric needs to be clarified.
minor comments (4)
  1. [Appendix B.4, Table 3] The hyperparameter table lists 'temporal decay 0.5' but the main text refers to 'decay factor δ'; please unify the nomenclature to avoid confusion.
  2. [Appendix B.3, Table 2] Several medication names contain typographical errors, e.g., 'RIV ASTIGMINE' should be 'RIVASTIGMINE'. These should be corrected for reproducibility since the codes are used for label construction.
  3. [Appendix C.2] The LLaMA comparison reports only precision and recall for dementia at 5 years, with no AUROC/AUPRC, no number of test patients, and no threshold details; a fuller description would strengthen the comparison.
  4. [§3.2] The exclusion criterion 'the disease onset occurred within one year following the prediction time point' is described, but it is not stated whether this exclusion is applied to both GPT and BERT evaluations identically; please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular dependency: the zero-shot claims are tested against an external fine-tuned baseline and labels not used to train the model.

full rationale

The central derivation is self-contained. The pretraining objective (Section 2.3, Appendix B.2) predicts tokens of the next visit from history and the next-visit time encoded in the <sep> positional embedding. The zero-shot evaluations (Section 3.2, Appendix B.3) use externally defined onset labels and compare against a fully fine-tuned BERT baseline and LLaMA-3.3-70B, so the reported AUROC/AUPRC numbers are not constructed from the model's own training targets. The only overlapping-author citation (Zhu et al. 2024) is used for the general clinical motivation of early detection and is not load-bearing for any equation or evaluation. The repeat-token regularization hyperparameter δ is selected on the validation split and then applied to the test split, which is standard model selection, not a fit of the reported quantities. The paper does leave the inference-time horizon encoding (how t_{i+1} is set for 2- and 5-year windows) underspecified; that is a reproducibility and extrapolation concern, not a circularity, because no step defines the evaluation labels as the model's training objective or makes the test result true by construction. No circular step can be quoted from the paper.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim depends on two tunable hyperparameters (delta, w_min) and on assumptions about the time-conditioning mechanism, label validity, and baseline fairness. No new physical entities are introduced.

free parameters (2)
  • decay factor delta = 0.5
    Controls the strength of repeat-token regularization; chosen as a balanced operating point from pretraining validation (Section 3.1, Figure 2). The zero-shot results all use this value.
  • minimum weight w_min = not reported
    Floor for repeat-token weights in Equation 1 (Appendix B.2); the chosen value is not stated, so its influence on the loss and downstream results is unknown.
assumptions (3)
  • ad hoc to paper The positional embedding of the <sep> token can be set to an arbitrary future time to condition zero-shot prediction at 2- and 5-year horizons.
    Section 2.1 introduces the <sep> positional embedding of the next visit; Section 3.2 uses it for zero-shot horizons without demonstrating extrapolation beyond observed inter-visit intervals.
  • domain assumption The curated sets of ICD-10 and medication codes in Appendix B.3 accurately identify first incidence of dementia and knee OA.
    The labels are defined solely by these codes; any misclassification biases both the model and baseline equally, but the absolute performance numbers depend on this coding accuracy.
  • ad hoc to paper The fine-tuned BERT baseline is a strong and fair comparator, despite its configuration being unspecified.
    Without details on BERT's architecture, pretraining corpus, and fine-tuning protocol, the comparison's validity is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Foundation Models for Clinical Records at Health System Scale." pith.science (2026). https://pith.science/paper/KAXOUSY4

@misc{pith2026250700574,
  author       = {Pith},
  title        = {Pith review of: Foundation Models for Clinical Records at Health System Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KAXOUSY4}},
  note         = {Machine review of arXiv:2507.00574}
}
read the original abstract

Large-scale pretraining has transformed modeling of language and other data types, but its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present a novel generative pretraining strategy for sequential EHR data using next-visit event prediction. Our model learns to autoregressively generate various tokenized clinical events for the next visit based on patient history and inherently handles the joint prediction of heterogeneous data types. Additionally, we introduce regularization on predicting repeated events and highlight a key pitfall in EHR-based foundation model evaluations: repeated event tokens can inflate performance metrics when new onsets are not distinguished from subsequent occurrences. Our model is evaluated via zero-shot prediction for forecasting dementia and knee osteoarthritis incidence within 2 and 5 years, and the model performance rivals a fully fine-tuned masked pretrained Transformer baseline, demonstrating that our approach captures complex clinical dependencies without requiring costly task-specific fine-tuning.

Figures

Figures reproduced from arXiv: 2507.00574 by the authors.

Figure 1
Figure 1. Illustrative examples to outline the GPT-EHR system, showing both the pretraining setup (a) and the evaluation methodology (b). performance through pretraining evaluations and propose regularization for repeated events to encourage the learning of new events. Compared to a fully fine-tuned BERT-based foundation model baseline, we demonstrate strong zero-shot generalization on forecasting dementia and knee osteoarthr… view at source ↗
Figure 2
Figure 2. Effect of decay regularization on repeated tokens. Show￾ing precision/recall against the primary (left) y-axis, while dashed lines show the on-time rate against the secondary (right) y-axis, for distinct decay factors. Left: dementia. Right: pancreatic cancer. Importantly, we define a new metric, the on-time rate r = TP≤t1 /TPtotal, which measures the proportion of true positive predictions that occur at or before t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 20 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    R., Cardoso, F., Siesling, S., et al

    Arnold, M., Morgan, E., Rumgay, H., Mafra, A., Singh, D., Laversanne, M., Vignat, J., Gralow, J. R., Cardoso, F., Siesling, S., et al. Current and future burden of breast cancer: Global statistics for 2020 and 2040. The Breast, 66: 0 15--23, 2022

  3. [3]

    F., and Sun, J

    Choi, E., Schuetz, A., Stewart, W. F., and Sun, J. Using recurrent neural network models for early detection of heart failure onset. Journal of the American Medical Informatics Association, 24 0 (2): 0 361--370, 2017

  4. [4]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp.\ 4171--4186, 2019

  5. [5]

    Timely diagnosis for alzheimer’s disease: a literature review on benefits and challenges

    Dubois, B., Padovani, A., Scheltens, P., Rossi, A., and Dell’Agnello, G. Timely diagnosis for alzheimer’s disease: a literature review on benefits and challenges. Journal of Alzheimer’s disease, 49 0 (3): 0 617--631, 2015

  6. [6]

    Ehrmamba: Towards generalizable and scalable foundation models for electronic health records

    Fallahpour, A., Alinoori, M., Ye, W., Cao, X., Afkanpour, A., and Krishnan, A. Ehrmamba: Towards generalizable and scalable foundation models for electronic health records. arXiv preprint arXiv:2405.14567, 2024

  7. [7]

    L., Steinberg, E., Fleming, S

    Guo, L. L., Steinberg, E., Fleming, S. L., Posada, J., Lemmon, J., Pfohl, S. R., Shah, N., Fries, J., and Sung, L. Ehr foundation models improve robustness in the presence of temporal distribution shift. Scientific Reports, 13 0 (1): 0 3767, 2023

  8. [8]

    L., Fries, J., Steinberg, E., Fleming, S

    Guo, L. L., Fries, J., Steinberg, E., Fleming, S. L., Morse, K., Aftandilian, C., Posada, J., Shah, N., and Sung, L. A multi-center study on the adaptability of a shared foundation model for electronic health records. NPJ Digital Medicine, 7 0 (1): 0 171, 2024

Show all 27 references
  1. [9]

    A., and Mark, R

    Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., and Mark, R. Mimic-iv. PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021), pp.\ 49--55, 2020

  2. [10]

    Disease-modifying treatments for osteoarthritis (dmoads) of the knee and hip: lessons learned from failures and opportunities for the future

    Karsdal, M., Michaelis, M., Ladel, C., Siebuhr, A., Bihlet, A., Andersen, J., Guehring, H., Christiansen, C., Bay-Jensen, A., and Kraus, V. Disease-modifying treatments for osteoarthritis (dmoads) of the knee and hip: lessons learned from failures and opportunities for the fut...

  3. [11]

    A., Deng, A., Baston, A., Ross, J., Idowu, E., et al

    Kraljevic, Z., Bean, D., Shek, A., Bendayan, R., Hemingway, H., Yeung, J. A., Deng, A., Baston, A., Ross, J., Idowu, E., et al. Foresight--generative pretrained transformer (gpt) for modelling of patient timelines using ehrs. arXiv preprint arXiv:2212.08072, 2022

  4. [12]

    Li, Y., Rao, S., Solares, J. R. A., Hassaine, A., Ramakrishnan, R., Canoy, D., Zhu, Y., Rahimi, K., and Salimi-Khorshidi, G. Behrt: transformer for electronic health records. Scientific reports, 10 0 (1): 0 7155, 2020

  5. [13]

    Hi-behrt: hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records

    Li, Y., Mamouei, M., Salimi-Khorshidi, G., Rao, S., Hassaine, A., Canoy, D., Lukasiewicz, T., and Rahimi, K. Hi-behrt: hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records. IEEE journal of biome...

  6. [14]

    McDermott, M., Nestor, B., Argaw, P., and Kohane, I. S. Event stream gpt: a data pre-processing and modeling library for generative, pre-trained transformers over continuous-time sequences of complex events. Advances in Neural Information Processing Systems, 36: 0 24322--24334, 2023

  7. [15]

    H., Levaux, H

    Nordo, A. H., Levaux, H. P., Becnel, L. B., Galvez, J., Rao, P., Stem, K., Prakash, E., and Kush, R. D. Use of ehrs data for clinical research: historical progress and current applications. Learning health systems, 3 0 (1): 0 e10076, 2019

  8. [16]

    S., Spotnitz, M., Chen, R., Perotte, A., and Natarajan, K

    Pang, C., Jiang, X., Kalluri, K. S., Spotnitz, M., Chen, R., Perotte, A., and Natarajan, K. Cehr-bert: Incorporating temporal information from structured ehr data to improve prediction tasks. In Machine Learning for Health, pp.\ 239--260. PMLR, 2021

  9. [17]

    Improving language understanding by generative pre-training

    Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018

  10. [18]

    Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction

    Rasmy, L., Xiang, Y., Xie, Z., Tao, C., and Zhi, D. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine, 4 0 (1): 0 86, 2021

  11. [19]

    E., Was, J., Li, Q., Bates, D

    Renc, P., Jia, Y., Samir, A. E., Was, J., Li, Q., Bates, D. W., and Sitek, A. Zero shot health trajectory prediction using transformer. NPJ Digital Medicine, 7 0 (1): 0 256, 2024

  12. [20]

    Pre-training of graph augmented transformers for medication recommendation

    Shang, J., Ma, T., Xiao, C., and Sun, J. Pre-training of graph augmented transformers for medication recommendation. arXiv preprint arXiv:1906.00346, 2019

  13. [21]

    J., Bihorac, A., and Rashidi, P

    Shickel, B., Tighe, P. J., Bihorac, A., and Rashidi, P. Deep ehr: a survey of recent advances in deep learning techniques for electronic health record (ehr) analysis. IEEE journal of biomedical and health informatics, 22 0 (5): 0 1589--1604, 2017

  14. [22]

    A., Corbin, C

    Steinberg, E., Jung, K., Fries, J. A., Corbin, C. K., Pfohl, S. R., and Shah, N. H. Language models are an effective representation learning technique for electronic health record data. Journal of biomedical informatics, 113: 0 103637, 2021

  15. [23]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

  16. [24]

    Ehrshot: An ehr benchmark for few-shot evaluation of foundation models

    Wornow, M., Thapa, R., Steinberg, E., Fries, J., and Shah, N. Ehrshot: An ehr benchmark for few-shot evaluation of foundation models. Advances in Neural Information Processing Systems, 36: 0 67125--67137, 2023

  17. [25]

    Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review

    Xiao, C., Choi, E., and Sun, J. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. Journal of the American Medical Informatics Association, 25 0 (10): 0 1419--1428, 2018

  18. [26]

    Transformehr: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records

    Yang, Z., Mitra, A., Liu, W., Berlowitz, D., and Yu, H. Transformehr: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records. Nature communications, 14 0 (1): 0 7857, 2023

  19. [27]

    R., Huang, S.-L., Ma, X., Chaudhari, A., Madaan, D., Almahmoud, E., Chopra, S., et al

    Zhu, W., Tang, H., Zhang, H., Rajamohan, H. R., Huang, S.-L., Ma, X., Chaudhari, A., Madaan, D., Almahmoud, E., Chopra, S., et al. Predicting risk of alzheimer’s diseases and related dementias with ai foundation model on electronic health records. medRxiv, 2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.