Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Exploring Scaling Laws for EHR Foundation Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Scaling laws hold for EHR foundation models: compute, data, and clinical accuracy follow predictable power laws.

desk verdict Useful first scaling-law study for EHR foundation models, but the headline exponents rest on a missing epoch/token accounting that makes them fragile. read the letter →

arxiv 2505.22964 v2 pith:EXDI54D5 submitted 2025-05-29 cs.CL cs.AIcs.LG

Sheng Zhang , Qin Liu , Naoto Usuyama , Cliff Wong , Tristan Naumann , Hoifung Poon This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords scalinglawselectronichealthrecordsfoundationmodelsIsoFLOPanalysisMIMIC-IVpowerlawzero-shotpredictionnext-token
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether transformer models trained on electronic health records improve predictably with compute and data, the way large language models do. Using patient timelines from MIMIC-IV, it trains a family of decoder-only transformers from about 1 million to 982 million parameters and shows that, under fixed compute budgets, validation loss traces parabolic IsoFLOP curves whose minima follow power laws in compute. It further shows zero-shot performance on ICU mortality and 30-day readmission tracks validation loss, and that gains saturate beyond 28 million parameters because the dataset's roughly $2.67 \times 10^8$ tokens run out. If correct, these scaling laws would let researchers choose compute-optimal model sizes for EHR foundation models and forecast the data needed for better clinical predictions.

What carries the argument

The central machinery is the IsoFLOP analysis from Hoffmann et al. (2022), transplanted to EHR timelines: fix total FLOPs per run, vary model size, and record validation loss; fit a parabola to each budget's loss-versus-size curve, then fit power laws to the resulting minima. This is carried by an autoregressive Llama-style transformer trained on ETHOS-style patient timelines, where clinical events are tokenized into ICD and ATC codes, quantile-binned lab values, and time-interval tokens, and each training batch respects patient boundaries. The power laws $N_{\mathrm{opt}} \propto C^{0.58}$ and $D_{\mathrm{opt}} \propto C^{0.44}$ are the objects that make resource planning predictive.

What would settle it

Rerun the IsoFLOP sweep while logging the exact number of epochs per configuration and the validation loss at each epoch. If, for the $8 \times 10^5$ TFLOP budget, models trained for many epochs start to overfit (validation loss rises while training loss falls), or if refitting the minima using only single-epoch runs changes the exponents $a \approx 0.58$ and $b \approx 0.44$ materially, then the claimed scaling-law exponents are artifacts of multi-epoch training rather than intrinsic EHR scaling behavior.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that autoregressive next-token prediction over tokenized patient timelines exhibits scaling behavior analogous to language modeling. For five fixed FLOP budgets, validation loss against parameter count forms a U-shaped IsoFLOP curve; fitting parabolas locates the optimal model size per budget. The optimal size and token count follow power laws $N_{\mathrm{opt}} \propto C^{0.58}$ and $D_{\mathrm{opt}} \propto C^{0.44}$, exponents that differ from the Chinchilla values, suggesting EHR scaling dynamics are distinct. Downstream, validation loss correlates strongly with zero-shot ICU-mortality and readmission AUC, and performance improves log-linearly with model size up to roughly 28M parameters before flattening when the model's data appetite exceeds the available MIMIC-IV corpus. The authors interpret the plateau as a data-scarcity effect, not a ceiling on EHR model scaling.

Load-bearing premise

The load-bearing premise is that the standard language-model cost formula (total FLOPs ≈ $6 \times \text{parameters} \times \text{tokens}$, one pass over the data) transfers to EHR timelines, even though MIMIC-IV's $2.67 \times 10^8$ tokens are too few for the largest IsoFLOP budgets without many repeated epochs, and the paper does not report those epoch counts.

Editorial extensions

If this is right

  • Compute-optimal EHR training: for a given FLOP budget, the fitted power laws specify the model size and token count that minimize validation loss, removing guesswork.
  • Validation loss can serve as a cheap proxy for downstream clinical utility in model selection, since it tracks zero-shot AUC for ICU mortality and 30-day readmission.
  • Scaling gains in clinical prediction are real but currently data-bound: performance stalls past roughly 28M parameters on MIMIC-IV's $2.67 \times 10^8$ tokens, and continuing gains require proportionally larger EHR corpora.
  • The exponents ($a \approx 0.58$, $b \approx 0.44$) imply EHR data has a different compute-optimal balance than natural language, so naively copying LLM ratios would waste compute.
  • The authors' boundary-preserving batching avoids cross-patient leakage that could otherwise corrupt next-token training and evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the power laws generalize to larger, multi-site EHR datasets, then a 'GPT moment' for structured clinical prediction would require pooling or federated corpora orders of magnitude larger than any single hospital system, since $D_{\mathrm{opt}}$ grows faster than linearly in compute.
  • The fitted exponents rest on single-pass Chinchilla cost accounting; because the MIMIC-IV token budget is far smaller than what the largest IsoFLOP runs call for, the paper's curves may blend multi-epoch training and overfitting. Repeating the IsoFLOP sweep with logged epoch counts, or with deduplicated or streamed tokens, would test whether the exponents are stable.
  • A natural next experiment is to test the same IsoFLOP methodology on other structured sequential data such as insurance claims, wearable sensor streams, or prescription registries, to see whether the EHR-specific exponents are a property of clinical codes or of modest data scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper trains decoder-only Llama-style transformers from scratch on tokenized MIMIC-IV patient timelines and studies their scaling behavior. It constructs IsoFLOP curves at five compute budgets (5e4 to 8e5 TFLOPs), fits parabolic loss-versus-model-size curves, and derives power laws N_opt ∝ C^0.58 and D_opt ∝ C^0.44. It then evaluates zero-shot ICU mortality and 30-day readmission performance, reporting log-linear gains up to 9M parameters and saturation beyond 28M parameters, which it attributes to insufficient training tokens.

Significance. If the scaling laws hold, the paper provides a valuable first step for EHR foundation models: a compute-optimal model-sizing recipe, a loss-based proxy for downstream clinical utility, and evidence that EHR data exhibit LLM-like scaling behavior. The work leverages a public benchmark (MIMIC-IV), preserves patient boundaries during batching, follows an established zero-shot evaluation protocol, and reports bootstrapped confidence intervals for downstream metrics. It also explicitly acknowledges that limited data scale may destabilize exponent estimates. However, the empirical core is currently vulnerable to a training-protocol ambiguity (the number of epochs per run) and to fitting choices that can bias the extracted exponents, so the central contribution is promising but not yet fully supported.

major comments (4)
  1. [§3 and §4.1, Table 1] The IsoFLOP curves set the number of training tokens as D = C/(6N) following the Chinchilla-style FLOPs estimate, but the entire training corpus contains only about 2.68e8 trainable tokens (Table 1). For the largest compute budgets, many plotted models imply D far exceeding the corpus size: at C = 8e5 TFLOPs, D is roughly 1.3e11 tokens for N = 1M and 1.5e10 tokens for N = 9M, corresponding to roughly 500 and 55 epochs, respectively. The paper never states whether these runs used multiple epochs, early stopping, or reduced effective compute, so the low-loss branches of the 4e5 and 8e5 TFLOPs curves may reflect overfitting rather than clean scaling. This directly affects the fitted minima and hence the exponents a ≈ 0.58 and b ≈ 0.44. Please report the actual number of tokens processed per run (or, equivalently, epochs and early-stopping criteria), and show that the results are robust when restricting to budgets where D is within a single pass over the corpus, or provide explicit multi-epoch analysis.
  2. [§4.1, first paragraph] The text states that, for each compute budget, the authors 'retain and plot the six models with the lowest validation loss' before fitting a parabola. Selecting the six lowest-loss points is a post-hoc truncation that can systematically skew the quadratic fit and, more importantly, the location of its minimum whenever the loss landscape is noisy or asymmetric. The manuscript does not report the total number of model sizes trained per budget, the losses of the discarded runs, or any alternative fitting procedure. Please provide the full IsoFLOP data (e.g., in supplementary tables or scatter plots with all runs), fit the parabolas on all runs, and demonstrate that the fitted minimum is stable under this selection; otherwise the optimal-size locus in the center and right panels of Figure 1 is not justified.
  3. [§4.1, Figure 1 center/right] The power-law exponents are obtained by fitting only five budget-level points, yet no confidence intervals, standard errors, or goodness-of-fit measures are reported. Given the authors' own observation in the same section that the limited data scale 'may reduce the precision and stability of exponent estimation,' the reader cannot assess whether a ≈ 0.58 and b ≈ 0.44 are meaningfully different from each other or from the Chinchilla values a = 0.5, b = 0.5. Please add uncertainty quantification (e.g., bootstrap or leave-one-budget-out analysis) and report residuals or an error metric for both fits.
  4. [§4.2, Figure 2 (right)] The saturation claim beyond 28M parameters compares models trained under different protocols: the smaller IsoFLOP-optimal models are trained under fixed compute budgets, while the larger models (28M to 982M) are trained 'until early stopping.' This protocol mismatch confounds the interpretation that saturation is caused purely by data scarcity. Please report the number of epochs and tokens consumed for the large models, and, if feasible, evaluate at least one larger model trained with its compute-optimal token count to separate data scarcity from training-protocol mismatch.
minor comments (4)
  1. [§2.1] Typo: 'transfomer' should be 'transformer'.
  2. [§4.1] The sentence 'the number of training tokens is determined the model’s FLOPs consumption' is missing a preposition; it should read 'determined by the model’s FLOPs consumption.'
  3. [Table 1] The row 'training example length' reports a minimum of 37 tokens, but §2.3 describes segmenting each patient timeline independently; please clarify how sequences shorter than the context length (2048) are handled during batching and loss computation.
  4. [General] No reproducibility statement is included for the training code, model configurations, or hyperparameter settings. Given that the paper's value depends on the scaling curves being reproducible, adding a statement about code/configuration release would strengthen the contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scaling exponents and IsoFLOP curves are empirical fits to the paper's own training runs, with no load-bearing self-citation or definitional reduction.

full rationale

The paper's central claims are empirical: it fixes compute budgets, trains transformer models of varied sizes on MIMIC-IV timelines, measures validation loss, fits parabolas to the resulting IsoFLOP profiles, and then fits power laws N_opt ∝ C^a and D_opt ∝ C^b to the minima, reporting a≈0.58 and b≈0.44. These exponents are extracted from measured loss curves rather than assumed by construction. The parsecs and downstream correlations are also direct measurements on held-out test data. The only externally imported methodology is the standard Chinchilla FLOPs accounting and the ETHOS tokenization/evaluation codebase, both of which are external prior work, not self-citations that smuggle in the conclusion. The saturation claim for models beyond 28M parameters is an extrapolation of the fitted D_opt power law, not a definitional identity: the paper explicitly hypothesizes data scarcity and notes that the MIMIC-IV corpus contains only about 2.67e8 tokens, far fewer than the implied optimal token counts. Whether the IsoFLOP runs actually consumed the nominal compute budgets given repeated epochs is a legitimate correctness and reproducibility concern, but it does not make the derivation circular. Because every claimed quantitative relationship is fit from data reported in the paper, the derivation chain is self-contained and no step reduces to its own inputs.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a handful of empirical fitting steps and domain assumptions inherited from ETHOS and Hoffmann et al. No new entities are introduced.

free parameters (2)
  • exponent_a (optimal model size vs compute) = 0.58
    Fitted from the optimal points of the IsoFLOP parabolic curves across 5 compute budgets (Section 4.1).
  • exponent_b (optimal tokens vs compute) = 0.44
    Fitted from the optimal token counts versus compute across 5 budgets (Section 4.1).
assumptions (5)
  • domain assumption Autoregressive next-token prediction over tokenized patient timelines captures clinically relevant structure
    The paper adopts ETHOS's tokenization and the autoregressive objective (Section 2.3). If this tokenization discards important clinical context, the scaling behavior measured on these tokens may not reflect true EHR modeling.
  • domain assumption The Chinchilla FLOPs estimate (total FLOPs = 6 * N * D) is accurate for these models
    Used in Section 3 to convert compute budgets into token counts. The paper checks against the PaLM estimate and says the difference is small, but does not show the calculation.
  • domain assumption Validation loss on the held-out set is a reliable proxy for downstream clinical utility
    Tested in Section 4.2 via zero-shot AUC correlation, but the scaling law fits themselves rely on validation loss as the performance metric.
  • domain assumption The compute budget range (5e4 to 8e5 TFLOPs) is wide enough to reveal stable scaling exponents
    The exponents are fit from only 5 budgets; the paper acknowledges data scale may reduce precision.
  • domain assumption The models are trained for a single epoch over the token set, or that multiple epochs do not change the scaling law form
    Unstated; MIMIC-IV has 267M tokens and the largest budgets require more tokens than available, so multi-epoch training is likely.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Scaling Laws for EHR Foundation Models." pith.science (2026). https://pith.science/paper/EXDI54D5

@misc{pith2026250522964,
  author       = {Pith},
  title        = {Pith review of: Exploring Scaling Laws for EHR Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EXDI54D5}},
  note         = {Machine review of arXiv:2505.22964}
}
read the original abstract

The emergence of scaling laws has profoundly shaped the development of large language models (LLMs), enabling predictable performance gains through systematic increases in model size, dataset volume, and compute. Yet, these principles remain largely unexplored in the context of electronic health records (EHRs) -- a rich, sequential, and globally abundant data source that differs structurally from natural language. In this work, we present the first empirical investigation of scaling laws for EHR foundation models. By training transformer architectures on patient timeline data from the MIMIC-IV database across varying model sizes and compute budgets, we identify consistent scaling patterns, including parabolic IsoFLOPs curves and power-law relationships between compute, model parameters, data size, and clinical utility. These findings demonstrate that EHR models exhibit scaling behavior analogous to LLMs, offering predictive insights into resource-efficient training strategies. Our results lay the groundwork for developing powerful EHR foundation models capable of transforming clinical prediction tasks and advancing personalized healthcare.

Figures

Figures reproduced from arXiv: 2505.22964 by the authors.

Figure 1
Figure 1. (left) illustrates the IsoFLOP profiles. For each FLOP budget, we train all model configurations described in Section 3, then retain and plot the six models with the lowest validation loss against their corresponding parameter counts. This selection ensures a diverse yet representative sampling of model sizes while filtering out clear outliers with substantially higher losses. We observe that for each compute budget… view at source ↗
Figure 2
Figure 2. Zero-shot performance on downstream healthcare tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Zero-shot performance of models of different sizes on ICU mortality. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Zero-shot performance of models of different sizes on 30-day hospital readmission. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Pretraining EHR Foundation Models with Patient-Aware Sampling

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Patient-aware sampling of pretraining windows, with patients weighted by a tunable exponent, improves downstream AUROC/AUPRC over a global token-stream baseline in autoregressive EHR models.

  2. Autoregressive EHR Foundation Models with Multimodal Inputs

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Latent compression improves multimodal fusion, but ICU mortality predictions from EHR plus ECG, CXR, and notes are generally no better than EHR-only.

Reference graph

Works this paper leans on

17 extracted references · 5 canonical work pages · cited by 2 Pith papers

  1. [7]

    Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al

    URL https://arxiv.org/abs/2203.15556. Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1,

    arXiv

  2. [8]

    Zeljko Kraljevic, Dan Bean, Anthony Shek, Rebecca Bendayan, Harry Hemingway, Joshua Au Yeung, Alexander Deng, Alfred Balston, Jack Ross, Esther Idowu, et al

    URL https://arxiv.org/abs/2001.08361. Zeljko Kraljevic, Dan Bean, Anthony Shek, Rebecca Bendayan, Harry Hemingway, Joshua Au Yeung, Alexander Deng, Alfred Balston, Jack Ross, Esther Idowu, et al. Fore- sight—a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study. The Lancet D...

    arXiv 2001

  3. [9]

    URL https://arxiv.org/abs/2303.08774. OpenAI. Openai gpt-4.5 system card. https://cdn.openai.com/ gpt-4-5-system-card-2272025.pdf ,

    arXiv

  4. [10]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al

    Accessed: 2025-02-27. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training,

    work page 2025

  5. [11]

    doi: 10.1038/s41746-024-01235-0

    ISSN 2398-6352. doi: 10.1038/s41746-024-01235-0. URL https://www.nature.com/articles/s41746-024-01235-0 . 10 Preprint. Noam Shazeer. Glu variants improve transformer,

    doi:10.1038/s41746-024-01235-0

  6. [12]

    URL https://arxiv.org/abs/2002. 05202. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding,

    work page 2002

  7. [13]

    URL https://arxiv.org/ abs/2104.09864. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models,

    arXiv

  8. [14]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    URL https://arxiv.org/abs/2302.13971. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need,

    arXiv

Show all 17 references
  1. [15]

    URL https://arxiv.org/abs/1706.03762. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen ...

    2020 arXiv

  2. [16]

    URL https://www.aclweb.org/ anthology/2020.emnlp-demos.6

    Association for Computational Linguistics. URL https://www.aclweb.org/ anthology/2020.emnlp-demos.6. World Health Organization. ATC/DDD Classification. https://www.who.int/ tools/atc-ddd-toolkit/atc-classification ,

    2020

  3. [17]

    Accessed: 2025-05-16

    URL https://www.who.int/tools/ atc-ddd-toolkit/atc-classification . Accessed: 2025-05-16. Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A Pfeffer, Jason Fries, and Nigam H Shah. The shaky foundations of large language models and fo...

    2025

  4. [2019]

    org/abs/1810.04805

    URL https://arxiv. org/abs/1810.04805. Aaron Grattafiori and team. The llama 3 herd of models,

    arXiv

  5. [2020]

    Centers for Medicare & Medicaid Services

    URL https://arxiv.org/abs/2005.14165. Centers for Medicare & Medicaid Services. ICD-10 Codes. https://www.cms.gov/ medicare/coding-billing/icd-10-codes ,

    2005 arXiv

  6. [2022]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

    URL https://arxiv.org/abs/2204.02311. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding,

    arXiv

  7. [2023]

    Yoshua Bengio, R´ejean Ducharme, Pascal Vincent, and Christian Jauvin

    URL https://arxiv.org/abs/2305.13245. Yoshua Bengio, R´ejean Ducharme, Pascal Vincent, and Christian Jauvin. A neural proba- bilistic language model. Journal of machine learning research, 3(Feb):1137–1155,

    arXiv

  8. [2024]

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al

    URL https://arxiv.org/ abs/2407.21783. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autore- gressive generative modeling. arXiv preprint arXiv:2010.14701,

    2010 arXiv

  9. [2025]

    Accessed: 2025-05-16

    URL https://www.cms.gov/medicare/ coding-billing/icd-10-codes . Accessed: 2025-05-16. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.