{"id":"d1ab39ce-bfc4-4cc5-a364-6b796dfbe570","arxiv_id":"2608.02939","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Federated generative event models trained across three independent ICU datasets match most centralized multi-site performance and are substantially more transportable across hospitals than LightGBM.","lead":"Federated training of tokenized generative event models across 122,251 ICU stays from three health systems preserves most of the performance of centralized multi-site training while keeping records local. GEM models also transfer across hospitals with much smaller performance drops than standard supervised models, though multi-site training helps most when a hospital has little local data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MIMIC-frozen tokenizer is the load-bearing premise: if its decile bins misalign UCMC/NU raw values, the reported cross-site penalties and federated gains may partly reflect tokenization mismatch rather than transportability; the paper acknowledges this but the main claims still depend on it.","rationale":"The reader's weakest-assumption analysis identifies the MIMIC-learned tokenizer and frozen decile bins as the load-bearing premise, and I agree. The paper's own Figure B1 and the limitations paragraph support the reality of the risk: cross-site quantile-token distributions differ substantially, and the authors concede that fixing MIMIC bins 'may have affected institutions differently.' All of the headline quantitative claims - cross-site transfer penalties, federated-versus-centralized deficits, and the value of multi-site training at data-limited sites - are computed inside this fixed token space. If the bins misalign raw values across sites, the observed penalties could reflect tokenization artifact, and the apparent similarity between UCMC and NU could artificially lower the mean cross-site penalty. The concern is not that the authors hid this; they explicitly acknowledge it. But acknowledgment without quantification leaves the central claim conditional. I do not think the concern overturns the paper's qualitative conclusions, because the supervised baselines share the same token space and the federated-versus-centralized comparison is internally controlled. The remaining issues raised by the reader - the implausible FedAdam hyponatremia PR-AUC values and the lack of training-seed variance - are real but less load-bearing: correcting the FedAdam anomaly would make FedAdam look even worse, and seed variance would mostly affect the precision of small gaps rather than the large GEM-versus-LightGBM transportability margins. Therefore the appropriate verdict remains the reader's CONDITIONAL, and I would not move it.","tokens_in":23901,"tokens_out":14035,"duration_ms":137637,"concrete_test":"Rerun the main within-site/cross-site/federated comparison with a site-calibrated tokenizer: learn decile bins separately on each site's training split, then define a common token space by mapping each site's quantile bins onto MIMIC quantile boundaries, so that token Q_k denotes the k-th decile of that site's own training values. If GEM-*'s mean cross-site ROC-AUC penalty (Table 4) or the FedAvg/FedAvgM deficits versus GEM-all (Table 5) change by more than ~0.01, or if LightGBM's penalty drops comparably, then the MIMIC-frozen bins are a first-order driver of the transportability and federated-recovery claims. To keep the experiment tractable, restrict the recalibration to the lab/vital categories most predictive of the 12 outcomes: sodium, potassium, hemoglobin, creatinine, blood pressure, and heart rate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states that the 1344-token vocabulary and category-value decile cutoffs were learned on the MIMIC training split and then frozen and applied to all UCMC and NU splits. The central cross-site and federated comparisons all run through this shared token space. If a token such as LAB-RES//potassium_Q7 denotes different raw value ranges at different sites, then a model trained at UCMC and evaluated at NU can lose performance because of token-semantic misalignment rather than because of clinically meaningful transportability. Conversely, UCMC and NU are both non-MIMIC, contemporary health systems; they may share a common \"MIMIC-bin shift\" that artificially lowers the UCMC<->NU transfer penalties and hence the mean cross-site penalty in Table 4. The paper's own Figure B1 shows substantial cross-site quantile-bin distribution differences, and the limitations paragraph explicitly concedes that fixing MIMIC bins 'may have affected institutions differently.' Because LightGBM and LR consume the same tokenized features, the relative GEM-versus-baseline comparison is not fully confounded; however, the magnitude of the claimed transportability advantage (0.025 vs 0.079 ROC-AUC penalty) and the absolute claim that federated models 'preserve most centralized performance' (deficits 0.010-0.024) could shift if the binning were calibrated per site or to a pooled distribution. This is the most load-bearing premise among the paper's claims, because every cross-site conclusion is evaluated inside this token space.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript evaluates federated training of tokenized generative event models (GEMs) on intensive care data from three health systems (UCMC, NU, MIMIC) harmonized to CLIF-2.1, totaling 122,251 hospitalizations and 12 post-24-hour prediction tasks. Models are compared in within-site, cross-site, centralized, and federated (FedAvg, FedAvgM, FedAdam) configurations using representation-based inference. The principal claims are that GEM representations are substantially more transportable than LightGBM or logistic regression (cross-site ROC-AUC penalties of 0.025 versus 0.079), that FedAvg and FedAvgM recover most of the centralized multi-site performance (deficits of 0.010-0.024 ROC-AUC), that gains saturate within 5-10 communication rounds, and that multi-site training is most valuable when local data are limited.","tokens_in":24184,"tokens_out":5378,"duration_ms":41509,"significance":"If the quantitative claims hold, the paper provides a valuable multi-site benchmark separating federated optimization losses from representation transfer losses in a clinically realistic setting. The study has notable strengths: equal one-epoch training budgets for federated and centralized comparisons, per-outcome exclusion of patients with pre-24h events, bootstrap confidence intervals, a released code framework (coreopsis), and an evaluation design that does not fit constants to the held-out data. The main claims are falsifiable and of direct practical interest for ICU prediction. However, the magnitude of several headline numbers rests on a MIMIC-frozen tokenizer and on single-seed comparisons of small differences, so the quantitative conclusions need additional robustness work.","major_comments":[{"comment":"The tokenizer and decile bins are learned exclusively from MIMIC training data and then frozen for UCMC and NU (Section 3.2). Because every cross-site and federated comparison runs through this shared token space, the transfer penalties in Table 4 and the federated deficits in Table 5 could partly reflect token-semantic misalignment between MIMIC-derived bins and UCMC/NU raw values rather than model transportability. Figure B1 documents substantial cross-site differences in quantile-token distributions, and the limitations paragraph concedes that fixing MIMIC bins 'may have affected institutions differently.' I request a robustness analysis that re-learns bins per site or on a pooled sample and reports whether the GEM transportability advantage and the FedAvg deficits persist.","section":"Section 3.2, Figure B1"},{"comment":"The claim that centralized multi-site training yields only modest improvements is stated without significance testing. The site-specific ROC-AUC gains for GEM-* are 0.004 at UCMC, 0.009 at NU, and 0.003 at MIMIC, all well within the reported bootstrap confidence intervals of the local models. Please add paired bootstrap tests (as is already done for FedAvg versus FedAvgM) for centralized versus local training, at least for the headline ROC-AUC comparisons, and report the resulting p-values.","section":"Section 4.2, Tables 3-4"},{"comment":"The crossover claim that local models become competitive after 'approximately 3,000 training examples' is not accompanied by an estimator, confidence interval, or definition of competitiveness. Please define a margin (e.g., within 0.005 ROC-AUC of the multi-site comparator), estimate the crossover from the learning curves, and provide uncertainty for it.","section":"Section 4.4"},{"comment":"The FedAdam PR-AUC values for hyponatremia (0.501, 0.501, 0.422) are one to two orders of magnitude larger than those of every other model for the same outcome (typically 0.01-0.05). This appears to be an error in the appendix table or in the FedAdam evaluation, and it will distort the aggregate FedAdam PR-AUC deficits reported in Table 5. Please correct the anomaly and re-run the FedAdam comparisons.","section":"Table A6 (hyponatremia)"},{"comment":"All results are based on a single training run per configuration, so the confidence intervals reflect only patient-level resampling and not training stochasticity. For a 76.9M-parameter transformer, the key differences (FedAvg/FedAvgM deficits of 0.010-0.024 ROC-AUC, centralized gains of 0.003-0.009) are small enough that seed variation may change the qualitative conclusions. Please report at least three seeds for the central configurations, or demonstrate that seed-to-seed variability is negligible.","section":"Sections 3.3-3.5"}],"minor_comments":[{"comment":"The limitations paragraph contains the sentence 'The authors are incredibly grateful to the teams at Beth-Israel/MIT, Northwestern Medicine, and UCMC who have made data centralization possible for this experiment,' which reads as an acknowledgment and should be moved to the Acknowledgments section.","section":"Section 5 (limitations)"},{"comment":"The caption mentions 'mean Wasserstein-1 distance between sites' without defining how the distance is computed over quantile-token distributions; please specify the exact comparison procedure.","section":"Figure B1 caption"},{"comment":"The statement that no significant difference between FedAvg and FedAvgM was detected (p>0.25 for all six comparisons) is not accompanied by details of the bootstrap test or the p-values; please report the procedure and the full set of p-values.","section":"Table 5 and Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an empirical study whose central claims are logically coherent and not circular. The main risk is that the headline magnitudes are fragile to tokenization choices and seed variation; the requested robustness analyses are within scope. I would also suggest that the editor ask for a code and data audit of the FedAdam hyponatremia anomaly before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth your time: it is one of the first evaluations of federated GEM training on genuinely separate health systems (UCMC, NU, MIMIC-IV via CLIF), not a single-site simulation. The headline result—GEM representations transfer far better than LightGBM or LR (0.025 vs 0.079 ROC-AUC cross-site penalty)—is new and, as far as I can tell from the tables, supported. The decomposition of transfer penalty versus federated optimization deficit is the most useful contribution: it tells the field that the bottleneck is transportability, not aggregation.\n\nThe work is careful in the ways that matter: fair one-epoch training budgets, bootstrap CIs, per-outcome post-24-hour exclusions, held-out data used only for evaluation, and released code (coreopsis). Credit where due: this is a solid empirical benchmark, not a hype piece.\n\nSoft spots, in proportion:\n\n1. The MIMIC-frozen tokenizer is the main structural caveat. The stress-test note is right that decile bins learned on MIMIC could misalign raw values at UCMC and NU, and the paper's own Figure B1 shows substantial cross-site bin shifts. However, this does not sink the relative GEM-versus-baseline comparison, since LightGBM and LR consume the same token space. It could shift the absolute penalties and the federated-deficit numbers, and the paper says so in the limitations. A sensitivity check with per-site or pooled bins would settle it. Fair to ask for; not fatal.\n\n2. The FedAdam hyponatremia PR-AUC rows in Table A6 look wrong: 0.501 at all three sites versus ~0.01 for every other model. That smells like an implementation bug or label leak, not a real result. It needs an explanation before publication.\n\n3. No multiple training seeds. The bootstrap CIs cover patient resampling but not optimization stochasticity, and one-epoch training from scratch can be seed-sensitive. Minor-to-moderate.\n\n4. Centralized gains are not significance-tested, and the \"~3,000 examples\" crossover is asserted without an estimator. Minor, but easy to tighten.\n\nBottom line: the qualitative conclusions—GEMs are more portable, FedAvg/FedAvgM approach centralized training, and multi-site benefit concentrates at data-limited sites—hold up despite these issues. The paper is appropriate for a serious ML-for-health venue and would be a useful citation for anyone working on federated EHR foundation models. I'd send it to peer review rather than desk-reject, asking for the FedAdam anomaly to be fixed, seed variance reported, and a tokenizer sensitivity analysis added.\n\nBest.","headline":"A real three-health-system evaluation of federated GEMs whose central claims largely hold; the MIMIC-frozen tokenizer is a legitimate caveat, not a fatal flaw.","tokens_in":24803,"tokens_out":1566,"would_cite":true,"duration_ms":16750,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated training of tokenized generative event models nearly matches centralized training on pooled data, with most gains in 5-10 rounds.","keywords":["federated learning","generative event models","electronic health records","representation-based inference","cross-site transportability","clinical prediction","FedAvg","tokenization"],"falsifier":"Re-tokenize each hospital's records with bins and vocabulary learned on that hospital's own training split, retrain GEMs, and recompute the cross-site transfer penalties and FedAvg-versus-centralized gaps; if the GEM transportability advantage shrinks or the federated gap changes materially, the paper's headline numbers are in part artifacts of the fixed tokenizer rather than properties of the model or federated algorithm.","tokens_in":23681,"feed_emoji":"🏥","tokens_out":7995,"duration_ms":68856,"temperature":0.7,"pith_summary":"This paper asks whether hospitals can jointly train a generative model of patient event sequences without sharing records, and whether the resulting model is worth using at any single hospital. It reports that on 122,251 ICU stays from three independent health systems, a generative event model (GEM) trained to predict the next clinical event transfers across sites with a ROC-AUC penalty of 0.025, versus 0.079 for a gradient-boosted baseline, and that FedAvg and FedAvgM recover most of the performance of training on centrally pooled data within 5-10 communication rounds. The paper's main boundary condition is that multi-site data helps most when the target hospital has little local data: after a few thousand local examples, site-trained models catch up, and centralized pooling offers only modest gains over strong local models. A sympathetic reader would take the central claim to be that federated aggregation is technically solved for this setting and that the remaining bottleneck is learning representations that actually improve a specific target site.","feed_headline":"Federated models nearly match centralized EHR training","feed_subtitle":"Cross-site penalty for generative event models is 0.025 ROC-AUC versus 0.079 for gradient-boosted trees, and 5-10 rounds suffice.","key_machinery":"The load-bearing object is the tokenized generative event model (GEM): a 76.9-million-parameter transformer trained from scratch to predict the next token in a hospitalization's event sequence, where tokens encode demographics, transfers, lab orders and results, vitals, medications, and other care events in a shared ICU data format with numerical values binned into deciles. The companion mechanism is representation-based inference: the sequence is truncated at 24 hours, the model's final hidden layer is extracted as a fixed vector, and a logistic regression per outcome is trained on those vectors. Federated learning enters as weighted averaging of model weights (FedAvg, plus momentum and adaptive variants), with each site training on a one-nth fraction of its data per round; the measured saturation by 5-10 rounds is what makes the federated claim practical.","core_discovery":"On the paper's own terms, the discovery is that pretraining a transformer to predict the next token in a shared, expert-mapped clinical event sequence produces patient representations that are markedly more portable across hospitals than representations from conventional supervised models. This portability is quantified as a 0.025 ROC-AUC average transfer penalty for the extended-trained GEM, compared with 0.079 for gradient-boosted trees and 0.077 for logistic regression; the PR-AUC penalty is 0.027 versus 0.089. The paper also claims that federated averaging (FedAvg and FedAvgM) comes within 0.010-0.024 ROC-AUC of centralized GEM training, while FedAdam fails by 0.13-0.17 ROC-AUC, and that nearly all federated benefit appears within 5-10 rounds. It further claims that centralized multi-site training improves on complete local training by only small amounts (0.003-0.009 ROC-AUC for the optimized GEM), so the practical payoff of multi-site learning is concentrated at data-limited sites.","pith_inferences":["One testable consequence not pursued in the paper: if the cross-site penalty is driven by the frozen tokenizer, then learning site-specific value bins with a shared token semantics (or matching bins by rank across sites) should reduce the already-small GEM penalty further; this could be checked by retraining with per-site bins.","The 5-10 round saturation suggests communication schedules could be made adaptive, stopping federated updates once client weights stabilize, without losing accuracy; this is not tested in the paper.","Because only representation-based inference was used, the transportability conclusion may not extend to generative inference or supervised fine-tuning, where cross-site differences in documentation style could be amplified; that is an open question the paper acknowledges.","A practical deployment reading, implicit in the results, is that federated models should be positioned as onboarding tools for data-limited sites rather than permanent replacements for local training; if correct, procurement decisions should emphasize easy participation over massive federation scale."],"forward_implications":["A hospital joining a federation with little local data can expect a usable head start from a model trained at other sites; after a few thousand local examples, models trained on local data alone become competitive.","Federated GEM training requires only about 5-10 communication rounds to capture most of the benefit, so communication overhead need not be a barrier.","Server-side adaptive optimization such as FedAdam is counterproductive for these models; simple weighted averaging with or without momentum is the right default.","The main remaining gap to centralized training is not the federated aggregation step but the limited transferability of heterogeneous multi-site data to a target institution.","Representations from generative pretraining, even with only one epoch of training, transfer across institutions far better than supervised baselines, though extended training raises absolute performance."],"supporting_citations":[{"why":"Defines the common ICU data format whose expert mappings let all sites share one event vocabulary.","marker":"14,15"},{"why":"Establishes the GEM pretraining objective and event-token workflow on which the model is built.","marker":"16,17"},{"why":"Supplies the representation-based inference setup: fixed-length vectors from truncated sequences feed per-outcome classifiers.","marker":"19-23"},{"why":"One of the three datasets; its training split is used to learn the token vocabulary and decile bins.","marker":"33"},{"why":"FedAvg, the weighted-averaging algorithm that defines the federated training protocol.","marker":"36"},{"why":"FedAvgM, the momentum variant that comes closest to centralized performance.","marker":"37"},{"why":"FedAdam, the adaptive server optimizer whose large deficits set the comparison for averaging methods.","marker":"60"},{"why":"The gradient-boosted tree baseline that anchors the within-site and cross-site comparisons.","marker":"63"}],"fun_headline_variants":["Federated EHR training: near-central accuracy in 5-10 rounds","Cross-site EHR penalty: 0.025 for generative, 0.079 for trees","Generative event models transfer better across hospitals than supervised","Federated GEMs keep cross-site AUC loss to just 0.025","Federated GEMs: small cross-site loss, fast convergence to central quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire cross-site comparison assumes that a token vocabulary and numerical value bins learned on one hospital's training split and then frozen for the other two hospitals preserve clinical meaning equally at all sites, so that differences in performance reflect model transportability rather than tokenization mismatch.","fun_headline_variants_meta":{"raw":{"variants":["Federated EHR training: near-central accuracy in 5-10 rounds","Cross-site EHR penalty: 0.025 for generative, 0.079 for trees","Generative event models transfer better across hospitals than supervised","Federated GEMs keep cross-site AUC loss to just 0.025","Federated GEMs: small cross-site loss, fast convergence to central quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000907,"raw_usage":{"total_tokens":3933,"prompt_tokens":1014,"completion_tokens":2919,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":2816}},"tokens_in":630,"tokens_out":2919,"duration_ms":19558,"temperature":1.0,"reasoning_tokens":2816,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:54:59.445344+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-tokenize each hospital's records with bins and vocabulary learned on that hospital's own training split, retrain GEMs, and recompute the cross-site transfer penalties and FedAvg-versus-centralized gaps; if the GEM transportability advantage shrinks or the federated gap changes materially, the paper's headline numbers are in part artifacts of the fixed tokenizer rather than properties of the model or federated algorithm.","supporting_citations":[{"cited_title":"MIMIC-IV, a freely accessible electronic health record dataset,","cited_arxiv_id":null,"evidence_quote":"One of the three datasets; its training split is used to learn the token vocabulary and decile bins."},{"cited_title":"Communication- efficient learning of deep networks from decentralized data,","cited_arxiv_id":null,"evidence_quote":"FedAvg, the weighted-averaging algorithm that defines the federated training protocol."},{"cited_title":"Measuring the effects of non-identical data distribution for federated visual classification,","cited_arxiv_id":null,"evidence_quote":"FedAvgM, the momentum variant that comes closest to centralized performance."},{"cited_title":"Adaptive federated optimization,","cited_arxiv_id":null,"evidence_quote":"FedAdam, the adaptive server optimizer whose large deficits set the comparison for averaging methods."},{"cited_title":"Lightgbm: A highly efficient gradient boosting decision tree,","cited_arxiv_id":null,"evidence_quote":"The gradient-boosted tree baseline that anchors the within-site and cross-site comparisons."}],"review_version":1}