REVIEW 6 minor 3 cited by
EEG foundation models encode dataset identity, not portable disease information, on clinical benchmarks
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 01:27 UTC pith:IGP2XJR4
load-bearing objection Careful, honest negative-control benchmark showing frozen EEG foundation models underperform classical features on an external Korean cohort and encode dataset identity at ceiling; credible as stated, but the headline CAUEEG gap rests on recording-level folds and awaits code release.
What EEG Foundation Models Encode: Dataset Identity and a Negative-Control Suite for Clinical Benchmarks
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Apparent transfer gains of EEG foundation models do not survive symmetric, controlled evaluation. On the Korean CAUEEG dementia-staging cohort, classical features reach 0.734 macro-AUROC versus 0.699 (BIOT), 0.669 (CBraMod), and 0.568 (REVE) under identical 19-channel five-fold splits; a patient-disjoint sensitivity preserves the ordering. Frozen REVE embeddings let a linear probe decode dataset identity at or near ceiling across Western–Korean and Western–Western pairs, even after PCA-50 and after removing line-frequency and amplitude-scale information, establishing dataset membership rather than a causal site or population effect. A matched random-initialized encoder exceeds pretrained REV
What carries the argument
The paper's central instrument is a negative-control suite: symmetric clean-montage construction so foundation-model and classical features see identical channels and folds; stronger classical comparators including enhanced spectral features and an MLP head; label permutation and scrambled-label fine-tuning to expose label-independent gains; matched random initialization to separate architecture from pretraining; and a dataset-identity probe, a linear classifier that decodes which dataset produced a recording from frozen embeddings. The dataset-identity probe does the load-bearing work: near-ceiling decodability across Western–Korean and Western–Western pairs, robust to PCA-50 and to removal
Load-bearing premise
The headline external-cohort result depends on recording-level five-fold validation because CAUEEG exposes no patient identifier, so the central comparison holds only if the public patient-overlap annotation correctly identifies all repeated recordings; if overlap differs between the classical and foundation-model pipelines, the reported gap could be partly a leakage artifact.
What would settle it
Run a fully patient-disjoint evaluation on CAUEEG with a verified patient map, or on a new external cohort with explicit patient identifiers and identical 19-channel preprocessing; if any frozen foundation model matches or exceeds the classical 0.734 macro-AUROC, and the classical-over-REVE ordering reverses, the paper's central claim is falsified.
If this is right
- Published clinical gains of EEG foundation models should be re-audited with montage-matched folds, patient-disjoint splits, and stronger classical baselines before being attributed to portable disease knowledge.
- Cross-population deployment of frozen EEG foundation models would be expected to underperform hand-crafted features unless specific adaptation and scrambled-label controls are passed.
- The task-specific seizure-detection result (pretrained vs random-init separation, label-permutation collapse) indicates that frozen representations may still carry event-related signal, but its scope should be verified with an amplitude-aware classical comparator and paired subject-level statistics.
- Dataset identity being decodable at ceiling implies that benchmark scores across datasets are not directly comparable; recording provenance must be treated as a confound in any cross-dataset leaderboard.
- A minimal reporting protocol — clean symmetric channels, patient-disjoint evaluation or leakage sensitivity, stronger comparators, random init, label permutation, scrambled-label adaptation — becomes the standard bar for clinical EEG foundation-model claims.
Where Pith is reading between the lines
- A direct extension would be to test whether the dataset-identity signal is driven by country, device, or recording style by training a domain-adversarial classifier that removes dataset-specific directions and then re-measuring diagnosis decodability; the paper leaves this causal decomposition untested.
- If dataset membership is near-ceiling decodable, multi-site clinical trials should publish per-site stratified results; a foundation model that wins on average could still be reading site-specific artifacts that happen to correlate with outcomes.
- A testable prediction is that on a truly patient-disjoint, multi-site cohort with identical hardware and protocol, the classical-over-foundation-model gap on dementia staging would shrink or reverse; collecting such a cohort would settle whether the CAUEEG deficit is about population or about dataset shift.
- The scrambled-label LoRA result (real equals null) suggests that any apparent cross-population adaptation gain should be reported with a label-shuffled control; future studies that omit this control should be read with caution.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper is an audit-style benchmark of seven EEG foundation models (LaBraM, EEGMamba, CBraMod, REVE, LEAD, BENDR, BIOT) on five clinical tasks across four benchmark datasets, with a Korean external-cohort stress test (CAUEEG). Primary comparisons use frozen embeddings with linear probes under subject-disjoint LOSO or grouped five-fold CV where identifiers exist, and recording-level five-fold CV with a patient-disjoint sensitivity for CAUEEG. The headline results are: (i) on a symmetric clean 19-channel CAUEEG Normal/MCI/Dementia evaluation, classical features reach 0.734 macro-AUROC versus 0.699 for BIOT, 0.669 for CBraMod, and 0.568 for REVE, with the classical-over-REVE ordering preserved on an n=172 patient-disjoint subset; (ii) dataset identity is linearly decoded from frozen REVE embeddings at or near ceiling across Western–Korean and Western–Western dataset pairs, even after PCA-50 and removal of line-frequency/amplitude information; (iii) on CHB-MIT cross-subject ictal detection, pretrained REVE reaches 0.793 versus 0.701 for random initialisation and 0.505 for raw random features, while label permutation collapses the probe to chance. The paper also documents that scrambled-label LoRA reproduces apparent adaptation gains, that enhanced classical features match or exceed REVE on Alzheimer's detection, and that many apparent findings change after montage matching and stronger comparators. The authors are unusually transparent: they correct an earlier asymmetric CAUEE
Significance. If the results are taken at face value, this is a significant contribution to the clinical EEG foundation-model literature. The paper provides direct evidence that frozen EEG-FM representations can encode dataset identity rather than portable disease information, and it demonstrates that several reported gains disappear under symmetric preprocessing, stronger classical baselines, and negative controls. The negative-control suite — label permutation, scrambled-label adaptation, matched random initialisation, and symmetric montage construction — is a reusable methodological contribution that goes beyond the usual leaderboard-style benchmark. The manuscript is also exemplary in its self-correction practices: it explicitly retires unreliable estimates, discloses post-hoc rows, and attaches precise caveats to each claim (e.g., that the CHB-MIT result does not establish superiority over amplitude-aware handcrafted baselines). The main limitations are clearly stated, in particular the absence of patient identifiers for CAUEEG and the descriptive nature of several comparative statistics. The paper is likely to be influential as a cautionary benchmark and as a template for future EEG-FM eva
minor comments (6)
- [Abstract and §4.6, Table 9] The headline CAUEEG value 0.734 is the best over the three probe families (SVM-RBF), while the stated primary protocol is frozen linear probing. The all-LogReg comparison (Classical 0.720 vs REVE 0.528) preserves the direction, but the abstract should either state 'best-of-probe-family' or headline the linear-probe value to avoid implying the 0.734 number comes from the primary linear probe.
- [§3.2 and §4.6] Please provide more detail on the provenance and availability of 'dementia-no-overlap.json' and the 'authors' fixed split': where they are obtained, whether they cover all three diagnostic groups, and how the 1,187-recording CAUEEG analysis cohort is derived from the 1,379 released recordings. This is important for reproducibility, since the patient-disjoint sensitivity depends entirely on this file.
- [§3.4, footnote] There is an internal inconsistency between the stated feature count '9×N_ch + 2 features per epoch (spectral/temporal features plus hemispheric alpha asymmetry and frontal theta power)' and the footnote saying Tables 3–5 were computed without alpha asymmetry and frontal theta. Please state explicitly which tables use which feature set and report the exact feature dimensionality for each classical row (e.g., 9×19 vs 9×19+2).
- [Abstract and Conclusion] The CAUEEG gap is presented in the abstract as 'classical features achieve 0.734 ... versus ... 0.568 for REVE' without the immediately relevant caveat that the primary CAUEEG folds are recording-level. The body (§5.4 #12) is appropriately candid, but the abstract should carry a short qualifier such as 'recording-level primary folds' so readers do not over-interpret the point estimate.
- [§2.2 and References] References [37], [41], and [42] appear in the reference list but are not cited in the visible text. Either cite them where relevant (e.g., analysis variability, scaling laws, the companion manuscript) or remove them from the list.
- [§4.7] The conclusion that scrambled labels reproduce the LoRA movement relies on near-equality of means (0.7039 vs 0.7050 and 0.8392 vs 0.8397). Reporting seed-level or fold-level paired values would strengthen this claim; as written, the reader cannot assess whether the tiny 0.1 pp differences are within seed variability.
Circularity Check
No significant circularity: the paper is an empirical audit/benchmark whose claims are tested against independent controls, not derived from its own inputs.
full rationale
This is a benchmarking and negative-control paper, not a derivation. The main claims are empirical comparisons: frozen embeddings from pretrained models are scored by linear probes against classical features and randomized controls. No reported quantity is defined in terms of the quantity it is claimed to predict, and no fitted parameter is renamed as a prediction. The CAUEEG external-cohort comparison is explicitly recording-level and is accompanied by a patient-disjoint sensitivity; reliance on the authors' dementia-no-overlap.json is an acknowledged data-validity limitation (leakage risk), not a circular step, because the paper does not claim that file proves the gap. The dataset-identity probe trains a linear classifier on dataset labels, but the reported result is held-out/out-of-pair generalization at or near ceiling; this is an empirical measurement of representation content, not an equivalence-by-construction. The scrambled-label fine-tuning and label-permutation controls are designed to test whether label information explains observed movements, and the null results are presented as null results. The only self-citation is reference [42], a companion manuscript under review, which is not load-bearing in the provided text: no central premise is justified by it, and no uniqueness theorem or ansatz is imported from it. All benchmark model scores are produced independently from public weights and external datasets, and the paper explicitly flags its own limitations, including in-domain pretraining exposure, missing patient identifiers, and unpaired uncertainty estimates. Consequently there is no circular step to report under the criteria of this review. Correctness risks, such as possible unannotated patient overlap in CAUEEG or probe-family selection favoring the classical comparator, are separate from circularity and do not raise the score.
Axiom & Free-Parameter Ledger
free parameters (5)
- Probe regularization C =
C=1.0 (CHB-MIT, TUAB); C=0.1 (ds004504, CAUEEG)
- In-fold PCA-200 on FM embeddings =
200 components (CHB-MIT)
- Ictal:interictal ratio with same-session negatives =
1:3
- TUAB epoch subsample =
4,807 of 8,130 epochs
- Per-epoch z-score normalization and clip =
z-score per channel, clip ±8 SD
axioms (6)
- domain assumption One shared preprocessing pipeline is a fair interface to all seven models
- domain assumption CHB-MIT is absent from REVE's pretraining inventory
- domain assumption dementia-no-overlap.json is a faithful patient-overlap trace
- domain assumption Approximate p-values from overlapping LOSO training sets are meaningful
- domain assumption The pure-PyTorch Mamba2 port is faithful
- standard math Standard statistics and DSP machinery behave as assumed
read the original abstract
Pretrained EEG foundation models are proposed for clinical decoding, but whether reported gains transfer across populations or survive negative controls is unclear. We benchmark LaBraM, EEGMamba, CBraMod, REVE, LEAD, BENDR, and BIOT on five clinical tasks across four datasets. Primary analyses use frozen linear probes with subject-disjoint LOSO or grouped five-fold validation. Because CAUEEG releases no patient identifiers, it is evaluated at recording level with a patient-disjoint sensitivity. We challenge apparent gains using stronger classical comparators, label permutation, scrambled-label fine-tuning, and random-initialisation controls. In a matched 19-channel CAUEEG evaluation (Normal/MCI/Dementia; N = 1,187 recordings), classical features achieve 0.734 macro-AUROC versus 0.699 for BIOT, 0.669 for CBraMod, and 0.568 for REVE. A patient-disjoint sensitivity retains the classical-over-REVE ordering (0.717 versus 0.565). Dataset identity is decoded from frozen REVE embeddings at or near ceiling across Western-Korean and Western-Western pairs, including after PCA-50 and removal of line-frequency and amplitude-scale information. This establishes dataset membership, not a causal site or population effect. A matched random-initialised encoder exceeds pretrained REVE on CAUEEG (0.659 versus 0.570). On CHB-MIT cross-subject ictal detection (n = 23), REVE reaches 0.793, versus 0.739 for the best enhanced nonlinear comparator, 0.701 for random initialisation, and 0.505 for raw-signal random features. Because preprocessing removes absolute amplitude, this does not establish superiority over every plausible handcrafted baseline. Conclusions change materially after montage matching, patient-overlap checks, stronger comparators, and representation controls. We distill these checks into a reporting protocol for clinical EEG foundation-model studies.
Figures
Forward citations
Cited by 3 Pith papers
-
Cross-Cohort Spectral-Temporal Dissociation in Frozen EEG Foundation-Model Representations
EEG foundation models fail to encode the alpha-envelope DFA exponent, a disease-relevant temporal-scaling feature, while spectral-input models still encode the static 1/f slope.
-
Cross-Cohort Spectral-Temporal Dissociation in Frozen EEG Foundation-Model Representations
Frozen EEG foundation models can decode the aperiodic spectral slope, but not the alpha-envelope DFA exponent reproducibly across two cohorts.
-
Technological Advances in Detecting and Managing Cognitive Impairment in Older Adults: Trends, Challenges, and Future Directions
Narrative review of cognitive-impairment detection technologies concludes that reported accuracies are often inflated by weak validation and that progress depends on multimodal, longitudinally validated, externally te...
Reference graph
Works this paper leans on
-
[1]
A., Grabowski, H
DiMasi, J. A., Grabowski, H. G., & Hansen, R. W. (2016). Innovation in the pharmaceutical industry: New estimates of R&D costs.Journal of Health Economics, 47, 20–33
2016
-
[2]
W., Craighead, J
Hay, M., Thomas, D. W., Craighead, J. L., et al. (2014). Clinical development success rates for investigational drugs.Nature Biotechnology, 32(1), 40–51
2014
-
[3]
& Barachant, A
Jayaram, V. & Barachant, A. (2018). MOABB: Trustworthy algorithm benchmarking for BCIs.Journal of Neural Engineering, 15(6), 066011
2018
-
[4]
Bommasani, R., et al. (2021). On the opportunities and risks of foundation models. arXiv:2108.07258
Pith/arXiv arXiv 2021
-
[5]
He, K., Chen, X., Xie, S., et al. (2022). Masked autoencoders are scalable vision learners. Proc. CVPR, pp. 16000–16009
2022
-
[6]
B., Zhao, L
Jiang, W. B., Zhao, L. M., & Lu, B. L. (2024). Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI.Proc. ICLR(Spotlight)
2024
-
[7]
Wang, J., et al. (2025). EEGMamba: An EEG Foundation Model with Mamba.Neural Networks, 192, 107816
2025
-
[8]
Wang, J., et al. (2025). CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding. Proc. ICLR. 24
2025
-
[9]
El Ouahidi, Y., Lys, J., Th¨ olke, P., Farrugia, N., Pasdeloup, B., Gripon, V., Jerbi, K., & Lioi, G. (2025). REVE: A Foundation Model for EEG, Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjects.Advances in Neural Information Processing Systems (NeurIPS). arXiv:2510.21585
arXiv 2025
-
[10]
Wu, J., et al. (2025). AdaBrain-Bench: Benchmarking Brain Foundation Models for Brain- Computer Interface Applications.arXiv:2507.09882
Pith/arXiv arXiv 2025
-
[11]
Wang, Y., Huang, N., Mammone, N., Cecchi, M., & Zhang, X. (2025). LEAD: An EEG Foundation Model for Alzheimer’s Disease Detection.arXiv preprint arXiv:2502.01678 [cs.LG, eess.SP]
arXiv 2025
-
[12]
Kostas, D., Aroca-Ouellette, S., & Rudzicz, F. (2021). BENDR: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data. Frontiers in Human Neuroscience, 15, 653659
2021
-
[13]
B., & Sun, J
Yang, C., Westover, M. B., & Sun, J. (2023). BIOT: Biosignal Transformer for Cross-data Learning in the Wild.Advances in Neural Information Processing Systems (NeurIPS), 36, 78240–78260
2023
-
[14]
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A simple framework for contrastive learning of visual representations.Proc. ICML, pp. 1597–1607
2020
-
[15]
& Dao, T
Gu, A. & Dao, T. (2024). Mamba: Linear-time sequence modeling with selective state spaces.Proc. COLM
2024
-
[16]
Dao, T. & Gu, A. (2024). Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality.Proc. ICML
2024
-
[17]
Kuruppu, G., Wagh, N., Kremen, V., Pati, S., Worrell, G., & Varatharajah, Y. (2025). EEG Foundation Models: A Critical Review of Current Progress and Future Directions. arXiv:2507.11783
arXiv 2025
-
[18]
Liu, D., Chen, Y., Chen, Z., Cui, Z., Wen, Y., An, J., Luo, J., & Wu, D. (2026). EEG Foundation Models: Progresses, Benchmarking, and Open Problems.arXiv:2601.17883
arXiv 2026
-
[19]
A., Laskaris, N., & Zafeiriou, S
Lee, N., Bakas, S., Barmpas, K., Panagakis, Y., Adamos, D. A., Laskaris, N., & Zafeiriou, S. (2025). Assessing the Capabilities of Large Brainwave Foundation Models.2025 IEEE 35th International Workshop on Machine Learning for Signal Processing (MLSP)
2025
-
[20]
Wang, X., Yang, Y., & Coyle, D. (2026). EEG-FM-Audit: A Systematic Evaluation and Analysis Pipeline for EEG Foundation Models.arXiv:2605.26910
Pith/arXiv arXiv 2026
-
[21]
Kastrati, A., B¨ urki, J., Lauer, J., Xuan, C., Iaquinto, R., & Wattenhofer, R. (2025). EEG- Bench: A Benchmark for EEG Foundation Models in Clinical Applications.Foundation Models for the Brain and Body Workshop (BrainBodyFM), NeurIPS 2025. arXiv:2512.08959
arXiv 2025
-
[22]
G., Karaiskou, A.-I., Gagliardi, G., Strypsteen, T., Badiei, M
Kontras, K., Osselaer, T., Mouslech, S. G., Karaiskou, A.-I., Gagliardi, G., Strypsteen, T., Badiei, M. H., Rani, A., Vanmarcke, M., Bhagubai, M., Ekbote, C., Hwang, J., Chatzichristos, C., Liang, P. P., & De Vos, M. (2026). NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces.arXiv:2605.14698
Pith/arXiv arXiv 2026
-
[23]
Xiong, W., Li, J., Li, J., Zhu, K., & Jiang, C. (2026). EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation and Diagnostic Analyses of EEG Foundation Models.International Conference on Machine Learning (ICML 2026). arXiv:2508.17742. 25
arXiv 2026
-
[24]
Lu, Z., Li, Z., Shen, X., Lou, K., Xin, Y., Chen, X., Wang, S., Chen, X., Fan, J., Huang, C., Xu, X., Hou, Z., Wei, C., & Liu, Q. (2026). OmniEEG-Bench: A Standardized Evaluation Benchmark for EEG Foundation Models.arXiv:2606.00815
Pith/arXiv arXiv 2026
-
[25]
ˇSirca, U., Alimardani, M., Zafeiriou, S., & Barmpas, K. (2026). Beyond Accuracy: Robust- ness, Interpretability and Expressiveness of EEG Foundation Models.arXiv:2605.17562
Pith/arXiv arXiv 2026
-
[26]
T., et al
Schirrmeister, R. T., et al. (2017). Deep learning with convolutional neural networks for EEG decoding and visualization.Human Brain Mapping, 38(11), 5391–5420
2017
-
[27]
Dosovitskiy, A., et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale.Proc. ICLR
2021
-
[28]
Shoeb, A. H. (2009). Application of machine learning to epileptic seizure onset detection and treatment.PhD thesis, MIT
2009
-
[29]
& Picone, J
Obeid, I. & Picone, J. (2016). The Temple University Hospital EEG data corpus.Frontiers in Neuroscience, 10, 196
2016
-
[30]
Miltiadous, A., et al. (2023). A dataset of scalp EEG recordings of Alzheimer’s disease, frontotemporal dementia and healthy subjects from routine EEG.Data, 8(6), 95
2023
-
[31]
Kemp, B., et al. (2000). Analysis of a sleep-dependent neuronal feedback loop.IEEE Trans. Biomedical Engineering, 47(9), 1185–1194
2000
-
[32]
C., & Paik, J
Kim, M.-J., Youn, Y. C., & Paik, J. (2023). Deep learning-based EEG analysis to classify normal, mild cognitive impairment, and dementia: Algorithms and dataset.NeuroImage, 272, 120054
2023
-
[33]
Gramfort, A., et al. (2013). MEG and EEG data analysis with MNE-Python.Frontiers in Neuroscience, 7, 267
2013
-
[34]
Welch, P. (1967). The use of fast Fourier transform for the estimation of power spectra. IEEE Trans. Audio and Electroacoustics, 15(2), 70–73
1967
-
[35]
Hjorth, B. (1970). EEG analysis based on time domain properties.Electroencephalography and Clinical Neurophysiology, 29(3), 306–310
1970
-
[36]
J., et al
Donoghue, T., Haller, M., Peterson, E. J., et al. (2020). Parameterizing neural power spectra into periodic and aperiodic components.Nature Neuroscience, 23(12), 1655–1665
2020
-
[37]
Botvinik-Nezer, R., et al. (2020). Variability in the analysis of a single neuroimaging dataset by many teams.Nature, 582, 84–88. DOI: 10.1038/s41586-020-2314-9
-
[38]
van Dijk, H., van Wingen, G., Denys, D., Olbrich, S., van Ruth, R., & Arns, M. (2022). The two decades brainclinics research archive for insights in neurophysiology (TDBRAIN) database.Scientific Data, 9, 333
2022
-
[39]
Cavanagh, J. F. (2021). EEG: 3-Stim Auditory Oddball and Rest in Parkinson’s Disease. OpenNeuro, dataset ds003490
2021
-
[40]
J., Shen, Y., Wallis, P., et al
Hu, E. J., Shen, Y., Wallis, P., et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models.Proc. ICLR
2022
-
[41]
Kaplan, J., et al. (2020). Scaling laws for neural language models.arXiv:2001.08361
Pith/arXiv arXiv 2020
-
[42]
Zare, M. (2026). Foundation Models for EEG Are Blind to Long-Range Temporal Cor- relations: A Spectral–Temporal Dissociation Behind Their Cross-Population Fragility. Companion manuscript, under review, 2026. 26
2026
-
[43]
L., et al
Goldberger, A. L., et al. (2000). PhysioBank, PhysioToolkit, and PhysioNet.Circulation, 101(23), e215–e220. 27
2000
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.