Pith. sign in

REVIEW 2 major objections 5 minor 81 references

A foundation model with multi-variate parallel attention to generate neuronal activity

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single pre-trained transformer detects seizures in 50 never-seen patients with the same average inter-rater agreement as human experts (Kappa 0.57).

desk verdict A genuinely useful architecture and dataset, but the expert-level zero-shot claim rests on a manual channel-selection step that the paper itself shows is worth ~0.18 Kappa. read the letter →

arxiv 2506.20354 v2 pith:PEYJ2AFP submitted 2025-06-25 cs.LG cs.AI

Francesco Carzaniga , Michael Hersche , Abu Sebastian , Kaspar Schindler , Abbas Rahimi This is my paper
classification cs.LGcs.AI
keywords multi-variateparallelattentionintracranialEEGfoundationmodelseizuredetectionheterogeneouschannelconfigurationsgenerativepre-trainingtime-seriesforecastingSWECidataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether one pre-trained model can handle intracranial EEG (iEEG) recordings whose electrode layouts differ completely from patient to patient, and whether that model can then do clinically useful work on patients it has never seen. The authors answer by building MVPFormer, a transformer whose new attention mechanism, MVPA, splits attention into content, time, and channel components so that no fixed channel geometry is assumed. Trained generatively to predict the next five-second segment of brain signal on the newly released SWEC dataset (68 subjects, nearly 10,000 hours), and fine-tuned with only 0.1% of its parameters, MVPFormer reaches a Cohen's Kappa of 0.57 against a board-certified neurologist across 50 unseen subjects. That matches reported human-expert agreement, while the vanilla-attention ablation collapses to 0.05 and the prior iEEG foundation model to 0.08. If the result holds, clinical seizure monitoring could be run by a single open model across arbitrary hospital setups, and the released dataset unblocks further iEEG foundation-model research.

What carries the argument

The load-bearing object is multi-variate parallel attention (MVPA), a self-attention variant whose query-key score separates into three additive components: content-based attention between raw segment embeddings (no positional encoding), time-based attention depending only on the relative temporal distance between segments, and channel-based attention depending only on the relative spatial distance between channels. Because the time term is identical across channels and the channel term is identical across time steps, each is computed once per query and broadcast, giving a subquadratic complexity of $O(T^2 C + T C^2)$ in the context length; a Transformer-XL-style shifting trick computes all relative embeddings in one pass, and a custom Triton kernel (FlashMVPA) keeps memory linear. Around this core sits the rest of the recipe: a db4 wavelet encoder that turns each five-second segment into a continuous embedding (wav2vec-inspired), a contrastive next-segment prediction objective with 30 in-batch confounders during pre-training, and LoRA fine-tuning (rank 8) that updates only about 0.1% of parameters for downstream tasks.

What would settle it

Re-run the zero-shot seizure detection with channel selection done by a fixed, purely algorithmic rule (for example, ranking electrodes by artifact amplitude or by predictive loss on held-out signal) applied identically to every one of the 50 test subjects; if average Kappa falls toward the 0.36 all-channel value rather than the reported 0.54-0.57, the expert-level claim is carried by the human picking channels, not by the model alone. A second check is to have a second board-certified neurologist re-annotate a sample of the same episodes: if the human-human Kappa on those episodes is well below 0.57, the model has not actually matched the expert ceiling.

Watch

Extended reading notes

Core claim

The paper's central claim is that disentangling content, temporal, and spatial attention is what lets a generative transformer generalize across heterogeneous electrode arrays. In MVPFormer, iEEG is chopped into five-second segments, mapped through a wavelet encoder to continuous embeddings, and processed by MVPA, whose attention score is the sum of three terms: a content term with no positional encoding, a time term depending only on relative temporal distance, and a channel term depending only on relative spatial distance. Pre-trained to predict the next segment in time against contrastive confounders, then fine-tuned on 18 subjects, the model is evaluated zero-shot on 50 unseen subjects and achieves average inter-rater agreement of 0.57 (0.54 for the 75M-parameter version), matching human expert Kappa values. The same model outperforms state-of-the-art transformers on seizure detection on the MAYO and FNUSA datasets, sets state-of-the-art results on pitch and volume decoding in Brain TreeBank and matches the electrode-location-aware PopT elsewhere, and matches or beats PatchTST and TimeMixer on long-term forecasting benchmarks. The authors present this as evidence that MVPA is a general-purpose attention mechanism for heterogeneous multivariate time-series, and that the generative foundation-model recipe transfers to electrophysiology.

Load-bearing premise

The expert-level seizure-detection result assumes that a non-expert's quick visual choice of the 30 to 50 least noisy electrodes per test subject is a valid preprocessing step, since running the same model on all available channels drops its Kappa agreement from 0.54 to 0.36.

Editorial extensions

If this is right

  • A single open model can be dropped onto a new epilepsy patient's electrode montage and perform seizure detection at expert-level agreement without any patient-specific retraining or electrode-location information.
  • The SWEC dataset (68 subjects, 9,328 hours, 704 ictal events) becomes the largest public iEEG corpus, removing the data bottleneck the paper identifies as the main obstacle to iEEG foundation models.
  • Generative pre-training on next-segment prediction transfers to downstream classification better than purely discriminative training (Kappa 0.54 vs 0.52), supporting the foundation-model paradigm for electrophysiology.
  • Because MVPA's channel attention learns an implicit connectivity map from relative channel distances, the mechanism transfers to any montage, as demonstrated by competitive results on MAYO, FNUSA, and Brain TreeBank without using electrode coordinates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own all-channel experiment (Kappa 0.54 to 0.36 with every electrode included) suggests the headline result depends on the noise-reducing effect of channel preselection, not on anything anatomical; an automated denoising or channel-scoring front-end could plausibly recover expert-level agreement without human visual inspection.
  • The learned channel-attention map, dominated by a diagonal with distinct clusters and long-range connections, is a data-driven candidate functional-connectivity estimate; comparing it against anatomical distance or known epileptogenic networks would test whether MVPA is learning brain structure or merely signal statistics.
  • Agreement of 0.57 is scored against a single expert's labels; a two-expert annotation of the same 50-subject episodes would show how much of the residual disagreement is model error versus genuine inter-rater variability, and would set a fairer ceiling for what 'expert-level' means.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces multi-variate parallel attention (MVPA), a factorized self-attention mechanism with separate content, temporal, and spatial components, and uses it to build MVPFormer, a generative foundation model for intracranial EEG (iEEG). MVPFormer is pre-trained with a contrastive next-embedding objective on the newly released SWEC dataset (68 subjects, 9328 hours) and then fine-tuned with LoRA for downstream tasks. The authors report expert-level Cohen's Kappa of 0.57 (MVPFormer-M) and 0.54 (MVPFormer-S) for zero-shot seizure detection on 50 unseen SWEC subjects, results on the MAYO and FNUSA iEEG datasets, results on four Brain TreeBank decoding tasks, and competitive forecasting/classification results on standard time-series benchmarks. Code, weights, and the SWEC dataset are released.

Significance. If the results hold, the paper makes substantial contributions: the largest public iEEG dataset to date, an open foundation model for heterogeneous multi-channel time series, and controlled ablations (MV-Llama and a no-pretraining variant) showing the value of MVPA and generative pre-training. The per-subject result tables are unusually detailed, and the external evaluations on Brain TreeBank, MAYO, and FNUSA provide independent grounding that mitigates circularity concerns. However, the headline expert-level zero-shot claim is conditioned on a non-reproducible manual channel-selection step, and the all-channel result for the larger model is missing; this must be addressed before the central generalization claim can be accepted.

major comments (2)
  1. [Section 5.1; Appendix G.8] The headline zero-shot Kappa of 0.57 (MVPFormer-M) and 0.54 (MVPFormer-S) is obtained after a non-expert visually selects 30-50 'least noisy' channels per test subject, as described in Section 5.1. Appendix G.8 shows that evaluating MVPFormer-S on all channels reduces Kappa from 0.54 to 0.36 (Table S17), yet no all-channel result is reported for MVPFormer-M, the model behind the headline number. The selection is not specified by a reproducible rule (e.g., an SNR threshold or an automated algorithm), and no inter-annotator reliability for the visual selection is reported. Because the selector inspects the raw test recordings, the selection can inadvertently favor channels where ictal activity is visually more apparent, potentially leaking outcome information into preprocessing. This makes the central expert-level zero-shot claim contingent on a non-reproducible and possibly label-influenced step. Please provide a fully specified selection procedure (or release the per-subject channel indices), report all-channel results for both model sizes, and analyze the sensitivity of Kappa to the selection.
  2. [Section 5.2.1; Figure 3] The characterization of a Kappa of 0.57 as 'matching human expert performance' should be conditioned on the channel-selection protocol and accompanied by uncertainty quantification. Under the Landis and Koch criteria used in the paper, the all-channel value of 0.36 for MVPFormer-S falls in the 'fair' range and is below the 0.53 threshold the authors use for expert-level agreement; reporting only the selected-channel Kappas in the main figure inflates the apparent robustness. The paper should report confidence intervals (e.g., bootstrap) for the mean Kappa, state explicitly that the expert-level claim applies to the manually selected channel subset, and provide the selected-channel versus all-channel comparison for MVPFormer-M before the abstract claims expert-level generalization.
minor comments (5)
  1. [Abstract; Section 5.2.2; Table S10] The abstract states 'state-of-the-art performance on four Brain TreeBank iEEG decoding tasks,' but Table S10 shows that PopT (BrainBERT) achieves higher accuracy than MVPFormer on Onset (0.90 vs 0.87) and Speech (0.93 vs 0.90). The main text correctly qualifies this; the abstract should be aligned to say state-of-the-art on two of the four tasks and competitive on the remaining two.
  2. [Table 1; Appendix G.11] The MAYO and FNUSA results are obtained after fine-tuning a classification head on the first four patients of each dataset, as described in Appendix G.11. The main text says the models are 'applied' to these datasets and should state the fine-tuning step explicitly to avoid implying zero-shot transfer.
  3. [Section 5.3; Table S24] The sentence 'MVPFormer always equals or surpasses the baselines' is not supported by the full table: TimeMixer achieves lower MSE than MVPFormer on ETTh1-96 (0.37 vs 0.38) and ETTh2-96 (0.29 vs 0.30). Consider reporting the full table in the main text and softening this claim.
  4. [Section 5.1] The relation between the '39B total training segments' and '390M unique' segments should be clarified, since without the sliding-window overlap explanation readers may misread the dataset scale.
  5. [Section 5.2.1; Algorithm 1; Figure S20] There is a typo 'We being by evaluating' in Section 5.2.1, Algorithm 1 uses inconsistent symbols (u, y, w; v for both value and bias), and the caption of Figure S20 appears to be copied from Figure S19. These should be corrected during revision.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central claims rest on unseen-subject evaluation and external benchmarks; the only self-citation is a non-load-bearing limitations pointer.

full rationale

The paper's claimed derivation chain is self-contained. MVPA is defined directly in Eqs. (1)-(3) as a sum of content-, time-, and channel-based attention terms; this is an architectural construction, not a fitted prediction. MVPFormer is pre-trained with the contrastive loss in Eq. (4) on 18 SWEC subjects and then evaluated zero-shot on 50 held-out SWEC subjects for seizure detection, on the external MAYO and FNUSA datasets, and on the external Brain TreeBank and standard forecasting/classification benchmarks. The headline Kappa of 0.57 is computed against expert annotations on test subjects that were not used in fine-tuning, so it is not equivalent to any training input. The pre-training validation in Appendix F compares cosine similarity to true, two-step, and random targets; this is a sanity check, not a circular prediction. The all-channel ablation in Appendix G.8 (Kappa drop from 0.54 to 0.36) and the manual selection of 30-50 least noisy channels described in Section 5.1 are reproducibility and validity concerns, but no equation or fitted parameter reduces the reported score to the selection rule. The only self-citation, reference [56], appears in the Limitations section as a pointer to prior work on data quality; it is not load-bearing for any experimental claim. Consequently, there is no circular step.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The ledger shows that the central empirical claims rest on domain-specific preprocessing choices (manual channel selection, episodic thresholds) and on assumptions about the wavelet encoder, relative channel distance, and contrastive negatives. No new physical entities are introduced. The MVPA mechanism itself is a modeling choice, not an axiom, and it is evaluated against external benchmarks.

free parameters (4)
  • Test channel subset (30-50 per subject) = varies by subject
    Channels selected by visual inspection by a non-expert to include only least noisy ones (Section 5.1); using all channels reduces Kappa from 0.54 to 0.36 (Appendix G.8).
  • Episodic post-processing thresholds = merge within 5 min; min event 20s; min 5 positives; threshold 3/10 positive seconds
    Hand-set rules in Appendix E affect episodic F1, fp/h, and the reported Kappa; these are not independently justified.
  • Pre-training hyperparameters = tau=0.1, n_negatives=30, L=10 segments, segment=5s, window=500s
    Chosen settings in Section 3.2 and Appendix A define the contrastive objective and context length; no sensitivity analysis is provided.
  • LoRA rank and alpha = rank 8, alpha 16
    Fine-tuning configuration in Section 3.2 that limits trainable parameters to roughly 0.1% of the base model.
assumptions (5)
  • domain assumption db4 wavelet encoding preserves sufficient discriminative information in 5-second segments for future prediction and downstream classification.
    Encoder in Section 3.1; no analysis of information loss or alternative encodings.
  • domain assumption Relative channel distance, without absolute electrode locations, is sufficient to learn an implicit brain connectivity map.
    Section 2.1 and Appendix G.10; this is why MVPA ignores electrode coordinates even on Brain TreeBank where they are available.
  • domain assumption Contrastive negatives sampled from other windows are plausible but distinct confounders.
    Section 3.2 and Appendix C.1; the loss assumes the true next segment is the only positive among negatives.
  • domain assumption iEEG signals are sufficiently stationary that predicting the next 5-second embedding is a well-posed generative objective.
    Section 3.2; validation uses cosine similarity to the true next segment versus random and two-step references.
  • domain assumption Landis and Koch thresholds and literature expert Kappa values transfer to this dataset and labeling protocol.
    Section 5.2.1 compares Kappa 0.57 to external expert-expert agreements, not to same-dataset expert agreement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A foundation model with multi-variate parallel attention to generate neuronal activity." pith.science (2026). https://pith.science/paper/PEYJ2AFP

@misc{pith2026250620354,
  author       = {Pith},
  title        = {Pith review of: A foundation model with multi-variate parallel attention to generate neuronal activity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PEYJ2AFP}},
  note         = {Machine review of arXiv:2506.20354}
}
read the original abstract

Learning from multi-variate time-series with heterogeneous channel configurations remains a fundamental challenge for deep neural networks, particularly in clinical domains such as intracranial electroencephalography (iEEG), where channel setups vary widely across subjects. In this work, we introduce multi-variate parallel attention (MVPA), a novel self-attention mechanism that disentangles content, temporal, and spatial attention, enabling flexible, generalizable, and efficient modeling of time-series data with varying channel counts and configurations. We use MVPA to build MVPFormer, a generative foundation model for human electrophysiology, trained to predict the evolution of iEEG signals across diverse subjects. To support this and future efforts by the community, we release the SWEC iEEG dataset, the largest publicly available iEEG dataset to date, comprising nearly 10,000 hours of recordings from heterogeneous clinical sources. MVPFormer leverages MVPA to achieve strong generalization across subjects, demonstrating expert-level performance in several iEEG tasks. MVPFormer surpasses state-of-the-art Transformer baselines in seizure detection across the SWEC, the MAYO, and the FNUSA datasets, while also achieving state-of-the-art performance on four Brain TreeBank iEEG decoding tasks. We further validate MVPA on standard time-series forecasting and classification tasks, where it matches or exceeds the performance of existing attention-based models. Together, our contributions establish MVPA as a general-purpose attention mechanism for heterogeneous time-series and MVPFormer as the first open-source, open-weights, and open-data iEEG foundation model with SOTA clinical performance. The code is available at https://github.com/IBM/multi-variate-parallel-transformer. The SWEC iEEG dataset is available at https://huggingface.co/datasets/NeuroTec/SWEC_iEEG_Dataset.

Figures

Figures reproduced from arXiv: 2506.20354 by the authors.

Figure 1
Figure 1. Multi-variate parallel attention (MVPA). (a) The input signal is divided into temporal and spatial segments. Each query-key interaction is computed for all keys within a local window. (b) MVPA decomposes attention into three components: content-based, computed per segment without positional encoding; time-based, shared across channels and dependent only on temporal distance; and channel-based, shared across time ste… view at source ↗
Figure 2
Figure 2. MVPFormer architecture and forward pass. iEEG signals are segmented in time and space, encoded via a wavelet-based encoder, and arranged into a 2D embedding grid. These continu￾ous embeddings are processed by MVPA to model temporal, spatial, and content-based dependencies. MVPFormer predicts the next-in-time embedding while reducing similarity to confounders from the same or other subjects. Notched in the bottom rig… view at source ↗
Figure 3
Figure 3. MVPFormer’s inter-rater agreement on seizure detection. The average Kappa score between MVPFormer-M and the human expert is 0.57, competitive with the values obtained between human experts. subjects and ictal events than are reported for human experts, making this task more challenging for MVPFormer. MVPFormer achieves an average Kappa of 0.57 across 50 unseen subjects from the SWEC iEEG dataset, matching human expe… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

81 extracted references · 65 canonical work pages

  1. [1]

    Time-LLM: Time series forecasting by reprogramming large language models,

    M. Jin, S. Wang, L. Ma, Z. Chu, J. Y . Zhang, X. Shi, P.-Y . Chen, Y . Liang, Y .-F. Li, S. Pan, and Q. Wen, “Time-LLM: Time series forecasting by reprogramming large language models,” in Iternational Conference on Learning Representations (ICLR), 2024

    2024

  2. [2]

    Timemixer: Decomposable multiscale mixing for time series forecasting,

    S. Wang, H. Wu, X. Shi, T. Hu, H. Luo, L. Ma, J. Y . Zhang, and J. ZHOU, “Timemixer: Decomposable multiscale mixing for time series forecasting,” in International Conference on Learning Representations (ICLR), 2024

    work page 2024

  3. [3]

    A time series is worth 64 words: Long- term forecasting with transformers,

    Y . Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, “A time series is worth 64 words: Long- term forecasting with transformers,” in International Conference on Learning Representations (ICLR), 2023

    work page 2023

  4. [4]

    P. L. Nunez and R. Srinivasan, Electric Fields of the Brain. Oxford University Press, Jan. 2006

    work page 2006

  5. [5]

    Seizure prediction — ready for a new era,

    L. Kuhlmann, K. Lehnertz, M. P. Richardson, B. Schelter, and H. P. Zaveri, “Seizure prediction — ready for a new era,” Nature Reviews Neurology, vol. 14, no. 10, 2018

    work page 2018

  6. [6]

    Comparison of different input modalities and network structures for deep learning-based seizure detection,

    K.-O. Cho and H.-J. Jang, “Comparison of different input modalities and network structures for deep learning-based seizure detection,” Scientific Reports, 2020

    work page 2020

  7. [7]

    EEGWaveNet: Multiscale CNN-based spatiotemporal feature extraction for EEG seizure detection,

    P. Thuwajit, P. Rangpong, P. Sawangjai, P. Autthasan, R. Chaisaen, N. Banluesombatkul, P. Boonchit, N. Tatsaringkansakul, T. Sudhawiyangkul, and T. Wilaiprasitporn, “EEGWaveNet: Multiscale CNN-based spatiotemporal feature extraction for EEG seizure detection,” IEEE Transactions on Industrial Informatics, vol. 18, no. 8, 2022

    work page 2022

  8. [8]

    Brain- BERT: Self-supervised representation learning for intracranial recordings,

    C. Wang, V . Subramaniam, A. U. Yaari, G. Kreiman, B. Katz, I. Cases, and A. Barbu, “Brain- BERT: Self-supervised representation learning for intracranial recordings,” in International Conference on Learning Representations (ICLR), 2023

    work page 2023

Show all 81 references
  1. [9]

    Towards trustworthy seizure onset detection using workflow notes,

    K. Saab, S. Tang, M. Taha, C. Lee-Messer, C. Ré, and D. L. Rubin, “Towards trustworthy seizure onset detection using workflow notes,” npj Digital Medicine, vol. 7, no. 1, 2024

    2024

  2. [10]

    Mul- ticenter intracranial EEG dataset for classification of graphoelements and artifactual signals,

    P. Nejedly, V . Kremen, V . Sladky, J. Cimbalnik, P. Klimes, F. Plesinger, F. Mivalt, V . Travnicek, I. Viscor, M. Pail, J. Halamek, B. H. Brinkmann, M. Brazdil, P. Jurak, and G. Worrell, “Mul- ticenter intracranial EEG dataset for classification of graphoelements and artifact...

    2020

  3. [11]

    Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli,

    C. Wang, A. U. Yaari, A. K. Singh, V . Subramaniam, D. Rosenfarb, J. DeWitt, P. Misra, J. R. Madsen, S. Stone, G. Kreiman, B. Katz, I. Cases, and A. Barbu, “Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli,” in Advances in Neural Informati...

    2024

  4. [12]

    Informer: Beyond efficient transformer for long sequence time-series forecasting,

    H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12. AAAI Press, 2021, pp. 11 106–11 115

    2021

  5. [13]

    Autoformer: Decomposition transformers with Auto- Correlation for long-term series forecasting,

    H. Wu, J. Xu, J. Wang, and M. Long, “Autoformer: Decomposition transformers with Auto- Correlation for long-term series forecasting,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021

    2021

  6. [14]

    Deep time series forecasting models: A comprehensive survey,

    X. Liu and W. Wang, “Deep time series forecasting models: A comprehensive survey,”Mathe- matics, vol. 12, no. 10, 2024

    2024

  7. [15]

    Brant-2: Foundation model for brain signals,

    Z. Yuan, F. Shen, M. Li, Y . Yu, C. Tan, and Y . Yang, “Brant-2: Foundation model for brain signals,” arXiv preprint arXiv:2402.1025, 2024

    2024

  8. [16]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017

    2017

  9. [17]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning...

    2021

  10. [18]

    Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting,

    Y . Zhang and J. Yan, “Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting,” in International Conference on Learning Representations (ICLR), 2023

    2023

  11. [19]

    Transformers in time series: A survey,

    Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, and L. Sun, “Transformers in time series: A survey,”arXiv preprint arXiv:2202.07125, 2022

    2022 arXiv

  12. [20]

    Transformer-XL: Attentive language models beyond a fixed-length context,

    Z. Dai, Z. Yang, Y . Yang, J. Carbonell, Q. Le, and R. Salakhutdinov, “Transformer-XL: Attentive language models beyond a fixed-length context,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. 11

    2019

  13. [21]

    Generating long sequences with sparse transformers,

    R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” arXiv preprint arXiv:1904.10509, 2019

    1904 arXiv

  14. [22]

    GQA: Training generalized multi-query transformer models from multi-head checkpoints,

    J. Ainslie, J. Lee-Thorp, M. de Jong, Y . Zemlyanskiy, F. Lebron, and S. Sanghai, “GQA: Training generalized multi-query transformer models from multi-head checkpoints,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  15. [23]

    FlashAttention: Fast and memory-efficient exact attention with IO-awareness,

    T. Dao, D. Y . Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022

    2022

  16. [24]

    FlashAttention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “FlashAttention-2: Faster attention with better parallelism and work partitioning,” in International Conference on Learning Representations (ICLR), 2024

    2024

  17. [25]

    A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27,

    Y . LeCun, “A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27,”Open Review, vol. 62, no. 1, pp. 1–62, 2022

    2022

  18. [26]

    LLM pretraining with continuous concepts,

    J. Tack, J. Lanchantin, J. Yu, A. Cohen, I. Kulikov, J. Lan, S. Hao, Y . Tian, J. Weston, and X. Li, “LLM pretraining with continuous concepts,” arXiv preprint arXiv:2502.08524, 2025

    2025 arXiv

  19. [27]

    GIVT: Generative infinite-vocabulary trans- formers,

    M. Tschannen, C. Eastwood, and F. Mentzer, “GIVT: Generative infinite-vocabulary trans- formers,” in Computer Vision – ECCV 2024. Cham: Springer Nature Switzerland, 2025, pp. 292–309

    2024

  20. [28]

    Training large language models to reason in a continuous latent space,

    S. Hao, S. Sukhbaatar, D. Su, X. Li, Z. Hu, J. Weston, and Y . Tian, “Training large language models to reason in a continuous latent space,” arXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  21. [29]

    Scaling up test-time compute with latent reasoning: A recurrent depth approach,

    J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein, “Scaling up test-time compute with latent reasoning: A recurrent depth approach,” arXiv preprint arXiv:2502.05171, 2025

    2025 arXiv

  22. [30]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations (ICLR), 2022

    2022

  23. [31]

    Analysis of EEG records in an epileptic patient using wavelet transform,

    H. Adeli, Z. Zhou, and N. Dadmehr, “Analysis of EEG records in an epileptic patient using wavelet transform,” Journal of neuroscience methods, vol. 123, no. 1, pp. 69–87, 2003

    2003

  24. [32]

    An EEG based real-time epilepsy seizure detection approach using discrete wavelet transform and machine learning methods,

    M. Shen, P. Wen, B. Song, and Y . Li, “An EEG based real-time epilepsy seizure detection approach using discrete wavelet transform and machine learning methods,” Biomedical Signal Processing and Control, vol. 77, 2022

    2022

  25. [33]

    wav2vec: Unsupervised pre-training for speech recognition,

    S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” in Interspeech 2019, 2019

    2019

  26. [34]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  27. [35]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  28. [36]

    Review of the BCI competition IV,

    M. Tangermann, K.-R. Müller, A. Aertsen, N. Birbaumer, C. Braun, C. Brunner, R. Leeb, C. Mehring, K. J. Miller, G. R. Müller-Putz, G. Nolte, G. Pfurtscheller, H. Preissl, G. Schalk, A. Schlögl, C. Vidaurre, S. Waldert, and B. Blankertz, “Review of the BCI competition IV,” Fron...

    2012

  29. [37]

    CHB-MIT scalp EEG database,

    A. Shoeb, “CHB-MIT scalp EEG database,” 2010

    2010

  30. [38]

    The temple university hospital eeg data corpus,

    I. Obeid and J. Picone, “The temple university hospital eeg data corpus,” Frontiers in Neuro- science, vol. 10, 2016

    2016

  31. [39]

    Population Transformer: Learning population-level representations of neural activity,

    G. Chau, C. Wang, S. J. Talukder, V . Subramaniam, S. Soedarmadji, Y . Yue, B. Katz, and A. Barbu, “Population Transformer: Learning population-level representations of neural activity,” in International Conference on Learning Representations (ICLR), 2025

    2025

  32. [40]

    Interrater reliability between scorers from eight european sleep laboratories in subjects with different sleep disorders,

    H. Danker-Hopfe, D. Kunz, G. Gruber, G. Klösch, J. L. Lorenzo, S. L. Himanen, B. Kemp, T. Penzel, J. Röschke, H. Dorn, A. Schlögl, E. Trenker, and G. Dorffner, “Interrater reliability between scorers from eight european sleep laboratories in subjects with different sleep disor...

    2004

  33. [41]

    Characterization of four-class motor imagery EEG data for the BCI-competition 2005,

    A. Schlögl, F. Lee, H. Bischof, and G. Pfurtscheller, “Characterization of four-class motor imagery EEG data for the BCI-competition 2005,” Journal of Neural Engineering, vol. 2, no. 4, pp. L14–L22, Aug. 2005

    2005

  34. [42]

    Interrater reliability: the kappa statistic,

    M. L. McHugh, “Interrater reliability: the kappa statistic,” Biochemia Medica, 2012

    2012

  35. [43]

    The measurement of observer agreement for categorical data,

    J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, no. 1, p. 159, Mar. 1977. 12

    1977

  36. [44]

    Inter-rater agreement on identification of electrographic seizures and periodic discharges in icu eeg recordings,

    J. Halford, D. Shiau, J. Desrochers, B. Kolls, B. Dean, C. Waters, N. Azar, K. Haas, E. Kutluay, G. Martz, S. Sinha, R. Kern, K. Kelly, J. Sackellares, and S. LaRoche, “Inter-rater agreement on identification of electrographic seizures and periodic discharges in icu eeg record...

    2015

  37. [45]

    EEG interpretation reliability and interpreter confidence: A large single-center study,

    A. C. Grant, S. G. Abdel-Baki, J. Weedon, V . Arnedo, G. Chari, E. Koziorynska, C. Lushbough, D. Maus, T. McSween, K. A. Mortati, A. Reznikov, and A. Omurtag, “EEG interpretation reliability and interpreter confidence: A large single-center study,”Epilepsy & Behavior, vol. 32, 2014

    2014

  38. [46]

    Interrater reliability in interpretation of electrocorticographic seizure detections of the responsive neurostimulator,

    M. Quigg, F. Sun, N. B. Fountain, B. C. Jobst, V . S. S. Wong, E. Mirro, S. Brown, and D. C. Spencer, “Interrater reliability in interpretation of electrocorticographic seizure detections of the responsive neurostimulator,”Epilepsia, vol. 56, no. 6, 2015

    2015

  39. [47]

    Brant: Foundation model for intracranial neural signal,

    D. Zhang, Z. Yuan, Y . Yang, J. Chen, J. Wang, and Y . Li, “Brant: Foundation model for intracranial neural signal,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023

    2023

  40. [48]

    A decoder-only foundation model for time-series forecasting,

    A. Das, W. Kong, R. Sen, and Y . Zhou, “A decoder-only foundation model for time-series forecasting,” in International Conference on Machine Learning (ICML), 2024

    2024

  41. [49]

    Wpmixer: Efficient multi-resolution mixing for long-term time series forecasting,

    M. M. N. Murad, M. Aktukmak, and Y . Yilmaz, “Wpmixer: Efficient multi-resolution mixing for long-term time series forecasting,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 18, 2025

    2025

  42. [50]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu et al., “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020

    2005 arXiv

  43. [51]

    FEDformer: Frequency enhanced de- composed transformer for long-term series forecasting,

    T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin, “FEDformer: Frequency enhanced de- composed transformer for long-term series forecasting,” in Proc. 39th International Conference on Machine Learning (ICML 2022), 2022

    2022

  44. [52]

    Scaling data-constrained language models,

    N. Muennighoff, A. M. Rush, B. Barak, T. L. Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, and C. Raffel, “Scaling data-constrained language models,” in Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  45. [53]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, ...

    2022

  46. [54]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  47. [55]

    The fineweb datasets: Decanting the web for the finest text data at scale,

    G. Penedo, H. Kydlí ˇcek, L. B. allal, A. Lozhkov, M. Mitchell, C. Raffel, L. V . Werra, and T. Wolf, “The fineweb datasets: Decanting the web for the finest text data at scale,” inAdvances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024

    2024

  48. [56]

    The case for cleaner biosignals: High-fidelity neural compressor enables transfer from cleaner iEEG to noisier EEG,

    F. S. Carzaniga, G. T. Hoppeler, M. Hersche, K. Schindler, and A. Rahimi, “The case for cleaner biosignals: High-fidelity neural compressor enables transfer from cleaner iEEG to noisier EEG,” in International Conference on Learning Representations (ICLR), 2025

    2025

  49. [57]

    Artificial intelligence as an emerging technology in the current care of neurological disorders,

    U. K. Patel, A. Anwar, S. Saleem, P. Malik, B. Rasul, K. Patel, R. Yao, A. Seshadri, M. Yousu- fuddin, and K. Arumaithurai, “Artificial intelligence as an emerging technology in the current care of neurological disorders,” Journal of Neurology, vol. 268, no. 5, 2019

    2019

  50. [58]

    Long-term treatment with responsive brain stimulation in adults with refractory partial seizures,

    G. K. Bergey, M. J. Morrell, E. M. Mizrahi, A. Goldman, D. King-Stephens, D. Nair, S. Srini- vasan, B. Jobst, R. E. Gross, D. C. Shields, G. Barkley, V . Salanova, P. Olejniczak, A. Cole, S. S. Cash, K. Noe, R. Wharen, G. Worrell, A. M. Murro, J. Edwards, M. Duchowny, D. Spenc...

    2015

  51. [59]

    Wearable digital health technology for epilepsy,

    E. Donner, O. Devinsky, and D. Friedman, “Wearable digital health technology for epilepsy,” New England Journal of Medicine, vol. 390, no. 8, 2024

    2024

  52. [60]

    Emerging insights into the genesis of epilepsy,

    J. O. McNamara, “Emerging insights into the genesis of epilepsy,”Nature, vol. 399, no. 6738, pp. A15–A22, Jun. 1999

    1999

  53. [61]

    Patient-independent seizure detection based on long-term iEEG and a novel lightweight CNN,

    X. Si, Z. Yang, X. Zhang, Y . Sun, W. Jin, L. Wang, S. Yin, and D. Ming, “Patient-independent seizure detection based on long-term iEEG and a novel lightweight CNN,”Journal of Neural Engineering, vol. 20, no. 1, 2023. 13

    2023

  54. [62]

    Neuro-GPT: Developing a foundation model for EEG,

    W. Cui, W. Jeong, P. Thölke, T. Medani, K. Jerbi, A. A. Joshi, and R. M. Leahy, “Neuro-GPT: Developing a foundation model for EEG,” arXiv preprint arXiv:2311.03764, 2023

    2023 arXiv

  55. [63]

    DeBERTa: Decoding-enhanced bert with disentangled attention,

    P. He, X. Liu, J. Gao, and W. Chen, “DeBERTa: Decoding-enhanced bert with disentangled attention,” in International Conference on Learning Representations (ICLR), 2021

    2021

  56. [64]

    Axial attention in multidimensional transformers,

    J. Ho, N. Kalchbrenner, D. Weissenborn, and T. Salimans, “Axial attention in multidimensional transformers,” arXiv preprint arXiv:1912.12180, 2019

    1912 arXiv

  57. [65]

    CCNet: Criss-cross attention for semantic segmentation,

    Z. Huang, X. Wang, L. Huang, C. Huang, Y . Wei, and W. Liu, “CCNet: Criss-cross attention for semantic segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019

    2019

  58. [66]

    Reformer: The efficient transformer,

    N. Kitaev, L. Kaiser, and A. Levskaya, “Reformer: The efficient transformer,” inInternational Conference on Learning Representations (ICLR), 2020

    2020

  59. [67]

    Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX,

    B. Wang, “Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX,” 2021. [Online]. Available: https://github.com/kingoflolz/ mesh-transformer-jax

    2021

  60. [68]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  61. [69]

    Neurons that fire together also conspire together: Is normal sleep circuitry hijacked to generate epilepsy?

    M. P. Beenhakker and J. R. Huguenard, “Neurons that fire together also conspire together: Is normal sleep circuitry hijacked to generate epilepsy?” Neuron, 2009

    2009

  62. [70]

    Controversies on the network theory of epilepsy: Debates held during the ictals 2019 conference,

    H. P. Zaveri, B. Schelter, C. A. Schevon, P. Jiruska, J. G. Jefferys, G. Worrell, A. Schulze- Bonhage, R. B. Joshi, V . Jirsa, M. Goodfellow, C. Meisel, and K. Lehnertz, “Controversies on the network theory of epilepsy: Debates held during the ictals 2019 conference,” Seizure,...

    2019

  63. [71]

    Deep anomaly detection of seizures with paired stereoelectroencephalography and video recordings,

    M. L. Martini, A. A. Valliani, C. Sun, A. B. Costa, S. Zhao, F. Panov, S. Ghatan, K. Rajan, and E. K. Oermann, “Deep anomaly detection of seizures with paired stereoelectroencephalography and video recordings,” Scientific Reports, vol. 11, no. 1, 2021

    2021

  64. [72]

    Objective evaluation metrics for automatic classification of EEG events,

    S. Ziyabari, V . Shah, M. Golmohammadi, I. Obeid, and J. Picone, “Objective evaluation metrics for automatic classification of EEG events,” arXiv preprint arXiv:1712.10107, 2017

    2017 arXiv

  65. [73]

    Validation of temporal scoring metrics for automatic seizure detection,

    V . Shah, I. Obeid, J. Picone, G. Ekladious, R. Iskander, and Y . Roy, “Validation of temporal scoring metrics for automatic seizure detection,” in 2020 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), 2020

    2020

  66. [74]

    Critical evaluation of four different seizure detection systems tested on one patient with focal and generalized tonic and clonic seizures,

    A. Van de Vel, K. Verhaert, and B. Ceulemans, “Critical evaluation of four different seizure detection systems tested on one patient with focal and generalized tonic and clonic seizures,” Epilepsy & Behavior, vol. 37, 2014

    2014

  67. [75]

    Seizure detection at home: Do devices on the market match the needs of people living with epilepsy and their caregivers?

    E. Bruno, P. F. Viana, M. R. Sperling, and M. P. Richardson, “Seizure detection at home: Do devices on the market match the needs of people living with epilepsy and their caregivers?” Epilepsia, 2020

    2020

  68. [76]

    Intracranial eeg seizure onset and termination patterns and their association,

    L. A. Shokooh, D. H. Toffa, P. Pouliot, F. Lesage, and D. K. Nguyen, “Intracranial eeg seizure onset and termination patterns and their association,” Epilepsy Research, vol. 176, 2021

    2021

  69. [77]

    A few thoughts on “what is a seizure?

    J. Gotman, “A few thoughts on “what is a seizure?”,”Epilepsy & Behavior, vol. 22, pp. S2–S3, Dec. 2011

    2011

  70. [78]

    Structural, geometric and genetic factors predict interregional brain connectivity patterns probed by electrocorticography,

    R. F. Betzel, J. D. Medaglia, A. E. Kahn, J. Soffer, D. R. Schonhaut, and D. S. Bassett, “Structural, geometric and genetic factors predict interregional brain connectivity patterns probed by electrocorticography,”Nature Biomedical Engineering, 2019

    2019

  71. [79]

    Geometric constraints on human brain function,

    J. C. Pang, K. M. Aquino, M. Oldehinkel, P. A. Robinson, B. D. Fulcher, M. Breakspear, and A. Fornito, “Geometric constraints on human brain function,” Nature, vol. 618, no. 7965, 2023

    2023

  72. [80]

    Report of the committee on methods of clinical examination in electroen- cephalography: 1957,

    H. H. Jasper, “Report of the committee on methods of clinical examination in electroen- cephalography: 1957,” Electroencephalography and Clinical Neurophysiology, vol. 10, no. 2, pp. 370–375, 1958. 14 A Details on multi-variate parallel attention (MVPA) Algorithm 1 illustrates...

    1957

  73. [720]

    In bold are the best MSE results, in italics are the second best

    The vanilla Transformer is also included as a point of comparison. In bold are the best MSE results, in italics are the second best. MVPFormer notably outperforms the vanilla Transformer and is competitive with all baselines, having either the best or second best result in mos...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.