REVIEW 3 major objections 6 minor 45 references
The 2025 PNPL Competition: Speech Detection and Phoneme Classification in the LibriBrain Dataset
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This competition paper presents the largest within-subject MEG dataset to date, over 50 hours, paired with two standardized speech-decoding tasks to spark an 'ImageNet moment' for non-invasive brain-computer interfaces.
desk verdict First standardized benchmark for non-invasive speech decoding on a 50+ hour MEG dataset, with a real infrastructure contribution; but the validity of the labels is unverified and the baselines lack detail—worth refereeing conditionally. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is LibriBrain itself: more than 50 hours of 306-channel magnetoencephalography from a single subject listening to audiobooks, sampled at 250 Hz and aligned to word- and phoneme-level transcriptions inherited from the source speech corpus. Around this dataset, the paper's machinery is standardization: a Python library that downloads and batches the data into deep-learning-ready tensors, public train, validation, test, and competition holdout splits, the F1-macro metric, and reference models that establish baseline scores of 68.04% and 60.39%. The speech-detection task labels every temporal sample as speech or non-speech, making efficient use of each recording at 250 labels per second; the phoneme task uses more than 1.5 million examples across 39 classes, an order of magnitude more examples per class than word classification. Together these components turn a raw data release into a competition with comparable endpoints.
What would settle it
Permute the phoneme labels across training windows and retrain the reference model: if the F1-macro stays materially above chance, the model is exploiting a non-phoneme artifact; likewise, if shifting the speech labels by one 4 ms sample on a held-out subset does not drop the reference model's speech-detection score, the labels are too coarse or mistimed to pin the task to the neural signal.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that LibriBrain is the largest within-subject MEG dataset recorded to date, over 50 hours of 306-sensor data from one healthy participant listening to audiobooks, downsampled to 250 Hz, and that it is large enough to support a new kind of benchmark. The paper pairs the dataset with two foundational tasks that have direct analogues in automatic speech recognition: per-sample binary speech detection and fixed-window phoneme classification over the 39 standard phoneme classes. It reports reference models reaching F1-macro scores of 68.04% for speech detection and 60.39% for phoneme classification, compared with naive majority-class baselines of 45.30% and 0.47%, and argues that these tasks are far more label-efficient than word classification. The contribution is not a new model or algorithm; it is the assertion that a deep single-subject dataset plus standardized evaluation can convert a field where end-to-end non-invasive decoders have hovered at chance into a measurable, comparable research program.
Load-bearing premise
The whole benchmark rests on the assumption that the word and phoneme alignments inherited by LibriBrain are correct and that the audiobook audio stayed synchronized with the MEG clock; if those labels are wrong or drift, the reference scores and every leaderboard entry measure alignment error rather than neural decoding.
Editorial extensions
If this is right
- Any research group can now evaluate a speech decoder on the same held-out MEG data and report an F1-macro score directly comparable with the published 68.04% and 60.39% baselines.
- If the reference models are reproducible, the two tasks give the field sanity baselines: speech-detection systems should score well above 50%, and phoneme systems well above the 2.56% chance level, replacing near-chance end-to-end results reported for MEG-to-text.
- The Standard track, restricted to LibriBrain data, tests whether algorithmic innovation alone can push decoding performance on a fixed corpus.
- The Extended track, which permits any external data, provides a controlled test of whether pooling additional brain recordings improves decoding under the same evaluation metric.
- A successful competition would make future non-invasive speech-decoding papers report comparable benchmark numbers, the same way speech recognition moved to shared evaluation corpora.
Reading between the lines
- Because LibriBrain is a single participant, the benchmark's conclusions about decoding accuracy may not transfer to other brains; a natural next step would be a multi-speaker extension, which the paper's planned multi-year curriculum already gestures toward.
- The label-correctness assumption could be tested cheaply: if a one-hour subset is re-aligned with an independent forced aligner and the reference model's scores change materially, the leaderboard would be measuring alignment quality as much as neural decoding.
- The competition structure could also serve as a probe for scaling laws: if Extended-track entries with more compute consistently outscore Standard-track entries on the same holdout, that would be direct evidence that non-invasive speech decoding benefits from data and compute scaling rather than only architecture.
- If the two foundational tasks prove solvable at the reported levels, it would suggest that the information needed for speech decoding is present in MEG and that the bottleneck for full text decoding lies in sequence-level decoding, not in the absence of a decodable signal.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes the 2025 PNPL competition, the first competition for language decoding from non-invasive brain data. It introduces LibriBrain, a single-subject MEG corpus of over 50 hours of audiobook-listening data, and the pnpl Python library for data access and integration with PyTorch. The competition defines two supervised tasks, Speech Detection (binary per-sample classification) and Phoneme Classification (39 CMU/ARPAbet classes), with fixed train/validation/test/competition-holdout splits and an F1-macro evaluation metric. Reference models achieve F1-macro scores of 68.04% for Speech Detection and 60.39% for Phoneme Classification, compared with naive baselines of 45.30% and 0.47%. The paper also details the competition tracks, rules, prizes, schedule, and community infrastructure.
Significance. If the label synchronization and baselines are trustworthy, this paper provides a valuable community resource: the largest within-subject MEG dataset for speech decoding, standardized tasks and splits, an accessible data-loading library, tutorials, a leaderboard, and a clear evaluation protocol. These infrastructure contributions are concrete and will likely accelerate comparable work across research groups. The central claim is not a mathematical derivation but an empirical benchmark, so the paper's value rests on the correctness of the dataset labels and the reproducibility of the reference scores.
major comments (3)
- [1.3 (Data) and Table 2] The benchmark's validity rests on the inherited LibriSpeech alignments being correctly synchronized with the MEG samples, but Section 1.3 provides no trigger description, no clock-drift measurement, and no alignment validation. A misalignment of even tens of milliseconds would systematically mislabel phoneme boundaries and short non-speech pauses, and then the reference scores in Table 2 and all leaderboard rankings could reflect temporal autocorrelation of labels rather than neural decoding. Please add synchronization evidence, for example: trigger timing reports, estimated audio-to-MEG clock drift, an audio-envelope-to-MEG cross-correlation or temporal-response-function analysis on held-out segments, or manual QA on a sample of phoneme boundaries. This is load-bearing because every supervised label and both reference scores depend on it.
- [1.6 (Baselines) and Table 2] The reference model scores are reported as single F1-macro numbers with no architecture description, training hyperparameters, or uncertainty estimates. Since Section 2.2 makes beating 68.04% for Speech Detection and 60.39% for Phoneme Classification a condition for prizes, these numbers must be reproducible and statistically credible. Please report the model architectures, training details, number of independent runs, and confidence intervals or standard deviations. In addition, clarify what "averaged over sets of 100 samples" means for the phoneme classification model: is the input a fixed 100-sample window, or are predictions averaged over 100 raw sample labels, or both?
- [Table 1 and Section 1.6] The public validation and test sets are very small relative to the training set (0.36 and 0.38 hours out of 51.57 hours). For sample-level Speech Detection this yields on the order of 3e5 labeled samples, but for Phoneme Classification, when labels are averaged over 100-sample windows, this reduces to roughly 3e3 windows across 39 classes, which can produce wide error bars on F1-macro. Since rankings and prize eligibility are based on these splits, please report per-class counts, bootstrap confidence intervals for the reference scores, and a brief analysis of whether the fixed small holdout is sufficient for reliable discrimination among submissions.
minor comments (6)
- [1.5 (Metrics)] Typo: "prepose" should be "propose" in the sentence "...which we prepose to do...".
- [1.4 (Tasks and application scenarios)] Typo: "attemps" should be "attempts" in the sentence "early attemps in EEG and MEG have yielded weak results".
- [2.3 (Schedule and readiness)] The date line "10 May 26 May, 2025" appears garbled; it should likely read "10 May 2025" and "26 May 2025" as separate events.
- [Organizing team] Typo: "researchs" should be "researchers" in the sentence "a strategic collaboration between researchs in Oxford, Montreal, Switzerland, Google, and Meta".
- [1.2 (Novelty) and References] The claim of the "largest within-subject MEG dataset recorded to date" is cited only to the companion preprint [42], which shares several authors with this paper. Please state the overlap explicitly and, if possible, provide independent or concurrently reviewed evidence for the scale comparison shown in Figure 1B.
- [1.3 and 1.4] Consider listing the 39 phoneme classes or citing the exact ARPAbet inventory, since the phoneme label set is a core element of the task. Also, define the "sets of 100 samples" windowing exactly and state whether windows are overlapping or disjoint.
Circularity Check
No circular derivation: the paper reports a new dataset, defines two supervised tasks, and gives reference-model baselines; no claim reduces to its own inputs by construction.
full rationale
This paper is a competition/benchmark proposal rather than a derivation. The central claims are (i) the existence and scale of LibriBrain, (ii) the definition of two supervised tasks with fixed splits, and (iii) reference baseline scores. None of these is derived from an assumed input in a way that would constitute circularity. The 68.04% and 60.39% reference F1 scores (Table 2, Section 1.6) are presented as trained reference models; they are empirical benchmark results, not fitted parameters renamed as predictions. The dataset size claim is supported internally by Table 1 (51.57 train hours plus validation/test splits) and by the companion dataset paper [42]; citing the companion paper is a normal self-citation, and although the 'largest within-subject MEG dataset recorded to date' superlative relies partly on the authors' own reporting, it is an empirical assertion backed by the paper's own duration table and comparison to prior datasets, not a mathematical consequence that reduces to the citation. The Extended-track motivation cites the organizers' own prior work [17, 18, 31] on pooling MEG data, but that citation is used to justify a competition design choice (permitting external training data), not to establish a predicted result. No uniqueness theorem, ansatz, or fitted input is imported from self-citations. The unvalidated LibriSpeech-alignment/synchronization assumption flagged in review is a correctness/validity risk for the benchmark, but it is not circularity: the labels are treated as ground-truth inputs to the supervised tasks, not as quantities derived from MEG predictions or from the competition design. Overall, no step in the paper's reasoning chain reduces by construction to its own inputs.
Assumptions & free parameters
free parameters (2)
- Reference Speech Detection F1-macro =
68.04%
- Reference Phoneme Classification F1-macro =
60.39%
assumptions (4)
- domain assumption Deep, within-subject data yields the largest gains in decoding performance
- domain assumption Word- and phoneme-level alignments from LibriSpeech apply to the MEG recordings
- domain assumption MEG can approach or exceed invasive precision for speech decoding under ideal conditions
- domain assumption F1-macro with balanced-class chance assumptions is the right evaluation
Cite this review
Pith. "Pith review of The 2025 PNPL Competition: Speech Detection and Phoneme Classification in the LibriBrain Dataset." pith.science (2026). https://pith.science/paper/NSOSXHTK
@misc{pith2026250610165,
author = {Pith},
title = {Pith review of: The 2025 PNPL Competition: Speech Detection and Phoneme Classification in the LibriBrain Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/NSOSXHTK}},
note = {Machine review of arXiv:2506.10165}
}
read the original abstract
The advance of speech decoding from non-invasive brain data holds the potential for profound societal impact. Among its most promising applications is the restoration of communication to paralysed individuals affected by speech deficits such as dysarthria, without the need for high-risk surgical interventions. The ultimate aim of the 2025 PNPL competition is to produce the conditions for an "ImageNet moment" or breakthrough in non-invasive neural decoding, by harnessing the collective power of the machine learning community. To facilitate this vision we present the largest within-subject MEG dataset recorded to date (LibriBrain) together with a user-friendly Python library (pnpl) for easy data access and integration with deep learning frameworks. For the competition we define two foundational tasks (i.e. Speech Detection and Phoneme Classification from brain data), complete with standardised data splits and evaluation metrics, illustrative benchmark models, online tutorial code, a community discussion board, and public leaderboard for submissions. To promote accessibility and participation the competition features a Standard track that emphasises algorithmic innovation, as well as an Extended track that is expected to reward larger-scale computing, accelerating progress toward a non-invasive brain-computer interface for speech.
Figures
Reference graph
Works this paper leans on
-
[1]
B. Accou et al. SparrKULee: A speech-evoked auditory response repository of the KU Leuven, containing EEG of 85 participants. bioRxiv, 2023. URL https://doi.org/10.1101/2023. 07.24.550310
doi:10.1101/2023 2023
- [2]
-
[3]
E. L. Barratt, S. T. Francis, P. G. Morris, and M. J. Brookes. Mapping the topological or- ganisation of beta oscillations in motor cortex using MEG. NeuroImage, 181:831–844, 2018. doi: 10.1016/j.neuroimage.2018.06.041. URL https://doi.org/10.1016/j.neuroimage. 2018.06.041
- [4]
-
[5]
T. M. Blakely, K. J. Miller, R. P. N. Rao, M. D. Holmes, and J. G. Ojemann. Localization and classification of phonemes using high spatial resolution electrocorticography (ECoG) grids. In 2008 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 4964–4967. IEEE, 2008. doi: 10.1109/IEMBS.2008.4650328. URL https://d...
-
[6]
M. P. Broderick, A. J. Anderson, G. M. Di Liberto, M. J. Crosse, and E. C. Lalor. Electrophys- iological correlates of semantic dissimilarity reflect the comprehension of natural, narrative speech. Current Biology, 28:803–809, 2018
work page 2018
-
[7]
J. S. Brumberg, E. J. Wright, D. S. Andreasen, F. H. Guenther, and P. R. Kennedy. Classification of intended phoneme production from chronic intracortical microelectrode recordings in speech- motor cortex. Frontiers in Neuroscience, 5:65, May 2011. doi: 10.3389/fnins.2011.00065. URL https://doi.org/10.3389/fnins.2011.00065
work page Pith review arXiv 2011
-
[8]
N. S. Card, M. Wairagkar, C. Iacobacci, X. Hou, T. Singer-Clark, F. R. Willett, E. M. Kunz, and D. M. Brandman. An accurate and rapidly calibrating speech neuroprosthesis. New England Journal of Medicine, 391(7):609–618, 2024
work page 2024
Show all 45 references
-
[9]
The CMU pronouncing dictionary, 1993
Carnegie Mellon University. The CMU pronouncing dictionary, 1993. http://www.speech. cs.cmu.edu/cgi-bin/cmudict
1993
-
[10]
d’Ascoli, C
S. d’Ascoli, C. Bel, J. Rapin, H. Banville, Y . Benchetrit, C. Pallier, and J.-R. King. Decoding individual words from non-invasive brain recordings across 723 participants. arXiv preprint,
-
[11]
Défossez, C
A. Défossez, C. Caucheteux, J. Rapin, O. Kabeli, and J.-R. King. Decoding speech per- ception from non-invasive brain recordings. Nature Machine Intelligence, 5(10):1097–1107,
-
[12]
S. Furui. Speech recognition: Technology and applications. Academic Press, 2000
2000
-
[13]
J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren. TIMIT acoustic-phonetic continuous speech corpus. Linguistic Data Consortium, 1993. https: //catalog.ldc.upenn.edu/LDC93S1
1993
-
[14]
J. J. Godfrey, E. C. Holliman, and J. McDaniel. Switchboard: Telephone speech corpus for research and development. In Proc. IEEE ICASSP, pages 517–520, 1992
1992
-
[15]
F. H. Guenther, J. S. Brumberg, E. J. Wright, A. Nieto-Castanon, J. A. Tourville, M. Panko, R. Law, S. A. Siebert, J. L. Bartels, D. S. Andreasen, P. Ehirim, H. Mao, and P. R. Kennedy. 10 A wireless brain-machine interface for real-time speech synthesis. PLoS ONE, 4(12):e8218,...
2009 doi
-
[16]
Gwilliams, G
L. Gwilliams, G. Flick, A. Marantz, L. Pylkkanen, D. Poeppel, and J.-R. King. Introducing MEG-MASC: A high-quality magneto-encephalography dataset for evaluating natural speech processing. Scientific Data, 10(1):862, 2023. doi: 10.1038/s41597-023-02752-5. URL https://www.natur...
2023 doi
-
[17]
Jayalath, G
D. Jayalath, G. Landau, and O. Parker Jones. Unlocking non-invasive brain-to-text. arXiv preprint arXiv:2505.13446, 2025
2025 arXiv
-
[18]
Jayalath, G
D. Jayalath, G. Landau, B. Shillingford, M. Woolrich, and O. Parker Jones. The Brain’s Bitter Lesson: Scaling speech decoding with self-supervised learning. Forty-second International Conference on Machine Learning, ICML, 2025
2025
-
[19]
H. Jo, Y . Yang, J. Han, Y . Duan, H. Xiong, and W. H. Lee. Are EEG-to-text models working? arXiv preprint, 2024. URL https://arxiv.org/abs/2405.06459
2024 arXiv
-
[20]
Krizhevsky
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009
2009
-
[21]
Lee and H.-W
K.-F. Lee and H.-W. Hon. Speaker-independent phone recognition using hidden markov models. IEEE Transactions on Acoustics, Speech, and Signal Processing, 37(11):1641–1648, 1989
1989
-
[22]
E. C. Leuthardt, C. Gaona, M. Sharma, N. Szrama, J. Roland, Z. Freudenberg, J. Solis, J. Bres- hears, and G. Schalk. Using the electrocorticographic speech network to control a brain- computer interface in humans. Journal of Neural Engineering, 8(3):036004, Jun 2011. doi: 10. ...
2011 doi
-
[23]
Librivox: Free public domain audiobooks, 2024.https://librivox
LibriV ox V olunteers. Librivox: Free public domain audiobooks, 2024.https://librivox. org
2024
-
[24]
S. L. Metzger, K. T. Littlejohn, A. B. Silva, D. A. Moses, M. P. Seaton, R. Wang, M. E. Dougherty, J. R. Liu, P. Wu, M. A. Berger, I. Zhuravleva, A. Tu-Chan, K. Ganguly, G. K. Anumanchipalli, and E. F. Chang. A high-performance neuroprosthesis for speech decoding and avatar co...
2023
-
[25]
D. A. Moses, S. L. Metzger, J. R. Liu, G. K. Anumanchipalli, J. G. Makin, P. F. Sun, J. Chartier, M. E. Dougherty, P. M. Liu, G. M. Abrams, A. Tu-Chan, K. Ganguly, and E. F. Chang. Neuroprosthesis for decoding speech in a paralyzed person with anarthria. New England Journal of...
2021 doi
-
[26]
M. S. Nieuwland et al. Large-scale replication study reveals a limit on probabilistic prediction in language comprehension. eLife, Apr. 2018
2018
-
[27]
Panayotov, G
V . Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. IEEE, 2015
2015
-
[28]
Paszke, S
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-p...
2019
-
[29]
X. Pei, D. L. Barbour, E. C. Leuthardt, and G. Schalk. Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. Journal of Neural Engineering, 8:046028, 2011
2011
-
[30]
L. R. Rabiner and B.-H. Juang. Fundamentals of Speech Recognition. Prentice-Hall, 1993
1993
-
[31]
Ridge and O
J. Ridge and O. Parker Jones. Resolving domain shift for representations of speech in non- invasive brain recordings. arXiv preprint, 2024. URL https://arxiv.org/abs/2410. 19986
2024
-
[32]
Russakovsky, J
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211–252, 2015. 11
2015
-
[33]
Schoffelen et al
J.-M. Schoffelen et al. A 204-subject multimodal neuroimaging dataset to study language processing. Scientific Data, 6:1–13, Apr. 2019
2019
-
[34]
M. W. J. van Es, C. Gohil, A. J. Quinn, and M. W. Woolrich. osl-ephys: a python toolbox for the analysis of electrophysiology data. Frontiers in Neuroscience, 19:1522675, Feb. 2025. doi: 10.3389/fnins.2025.1522675
2025
-
[35]
F. R. Willett, E. M. Kunz, C. Fan, D. T. Avansino, G. H. Wilson, E. Y . Choi, F. Kam- dar, M. F. Glasser, L. R. Hochberg, S. Druckmann, K. V . Shenoy, and J. M. Hender- son. A high-performance speech neuroprosthesis. Nature, 620:1031–1036, 2023. doi: 10.1038/s41586-023-06377-x
2023 doi
-
[36]
F. R. Willett, J. Li, T. Le, C. Fan, M. Chen, E. Shlizerman, Y . Chen, X. Zheng, T. S. Okubo, T. Benster, H. D. Lee, M. Kounga, E. K. Buchanan, D. Zoltowski, S. W. Linderman, and J. M. Henderson. Brain-to-Text Benchmark ’24: Lessons learned. arXiv preprint arXiv:2412.17227,
-
[37]
G. H. Wilson, S. D. Stavisky, F. R. Willett, D. T. Avansino, J. N. Kelemen, L. R. Hochberg, J. M. Henderson, S. Druckmann, and K. V . Shenoy. Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus. Journal of Neural Engineering, 17(6):066007, 20...
2020 doi
-
[38]
Xiong, J
W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig. Achieving human parity in conversational speech recognition. arXiv preprint arXiv:1610.05256, 2016. URL https://arxiv.org/abs/1610.05256
2016 arXiv
- [39]
-
[40]
Y . Yang, Y . Duan, Q. Zhang, R. Xu, and H. Xiong. NeuSpeech: Decode neural signal as speech. arXiv preprint, 2024. URL https://arxiv.org/abs/2403.01748
2024 arXiv
-
[41]
Y . Yang, H. Jo, Y . Duan, Q. Zhang, J. Zhou, W. H. Lee, R. Xu, and H. Xiong. MAD: Multi- alignment MEG-to-text decoding. arXiv preprint, 2024. URL https://arxiv.org/abs/ 2406.01512
2024
-
[42]
Y . Yang, Y . Duan, H. Jo, Q. Zhang, R. Xu, O. Parker Jones, X. Hu, C.-T. Lin, and H. Xiong. NeuGPT: Unified multi-modal neural GPT. arXiv preprint, 2024. URL https://arxiv.org/ abs/2410.20916
2024 arXiv
-
[45]
Özdogan, G
M. Özdogan, G. Landau, G. Elvers, D. Jayalath, P. Somaiya, F. Mantegna, M. Woolrich, and O. Parker Jones. LibriBrain: Over 50 hours of within-subject MEG to improve speech decoding methods at scale. arXiv preprint, 2025. URL https://arxiv.org/abs/2506.02098. 12
2025 arXiv
-
[2023]
URL https://www.nature.com/articles/ s42256-023-00714-5
doi: 10.1038/s42256-023-00714-5. URL https://www.nature.com/articles/ s42256-023-00714-5
-
[2024]
URL https://arxiv.org/abs/2412.17829
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.