{"id":"bf51565d-3fec-4af5-9871-e68d2b1c5ab8","arxiv_id":"2512.02978","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"No universal EEG decoding pipeline exists: covariance tangent-space projection and CSP rank best on average across three datasets, but per-subject winners vary — and the claimed 340,000+ unique configurations is really the count of subject-level evaluations.","lead":"This paper benchmarked EEG motor-imagery decoding pipelines from three public datasets and found that no single method wins for everyone: tangent-space projection and CSP lead on average, while nonlinear features win for specific individuals. The headline '340,000+ unique configurations' actually counts per-subject evaluations, and the per-subject wins are not tested against chance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-subject 'personalized' wins are likely max-over-pipeline selection noise; no permutation baseline is reported, and the protocol description is internally inconsistent.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the personalization conclusion rests on per-subject winners that are maximal over hundreds of pipelines, with no null model to control for selection noise. I agree with this diagnosis. The average ranking of cov_tgsp and CSP is coherent with the data and with prior Riemannian benchmarking, so the core descriptive finding does not warrant rejection. But the stronger claim—that particular nonlinear methods genuinely beat spatial defaults for particular participants—requires a chance-level comparison that the paper does not provide. The internally inconsistent protocol descriptions (single deterministic split vs. repeated k-fold) further undermine confidence in the exact scores used to identify those winners. The inflated '342,604 unique configurations' count and absence of code/supplementary material are additional transparency problems, but they are secondary to the selection-noise issue. Since the reader already returned a CONDITIONAL verdict that captures this fragility, my stress-test does not move the verdict; it confirms the condition. No change is needed.","tokens_in":19520,"tokens_out":4583,"duration_ms":46918,"concrete_test":"Permutation test on PhysionetMI using the identical protocol (fix whether it is the deterministic MOABB split or repeated k-fold): shuffle trial labels within each training/test split, rerun the full pipeline grid, and for each subject record the maximum accuracy over all pipelines and the identity of the winning family. Repeat this null procedure enough times (e.g., 100 permutations) to form the distribution of best-of-k accuracies and winner-family frequencies. Then locate subjects #57, #59, #17, #73 in these null distributions. If the observed best-pipeline accuracy for these subjects is below the 95th percentile of the null max, or if the winning-family identity is not reproduced in the permutation tail, the per-subject personalization wins are consistent with selection noise and Fig. 7's conclusion should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The fragile hinge is the claim that nonlinear/connectivity pipelines win for specific PhysionetMI subjects (#57/#59 with Hjorth, #17/#73 with instantaneous connectivity in Fig. 7), which is used to conclude that personalized pipeline selection is needed. These per-subject winners are selected as maxima over ~2,200 subject-level pipeline evaluations. Under the null that all pipelines are equivalent, max-over-pipeline accuracy grows with the number of pipelines, so apparent per-subject 'best methods' are expected. The paper nowhere reports a permutation test, a chance-level max baseline, or an adjustment for multiple comparisons, even though its Introduction explicitly lists nonparametric permutation testing as a requirement for rigorous benchmarking. The problem is compounded by contradictory protocol descriptions: the Assessment section states MOABB within-session uses 'a single, deterministic split per session,' while the Results section says the same evaluation uses 'repeated k-fold cross-validation.' A deterministic split and repeated k-fold have different variance and different effective multiplicity; the paper's assertion that differences are attributable to feature extraction/scaling/classifier 'rather than to idiosyncrasies in data partitioning' is therefore not grounded. Without a null distribution for the per-subject maximum, the personalization conclusion cannot be separated from selection noise. The average finding (cov_tgsp/CSP best on average, weaker on PhysionetMI) is not the issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a large-scale benchmark of EEG motor-imagery decoding pipelines on three public datasets (PhysionetMI, Zhou2016, Cho2017), comparing spatial methods (CSP, covariance tangent-space projection), instantaneous coherence plus tangent-space projection, and nonlinear features (Hjorth parameters, Higuchi fractal dimension, SVD entropy, among others), combined with several scalers and classical classifiers. Using the MOABB within-session protocol and per-subject evaluation, the authors report that cov_tgsp and CSP achieve the highest average accuracies across datasets and frequency bands, while nonlinear/connectivity methods perform worse on average but appear to win for specific PhysionetMI participants. The paper interprets this as evidence against a universal one-size-fits-all pipeline and in favor of personalized pipeline selection.","tokens_in":19725,"tokens_out":6078,"duration_ms":55757,"significance":"If the empirical ranking is accepted, the paper provides a useful descriptive benchmark: the average advantage of Riemannian tangent-space and CSP over nonlinear/connectivity feature families is consistent across tables and datasets, and the per-dataset variation is informative. Strengths include the use of three public datasets through MOABB, fixed random states for reproducibility, and subject-level reporting. However, the paper's headline scale claim ('342,604 unique configurations') is not supported by its own reporting, and the personalized-pipeline conclusion rests on per-subject best-pipeline maxima without a null model. These issues need to be addressed before the benchmark's conclusions can be considered established.","major_comments":[{"comment":"The paper repeatedly calls 342,604 the number of 'unique combinations' or 'unique algorithmic configurations.' The Results arithmetic, however, is 243,250 + 98,730 + 624 = 342,604, where the first two numbers are explicitly described as 'evaluations' for 109 and 4 subjects, respectively. An evaluation instance (subject × pipeline × band) is not a unique configuration. The actual grid of feature extractors, scalers, classifiers, and frequency bands is far smaller. Please correct the count in the title/abstract and report the true number of unique pipeline configurations separately from the total number of subject-level evaluations.","section":"Abstract; Results (General benchmark)"},{"comment":"The personalization conclusion is built on per-subject winners such as Hjorth for PhysionetMI subjects #57/#59 and connectivity for #17/#73. These are selected as maxima over a very large number of pipeline evaluations per subject. Under the null that all pipelines are equivalent, the maximum over many pipelines will systematically exceed the mean and will produce apparent 'best methods' by chance. The paper reports no permutation test, no chance-level max distribution, and no correction for multiple comparisons, even though the Introduction explicitly calls for nonparametric permutation testing in benchmarking. The problem is compounded by an internal contradiction: the Assessment section states that MOABB within-session evaluation uses 'a single, deterministic split per session,' while the Results section states that it 'performs repeated k-fold cross-validation.' These two protocols h","section":"Assessment vs. Results (evaluation protocol); Figures 5-7"},{"comment":"The choice of the 8-15 Hz band appears to be post hoc: 'In our preliminary experiments, we explored a broad spectrum of frequencies ranging from 8 to 30 Hz. Subsequently, we refined our methodology by implementing specific bandpass filtering, concentrating our analysis on sensorimotor rhythms, particularly the mu rhythm which spans 8 to 15 Hz.' If the band was chosen after inspecting results on the same datasets, then the comparison between 8-15 and 8-30 Hz is not a confirmatory evaluation and the reported performance advantage could be inflated by selection. Please clarify whether the band was specified a priori, or treat the band choice as a tuned hyperparameter and report the appropriate conditional analysis.","section":"Materials & Methods (EEG Signal Processing, Filtering)"},{"comment":"The pooled 'across all datasets' mean accuracy is computed over all subjects, so PhysionetMI (109 subjects) dominates the average relative to Zhou2016 (4 subjects) and Cho2017 (52 subjects). Because PhysionetMI is also the dataset with the weakest absolute performance (best pipeline 0.64 vs 0.74/0.87), the pooled mean of 0.69 is not a balanced summary of the three datasets. Please report dataset-weighted means or clearly label the pooled figure as subject-weighted.","section":"Results (Overall assessment); Tables 1-2"}],"minor_comments":[{"comment":"There are typographical errors: 'ASTRACT' in the main text, 'Riemannan' in the Introduction, and 'classificaiton' in the Discussion. A careful proofreading pass is needed.","section":"General"},{"comment":"The Results list Petrosian fractal dimension, approximate entropy, and detrended fluctuation analysis as evaluated features, but these are absent from the reported tables, figures, and family-level summaries. Please either include these results or state that they were excluded from the final benchmark.","section":"Results"},{"comment":"The Code availability section lists software dependencies but does not provide a link or DOI to the actual benchmarking code. For a study whose contribution is a benchmark, please provide a public repository or an archival release with the exact pipeline configuration.","section":"Materials & Methods (Code availability)"},{"comment":"The numbers 243,250 and 98,730 are described as evaluations for 109 and 4 subjects respectively, but these counts do not divide evenly by the subject counts. Please clarify exactly what is included in these counts (subjects × pipelines × frequency bands × cross-validation folds, or another product).","section":"Results (General benchmark)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a genuinely useful descriptive core, but the title and abstract overstate the scale of the benchmark, and the personalization inference currently rests on an uncontrolled max-over-pipelines analysis. I would ask the authors to correct the count, resolve the evaluation-protocol contradiction, and supply a permutation-based null analysis for per-subject best-pipeline scores. A code/data release would also substantially increase the value of this work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The useful core is the per-subject scale: three public datasets, two bands, all the standard feature extractors and scalers, and a clear table showing cov_tgsp and CSP on top with nonlinear features trailing. That matches the MOABB consensus, so I'm not worried about the central tendency. The dataset differences (weakest on PhysionetMI, best on Zhou2016) are also plausible.\n\nWhat's new is the per-subject map itself, and the observation that for a few PhysionetMI subjects the best pipeline is Hjorth or instantaneous connectivity. But that observation is not backed by statistics. Each subject has roughly 2,200 pipeline evaluations (243,250 / 109). If you take the max over that many pipelines, you expect some nonlinear method to win even when all pipelines are equivalent. The paper never reports a permutation test, a chance-level max baseline, or a correction for multiple comparisons — even though its own introduction says nonparametric testing is required. So the personalization conclusion is a fragile hinge. The average-level claim does not depend on that hinge, but the 'BCI illiteracy as mismatch' interpretation does.\n\nTwo more things. The title says '342,604+ unique algorithmic configurations,' but the paper's own arithmetic (243,250 + 98,730 + 624) is the number of subject-level evaluation runs, not unique configurations. The unique grid is at most a few thousand. That needs to be fixed in title, abstract, and text. Second, the evaluation protocol is described inconsistently: the Assessment section says MOABB uses a single deterministic split per session, and the Results section says repeated k-fold cross-validation. Those are different procedures with different variance and different multiplicity. The paper's claim that differences are attributable to feature extractor/scaler/classifier 'rather than idiosyncrasies in data partitioning' is not grounded on that contradiction.\n\nNo code and no supplementary tables are provided, so independent verification is impossible. That is a real problem for a benchmarking paper. There is also a stray section on quantum machine learning and barren plateaus that doesn't belong, and some references are mismatched (e.g., ref. 31 in the Introduction).\n\nOn balance: the average descriptive result is solid and useful. The headline count and the personalized-winner claim need either stronger evidence (permutation nulls, code release) or more modest wording. A serious referee should see this, but it needs substantial revision before I would trust the personalization angle.","headline":"The average finding (cov_tgsp/CSP are the best defaults) is probably right, but the headline 342,604 figure counts subject-level evaluations, not configurations, and the per-subject 'personalized winners' aren't separated from selection noise.","tokens_in":20406,"tokens_out":2829,"would_cite":false,"duration_ms":26834,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across 340,000+ EEG decoding pipelines, covariance tangent-space projection averages highest — but no single method wins for every user.","keywords":["EEG decoding","motor imagery","brain-computer interface","Common Spatial Patterns","Riemannian geometry","tangent space projection","nonlinear features","personalization"],"falsifier":"For each subject on the heterogeneous dataset, compare the reported best-pipeline accuracy against the distribution of accuracies of all equivalent pipelines for that subject: if the maximum over hundreds of configurations is expected to exceed the reported winner by chance alone (a multiple-comparisons effect), the individual-level exceptions no longer support personalization. A cleaner test: rerun evaluation with a permutation-based chance level and with repeated cross-validation across multiple splits; if the per-subject nonlinear winners do not replicate across splits, the personalization","tokens_in":19285,"feed_emoji":"🧠","tokens_out":6134,"duration_ms":53987,"temperature":0.7,"pith_summary":"The paper sets out to answer two questions: which EEG decoding pipeline gives the best average performance for motor-imagery brain-computer interfaces, and whether any one configuration works for everyone. Benchmarking 342,604 feature-extraction, scaling, and classifier combinations across three public datasets and two frequency bands, at the level of individual participants, it finds that tangent-space projection of covariance matrices is the strongest average performer (0.69), with Common Spatial Patterns close behind. The paper's central conclusion is negative: performance is dataset-dependent and per-user winners vary, with nonlinear methods beating spatial defaults for specific individuals on the most heterogeneous dataset. The authors argue this means 'BCI illiteracy' is often an algorithm-user mismatch, and that practical systems should personalize or adapt their decoding pipeline.","feed_headline":"340,000+ pipelines ranked: no universal EEG decoder wins","feed_subtitle":"Tangent-space covariance projection tops average accuracy, but per-user winners vary — a case for personalized BCIs.","key_machinery":"The load-bearing machinery is the pipeline grid: every combination of a feature extractor, a feature scaler, and a classifier is treated as a candidate decoder. The two feature extractors that carry the main result are Common Spatial Patterns, which learns spatial filters maximizing variance contrast between classes, and tangent-space projection, which maps symmetric positive definite covariance matrices (and coherence matrices) into a Euclidean space where standard classifiers apply. Evaluating every configuration per subject and per frequency band is what allows the paper to report both an average ranking and the individual exceptions to it.","core_discovery":"The discovery is a ranking plus an exception: on average, spatial methods dominate — covariance tangent-space projection reaches a mean accuracy of 0.69, and CSP reaches 0.64–0.66 — while nonlinear and connectivity pipelines cluster at 0.53–0.61. But the ranking is not stable. On the largest, most heterogeneous dataset the top pipeline only reaches 0.64, and per-subject analysis shows individuals where Hjorth parameters or connectivity-based tangent-space features outperform the spatial defaults. The paper takes this as evidence that no universal decoder exists and that individualized pipeline selection — or adaptive systems that match each user's neurophysiological profile — is the route to","pith_inferences":["A calibration-phase design that tests a small set of diverse pipelines per user could turn the paper's per-subject variation into a practical personalization strategy without needing a universal winner.","The conclusion that no universal pipeline exists would be sharpened by extending the same exhaustive benchmarking to other BCI paradigms (e.g., P300 or SSVEP) and to invasive recordings, where signal characteristics differ.","A natural follow-up experiment is to check whether the per-subject 'winning' pipelines (Hjorth for some users, connectivity for others) are stable across independent sessions; stability is what would make personalization clinically usable.","The 0.69 average of the top method leaves substantial headroom, suggesting that combining spatial and nonlinear features in ensembles, rather than choosing one family, may yield the largest gains."],"forward_implications":["Covariance tangent-space projection is the best default choice for EEG motor-imagery decoding when no user information is available.","Average rankings underestimate the difficulty of real-world deployment: on heterogeneous populations the best pipeline's accuracy can drop well below its average.","Nonlinear and connectivity features carry complementary information for some individuals, so fixed spatial-only pipelines will under-serve those users.","'BCI illiteracy' should be treated as a solvable engineering problem of matching decoder to user, not an inherent user trait.","Adaptive systems that switch or combine pipelines per user are a concrete path suggested by the data."],"fun_headline_variants":["340k EEG pipelines tested: no one-size-fits-all decoder","Top EEG decoder? Depends on the user, benchmark finds","Adaptive BCIs needed: 340k pipelines, no universal winner","Per-user EEG decoding wins: benchmark shows no universal method","Personalized BCIs urged: 340k+ pipelines, no global champ"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's within-session evaluation protocol fairly ranks the 342,604 pipelines, and the per-subject winners reflect genuine pipeline-user matches rather than noise from choosing the best of many configurations per subject.","fun_headline_variants_meta":{"raw":{"variants":["340k EEG pipelines tested: no one-size-fits-all decoder","Top EEG decoder? Depends on the user, benchmark finds","Adaptive BCIs needed: 340k pipelines, no universal winner","Per-user EEG decoding wins: benchmark shows no universal method","Personalized BCIs urged: 340k+ pipelines, no global champ"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1172,"prompt_tokens":800,"completion_tokens":372,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":281}},"tokens_in":544,"tokens_out":372,"duration_ms":4536,"temperature":1.0,"reasoning_tokens":281,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T18:52:35.591989+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For each subject on the heterogeneous dataset, compare the reported best-pipeline accuracy against the distribution of accuracies of all equivalent pipelines for that subject: if the maximum over hundreds of configurations is expected to exceed the reported winner by chance alone (a multiple-comparisons effect), the individual-level exceptions no longer support personalization. A cleaner test: rerun evaluation with a permutation-based chance level and with repeated cross-validation across multiple splits; if the per-subject nonlinear winners do not replicate across splits, the personalization","supporting_citations":[],"review_version":1}