{"id":"2bf58cfd-6fe0-4d8e-971a-21d108eb6981","arxiv_id":"2505.10536","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper reports high rest/task classification accuracy for fNIRS during a tennis game, but asymmetric augmentation of the rest class confounds the benchmark.","lead":"A benchmark study classifies rest versus task brain states from fNIRS signals recorded while participants play a tennis video game, reporting accuracies near 97 percent. The headline numbers are undercut by a preprocessing step: rest segments were artificially stretched from 30 to 60 seconds, so the classifiers may be detecting real-versus-synthetic signal differences instead of brain-state differences.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic rest class is the load-bearing flaw: task trials are real 60-s recordings, rest trials are 30-s recordings upsampled to 60 s, and the paper's own control that equalizes real/synthetic structure drops to chance.","rationale":"In good faith, the paper aims to show that fNIRS distinguishes rest from task states in a naturalistic tennis game and that deep models perform strongly. For that claim to hold, the rest class must represent authentic resting brain activity at the same recording length as the task class. That is exactly what is missing: rest is generated by upsampling 30-s recordings to 60 s. The reader's weakest assumption identifies this same confound, and I agree with the rejection. My partial rather than full agreement is on one point of interpretation: Table 6 removes the real-versus-synthetic asymmetry but also shortens the real task half, so it strongly supports the confound interpretation without isolating it in a single controlled comparison. Even with that caveat, the central claim fails as stated because no analysis demonstrates generalization to real 60-s rest. The leave-participants-out check preserves the asymmetry and therefore cannot repair this. The paper deserves credit for reporting the control and for acknowledging in the Discussion that synthetic signals may lack nuanced temporal dynamics; the problem is that the report treats the augmentation as validated while the decisive real-rest comparison was not run. Because the reader already recommended REJECT and my stress test supports that same verdict, no adjustment is needed; I would keep the verdict unchanged.","tokens_in":18433,"tokens_out":7914,"duration_ms":80678,"concrete_test":"Collect or release true 60-s rest recordings from held-out participants. Rerun the same feature-engineering, LSTM, and ResNet pipelines on real 60-s task versus real 60-s rest, keeping participant separation as in Table 5. If accuracy and AUC remain near the reported 93-98% and 0.99 values, the synthetic-rest concern would not be load-bearing; if performance falls toward chance, the headline numbers were inflated by real-versus-synthetic statistics. As a secondary check, train a classifier with labels real 60-s task versus synthetically extended rest and test it on held-out real 60-s rest; above-chance transfer would confirm the artifact is learnable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that classifiers separate authentic rest and task brain states. This condition is not met: task trials are real 60-s recordings, while rest trials are real 30-s recordings extended to 60 s by Fourier resampling, nonlinear time warping, and injected noise (Section 2.7). Every rest sample is synthetic, so the high accuracies in Tables 1-4 and the leave-participants-out results in Table 5 may reflect learnable differences between real and synthetic signal statistics (spectral structure, warping artifacts, noise floor) rather than cognitive state. The feature-importance results do not resolve this, because slope and RMS are exactly the summary statistics that augmentation can alter. The paper's own control is the strongest evidence: with 30 s real plus 30 s synthetic input in both classes (Section 4.5, Table 6), accuracy drops to chance. One caveat is that this control also shortens real task content from 60 s to 30 s, so it does not isolate the real-versus-synthetic cue by itself; nevertheless, it is inconsistent with the claim that the original classifiers decode robust rest/task physiology, and the Discussion's stated limitation that synthetic signals may lack nuanced temporal dynamics acknowledges the unresolved risk. Without real 60-s rest recordings, the reported accuracies cannot be interpreted as neural-state decoding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports a benchmark of classical machine-learning classifiers, an LSTM-attention network, and CNN architectures (ResNet50, EfficientNet-B0) for binary classification of rest versus task states from 6-channel fNIRS recordings acquired during an interactive tennis video game. After quality exclusion, 45 samples were analyzed. Because task trials lasted 60 s and rest trials 30 s, the authors synthesized 60 s rest trials from the 30 s recordings using Fourier resampling, nonlinear time warping, and injected noise (Section 2.7). The paper reports near-ceiling accuracy for Extra Trees (97.6%), Gradient Boosting (97.3%), and ResNet (95.0%, AUC 99.2%), interprets these as evidence that fNIRS can decode cognitive state in dynamic real-world environments, and includes feature-importance analyses and a leave-participants-out check. Section 4.5 reports a control experiment with 30 s real plus 30 s synthetic input in both classes in which classification performance falls to chance.","tokens_in":18673,"tokens_out":7733,"duration_ms":69117,"significance":"The study has useful elements: the interactive gaming paradigm is ecologically motivated, the benchmark covers several model families, and the paper includes a leave-participants-out validation and three interpretability analyses. However, the central claim is not supported because the rest class is entirely synthetic while the task class is real 60 s data; the classifiers can separate the classes using the real-versus-synthetic structure rather than authentic neural states. The authors' own control (Table 6) shows chance-level performance when both classes are mixtures of 30 s real and 30 s synthetic data, which is strong evidence for this confound. The reported accuracies are therefore not trustworthy as neural-decoding benchmarks, and the issue cannot be repaired without real 60 s rest recordings.","major_comments":[{"comment":"The rest class is not real data. Each 30 s rest recording is extended to 60 s by Fourier-based resampling, nonlinear time warping, and injected noise, while each task trial is a genuine 60 s recording. This construction introduces a learnable real-versus-synthetic cue into the two classes, so the high accuracies in Tables 1-4 and Table 5 and the near-perfect AUC in the Abstract cannot be attributed to discrimination of rest and task brain states.","section":"§2.7"},{"comment":"The control experiment is the decisive evidence: when both classes are 30 s real plus 30 s synthetic, accuracy drops to approximately chance (mean accuracy about 0.51, mean AUC about 0.48). The authors interpret this as showing that synthetic data cannot replace real data, but the more direct reading is that the original classifiers relied on the difference between real 60 s task signals and synthetic 60 s rest signals. Even though this control also shortens the real task content, it does not resolve the ambiguity, because real 60 s rest recordings were never collected.","section":"§4.5, Table 6"},{"comment":"The leave-participants-out analysis does not address the confound: the real-versus-synthetic distinction is independent of participant identity, so high accuracy on unseen participants is equally compatible with a model that has learned an artifact. The surrounding text argues that the model learns rest- and task-related patterns, but no control with real 60 s rest data is offered, so Table 5 does not validate neural decoding.","section":"Robustness Check (Table 5)"},{"comment":"The feature-importance analyses do not resolve the problem. Slope, RMS, maximum, and mean are exactly the summary statistics that Fourier resampling, time warping, and noise injection can modify, so the coefficient-, SHAP-, and tree-based rankings of these features are consistent with a classifier that separates real from synthetic signals rather than rest from task states.","section":"§4.4"},{"comment":"The Discussion states that synthetic signals 'may still lack the nuanced temporal dynamics present in genuine neural activity' and that 'careful validation is necessary.' This acknowledged limitation directly contradicts the Abstract's claim that the augmentation strategy 'preserved physiological integrity' and further undermines the conclusion that the benchmark measures neural-state decoding.","section":"Challenges and Future Perspectives"}],"minor_comments":[{"comment":"The LSTM section describes a softmax output with categorical cross-entropy although the task is binary and the rest of the paper uses sigmoid activation with binary cross-entropy; please reconcile the description.","section":"§3.2"},{"comment":"There are formatting inconsistencies such as 'as shown in figure.1'; the caption referencing should be cleaned up.","section":"Figure 1"},{"comment":"Please clarify whether the '45 samples' after quality exclusion are participants, sessions, or individual trials, since the cross-validation unit is otherwise ambiguous.","section":"§2.2/§2.5"},{"comment":"The 10-fold cross-validation description does not state whether the StandardScaler and other preprocessing steps were fit on the training folds only; this should be specified to avoid hidden data leakage.","section":"§3.1"},{"comment":"The precision in Table 5 is inconsistent (e.g., 0.925 versus 0.920792079); please standardize the number of decimals and include standard deviations as in Table 1.","section":"Table 5"}],"recommendation":"reject","confidential_remarks":"The core problem is structural: no real 60 s rest recordings exist in the study, and the authors' own control experiment (Table 6) points directly to a real-versus-synthetic confound. The only adequate fix would be new data collection or a complete reframing of the claims, which is outside the scope of a revision. The decision letter should also note that the interpretation of Table 6 in the manuscript inverts its logical force."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the headline accuracies don't survive the paper's own control. Rest trials are 30 s real recordings stretched to 60 s by Fourier resampling, time warping, and injected noise; task trials are real 60 s recordings. When the authors equalize the mix so both classes are 30 s real plus 30 s synthetic (Table 6), accuracy collapses to chance. The 93–98% numbers in Tables 1–5 therefore cannot be read as neural decoding of rest versus task. The parsimonious story is that the classifiers latched onto learnable differences between real and synthetic signal statistics.\n\nCredit where it is due. The authors ran that control, reported the chance-level result in full, and acknowledged in the Discussion that synthetic signals may lack the nuanced temporal dynamics of genuine neural activity. That is more transparent than most papers. The dataset is new and decently sized (45 participants, six prefrontal channels, solo and dyadic tennis play), and the per-fold tables are detailed enough to be genuinely useful. The leave-participants-out check (Table 5) is a robustness step many benchmark papers skip, even though it does not fix the confound: augmentation is applied identically across participants, so the real-versus-synthetic cue survives. The citation pattern looks normal — standard fNIRS-BCI literature plus their own earlier paper where relevant.\n\nThe remaining soft spots are minor by comparison. The Conclusion crowns ResNet as the top performer, but Table 1 shows Extra Trees and Gradient Boosting with higher accuracy and higher AUC; the Abstract is more careful, so this reads as a text-level slip. The Abstract's claim that the augmentation strategy preserves physiological integrity also sits uneasily next to Table 6. And there is no statistical comparison across models, so the deep-versus-classical ordering is descriptive, not inferential.\n\nOne fair caveat on the control: it also shortens the real task content from 60 s to 30 s, so it does not cleanly isolate the real-versus-synthetic cue by itself. But combined with the fact that rest was entirely synthetic, the most straightforward reading is that the original numbers were inflated by class-specific signal statistics; at minimum, the control is inconsistent with 'robust decoding of rest/task physiology' as claimed.\n\nWho this is for: fNIRS-BCI researchers and anyone working on augmentation in physiological signals. The paper works as a cautionary methods example, not as evidence for high decoding accuracy. I would send it to peer review — the flaw is identifiable and fixable (record real 60 s rest data, or reframe as a negative-result paper about augmentation artifacts), and the transparency of the reporting suggests a revision could succeed. As submitted, however, the central claim is unsupported.","headline":"The benchmark's headline accuracies are confounded by a fully synthetic rest class—the authors' own control collapses to chance—but the dataset, transparent reporting, and instructive negative control make it a viable cautionary-methods paper.","tokens_in":19282,"tokens_out":10133,"would_cite":false,"duration_ms":88820,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In an interactive tennis game, fNIRS signals can be classified as task or rest with above-97% accuracy, and a ResNet CNN reaches 95% accuracy with a 99.2% AUC, the paper claims.","keywords":["fNIRS","brain-computer interface","rest-task classification","Gramian Angular Field","convolutional neural network","data augmentation","tennis simulation","ensemble learning"],"falsifier":"Collect a dataset with real 60-second rest recordings and real 60-second task recordings under the same protocol, train the same models, and check whether accuracy stays near 95–97%; the paper's own control with mixed real-plus-synthetic inputs in both classes already drops accuracy to roughly chance.","tokens_in":18195,"feed_emoji":"🎾","tokens_out":8537,"duration_ms":73581,"temperature":0.7,"pith_summary":"This paper tries to establish that functional near-infrared spectroscopy (fNIRS) can decode whether someone is at rest or actively playing an interactive tennis video game, using only prefrontal scalp recordings. It benchmarks classical feature-based models, an attention-augmented LSTM, and convolutional networks fed with Gramian Angular Field images, reporting that Extra Trees and Gradient Boosting exceed 97% accuracy and that a ResNet CNN reaches 95% accuracy with a 99.2% AUC. Because rest trials lasted 30 seconds and task trials 60 seconds, the authors introduce a hybrid augmentation pipeline—Fourier resampling plus nonlinear time warping with injected noise—to stretch rest segments to match task length. The paper also reports feature-importance analyses pointing to slope and RMS features of both oxygenated and deoxygenated hemoglobin as the main drivers. If the claims hold, fNIRS-based BCIs would be viable in dynamic, real-world environments such as adaptive gaming and rehabilitation.","feed_headline":"fNIRS rest-vs-play decoding tops 97% in a tennis game","feed_subtitle":"Ensemble models and a ResNet CNN hit near-ceiling scores, but the rest data were synthetically stretched to match task length.","key_machinery":"The load-bearing mechanism is a hybrid upsampling pipeline that equalizes trial durations: each 30-second rest segment is extended to 60 seconds by randomly assigning, per channel, either Fourier-based resampling or nonlinear time warping, with low-amplitude band-limited noise injected, so that deep networks receive equal-length inputs without truncating the 60-second task trials. The other central component is the Gramian Angular Field transform, which converts each channel's time series into a 224-pixel image and arranges the channel images in a 4×3 RGB grid; this composite image is what the ResNet50 and EfficientNet classifiers see. Together the pipeline and image encoding allow the CNNs to operate on spatially arranged temporal-correlation patterns, while the classical models instead consume engineered time-domain features such as slope, RMS, and higher moments.","core_discovery":"On the paper's own account, rest and task states during realistic tennis gameplay are separable from six-channel prefrontal fNIRS recordings with near-ceiling performance, and deep models are the strongest decoders. The ResNet50 CNN, trained on composite Gramian Angular Field images of the multichannel signals, reaches 95.0% mean accuracy and 99.2% mean AUC; Extra Trees and Gradient Boosting reach 97.3–97.6% accuracy; the LSTM-attention model reaches 93.1%. The paper attributes the high separability to genuine hemodynamic differences, noting that both HbO and HbR slopes and RMS features matter, and that the augmentation preserves the canonical HbO–HbR anti-correlation. It also reports a control experiment in which rest and task inputs each combine 30 seconds of real signal with 30 seconds of synthetic signal, and accuracy falls to roughly chance, a result it interprets as showing that synthetic data cannot replace real neural signal content.","pith_inferences":["The paper's own mixed-real/synthetic control hints that the headline accuracies may reflect a real-versus-synthetic distinction rather than a rest-versus-task distinction; a definitive test would require real 60-second rest recordings collected under the same protocol.","If the augmentation confound is confirmed, the GAF/CNN results would still be informative about signal-level discriminability, but the ecological-validity conclusion would need to be scaled back.","A natural extension would be to train the same classifiers on fully real equal-length segments—for example, by collecting 60-second rest blocks or by comparing the first 30 seconds of task with a real 30-second rest block—and check whether accuracy survives.","The feature-importance findings (HbO slope dominating) could be tested in a closed-loop BCI where the classifier's confidence is used to drive game difficulty, providing a direct behavioral validation."],"forward_implications":["If the claimed accuracies hold under fully real, equal-length recordings, fNIRS-based BCIs could operate reliably in interactive gaming and other naturalistic environments without restricting users to short, scripted trials.","ResNet on Gramian Angular Field images would be a preferred architecture for decoding fNIRS when spatial layout matters, while Extra Trees and Gradient Boosting remain competitive when only engineered features are available.","Including both oxygenated and deoxygenated hemoglobin, and especially slope and RMS features, becomes a recommended practice for fNIRS state classification.","The data-augmentation pipeline would allow researchers to keep full-length task trials while still training duration-sensitive deep models, increasing usable data in fNIRS studies.","Leave-subject-out validation at comparable accuracy would support cross-participant generalization of the classifiers."],"supporting_citations":[{"why":"Supplies the tennis-simulation task paradigm and the earlier comparison of human versus AI opponents that this study extends to rest/task classification.","marker":"[Ghalavand et al., 2024]"},{"why":"Provides the Gramian Angular Field method used to convert fNIRS time series into the images fed to the CNN models.","marker":"[Wang and Oates, 2015]"},{"why":"Gives the Fourier-based resampling technique used in the hybrid augmentation pipeline for stretching rest segments.","marker":"[Rasche et al., 1999]"},{"why":"Supplies the nonlinear time-warping method used as the second augmentation arm for the rest signals.","marker":"[Zhang et al., 2023]"},{"why":"Underpins the decision to include both oxygenated and deoxygenated hemoglobin in the analysis.","marker":"[Yücel et al., 2021]"},{"why":"Defines the Extra Trees ensemble method that achieves one of the highest accuracies in the benchmark.","marker":"[Geurts et al., 2006]"},{"why":"Defines gradient boosting, the other top-performing classical model.","marker":"[Natekin and Knoll, 2013]"},{"why":"Documents the ResNet50 architecture used for the top-performing CNN.","marker":"[Koonce, 2021]"}],"fun_headline_variants":["fNIRS brain game: AI scores 97% on rest vs task","Deep nets win in fNIRS gaming BCI: 95% accuracy, 99% AUC","Rest-vs-task decoding in fNIRS hits 97% with classic models","Synthetic stretch kills fNIRS BCI accuracy in control test","fNIRS BCI: near-perfect AUC, but augmentation is synthetic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rest class is built by stretching 30-second real rest recordings to 60 seconds with Fourier resampling, time warping, and injected noise, so the classifiers may be learning to separate real signals from synthetic ones rather than true rest from task states.","fun_headline_variants_meta":{"raw":{"variants":["fNIRS brain game: AI scores 97% on rest vs task","Deep nets win in fNIRS gaming BCI: 95% accuracy, 99% AUC","Rest-vs-task decoding in fNIRS hits 97% with classic models","Synthetic stretch kills fNIRS BCI accuracy in control test","fNIRS BCI: near-perfect AUC, but augmentation is synthetic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000901,"raw_usage":{"total_tokens":3888,"prompt_tokens":965,"completion_tokens":2923,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":2818}},"tokens_in":581,"tokens_out":2923,"duration_ms":20591,"temperature":1.0,"reasoning_tokens":2818,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:08:23.598934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a dataset with real 60-second rest recordings and real 60-second task recordings under the same protocol, train the same models, and check whether accuracy stays near 95–97%; the paper's own control with mixed real-plus-synthetic inputs in both classes already drops accuracy to roughly chance.","supporting_citations":[],"review_version":1}