REVIEW 4 major objections 5 minor 1 cited by
A Multi-Label EEG Dataset for Mental Attention State Classification in Online Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The paper introduces MEMA, a public multi-label EEG dataset for classifying mental attention states—neutral, relaxing, and concentrating—during online learning, and validates it through 1,060 minutes of recordings from 20 subjects…
desk verdict MEMA is a useful public EEG dataset for online-learning attention, but the single-video-per-state design means the labels are as much about stimulus identity as about attention, and the paper needs a more careful validation story. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the MEMA dataset together with its standardized three-task collection paradigm. Each attention state is induced by a specific video—neutral, relaxing, and concentrating—and each trial ends with self-assessment of attention and emotion, so the labels are self-reports anchored to a controlled stimulus. Around this, the validation machinery includes EEG preprocessing (notch filtering, band-pass filtering, ICA artifact removal), topographic analysis of $\alpha$ and $\beta$ power over frontal sites, six baseline classifiers evaluated in subject-dependent and cross-subject settings, a $\chi^2$ test for the attention-emotion association, and hard parameter sharing multi-task learning to test which emotion dimension best pairs with attention classification.
What would settle it
A direct test would be to rerun the same paradigm with an independent objective measure of attention—for instance, recording eye-gaze patterns, reaction times to intermittent probes, or a second EEG-based attention index—and compare those measures across the neutral, relaxing, and concentrating conditions. If the relaxing and concentrating conditions produce indistinguishable objective attention, or if self-report labels frequently disagree with the objective measure, the dataset's core labeling assumption would fail.
Extended reading notes
Core claim
The central claim is that MEMA is a validated, publicly available multi-label EEG dataset for mental attention state classification in online learning, with three classes—neutral, relaxing, and concentrating—and auxiliary labels that make it more than a single-task collection. The collection paradigm assigns a distinct video task to each state: a blank-screen neutral clip, a soothing scenic clip with music for relaxing, and a machine-learning lecture clip for concentrating, followed by a comprehension question. After each trial, subjects self-report their attention state and rate emotion on the valence-arousal-dominance model. The paper demonstrates that the dataset supports reliable attention classification with classical and deep learning models, that $\alpha$ power rises and $\beta$ power falls as attention decreases, and that attention is statistically associated with valence and arousal; in particular, multi-task learning that pairs attention with valence improves classification accuracy.
Load-bearing premise
The load-bearing premise is that the three video tasks reliably induce the intended attention states and that subjects' self-reported attention labels accurately capture those states; if the videos fail to induce the intended states, or if self-reports are inaccurate, then the classification results and attention-emotion correlations describe the videos or the reports rather than actual brain-state attention.
Editorial extensions
If this is right
- Researchers gain a public, multi-label benchmark for EEG-based attention classification in online learning, addressing the scarcity that has limited reproducibility and comparability.
- The presence of emotion labels, personality traits, and personal information enables studies of how attention interacts with affect and individual differences, not just single-label classification.
- The standardized paradigm—three states, randomized trial order, fixed durations informed by physiological and psychological research—offers a template for future EEG data collections in learning contexts.
- The reported baselines (up to 85.12% subject-dependent and 64.84% cross-subject accuracy) provide concrete reference points for evaluating new attention classifiers.
- Because valence is the emotion dimension statistically tied to attention, the paper's multi-task results imply that attention and valence should be modeled together, not paired arbitrarily with arousal or dominance.
Reading between the lines
- A natural extension would be to use MEMA to train real-time attention monitors for lecture-style video content, since the concentrating task uses a genuine course video and the labels are trial-level.
- Because the attention labels are self-reports, an independent behavioral or physiological check—such as eye tracking or response-time measures during the concentrating task—could strengthen the validity of the ground truth, something the paper itself does not report.
- With the included personality and personal-information fields, the dataset could support individual-difference analyses, for example whether personality traits modulate the strength of the alpha-beta attention signature, but such analyses are not carried out in this paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MEMA, a public multi-label EEG dataset for mental attention state classification in online learning. Twenty subjects completed 12 trials each across three attention states (neutral, relaxing, concentrating), with auxiliary emotional VAD labels, Big Five personality data, and personal information. The authors provide baselines for subject-dependent and cross-subject classification of attention and emotion using classical and deep learning models, reporting attention accuracies up to 85.12% and 64.84%, respectively, and a multi-label correlation analysis between attention and emotion. The central claim is that MEMA is a validated, high-quality resource that fills a gap in public EEG attention datasets.
Significance. If the attention labels indeed reflect the intended mental states, MEMA would be a valuable public resource, being one of the few public EEG datasets for attention in online learning and the only one with multi-label emotion annotations, personality traits, and personal information. The paper ships a publicly available dataset and a reproducible baseline suite, including both classical and deep learning models, which is a practical strength. However, the validation claim rests on the assumptions that the task stimuli evoke separable attention states and that self-reports are reliable; the current evidence does not conclusively establish these, so the dataset's scientific value is present but not yet fully demonstrated.
major comments (4)
- [Section II.C] The three attention states are each instantiated by exactly one video type: a one-minute blank screen for neutral, a five-minute nature-scenery video with music for relaxing, and a five-minute machine-learning lecture for concentrating. Because the stimuli differ in low-level visual content, audio content, and duration, the high classification accuracies in Table II could reflect classifiers recognizing the specific stimulus rather than a generalizable internal attention state. The paper's central claim that MEMA is a validated attention-state dataset requires evidence of cross-content generalization, such as multiple videos per state or an analysis that removes stimulus-locked features; as it stands, this confounding is unresolved.
- [Section II.C] The attention label is determined by the task condition and only corroborated by a 15-second self-assessment after each video; no objective behavioral measure (quiz accuracy, reaction time, eye tracking) is reported, and the quiz answers from the concentrating task are not analyzed. Because participants were told which state each video was intended to induce, their self-reports may reflect demand characteristics rather than independent verification. This is load-bearing for the dataset's label validity, and the authors should either provide an objective validation of the self-reports or explicitly document and discuss the limitation.
- [Section III.C.1 and Table II] Table II reports only mean accuracy and F1 without standard deviations, confidence intervals, or significance tests. The subject-dependent setup uses one fixed split (first 9 trials for training, last 3 for testing), which is highly sensitive to trial order and within-session fatigue or learning effects. For a dataset paper, these numbers are the principal quantitative evidence of data quality; the absence of variability measures and statistical comparisons makes the evidence incomplete. Standard deviations across subjects/folds, per-class results, and chance-level comparisons should be reported.
- [Section III.B] The qualitative validation in Section III.B is performed on one subject only (Figure 2). The observation that alpha power increases and beta power decreases with lower attention, while consistent with the literature, is not established across the 20-subject cohort and therefore does not, by itself, support the dataset-level quality claim. The authors should either extend the qualitative analysis to multiple subjects or clearly present it as an illustrative example rather than part of the validation.
minor comments (5)
- [Abstract vs. full text] The dataset URL in the abstract (https://github.com/GuanjianLiu/MEMA) differs from the URL in the full text (https://github.com/XJTU-EEG/MEMA); the authors should ensure a single working link is used consistently.
- [Section II.C] The heading 'Task Design for Each Trail' should read 'Each Trial'.
- [Section III.C.1] The sentence 'the first 9 of one subject's trials used for training' is missing a verb; it should be 'were used for training'.
- [Table III] Table III reports chi-square values without degrees of freedom or p-values; these should be added to support the stated association claims.
- [References] Reference [21] is cited for the duration of the concentrating task through the Continuous Performance Test, but [21] concerns visual sustained attention degradation, not the CPT; please verify the citation.
Circularity Check
No circular derivation: the dataset report rests on supervised benchmarking and descriptive analysis, not on a fitted input renamed as a prediction.
full rationale
This is a data-reporting paper rather than a derivation paper. The central claim is that MEMA is a validated multi-label EEG dataset for attention state classification in online learning. Attention labels are assigned from the experimental task design (neutral, relaxing, concentrating) and then corroborated by 15-second self-assessments; EEG classification accuracies in Table II are standard supervised benchmarks on those labels using held-out trials or leave-one-subject-out cross-validation. There is no equation or fitted parameter that is subsequently renamed as a prediction, and no derived quantity is equivalent by construction to an input. The qualitative alpha/beta power observations are descriptive and are compared with prior external studies rather than presented as a theorem. The paper contains several self-citations (e.g., [11], [12], [13], [22], [23], [24]), but these support emotion-recognition methodology and auxiliary analysis; they are not load-bearing for the core dataset validity claim. A genuine scientific limitation is that the paradigm confounds attention state with the particular video stimuli and relies on unaudited self-reports, so the high classification accuracies may reflect stimulus identity or self-report consistency rather than a generalizable internal attention state. That is a validity threat, not a circularity, because the paper does not claim to derive the labels from the EEG signals or to derive the EEG signals from the labels. Accordingly, no circular step can be quoted with a specific reduction, and the appropriate score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption The three video tasks reliably induce neutral, relaxing, and concentrating states.
- domain assumption Subjects' self-assessed attention and VAD emotion labels are accurate ground truth.
- domain assumption A band-pass of 8 to 30 Hz plus 50 Hz notch and ICA removes artifacts while preserving attention-relevant EEG.
- domain assumption Alpha and beta power changes reflect attention state in the expected direction.
Cite this review
Pith. "Pith review of A Multi-Label EEG Dataset for Mental Attention State Classification in Online Learning." pith.science (2026). https://pith.science/paper/4SNKWG2X
@misc{pith2026241109879,
author = {Pith},
title = {Pith review of: A Multi-Label EEG Dataset for Mental Attention State Classification in Online Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/4SNKWG2X}},
note = {Machine review of arXiv:2411.09879}
}
read the original abstract
Attention is a vital cognitive process in the learning and memory environment, particularly in the context of online learning. Traditional methods for classifying attention states of online learners based on behavioral signals are prone to distortion, leading to increased interest in using electroencephalography (EEG) signals for authentic and accurate assessment. However, the field of attention state classification based on EEG signals in online learning faces challenges, including the scarcity of publicly available datasets, the lack of standardized data collection paradigms, and the requirement to consider the interplay between attention and other psychological states. In light of this, we present the Multi-label EEG dataset for classifying Mental Attention states (MEMA) in online learning. We meticulously designed a reliable and standard experimental paradigm with three attention states: neutral, relaxing, and concentrating, considering human physiological and psychological characteristics. This paradigm collected EEG signals from 20 subjects, each participating in 12 trials, resulting in 1,060 minutes of data. Emotional state labels, basic personal information, and personality traits were also collected to investigate the relationship between attention and other psychological states. Extensive quantitative and qualitative analysis, including a multi-label correlation study, validated the quality of the EEG attention data. The MEMA dataset and analysis provide valuable insights for advancing research on attention in online learning. The dataset is publicly available at \url{https://github.com/GuanjianLiu/MEMA}.
Figures
Forward citations
Cited by 1 Pith paper
-
Comprehensive Review of EEG-to-Output Research: Decoding Neural Signals into Images, Videos, and Audio
A PRISMA-style review of EEG-to-output decoding claims to analyze 1,800 studies but omits the flow diagram, study list, and quantitative synthesis needed to back that claim.
Reference graph
Works this paper leans on
-
[1]
Recent theoretical, neural, and clinical advances in sustained attention research,
F. C. Fortenbaugh, J. DeGutis, and M. Esterman, “Recent theoretical, neural, and clinical advances in sustained attention research,” Annals of the New York Academy of Sciences , vol. 1396, no. 1, pp. 70–91, 2017
work page 2017
-
[2]
The attention system of the human brain: 20 years after,
S. E. Petersen and M. I. Posner, “The attention system of the human brain: 20 years after,” Annual review of neuroscience, vol. 35, pp. 73–89, 2012
work page 2012
-
[3]
Attentive face detection and recognition,
V . Kr ¨uger, U. Mahlmeister, and G. Sommer, “Attentive face detection and recognition,” in Mustererkennung 1998: 20. DAGM-Symposium Stuttgart, 29. September–1. Oktober 1998. Springer, 1998, pp. 279–286
work page 1998
-
[4]
The impact of focused attention on emotional evaluation: An eye-tracking investigation
F. Dolcos, P. C. Bogdan, M. O’Brien, A. D. Iordan, A. Madison, S. Buetti, A. Lleras, and S. Dolcos, “The impact of focused attention on emotional evaluation: An eye-tracking investigation.” Emotion, vol. 22, no. 5, p. 1088, 2022
work page 2022
-
[5]
An efficient lstm network for emotion recognition from multichannel eeg signals,
X. Du, C. Ma, G. Zhang, J. Li, Y .-K. Lai, G. Zhao, X. Deng, Y .-J. Liu, and H. Wang, “An efficient lstm network for emotion recognition from multichannel eeg signals,” IEEE Transactions on Affective Computing , vol. 13, no. 3, pp. 1528–1540, 2020
2020
-
[6]
Attention recognition system in online learning platform using eeg signals,
S. Gupta and P. Kumar, “Attention recognition system in online learning platform using eeg signals,” in Emerging Technologies for Smart Cities: Select Proceedings of EGTET 2020 . Springer, 2021, pp. 139–152
work page 2020
-
[7]
Detection of sustained auditory attention in students with visual impairment,
H. Ghasemy, M. Momtazpour, and S. H. Sardouie, “Detection of sustained auditory attention in students with visual impairment,” in 2019 27th Iranian Conference on Electrical Engineering (ICEE) . IEEE, 2019, pp. 1798–1801
work page 2019
-
[8]
Eeg-based closed-loop neurofeed- back for attention monitoring and training in young adults,
B. Wang, Z. Xu, T. Luo, and J. Pan, “Eeg-based closed-loop neurofeed- back for attention monitoring and training in young adults,” Journal of Healthcare Engineering, vol. 2021, no. 1, p. 5535810, 2021
work page 2021
Show all 39 references
-
[9]
Detection of human attention using eeg signals,
M. Alirezaei and S. H. Sardouie, “Detection of human attention using eeg signals,” in 2017 24th National and 2nd International Iranian Conference on biomedical engineering (ICBME). IEEE, 2017, pp. 1–5
2017
-
[10]
Attention recognition in eeg- based affective learning research using cfs+ knn algorithm,
B. Hu, X. Li, S. Sun, and M. Ratcliffe, “Attention recognition in eeg- based affective learning research using cfs+ knn algorithm,” IEEE/ACM transactions on computational biology and bioinformatics, vol. 15, no. 1, pp. 38–45, 2016
2016
-
[11]
Libeer: A comprehensive benchmark and algorithm library for eeg-based emotion recognition,
H. Liu, S. Yang, Y . Zhang, M. Wang, F. Gong, C. Xie, G. Liu, Z. Liu, Y .-J. Liu, B.-L. Lu et al. , “Libeer: A comprehensive benchmark and algorithm library for eeg-based emotion recognition,” arXiv preprint arXiv:2410.09767, 2024
-
[12]
Cross-modal credibility modelling for eeg-based multimodal emotion recognition,
Y . Zhang, H. Liu, D. Wang, D. Zhang, T. Lou, Q. Zheng, and C. Quek, “Cross-modal credibility modelling for eeg-based multimodal emotion recognition,” Journal of Neural Engineering , 2024
2024
-
[13]
Autoeer: au- tomatic eeg-based emotion recognition with neural architecture search,
Y . Wu, H. Liu, D. Zhang, Y . Zhang, T. Lou, and Q. Zheng, “Autoeer: au- tomatic eeg-based emotion recognition with neural architecture search,” Journal of Neural Engineering , vol. 20, no. 4, p. 046029, 2023
2023
-
[14]
Exploring eeg features in cross-subject emotion recognition,
X. Li, D. Song, P. Zhang, Y . Zhang, Y . Hou, and B. Hu, “Exploring eeg features in cross-subject emotion recognition,”Frontiers in neuroscience, vol. 12, p. 162, 2018
2018
-
[15]
Handedness, language areas and neuropsychiatric diseases: insights from brain imaging and genetics,
A. Wiberg, M. Ng, Y . Al Omran, F. Alfaro-Almagro, P. McCarthy, J. Marchini, D. L. Bennett, S. Smith, G. Douaud, and D. Furniss, “Handedness, language areas and neuropsychiatric diseases: insights from brain imaging and genetics,” Brain, vol. 142, no. 10, pp. 2938– 2947, 2019
2019
-
[16]
Electroencephalogram-based attention level classification using convolution attention memory neural network,
C. K. Toa, K. S. Sim, and S. C. Tan, “Electroencephalogram-based attention level classification using convolution attention memory neural network,” IEEE Access, vol. 9, pp. 58 870–58 881, 2021
2021
-
[17]
Eeg-based attention feedback to improve focus in e-learning,
C. Sethi, H. Dabas, C. Dua, M. Dalawat, and D. Sethia, “Eeg-based attention feedback to improve focus in e-learning,” in Proceedings of the 2018 2nd international conference on computer science and artificial intelligence, 2018, pp. 321–326
2018
-
[18]
The eeg-based attention analysis in multimedia m-learning,
D. Ni, S. Wang, and G. Liu, “The eeg-based attention analysis in multimedia m-learning,” Computational and Mathematical Methods in Medicine, vol. 2020, no. 1, p. 4837291, 2020
2020
-
[19]
Emotion analysis for personality inference from eeg signals,
G. Zhao, Y . Ge, B. Shen, X. Wei, and H. Wang, “Emotion analysis for personality inference from eeg signals,” IEEE transactions on affective computing, vol. 9, no. 3, pp. 362–371, 2017
2017
-
[20]
A dynamic model of stress and sustained attention,
P. A. Hancock, “A dynamic model of stress and sustained attention,” Human factors, vol. 31, no. 5, pp. 519–537, 1989
1989
-
[21]
Visual sustained attention: Image degradation produces rapid sensitivity decrement over time,
K. H. Nuechterlein, R. Parasuraman, and Q. Jiang, “Visual sustained attention: Image degradation produces rapid sensitivity decrement over time,” Science, vol. 220, no. 4594, pp. 327–329, 1983
1983
-
[22]
Eeg- based emotion recognition with emotion localization via hierarchical self-attention,
Y . Zhang, H. Liu, D. Zhang, X. Chen, T. Qin, and Q. Zheng, “Eeg- based emotion recognition with emotion localization via hierarchical self-attention,” IEEE Transactions on Affective Computing , 2022
2022
-
[23]
Eeg-based multimodal emotion recognition: A machine learning per- spective,
H. Liu, T. Lou, Y . Zhang, Y . Wu, Y . Xiao, C. S. Jensen, and D. Zhang, “Eeg-based multimodal emotion recognition: A machine learning per- spective,” IEEE Transactions on Instrumentation and Measurement , 2024
2024
-
[24]
Fixing our focus: Training attention to regulate emotion,
H. A. Wadlinger and D. M. Isaacowitz, “Fixing our focus: Training attention to regulate emotion,” Personality and social psychology review, vol. 15, no. 1, pp. 75–102, 2011
2011
-
[25]
Three dimensions of emotion
H. Schlosberg, “Three dimensions of emotion.” Psychological review, vol. 61, no. 2, p. 81, 1954
1954
-
[26]
Detecting frontal eeg activities with forehead electrodes,
J.-R. Duann, P.-C. Chen, L.-W. Ko, R.-S. Huang, T.-P. Jung, and C.- T. Lin, “Detecting frontal eeg activities with forehead electrodes,” in Foundations of Augmented Cognition. Neuroergonomics and Opera- tional Neuroscience: 5th International Conference, FAC 2009 Held as Part o...
2009
-
[27]
Resting state eeg power research in attention-deficit/hyperactivity disorder: A review update,
A. R. Clarke, R. J. Barry, and S. Johnstone, “Resting state eeg power research in attention-deficit/hyperactivity disorder: A review update,” Clinical Neurophysiology, vol. 131, no. 7, pp. 1463–1479, 2020
2020
-
[28]
Eeg alpha power is modulated by attentional changes during cognitive tasks and virtual reality immersion,
E. Magosso, F. De Crescenzio, G. Ricci, S. Piastra, and M. Ursino, “Eeg alpha power is modulated by attentional changes during cognitive tasks and virtual reality immersion,” Computational intelligence and neuroscience, vol. 2019, no. 1, p. 7051079, 2019
2019
-
[29]
Frontal and parietal alpha oscillations reflect attentional modulation of cross-modal matching,
J. Misselhorn, U. Friese, and A. K. Engel, “Frontal and parietal alpha oscillations reflect attentional modulation of cross-modal matching,” Scientific reports, vol. 9, no. 1, p. 5030, 2019
2019
-
[30]
Least squares support vector machine classifiers,
J. A. Suykens and J. Vandewalle, “Least squares support vector machine classifiers,” Neural Processing Letters, vol. 9, no. 3, pp. 293–300, 1999
1999
-
[31]
Induction of decision trees,
J. R. Quinlan, “Induction of decision trees,” Machine learning, vol. 1, pp. 81–106, 1986
1986
-
[32]
Random forests,
L. Breiman, “Random forests,” Machine Learning , vol. 45, no. 1, pp. 5–32, 2001
2001
-
[33]
Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces,
V . J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, “Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces,” Journal of neural engineering , vol. 15, no. 5, p. 056013, 2018
2018
-
[34]
Eeg- based emotion recognition via channel-wise attention and self attention,
W. Tao, C. Li, R. Song, J. Cheng, Y . Liu, F. Wan, and X. Chen, “Eeg- based emotion recognition via channel-wise attention and self attention,” IEEE Transactions on Affective Computing, vol. 14, no. 1, pp. 382–393, 2020
2020
-
[35]
Dgcnn: A convolutional neural network over large-scale labeled graphs,
A. V . Phan, M. Le Nguyen, Y . L. H. Nguyen, and L. T. Bui, “Dgcnn: A convolutional neural network over large-scale labeled graphs,” Neural Networks, vol. 108, pp. 533–543, 2018
2018
-
[36]
The chi-square test of independence,
M. L. McHugh, “The chi-square test of independence,” Biochemia medica, vol. 23, no. 2, pp. 143–149, 2013
2013
-
[37]
A survey on multi-task learning,
Y . Zhang and Q. Yang, “A survey on multi-task learning,” IEEE Transactions on Knowledge and Data Engineering , vol. 34, no. 12, pp. 5586–5609, 2021
2021
-
[38]
Multitask learning,
R. Caruana, “Multitask learning,” Machine learning, vol. 28, pp. 41–75, 1997
1997
-
[39]
Taskonomy: Disentangling task transfer learning,
A. R. Zamir, A. Sax, W. Shen, L. J. Guibas, J. Malik, and S. Savarese, “Taskonomy: Disentangling task transfer learning,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 3712–3722
2018
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.