Performance Evaluation of Supervised Machine Learning Techniques for Efficient Detection of Emotions from Online Content

Amir Mosavi; Annamaria R. Varkonyi-Koczy; Fazal Masud Kundi; Fazli Subhan; Muhammad Imran; Muhammad Zubair Asghar; Peter Csiba; Shahboddin Shamshirband

REVIEW 5 major objections 5 minor 7 references

Performance Evaluation of Supervised Machine Learning Techniques for Efficient Detection of Emotions from Online Content

T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On the ISEAR emotion dataset, a back-propagation neural classifier reaches the highest accuracy at 71.27%, with logistic regression at 66.58% accuracy and the best joy recall (83%); the paper recommends these two.

desk verdict Routine ISEAR classifier benchmark whose 'best classifier' recommendation is undermined by inconsistent dataset-size and accuracy reporting; useful mainly as a cautionary example. read the letter →

arxiv 1908.01587 v1 pith:GDX5YFEJ submitted 2019-08-05 cs.IR cs.CLcs.LG

Muhammad Zubair Asghar , Fazli Subhan , Muhammad Imran , Fazal Masud Kundi , Shahboddin Shamshirband , Amir Mosavi , Peter Csiba , Annamaria R. Varkonyi-Koczy This is my paper · ORCID

classification cs.IRcs.CLcs.LG

keywords emotiondetectionsupervisedmachinelearningISEARdatasettextclassificationlogisticregressionback-propagationneuralclassifiersentimentanalysisperformanceevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish which supervised machine-learning classifier should be used to detect emotions in written text, and it argues that, on the ISEAR benchmark corpus, the answer is a back-propagation neural classifier (BPN) and logistic regression. It reports BPN as the accuracy leader at 71.27% and logistic regression as the balanced leader at 66.58% accuracy, with the top average precision and recall (0.67) and an 83% recall for joy. This matters because emotion detection from online content is a practical task for which earlier studies often tested only a single classifier or a smaller classifier set. The paper's contribution is a direct eight-classifier comparison under one feature-engineering and evaluation protocol, ending in a concrete recommendation.

What carries the argument

The central machinery is the ISEAR corpus—a benchmark collection of self-reported emotional episodes labeled by emotion—combined with a uniform pipeline: tokenization, stop-word and punctuation removal, word-count and TF-IDF feature encoding, an 80/20 random split, and evaluation with precision, recall, F1, and accuracy. This single protocol is what lets the paper compare eight classifiers on the same footing; the ranking in Table 12 is the argument's load-bearing output.

What would settle it

Counting the rows in the exact dataset file the experiments used and repeating the 80/20 split would settle the claim: if the file contains 2,273 reviews rather than 5,477, or if the test partition is not 20 percent of the stated total, then the per-class counts in Table 1 and the reported accuracies cannot be reproduced, and the BPN and logistic-regression recommendation would need to be recomputed.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a supervised emotion-classification pipeline—tokenization, stop-word removal, word-count vectors and TF-IDF weighting, an 80/20 random train/test split, and evaluation by precision, recall, F1, and accuracy—yields a clear ranking of eight classifiers on five emotion categories from the ISEAR corpus. Back-propagation neural classifier produces the best overall accuracy (71.27%) and the best precision for guilt (0.72); logistic regression produces the best balanced averages (0.67 precision, 0.67 recall, 66.58% accuracy) and the highest single-class recall (0.83 for joy). SVM is strongest for joy by precision and F1 (0.76 and 0.77), while KNN is weakest overall (57.81%). The paper concludes by recommending BPN and logistic regression for emotion detection from text.

Load-bearing premise

The results stand on the assumption that the dataset used in the experiments is the standard 5,477-instance ISEAR subset split 80/20 into training and test sets; the paper elsewhere describes the dataset as 2,273 reviews and mentions 750 test instances, so the reported class counts and accuracies are only as solid as that 5,477/80-20 premise.

Editorial extensions

If this is right

A practitioner building an emotion detector for text that resembles ISEAR self-reports can start with logistic regression or BPN rather than KNN or XGBoost and expect better accuracy and F1 on the five tested emotion categories.
The reported figures give concrete baselines—BPN 71.27%, logistic regression 66.58%, SVM 64.66%—that later systems using deep learning or word embeddings would need to beat on ISEAR.
If the goal is to avoid missing joy specifically, logistic regression's 83% recall makes it the better choice among the tested classifiers.
Because the tests use a random split rather than cross-validation, the ranking is a point estimate for that split rather than a certified guarantee for other samples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper does not run is cross-corpus transfer: train the same eight classifiers on ISEAR and evaluate on tweets or product reviews, where emotion vocabulary and style differ; the ranking could shift even if the within-corpus ranking holds.
The 80/20 random split with no repeated trials means the gaps between neighboring classifiers (for example, logistic regression at 66.58% versus SGD at 65.57%) may be within random fluctuation, so repeated stratified runs would turn the ranking into a firmer recommendation.
The metric-dependent winners suggest a decision rule rather than a single champion: choose BPN when overall accuracy matters most, logistic regression when recall for joy matters most, and SVM when per-class precision and F1 for joy are the target.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick summary: this is a routine eight-classifier benchmark on the public ISEAR emotion dataset, and the main recommendation (BPN and logistic regression are best) is not supported by the paper's own tables. The dataset size is given as 2273 in Section 3.1, 5477 in Table 1, and 750 test instances in Section 4.2, so no result can be independently checked.

What it does well: it uses a standard public dataset, reports per-class precision/recall/F1 for each classifier, and includes pseudocode for preprocessing and feature engineering. The comparison with Thomas et al. (2014) and Danisman & Alpkocak (2008) is sensible. The limitations section is candid about random splitting and using a single dataset. That is the extent of the credit; this is an incremental parameter scan with no new method, feature, or theory.

The soft spots are not cosmetic. The 2273/5477/750 contradiction is load-bearing because the accuracies in Tables 4-12 are the whole point. A 20% split of 5477 gives roughly 1095 test examples, not 750; a 20% split of 2273 gives roughly 455. Additionally, Table 12 lists Naive Bayes accuracy as 66.58%, identical to logistic regression, while Table 7 says 63.6%. The recommendation section names both BPN (71.27%) and LR (66.58%) as best, which is odd given BPN is five points higher; then Section 4.4 only compares LR with prior work. The paper also says five classifiers in the introduction and runs eight, and fluctuates between five and seven emotion categories. There are no significance tests, no cross-validation, no hyperparameters (other than RF(200) and KNN(25)), and no code or data release. These are not minor polish items; they make RQ3's answer unreproducible.

Overall: the paper is for someone wanting a quick list of baseline numbers on ISEAR, but the numbers are not reliable as reported. A serious referee would need the corrected dataset size, consistent tables, code, and ideally repeated runs with variance before the comparison can be trusted. I would not cite it, and I would not send it to review in its current form; a desk reject with an invitation to resubmit after fixing the reporting and adding reproducibility material is the proportionate response.

Referee Report

5 major / 5 minor

Summary. The paper compares eight supervised machine learning classifiers (SVM, logistic regression, KNN, Naïve Bayes, random forest, XGBoost, SGD, and BPN) for emotion classification on the ISEAR dataset. The authors evaluate classifiers by accuracy, precision, recall, and F1-score, report per-emotion performance tables, and recommend BPN and logistic regression as the best classifiers for detecting the five emotion categories. The paper also includes pseudocode for preprocessing, feature engineering, classifier training, and evaluation.

Significance. If the comparative results were trustworthy, the paper would provide a useful practical benchmark for emotion classification on a standard public dataset, with the notable strength of reporting multiple evaluation metrics and per-class breakdowns. The use of a public dataset and the inclusion of algorithmic pseudocode are also positive features. However, the significance is severely limited by internal inconsistencies in the reported dataset size and in the results tables, which make the headline recommendation unsupportable and the empirical comparison unreproducible.

major comments (5)

[Section 3.1 and Table 1 vs Section 4.2] There is a direct contradiction in the dataset size: Section 3.1 states the ISEAR dataset comprises 2273 reviews, while Table 1 reports 5477 reviews with per-class counts summing to 5477 (1094+1095+1096+1096+1096 = 5477). Section 4.2 then states that CPU times are for classifying 750 test instances, which is neither 20% of 5477 (approximately 1095) nor 20% of 2273 (approximately 455). Because the evaluation split is not well-defined, the accuracies in Tables 4-12 cannot be independently checked, and the central claim in Section 4.3 is therefore ungrounded.
[Table 12 vs Table 7] Table 12 lists Naïve Bayes accuracy as 66.58%, identical to the logistic regression accuracy, whereas Table 7 reports the Naïve Bayes accuracy as 63.6%. Since Table 12 is presented as the overall results table that supports the recommendation, this internal inconsistency means the comparison table itself is not reliable and the reported performance ordering is suspect.
[Section 4.3 and Section 4.4] The recommendation that both BPN and logistic regression 'have produced best results' is not derivable from the stated metrics: BPN achieves the highest accuracy (71.27%), but logistic regression achieves 66.58% accuracy, and no explicit selection criterion (e.g., accuracy, recall, F1, or a combination) is given that would select both classifiers. Moreover, Section 4.4 benchmarks only logistic regression against prior work, not BPN, leaving the co-recommendation unjustified.
[Section 3.3 and Section 4.1] The set of classifiers is inconsistently described. The introduction says the study applies 'five machine learning classifiers' to detect 'seven categories of emotions', but Section 3.3 lists six classifiers (including both 'Naïve Bayesian' twice) and Section 4.1 lists eight classifiers including BPN. Algorithm 6's ML-classifiers list omits BPN entirely, even though BPN is part of the final recommendation. This inconsistency further hampers reproducibility.
[Section 5.1] Limitation 2 in Section 5.1 says the experiments used 'a subsample of five thousand records', which contradicts Section 3.1's statement of 2273 reviews and Table 1's total of 5477. This is not a minor wording issue because it directly affects which dataset variant was used and how the reported results should be interpreted.

minor comments (5)

[Section 3.3.5 and 3.3.6] The manuscript contains stray editorial text such as 'Eq. 1 for an example. The number should be aligned to the right margin.' and 'Eq. 1 for an example.' These should be removed and proper equation numbering used.
[Table 3] In Table 3, the review numbered 28 is labeled 'Guilt', but the described content ('I felt ashamed in front of the whole class') suggests the label should be 'Shame'; this cast doubt on the quality of the annotation in the sample, and the authors should verify labels in the full dataset.
[Throughout] There are numerous typographical and referencing errors, including 'Jupiter notebook' (should be Jupyter), 'Wile' in Algorithm 2, 'adoptive' in Section 2, and many references listed in the bibliography (e.g., Bagić and Podobnik 2016, Lin and Utz 2015, Podobnik 2013, Siersdorfer et al. 2010) that are never cited in the body.
[Section 4.2.2 and Table 5] The text says logistic regression achieved 83% recall for 'Joy' and 73% precision for 'sadness', but the per-class averages in Table 12 (precision 0.67, recall 0.67) are not reconciled with the claim that logistic regression is the best-performing classifier on these metrics.
[Table 13] The comparison in Table 13 reports 'Our Work' accuracy of 66.5%, but the earlier results tables give 66.58%; the discrepancy is minor but should be made consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the classifier rankings are empirical measurements on a held-out test split, not derivations that reduce to their inputs.

full rationale

This paper is a benchmark-style performance comparison, not a derivation from first principles, and I found no step in which a claimed prediction or result is equivalent to its inputs by construction. The central claim (Section 4.3) recommends BPN and logistic regression as best classifiers based on accuracy, precision, recall, and F-measure values reported in Tables 4-12. These values are presented as experimental measurements obtained by training supervised classifiers on an 80% split of the ISEAR dataset and evaluating on the remaining 20% test split (Sections 3.3.2 and 4.1). There is no fitted parameter that is later renamed as a prediction, no definition that embeds the target result, and no uniqueness or existence theorem imported from the authors' own prior work to force the choice of classifier. The self-citations to Asghar et al. (2017a, 2017b) occur in the introduction and related work as background references for emotion definitions and prior sentiment-classification systems; they are not load-bearing for the experimental comparison or for the final recommendation. The comparison with similar studies in Section 4.4 uses the paper's own logistic-regression results against published results from Danisman and Alpkocak (2008) and Thomas et al. (2014), which is an external benchmarking step rather than a circular reuse of the paper's own output as evidence for itself. The paper does contain serious internal consistency problems that affect reproducibility: Section 3.1 says the ISEAR dataset comprises 2273 reviews while Table 1 reports 5477 reviews; Section 4.2 mentions classifying 750 test instances, which matches neither a 20% split of 2273 (roughly 455) nor a 20% split of 5477 (roughly 1095); and Table 12 lists Naive Bayes accuracy as 66.58%, identical to logistic regression, while Table 7 reports 63.6% for Naive Bayes. These are correctness and reporting concerns, not circularity: they do not show that any reported result is forced by definition or by the paper's own assumptions. Accordingly, the appropriate circularity score is 0, with no specific circular steps identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper is a straightforward supervised-learning benchmark. Its central claim depends on the correctness of the ISEAR dataset description (which is internally inconsistent), the adequacy of TF-IDF features, the correctness of unspecified library implementations, and the validity of a single random split. There are no invented entities. The hand-chosen hyperparameters (K=25, RF=200, 80/20 split, TF-IDF) are free parameters that affect the reported numbers but are not optimized or justified.

free parameters (4)

K (number of neighbors) for KNN = 25
Set by hand for KNN; no tuning procedure is described. Appears as 'K-Nearest Neighbor (KNN 25)' in Table 12.
Number of trees for Random Forest = 200
Set by hand, stated as 'RF (200)' in Table 8; no search over tree counts is reported.
Train/test split ratio = 80/20
Chosen by hand in Section 3.3.2; the authors note in Section 5.1 that random splitting is used and cross-validation is left to future work.
Feature representation = TF-IDF
Chosen by hand in Section 3.3.1; no comparison with other feature sets is performed, and Section 5.1 flags that traditional feature selection needs replacement.

assumptions (4)

domain assumption The ISEAR dataset used for experiments is the standard dataset with the class distribution given in Table 1 (approximately 1095 instances per class, 5477 total).
The paper relies on ISEAR as the benchmark dataset, but Section 3.1 states the dataset is comprised of 2273 reviews, conflicting with Table 1's 5477 total and with the 750 test instances in Section 4.2. If the dataset is not as described, all reported results are ungrounded.
domain assumption TF-IDF bag-of-words features are sufficient to train the classifiers for emotion classification.
All classifiers use the TF-IDF representation described in Section 3.3.1; no alternative feature sets are tested, and the authors list this as a limitation in Section 5.1.
domain assumption The NLTK and scikit-learn (or equivalent) implementations of the classifiers are correct and used with default parameters where not specified.
No code or configuration files are provided, and hyperparameters for SVM, XGBoost, and other classifiers are not reported, so the results depend on unspecified implementation details.
domain assumption A single random 80/20 split provides a valid basis for comparing classifier performance.
The paper uses one random split (Algorithm 3) and no cross-validation or repeated runs; Section 5.1 acknowledges this as a limitation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Performance Evaluation of Supervised Machine Learning Techniques for Efficient Detection of Emotions from Online Content." pith.science (2026). https://pith.science/paper/GDX5YFEJ

@misc{pith2026190801587,
  author       = {Pith},
  title        = {Pith review of: Performance Evaluation of Supervised Machine Learning Techniques for Efficient Detection of Emotions from Online Content},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDX5YFEJ}},
  note         = {Machine review of arXiv:1908.01587}
}

read the original abstract

Emotion detection from the text is an important and challenging problem in text analytics. The opinion-mining experts are focusing on the development of emotion detection applications as they have received considerable attention of online community including users and business organization for collecting and interpreting public emotions. However, most of the existing works on emotion detection used less efficient machine learning classifiers with limited datasets, resulting in performance degradation. To overcome this issue, this work aims at the evaluation of the performance of different machine learning classifiers on a benchmark emotion dataset. The experimental results show the performance of different machine learning classifiers in terms of different evaluation metrics like precision, recall ad f-measure. Finally, a classifier with the best performance is recommended for the emotion classification.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 6 canonical work pages

[1]

Anitescu, C.; Atroshchenko, E.; Alajlan, N.; & Rabczuk, T. (2019). Artificial neural network methods for the solution of second order boundary value problems. Computers, Materials & Continua, 59(1), 345-359. Asghar, M. Z.; Khan, A.; Bibi, A.; Kundi, F. M.; Ahmad, H. (2017): Sentence-Level Emotion Detection Framework Using Rule-Based Classification. Cognit...

work page 2019
[2]

Uğuz, H. (2011). A two-stage feature selection method for text categorization by using information gain, principal component analysis and genetic algorithm. Knowledge-Based Systems, vol. 24, no. 7, pp. 1024-1032. Varathan, K. D.; Giachanou, A.; Crestani, F. (2017). Comparative opinion mining: a review. Journal of the Association for Information Science an...

work page 2011
[6]

Wysocki, M. (2012). The role of social media in sports communication: An analysis of NBA Teams’ strategy. Yüksek Lisans Tezi, American University, Washington. Yu, Y.; Wang, X. (2015). World Cup 2014 in the Twitter World: A big data analysis of sentiments in US sports fans’ tweets. Computers in Human Behavior , vol. 48, pp. 392 -

work page 2012
[53]

Davletcharova, A.; Sugathan, S.; Abraham, B.; James, A. P. (2015). Detection and analysis of emotion from speech signals. Procedia Computer Science, vol. 58, pp. 91-96. De Albornoz; J. C.; Chugur, I.; Amigó, E. (2012). Using an Emotion-based Model and Sentiment Analysis Techniques to Classify Polarity for Reputation. In CLEF (Online Working Notes/Labs/Wor...

work page 2015
[400]

Winarsih, N. A. S.; Supriyanto, C. (2016). Evaluation of classification methods for Indonesian text emotion detection. International Seminar on Application for Technology of Information and Communication (ISemantic) , pp. 130-133. ISEAR Dataset, available at: http://emotionresearch.net/toolbox/toolboxdatabase. Last accessed 10-nov-2018. Retrieved from htt...

work page 2016
[983]

H.; Wu, C.; Potdar, V

Binali, H. H.; Wu, C.; Potdar, V. (2009). A new significant area: Emotion detection in e-learning using opinion mining techniques. 2009 3rd IEE E International Conference on Digital Ecosystems and Technologies, pp. 259-264. Cambria, E. (2016). Affective computing and sentiment analysis. IEEE Intelligent Systems, vol. 31, no. 2, pp. 102-107. Cambria, E.; G...

work page 2009
[1997]

ICICS., Proceedings of 1997 International Conference on, vol. 1, pp. 397-401. Gao, W.; Ai, H. (2009). Face gender classification on consumer images in a multi-ethnic environment. International Conference on Biometrics, pp. 169-178. Gratch, J. ; Lucas, G. ; Malandrakis, N. ; Szablowski, E .; Fessler, E. ; Nichols, J . (2015). GOAALLL!: Using sentiment in t...

arXiv 2009

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Anitescu, C.; Atroshchenko, E.; Alajlan, N.; & Rabczuk, T. (2019). Artificial neural network methods for the solution of second order boundary value problems. Computers, Materials & Continua, 59(1), 345-359. Asghar, M. Z.; Khan, A.; Bibi, A.; Kundi, F. M.; Ahmad, H. (2017): Sentence-Level Emotion Detection Framework Using Rule-Based Classification. Cognit...

work page 2019

[2] [2]

Uğuz, H. (2011). A two-stage feature selection method for text categorization by using information gain, principal component analysis and genetic algorithm. Knowledge-Based Systems, vol. 24, no. 7, pp. 1024-1032. Varathan, K. D.; Giachanou, A.; Crestani, F. (2017). Comparative opinion mining: a review. Journal of the Association for Information Science an...

work page 2011

[3] [6]

Wysocki, M. (2012). The role of social media in sports communication: An analysis of NBA Teams’ strategy. Yüksek Lisans Tezi, American University, Washington. Yu, Y.; Wang, X. (2015). World Cup 2014 in the Twitter World: A big data analysis of sentiments in US sports fans’ tweets. Computers in Human Behavior , vol. 48, pp. 392 -

work page 2012

[4] [53]

Davletcharova, A.; Sugathan, S.; Abraham, B.; James, A. P. (2015). Detection and analysis of emotion from speech signals. Procedia Computer Science, vol. 58, pp. 91-96. De Albornoz; J. C.; Chugur, I.; Amigó, E. (2012). Using an Emotion-based Model and Sentiment Analysis Techniques to Classify Polarity for Reputation. In CLEF (Online Working Notes/Labs/Wor...

work page 2015

[5] [400]

Winarsih, N. A. S.; Supriyanto, C. (2016). Evaluation of classification methods for Indonesian text emotion detection. International Seminar on Application for Technology of Information and Communication (ISemantic) , pp. 130-133. ISEAR Dataset, available at: http://emotionresearch.net/toolbox/toolboxdatabase. Last accessed 10-nov-2018. Retrieved from htt...

work page 2016

[6] [983]

H.; Wu, C.; Potdar, V

Binali, H. H.; Wu, C.; Potdar, V. (2009). A new significant area: Emotion detection in e-learning using opinion mining techniques. 2009 3rd IEE E International Conference on Digital Ecosystems and Technologies, pp. 259-264. Cambria, E. (2016). Affective computing and sentiment analysis. IEEE Intelligent Systems, vol. 31, no. 2, pp. 102-107. Cambria, E.; G...

work page 2009

[7] [1997]

ICICS., Proceedings of 1997 International Conference on, vol. 1, pp. 397-401. Gao, W.; Ai, H. (2009). Face gender classification on consumer images in a multi-ethnic environment. International Conference on Biometrics, pp. 169-178. Gratch, J. ; Lucas, G. ; Malandrakis, N. ; Szablowski, E .; Fessler, E. ; Nichols, J . (2015). GOAALLL!: Using sentiment in t...

arXiv 2009