{"id":"848dda98-1dd1-44ce-97a2-164219cc11ea","arxiv_id":"1908.01587","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A benchmark of eight standard classifiers on the ISEAR emotion dataset shows the back-propagation neural network (71.27% accuracy) and logistic regression (66.58%) score highest, but reporting errors in the paper make the ranking unreliable.","lead":"This study runs eight off-the-shelf machine learning classifiers on the ISEAR text dataset to see which best detects emotions such as joy, fear, and guilt. The paper recommends logistic regression and a neural network, but its own results tables contain conflicting numbers, so the recommendation is not reliable.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset-size contradiction (§3.1 vs Table 1 vs §4.2) makes the reported classifier ranking and §4.3 recommendation unreproducible; Naive Bayes accuracy also contradicts itself (Table 7 vs Table 12).","rationale":"I read the paper as a comparative benchmark whose stated contribution is the recommendation in Section 4.3. The reader's weakest assumption is the same one I would flag: the experimental setup is internally inconsistent in dataset size and split. That inconsistency is load-bearing because every accuracy, precision, recall, and F-measure in Tables 4-12 is a function of the training/test partition; if the dataset is not the 5477-row five-class ISEAR set described in Table 1, the ranking of classifiers and the recommendation cannot be trusted. I found independent evidence beyond the reader's: Table 12's Naive Bayes accuracy (66.58%) contradicts Table 7 (63.6%), and Section 4.4 only compares logistic regression despite BPN having the best raw accuracy; this suggests the recommendation text was not generated from a single consistent analysis. The paper has almost no independent support: no code or appendix implementation is included in the manuscript, no random seed is reported, and hyperparameters are mostly absent, so the experiments cannot be rerun from the text. The limitations section is candid about random splitting, the single dataset, and classical feature choices, but candor does not repair the unspecified dataset identity. For these reasons I maintain the rejection; no verdict adjustment is needed.","tokens_in":15580,"tokens_out":6169,"duration_ms":61374,"concrete_test":"Download the cited ISEAR dataset, keep only the five emotion labels listed in Table 1, count the records, split 80/20 with a fixed random seed, and reproduce the Naive Bayes accuracy on that split. If the row count is not 5477, or the test set is not about 1095, or the reproduced NB accuracy does not equal 63.6% (Table 7), then the paper's reported 2273 reviews, 750 test instances, Table 12 values, and the Section 4.3 recommendation cannot all be valid.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is Section 4.3's recommendation that BPN and logistic regression are the best classifiers. That claim depends entirely on the benchmark numbers in Tables 4-12, but those numbers are not anchored to a well-defined dataset. Section 3.1 states the ISEAR dataset comprises 2273 reviews; Table 1 reports 5477 reviews, with per-class counts summing to 5477; Section 4.2 reports CPU time for classifying 750 test instances. These figures are mutually incompatible: a 20% test split of 5477 gives about 1095 test instances, while a 20% split of 2273 gives about 455. The text also fluctuates between five and seven emotion categories, and the introduction says five classifiers while the experiments cover eight. Without knowing which dataset variant and split produced each row, no reported accuracy can be independently checked. A second, independent reproducibility failure is that Table 12 lists Naive Bayes accuracy as 66.58%, identical to logistic regression, while Table 7 gives the same classifier 63.6%. The recommendation itself is also not derivable from the stated numbers: BPN has the highest accuracy (71.27%) but is co-recommended with logistic regression (66.58%), and Section 4.4 only benchmarks logistic regression. These are not cosmetic issues: if the underlying measurements are ungrounded, the 'best classifier' answer to RQ3 is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares eight supervised machine learning classifiers (SVM, logistic regression, KNN, Naïve Bayes, random forest, XGBoost, SGD, and BPN) for emotion classification on the ISEAR dataset. The authors evaluate classifiers by accuracy, precision, recall, and F1-score, report per-emotion performance tables, and recommend BPN and logistic regression as the best classifiers for detecting the five emotion categories. The paper also includes pseudocode for preprocessing, feature engineering, classifier training, and evaluation.","tokens_in":15936,"tokens_out":3562,"duration_ms":31578,"significance":"If the comparative results were trustworthy, the paper would provide a useful practical benchmark for emotion classification on a standard public dataset, with the notable strength of reporting multiple evaluation metrics and per-class breakdowns. The use of a public dataset and the inclusion of algorithmic pseudocode are also positive features. However, the significance is severely limited by internal inconsistencies in the reported dataset size and in the results tables, which make the headline recommendation unsupportable and the empirical comparison unreproducible.","major_comments":[{"comment":"There is a direct contradiction in the dataset size: Section 3.1 states the ISEAR dataset comprises 2273 reviews, while Table 1 reports 5477 reviews with per-class counts summing to 5477 (1094+1095+1096+1096+1096 = 5477). Section 4.2 then states that CPU times are for classifying 750 test instances, which is neither 20% of 5477 (approximately 1095) nor 20% of 2273 (approximately 455). Because the evaluation split is not well-defined, the accuracies in Tables 4-12 cannot be independently checked, and the central claim in Section 4.3 is therefore ungrounded.","section":"Section 3.1 and Table 1 vs Section 4.2"},{"comment":"Table 12 lists Naïve Bayes accuracy as 66.58%, identical to the logistic regression accuracy, whereas Table 7 reports the Naïve Bayes accuracy as 63.6%. Since Table 12 is presented as the overall results table that supports the recommendation, this internal inconsistency means the comparison table itself is not reliable and the reported performance ordering is suspect.","section":"Table 12 vs Table 7"},{"comment":"The recommendation that both BPN and logistic regression 'have produced best results' is not derivable from the stated metrics: BPN achieves the highest accuracy (71.27%), but logistic regression achieves 66.58% accuracy, and no explicit selection criterion (e.g., accuracy, recall, F1, or a combination) is given that would select both classifiers. Moreover, Section 4.4 benchmarks only logistic regression against prior work, not BPN, leaving the co-recommendation unjustified.","section":"Section 4.3 and Section 4.4"},{"comment":"The set of classifiers is inconsistently described. The introduction says the study applies 'five machine learning classifiers' to detect 'seven categories of emotions', but Section 3.3 lists six classifiers (including both 'Naïve Bayesian' twice) and Section 4.1 lists eight classifiers including BPN. Algorithm 6's ML-classifiers list omits BPN entirely, even though BPN is part of the final recommendation. This inconsistency further hampers reproducibility.","section":"Section 3.3 and Section 4.1"},{"comment":"Limitation 2 in Section 5.1 says the experiments used 'a subsample of five thousand records', which contradicts Section 3.1's statement of 2273 reviews and Table 1's total of 5477. This is not a minor wording issue because it directly affects which dataset variant was used and how the reported results should be interpreted.","section":"Section 5.1"}],"minor_comments":[{"comment":"The manuscript contains stray editorial text such as 'Eq. 1 for an example. The number should be aligned to the right margin.' and 'Eq. 1 for an example.' These should be removed and proper equation numbering used.","section":"Section 3.3.5 and 3.3.6"},{"comment":"In Table 3, the review numbered 28 is labeled 'Guilt', but the described content ('I felt ashamed in front of the whole class') suggests the label should be 'Shame'; this cast doubt on the quality of the annotation in the sample, and the authors should verify labels in the full dataset.","section":"Table 3"},{"comment":"There are numerous typographical and referencing errors, including 'Jupiter notebook' (should be Jupyter), 'Wile' in Algorithm 2, 'adoptive' in Section 2, and many references listed in the bibliography (e.g., Bagić and Podobnik 2016, Lin and Utz 2015, Podobnik 2013, Siersdorfer et al. 2010) that are never cited in the body.","section":"Throughout"},{"comment":"The text says logistic regression achieved 83% recall for 'Joy' and 73% precision for 'sadness', but the per-class averages in Table 12 (precision 0.67, recall 0.67) are not reconciled with the claim that logistic regression is the best-performing classifier on these metrics.","section":"Section 4.2.2 and Table 5"},{"comment":"The comparison in Table 13 reports 'Our Work' accuracy of 66.5%, but the earlier results tables give 66.58%; the discrepancy is minor but should be made consistent.","section":"Table 13"}],"recommendation":"reject","confidential_remarks":"The manuscript has significant internal inconsistencies in the reported dataset size and classifier results that make the empirical claims unreproducible. The reference list also contains many entries that are never cited in the text, suggesting incomplete manuscript preparation. Even after correcting local inconsistencies, the central comparison would require re-running all experiments with a clearly stated dataset split; this is beyond the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick summary: this is a routine eight-classifier benchmark on the public ISEAR emotion dataset, and the main recommendation (BPN and logistic regression are best) is not supported by the paper's own tables. The dataset size is given as 2273 in Section 3.1, 5477 in Table 1, and 750 test instances in Section 4.2, so no result can be independently checked.\n\nWhat it does well: it uses a standard public dataset, reports per-class precision/recall/F1 for each classifier, and includes pseudocode for preprocessing and feature engineering. The comparison with Thomas et al. (2014) and Danisman & Alpkocak (2008) is sensible. The limitations section is candid about random splitting and using a single dataset. That is the extent of the credit; this is an incremental parameter scan with no new method, feature, or theory.\n\nThe soft spots are not cosmetic. The 2273/5477/750 contradiction is load-bearing because the accuracies in Tables 4-12 are the whole point. A 20% split of 5477 gives roughly 1095 test examples, not 750; a 20% split of 2273 gives roughly 455. Additionally, Table 12 lists Naive Bayes accuracy as 66.58%, identical to logistic regression, while Table 7 says 63.6%. The recommendation section names both BPN (71.27%) and LR (66.58%) as best, which is odd given BPN is five points higher; then Section 4.4 only compares LR with prior work. The paper also says five classifiers in the introduction and runs eight, and fluctuates between five and seven emotion categories. There are no significance tests, no cross-validation, no hyperparameters (other than RF(200) and KNN(25)), and no code or data release. These are not minor polish items; they make RQ3's answer unreproducible.\n\nOverall: the paper is for someone wanting a quick list of baseline numbers on ISEAR, but the numbers are not reliable as reported. A serious referee would need the corrected dataset size, consistent tables, code, and ideally repeated runs with variance before the comparison can be trusted. I would not cite it, and I would not send it to review in its current form; a desk reject with an invitation to resubmit after fixing the reporting and adding reproducibility material is the proportionate response.","headline":"Routine ISEAR classifier benchmark whose 'best classifier' recommendation is undermined by inconsistent dataset-size and accuracy reporting; useful mainly as a cautionary example.","tokens_in":16467,"tokens_out":2498,"would_cite":false,"duration_ms":25558,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On the ISEAR emotion dataset, a back-propagation neural classifier reaches the highest accuracy at 71.27%, with logistic regression at 66.58% accuracy and the best joy recall (83%); the paper recommends these two.","keywords":["emotion detection","supervised machine learning","ISEAR dataset","text classification","logistic regression","back-propagation neural classifier","sentiment analysis","performance evaluation"],"falsifier":"Counting the rows in the exact dataset file the experiments used and repeating the 80/20 split would settle the claim: if the file contains 2,273 reviews rather than 5,477, or if the test partition is not 20 percent of the stated total, then the per-class counts in Table 1 and the reported accuracies cannot be reproduced, and the BPN and logistic-regression recommendation would need to be recomputed.","tokens_in":15321,"feed_emoji":"😊","tokens_out":8240,"duration_ms":76165,"temperature":0.7,"pith_summary":"The paper tries to establish which supervised machine-learning classifier should be used to detect emotions in written text, and it argues that, on the ISEAR benchmark corpus, the answer is a back-propagation neural classifier (BPN) and logistic regression. It reports BPN as the accuracy leader at 71.27% and logistic regression as the balanced leader at 66.58% accuracy, with the top average precision and recall (0.67) and an 83% recall for joy. This matters because emotion detection from online content is a practical task for which earlier studies often tested only a single classifier or a smaller classifier set. The paper's contribution is a direct eight-classifier comparison under one feature-engineering and evaluation protocol, ending in a concrete recommendation.","feed_headline":"Two classifiers top the ISEAR emotion-detection benchmark","feed_subtitle":"Back-prop nets score 71.27% accuracy; logistic regression reaches 83% recall for joy.","key_machinery":"The central machinery is the ISEAR corpus—a benchmark collection of self-reported emotional episodes labeled by emotion—combined with a uniform pipeline: tokenization, stop-word and punctuation removal, word-count and TF-IDF feature encoding, an 80/20 random split, and evaluation with precision, recall, F1, and accuracy. This single protocol is what lets the paper compare eight classifiers on the same footing; the ranking in Table 12 is the argument's load-bearing output.","core_discovery":"On its own terms, the paper's discovery is that a supervised emotion-classification pipeline—tokenization, stop-word removal, word-count vectors and TF-IDF weighting, an 80/20 random train/test split, and evaluation by precision, recall, F1, and accuracy—yields a clear ranking of eight classifiers on five emotion categories from the ISEAR corpus. Back-propagation neural classifier produces the best overall accuracy (71.27%) and the best precision for guilt (0.72); logistic regression produces the best balanced averages (0.67 precision, 0.67 recall, 66.58% accuracy) and the highest single-class recall (0.83 for joy). SVM is strongest for joy by precision and F1 (0.76 and 0.77), while KNN is weakest overall (57.81%). The paper concludes by recommending BPN and logistic regression for emotion detection from text.","pith_inferences":["A natural extension the paper does not run is cross-corpus transfer: train the same eight classifiers on ISEAR and evaluate on tweets or product reviews, where emotion vocabulary and style differ; the ranking could shift even if the within-corpus ranking holds.","The 80/20 random split with no repeated trials means the gaps between neighboring classifiers (for example, logistic regression at 66.58% versus SGD at 65.57%) may be within random fluctuation, so repeated stratified runs would turn the ranking into a firmer recommendation.","The metric-dependent winners suggest a decision rule rather than a single champion: choose BPN when overall accuracy matters most, logistic regression when recall for joy matters most, and SVM when per-class precision and F1 for joy are the target."],"forward_implications":["A practitioner building an emotion detector for text that resembles ISEAR self-reports can start with logistic regression or BPN rather than KNN or XGBoost and expect better accuracy and F1 on the five tested emotion categories.","The reported figures give concrete baselines—BPN 71.27%, logistic regression 66.58%, SVM 64.66%—that later systems using deep learning or word embeddings would need to beat on ISEAR.","If the goal is to avoid missing joy specifically, logistic regression's 83% recall makes it the better choice among the tested classifiers.","Because the tests use a random split rather than cross-validation, the ranking is a point estimate for that split rather than a certified guarantee for other samples."],"supporting_citations":[{"why":"Supplies the benchmark corpus of self-reported emotional episodes that all classifiers are trained and tested on.","marker":"(ISEAR, 2018)"},{"why":"Is the single-classifier baseline on the same corpus that this work extends by testing more classifiers and more emotion signals.","marker":"Thomas et al. (2014)"},{"why":"Supplies the Naïve Bayes classifier method and a comparable ISEAR result used in the evaluation table.","marker":"Danisman and Alpkocak, 2008"},{"why":"Provides the NLTK Python toolkit used to implement the machine-learning classifiers.","marker":"Loper and Bird, 2002"},{"why":"Supplies the TF-IDF feature representation scheme used to encode the text before classification.","marker":"Patil, L. H., & Atique, M. 2013"},{"why":"Supplies the XGBoost gradient-boosting algorithm included in the comparison.","marker":"Babajide Mustapha & Saeed 2016"},{"why":"Provides the precision, recall, and F-measure evaluation framework used to rank the classifiers.","marker":"Quan & Ren, 2010"}],"fun_headline_variants":["BPN and logistic regression lead emotion detection benchmark","Neural net tops emotion detection accuracy, logistic regression top recall","Emotion detection: BPN and logistic regression outperform SVMs and KNN","Which ML classifier detects emotions best? BPN and logistic regression","ISEAR benchmark: BPN best precision, logistic regression best joy recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand on the assumption that the dataset used in the experiments is the standard 5,477-instance ISEAR subset split 80/20 into training and test sets; the paper elsewhere describes the dataset as 2,273 reviews and mentions 750 test instances, so the reported class counts and accuracies are only as solid as that 5,477/80-20 premise.","fun_headline_variants_meta":{"raw":{"variants":["BPN and logistic regression lead emotion detection benchmark","Neural net tops emotion detection accuracy, logistic regression top recall","Emotion detection: BPN and logistic regression outperform SVMs and KNN","Which ML classifier detects emotions best? BPN and logistic regression","ISEAR benchmark: BPN best precision, logistic regression best joy recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1474,"prompt_tokens":840,"completion_tokens":634,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":546}},"tokens_in":456,"tokens_out":634,"duration_ms":6175,"temperature":1.0,"reasoning_tokens":546,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:05:29.487646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Counting the rows in the exact dataset file the experiments used and repeating the 80/20 split would settle the claim: if the file contains 2,273 reviews rather than 5,477, or if the test partition is not 20 percent of the stated total, then the per-class counts in Table 1 and the reported accuracies cannot be reproduced, and the BPN and logistic-regression recommendation would need to be recomputed.","supporting_citations":[],"review_version":1}