Speech Emotion Detection Based on MFCC and CNN-LSTM Architecture

Qianhe Ouyang

REVIEW 5 major objections 5 minor 1 cited by

Speech Emotion Detection Based on MFCC and CNN-LSTM Architecture

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A CNN-LSTM trained on MFCC and spectral features from 2,459 acted utterances classifies seven speech emotions with 61.07% accuracy, with anger and neutral most reliable.

desk verdict A plausible 61% SER result that is unverifiable as reported, because the data split is undocumented and the test set appears to have been used for model selection. read the letter →

arxiv 2501.10666 v1 pith:KQHP6KTN submitted 2025-01-18 cs.SD cs.LGeess.AS

Qianhe Ouyang This is my paper · ORCID

classification cs.SDcs.LGeess.AS

keywords speechemotionrecognitionCNN-LSTMMFCCSAVEERAVDESSaudioclassificationconfusionspectralfeatures

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a hybrid CNN-LSTM can recognize seven emotions from speech with about 61% accuracy using MFCC and related spectral features on a combined acted-speech dataset. It matters because speech emotion detection remains an open problem, and a working seven-way classifier with per-emotion breakdown gives a concrete baseline for which emotions are easy or hard to recognize. The author's main empirical finding is that recognition quality is emotion-dependent: anger and neutral are detected well above the average, while disgust falls well below it, and the confusion matrix shows negative emotions mixing with each other and surprise straddling positive and negative contexts. The paper argues this reflects the acoustic properties of the emotions themselves rather than only model limitations.

What carries the argument

The carrying mechanism is the CNN-LSTM pipeline applied to spectral image inputs. Each utterance is converted to a waveplot or spectrogram and a feature vector containing MFCCs, pitch statistics, formant frequencies, and peak frequency information; four convolutional layers extract local spatial patterns, three LSTM layers model the temporal sequence of those features, and a dense layer outputs one of seven emotion classes. The MFCC step is the load-bearing representation because it maps audio to a Mel-frequency axis that mirrors human pitch perception, which is what makes the classifier's emotion distinctions possible.

What would settle it

Run the same CNN-LSTM on the same 2,459 utterances using a leave-one-speaker-out split, where no speaker appears in both training and test. If overall accuracy falls well below 61.07%, the original figure partly reflected speaker identity rather than emotion recognition.

Watch

Extended reading notes

Core claim

The paper's central claim is that a hybrid CNN-LSTM, trained on MFCCs and related spectral features extracted from 2,459 acted utterances drawn from SAVEE and RAVDESS, classifies seven emotions with 61.07% overall test accuracy, with anger at 75.31% and neutral at 71.70%, and disgust at only 38.33%. The author further argues that these differences are not random: emotions that are frequent and acoustically distinct, such as anger, happy, and neutral, are misclassified least, while context-dependent emotions like surprise are confused with both positive and negative categories, and negative emotions such as sad, fear, and disgust are often mixed with one another. The paper presents this as evidence that emotion recognition accuracy depends on the properties of the emotion itself, not solely on the classifier.

Load-bearing premise

The reported accuracy presupposes that the training and test split separates speakers and recording sessions and that class imbalance does not distort the average; the paper does not describe this split, so the 61.07% score could be inflated by the model recognizing voices instead of emotions.

Editorial extensions

If this is right

A seven-way speech emotion classifier can reach 61.07% accuracy on a combined acted dataset of 2,459 utterances, providing a concrete baseline for later work.
Anger and neutral are anchor emotions in this feature space, both above 70%, so they can be detected with reasonable confidence in practical systems.
Disgust is the weak class at 38.33%; a deployed system should either gather more disgust-specific data or accept confusion with other negative emotions.
The confusion matrix implies that surprise should not be treated as a single discrete label, since it is split between positive and negative interpretations.
The next improvement target is the negative-emotion cluster (sad, fear, disgust), not the overall architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: a speaker-disjoint split would likely lower the 61.07% score, because acted corpora like these include highly recognizable voices; the paper's split is unspecified.
Inference: the reported roughly 10% gain from separating male and female voices suggests pitch normalization across gender would make emotion recognition more robust and is a testable next step.
Inference: the paper's confusability story could be checked by training the same CNN-LSTM on continuous valence or arousal ratings; if surprise sits at the boundary, the discrete seven-way task is the reason for its mixed misclassifications.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: this paper reports 61.07% accuracy on a 7-class SAVEE/RAVDESS subset with a CNN-LSTM on MFCC, but the number is not trustworthy as reported. The train/test split is never defined, and Figure 4 suggests the test set was used to monitor training. That's a load-bearing issue, not a cosmetic one.

What's genuinely here: the paper combines selected parts of two public datasets, drops calm, and runs a standard CNN-LSTM with four conv layers and three LSTM layers. The per-class results are the only real content: anger 75.31%, neutral 71.70%, disgust 38.33%. The confusion-matrix observation that negative emotions (sad, fear, disgust) mix with each other is plausible and backed by the matrix, even if the subsequent generalization about emotion semantics is speculative. The writing is clear and the feature-extraction description, while high-level, is coherent.

The soft spots are significant. First, no data partitioning details: there's no statement about random vs. stratified split, no speaker-independent split, and no indication of whether the same speakers appear in both train and test. With two small datasets, speaker overlap is a real risk and would let the model exploit speaker identity. Second, the test-loss curve in Fig. 4 is described as stabilizing after about 200 epochs, which implies the test set was observed during training, likely for early stopping. That compromises the reported accuracy as an unbiased estimate. Third, calm was removed after seeing an "unsatisfactory training result"—this is post-hoc pruning, which alters the task and biases the reported numbers upward. Fourth, there are no error bars, no cross-validation, no baseline comparisons with existing SER systems. The feature selection "heuristics" are also left unspecified.

On the citation side, the paper cites generic SER work but misses the many prior CNN-LSTM SER papers, so it doesn't position itself relative to the actual state of the art.

The math is not the problem; the reporting is. The paper is internally coherent and the architecture is reasonable, but the empirical claim cannot be reproduced or verified from the text.

This is a paper for someone who wants a hint of what a CNN-LSTM might get on this dataset mix. It is not a citable result. I would desk reject it, with clear guidance on what would make it salvageable: a speaker-independent split, a written hold-out protocol, error bars or cross-validation, and a baseline comparison.

Recommendation: no refs. Not ready for peer review as is.

Referee Report

5 major / 5 minor

Summary. The paper proposes a hybrid CNN-LSTM architecture for speech emotion recognition on a combined subset of the SAVEE and RAVDESS datasets. The pipeline extracts MFCC, pitch, and other statistical features using Librosa, then feeds spectrogram-style image inputs through four convolutional layers and three LSTM layers. The authors report a test-set accuracy of 61.07% across seven emotions (calm is excluded from the original RAVDESS labels), with per-class accuracies from 38.33% (disgust) to 75.31% (anger), and they interpret the confusion matrix as evidence that emotion properties influence classifiability. The paper includes no comparison to baselines, no error bars or repeated runs, and no explicit description of the train/test split.

Significance. If the reported accuracy were established through a clean, reproducible evaluation protocol, the result would be an incremental data point for the common CNN-LSTM approach to speech emotion recognition: it would confirm that MFCC-based features plus a convolutional-recurrent stack can reach roughly 60% on a seven-class SAVEE/RAVDESS mix, with anger and neutral being relatively easy classes and disgust being hard. The paper's strength is that it uses two widely available public datasets and reports a full confusion matrix, which facilitates future comparisons. However, the contribution is primarily empirical, and the current evaluation is too underspecified to support the headline number; the absence of baselines also prevents the paper from demonstrating any advantage attributable to the hybrid architecture.

major comments (5)

[Section 2.1 and Section 2.3] The manuscript never specifies how the 2,459 samples are partitioned into training and test sets, nor whether the partition prevents the same speaker from appearing in both sets; because the reported 61.07% accuracy is the central empirical claim, this missing information leaves the result unreproducible and the risk of speaker-identity leakage unaddressed.
[Figure 4 and Section 3] The statement that 'the loss for the test set tends towards stability after the epochs reaching approximately 200' indicates that the test set was evaluated during training, presumably for early stopping or hyperparameter selection; if the test set influenced any training decision, the reported test accuracy is optimistically biased, and the claim that overfitting was 'reduced as much as possible' is not supported.
[Section 2.1] The removal of calm utterances from the dataset because of an 'unsatisfactory training result' is an outcome-based selection: the task is altered after observing performance, which invalidates the reported seven-class accuracy as an estimate of generalization to the original seven-emotion label set and weakens the conclusion that the model performs well on 'common emotions'.
[Table 1] The overall accuracy of 61.07% does not match the unweighted mean of the per-class accuracies (approximately 59.8%), and no class sizes or weighted aggregation formula are given; without this information the reader cannot verify the headline number or interpret the per-class rates consistently.
[Section 3] No comparison to any baseline (e.g., a CNN-only model, an LSTM-only model, or a standard MFCC plus shallow classifier) is provided, and no cross-validation or repeated runs with error bars are reported; consequently, the paper does not demonstrate that the CNN-LSTM architecture is responsible for the observed accuracy rather than the feature set or dataset construction choices.

minor comments (5)

[Section 2.2] The architecture description gives the number of convolutional and LSTM layers but not the number of filters, kernel sizes, or LSTM units, so the architecture cannot be reproduced from the text.
[Figure 5] The confusion matrix is described in the text but the axes are not labeled in the figure caption; please add class names and indicate whether the entries are counts or percentages.
[Section 2.1] The feature-heuristic selection is stated only as 'heuristics are implemented to score each feature'; without the scoring criterion or the number of features retained, this step is not reproducible.
[References [7] and [8]] The dataset references point to Kaggle URLs without version identifiers or access dates; please provide a formal citation or a permanent repository identifier.
[Abstract and Section 3] The term 'comprehensively' is undefined; specify whether the 61.07% figure is a macro-average, micro-average, or class-weighted accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the 61.07% accuracy is an empirical measurement, and no load-bearing claim reduces to its inputs or to a self-citation.

full rationale

Walking the derivation chain: the paper's central claim is the measured 61.07% test-set accuracy of a trained CNN-LSTM model (Abstract; Section 3, Table 1). This is an empirical result of a fit, not a quantity derived from a premise, so there is no fitted parameter renamed as a prediction, no equation that equates an output to an input by construction, and no uniqueness or existence theorem imported from the authors' prior work. All cited references ([7] SAVEE, [8] RAVDESS, [9] librosa, [11] Donahue et al., [12] Palo et al.) are external sources used for data provenance and architecture motivation, and none is authored by the present author, so no self-citation is load-bearing. The closing statements that classification accuracy depends on emotion properties, and that surprise confuses with positive/negative emotions, are post-hoc interpretations of the confusion matrix in Figure 5 rather than circular derivations: the per-class accuracies in Table 1 are the measured data, and the prose restates them in causal-sounding language, which is interpretive gloss rather than a reduction of the claim to its own input. The weaknesses flagged by a skeptical reader — unspecified train/test partitioning (Section 2.1, Section 2.3), test loss being monitored during training (Figure 4: 'the loss for the test set tends towards stability after the epochs reaching approximately 200'), and removal of calm utterances due to 'unsatisfactory training result' (Section 2.1) — are correctness, reproducibility, and validity concerns about whether 61.07% reflects true generalization, not instances of the claim becoming equivalent to its inputs by definition. Accordingly, the honest finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central accuracy claim depends on the dataset construction (including the outcome-based removal of calm), the unstated data split, and the hand-set hyperparameters. The paper introduces no new theoretical entities, so the ledger is short but the unstated split is the largest burden.

free parameters (2)

Model hyperparameters (learning rate, decay, dropout, epochs, batch size) = lr=1e-5, decay=1e-6, CNN dropout=0.1, LSTM dropout=0.2, epochs=370, batch_size=16
Chosen by hand in Section 2.3 without a described validation or tuning procedure; the reported test accuracy depends on these choices.
Feature selection heuristic = not specified
Section 2.1 states a heuristic scores roughly 40,000 features, but the scoring rule and number of selected features are not given, so the feature set is an unquantified modeling choice.

assumptions (4)

domain assumption The emotion labels in SAVEE and RAVDESS are treated as ground truth.
Used throughout the dataset construction; actor portrayals are assumed to reflect the intended emotion.
domain assumption The chosen audio features (MFCC, pitch statistics, energy) capture emotion-relevant information.
Section 2.1 selects these features without evidence that they are sufficient for the seven classes.
domain assumption The unspecified train/test split is speaker-disjoint and representative.
The paper never describes the split, so the reported 61.07% assumes generalization rather than speaker or recording leakage.
ad hoc to paper Removing calm emotion is valid because it improves training results.
Section 2.1 excludes calm utterances based on observed poor performance, an outcome-driven selection not justified independently of the results.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Speech Emotion Detection Based on MFCC and CNN-LSTM Architecture." pith.science (2026). https://pith.science/paper/KQHP6KTN

@misc{pith2026250110666,
  author       = {Pith},
  title        = {Pith review of: Speech Emotion Detection Based on MFCC and CNN-LSTM Architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KQHP6KTN}},
  note         = {Machine review of arXiv:2501.10666}
}

read the original abstract

Emotion detection techniques have been applied to multiple cases mainly from facial image features and vocal audio features, of which the latter aspect is disputed yet not only due to the complexity of speech audio processing but also the difficulties of extracting appropriate features. Part of the SAVEE and RAVDESS datasets are selected and combined as the dataset, containing seven sorts of common emotions (i.e. happy, neutral, sad, anger, disgust, fear, and surprise) and thousands of samples. Based on the Librosa package, this paper processes the initial audio input into waveplot and spectrum for analysis and concentrates on multiple features including MFCC as targets for feature extraction. The hybrid CNN-LSTM architecture is adopted by virtue of its strong capability to deal with sequential data and time series, which mainly consists of four convolutional layers and three long short-term memory layers. As a result, the architecture achieved an accuracy of 61.07% comprehensively for the test set, among which the detection of anger and neutral reaches a performance of 75.31% and 71.70% respectively. It can also be concluded that the classification accuracy is dependent on the properties of emotion to some extent, with frequently-used and distinct-featured emotions having less probability to be misclassified into other categories. Emotions like surprise whose meaning depends on the specific context are more likely to confuse with positive or negative emotions, and negative emotions also have a possibility to get mixed with each other.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Explainable Lightweight Compact Deep Models for Speech Emotion Recognition
cs.SD 2026-07 conditional novelty 3.0 of 10

A 33k-parameter CNN with attentive statistics pooling and Grad-CAM reaches 96.9% accuracy on SAVEE speech emotion recognition, but the evaluation rests on one speaker-independent split.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · cited by 1 Pith paper

[1]

The emotion detection services could enable users to have enhanced experiences to manipulate and adjust their real-time emotions, receiving feedback timely

Introduction Emotion detection currently has become a nascent and attractive field where many researchers applied different applications, and the potential necessity and significance have gradually emerged with the increasing appearance of human-computer circumstances. The emotion detection services could enable users to have enhanced experiences to manip...

work page 2015
[2]

Dataset description and preprocessing For the dataset, two relatively common-used datasets are taken into consideration as the original speech signal

Methods 2.1. Dataset description and preprocessing For the dataset, two relatively common-used datasets are taken into consideration as the original speech signal. The Surrey Audio-Visual Expressed Emotion (SAVEE) dataset was recorded from four male native English speakers (identified as DC, JE, JK, KL) who are also postgraduate students aged from 27 to 3...

work page 2000
[3]

The model loss for training and test set Firstly, the basic training effects of the model using the architectures designed above are illustrated in Figure 4

Result and discussion Figure 4. The model loss for training and test set Firstly, the basic training effects of the model using the architectures designed above are illustrated in Figure 4. It could be illustrated in Fig.4 that the loss of the model decreases in the whole process with the increase of epochs for the training set and continues to decline wi...

work page
[4]

Regarding the feature extraction, features from four aspects are adopted for further analysis and the MFCC processing provided by the Librosa is specifically vital

Conclusion In this paper, the problem of emotion detection in speech cases is tackled using the designed feature extraction methods and the CNN-LSTM network as mentioned above. Regarding the feature extraction, features from four aspects are adopted for further analysis and the MFCC processing provided by the Librosa is specifically vital. After the prepr...

work page
[5]

Ruiz L Z et al. 2017 Human emotion detection through facial expressions for commercial analysis 2017IEEE 9th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment and Management (HNICEM) pp 1-6

work page 2017
[6]

2015 Is virtual reality emotionally arousing? Investigating five emotion inducing virtual park scenarios International journal of human-computer studies 82 pp 48-56

Felnhofer A et al. 2015 Is virtual reality emotionally arousing? Investigating five emotion inducing virtual park scenarios International journal of human-computer studies 82 pp 48-56

work page 2015
[7]

Yu J 2014 A video, text, and speech-driven realistic 3-D virtual head for human machine interface IEEE transactions on cybernetics 45(5) pp 991-1002

work page 2014
[8]

2015 Performance evaluation of different support vector machine kernels for face emotion recognition 2015 SAI Intelligent Systems Conference (IntelliSys) pp 804-806

Adeyanju I A et al. 2015 Performance evaluation of different support vector machine kernels for face emotion recognition 2015 SAI Intelligent Systems Conference (IntelliSys) pp 804-806

work page 2015

Show all 14 references

[9]

2015 Improved emotion recognition using GMM-UBMs 2015 International Conference on Signal Processing and Communication Engineering Systems pp 53-57

Vydana H K et al. 2015 Improved emotion recognition using GMM-UBMs 2015 International Conference on Signal Processing and Communication Engineering Systems pp 53-57

2015
[10]

Shahin I et al. 2019 Emotion recognition using hybrid Gaussian mixture model and deep neural network IEEE access 7 pp 26777-26787 [7]Eu J L 2019 Surrey Audio-Visual Expressed Emotion 2022 https://www.kaggle.com/datasets/ejlok1/surrey-audiovisual-expressed- emotion-savee [8]Ste...

2019
[11]

2015 librosa: Audio and music signal analysis in python Proceedings of the 14th python in science conference Vol 8 pp 18-25

McFee B et al. 2015 librosa: Audio and music signal analysis in python Proceedings of the 14th python in science conference Vol 8 pp 18-25

2015
[12]

Logan B 2000 Mel frequency cepstral coefficients for music modeling International Symposium on Music Information Retrieval

2000
[13]

2015 Long-term recurrent convolutional networks for visual recognition and description Proceedings of the IEEE conference on computer vision and pattern recognition pp 2625- 2634

Donahue J et al. 2015 Long-term recurrent convolutional networks for visual recognition and description Proceedings of the IEEE conference on computer vision and pattern recognition pp 2625- 2634

2015
[14]

2015 Classification of emotions of angry and disgust SmartCR 5(3) pp 151-158

Palo H K et al. 2015 Classification of emotions of angry and disgust SmartCR 5(3) pp 151-158

2015

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

The emotion detection services could enable users to have enhanced experiences to manipulate and adjust their real-time emotions, receiving feedback timely

Introduction Emotion detection currently has become a nascent and attractive field where many researchers applied different applications, and the potential necessity and significance have gradually emerged with the increasing appearance of human-computer circumstances. The emotion detection services could enable users to have enhanced experiences to manip...

work page 2015

[2] [2]

Dataset description and preprocessing For the dataset, two relatively common-used datasets are taken into consideration as the original speech signal

Methods 2.1. Dataset description and preprocessing For the dataset, two relatively common-used datasets are taken into consideration as the original speech signal. The Surrey Audio-Visual Expressed Emotion (SAVEE) dataset was recorded from four male native English speakers (identified as DC, JE, JK, KL) who are also postgraduate students aged from 27 to 3...

work page 2000

[3] [3]

The model loss for training and test set Firstly, the basic training effects of the model using the architectures designed above are illustrated in Figure 4

Result and discussion Figure 4. The model loss for training and test set Firstly, the basic training effects of the model using the architectures designed above are illustrated in Figure 4. It could be illustrated in Fig.4 that the loss of the model decreases in the whole process with the increase of epochs for the training set and continues to decline wi...

work page

[4] [4]

Regarding the feature extraction, features from four aspects are adopted for further analysis and the MFCC processing provided by the Librosa is specifically vital

Conclusion In this paper, the problem of emotion detection in speech cases is tackled using the designed feature extraction methods and the CNN-LSTM network as mentioned above. Regarding the feature extraction, features from four aspects are adopted for further analysis and the MFCC processing provided by the Librosa is specifically vital. After the prepr...

work page

[5] [5]

Ruiz L Z et al. 2017 Human emotion detection through facial expressions for commercial analysis 2017IEEE 9th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment and Management (HNICEM) pp 1-6

work page 2017

[6] [6]

2015 Is virtual reality emotionally arousing? Investigating five emotion inducing virtual park scenarios International journal of human-computer studies 82 pp 48-56

Felnhofer A et al. 2015 Is virtual reality emotionally arousing? Investigating five emotion inducing virtual park scenarios International journal of human-computer studies 82 pp 48-56

work page 2015

[7] [7]

Yu J 2014 A video, text, and speech-driven realistic 3-D virtual head for human machine interface IEEE transactions on cybernetics 45(5) pp 991-1002

work page 2014

[8] [8]

2015 Performance evaluation of different support vector machine kernels for face emotion recognition 2015 SAI Intelligent Systems Conference (IntelliSys) pp 804-806

Adeyanju I A et al. 2015 Performance evaluation of different support vector machine kernels for face emotion recognition 2015 SAI Intelligent Systems Conference (IntelliSys) pp 804-806

work page 2015