REVIEW 3 major objections 6 minor 43 references
Emotion Recognition for Low-Resource Turkish: Fine-Tuning BERTurk on TREMO and Testing on Xenophobic Political Discourse
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Fine-tuning BERTurk on TREMO yields 92.62% Turkish emotion accuracy, and applying it to 'sessiz istila' tweets shows anger and surprise dominating anti-refugee discourse.
desk verdict A standard BERTurk fine-tune with a potentially useful descriptive case study, but the central emotion distribution rests on unvalidated out-of-domain labels and the reported numbers do not add up. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a fine-tuning pipeline: BERTurk, a transformer pretrained on Turkish text, is trained for three epochs on the TREMO emotion dataset, which has been balanced to 3,003 labeled sentences per emotion across six categories. Tweets are normalized by replacing retweets, URLs, mentions, hashtags, and emojis with fixed Turkish tokens, and the classifier's predictions are kept only when the highest softmax probability is at least 0.6, with lower-confidence predictions marked ambiguous. The resulting model is then run over the 47,024-tweet 'sessiz istila' corpus, and the monthly and yearly counts of each emotion are compared.
What would settle it
Have Turkish-speaking annotators label a random sample of the 47,024 'sessiz istila' tweets into the same six emotion categories, then compare their labels with the model's confident predictions; if agreement on anger and surprise is near chance, the temporal trends are artifacts of the model rather than public sentiment.
Extended reading notes
Core claim
The paper's central claim is that fine-tuning BERTurk on a balanced version of the TREMO dataset produces a reliable Turkish emotion-recognition model for six basic emotion categories. On the held-out test set the model reaches 92.62% overall accuracy, with F1 scores per emotion ranging from 0.9091 to 0.9505 and with happiness and disgust easiest to recognize, sadness and anger hardest. Applied to the 'sessiz istila' corpus, the model reports anger in 43.6% and surprise in 33.4% of confident predictions, and it tracks a year-by-year shift: surprise dominates 2021, anger dominates 2022, with the highest tweet volumes and strongest anger appearing in May and June 2022. The paper presents this as evidence that a localized transformer model can capture emotional dynamics in Turkish anti-refugee discourse.
Load-bearing premise
The paper assumes that the emotions the model learned from the TREMO training sentences are the same emotions expressed in real Turkish tweets about refugees, without human-checked labels on those tweets to confirm it.
Editorial extensions
If this is right
- A Turkish emotion classifier at this accuracy level can support real-time monitoring of social-media sentiment for marketing, public relations, and crisis management.
- The emotion timeline—surprise in 2021, anger in 2022, peaking with the film's release—shows that counting emotions by month is a practical way to track how anti-refugee discourse intensifies around events.
- Balancing TREMO to 3,003 sentences per emotion before fine-tuning is a straightforward recipe for improving emotion classification in low-resource languages with skewed label distributions.
- The difficulty the model has separating anger from fear pinpoints where future Turkish emotion models need better contextual representations.
Reading between the lines
- Editorial inference: the surprise-to-anger shift could be read as a two-stage public response—initial shock, then sustained moral outrage—but the paper does not test that causal story.
- Editorial inference: because the model discards predictions below a 0.6 confidence threshold, the reported anger and surprise percentages apply only to confident predictions; if the ambiguous tweets differ, the true emotion proportions in the full 47,024-post corpus could shift.
- Editorial inference: the same fine-tuning recipe could generalize to other morphologically rich low-resource languages that have a small emotion-labeled dataset, though the paper does not claim this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a BERTurk-based emotion recognition model fine-tuned on the balanced TREMO dataset for Turkish, claiming 92.62% accuracy across six Ekman emotion categories. The model is then applied to 47,024 Turkish X posts containing the keyword "sessiz istila" from June 2021 to December 2022, yielding a reported distribution with anger at 43.6% and surprise at 33.4%, and a shift from surprise-dominated discourse in 2021 to anger-dominated discourse in 2022. The paper presents this as a contribution to low-resource Turkish emotion recognition and to computational social science, with implications for monitoring social sentiment in crisis- and policy-related contexts.
Significance. If the central claims were fully supported, the paper would provide a useful demonstration of fine-tuning a transformer model for Turkish emotion classification and a substantive case study of emotional dynamics in anti-refugee discourse. The strengths are the relevant task, the use of a language-specific pretrained model (BERTurk), and the effort to balance the TREMO dataset. However, several load-bearing issues currently undermine the claims: the reported accuracy is internally inconsistent with the described train/test split, the target-domain emotion counts are not validated against human annotation, and the paper's own limitation statement acknowledges confusion between anger and fear, which is the headline emotion. The paper also does not provide code, data, seeds, or baselines, so the reported figures are not independently checkable. As presented, the contribution is not yet established.
major comments (3)
- [§3.3, §4.2, Eq. (2)] The confusion matrix total of 3,605 predictions is inconsistent with the stated 90/10 split on the balanced 18,018-instance TREMO dataset, which should yield approximately 1,802 test predictions. Moreover, the reported diagonal sum of 3,338 exceeds the entire expected test set, so the computed accuracy of 92.62% cannot be correct as described. The Discussion section also states 92.47% accuracy instead. Please provide the actual test-set size and confusion-matrix dimensions, correct the accuracy calculation, and report all metrics consistently (or make the evaluation code and split indices public so the discrepancy can be resolved).
- [§3.5, §4.3, Table 4] The emotion percentages for the target corpus are based on counts that sum to 40,880, whereas the collected dataset is described as 47,024 tweets. Predictions below the 0.6 confidence threshold are labeled as ambiguous (-1) and are excluded from the reported counts, but the number of excluded tweets is never given, and no analysis shows that exclusion is not systematically correlated with particular emotions or time periods. The reported anger (43.6%) and surprise (33.4%) distribution is therefore defined over an uncharacterized subset and cannot be taken as the emotion distribution of the collected discourse without reporting the excluded counts and a sensitivity analysis.
- [§4.3, §5.1, Table 4] The headline finding that anger dominates the sessiz istila corpus (43.6%) is directly threatened by the model's admitted difficulty in differentiating anger from fear, as stated in Section 5.1. Fear is the fourth-largest category at 11.2%, so even a modest error rate between these two classes could materially change the reported dominance. The paper provides no target-domain validation: there is no comparison of model labels with human annotations on a sample of sessiz istila tweets, no inter-annotator agreement, and no error analysis on political or Xenophobic discourse. Without such validation, the temporal anger/surprise findings remain possible artifacts of applying a TREMO-trained model to out-of-domain text. Please add a target-domain evaluation with human-annotated samples and report confusion and agreement metrics for those samples.
minor comments (6)
- [§2] The paper refers to "Sessis Istila" in the paragraph on Erbaysal Filibeli & Öneren Özbek; this appears to be a typo for "Sessiz Istila" and should be corrected.
- [§4.3] The text states that "fear was recorded 5,581 times" while Table 4 lists the fear count as 4,581; these numbers must be reconciled.
- [§3.1] The data collection is described as occurring on January 26, 2022, yet the corpus extends to December 31, 2022; clarify whether the collection was retrospective through the academic API or whether multiple collection rounds were performed.
- [§3.2] The normalization rules in Table 1 are expressed as R code, while the emoji conversion is described as a Python program; the paper would benefit from a unified description of the preprocessing pipeline, including the language/toolchain used at each step.
- [§4.2, Figure 6] The confusion matrix in Figure 6 would be much clearer with explicit emotion labels on both axes; currently the reader has to infer the class ordering from the text.
- [References] The reference list uses inconsistent name spellings (e.g., "Tocoglu" vs. "Toçoglu") and inconsistent entry formatting; please unify the style and check that all cited works, including those with DOIs, are listed with complete information.
Circularity Check
No significant circularity: the evaluation accuracy is computed on held-out TREMO labels, and the target-corpus emotion counts are model outputs, not derivations from the paper's inputs.
full rationale
The paper's central technical claim is the held-out accuracy of a fine-tuned BERTurk on the TREMO test split. Equation (2) computes 92.62% directly from the confusion-matrix diagonal (3338/3605), and the accuracy is therefore a measured performance on independent TREMO labels rather than a quantity defined by the model. The subsequent 'sessiz istila' emotion percentages (Table 4) are produced by applying the trained model to new tweets and are not used to define or fit the model, so they do not reduce to the training input by construction. The absence of human-validated labels on the target political discourse is a real external-validity and domain-transfer limitation, explicitly acknowledged in part by Section 5.1's admission that the model 'struggled to differentiate between anger and fear,' but this is not circular reasoning. The only self-citation, the first author's Turkish-Emoji resource [39], is a preprocessing lexicon rather than a load-bearing theoretical premise, and no uniqueness theorem or fitted parameter is renamed as a prediction. Internal inconsistencies (92.62% vs. 92.47%; confusion-matrix total 3605 vs. the expected ~1801 test instances; data-collection dates) are correctness and reproducibility concerns that do not fit any circularity pattern.
Assumptions & free parameters
free parameters (4)
- Confidence threshold for emotion labels =
0.6
- Train/test split ratio =
90/10
- Per-class balancing cap =
3,003 sentences per emotion
- Number of fine-tuning epochs =
3
assumptions (5)
- domain assumption Ekman's six basic emotions are an adequate label scheme for Turkish political discourse.
- domain assumption TREMO's ValidatedEmotion labels are accurate ground truth for emotion classification.
- domain assumption BERTurk's pretrained representations transfer to Turkish social media text, and no stop-word removal is needed.
- ad hoc to paper The fine-tuned TREMO classifier generalizes to the 'sessiz istila' X corpus without domain adaptation.
- domain assumption Tweets containing the keyword 'sessiz istila' are a representative sample of the relevant public discourse.
Cite this review
Pith. "Pith review of Emotion Recognition for Low-Resource Turkish: Fine-Tuning BERTurk on TREMO and Testing on Xenophobic Political Discourse." pith.science (2026). https://pith.science/paper/YUI6FJLW
@misc{pith2026250512160,
author = {Pith},
title = {Pith review of: Emotion Recognition for Low-Resource Turkish: Fine-Tuning BERTurk on TREMO and Testing on Xenophobic Political Discourse},
year = {2026},
howpublished = {\url{https://pith.science/paper/YUI6FJLW}},
note = {Machine review of arXiv:2505.12160}
}
read the original abstract
Social media platforms like X (formerly Twitter) play a crucial role in shaping public discourse and societal norms. This study examines the term Sessiz Istila (Silent Invasion) on Turkish social media, highlighting the rise of anti-refugee sentiment amidst the Syrian refugee influx. Using BERTurk and the TREMO dataset, we developed an advanced Emotion Recognition Model (ERM) tailored for Turkish, achieving 92.62% accuracy in categorizing emotions such as happiness, fear, anger, sadness, disgust, and surprise. By applying this model to large-scale X data, the study uncovers emotional nuances in Turkish discourse, contributing to computational social science by advancing sentiment analysis in underrepresented languages and enhancing our understanding of global digital discourse and the unique linguistic challenges of Turkish. The findings underscore the transformative potential of localized NLP tools, with our ERM model offering practical applications for real-time sentiment analysis in Turkish-language contexts. By addressing critical areas, including marketing, public relations, and crisis management, these models facilitate improved decision-making through timely and accurate sentiment tracking. This highlights the significance of advancing research that accounts for regional and linguistic nuances.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
A., Nunoo-Mensah, H., and Chen, W
Acheampong, F. A., Nunoo-Mensah, H., and Chen, W. Transformer models for text-based emotion detec- tion: a review of BERT-based approaches.Artificial Intelligence Review, 54(8), pages 5789–5829, 2021. DOI: 10.1007/s10462-021-09958-2
-
[2]
Emotion-enriched word embeddings for Turkish.Expert Systems with Applications., page 225, 2023
Aka Uymaz, H., and Kumova Metin, S. Emotion-enriched word embeddings for Turkish.Expert Systems with Applications., page 225, 2023. DOI: 10.1016/j.eswa.2023.120011
-
[3]
Emotion Analysis on Turkish Texts: A Systematic Mapping Study
Altmay, G., and Turhan, C. Emotion Analysis on Turkish Texts: A Systematic Mapping Study. InInternational Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT)., pages 523–527, 2022. DOI: 10.1109/ISMSIT56059.2022.9932858
arXiv 2022
-
[4]
Alzahrani, E., and Jololian, L. How Different Text-Preprocessing Techniques using the Bert Model Affect the Gender Profiling of Authors.ArXiv preprint., pages 01–08, 2021. DOI: 10.5121/csit.2021.111501
-
[5]
Populist Nationalism and Anti-refugee Sentiment in Turkey: The Case of the Victory Party
Bahadır Türk, H. Populist Nationalism and Anti-refugee Sentiment in Turkey: The Case of the Victory Party. Nationalism and Ethnic Politics., 2023. DOI: 10.1080/13537113.2023.2248792
-
[6]
(2023, February 3).Twitter Ends Its Free API: Here’s Who Will Be Affected
Barnes, J. (2023, February 3).Twitter Ends Its Free API: Here’s Who Will Be Affected. Forbes. forbes.com article
work page 2023
-
[7]
Barrie, C., and Ho, J. academictwitteR: an R package to access the Twitter Academic Research Product Track v2 API endpoint.Journal of Open Source Software., 6(62), page 3272, 2021. DOI: 10.21105/joss.03272
-
[8]
Emotion analysis on Turkish texts
Boynukalin, Z., and Karagoz, P. Emotion analysis on Turkish texts. InE. Gelenbe and R. Lent (Eds.), Information Sciences and Systems 2013. Lecture Notes in Electrical Engineering.: Vol. 264 (LNEE)., pages 159–168, Springer, Cham., 2013. DOI: 10.1007/978-3-319-01604-7_16
Show all 43 references
-
[9]
Buckley, M., Chen, J., Durkheimer, M., Krohnstad, H., Liu, H., Mcdevitt, L., Pirani, A., Pollack, J., Wang, R., Bazerman, M., Beshears, J., Haidt, J., Heath, C., Kashima, Y ., Keltner, D., Peters, K., Schaller, M., Small, D., Stephen, A., and Milkman, K. L. What Makes Online C...
2012 doi
- [10]
-
[11]
Dehghan, S., and Yanikoglu, B. (2024). Evaluating ChatGPT’s Ability to Detect Hate Speech in Turkish Tweets. InA. Hürriyeto˘ glu, G. Uludo˘ gan, S. Thapa, and H. Tanev (Eds.), Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-polit...
2024
-
[12]
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.ArXiv preprint., 2019
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.ArXiv preprint., 2019. DOI: 10.48550/arXiv.1810.04805
-
[13]
Deep Learning Based Sentiment Analysis and Text Summarization in Social Networks
Do˘gan, E., and Kaya, B. Deep Learning Based Sentiment Analysis and Text Summarization in Social Networks. In2019 International Artificial Intelligence and Data Processing Symposium (IDAP)., pages 1–6, 2019. DOI: 10.1109/IDAP.2019.8875879
2019
-
[14]
Basic Emotions
Ekman, P. Basic Emotions. Handbook of cognition and emotion (T. Dalgleish and M. J. Power, Eds.; 1st ed.). John Wiley and Sons., (1999). 15 September 11, 2025
1999
-
[15]
Unicode Code Points., 2024
EmojiTerra. Unicode Code Points., 2024. https://emojiterra.com/blog/code-points/
2024
-
[16]
Populism, Disinformation andSurveillance
Erbaysal Filibeli, T., and Öneren Özbek, M. Populism, Disinformation andSurveillance. Mapping Lies in the Global Media Sphere (1st ed.)., Routledge, 2023
2023
-
[17]
Han, W., Wang, Y ., and Scott, M. Social media activation of pro-environmental personal norms: an exploration of informational, normative and emotional linkages to personal norm activation.Journal of Travel and Tourism Marketing., 38(6), pages 568–581, 2021. DOI: 10.1080/10548...
2021
-
[18]
E., and Samavi, R
Heydarian, M., Doyle, T. E., and Samavi, R. MLCM: Multi-Label Confusion Matrix.IEEE Access., 10, pages 19083–19095, 2022. DOI: 10.1109/ACCESS.2022.3151048
2022
-
[19]
S., Jim, J
Islam, T., Hafiz, M. S., Jim, J. R., Kabir, M. M., and Mridha, M. F. A systematic review of deep learning data augmentation in medical imaging: Recent advances and future research directions.Healthcare Analytics., page 5,
-
[20]
Turkish Emotional Word Norms for Arousal, Valence, and Dis- crete Emotion Categories.Psychological Reports., 124(1), pages 188–209, 2021
Kapucu, A., Kılıç, A., Özkılıç, Y ., and Sarıbaz, B. Turkish Emotional Word Norms for Arousal, Valence, and Dis- crete Emotion Categories.Psychological Reports., 124(1), pages 188–209, 2021. DOI: 10.1177/0033294118814722
2021 doi
-
[21]
Discovering Public Opinions by Performing Sentimental Analysis on Real Time Twitter Data
Kavitha, G., Saveen, B., and Imtiaz, N. Discovering Public Opinions by Performing Sentimental Analysis on Real Time Twitter Data. In2018 International Conference on Circuits and Systems in Digital Enterprise Technology (ICCSDET)., pages 1–4, 2019. DOI: 10.1109/ICCSDET.2018.8821105
2019
-
[23]
H., and Ozgur, A
Masarifoglu, M., Tigrak, U., Hakyemez, S., Gul, G., Bozan, E., Buyuklu, A. H., and Ozgur, A. Sentiment analysis of customer comments in banking using BERT-based approaches. InSIU 2021 - 29th IEEE Conference on Signal Processing and Communications Applications, Proceedings., 20...
2021
-
[24]
Muraina, I. O. Ideal dataset splitting ratios in machine learning algorithms: General concerns for data scientists and data analysts. In7th International Mardin Artuklu Scientific Research Conference., pages 496–504, 2022. researchgate.net publication
2022
- [25]
-
[26]
Onan, A., and Balbal, K. F. Improving Turkish Text Sentiment Classification Through Task-Specific and Universal Transformations: An Ensemble Data Augmentation Approach.IEEE Access., 12, pages 4413–4458, 2024. DOI: 0.1109/ACCESS.2024.3349971
2024
-
[27]
How to get angry online
Palmer, A. How to get angry online. . . properly: Creating online deliberative systems that harness political anger’s power and mitigate its costs.Politics, Philosophy and Economics., 23(3), pages 295–318, 2024. DOI: 10.1177/1470594X231222539
2024 doi
-
[28]
Multilingual evaluation of pre-processing for BERT-based sentiment analysis of tweets.Expert Systems with Applications., page 181, 2021
Pota, M., Ventura, M., Fujita, H., and Esposito, M. Multilingual evaluation of pre-processing for BERT-based sentiment analysis of tweets.Expert Systems with Applications., page 181, 2021. DOI: 10.1016/j.eswa.2021.115119
2021
-
[29]
Data Splitting
Reitermanová, Z. Data Splitting. InJ. + Šafránková and J. Pavl˚ u (Eds.), Proceedings of the 19th Annual Conference of Doctoral Students - WDS 2010., pages 31–36, Matfyzpress, 2010
2010
-
[30]
BERTurk - BERT models for Turkish (1.0.0).Zenodo., 2020
Schweter, S. BERTurk - BERT models for Turkish (1.0.0).Zenodo., 2020. DOI: 10.5281/zenodo.3770923
2020 doi
-
[31]
Emotions and information diffusion in social media - Sentiment of mi- croblogs and sharing behavior.Journal of Management Information Systems., 29(4), pages 217–248, 2013
Stieglitz, S., and Dang-Xuan, L. Emotions and information diffusion in social media - Sentiment of mi- croblogs and sharing behavior.Journal of Management Information Systems., 29(4), pages 217–248, 2013. DOI: 10.2753/MIS0742-1222290408
2013 doi
-
[32]
Subarkah, P., Rozaq, H. A. A., Arsi, P., Sholikhatin, S. A., Riyanto, R., and Marcos, H. Implementation of Text Mining to Detect Emotions of Fuel Price Increase Using BERT-LSTM Method.GAZI UNIVERSITY JOURNAL OF SCIENCE., 2024. DOI: 10.35378/gujs.1424742
2024 doi
-
[33]
Interpreting the public sentiment variations on Twitter.IEEE Transactions on Knowledge and Data Engineering., 26(5), pages 1158–1170, 2014
Tan, S., Li, Y ., Sun, H., Guan, Z., Yan, X., Bu, J., Chen, C., and He, X. Interpreting the public sentiment variations on Twitter.IEEE Transactions on Knowledge and Data Engineering., 26(5), pages 1158–1170, 2014. DOI: 10.1109/TKDE.2013.116
2014 doi
-
[34]
A., and Alpkocak, A
Tocoglu, M. A., and Alpkocak, A. Emotion extraction from turkish text. InProceedings - 2014 European Network Intelligence Conference, ENIC 2014., pages 130–133, 2014. DOI: 10.1109/ENIC.2014.17
2014 doi
-
[35]
A., and Alpkocak, A
Tocoglu, M. A., and Alpkocak, A. TREMO: A dataset for emotion analysis in Turkish.Journal of Information Science., 44(6), pages 848–860, 2018. DOI: 10.1177/0165551518761014 16 September 11, 2025
2018 doi
-
[36]
A., and Alpkoçak, A
Toçoglu, M. A., and Alpkoçak, A. Lexicon-based emotion analysis in Turkish.Turkish Journal of Electrical Engineering and Computer Sciences., 27(2), pages 1213–1227, 2019. DOI: 10.3906/elk-1807-41
2019 doi
-
[37]
A., Ozturkmenoglu, O., and Alpkocak, A
Tocoglu, M. A., Ozturkmenoglu, O., and Alpkocak, A. Emotion Analysis from Turkish Tweets Using Deep Neural Networks.IEEE Access., 7, pages 183061–183069, 2019. DOI: 10.1109/ACCESS.2019.2960113
2019
-
[38]
C., and Özbek, S
Türkmen, H., Dikenelli, O., Eraslan, C., Çallı, M. C., and Özbek, S. S. Harnessing the Power of BERT in the Turkish Clinical Domain: Pretraining Approaches for Limited Data Scenarios.ArXiv preprint., 2023. http://arxiv.org/abs/2305.03788
2023 arXiv
-
[39]
Turkish-Emoji: Detailed list of emoji symbols and unicode code points in Turkish, 2024
Wicaksono, D. Turkish-Emoji: Detailed list of emoji symbols and unicode code points in Turkish, 2024. https://github.com/dwicak/Turkish-Emoji
2024
-
[40]
Fine-tuning Transformer-based Encoder for Turkish Language Understanding Tasks.ArXiv preprint.,
Yildirim, S. Fine-tuning Transformer-based Encoder for Turkish Language Understanding Tasks.ArXiv preprint.,
-
[41]
Yucalar, F. Developing an Advanced Software Requirements Classification Model Using BERT: An Empir- ical Evaluation Study on Newly Generated Turkish Data.Applied Sciences., 13(20), page 11127, 2023. DOI: 10.3390/app132011127
2023 doi
- [42]
- [44]
-
[2024]
DOI: 10.1016/j.health.2024.100340
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.