REVIEW 2 major objections 6 minor 29 references
JNLP at SemEval-2025 Task 11: Cross-Lingual Multi-Label Emotion Detection Using Generative Models
T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Instruction-tuning a 32-billion-parameter generative model with a pairwise, one-emotion-at-a-time prompt and training on all languages together places it in the top four for ten languages in SemEval-2025 Task 11 emotion detection, first…
desk verdict A credible SemEval-2025 system description whose leaderboard results stand, but the printed prompts omit one of the six emotion labels, so the paper needs a correction before it can serve as a reproducible recipe. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the pairwise instruction template: a system prompt that names the emotion label set, a user turn of the form 'Given the sentence: ..., is the emotion anger expressed in it?' (Track A) or 'what is the intensity of the emotion fear?' (Track B), and an assistant answer of 'yes'/'no' or 'low'/'moderate'/'high'. LoRA fine-tuning of Qwen2.5-32B under a causal language-modeling objective on all languages concatenated is the training mechanism, and the base template—asking for all labels at once—is the control that isolates the benefit of per-emotion decomposition.
What would settle it
Run the exact Table 5 and Table 6 prompts—with the five-emotion label set printed in the paper—through a LoRA-fine-tuned Qwen2.5-32B on the BRIGHTER development set and compare macro-F1 and Pearson values with the development columns of Tables 3 and 4. A second check: if the model ever emits 'disgust' despite the prompt not listing it, the printed template cannot be the deployed prompt, and the reported numbers are not reproducible from the paper alone.
Extended reading notes
Core claim
The central discovery is that multi-label emotion detection in many languages can be reformulated as a set of per-emotion generation questions and solved by a single instruction-tuned LLM: for Track A, ask 'is emotion ye expressed?' and read 'yes' or 'no'; for Track B, ask for the intensity of ye and read 'low', 'moderate', or 'high'. With Qwen2.5-32B fine-tuned under LoRA on all training languages jointly, this pairwise strategy achieves the paper's reported rankings, while a base strategy that maps a sentence directly to all labels is competitive only where most sentences have zero or one emotion. Mixed-language training consistently improves over separate per-language models, and the generative model generally outperforms a fine-tuned XLM-RoBERTa encoder. The authors thus claim the pairwise decomposition, not model size or language-specific tuning, is the ingredient that carries the generalization.
Load-bearing premise
The reported rankings are only reproducible if the printed instruction templates are the exact prompts that produced the submitted predictions, and the paper's own tables show a mismatch—the printed label set has five emotions while outputs and evaluation include 'disgust' as a sixth.
Editorial extensions
If this is right
- A single mixed-language instruction-tuned model can replace per-language pipelines for multilingual emotion detection, which matters most for low-resource languages where separate training data are thin.
- Teams facing multi-label emotion data should default to the pairwise question format when sentences regularly carry several emotions, and to the base format when labels are sparse.
- The same pairwise prompt structure covers both label detection and intensity prediction, so one fine-tuned model can serve both tasks without architectural changes.
- Instruction-tuned generative models of the Qwen2.5-32B class are a viable alternative to fine-tuned encoders for this task, with the main trade-off being higher inference cost.
Reading between the lines
- A likely reason mixed-language training helps is that it balances the emotion-label distribution across languages; a testable corollary is that per-language models should catch up when trained on resampled or augmented data that equalizes label frequencies.
- The five-emotion prompt printed in Table 1 cannot explain the 'disgust' rows in Figure 3 and Table 2, so the deployed prompt almost certainly listed six emotions; if so, the paper's template is illustrative rather than the exact submission prompt.
- The authors' suggested logit-aggregation variant is an immediate cheap improvement: instead of decoding 'yes' or 'no', sum the logits of both tokens and threshold the softmax probability, which should yield calibrated per-language thresholds.
- The pairwise recipe should transfer to other multi-label text-generation problems, such as aspect-based sentiment or emotion-in-conversation, because the conditioning mechanism (one label at a time) is independent of the label inventory.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes a system for SemEval-2025 Task 11, covering Track A (multi-label emotion detection across 28 languages in the BRIGHTER dataset) and Track B (emotion intensity prediction across 11 languages). Two architectures are compared: a fine-tuned BERT-based classifier and instruction-tuned generative LLMs (Llama-2 and Qwen-2.5), under two multi-label strategies: a base strategy that generates all emotion labels at once, and a pairwise strategy that considers each emotion separately. The authors also compare per-language training against mixed-language training. The reported results are external leaderboard scores: top-four rankings in 10 Track A languages including first place in Hindi, and top-five rankings in 7 Track B languages. The paper additionally analyzes when pairwise versus base strategies help, the effect of label imbalance, and the role of mixed-language training.
Significance. If the reported leaderboard scores are taken at face value, the paper provides a credible and simple competitive recipe for multilingual multi-label emotion detection: instruction-tuned Qwen-32B with pairwise prompting performed strongly across many languages, and the code is publicly available. The central empirical claim rests on an external benchmark with no fitted constants, so circularity is not a concern. The main weaknesses are internal-consistency problems in the system description and two overgeneralized auxiliary findings; these are fixable and do not undermine the external leaderboard evidence.
major comments (2)
- [Table 1; Appendix A (§3.2)] The instruction templates printed in Table 1 and in Appendix A (Tables 5 and 6) define the emotion label set as {anger, fear, joy, sadness, surprise}, omitting 'disgust'. This contradicts Table 2, whose Track A output example is 'disgust, sadness'; Figure 3, which reports performance for 'disgust'; and Section 5.3, which discusses 'disgust' as a class. Since the official BRIGHTER label set includes disgust, the reader cannot tell from the text whether the submitted runs used the printed prompt (in which case the model would have to ignore the system instruction to produce disgust) or a different prompt that included disgust (in which case the paper does not describe the actual submitted system). Please correct the label set in all templates and state explicitly the exact prompts used for the submitted Track A and Track B runs, including how 'disgust' was handled.
- [§5.3, Tables 3–4] The claim that 'mixed-language training... demonstrates superior performance compared to training separate models' is not supported for Track A by the data in Table 3. The only mixed-language Track A row reported (Qwen-32b base mixed langs) has average 0.5933, identical to the Qwen-32b pairwise separated langs average of 0.5933, and no pairwise-mixed Track A result is shown; in contrast, Table 4 does show pairwise mixed (0.7363) beating pairwise separated (0.7234) in Track B. Please either report the pairwise-mixed Track A result or qualify this claim to Track B only.
minor comments (6)
- [§5.1, Table 3] The statement that 'in low-resource languages, LLMs performed poorly, whereas the RoBERTa-based approach yielded better results' is contradicted by several rows in Table 3: for afr, arq, ibo, kin, pcm, vmw, and yor, the Qwen-32B models beat xml-roberta by a clear margin. Please qualify the claim to specific languages or report a systematic comparison.
- [§3.2, Eq. (4)] The BERT-based multi-label classifier is described as using a sigmoid output layer, but the loss is said to be cross-entropy. For multi-label classification with independent sigmoid outputs, the appropriate objective is binary cross-entropy per label; please correct this.
- [Table 3] The column header 'orm(development)' appears to be a typo for 'orm'. Please also clarify in the caption whether the reported development results are on the official development set or on the 10% internal held-out set described in Section 4.
- [§3.1, Eqs. (1)–(2)] The notation in equations (1) and (2) is ambiguous (for example, P_A({0,1} | x, y_e) mixes set notation with probability notation). Please define all symbols and describe the decoding procedure that maps generated text to predicted labels and intensity values.
- [Abstract, §5.1] The phrase 'Top 4 performance across 10 languages' is ambiguous because Section 5.1 reports a mix of 1st, 2nd, 3rd, and 4th place rankings. Please rephrase as 'ranked among the top four in 10 languages' for clarity.
- [§2] There is a grammatical error in the sentence 'their effectiveness in multilingual sentiment analysis is often limited without additional fine-tuning is performed'; it should read '...without additional fine-tuning being performed'.
Circularity Check
No circularity: reported results are leaderboard scores on the official SemEval-2025 Task 11 test set, produced by standard fine-tuning; the derivation contains no fitted constants, no prediction that is a fitted input in disguise, and no load-bearing self-citation.
full rationale
The paper's central claims are empirical leaderboard rankings (Track A top 4 in 10 languages, 1st in Hindi; Track B top 5 in 7 languages) on the external SemEval-2025 Task 11 benchmark. The method (Section 3) is described as standard BERT fine-tuning and instruction-tuned LLM generation with LoRA, trained on the official BRIGHTER training data and evaluated on the held-out development and test sets. There is no equation that defines a quantity in terms of the target output, and no parameter fitted to the test set that is later renamed as a prediction. The pairwise/base strategies are presented as modeling choices and are compared empirically rather than derived from the results they supposedly predict. The only self-citation, Xue et al. (2024), is invoked as general inspiration ('building upon recent SOTA methods in the field of emotion recognition') and is not used to justify the reported rankings or to exclude alternative methods; the actual training recipe is standard instruction fine-tuning (Chung et al., 2022; Hu et al., 2022). Therefore the self-citation is not load-bearing. One descriptive inconsistency exists: Table 1's prompt template restricts the emotion set to {anger, fear, joy, sadness, surprise}, while Table 2 and Figure 3 include 'disgust', and the official task label set has six emotions. This is a reproducibility/correctness concern about whether the printed prompts match the deployed system, not a circularity of derivation; it does not make any claim equivalent to its input. Under the hard rules, no specific reduction can be exhibited, so the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- learning rate =
3e-4
- training epochs =
5-6
- LoRA configuration =
not reported
assumptions (3)
- domain assumption The official BRIGHTER and SemEval test labels are the ground truth, and the reported test scores are official results rather than recomputed values.
- domain assumption Generated label text from the LLM was mapped unambiguously to the predefined label set without unstated decoding rules.
- domain assumption The printed instruction prompt is the prompt actually used for the submitted predictions.
Cite this review
Pith. "Pith review of JNLP at SemEval-2025 Task 11: Cross-Lingual Multi-Label Emotion Detection Using Generative Models." pith.science (2026). https://pith.science/paper/3TXGF5UN
@misc{pith2026250513244,
author = {Pith},
title = {Pith review of: JNLP at SemEval-2025 Task 11: Cross-Lingual Multi-Label Emotion Detection Using Generative Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/3TXGF5UN}},
note = {Machine review of arXiv:2505.13244}
}
read the original abstract
With the rapid advancement of global digitalization, users from different countries increasingly rely on social media for information exchange. In this context, multilingual multi-label emotion detection has emerged as a critical research area. This study addresses SemEval-2025 Task 11: Bridging the Gap in Text-Based Emotion Detection. Our paper focuses on two sub-tracks of this task: (1) Track A: Multi-label emotion detection, and (2) Track B: Emotion intensity. To tackle multilingual challenges, we leverage pre-trained multilingual models and focus on two architectures: (1) a fine-tuned BERT-based classification model and (2) an instruction-tuned generative LLM. Additionally, we propose two methods for handling multi-label classification: the base method, which maps an input directly to all its corresponding emotion labels, and the pairwise method, which models the relationship between the input text and each emotion category individually. Experimental results demonstrate the strong generalization ability of our approach in multilingual emotion recognition. In Track A, our method achieved Top 4 performance across 10 languages, ranking 1st in Hindi. In Track B, our approach also secured Top 5 performance in 7 languages, highlighting its simplicity and effectiveness\footnote{Our code is available at https://github.com/yingjie7/mlingual_multilabel_emo_detection.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Tadesse Destaw Belay, Israel Abebe Azime, Abinew Ali Ayele, Grigori Sidorov, Dietrich Klakow, Philip Slusallek, Olga Kolesnikova, and Seid Muhie Yimam. 2025. https://aclanthology.org/2025.coling-main.237/ Evaluating the capabilities of large language models for multi-label emotion understanding . In Proceedings of the 31st International Conference on Comp...
2025
-
[4]
Prerna Chikersal, Soujanya Poria, and Erik Cambria. 2015. Sentu: sentiment analysis of tweets by combining a rule-based classifier with supervised learning. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pages 647--651
work page 2015
-
[5]
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H...
arXiv 2022
-
[6]
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...
-
[7]
Pankaj Dadure, Ananya Dixit, Kunal Tewatia, Nandini Paliwal, and Anshika Malla. 2025. https://aclanthology.org/2025.abjadnlp-1.10/ Sentiment analysis of A rabic tweets using large language models . In Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pages 88--94, Abu Dhabi, UAE. Association for Computational Linguistics
work page 2025
-
[8]
Jiawen Deng and Fuji Ren. 2020. Multi-label emotion detection via emotion-specified feature extraction and emotion correlation learning. IEEE Transactions on Affective Computing, 14(1):475--486
work page 2020
Show all 29 references
-
[9]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...
2019 doi
-
[10]
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations
2022
-
[11]
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411--4421. PMLR
2020
-
[12]
Upma Kumari, Arvind K Sharma, and Dinesh Soni. 2017. Sentiment analysis of smart phone product review using svm classification technique. In 2017 International conference on energy, communication, data analytics and soft computing (ICECDS), pages 1469--1474. IEEE
2017
-
[13]
Zheng Li, Ying Wei, Yu Zhang, and Qiang Yang. 2018. https://doi.org/10.1609/aaai.v32i1.12055 Hierarchical attention transfer network for cross-domain sentiment classification . Proceedings of the AAAI Conference on Artificial Intelligence, 32(1)
2018 doi
-
[14]
Lany Laguna Maceda, Jennifer Laraya Llovido, Miles Biago Artiaga, and Mideth Balawiswis Abisado. 2024. https://doi.org/10.1145/3639233.3639353 Classifying sentiments on social media texts: A gpt-4 preliminary study . In Proceedings of the 2023 7th International Conference on N...
2024
-
[15]
Shervin Minaee, Elham Azimi, and AmirAli Abdolrashidi. 2019. Deep-sentiment: Sentiment analysis using ensemble of cnn and bi-lstm models. arXiv preprint arXiv:1904.04206
2019 arXiv
-
[16]
Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nirmal Surange, Daniela Teodorescu, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino D. M. A. Ali, Ilseyar Alimova, Vladimir...
2025 arXiv
-
[17]
Mohammad
Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Seid Muhie Yimam, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine De Kock, Tadesse Destaw Belay, Ibrahim Said Ahmad, Nirmal Surange, Daniela Teodorescu, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino...
2025
-
[18]
Muhammad Mujahid, Khadija Kanwal, Furqan Rustam, Wajdi Aljedaani, and Imran Ashraf. 2023. https://doi.org/10.1145/3605889 Arabic chatgpt tweets classification using roberta and bert ensemble model . ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(8)
2023 doi
-
[19]
Pansy Nandwani and Rupali Verma. 2021. A review on sentiment analysis and emotion detection from text. Social network analysis and mining, 11(1):81
2021
-
[20]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67
2020
-
[21]
Shivam Sharma, Rahul Aggarwal, and M. Kumar. 2023. https://api.semanticscholar.org/CorpusID:259217117 Mining twitter for insights into chatgpt sentiment: A machine learning approach . 2023 International Conference on Distributed Computing and Electrical Circuits and Electronic...
2023
-
[22]
Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cant \'o n Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes,...
2023 arXiv
-
[23]
Abinash Tripathy, Ankit Agrawal, and Santanu Kumar Rath. 2016. Classification of sentiment reviews using n-gram machine learning approach. Expert Systems with Applications, 57:117--126
2016
-
[24]
A Upadhye. 2024. Sentiment analysis using large language models: Methodologies, applications, and challenges. Int. J. Comput. Appl, 186:30--34
2024
-
[25]
Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. 2022. A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7):5731--5780
2022
-
[26]
Jieying Xue, Minh-Phuong Nguyen, Blake Matheny, and Le-Minh Nguyen. 2024. Bioserc: Integrating biography speakers supported by llms for erc tasks. In International Conference on Artificial Neural Networks, pages 277--292. Springer
2024
-
[27]
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...
2021 doi
-
[28]
Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.491 Don`t trust C hat GPT when your question is not in E nglish: A study of multilingual abilities and types of LLM s . In Proceedings of the 2023 Conference...
2023 doi
-
[29]
Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2016. Attention-based lstm network for cross-lingual sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 247--256
2016
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.