Pith. sign in

REVIEW 5 major objections 5 minor 79 references

Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Current large language models still cannot reliably identify all emotions expressed in a text, even in English, and the gap is much larger for four Ethiopian languages; the paper supports this with a new benchmark called EthioEmo.

desk verdict New dataset, real contribution, but the headline high-vs-low-resource gap rests on a filtered sample; treat the benchmark as controlled, not naturalistic. read the letter →

arxiv 2412.17837 v2 pith:M5NDXTVN submitted 2024-12-17 cs.CL

Tadesse Destaw Belay , Israel Abebe Azime , Abinew Ali Ayele , Grigori Sidorov , Dietrich Klakow , Philipp Slusallek , Olga Kolesnikova , Seid Muhie Yimam This is my paper · ORCID
classification cs.CL
keywords multi-labelemotionclassificationEthioEmoEthiopianlanguageslargelanguagemodelslow-resourceNLPbenchmarkin-contextlearningfine-tuningversuszero-shotprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces EthioEmo, a multi-label emotion classification dataset for Amharic, Afan Oromo, Somali, and Tigrinya, and uses it to test how well large language models recognize the emotions people actually express. Its central finding is that accurate multi-label emotion classification is still not solved even for English, and that the performance gap between high-resource and low-resource languages remains wide. On the EthioEmo test sets, fine-tuned encoder-only models clearly outperform zero-shot and few-shot LLMs, which the paper reads as evidence that emotion understanding has not emerged from scale alone. A sympathetic reader should care because the dataset and its baselines give the field a concrete instrument for tracking whether future models improve.

What carries the argument

The load-bearing object is EthioEmo itself, a multi-label dataset built by filtering millions of collected posts and news headlines through per-language emotion keyword lexicons, then annotating each selected instance with any subset of six Ekman emotion classes plus neutral. The evaluation protocol is the other half: fine-tuned encoder-only models are compared with zero- and few-shot decoder-only models using log-likelihood scoring, and a translate-test variant converts the Ethiopian test sets into English to separate task difficulty from language understanding. The multi-label agreement scores reported for the annotation are what make the performance differences interpretable as model failures rather than noise in the gold labels.

What would settle it

Take a fresh sample of social media posts in the four languages without applying the emotion-keyword filter, annotate it with the same protocol, and rerun the same models. If zero-shot LLM performance on the unfiltered sample is comparable to or better than the fine-tuned baselines, the claim that LLMs are insufficient would not generalize to natural text; if performance collapses, the benchmark's keyword-based selection is driving the apparent gap.

Watch

Extended reading notes

Core claim

The paper's central claim is that current large language models are insufficient for multi-label emotion understanding, and this insufficiency is most visible in low-resource languages. It supports this with EthioEmo, a new benchmark of 23,321 annotated instances across four Ethiopian languages, evaluated alongside the English SemEval-2018 Task 1 set. The best fine-tuned encoder-only models reach weighted F1 scores of roughly 58-73% on Ethiopian languages, while zero-shot open LLMs stay around 25-49% and a commercial model averages 52%. Translating the Ethiopian test sets into English does not close the gap, so the paper concludes the difficulty is not mainly a translation problem: it is a problem of how emotion is modeled and how well languages are represented in pretraining.

Load-bearing premise

The paper's conclusion depends on the assumption that filtering texts by emotion keywords yields a sample that represents how emotion is actually expressed in these languages, rather than a sample in which emotion words are unusually easy to spot.

Editorial extensions

If this is right

  • Fine-tuned encoder-only models remain the stronger practical baseline for low-resource multi-label emotion classification, at least until LLM prompting improves substantially.
  • Because translating the test set into English does not erase the gap, progress on emotion understanding cannot be achieved by translation alone; it requires language-specific training data or better multilingual pretraining.
  • Showing a few examples helps most LLMs, but the gains are inconsistent, so in-context learning is not a reliable substitute for fine-tuning on this task.
  • EthioEmo gives future work a public benchmark with per-source and per-language breakdowns, making it possible to measure whether a new model is actually better at emotion or just better at English.
  • Even English multi-label emotion classification remains unsolved, so emotion understanding should be treated as an open evaluation target rather than a solved capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: because the corpus was filtered by emotion keywords, EthioEmo may overrepresent explicit emotion vocabulary; true performance on implicit, sarcastic, or context-dependent emotion could be lower than the benchmark numbers suggest.
  • My inference: the pattern across model types suggests that what matters most is whether the target language appeared in pretraining, not model size or reasoning ability; a testable extension would be to compare models pretrained on the same data but with different parameter counts.
  • My inference: the same evaluation design could be exported to other low-resource language families to see whether the high-resource/low-resource emotion gap is universal or specific to these four languages.
  • My inference: the failure modes reported, with generative models over-predicting emotion labels and discriminative models collapsing to single labels, point toward calibration-aware metrics as a next step for multi-label emotion evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces EthioEmo, a new multi-label emotion classification dataset for Amharic, Afan Oromo, Somali, and Tigrinya, using Ekman's six basic emotions plus a neutral class. Texts were collected from Twitter/X, Facebook, YouTube, and news headlines, filtered via emotion lexicons, and annotated by at least three native speakers per instance, with agreement metrics reported. The authors evaluate fine-tuned Afri-centric encoder-only models, zero-shot and few-shot decoder-only models, a translated-to-English test condition, and prompt-sensitivity variants. They report that even high-resource English emotion classification is not solved, that there is a large gap between high-resource and low-resource languages, and that fine-tuned encoder-only models remain strong baselines relative to zero-shot/few-shot LLMs.

Significance. If the central claims hold, EthioEmo is a valuable public resource for an under-resourced area: it adds four Ethiopian languages to multi-label emotion research, uses multiple data sources and a detailed annotation protocol with moderate agreement, and releases lexicons and annotation guidelines. The paper also provides a broad comparison across encoder-only, encoder-decoder, and decoder-only models and includes a translate-test setting that separates language understanding from task difficulty. The main significance lies in the benchmark itself and the reproducible baseline numbers, which will likely be used by later work. However, the paper's headline claims about LLM insufficiency and the high- versus low-resource gap depend on two design choices that are not fully defended: lexicon-filtered sampling and a cross-dataset comparison with an underspecified English benchmark.

major comments (5)
  1. [Section 3.1, 3.3, Table 3; Section 6] The dataset is built by first filtering candidate posts with emotion keyword lexicons and then excluding neutral-only instances from all experiments, as stated in Section 3.1 and Table 3. The paper's own error analysis in Section 6 notes that annotators sometimes depend only on emotion keywords and that models often classify from keywords/emojis rather than context. As a result, EthioEmo is a conditional sample of lexicon-positive, non-neutral posts, not a random sample of how emotion is expressed in natural social-media text in these languages. This does not invalidate the benchmark, but it does mean the abstract's unqualified statement that 'accurate multi-label emotion classification is still insufficient' for natural text in these languages is not established. The authors should either restrict the claim to the lexicon-filtered, non-neutral distribution or add an experiment on a random, unfiltered sample to quantify the selection bias.
  2. [Section 4, Table 3] The English comparison used as the 'high-resource anchor' is underspecified: the paper refers to 'an additional English multi-label emotion dataset from SemEval 2018 Task 1' but does not state which subtask, which label inventory, or which preprocessing steps were used. If, as in the standard SemEval-2018 Task 1 emotion-classification subtask, the English data use a larger and different label set than the six Ekman emotions, then the reported 'large gap between high-resource and low-resource languages' is confounded with the number and type of classes, the annotation protocol, and the data source. The authors should specify the exact English dataset and, ideally, evaluate English under the same six-emotion label scheme to make the cross-language gap interpretable.
  3. [Section 5.2, Table 5] The text says that 'Encoder-only models still have an advantage over the recently popular open-source decoder-only models for low-resource languages' and that 'LLMs under-perform compared to encoder-only models.' This is contradicted by the zero-shot numbers in Table 5: averaging the columns, Gemma-1.1-7b-it (34.79) and LLaMA-3-8B-Instruct (34.17) both exceed the best zero-shot encoder-only average, AfroLM-active-l (30.20), and the open-weight encoder-decoder Cohere-aya-101 reaches 46.12. The statement appears to conflate zero-shot and fine-tuned encoder-only results. Please correct the claim so that it is consistent with the reported table, for example by restricting it to fine-tuned encoder-only models.
  4. [Section 4.4, 5.5, Tables 5 and 9] The LLM evaluation protocol does not specify how free-form or log-likelihood outputs are converted to a multi-label prediction. Section 4.4 mentions log-likelihood-based evaluation with lm-evaluation-harness, and the prompts in Figure 2 ask the model to 'Answer:' with applicable emotions, but the paper does not state whether labels are predicted by per-class yes/no likelihood, by joint likelihood ranking, by a threshold, or by parsing generated text with post-processing. Since the central comparisons of decoder-only and encoder-decoder models depend entirely on these numbers, the lack of an exact output-to-label mapping is a reproducibility gap. Please provide the full decoding/parsing specification and, if possible, release the evaluation scripts.
  5. [Tables 4-5, Sections 5.1-5.4] All performance numbers appear to be from single runs with no confidence intervals, bootstrap estimates, or significance tests. Some of the comparative statements rely on small differences, such as the claim that AfroXLMR-76L outperforms for Amharic (69.9 vs. 68.3 for AfroXLMR-61L in Table 4) or the per-language rankings in Table 5. Fine-tuning is stochastic and the k-shot demonstrations are randomly selected (Section 4.4), so the paper should report variance across seeds or example sets before drawing conclusions about which model is best. This is not a demand for exhaustive significance testing, but for the specific comparative claims made in Sections 5.1-5.4, at least a small number of repeated runs or bootstrap intervals would substantiate them.
minor comments (5)
  1. [Section 5.4] The text says 'All models benefit from two-shot examples compared to zero-shot tests' and then immediately states that 'this improvement is not shown in Gemma-1.1-7b-it.' This is contradictory; please rephrase to distinguish the general trend from the exception.
  2. [Table 3] The relation between the train/test/dev counts and the 'final annotated dataset statistics' (Amharic 5,891, etc.) is not explained; it is unclear how many instances were excluded as neutral-only or unresolved before the splits, and whether the split was performed before or after that exclusion. A sentence clarifying this workflow would help.
  3. [Figure 1] The y-axis ranges differ across panels (e.g., English starts near 50 while the Ethiopian languages start near 0), which makes cross-language comparisons visually misleading. Consider using a common scale.
  4. [References] There are small typographical issues in the references, such as 'Accessed Augest-07-2024' for OpenAI (2024) and 'pertaining phase' in Section 5.1 for 'pre-training phase.'
  5. [Appendix G.3] The prompt sensitivity tables are extensive, but the relationship between the per-prompt numbers and the averaged numbers in Table 5 is not explicitly stated; please note in the text that Table 5 reports the average over the three prompts.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper constructs a benchmark dataset and empirically evaluates models on it, with no fitted parameter renamed as a prediction and no derivation that reduces to its own inputs.

full rationale

The paper's central activity is dataset construction followed by standard supervised evaluation: emotion lexicons (Section 3.1) are inputs to corpus filtering and annotation, while the reported F1 scores come from held-out test splits that were annotated independently of any model prediction. No equation in the paper defines an output in terms of the quantity it claims to predict; fine-tuning encoder-only models and zero-/few-shot LLM prompting are ordinary empirical comparisons rather than fitted-input-called-prediction. The self-citations (e.g., Yimam et al. 2020, Ayele et al. 2023, Belay et al. 2021, EthioLLM) supply prior lexicons, semantic models, and pretrained backbones, but none of these is invoked as the justification for the paper's headline claim that LLMs remain insufficient for multi-label emotion classification; that claim is supported by directly measured test-set results, including an external English benchmark (SemEval-2018 Task 1). Section 6's observation that annotators sometimes depended on emotion keywords is an explicitly stated limitation about annotation behavior and about models classifying from keywords or emojis; it is a data-quality concern, not a circular reduction. The lexicon-filtered, neutral-excluded sampling may limit representativeness of natural text, but representativeness is a validity risk, not a circularity. Thus the derivation chain is self-contained, and an honest non-finding with score 0 is appropriate.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No parameters were fitted to data. The five axioms are the load-bearing assumptions of the benchmark; the first two are standard annotation practice, the third is a sampling design choice, and the last two affect the LLM evaluation protocols.

assumptions (5)
  • domain assumption Ekman's six basic emotions plus neutral form a complete label set for emotion understanding in these languages
    Section 3 introduces the label set; if important culturally specific emotions are missing, the benchmark's scope is narrower than 'emotion understanding'.
  • domain assumption Gold labels are reliable when at least two of three annotators agree on a class; unresolved disagreements are excluded
    Section 3.3 describes this rule; all model scores inherit any noise in these labels.
  • domain assumption Lexicon-filtered instances are representative of natural emotion expression
    Section 3.1 filters via emotion keywords; this is the paper's weakest sampling premise.
  • domain assumption English prompts are a fair interface for evaluating non-English texts
    Section 4.2 uses English prompts for all decoder-only models, citing prior work; prompt language can change results.
  • domain assumption NLLB-200 translation preserves the emotion content of the test set
    Section 4.3 and the Limitations section note that translation quality and cultural emotion transfer are unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding." pith.science (2026). https://pith.science/paper/M5NDXTVN

@misc{pith2026241217837,
  author       = {Pith},
  title        = {Pith review of: Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M5NDXTVN}},
  note         = {Machine review of arXiv:2412.17837}
}
read the original abstract

Large Language Models (LLMs) show promising learning and reasoning abilities. Compared to other NLP tasks, multilingual and multi-label emotion evaluation tasks are under-explored in LLMs. In this paper, we present EthioEmo, a multi-label emotion classification dataset for four Ethiopian languages, namely, Amharic (amh), Afan Oromo (orm), Somali (som), and Tigrinya (tir). We perform extensive experiments with an additional English multi-label emotion dataset from SemEval 2018 Task 1. Our evaluation includes encoder-only, encoder-decoder, and decoder-only language models. We compare zero and few-shot approaches of LLMs to fine-tuning smaller language models. The results show that accurate multi-label emotion classification is still insufficient even for high-resource languages such as English, and there is a large gap between the performance of high-resource and low-resource languages. The results also show varying performance levels depending on the language and model type. EthioEmo is available publicly to further improve the understanding of emotions in language models and how people convey emotions through various languages.

Figures

Figures reproduced from arXiv: 2412.17837 by the authors.

Figure 1
Figure 1. In-context learning (ICL) experiments with k-shots and languages. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. The three prompts used for decoder-only zero [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Backgrounds of Annotators: gender, language participated, academic qualification, and field of study. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Number of emotion labels per instance across languages [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Emotion distribution in the data sources for EthioEmo dataset [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Emotion distribution for EthioEmo dataset across languages [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Facebook comment emotions distribution across languages from the given quota [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: YouTube comment emotions distribution across languages from the given quota [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Twitter (X) post emotions distribution across languages from the given quota [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: News headlines emotions distribution across languages from the given quota [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 33 canonical work pages

  1. [1]

    David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiun Lee. 2024 a . https://aclanthology.org/2024.eacl-long.14 SIB -200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects . In Proceedings of the 18th Conference of the European Chapter of the Assoc...

    work page 2024

  2. [2]

    David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Chukwuneke, Happy Buzaaba, Blessing Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuy...

    arXiv 2024

  3. [3]

    Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D

    Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. 2024. https://arxiv.org/abs/2404.11018 Many-shot in-context learning . arXiv preprint arXiv:2404.11018

    arXiv 2024

  4. [4]

    Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. https://doi.org/10.18653/v1/N19-4010 FLAIR : An easy-to-use framework for state-of-the-art NLP . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics (Demonstrations) , pages 54--59, Minneapo...

    doi:10.18653/v1/n19-4010 2019

  5. [5]

    Iqra Ameer, Noman Ashraf, Grigori Sidorov, and Helena G \'o mez Adorno. 2020. https://doi.org/10.13053/CyS-24-3-3476 Multi-label emotion classification using content-based features in twitter . Computaci \'o n y Sistemas , 24(3):1159--1164

    work page doi:10.13053/cys-24-3-3476 2020

  6. [6]

    o l \"u c \

    Iqra Ameer, Necva B \"o l \"u c \"u , Hua Xu, and Ali Al Bataineh. 2023. https://doi.org/10.18653/v1/2023.wassa-1.56 Findings of WASSA 2023 shared task: Multi-label and multi-class emotion classification on code-mixed text messages . In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis , pages...

    doi:10.18653/v1/2023.wassa-1.56 2023

  7. [7]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, and others

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, and others . 2023. https://arxiv.org/abs/2305.10403 PaLM 2 technical report . Preprint, arXiv:2305.10403

    arXiv 2023

  8. [8]

    Geetha A.V., Mala T., Priyanka D., and Uma E. 2024. https://doi.org/10.1016/j.inffus.2023.102218 Multimodal emotion recognition with deep learning: Advancements, challenges, and future directions . Information Fusion, 105:102218

    arXiv 2024

Show all 79 references
  1. [9]

    Abinew Ali Ayele, Seid Muhie Yimam, Tadesse Destaw Belay, Tesfa Asfaw, and Chris Biemann. 2023. https://aclanthology.org/2023.ranlp-1.6 Exploring A mharic hate speech data collection and classification approaches . In Proceedings of the 14th International Conference on Recent ...

    2023

  2. [10]

    Tadesse Destaw Belay, Abinew Ali Ayele, Getie Gelaye, Seid Muhie Yimam, and Chris Biemann. 2021. https://doi.org/10.1109/ICT4DA53266.2021.9672229 Impacts of homophone normalization on semantic models for A mharic . In 2021 International Conference on Information and Communicat...

    2021

  3. [11]

    Tadesse Destaw Belay, Seid Muhie Yimam, Abinew Ayele, and Chris Biemann. 2022. https://aclanthology.org/2022.sigul-1.18 Question answering classification for A mharic social media community based questions . In Proceedings of the 1st Annual Meeting of the ELRA/ISCA Special Int...

    2022

  4. [12]

    Federico Bianchi, Debora Nozza, and Dirk Hovy. 2022. https://doi.org/10.18653/v1/2022.wassa-1.18 XLM - EMO : Multilingual emotion prediction in social media text . In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysi...

    2022 doi

  5. [13]

    Gioele Cageggi, Emanuele Di Rosa, and Asia Uboldi. 2023. https://ceur-ws.org/Vol-3473/paper3.pdf App2check at emit: Large language models for multilabel emotion classification . In Proceedings of the Eighth Evaluation Campaign of Natural Language Processing and Speech Tools fo...

    2023

  6. [14]

    Lihu Chen and Gaël Varoquaux. 2024. https://arxiv.org/abs/2409.06857 What is the role of small models in the LLM era: A survey . Preprint, arXiv:2409.06857

    2024 arXiv

  7. [15]

    Dinu, and Stefan Dumitrescu

    Alexandra Ciobotaru, Mihai Vlad Constantinescu, Liviu P. Dinu, and Stefan Dumitrescu. 2022. https://aclanthology.org/2022.lrec-1.149 RED v2: Enhancing RED dataset for multi-label emotion detection . In Proceedings of the Thirteenth Language Resources and Evaluation Conference,...

    2022

  8. [16]

    Jacob Cohen. 1960. https://doi.org/10.1177/0013164460020001 A coefficient of agreement for nominal scales . Educational and psychological measurement, 20(1):37--46

    1960 doi

  9. [17]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning ...

    2020 doi

  10. [18]

    Didar Dadebayev, Wei Wei Goh, and Ee Xion Tan. 2022. https://doi.org/10.1016/j.jksuci.2021.03.009 Eeg-based emotion recognition: Review of commercial eeg devices and machine learning techniques . Journal of King Saud University - Computer and Information Sciences, 34(7):4385--4401

    2022 doi

  11. [19]

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. https://doi.org/10.18653/v1/2020.acl-main.372 G o E motions: A dataset of fine-grained emotions . In Proceedings of the 58th Annual Meeting of the Association for Computati...

    2020 doi

  12. [20]

    Jiawen Deng and Fuji Ren. 2020. https://doi.org/10.1109/TAFFC.2020.3034215 Multi-label emotion detection via emotion-specified feature extraction and emotion correlation learning . IEEE Transactions on Affective Computing, 14(1):475--486

    2020

  13. [21]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  14. [22]

    Bonaventure F. P. Dossou, Atnafu Lambebo Tonja, Oreen Yousuf, Salomey Osei, Abigail Oppong, Iyanuoluwa Shode, Oluwabusayo Olufunke Awoyomi, and Chris Emezue. 2022. https://doi.org/10.18653/v1/2022.sustainlp-1.11 A fro LM : A self-active learning-based multilingual pretrained l...

    2022 doi

  15. [23]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Zhenyu Yang, and Zhiwei Zhao. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:...

    2024 arXiv

  16. [24]

    Eberhard, Gary F

    David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2024. http://www.ethnologue.com/ Ethnologue: Languages of the World. Twenty-third edition. Dallas, Texas: SIL International. http://www.ethnologue.com/. [Accessed 01-06-2024]

    2024

  17. [25]

    Aleksandra Edwards and Jose Camacho-Collados. 2024. https://aclanthology.org/2024.lrec-main.879 Language models for text classification: Is in-context learning enough? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources an...

    2024

  18. [26]

    Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 6(3-4):169--200

    1992

  19. [27]

    Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. 2023. Do multilingual language models think better in english? arXiv preprint arXiv:2308.01223

    2023 arXiv

  20. [28]

    Mauajama Firdaus, Hardik Chauhan, Asif Ekbal, and Pushpak Bhattacharyya. 2020. https://doi.org/10.18653/v1/2020.coling-main.393 MEISD : A multimodal multi-label emotion, intensity and sentiment dialogue dataset for emotion recognition and sentiment analysis in conversations . ...

    2020 doi

  21. [29]

    Joseph L Fleiss. 1971. https://doi.org/10.1037/h0031619 Measuring nominal scale agreement among many raters . Psychological bulletin, 76(5):378

    1971 doi

  22. [30]

    Fitsum Gaim, Wonsuk Yang, and Jong C. Park. 2022. https://aclanthology.org/2022.lrec-1.707 G eez S witch: Language identification in typologically related low-resourced E ast A frican languages . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pa...

    2022

  23. [31]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023 doi

  24. [32]

    Team Gemma, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and others . 2024. https://arxiv.org/abs/2403.08295 Gemma: Open models based on gemini research and technology . Prep...

    2024 arXiv

  25. [33]

    Daiva Goštautaitė and Leonidas Sakalauskas. 2022. https://doi.org/10.3390/app12115396 Multi-label classification and explanation methods for students’ learning style prediction and interpretation . Applied Sciences, 12(11)

    2022 doi

  26. [34]

    Chenyang Huang, Amine Trabelsi, Xuebin Qin, Nawshad Farruque, Lili Mou, and Osmar Za \" ane. 2021. https://doi.org/10.18653/v1/2021.naacl-main.375 S eq2 E mo: A sequence to multi-label emotion classification model . In Proceedings of the 2021 Conference of the North American C...

    2021 doi

  27. [35]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  28. [36]

    Klaus Krippendorff. 2011. Computing K rippendorff’s alpha-reliability

    2011

  29. [37]

    Sheetal Kusal, Shruti Patil, Jyoti Choudrie, Ketan Kotecha, Deepali Vora, and Ilias Pappas. 2022. https://arxiv.org/abs/2205.03235 A review on text-based emotion detection--techniques, applications, datasets, and future directions . arXiv preprint arXiv:2205.03235

    2022 arXiv

  30. [38]

    Sheetal Kusal, Shruti Patil, Jyoti Choudrie, Ketan Kotecha, Deepali Vora, and Ilias Pappas. 2023. https://doi.org/10.1007/s10462-023-10509-0 A systematic review of applications of natural language processing and future challenges with special emphasis in text-based emotion det...

    2023 doi

  31. [39]

    Sanaa Laabar and Wajdi Zaghouani. 2024. https://aclanthology.org/2024.politicalnlp-1.3 Multi-dimensional insights: Annotated dataset of stance, sentiment, and emotion in F acebook comments on T unisia ' s J uly 25 measures . In Proceedings of the Second Workshop on Natural Lan...

    2024

  32. [40]

    Sheng Li, Rong Yan, Qing Wang, Juru Zeng, Xun Zhu, Yueke Liu, and Henghua Li. 2023. https://doi.org/10.1007/978-3-031-44696-2_3 Annotation quality measurement in multi-label annotations . In International Conference on Natural Language Processing and Chinese Computing, pages 3...

    2023 doi

  33. [41]

    Simon Liegl and Marco R Furtner. 2024. https://doi.org/10.1016/j.chb.2024.108148 Emotional leader communication in the digital age: An experimental investigation on the role of emoji . Computers in Human Behavior, 154:108148

    2024

  34. [42]

    Xuan Liu, Tianyi Shi, Guohui Zhou, Mingzhe Liu, Zhengtong Yin, Lirong Yin, and Wenfeng Zheng. 2023. https://doi.org/10.1057/s41599-023-01816-6 Emotion classification for short texts: an improved multi-label method . Humanities and Social Sciences Communications, 10(1):1--9

    2023 doi

  35. [43]

    Zhiwei Liu, Kailai Yang, Tianlin Zhang, Qianqian Xie, and Sophia Ananiadou. 2024. https://arxiv.org/abs/2401.08508 Emollms: A series of emotional large language models and annotation tools for comprehensive affective analysis . Preprint, arXiv:2401.08508

    2024 arXiv

  36. [44]

    Marian Marchal, Merel Scholman, Frances Yung, and Vera Demberg. 2022. https://aclanthology.org/2022.coling-1.322 Establishing annotation quality in multi-label annotations . In Proceedings of the 29th International Conference on Computational Linguistics, pages 3659--3668, Gye...

    2022

  37. [45]

    Mahmudul Haque, Zakaria Masud Jiyad, M

    Abdullah Al Maruf, Fahima Khanam, Md. Mahmudul Haque, Zakaria Masud Jiyad, M. F. Mridha, and Zeyar Aung. 2024. https://doi.org/10.1109/ACCESS.2024.3356357 Challenges and opportunities of text-based emotion detection: A survey . IEEE Access, 12:18416--18450

    2024

  38. [46]

    Meta. 2024. I ntroducing M eta L lama 3: T he most capable openly available L L M to date --- ai.meta.com. https://ai.meta.com/blog/meta-llama-3/. [Accessed 01-06-2024]

    2024

  39. [47]

    Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. https://doi.org/10.18653/v1/S18-1001 S em E val-2018 task 1: Affect in tweets . In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1--17, New Orleans, LA, USA

    2018 doi

  40. [48]

    Mohammad and Peter D

    Saif M. Mohammad and Peter D. Turney. 2013. https://arxiv.org/abs/1308.6297 Crowdsourcing a word-emotion association lexicon . Computational Intelligence, 29(3):436--465

    2013 arXiv

  41. [49]

    Shamsuddeen Muhammad, Idris Abdulmumin, Abinew Ayele, Nedjma Ousidhoum, David Adelani, Seid Yimam, Ibrahim Ahmad, Meriem Beloucif, Saif Mohammad, Sebastian Ruder, Oumaima Hourrane, Alipio Jorge, Pavel Brazdil, Felermino Ali, Davis David, Salomey Osei, Bello Shehu-Bello, Falalu...

    2023 doi

  42. [50]

    Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. https://aclanthology.org/2021.mrl-1.11 Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages . In Proceedings of the 1st Workshop on Multilingual Representation Lea...

    2021

  43. [51]

    OpenAI. 2024. https://openai.com/ GPT-4o mini: advancing cost-efficient intelligence . https://openai.com/. [Accessed Augest-07-2024]

    2024

  44. [52]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and others . 2024. https://arxiv.org/abs/2303.08774 GPT -4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  45. [53]

    Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. 2022. https://doi.org/10.18653/v1/2022.emnlp-demos.33 POTATO : The portable text annotation tool . In Proceedings of the 2022 Conference on Empirical Me...

    2022 doi

  46. [54]

    Randolph

    Justus J. Randolph. 2005. https://api.semanticscholar.org/CorpusID:59676845 Free-marginal multirater kappa (multirater k[free]): An alternative to fleiss' fixed-marginal multirater kappa

    2005

  47. [55]

    Liu, Jinfeng Zhou, Alvionna S

    Sahand Sabour, Siyang Liu, Zheyuan Zhang, June M. Liu, Jinfeng Zhou, Alvionna S. Sunaryo, Juanzi Li, Tatia M. C. Lee, Rada Mihalcea, and Minlie Huang. 2024. https://arxiv.org/abs/2402.12071 Emobench: Evaluating the emotional intelligence of large language models

    2024 arXiv

  48. [56]

    Kashfia Sailunaz, Manmeet Dhaliwal, Jon Rokne, and Reda Alhajj. 2018. https://doi.org/10.1007/s13278-018-0505-2 Emotion detection from text and speech: a survey . Social Network Analysis and Mining, 8(1):28

    2018 doi

  49. [57]

    Octavio S \'a nchez-Vel \'a zquez and Gerardo Sierra. 2016. https://doi.org/10.13053/rcs-110-1-1 Let's agree to disagree: Measuring agreement between annotators for opinion mining task . Research in Computing Science, 110:9--19

    2016 doi

  50. [58]

    Phakhawat Sarakit, Thanaruk Theeramunkong, Choochart Haruechaiyasak, and Manabu Okumura. 2015. https://doi.org/10.1109/ICTEmSys.2015.7110808 Classifying emotion in thai youtube comments . In 2015 6th International Conference of Information and Communication Technology for Embe...

    2015

  51. [59]

    Gopendra Vikram Singh, Priyanshu Priya, Mauajama Firdaus, Asif Ekbal, and Pushpak Bhattacharyya. 2022. https://aclanthology.org/2022.lrec-1.627 E mo I n H indi: A multi-label emotion and intensity annotated dataset in H indi for emotion recognition in dialogues . In Proceeding...

    2022

  52. [60]

    Nicolas Stefanovitch and Jakub Piskorski. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.6 Holistic inter-annotator agreement and corpus coherence estimation in a large-scale multilingual annotation campaign . In Proceedings of the 2023 Conference on Empirical Methods in Na...

    2023 doi

  53. [61]

    Carlo Strapparava and Rada Mihalcea. 2007. https://aclanthology.org/S07-1013 S em E val-2007 task 14: Affective text . In Proceedings of the Fourth International Workshop on Semantic Evaluations ( S em E val-2007) , pages 70--74, Prague, Czech Republic

    2007

  54. [62]

    Jiuding Sun, Chantal Shaib, and Byron C Wallace. 2023. Evaluating the zero-shot robustness of instruction-tuned language models. arXiv preprint arXiv:2306.11270

    2023 arXiv

  55. [63]

    Jie Tao and Xing Fang. 2020. https://doi.org/10.1186/s40537-019-0278-0 Toward multi-label sentiment analysis: a transfer learning based approach . Journal of Big Data, 7(1):1--26

    2020 doi

  56. [64]

    Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, and others

    NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, and others . 2022. https://arxiv.org/abs/2207.04672 No language left behind: Scaling human-centered machine translation . Prepr...

    2022 arXiv

  57. [65]

    Abrhalei Tela, Abraham Woubie, and Ville Hautamaki. 2020. https://arxiv.org/abs/2006.07698 Transferring monolingual model to low-resource language: the case of tigrinya . arXiv preprint arXiv:2006.07698

    2020 arXiv

  58. [66]

    Atnafu Lambebo Tonja, Israel Abebe Azime, Tadesse Destaw Belay, Mesay Gemeda Yigezu, Moges Ahmed Mehamed, Abinew Ali Ayele, Ebrahim Chekol Jibril, Michael Melese Woldeyohannis, Olga Kolesnikova, Philipp Slusallek, and others . 2024. https://arxiv.org/abs/2403.13737 EthioLLM : ...

    2024 arXiv

  59. [67]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and others . 2023. https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . arXiv preprint ...

    2023 arXiv

  60. [68]

    Fanfan Wang, Heqing Ma, Rui Xia, Jianfei Yu, and Erik Cambria. 2024 a . https://aclanthology.org/2024.semeval-1.277 S em E val-2024 task 3: Multimodal emotion cause analysis in conversations . In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-20...

    2024

  61. [69]

    Kaipeng Wang, Zhi Jing, Yongye Su, and Yikun Han. 2024 b . https://arxiv.org/abs/2403.06108 Large language models on fine-grained emotion detection dataset with data augmentation and transfer learning . Preprint, arXiv:2403.06108

    2024 arXiv

  62. [70]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and others . 2024 c . https://arxiv.org/abs/2406.01574 Mmlu-pro: A more robust and challenging multi-task language understanding benchmark . arXiv ...

    2024 arXiv

  63. [71]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...

    2021 doi

  64. [72]

    Seid Muhie Yimam, Hizkiel Mitiku Alemayehu, Abinew Ayele, and Chris Biemann. 2020. https://doi.org/10.18653/v1/2020.coling-main.91 Exploring A mharic sentiment analysis from social media texts: Building annotation tools and classification models . In Proceedings of the 28th In...

    2020 doi

  65. [73]

    Seid Muhie Yimam, Abinew Ali Ayele, Gopalakrishnan Venkatesh, Ibrahim Gashaw, and Chris Biemann. 2021. https://doi.org/10.3390/fi13110275 Introducing various semantic models for amharic: Experimentation and evaluation with multiple tasks and datasets . Future Internet, 13(11)

    2021 doi

  66. [74]

    Miaoran Zhang, Vagrant Gautam, Mingyang Wang, Jesujoba O Alabi, Xiaoyu Shen, Dietrich Klakow, and Marius Mosbach. 2024 a . https://arxiv.org/abs/2402.12976 The impact of demonstrations on multilingual in-context learning: A multidimensional analysis . arXiv preprint arXiv:2402.12976

    2024 arXiv

  67. [75]

    Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Pan, and Lidong Bing. 2024 b . https://aclanthology.org/2024.findings-naacl.246 Sentiment analysis in the era of large language models: A reality check . In Findings of the Association for Computational Linguistics: NAACL 2024, pages 38...

    2024

  68. [76]

    Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. https://arxiv.org/abs/2302.10198 Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert

    2023 arXiv

  69. [77]

    Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. https:/...

    2024 arXiv

  70. [78]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  71. [79]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.