{"id":"8b0b9177-e9bb-4b42-ac43-2b92b2fa9f85","arxiv_id":"2412.17837","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new multi-label emotion benchmark for four Ethiopian languages shows that fine-tuned encoder-only models outperform zero-shot and few-shot large language models, with large gaps between resource-rich and resource-poor languages.","lead":"This paper introduces EthioEmo, a multi-label emotion dataset for four Ethiopian languages, with about 23,000 annotated posts from Twitter, Facebook, YouTube, and news headlines. It benchmarks many language models and finds that fine-tuned multilingual encoders still outperform zero-shot and few-shot LLMs, especially for low-resource languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lexicon-filtered, neutral-excluded sampling may make EthioEmo unrepresentative; the reported high-vs-low-resource gap is not yet anchored to natural text.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: EthioEmo is built by lexicon filtering and then evaluated only on non-neutral instances, so it is not a random sample of natural emotion expression. This matters for the abstract's broad claim that multi-label emotion classification is insufficient and that a large high-resource/low-resource gap exists. The concern is not that the dataset is useless; it is that the reported numbers are conditional on an explicit-keyword, non-neutral distribution. The paper itself provides supporting evidence that keyword reliance is present in annotation and model errors, which makes the concern concrete rather than hypothetical. A direct random-sample test would settle whether the filter changes conclusions. The other issues raised by the reader (missing code/splits, no confidence intervals) are real but secondary; they affect reproducibility and precision, not the qualitative direction of the central claim. Therefore I keep the reader's CONDITIONAL verdict unchanged: acceptance should be conditioned on making the random-sample comparison or explicitly limiting the claim to the filtered benchmark distribution.","tokens_in":24862,"tokens_out":8086,"duration_ms":80317,"concrete_test":"Annotate a random sample (e.g., 500-1000 instances per language) drawn from the same source pools before the Section 3.1 lexicon filter, using the same annotation protocol and label set. Evaluate the best encoder-only model (AfroXLMR-76L) and GPT-4o-mini on this held-out random sample with the same six-label, neutral-excluded scoring. If the F1 difference relative to the reported EthioEmo test exceeds about 5 points, the lexicon/exclusion filter materially changes benchmark difficulty, and the 'large gap' claim must be re-scoped to lexicon-positive, non-neutral text.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 filters candidate posts using emotion keyword lexicons, and Section 3.3/Table 3 then excludes neutral-only instances from all experiments. The paper itself (Section 6) notes that annotators sometimes depended on emotion keywords and that models often classify from keywords/emojis rather than context. Consequently, the test distribution is a lexicon-positive, non-neutral conditional sample, not a random sample of how emotion is expressed in Amharic, Afan Oromo, Somali, or Tigrinya. The abstract's comparative claim ('large gap between high-resource and low-resource languages') uses English SemEval scores as the high-resource anchor, but the low-resource numbers come from this filtered sample. If the filter makes the Ethiopian task artificially easy, the gap is underestimated; if it makes the task artificially hard, the gap is overestimated. The paper does not quantify either direction. The dataset may still be useful as a controlled multi-label emotion benchmark, but it does not yet support the unqualified generalization that LLM emotion understanding is insufficient on natural social-media text in these languages.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EthioEmo, a new multi-label emotion classification dataset for Amharic, Afan Oromo, Somali, and Tigrinya, using Ekman's six basic emotions plus a neutral class. Texts were collected from Twitter/X, Facebook, YouTube, and news headlines, filtered via emotion lexicons, and annotated by at least three native speakers per instance, with agreement metrics reported. The authors evaluate fine-tuned Afri-centric encoder-only models, zero-shot and few-shot decoder-only models, a translated-to-English test condition, and prompt-sensitivity variants. They report that even high-resource English emotion classification is not solved, that there is a large gap between high-resource and low-resource languages, and that fine-tuned encoder-only models remain strong baselines relative to zero-shot/few-shot LLMs.","tokens_in":25031,"tokens_out":8130,"duration_ms":70193,"significance":"If the central claims hold, EthioEmo is a valuable public resource for an under-resourced area: it adds four Ethiopian languages to multi-label emotion research, uses multiple data sources and a detailed annotation protocol with moderate agreement, and releases lexicons and annotation guidelines. The paper also provides a broad comparison across encoder-only, encoder-decoder, and decoder-only models and includes a translate-test setting that separates language understanding from task difficulty. The main significance lies in the benchmark itself and the reproducible baseline numbers, which will likely be used by later work. However, the paper's headline claims about LLM insufficiency and the high- versus low-resource gap depend on two design choices that are not fully defended: lexicon-filtered sampling and a cross-dataset comparison with an underspecified English benchmark.","major_comments":[{"comment":"The dataset is built by first filtering candidate posts with emotion keyword lexicons and then excluding neutral-only instances from all experiments, as stated in Section 3.1 and Table 3. The paper's own error analysis in Section 6 notes that annotators sometimes depend only on emotion keywords and that models often classify from keywords/emojis rather than context. As a result, EthioEmo is a conditional sample of lexicon-positive, non-neutral posts, not a random sample of how emotion is expressed in natural social-media text in these languages. This does not invalidate the benchmark, but it does mean the abstract's unqualified statement that 'accurate multi-label emotion classification is still insufficient' for natural text in these languages is not established. The authors should either restrict the claim to the lexicon-filtered, non-neutral distribution or add an experiment on a random, unfiltered sample to quantify the selection bias.","section":"Section 3.1, 3.3, Table 3; Section 6"},{"comment":"The English comparison used as the 'high-resource anchor' is underspecified: the paper refers to 'an additional English multi-label emotion dataset from SemEval 2018 Task 1' but does not state which subtask, which label inventory, or which preprocessing steps were used. If, as in the standard SemEval-2018 Task 1 emotion-classification subtask, the English data use a larger and different label set than the six Ekman emotions, then the reported 'large gap between high-resource and low-resource languages' is confounded with the number and type of classes, the annotation protocol, and the data source. The authors should specify the exact English dataset and, ideally, evaluate English under the same six-emotion label scheme to make the cross-language gap interpretable.","section":"Section 4, Table 3"},{"comment":"The text says that 'Encoder-only models still have an advantage over the recently popular open-source decoder-only models for low-resource languages' and that 'LLMs under-perform compared to encoder-only models.' This is contradicted by the zero-shot numbers in Table 5: averaging the columns, Gemma-1.1-7b-it (34.79) and LLaMA-3-8B-Instruct (34.17) both exceed the best zero-shot encoder-only average, AfroLM-active-l (30.20), and the open-weight encoder-decoder Cohere-aya-101 reaches 46.12. The statement appears to conflate zero-shot and fine-tuned encoder-only results. Please correct the claim so that it is consistent with the reported table, for example by restricting it to fine-tuned encoder-only models.","section":"Section 5.2, Table 5"},{"comment":"The LLM evaluation protocol does not specify how free-form or log-likelihood outputs are converted to a multi-label prediction. Section 4.4 mentions log-likelihood-based evaluation with lm-evaluation-harness, and the prompts in Figure 2 ask the model to 'Answer:' with applicable emotions, but the paper does not state whether labels are predicted by per-class yes/no likelihood, by joint likelihood ranking, by a threshold, or by parsing generated text with post-processing. Since the central comparisons of decoder-only and encoder-decoder models depend entirely on these numbers, the lack of an exact output-to-label mapping is a reproducibility gap. Please provide the full decoding/parsing specification and, if possible, release the evaluation scripts.","section":"Section 4.4, 5.5, Tables 5 and 9"},{"comment":"All performance numbers appear to be from single runs with no confidence intervals, bootstrap estimates, or significance tests. Some of the comparative statements rely on small differences, such as the claim that AfroXLMR-76L outperforms for Amharic (69.9 vs. 68.3 for AfroXLMR-61L in Table 4) or the per-language rankings in Table 5. Fine-tuning is stochastic and the k-shot demonstrations are randomly selected (Section 4.4), so the paper should report variance across seeds or example sets before drawing conclusions about which model is best. This is not a demand for exhaustive significance testing, but for the specific comparative claims made in Sections 5.1-5.4, at least a small number of repeated runs or bootstrap intervals would substantiate them.","section":"Tables 4-5, Sections 5.1-5.4"}],"minor_comments":[{"comment":"The text says 'All models benefit from two-shot examples compared to zero-shot tests' and then immediately states that 'this improvement is not shown in Gemma-1.1-7b-it.' This is contradictory; please rephrase to distinguish the general trend from the exception.","section":"Section 5.4"},{"comment":"The relation between the train/test/dev counts and the 'final annotated dataset statistics' (Amharic 5,891, etc.) is not explained; it is unclear how many instances were excluded as neutral-only or unresolved before the splits, and whether the split was performed before or after that exclusion. A sentence clarifying this workflow would help.","section":"Table 3"},{"comment":"The y-axis ranges differ across panels (e.g., English starts near 50 while the Ethiopian languages start near 0), which makes cross-language comparisons visually misleading. Consider using a common scale.","section":"Figure 1"},{"comment":"There are small typographical issues in the references, such as 'Accessed Augest-07-2024' for OpenAI (2024) and 'pertaining phase' in Section 5.1 for 'pre-training phase.'","section":"References"},{"comment":"The prompt sensitivity tables are extensive, but the relationship between the per-prompt numbers and the averaged numbers in Table 5 is not explicitly stated; please note in the text that Table 5 reports the average over the three prompts.","section":"Appendix G.3"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about lexicon-filtered sampling is real and should be addressed; the circularity concern is not, since the lexicons are inputs to dataset construction rather than features in the evaluation. The dataset itself is a solid contribution, and the paper is likely publishable after the sampling representativeness, English comparison specification, and reporting rigor issues are handled."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EthioEmo is a real contribution: the first multi-label emotion dataset for Amharic, Afan Oromo, Somali, and Tigrinya, built from four sources with three annotators per instance and moderate agreement. The annotation protocol is one of the careful parts, and the paper deserves credit for reporting MLA, Cohen's kappa, and free-marginal kappa, and for doing expert error analysis. The experimental sweep is broad: fine-tuned Afri-centric encoders, encoder-decoder, decoder-only open models, GPT-4o-mini, zero/few-shot, translate-test, and prompt sensitivity. That is a solid benchmark paper.\n\nThe soft spot is the sampling. Section 3.1 filters candidate posts using emotion keyword lexicons, and Section 3.3 drops neutral-only instances from the evaluation. So the test distribution is a lexicon-positive, non-neutral conditional sample, not a random sample of emotion expression in these four languages. The English anchor is SemEval 2018 tweets, which were not filtered the same way. The paper's own Section 6 gives reason to worry: annotators sometimes depend on emotion keywords, and models often classify from keywords/emojis rather than context. A lexicon-filtered benchmark may overstate how easy the low-resource task is, or at least change its character. The abstract's claim that there is a large high-vs-low-resource gap is likely true in direction, but the size is not anchored to natural text. This does not invalidate EthioEmo as a controlled multi-label emotion benchmark; it does undercut the unqualified generalization about LLM emotion understanding on natural social media.\n\nOther soft spots are minor: all F1 numbers are point estimates with no error bars or significance tests, so differences of a few points between models should not be over-read. The evaluation details for LLM outputs are thin, and the paper doesn't ship the evaluation harness and exact splits, though the dataset and lexicons are public. The citation pattern is fine: the self-cited lexicon and sentiment resources are inputs to dataset construction, not to the evaluation, so there is no circularity.\n\nWho is this for? People working on multilingual affective computing or low-resource NLP benchmarks. It is a useful new resource and a fair baseline suite. It deserves serious peer review; a referee should push for a discussion of the sampling bias, variance estimates, and reproducibility artifacts. I would accept it with revisions, not desk-reject.","headline":"New dataset, real contribution, but the headline high-vs-low-resource gap rests on a filtered sample; treat the benchmark as controlled, not naturalistic.","tokens_in":25609,"tokens_out":2973,"would_cite":true,"duration_ms":28860,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current large language models still cannot reliably identify all emotions expressed in a text, even in English, and the gap is much larger for four Ethiopian languages; the paper supports this with a new benchmark called EthioEmo.","keywords":["multi-label emotion classification","EthioEmo","Ethiopian languages","large language models","low-resource NLP","emotion benchmark","in-context learning","fine-tuning versus zero-shot prompting"],"falsifier":"Take a fresh sample of social media posts in the four languages without applying the emotion-keyword filter, annotate it with the same protocol, and rerun the same models. If zero-shot LLM performance on the unfiltered sample is comparable to or better than the fine-tuned baselines, the claim that LLMs are insufficient would not generalize to natural text; if performance collapses, the benchmark's keyword-based selection is driving the apparent gap.","tokens_in":24673,"feed_emoji":"💬","tokens_out":7721,"duration_ms":60318,"temperature":0.7,"pith_summary":"The paper introduces EthioEmo, a multi-label emotion classification dataset for Amharic, Afan Oromo, Somali, and Tigrinya, and uses it to test how well large language models recognize the emotions people actually express. Its central finding is that accurate multi-label emotion classification is still not solved even for English, and that the performance gap between high-resource and low-resource languages remains wide. On the EthioEmo test sets, fine-tuned encoder-only models clearly outperform zero-shot and few-shot LLMs, which the paper reads as evidence that emotion understanding has not emerged from scale alone. A sympathetic reader should care because the dataset and its baselines give the field a concrete instrument for tracking whether future models improve.","feed_headline":"LLMs still fail multi-label emotion, even in English","feed_subtitle":"A four-language benchmark shows zero-shot LLMs trail fine-tuned small models by wide margins.","key_machinery":"The load-bearing object is EthioEmo itself, a multi-label dataset built by filtering millions of collected posts and news headlines through per-language emotion keyword lexicons, then annotating each selected instance with any subset of six Ekman emotion classes plus neutral. The evaluation protocol is the other half: fine-tuned encoder-only models are compared with zero- and few-shot decoder-only models using log-likelihood scoring, and a translate-test variant converts the Ethiopian test sets into English to separate task difficulty from language understanding. The multi-label agreement scores reported for the annotation are what make the performance differences interpretable as model failures rather than noise in the gold labels.","core_discovery":"The paper's central claim is that current large language models are insufficient for multi-label emotion understanding, and this insufficiency is most visible in low-resource languages. It supports this with EthioEmo, a new benchmark of 23,321 annotated instances across four Ethiopian languages, evaluated alongside the English SemEval-2018 Task 1 set. The best fine-tuned encoder-only models reach weighted F1 scores of roughly 58-73% on Ethiopian languages, while zero-shot open LLMs stay around 25-49% and a commercial model averages 52%. Translating the Ethiopian test sets into English does not close the gap, so the paper concludes the difficulty is not mainly a translation problem: it is a problem of how emotion is modeled and how well languages are represented in pretraining.","pith_inferences":["My inference: because the corpus was filtered by emotion keywords, EthioEmo may overrepresent explicit emotion vocabulary; true performance on implicit, sarcastic, or context-dependent emotion could be lower than the benchmark numbers suggest.","My inference: the pattern across model types suggests that what matters most is whether the target language appeared in pretraining, not model size or reasoning ability; a testable extension would be to compare models pretrained on the same data but with different parameter counts.","My inference: the same evaluation design could be exported to other low-resource language families to see whether the high-resource/low-resource emotion gap is universal or specific to these four languages.","My inference: the failure modes reported, with generative models over-predicting emotion labels and discriminative models collapsing to single labels, point toward calibration-aware metrics as a next step for multi-label emotion evaluation."],"forward_implications":["Fine-tuned encoder-only models remain the stronger practical baseline for low-resource multi-label emotion classification, at least until LLM prompting improves substantially.","Because translating the test set into English does not erase the gap, progress on emotion understanding cannot be achieved by translation alone; it requires language-specific training data or better multilingual pretraining.","Showing a few examples helps most LLMs, but the gains are inconsistent, so in-context learning is not a reliable substitute for fine-tuning on this task.","EthioEmo gives future work a public benchmark with per-source and per-language breakdowns, making it possible to measure whether a new model is actually better at emotion or just better at English.","Even English multi-label emotion classification remains unsolved, so emotion understanding should be treated as an open evaluation target rather than a solved capability."],"supporting_citations":[{"why":"Defines the six basic emotion categories that form the dataset's label set.","marker":"Ekman (1992)"},{"why":"Supplies the English NRC emotion lexicon that is translated and extended to create filtering keywords.","marker":"Mohammad and Turney (2013)"},{"why":"Establishes the prior Amharic sentiment lexicon method that the keyword-filtering pipeline builds on.","marker":"Yimam et al. (2020)"},{"why":"Provides the SemEval-2018 English multi-label emotion test set used as the high-resource comparison.","marker":"Mohammad et al. (2018)"},{"why":"Provides a fine-grained English emotion dataset whose inter-annotator agreement is the comparison point for EthioEmo's reliability.","marker":"Demszky et al. (2020)"},{"why":"Introduces the multi-label agreement measure used to compute annotation quality for the new dataset.","marker":"Li et al. (2023)"},{"why":"Motivates the translate-test design that separates language understanding from task difficulty.","marker":"Etxaniz et al. (2023)"},{"why":"Provides the evaluation harness used for zero-shot and in-context-learning scoring.","marker":"Gao et al. (2023)"}],"fun_headline_variants":["LLM emotion fails even in English, worse in 4 African tongues","EthioEmo reveals LLM multi-label emotion gap across languages","Zero-shot LLMs trail fine-tuned models on emotion tags","Multi-label emotion: English still hard, low-resource harder","New 4-language emotion benchmark exposes LLM limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusion depends on the assumption that filtering texts by emotion keywords yields a sample that represents how emotion is actually expressed in these languages, rather than a sample in which emotion words are unusually easy to spot.","fun_headline_variants_meta":{"raw":{"variants":["LLM emotion fails even in English, worse in 4 African tongues","EthioEmo reveals LLM multi-label emotion gap across languages","Zero-shot LLMs trail fine-tuned models on emotion tags","Multi-label emotion: English still hard, low-resource harder","New 4-language emotion benchmark exposes LLM limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1692,"prompt_tokens":892,"completion_tokens":800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":713}},"tokens_in":508,"tokens_out":800,"duration_ms":6847,"temperature":1.0,"reasoning_tokens":713,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:53:12.091073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh sample of social media posts in the four languages without applying the emotion-keyword filter, annotate it with the same protocol, and rerun the same models. If zero-shot LLM performance on the unfiltered sample is comparable to or better than the fine-tuned baselines, the claim that LLMs are insufficient would not generalize to natural text; if performance collapses, the benchmark's keyword-based selection is driving the apparent gap.","supporting_citations":[],"review_version":1}