Pith. sign in

REVIEW 4 major objections 6 minor 12 references

CSIRO-LT at SemEval-2025 Task 11: Adapting LLMs for Emotion Recognition for Multiple Languages

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-language LoRA fine-tuning is the most effective adaptation strategy for multilingual emotion recognition.

desk verdict Solid shared-task system description with a defensible narrow finding, but the abstract overstates the evidence and the test results undercut it in several languages. read the letter →

arxiv 2508.01161 v1 pith:B2KPNMM3 submitted 2025-08-02 cs.CL

Jiyu Chen , Necva Bölücü , Sarvnaz Karimi , Diego Mollá , Cécile L. Paris This is my paper
classification cs.CL
keywords emotionrecognitionmultilingualLLMLoRAfine-tuningparameter-efficientSemEval-2025Task11multi-labelclassificationintensitylow-resourcelanguages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a system for the SemEval-2025 shared task on detecting perceived emotions in text across 32 languages. It claims that the most effective adaptation strategy among those tried is to take a pre-trained multilingual large language model and fine-tune it separately for each language using low-rank adaptation (LoRA), rather than first instruction-tuning it, bridging through English, or using retrieval-based few-shot prompting. On the test set this per-language fine-tuning reached macro-F1 scores of 89.10 for Russian and 77.62 for English in multi-label emotion detection. The paper also frames each emotion as an independent binary or four-way intensity classification, which lets a single model be repurposed across tracks. The broader conclusion is that direct task-specific fine-tuning may be more valuable than elaborate prompt or instruction strategies for multilingual emotion recognition.

What carries the argument

The central mechanism is parameter-efficient supervised fine-tuning with LoRA (low-rank adaptation) on a per-language basis, applied to a pre-trained multilingual LLM. LoRA updates only a small set of low-rank weight matrices, so each language can receive its own adaptation cheaply; combined with 4-bit quantisation this makes per-language training practical. Around it, the paper formulates Track A as six independent binary classifications and Track B as six independent four-way intensity classifications, and it uses these formulations to compare direct SFT with instruction-tuned, English-bridged, marginalised-intensity, and BM25 few-shot alternatives.

What would settle it

Run the same development-set comparison on low-resource languages from the same task, such as Hausa, Oromo, Tigrinya, or Yoruba: if direct per-language SFT does not beat instruction-tuning-before-SFT, English-bridging, or BM25 few-shot there, the central claim is falsified. A second check is to rerun Track A using one multilingual model with a single shared fine-tune instead of per-language models; if the shared model matches or exceeds per-language scores, the 'separately for each language' part of the claim fails.

Watch

Extended reading notes

Core claim

For the emotion-detection track, the paper treats presence of each of six emotions as an independent binary decision, and for the intensity track as an independent four-class decision, then fine-tunes one multilingual LLM per language with LoRA and 4-bit quantisation. In development-set comparisons restricted to English, German, Portuguese and Russian, this direct supervised fine-tuning achieved the highest macro-F1 in every language, beating instruction-tuning on an external emotion dataset before fine-tuning, an English-bridging continual-fine-tuning strategy, and BM25-based one-shot prompting. The authors conclude from this that the best recipe for this task is direct per-language supervised fine-tuning of a pre-trained multilingual LLM, and they report that larger GPT-style models such as aya-32b-expanse tend to outperform smaller mT5-based aya-101, particularly for lower-resourced languages.

Load-bearing premise

The conclusion that fine-tuning a model separately for each language is the best approach comes from tests on just four well-resourced Indo-European languages, and assumes the result holds for all 32, including low-resource and structurally different languages.

Editorial extensions

If this is right

  • If direct per-language LoRA SFT is best, then elaborate multi-stage procedures such as instruction-tuning on external affective data, English bridging, and marginalising intensity predictions offer no benefit for the tested languages and can be omitted.
  • The same recipe should transfer to the other languages in the 32-language task, since the final test submissions used per-language SFT or zero-shot and reached competitive results such as 89.10 macro-F1 for Russian and 77.62 for English.
  • For intensity detection, fine-tuning also improves most languages, with instruction-tuned models most helpful in zero-shot settings and in languages with less training data.
  • Model choice matters: larger GPT-style models generally did better than the smaller mT5-based model, especially on lower-resourced languages, so per-language fine-tuning should be paired with the largest feasible multilingual base.
  • Across both tracks, the results suggest that when adapting LLMs for emotion recognition on most mid- and high-resourced languages, instruction-tuning is not as effective as directly supervised fine-tuning on task-specific data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The strategy ranking was established on four mid- to high-resource Indo-European languages; in lower-resource or non-Indo-European languages direct SFT may not be the winner, and retrieval-based few-shot prompting might matter more, a test the paper leaves open.
  • If the finding that instruction-tuning does not help after SFT generalises, it would simplify practice for classification-style multilingual tasks: skip the external instruction phase and spend the compute on per-language LoRA weights.
  • A natural testable extension is to run the same four-way strategy comparison on low-resource languages from the same dataset, such as Yoruba, Oromo, or Tigrinya, to see whether the ranking flips.
  • The paper's comparison used one-shot BM25 prompting as the few-shot baseline; scaling to more shots or using embedding-based retrieval could change the few-shot results and therefore the relative ranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports the CSIRO-LT submission to SemEval-2025 Task 11 (Tracks A and B), comparing strategies for adapting multilingual LLMs (aya-32b-expanse, aya-101, Llama3.1-8B-Instruct) for emotion recognition. Track A is formulated as per-emotion binary classification, and Track B as per-emotion intensity classification. The tested adaptation strategies include direct per-language LoRA SFT, instruction-tuning followed by SFT, English-bridged adaptation, marginalisation via a Track B model, and BM25-based one-shot ICL. The main claim, stated in the abstract and conclusions, is that per-language LoRA fine-tuning of a multilingual LLM is the most effective adaptation method. The supporting evidence is a development-set comparison (Table 2) over four languages (English, German, Portuguese, Russian), all mid- or high-resource Indo-European languages. Test-set results are reported for 26 languages in Track A and 11 languages in Track B, with comparisons to the organiser-provided baselines.

Significance. This is a standard shared-task system description. Its strengths are the use of task-provided splits and metrics, transparent reporting of hyperparameters (LoRA rank/alpha, epochs, learning rates, BM25 parameters), and a controlled comparison of several practical adaptation strategies on the same development set. If the central claim were restricted to the four evaluated languages, Table 2 convincingly supports the superiority of direct per-language SFT over the other tested strategies. However, the abstract's unqualified wording and the unsupported claim about low-resource languages limit the contribution. With careful scoping and removal or verification of the low-resource few-shot claim, the paper would be a useful empirical data point for LLM adaptation in multilingual emotion recognition.

major comments (4)
  1. [Abstract; §5 Conclusions; Table 2] The abstract states that 'the most effective method for this task is to fine-tune a pre-trained multilingual LLM with LoRA setting separately for each language,' but the evidence in Table 2 compares strategies only on English, German, Portuguese, and Russian, all mid- or high-resource Indo-European languages. Moreover, §5 explicitly narrows the conclusion to 'most languages, except for the lower-resourced languages where few-shot learning is more effective.' The abstract and the conclusion must be aligned by scoping the claim to the four evaluated languages, or the paper must add development-set strategy comparisons for low-resource and non-Indo-European languages.
  2. [§5 Conclusions; Table 2; Table 3] The conclusion that 'few-shot learning is more effective' in lower-resourced languages is not supported by any experiment in the paper. Table 2 shows 1-shot BM25 ICL underperforming direct SFT in all four languages, and Table 3 compares zero-shot versus SFT on several languages, including low-resource ones such as Amharic, Hausa, and Ukrainian, but it includes no few-shot condition. Either remove this claim or add the supporting comparison.
  3. [§4 Results; Table 1] The test-set results in Table 1 are not produced uniformly by per-language LoRA SFT: Ukrainian uses zero-shot emo-aya (marked with an asterisk), and many low-resource languages use the smaller aya-101 model. The paper does not explain how the per-language model and strategy were selected for languages outside the four-language development comparison, nor does it report which Table 1 entries come from SFT versus ICL. Without this information, the test-set scores cannot be used as evidence for the abstract's general claim.
  4. [§2.1 Methods; Table 2] The comparison set in Table 2 does not include a joint multilingual SFT baseline, i.e., fine-tuning a single model on the combined training data for all languages. The paper can therefore only claim that per-language SFT is the best among the four tested strategies, not that it is the most effective method for the task in any absolute sense. Please restrict the wording accordingly or add such a baseline.
minor comments (6)
  1. [Abstract] 'Semeval' should be written as 'SemEval' to be consistent with the task name used elsewhere in the paper.
  2. [§2.1 Methods] BM25 is a lexical retrieval method; describing it as 'semantic (bag-of-words) relevancy' is misleading. Please rephrase.
  3. [§4 Results (Track A development set)] The claim that 'all comparisons are statistically validated using hypothesis testing with a significance threshold of p < 0.05' lacks supporting details; please specify the test procedure (e.g., paired bootstrap or McNemar) and report p-values or significance markers for the key pairwise comparisons in Table 2.
  4. [Table 1] Add a note explaining what the dashes mean (no submission, no baseline, etc.) and, where feasible, indicate which entries are zero-shot versus SFT, since the current caption only says 'SFT or zero-shot (*)'.
  5. [§3.3 Hyperparameters] 'The training epoch size is 10' should read 'the number of epochs is 10' or 'we train for 10 epochs.'
  6. [§3.1 Experimental Setup] The description of the external instruction-tuning dataset is very brief; please include the data size, the specific subtasks used, and how the data were integrated into the instruction-tuning step.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claim is an empirical comparison against held-out task data, not a derivation that reduces to its inputs.

full rationale

The paper's core claim, that per-language LoRA fine-tuning is most effective, rests on direct development-set comparisons in Table 2 across four languages, with test-set results in Table 1 reported on held-out data provided by the task organisers. Hyperparameters (LoRA rank 32, alpha 64, learning rates, epochs, prompt templates) are fixed and reported; none are fitted to the test set. The test-set model selection is based on development performance, which is standard practice and does not constitute fitting a parameter and then predicting the same data. The abstract's unqualified phrasing overgeneralizes from a four-language comparison and conflicts with the paper's own Limitations and conclusion caveat that few-shot learning is more effective for lower-resourced languages, but that is a correctness/scope concern, not circularity. No load-bearing step reduces to a self-citation, a renamed input, or a definitional equivalence. The evaluation is self-contained against an external shared-task benchmark.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical one. The main free parameters are the fixed hyperparameters and the per-language model choices made using development-set results. The axioms are domain assumptions about dataset reliability, baseline comparability, and absence of leakage from the external instruction-tuning data. No invented entities are proposed.

free parameters (5)
  • LoRA rank and alpha = rank=32, alpha=64
    Chosen hyperparameters for all fine-tuning runs; not fitted to data but affect results. Section 3.3.
  • LoRA dropout, batch size, epochs = dropout=0.05, batch size=2, epochs=10
    Fixed hyperparameters for SFT; reported in Section 3.3.
  • Learning rates = 2e-5 (Track A), 5e-5 (Track B)
    Separate learning rates chosen for each track; Section 3.3.
  • BM25 parameters = k1=1.5, b=0.75
    Default parameters for BM25 retrieval in few-shot prompting; Section 3.3.
  • Per-language model selection = Varies by language (aya-101, aya-32b-expanse, Llama3.1-8B, or instruction-tuned variants)
    The choice of which LLM to use for each language on the test set was made based on development-set results; this manual selection affects the reported test numbers. Section 4, Table 1.
assumptions (4)
  • domain assumption The shared task dataset annotations are reliable and follow the task definition of collectively perceived emotion.
    The paper relies on the dataset provided by the task organisers without independent verification of annotation quality.
  • domain assumption The baseline results reported by Muhammad et al. (2025a) are computed under comparable conditions (same data split, same metrics).
    The baseline is not described in the paper; conclusions about outperforming the baseline depend on this.
  • domain assumption The external instruction-tuning dataset (Liu et al., 2024) does not overlap with the shared task test set.
    No leakage analysis is provided, yet this external data is used for EMO models.
  • domain assumption LoRA fine-tuning with the chosen ranks preserves the multilingual capabilities of the base models.
    The paper does not evaluate catastrophic forgetting or language coverage after fine-tuning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CSIRO-LT at SemEval-2025 Task 11: Adapting LLMs for Emotion Recognition for Multiple Languages." pith.science (2026). https://pith.science/paper/B2KPNMM3

@misc{pith2026250801161,
  author       = {Pith},
  title        = {Pith review of: CSIRO-LT at SemEval-2025 Task 11: Adapting LLMs for Emotion Recognition for Multiple Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B2KPNMM3}},
  note         = {Machine review of arXiv:2508.01161}
}
read the original abstract

Detecting emotions across different languages is challenging due to the varied and culturally nuanced ways of emotional expressions. The \textit{Semeval 2025 Task 11: Bridging the Gap in Text-Based emotion} shared task was organised to investigate emotion recognition across different languages. The goal of the task is to implement an emotion recogniser that can identify the basic emotional states that general third-party observers would attribute to an author based on their written text snippet, along with the intensity of those emotions. We report our investigation of various task-adaptation strategies for LLMs in emotion recognition. We show that the most effective method for this task is to fine-tune a pre-trained multilingual LLM with LoRA setting separately for each language.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 5 canonical work pages

  1. [3]

    arXiv preprint arXiv:2407.21783

    The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Scott A Golder and Michael W Macy

    arXiv

  2. [6]

    In Proceed- ings of the Eleventh International Conference on Lan- guage Resources and Evaluation (LREC 2018), pages 198–209

    Un- derstanding Emotions: A Dataset of Tweets to Study Interactions between Affect Categories. In Proceed- ings of the Eleventh International Conference on Lan- guage Resources and Evaluation (LREC 2018), pages 198–209. European Language Resources Association (ELRA). Saif M Mohammad

    work page 2018

  3. [8]

    arXiv preprint arXiv:2307.09288

    Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Elsbeth Turcan, Smaranda Muresan, and Kathleen McK- eown

    arXiv

  4. [10]

    arXiv preprint arXiv:2402.07827

    Aya Model: An instruction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827. Wikipedia

    arXiv

  5. [12]

    arXiv preprint arXiv:2308.10792

    Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792. 7

    arXiv

  6. [2018]

    In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1–17

    SemEval- 2018 Task 1: Affect in Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1–17. Saif Mohammad and Svetlana Kiritchenko

    work page 2018

  7. [2020]

    arXiv preprint arXiv:2005.14165

    Language models are few-shot learners. arXiv preprint arXiv:2005.14165. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yanni...

    arXiv 2005

  8. [2021]

    In Proceedings of the 2021 conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 2895–2909

    Emotion-infused models for explainable psychological stress detection. In Proceedings of the 2021 conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 2895–2909. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei- Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika S...

    work page 2021

Show all 12 references
  1. [2022]

    Compu- tational Linguistics, 48(2):239–278

    Ethics sheet for automatic emotion recognition and sentiment analysis. Compu- tational Linguistics, 48(2):239–278. Shamsuddeen Hassan Muhammad, Nedjma Ousid- houm, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nir- mal Surange, Daniela Teo...

    2025 arXiv

  2. [2023]

    In Findings of the Association for Computational Linguistics: EACL 2023, pages 1825–1836

    Best Practices in the Creation and Use of Emotion Lexicons. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1825–1836. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko

    2023

  3. [2024]

    Preprint, arXiv:2412.04261

    Aya expanse: Combining Re- search Breakthroughs for a New Multilingual Fron- tier. Preprint, arXiv:2412.04261. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al

    arXiv

  4. [2025]

    Accessed: 2025-02-26

    Shared heritage of German and En- glish. Accessed: 2025-02-26. Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tian- wei Zhang, Fei Wu, et al

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.