{"id":"b61f619a-7734-42a5-967e-8c3a0c6cc4a0","arxiv_id":"2505.13244","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuned multilingual LLMs with a pairwise per-emotion prompt achieved competitive results in multilingual multi-label emotion detection, including first place in Hindi at SemEval-2025 Task 11.","lead":"This paper describes the JNLP system that competed in SemEval-2025 Task 11, using Qwen and XLM-RoBERTa models fine-tuned to detect multiple emotions and their intensities in up to 28 languages. It reports top-four rankings in ten languages for emotion labels and top-five rankings in seven languages for emotion intensity.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The printed instruction templates omit the official emotion 'disgust' while outputs and evaluation include it, so the paper may not describe the system that actually produced the reported leaderboard results.","rationale":"The reader's weakest assumption is precisely this prompt mismatch, and I agree it is the most load-bearing local defect. The abstract's strong empirical claim is about a specific system; an internal inconsistency in the only artifact that defines that system (the instruction template) prevents a reader from knowing what was actually submitted. This is not an ad hominem or a disagreement with consensus: it is a reproducibility failure that is directly visible in the paper (Table 1 vs. Table 2/Fig. 3 and the official six-emotion label set). I considered the alternative concern that the JNLP(test) row aggregates per-language best configurations rather than a single method; that would further weaken 'simplicity,' but the prompt mismatch is more decisive because it blocks attribution of the results to any described system. The check is feasible: the authors release code, so verifying the deployed prompt and re-running the printed prompt settles the issue. If the code contains 'disgust,' the verdict stays conditional pending a prompt-template correction; if not, the central claim is unsupported. Therefore no verdict change is needed beyond the reader's CONDITIONAL.","tokens_in":10404,"tokens_out":10080,"duration_ms":94731,"concrete_test":"Download https://github.com/yingjie7/mlingual_multilabel_emo_detection and compare the exact Track A/Track B inference prompt templates against Table 1 and Appendix A; check specifically whether 'disgust' appears in the label set. Then run the printed Table 1 template on the development set for English and Hindi and compare macro-F1 with Table 3. If the printed template cannot output 'disgust' while the reported outputs/evaluation include it, the mismatch is substantive rather than typographical, and the central claim is not connected to the described system.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—Top 4 in Track A across 10 languages, 1st in Hindi—depends on 'our method' being the method described in §3.2. That description rests entirely on the prompt templates in Table 1 and Appendix A. Those templates define the emotion label set as {anger, fear, joy, sadness, surprise}, omitting 'disgust.' Yet Table 2 gives 'disgust, sadness' as a Track A output example, Figure 3 trains/evaluates 'disgust' in Track B, and the official SemEval-2025 Task 11 label set includes disgust. If the inference prompt actually used in the submission was the one printed, the model could not produce 'disgust' labels except by ignoring the system instruction; if, more plausibly, the deployed prompt included 'disgust,' then Table 1 and Appendix A are not the prompt that generated the results. Either way the paper does not currently specify the actual submitted system, so the leaderboard performance cannot be attributed to the described 'pairwise, mixed-language instruction-tuned' recipe from the text alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a system for SemEval-2025 Task 11, covering Track A (multi-label emotion detection across 28 languages in the BRIGHTER dataset) and Track B (emotion intensity prediction across 11 languages). Two architectures are compared: a fine-tuned BERT-based classifier and instruction-tuned generative LLMs (Llama-2 and Qwen-2.5), under two multi-label strategies: a base strategy that generates all emotion labels at once, and a pairwise strategy that considers each emotion separately. The authors also compare per-language training against mixed-language training. The reported results are external leaderboard scores: top-four rankings in 10 Track A languages including first place in Hindi, and top-five rankings in 7 Track B languages. The paper additionally analyzes when pairwise versus base strategies help, the effect of label imbalance, and the role of mixed-language training.","tokens_in":10613,"tokens_out":9880,"duration_ms":86569,"significance":"If the reported leaderboard scores are taken at face value, the paper provides a credible and simple competitive recipe for multilingual multi-label emotion detection: instruction-tuned Qwen-32B with pairwise prompting performed strongly across many languages, and the code is publicly available. The central empirical claim rests on an external benchmark with no fitted constants, so circularity is not a concern. The main weaknesses are internal-consistency problems in the system description and two overgeneralized auxiliary findings; these are fixable and do not undermine the external leaderboard evidence.","major_comments":[{"comment":"The instruction templates printed in Table 1 and in Appendix A (Tables 5 and 6) define the emotion label set as {anger, fear, joy, sadness, surprise}, omitting 'disgust'. This contradicts Table 2, whose Track A output example is 'disgust, sadness'; Figure 3, which reports performance for 'disgust'; and Section 5.3, which discusses 'disgust' as a class. Since the official BRIGHTER label set includes disgust, the reader cannot tell from the text whether the submitted runs used the printed prompt (in which case the model would have to ignore the system instruction to produce disgust) or a different prompt that included disgust (in which case the paper does not describe the actual submitted system). Please correct the label set in all templates and state explicitly the exact prompts used for the submitted Track A and Track B runs, including how 'disgust' was handled.","section":"Table 1; Appendix A (§3.2)"},{"comment":"The claim that 'mixed-language training... demonstrates superior performance compared to training separate models' is not supported for Track A by the data in Table 3. The only mixed-language Track A row reported (Qwen-32b base mixed langs) has average 0.5933, identical to the Qwen-32b pairwise separated langs average of 0.5933, and no pairwise-mixed Track A result is shown; in contrast, Table 4 does show pairwise mixed (0.7363) beating pairwise separated (0.7234) in Track B. Please either report the pairwise-mixed Track A result or qualify this claim to Track B only.","section":"§5.3, Tables 3–4"}],"minor_comments":[{"comment":"The statement that 'in low-resource languages, LLMs performed poorly, whereas the RoBERTa-based approach yielded better results' is contradicted by several rows in Table 3: for afr, arq, ibo, kin, pcm, vmw, and yor, the Qwen-32B models beat xml-roberta by a clear margin. Please qualify the claim to specific languages or report a systematic comparison.","section":"§5.1, Table 3"},{"comment":"The BERT-based multi-label classifier is described as using a sigmoid output layer, but the loss is said to be cross-entropy. For multi-label classification with independent sigmoid outputs, the appropriate objective is binary cross-entropy per label; please correct this.","section":"§3.2, Eq. (4)"},{"comment":"The column header 'orm(development)' appears to be a typo for 'orm'. Please also clarify in the caption whether the reported development results are on the official development set or on the 10% internal held-out set described in Section 4.","section":"Table 3"},{"comment":"The notation in equations (1) and (2) is ambiguous (for example, P_A({0,1} | x, y_e) mixes set notation with probability notation). Please define all symbols and describe the decoding procedure that maps generated text to predicted labels and intensity values.","section":"§3.1, Eqs. (1)–(2)"},{"comment":"The phrase 'Top 4 performance across 10 languages' is ambiguous because Section 5.1 reports a mix of 1st, 2nd, 3rd, and 4th place rankings. Please rephrase as 'ranked among the top four in 10 languages' for clarity.","section":"Abstract, §5.1"},{"comment":"There is a grammatical error in the sentence 'their effectiveness in multilingual sentiment analysis is often limited without additional fine-tuning is performed'; it should read '...without additional fine-tuning being performed'.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a shared-task system description whose core evidence is external leaderboard scores, so I am comfortable with major revision rather than rejection. The main blocker is the inconsistency between the printed prompts (which omit disgust) and the results that include disgust; the authors should be required to state exactly which prompts were used for the submitted runs and to correct the two overgeneralized claims in Sections 5.1 and 5.3."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid SemEval-2025 system description that earns its place mostly through official leaderboard results, not new method machinery. The components—LoRA fine-tuning, base vs pairwise prompting, mixed vs per-language training—are standard. The value is the comparison on a fresh 28-language dataset with external test scores. The headline ranks (Top 4 in 10 Track A languages, 1st in Hindi; Top 5 in 7 Track B languages) are credible and backed by detailed tables.\n\nThe best parts: the dev-set ablations in Tables 3 and 4 show real work; the analysis of pairwise vs base with respect to average number of labels (Figures 1-2) is a useful observation; the limitations paragraph is honest about compute cost and label imbalance. The code link is a plus.\n\nThe soft spots are real but mostly fixable. The biggest is the prompt-template discrepancy. Table 1 and Appendix A define the label set as {anger, fear, joy, sadness, surprise}, omitting 'disgust,' while Table 2, Figure 3, and the official task all include disgust. If the printed prompt is the one that ran, the model couldn't have produced disgust labels; if it wasn't, the paper doesn't document the actual system. Either way, the central recipe is not reproducible from the text as written. I'd treat this as a documentation error, likely a typo, but it is load-bearing for a system description, so it needs a correction and a re-check of every prompt template before publication.\n\nSecond, the claim that 'LLMs performed poorly in low-resource languages, whereas RoBERTa-based approach yielded better results' doesn't hold as written. Table 3 shows a mixed picture: xml-roberta wins some low-resource languages (amh, kin, ibo) but Qwen wins others (arq, ary, swe, tat). The averages are close. That sentence needs nuance, not a blanket generalization.\n\nThere's no variance or seed info, and no description of how output text is parsed into labels. For a system paper, parsing is part of the method; its absence makes replication harder.\n\nOn balance, the empirical claims are solid and checkable. The analysis is honest. The issues are documentation precision, not fake results. I would send this to peer review, but only after the authors fix the prompt table and soften the low-resource claim. For the right reader—someone building multilingual emotion detection systems—this is a useful comparison point.","headline":"A credible SemEval-2025 system description whose leaderboard results stand, but the printed prompts omit one of the six emotion labels, so the paper needs a correction before it can serve as a reproducible recipe.","tokens_in":11100,"tokens_out":2936,"would_cite":false,"duration_ms":27423,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Instruction-tuning a 32-billion-parameter generative model with a pairwise, one-emotion-at-a-time prompt and training on all languages together places it in the top four for ten languages in SemEval-2025 Task 11 emotion detection, first…","keywords":["multilingual emotion detection","multi-label classification","instruction fine-tuning","large language models","pairwise classification","LoRA","low-resource languages","cross-lingual transfer"],"falsifier":"Run the exact Table 5 and Table 6 prompts—with the five-emotion label set printed in the paper—through a LoRA-fine-tuned Qwen2.5-32B on the BRIGHTER development set and compare macro-F1 and Pearson values with the development columns of Tables 3 and 4. A second check: if the model ever emits 'disgust' despite the prompt not listing it, the printed template cannot be the deployed prompt, and the reported numbers are not reproducible from the paper alone.","tokens_in":10229,"feed_emoji":"🎭","tokens_out":8288,"duration_ms":76373,"temperature":0.7,"pith_summary":"The paper claims that a simple recipe—taking a 32-billion-parameter generative model, instruction-tuning it with LoRA, and asking it to judge each emotion separately rather than produce the full label set at once—generalizes across languages in the SemEval-2025 Task 11 emotion-detection shared task. The evidence offered is competitive standing: top-four macro-F1 in ten Track A languages, first in Hindi, and top-five Pearson correlation in seven Track B languages. The authors further report that training one model on all languages mixed together beats per-language models, and that the pairwise 'one emotion at a time' question beats a base prompt that asks for all labels when sentences carry multiple emotions, while the base prompt wins in sparse-label languages. The recipe also transfers from label detection (Track A) to intensity estimation (Track B) by changing only the expected answer format. If the recipe is right, multilingual multi-label emotion detection does not need language-specific engineering; the question decomposition and the training mixture do the work.","feed_headline":"Pairwise LLM prompt wins Hindi, top-4 in 10","feed_subtitle":"One mixed-language Qwen-32B model, prompted one emotion at a time, took first in Hindi and top-five in seven intensity tracks.","key_machinery":"The load-bearing object is the pairwise instruction template: a system prompt that names the emotion label set, a user turn of the form 'Given the sentence: ..., is the emotion anger expressed in it?' (Track A) or 'what is the intensity of the emotion fear?' (Track B), and an assistant answer of 'yes'/'no' or 'low'/'moderate'/'high'. LoRA fine-tuning of Qwen2.5-32B under a causal language-modeling objective on all languages concatenated is the training mechanism, and the base template—asking for all labels at once—is the control that isolates the benefit of per-emotion decomposition.","core_discovery":"The central discovery is that multi-label emotion detection in many languages can be reformulated as a set of per-emotion generation questions and solved by a single instruction-tuned LLM: for Track A, ask 'is emotion ye expressed?' and read 'yes' or 'no'; for Track B, ask for the intensity of ye and read 'low', 'moderate', or 'high'. With Qwen2.5-32B fine-tuned under LoRA on all training languages jointly, this pairwise strategy achieves the paper's reported rankings, while a base strategy that maps a sentence directly to all labels is competitive only where most sentences have zero or one emotion. Mixed-language training consistently improves over separate per-language models, and the generative model generally outperforms a fine-tuned XLM-RoBERTa encoder. The authors thus claim the pairwise decomposition, not model size or language-specific tuning, is the ingredient that carries the generalization.","pith_inferences":["A likely reason mixed-language training helps is that it balances the emotion-label distribution across languages; a testable corollary is that per-language models should catch up when trained on resampled or augmented data that equalizes label frequencies.","The five-emotion prompt printed in Table 1 cannot explain the 'disgust' rows in Figure 3 and Table 2, so the deployed prompt almost certainly listed six emotions; if so, the paper's template is illustrative rather than the exact submission prompt.","The authors' suggested logit-aggregation variant is an immediate cheap improvement: instead of decoding 'yes' or 'no', sum the logits of both tokens and threshold the softmax probability, which should yield calibrated per-language thresholds.","The pairwise recipe should transfer to other multi-label text-generation problems, such as aspect-based sentiment or emotion-in-conversation, because the conditioning mechanism (one label at a time) is independent of the label inventory."],"forward_implications":["A single mixed-language instruction-tuned model can replace per-language pipelines for multilingual emotion detection, which matters most for low-resource languages where separate training data are thin.","Teams facing multi-label emotion data should default to the pairwise question format when sentences regularly carry several emotions, and to the base format when labels are sparse.","The same pairwise prompt structure covers both label detection and intensity prediction, so one fine-tuned model can serve both tasks without architectural changes.","Instruction-tuned generative models of the Qwen2.5-32B class are a viable alternative to fine-tuned encoders for this task, with the main trade-off being higher inference cost."],"supporting_citations":[{"why":"Supplies the BRIGHTER training, development, and test data for Track A, the dataset the system is trained and evaluated on.","marker":"Muhammad et al., 2025a"},{"why":"Defines SemEval-2025 Task 11, its two tracks, and the evaluation metrics (macro-F1 and Pearson) against which the rankings are computed.","marker":"Muhammad et al., 2025b"},{"why":"Instruction fine-tuning methodology the LLM training follows, the foundation of the generative approach.","marker":"Chung et al. (2022)"},{"why":"LoRA, the parameter-efficient fine-tuning method used to adapt Qwen2.5-32B.","marker":"Hu et al., 2022"},{"why":"BERT, the architecture for the baseline classification model whose [CLS] vector is fed through the sigmoid output layer.","marker":"Devlin et al., 2019"},{"why":"XLM-RoBERTa, the multilingual encoder baseline compared against the generative models.","marker":"Conneau et al., 2020"},{"why":"LLaMA 2, the other generative model family tested, showing the pairwise recipe is not tied to one model.","marker":"Touvron et al., 2023"}],"fun_headline_variants":["Ask one emotion at a time: LLM tops Hindi, top-4 in 10","Per-emotion prompts beat big models for multilingual sentiment","One LLM, per-label queries: top-4 across 10 languages","Pairwise LLM queries win Hindi, top-4 in 10 languages","Generative pairwise method ranks #1 Hindi, top-4 in 10"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported rankings are only reproducible if the printed instruction templates are the exact prompts that produced the submitted predictions, and the paper's own tables show a mismatch—the printed label set has five emotions while outputs and evaluation include 'disgust' as a sixth.","fun_headline_variants_meta":{"raw":{"variants":["Ask one emotion at a time: LLM tops Hindi, top-4 in 10","Per-emotion prompts beat big models for multilingual sentiment","One LLM, per-label queries: top-4 across 10 languages","Pairwise LLM queries win Hindi, top-4 in 10 languages","Generative pairwise method ranks #1 Hindi, top-4 in 10"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1757,"prompt_tokens":993,"completion_tokens":764,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":665}},"tokens_in":609,"tokens_out":764,"duration_ms":7406,"temperature":1.0,"reasoning_tokens":665,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:16:38.243108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact Table 5 and Table 6 prompts—with the five-emotion label set printed in the paper—through a LoRA-fine-tuned Qwen2.5-32B on the BRIGHTER development set and compare macro-F1 and Pearson values with the development columns of Tables 3 and 4. A second check: if the model ever emits 'disgust' despite the prompt not listing it, the printed template cannot be the deployed prompt, and the reported numbers are not reproducible from the paper alone.","supporting_citations":[],"review_version":1}