Pith. sign in

REVIEW 2 major objections 6 minor 20 references

AIMA at SemEval-2024 Task 10: History-Based Emotion Recognition in Hindi-English Code-Mixed Conversations

T0 review · 2 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read This paper claims that context and sequential structure, not the current sentence alone, drive emotion recognition in code-mixed Hindi-English conversations, and that a four-model ensemble beats all baselines tested.

desk verdict A readable SemEval system description with a real gap: the test-time source of the previous-emotion feature is unspecified, which blocks reproduction and could shift the headline numbers. read the letter →

arxiv 2501.11166 v1 pith:PDIH3USC submitted 2025-01-19 cs.CL cs.AIcs.LG

classification cs.CLcs.AIcs.LG
keywords emotionrecognitioninconversationcode-mixedHindi-EnglishHinglishcontext-awareclassificationensemblemajorityvotingmachinetranslationpreprocessingGRU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles emotion recognition in Hindi-English code-mixed conversations, a setting where models trained on monolingual data often struggle. It argues that feeding a classifier the previous and next utterances, together with the conversation's sequential structure, substantially improves emotion prediction over sentence-only baselines. The claim is supported with four base models built on a pre-trained RoBERTa encoder and a final majority-vote ensemble that reaches weighted F1 0.4080 on the official test set, the best among the systems compared. A Hinglish-to-English translation pipeline is part of the recipe, converting code-mixed text to English before encoding.

What carries the argument

The central mechanism is the combination of a fixed pre-trained sentence encoder with a context module that lets the current utterance attend to the previous utterance's embedding through 8-head multi-head attention, plus a learned embedding of the previous emotion label. The strongest model replaces part of this with a stacked two-layer GRU that consumes the last few utterances and the next utterance, and it omits the previous-emotion embedding specifically to prevent error propagation during inference. A two-step translation pipeline, transliterating Hinglish to Hindi and then translating to English with SeamlessM4T, makes the code-mixed input compatible with the English pre-trained encoder.

What would settle it

Run the Simple History-Based and Full History-Based models on the official test set in two inference modes: first with the true emotion label of the previous utterance supplied as the history feature, then with the model's own predicted label for the previous utterance fed forward sequentially. If the second mode's weighted F1 falls materially below 0.4080, the ensemble's reported advantage depends on emotion history that is not available in live deployment.

Watch

Extended reading notes

Core claim

The authors claim that emotion in a code-mixed conversation is best predicted not from the current utterance alone but from a window of conversational context: the preceding sentence and its associated emotion, the immediately following sentence, and the longer dialogue history. Their four base architectures all share a pre-trained encoder and differ in how much context they consume; the Context-Aware GRU-Based model, which reads the last three previous sentences plus the current and next sentence and deliberately excludes the previous predicted emotion, achieves the best single-model weighted F1 of 0.4058. Majority voting over the four models yields the final ensemble at 0.4080, above every baseline in the paper's comparison table. The paper also reports that including all previous sentences adds little beyond the current and previous sentence, and that paraphrase-based data augmentation hurts rather than helps performance on this domain-specific data.

Load-bearing premise

The previous emotion label used as an input feature in the history-based models might come from true labels in the dataset, but the paper never explains how that label is obtained during test-time inference, so the reported scores may not be reproducible when the system runs on new conversations.

Editorial extensions

If this is right

  • Single-model performance is close enough that the previous and next utterances carry most of the predictive signal; adding the full conversation history gives only a small gain.
  • A zero-shot large language model baseline (GPT-3.5 Turbo) trails every proposed model by a wide margin in weighted F1, indicating that this task needs task-specific training rather than prompting alone.
  • Training the same architecture on paraphrase-augmented data reduces performance, suggesting that synthetic paraphrases distort the emotional style of the original code-mixed dialogue.
  • Majority voting over four models improves weighted F1 over each individual model, so ensemble diversity across different context windows is a simple source of gain.
  • Because the final models were fine-tuned on the concatenation of the train and validation splits, the reported test numbers reflect the full available supervision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open how the 'previous emotion' feature is obtained at test time; a fully autoregressive re-run with each model's own predicted emotion as history would reveal how much of the reported gain depends on true labels that are unavailable in deployment.
  • An untested architecture is implied: keep the GRU's sequential memory but re-introduce the previous emotion as a predicted, confidence-weighted input, combining the GRU model's robustness with the history models' extra signal.
  • The translation pipeline's contribution could be isolated by training the same models on romanized Hinglish with a code-mixed multilingual encoder, quantifying whether English translation is a bottleneck or an advantage.
  • The results suggest a cheap deployment recipe for low-resource code-mixed emotion recognition: translate to English, encode with a multilingual or English encoder, and use a context window of one utterance on each side.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper describes the AIMA system submitted to SemEval 2024 Task 10, Subtask 1, for emotion recognition in Hindi-English code-mixed conversations. The authors propose a preprocessing pipeline that transliterates code-mixed text to Hindi and then translates it to English using SeamlessM4T. They then design three base architectures: a Simple History-Based Model, a Full History-Based Model, and a Context-Aware GRU-Based Model, with a fourth model being the Simple History-Based Model trained on a Pegasus-paraphrased augmented dataset. Final predictions are obtained by majority voting over the four models. On the official test set, the final ensemble achieves a weighted F1 of 0.4080, and Table 1 reports that all proposed models outperform the four baselines (GPT-3.5 Turbo, Decision Tree, Linear Regression, and a fine-tuned sentence-level model). The paper concludes that incorporating previous and future context, along with sequential modeling, improves code-mixed ERC performance.

Significance. If the reported results are reproducible, the paper's main contribution is a practical demonstration that standard pre-trained encoders, combined with simple context-aware architectures and ensembling, can achieve competitive performance on code-mixed ERC. The system is evaluated on a shared-task benchmark and compared against several baselines, which is a useful reference for future work on code-mixed conversational emotion recognition. However, the methodological novelty is modest: the architectures combine well-known components (multi-head attention, GRU, pre-trained Transformers) and the paper does not include error analysis, ablation of the translation pipeline, or variance estimates. The primary scientific risk is the underspecified test-time use of the previous-emotion feature in two of the four base models, which directly affects the validity of the headline numbers. The paper is best viewed as a system description; its value depends on the reliability and clarity of the reported experimental protocol.

major comments (2)
  1. The Simple History-Based Model and the Full History-Based Model take the previous utterance's emotion as an input feature, but the paper never states how this label is obtained during test-time inference. Since the test set consists of held-out conversations with hidden labels, gold previous emotions are not available at evaluation time. Section 3.2.4 explicitly notes that the Context-Aware GRU-Based Model omits this feature 'to prevent error propagation during the inference phase,' confirming that the other two architectures are exposed to this concern. If gold labels were used at test time, the reported weighted F1 scores (0.4018 for the Simple model, 0.3992 for the Full model, and the ensemble 0.4080) are not attainable in deployment. If predicted labels were used, the sequential inference procedure, the decoding order, and the effect of error propagation need to be specified, because the numbers in Table 1 would then be dependent on that procedure. This is not an accusation of misconduct, but an underspecified architectural detail that blocks reproduction and could change the headline results. The authors should clarify the test-time protocol and re-run experiments accordingly if needed.
  2. [Table 1] The paper reports a single run per model, with no variance estimates, error bars, or significance tests. The difference between the Context-Aware GRU-Based Model (weighted F1 0.4058) and the final ensemble (0.4080) is only 0.0022, which is well within the range expected from random seed variation. The abstract's claim that the final model 'outperforms all other baselines' and Section 5's statement that 'This ensemble model outperforms each individual model' are therefore not statistically supported by the presented evidence. At minimum, the authors should report multiple runs with mean and standard deviation, or apply a statistical test (e.g., bootstrap or McNemar) to demonstrate that the differences are not due to chance.
minor comments (6)
  1. [Abstract and Section 3.2] The abstract states that the authors designed 'four different base models,' while Section 3.2 says 'three distinct architectures' and describes the second model as following the same architecture as the first but with data augmentation; this wording should be harmonized for clarity.
  2. [Section 3.1] The two-step translation pipeline (indic-trans transliteration followed by SeamlessM4T) is a central preprocessing component, but the paper provides no evaluation of translation quality or examples of translated utterances. At least a few qualitative examples, or a comparison of downstream performance with and without translation, would help readers assess the impact of this step.
  3. [Section 3.2.2] The data augmentation procedure is underspecified: it is not stated whether the Pegasus paraphrases were added only to the training set, how many total augmented examples were produced, or whether the emotion labels were assumed to be preserved after paraphrasing.
  4. [Section 4] The training details lack the number of epochs, the early stopping criterion, and the specific hyperparameters used for the classifier and attention modules; providing these would improve reproducibility.
  5. [Figure 1] The subfigures labeled (a), (b), and (c) in Figure 1 are not explicitly referenced from the text; the authors should refer to them in Sections 3.2.1, 3.2.3, and 3.2.4, respectively.
  6. [References and Table 1] There are minor formatting inconsistencies, for example 'GPT 3.5 Turbo' in Section 5 versus 'GPT-3.5 Turbo' in Table 1, and the reference to SeamlessM4T contains a space in 'V ogeit' that should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical system paper evaluated on an external held-out test set with no fitted quantity or self-citation chain that defines the reported results.

full rationale

This is an empirical system description paper for SemEval-2024 Task 10 subtask 1. The method uses pretrained encoders, a Hinglish-to-English translation pipeline, attention, GRU, and an ensemble of four models; the reported outcome is weighted F1 on the official held-out test split of the MaSaC dataset. There is no derivation in which a quantity is defined in terms of the result it is supposed to predict, no fitted parameter that is later renamed as a prediction, and no load-bearing self-citation. The baselines include GPT-3.5 Turbo, Decision Tree, Linear Regression, and a current-sentence-only fine-tuned model, and the claimed contribution is an architectural comparison plus an ensemble against that external benchmark. The most substantive reproducibility concern is that the Simple and Full History-Based models include the previous utterance's emotion as an input feature, and the paper does not specify how that feature is obtained at test time; Section 3.2.4 explicitly removes this feature to prevent error propagation, acknowledging the issue. This is an underspecified inference detail that could affect reported scores, but it is not circular reasoning under the defined criteria because the test labels are not used to construct the training objective or the evaluation metric. No self-citation or uniqueness theorem is invoked to force any modeling choice, and no component of the result reduces by definition to an input of the paper.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The system's central claim rests on hyperparameters selected by hand, on the reliability of the MaSaC annotations, on the assumption that the Hinglish-to-English translation pipeline preserves emotional content, on the contextual window hypothesis, and on an unspecified source for previous emotion labels at inference. No new entities are introduced.

free parameters (7)
  • encoder learning rate = 5e-6
    Set lower for the pretrained RoBERTa encoder to avoid destructive fine-tuning (Section 4); value chosen manually, no sweep reported.
  • classifier learning rate = 1e-4
    Used for non-encoder parameters (Section 4); chosen by hand.
  • batch size = 1 for GRU model, 4 for others
    Section 4; selected to fit memory and training, no analysis of effect on result.
  • previous emotion embedding dimension = 50
    Section 3.2.1; chosen by hand for the learned emotion embedding space.
  • GRU hidden dimension = 256
    Section 3.2.4; chosen by hand for the stacked GRU.
  • number of attention heads = 8
    Sections 3.2.1 and 3.2.4; standard choice, no ablation.
  • dropout rate = 0.25
    Section 3.2.4; chosen by hand.
assumptions (5)
  • domain assumption MaSaC emotion labels are correct ground truth
    System is trained and evaluated against annotations from Bedi et al. 2023; no annotation quality or inter-annotator agreement analysis is provided.
  • domain assumption Transliteration plus SeamlessM4T translation preserves emotion information
    Section 3.1 applies this pipeline to all input; the paper does not evaluate translation quality or emotion preservation.
  • domain assumption Previous and next utterance context contains the information needed for emotion prediction
    Core modeling choice in Section 3.2; the only context-free comparison is a single fine-tuned sentence encoder baseline.
  • ad hoc to paper Previous emotion label is available at inference for history-based models
    Sections 3.2.1 and 3.2.3 use previous emotion as input, but the paper does not state whether gold or predicted labels are used at test time.
  • standard math Backpropagation and AdamW optimize the cross-entropy objective
    Standard optimization assumed without proof in Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AIMA at SemEval-2024 Task 10: History-Based Emotion Recognition in Hindi-English Code-Mixed Conversations." pith.science (2026). https://pith.science/paper/PDIH3USC

@misc{pith2026250111166,
  author       = {Pith},
  title        = {Pith review of: AIMA at SemEval-2024 Task 10: History-Based Emotion Recognition in Hindi-English Code-Mixed Conversations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PDIH3USC}},
  note         = {Machine review of arXiv:2501.11166}
}
read the original abstract

In this study, we introduce a solution to the SemEval 2024 Task 10 on subtask 1, dedicated to Emotion Recognition in Conversation (ERC) in code-mixed Hindi-English conversations. ERC in code-mixed conversations presents unique challenges, as existing models are typically trained on monolingual datasets and may not perform well on code-mixed data. To address this, we propose a series of models that incorporate both the previous and future context of the current utterance, as well as the sequential information of the conversation. To facilitate the processing of code-mixed data, we developed a Hinglish-to-English translation pipeline to translate the code-mixed conversations into English. We designed four different base models, each utilizing powerful pre-trained encoders to extract features from the input but with varying architectures. By ensembling all of these models, we developed a final model that outperforms all other baselines.

Figures

Figures reproduced from arXiv: 2501.11166 by the authors.

Figure 1
Figure 1. Three proposed base model architectures for predicting the emotion of the current sentence. (a): This model [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 7 canonical work pages

  1. [1]

    Col Cj Antony, B Pariyath, Siti Noorfatimah Safar, Azharuddin Sahil, and Nair Ar. 2021. https://api.semanticscholar.org/CorpusID:234209801 Emotion recognition-based mental healthcare chat-bots: A survey . SSRN Electronic Journal

  2. [2]

    M. Bedi, S. Kumar, M. Akhtar, and T. Chakraborty. 2023. https://doi.org/10.1109/TAFFC.2021.3083522 Multi-modal sarcasm detection and humor classification in code-mixed conversations . IEEE Transactions on Affective Computing, 14(02):1363--1375

  3. [3]

    Irshad Ahmad Bhat, Vandan Mujadia, Aniruddha Tammewar, Riyaz Ahmad Bhat, and Manish Shrivastava. 2015. https://doi.org/10.1145/2824864.2824872 Iiit-h system submission for fire2014 shared task on transliterated search . In Proceedings of the Forum for Information Retrieval Evaluation, FIRE '14, pages 48--53, New York, NY, USA. ACM

  4. [4]

    Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. http://arxiv.org/abs/1406.1078 Learning phrase representations using rnn encoder-decoder for statistical machine translation

  5. [5]

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. http://arxiv.org/abs/1412.3555 Empirical evaluation of gated recurrent neural networks on sequence modeling

  6. [6]

    Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, B...

  7. [7]

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. http://arxiv.org/abs/2005.00547 Goemotions: A dataset of fine-grained emotions

  8. [8]

    Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. https://doi.org/10.18653/v1/2022.acl-long.62 Language-agnostic BERT sentence embedding . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878--891, Dublin, Ireland. Association for Computational Linguistics

Show all 20 references
  1. [9]

    Shivani Kumar, Md Shad Akhtar, Erik Cambria, and Tanmoy Chakraborty. 2024. https://arxiv.org/abs/2402.18944 Semeval 2024 -- task 10: Emotion discovery and reasoning its flip in conversation (ediref) . In Proceedings of the 2024 Annual Conference of the North American Chapter o...

  2. [10]

    Shivani Kumar, Ramaneswaran S, Md Akhtar, and Tanmoy Chakraborty. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.598 From multilingual complexity to emotional clarity: Leveraging commonsense to unveil emotions in code-mixed dialogues . In Proceedings of the 2023 Conference ...

  3. [11]

    Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, and Sirui Wang. 2023. http://arxiv.org/abs/2309.11911 Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework

  4. [12]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach

  5. [13]

    Ilya Loshchilov and Frank Hutter. 2019. http://arxiv.org/abs/1711.05101 Decoupled weight decay regularization

  6. [14]

    Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019. http://arxiv.org/abs/1905.02947 Emotion recognition in conversation: Research challenges, datasets, and recent advances

  7. [15]

    Geng Tu, Bin Liang, Bing Qin, Kam-Fai Wong, and Ruifeng Xu. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.813 An empirical study on multiple knowledge from C hat GPT for emotion recognition in conversations . In Findings of the Association for Computational Linguistics...

  8. [16]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

  9. [17]

    Anshul Wadhawan and Akshita Aggarwal. 2021. https://aclanthology.org/2021.wassa-1.21 Towards emotion recognition in H indi- E nglish code-mixed data: A transformer based approach . In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment a...

  10. [18]

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019. http://arxiv.org/abs/1912.08777 Pegasus: Pre-training with extracted gap-sentences for abstractive summarization

  11. [19]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  12. [20]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.