{"id":"a0d4e722-1406-4e6f-9376-22af98a52f7c","arxiv_id":"2412.12761","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding native-language samples and joint training with hate detection improves Hindi-English code-mixed humour and sarcasm detection, with multi-task learning giving the largest gains.","lead":"This paper tests three ways to improve AI detection of humour and sarcasm in Hindi-English mixed social media posts: adding monolingual English or Hindi training samples, training on multiple related tasks, and prompting or fine-tuning large multilingual language models. The multi-task approach gave the largest F1 gains (up to 10.7 points for humour and 12.4 for sarcasm), while large language models lagged behind fine-tuned multilingual models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 'up to 10.67%/12.35%' gains rest on unstated test-set selection and no variance estimates; best MTL configs are maxima over a large grid, so the claimed MTL advantage may be a selection artifact.","rationale":"The paper is an extensive empirical study, and its core experiments are internally consistent: the reported F1 scores across Tables 3 and 4 show that adding native samples and using MTL both move the needle relative to code-mixed-only training, and the paper is transparent about the synthetic Hindi translations losing humor and sarcasm (Appendix G). However, the central claim as stated in the abstract and conclusion—that MTL gives the largest gains, with 'up to 10.67%' and 'up to 12.35%' increments—is exactly the kind of claim that requires variance estimates and a clear model-selection protocol. Without them, the 'up to' numbers are unverifiable maxima over a large grid, and the unexplained p<0.05 markers cannot be interpreted. This is the reader's weakest assumption, and I agree it is the most load-bearing gap. A secondary concern is that MTL runs include additional native task data and hate data, so part of MTL's advantage could be attributed to data quantity rather than multi-task sharing; however, this is less central because the practical claim is about using those data sources. The concrete test I propose would settle whether the headline gains survive a validation-selected, multi-seed protocol. If they do, the central claim stands; if not, the paper would need to report more conservative numbers. This does not change the reader's CONDITIONAL verdict—it reinforces the conditions needed for acceptance.","tokens_in":39759,"tokens_out":10287,"duration_ms":102680,"concrete_test":"Fix the two headline comparisons (MuRIL humor, XLM-R sarcasm). Enumerate all Table 4 configurations (native dataset × auxiliary task × gate) plus the CM-only baseline; for each, select hyperparameters (learning rate, batch size, sequence length, lambda) that maximize validation F1. Evaluate the validation-selected model on the held-out test split, repeating with 10 independent seeds, and report mean ± std of test F1 for CM, native-sample-mixing best, and MTL best. Apply a paired test across seeds (e.g., Wilcoxon signed-rank or paired bootstrap) between MTL-best and NSM-best, and between MTL-best and CM. If the validation-selected test F1 difference is not significant, or the relative increments fall materially below the reported 'up to' values, the headline numbers are selection artifacts and the central claim should be reworded.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that MTL gives the largest gains hinges on two headline numbers: MuRIL MTL reaching F1 0.83 for humor (a 10.67% relative increment over MuRIL's CM-only 0.75) and XLM-R MTL reaching 0.91 for sarcasm (a 12.35% increment over XLM-R's CM-only 0.81). These 'up to' values are maxima taken over Table 4's large configuration space: three MLMs, four native dataset choices, seven auxiliary-task combinations, gated and ungated variants, plus the hyperparameter grid in Table 10 (six learning rates, three batch sizes, three sequence lengths, five regularization strengths). Section 4 states results are averaged over three seeds and marks p<0.05 results with '*', but it never describes the significance test, nor does it report variance or standard errors. Critically, the paper never states whether the best configurations in Tables 3 and 4 were chosen on the validation split or after inspecting test-set outcomes; Appendix E.1 only describes the train/val/test split. If test-set selection occurred, the reported 'up to' improvements are selected maxima expected to overstate the true effect. Even if validation was used, with three seeds and dozens of configurations, the probability that the maximum F1 exceeds the true effect by chance is substantial, and the per-configuration p-values would need multiple-comparison correction to support the 'most significant improvement' conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports experiments on Hindi-English code-mixed humour and sarcasm detection. Three strategies are compared: (i) mixing native monolingual (English and translated Hindi) task samples into the code-mixed training set, (ii) multi-task learning with hate detection as an auxiliary task, and (iii) prompting and instruction-finetuning of very large multilingual language models (VMLMs). The central empirical claim is that native sample mixing and multi-task learning improve positive-class F1 relative to code-mixed-only training, with MTL giving the largest gains ('up to 10.67%' for humour and 'up to 12.35%' for sarcasm), while VMLMs do not improve. The paper includes baseline reproductions, ablations, error analysis, and a public code repository.","tokens_in":40027,"tokens_out":4995,"duration_ms":44739,"significance":"If the claims hold, the paper provides useful evidence that scarce code-mixed annotations can be supplemented by monolingual data and a related auxiliary task, and it documents negative results for VMLM prompting in this code-mixed setting. The release of code and the reproduction of prior baselines are concrete strengths, as is the qualitative error analysis that connects MTL gains to hate-related contexts. However, the headline magnitudes are not yet supported by the reported statistics: the absence of variance estimates, an undefined significance test, and the lack of a stated model-selection protocol make it impossible to distinguish a genuine MTL advantage from selection over a large configuration grid.","major_comments":[{"comment":"The paper reports F1-scores 'averaged over three random seeds' and marks p<0.05 results with '*', but it never describes the significance test, reports no standard deviations or confidence intervals, and does not indicate whether any multiple-comparison correction was applied. Because the headline claims 'up to 10.67%' and 'up to 12.35%' are maxima over a large configuration space (three MLMs, four native dataset choices, seven auxiliary-task combinations, gated and ungated variants, plus the hyperparameter grid in Table 10), the reported gains may be selection artifacts rather than stable effects. Please report per-configuration variance and a complete significance-test procedure, or weaken the claims accordingly.","section":"Section 4, Tables 3 and 4"},{"comment":"It is never stated whether the best configurations in Tables 3 and 4 were chosen on the validation split or after inspecting test-set outcomes. If test-set selection occurred, the 'up to' improvements are optimistic by construction; even if validation was used, the number of configurations makes uncorrected per-configuration p-values unreliable. Please state the exact selection protocol, including whether the reported best configurations were fixed before the test set was used.","section":"Appendix E.1, Tables 3 and 4"},{"comment":"Appendix G concludes that Hindi translations 'didn't preserve the humorous and sarcastic context' and that the authors 'decided to use only English samples for further experiments,' but Section 3.2 and Table 3 present CM+Hi and CM+Hi+En as main experimental conditions, and the title and abstract emphasize 'synthetic native samples.' This is an internal inconsistency about what was actually evaluated. Please clarify which reported results use translated Hindi samples, whether the Hindi-augmented columns in Table 3 are included in the main claims, and whether translated Hindi samples were used in the MTL experiments.","section":"Appendix G vs. Section 3.2 and Table 3"}],"minor_comments":[{"comment":"The phrase 'upto' should be 'up to' in the abstract, contributions, and conclusion; also 'Futhermore' in Section 4.4 is a typo.","section":"Throughout"},{"comment":"The citation 'Swami et al. (1805)' is a date typo for 2018; please correct it.","section":"Table 1, Sarcasm row"},{"comment":"The phrase 'refer Appendix 5' should be 'refer Section 5', since the error analysis is in Section 5 of the main text.","section":"Section 7, Limitation bullet"},{"comment":"The reproduced F1 for Pandey and Singh (0.85) differs substantially from the originally reported 0.92; a brief explanation, such as a different evaluation split or metric conversion, would help readers interpret the baseline comparison.","section":"Table 2"},{"comment":"The table lists hyperparameter ranges, but the paper does not state which specific values (learning rate, batch size, sequence length, regularization strength) were used for the best configurations reported in Tables 3 and 4; please report the selected values for reproducibility.","section":"Table 10"}],"recommendation":"major_revision","confidential_remarks":"For the editor: this is an empirical benchmark paper rather than a methods contribution; its fit depends on whether the venue welcomes such studies. The central weakness is statistical selection and reporting, which appears fixable in a revision: the authors should report variance, describe the significance test, state the model-selection protocol, and clarify the role of synthetic Hindi data. I see no evidence of fabrication or of a circular derivation; the reliance on the authors' prior hate-detection result is motivational, not a methodological tautology."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a referee, but the headline \"up to\" gains should be re-examined. The paper applies native-sample mixing and MTL to Hindi-English code-mixed humor/sarcasm, and shows MTL with hate detection gives the biggest bumps. The direction is believable, and the work is honest: it reproduces baselines, releases code, and acknowledges the Google-translated Hindi samples lost humor and sarcasm (Appendix G). That last admission creates tension with the title and parts of the abstract—the paper frames \"synthetic native samples\" as a contribution while later saying they mostly used English-only mixes because the translations were bad. That framing needs to be straightened out.\n\nWhat is new is modest but real: native mixing was shown by the same group for hate; this extends it to two new tasks, and the MTL-with-hate result is the more interesting finding. The comparison against reproduced baselines is appropriate. The error analysis is useful.\n\nThe soft spots are the ones the stress-test identifies. Results are averaged over three seeds, but no variance or confidence intervals appear anywhere. The p<0.05 stars are unexplained—no test named, no correction for the dozens of configurations in Tables 3 and 4. The best rows are clearly \"up to\" maxima across a large grid, and the paper never states whether the configuration was chosen on validation or test. With three seeds and that many runs, the chance that the max F1 overshoots the true effect is real. The claim that MTL is \"the most significant improvement\" needs either a proper comparison or more cautious wording. Also, the synthetic Hindi samples are not released, so the negative result on Hindi mixing cannot be reproduced.\n\nNone of this kills the central direction, but it means the quantitative headline should be treated as optimistic until variance and selection details are provided.\n\nWho this is for: people working on low-resource/code-mixed classification, especially those considering data reuse and MTL. It deserves a serious referee, but it needs a revision that adds variance estimates, clarifies the selection procedure, and reconciles the synthetic-Hindi narrative.","headline":"Useful extension of the group's native-mixing idea to humor/sarcasm, but the headline MTL gains are maxima over a big grid with no variance estimates or selection details.","tokens_in":40597,"tokens_out":2061,"would_cite":true,"duration_ms":21623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For Hindi-English code-mixed humour and sarcasm detection, multi-task learning with a hate-detection auxiliary task and reusing monolingual task samples yields the largest F1-score gains, while prompting or instruction-finetuning very…","keywords":["code-mixed humour detection","code-mixed sarcasm detection","Hindi-English code-mixing","native sample mixing","multi-task learning","hate detection auxiliary task","multilingual language models","synthetic Hindi data"],"falsifier":"Re-run the three strategies with a fixed, validation-only model-selection protocol on the same Khandelwal et al. (2018) and Swami et al. (2018) test splits, and report per-seed F1 distributions or confidence intervals; if the MTL and native-mixing configurations selected without test-set peeking no longer beat the code-mixed-only baseline by the reported margins, the central claim is not supported.","tokens_in":39497,"feed_emoji":"😂","tokens_out":10334,"duration_ms":84535,"temperature":0.7,"pith_summary":"Hindi-English code-mixed tweets switch between Hindi and English inside a single utterance, and labelled data for detecting humour and sarcasm in them is scarce. This paper tests whether that scarcity can be bypassed by reusing existing resources: adding monolingual English and machine-translated Hindi humour or sarcasm samples to the code-mixed training set, jointly training with a semantically related hate-detection task, and prompting or instruction-finetuning very large multilingual models. Its central finding is that multi-task learning gives the largest improvement, with F1-score gains up to 10.67% for humour and 12.35% for sarcasm, followed by native sample mixing at up to 6.76% and 8.64%, while the very large models show no F1 improvement. The paper therefore argues that transfer from monolingual task data and a related task can substitute for expensive code-mixed annotation, at least when the base models are smaller multilingual language models.","feed_headline":"Multi-task learning lifts code-mixed humour and sarcasm scores","feed_subtitle":"Sharing hate-detection data and monolingual samples beats huge language models on Hindi-English code-mixed tweets.","key_machinery":"The argument is carried by two mechanisms. The first is native sample mixing: augmenting the code-mixed training set with monolingual task samples, in this case English datasets and synthetic Hindi versions produced by machine translation, so that a multilingual encoder can draw on linguistic knowledge from non-code-mixed text. The second is the multi-task framework, a 12-layer BERT-style architecture with a shared bottom module, separate task-specific top modules for humour, sarcasm, and hate detection, and a gating mechanism that blends the shared representation with each task representation; a regularizer softly pulls the task-specific weight matrices together. The gating mechanism is what lets the model exploit the hate-detection task on short and misspelled code-mixed inputs, and the paper's ablation shows that removing it lowers scores.","core_discovery":"On the paper's own terms, the discovery is that Hindi-English code-mixed humour and sarcasm detectors can be improved by training with data that is not code-mixed, and improved further by learning a related task at the same time. When monolingual English samples and translated Hindi samples are added to the code-mixed training set, the best multilingual models reach 0.79 F1 for humour and 0.88 for sarcasm, gains of up to 6.76% and 8.64%. When the same models are trained in a multi-task framework with hate detection as an auxiliary task, the best scores rise to 0.83 for humour and 0.91 for sarcasm, gains of up to 10.67% and 12.35%. The paper also finds that prompting or instruction-finetuning very large multilingual models with native samples does not improve them, largely because they over-assign the positive label, and that statistical classifiers are hurt rather than helped by native sample mixing.","pith_inferences":["The paper leaves implicit that its observation about low KL divergence could be turned into a selection rule: rank candidate native datasets by overlap between positive and negative word distributions, and prefer lower-divergence datasets before mixing.","A natural testable extension is to replace machine-translated Hindi with native Hindi humour and sarcasm data or with context-aware LLM translation; the paper's own error analysis predicts that gold Hindi data would push native sample mixing closer to the multi-task gains.","The VMLMs' tendency to over-predict humour and sarcasm suggests that calibration or explicit negative-example prompting, neither of which the paper tries, is the next lever for making large models competitive in this setting.","The hate-related errors in the multi-task models indicate that task interference is real; varying the strength of the soft-parameter-sharing regularizer or the hate task weight could trade off the observed gains and losses, a test the paper does not run."],"forward_implications":["Practitioners can improve code-mixed humour and sarcasm detectors without new code-mixed annotation by reusing English humour and sarcasm datasets and an existing hate-detection corpus.","The multi-task configuration with hate detection is the strongest of the three strategies, with the largest gains on short, misspelled inputs, and the gating mechanism is a load-bearing component of that gain.","Machine-translated Hindi samples are not a reliable source of humorous or sarcastic data, because wordplay and ironic tone are largely lost in translation; gold Hindi data or better context-aware translation would be needed.","Very large multilingual models prompted with native or code-mixed examples, or instruction-finetuned, do not match the smaller multilingual models on these tasks under the tested setups, due to a tendency to favour humour and sarcasm labels.","Statistical classifiers degrade when native samples are mixed in, so the benefit of native sample mixing is specific to pre-trained multilingual representations."],"supporting_citations":[{"why":"Supplies the Hindi-English code-mixed humour corpus whose training and test splits are used.","marker":"Khandelwal et al. (2018)"},{"why":"Supplies the Hindi-English code-mixed sarcasm corpus whose training and test splits are used.","marker":"Swami et al. (2018)"},{"why":"The prior native sample mixing result for hate detection that the paper extends to humour and sarcasm.","marker":"Mazumder et al. (2024)"},{"why":"Provides the multi-task architecture with shared bottom layers, task-specific modules, and gating that the MTL experiments adapt.","marker":"Rotman and Reichart (2022)"},{"why":"mBERT is one of the multilingual models evaluated and used in the multi-task framework.","marker":"Devlin et al. (2019)"},{"why":"XLM-R is one of the multilingual models evaluated and achieves the best sarcasm MTL result.","marker":"Conneau et al. (2020)"},{"why":"MuRIL is one of the multilingual models evaluated and achieves the best humour MTL result.","marker":"Khanuja et al. (2021)"},{"why":"IndicBERT is one of the multilingual models evaluated and achieves the best native-sample-mixing result.","marker":"Doddapaneni et al. (2023)"},{"why":"Supplies the ColBERT English humour dataset used as a native sample source.","marker":"Annamoradnejad and Zoghi (2024)"},{"why":"Supplies the News Headlines Dataset used as a native English sarcasm sample source.","marker":"Misra and Arora (2019)"}],"fun_headline_variants":["Multi-task learning boosts code-mixed humour and sarcasm scores","Adding native samples improves humour and sarcasm detection in code-mixed text","Hate detection data helps code-mixed humour and sarcasm models","Monolingual data boosts code-mixed humour and sarcasm F1 scores","Huge multilingual models lose to multi-task learning on code-mixed data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported F1 gains are not artifacts of configuration selection: the paper averages over three seeds but reports no variance or explicit test-selection protocol, so if the best configurations were chosen after looking at test-set scores, the headline improvements could be optimistic.","fun_headline_variants_meta":{"raw":{"variants":["Multi-task learning boosts code-mixed humour and sarcasm scores","Adding native samples improves humour and sarcasm detection in code-mixed text","Hate detection data helps code-mixed humour and sarcasm models","Monolingual data boosts code-mixed humour and sarcasm F1 scores","Huge multilingual models lose to multi-task learning on code-mixed data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":3135,"prompt_tokens":1008,"completion_tokens":2127,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":2033}},"tokens_in":624,"tokens_out":2127,"duration_ms":14100,"temperature":1.0,"reasoning_tokens":2033,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:45:15.688636+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the three strategies with a fixed, validation-only model-selection protocol on the same Khandelwal et al. (2018) and Swami et al. (2018) test splits, and report per-seed F1 distributions or confidence intervals; if the MTL and native-mixing configurations selected without test-set peeking no longer beat the code-mixed-only baseline by the reported margins, the central claim is not supported.","supporting_citations":[],"review_version":1}