{"id":"3aa8ead3-9938-4336-a808-2594552acbc3","arxiv_id":"2506.08147","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper introduces a trilingual English-Urdu-Spanish hate speech dataset and reports that LLMs like GPT-3.5 Turbo and Qwen 2.5 72B outperform SVM baselines, but evaluation inconsistencies and missing artifacts undercut the claims.","lead":"The paper reports a new trilingual dataset of about 10,000 tweets in English, Urdu, and Spanish labeled for hate speech, and compares many models from SVM to large language models like GPT-3.5 Turbo and Qwen 2.5 72B. The authors claim LLMs beat traditional baselines, especially for Urdu, which has little annotated hate speech data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-language confusion matrices all sum to the joint test-set size (2,039), contradicting the stated per-language 80/20 splits; the reported F1 scores are not supported by the paper's own numerical evidence.","rationale":"The reader's REJECT verdict is correct, but the most decisive problem is not the ambiguity in 'LLMs were trained on the entire dataset' versus 'evaluated on the test set using few-shot learning.' It is the fact that every per-language confusion matrix sums to 2,039, exactly the size of the joint 20% test set, while the stated per-language 80/20 splits imply test sets of roughly 767, 633, and 640. This is an internal numerical contradiction, not a wording ambiguity. Even if 'trained on the entire dataset' were clarified as few-shot prompting, the confusion matrices still cannot describe per-language held-out evaluation. The central empirical claim—per-language macro F1 of 0.87, 0.85, and 0.81—therefore lacks a valid evidentiary basis. The dataset and annotation effort may have value, but the reported benchmark results are not credible as presented. I partially agree with the reader's weakest_assumption: both concerns invalidate the empirical claim, but the confusion-matrix arithmetic is more objective and more easily checked.","tokens_in":20299,"tokens_out":4642,"duration_ms":55489,"concrete_test":"Recompute the expected test-set size for each language from the stated 20% split (English ≈ 767, Spanish ≈ 633, Urdu ≈ 640). Then re-derive macro F1 from each confusion matrix using only instances in that per-language test partition. If any monolingual confusion matrix contains more instances than its language test set, or if macro F1 computed from the matrix differs from Table 9, the headline scores are invalid. Request the per-language test IDs and the confusion matrices restricted to those IDs; this single check settles the contradiction.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing weakness is an internal arithmetic contradiction, independent of the train/test wording. Section 5.5 and Section 6 state an 80/20 train/test split, so the English test set should be about 20% of 3,834 = 767 instances, Spanish about 20% of 3,162 = 633, and Urdu about 20% of 3,197 = 640. Yet the confusion matrices for the best models sum to 920+854+99+166 = 2,039 for English, 900+833+119+187 = 2,039 for Spanish, 860+792+159+228 = 2,039 for Urdu, and 930+864+89+156 = 2,039 for Joint. Only the Joint matrix can plausibly contain 2,039 test instances. If the monolingual models were evaluated on the combined 2,039-instance test set, then the language-specific F1 scores in Table 9 do not measure language-specific generalization. If they were evaluated per language, the matrices are misreported. Either way, the central empirical claim of per-language macro F1 values 0.87, 0.85, and 0.81 is not supported by the paper's own evidence. This is more decisive than the Section 5.5 versus Section 7.4 ambiguity about whether LLMs were fine-tuned on all data or few-shot prompted, because it does not depend on interpreting the word 'trained.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a trilingual (English, Urdu, Spanish) hate speech detection dataset of 10,193 tweets and benchmarks a wide range of models, including SVM, BiLSTM, transformers, and LLMs (GPT-3.5 Turbo, Qwen 2.5 72B), claiming that attention-layer-integrated LLMs achieve macro F1 scores of 0.87 (English), 0.85 (Spanish), 0.81 (Urdu), and 0.88 (joint). The main claimed contributions are the novel dataset with high inter-annotator agreement (Fleiss' Kappa 0.821), a translation-based pipeline, and an attention-augmented multilingual framework. The paper reports consistent-sounding aggregate metrics, but the supporting numerical evidence contains internal contradictions that preclude verification of the central claims.","tokens_in":20580,"tokens_out":4615,"duration_ms":52325,"significance":"If the empirical claims were substantiated, this work would contribute a useful trilingual hate speech benchmark and a practical comparison of proprietary and open-weight LLMs against classical baselines, including a low-resource language (Urdu) that is genuinely underexplored. The dataset construction with three native-speaker annotators and reported agreement of 0.821 is a potentially valuable resource, and the explicit reporting of confusion matrices is a good-faith effort at transparency. However, the current manuscript does not provide the evidence needed to support its headline results: the confusion matrices contradict the stated experimental design, the LLM evaluation protocol is ambiguous on a point that determines validity, and the central attention-layer contribution is asserted without any ablation. As such, the scientific significance of the paper, as written, is not established.","major_comments":[{"comment":"The confusion matrices for the best English, Spanish, and Urdu models each sum to 2,039 instances (e.g., English: 920+854+99+166 = 2,039; Urdu: 860+792+159+228 = 2,039), which is exactly the size of the joint test set described in Section 5.5. Given the stated 80/20 split per language, the expected monolingual test-set sizes are approximately 767 (English), 633 (Spanish), and 640 (Urdu). This discrepancy means either the monolingual models were evaluated on the combined 2,039-instance test set, in which case the per-language F1 scores in Tables 8-9 do not measure language-specific generalization, or the confusion matrices are misreported. In either case, the central empirical claim of per-language macro F1 values 0.87, 0.85, and 0.81 is not supported by the paper's own numerical evidence.","section":"Section 8, Tables 10-13; Section 5.5"},{"comment":"Section 5.5 states that 'LLMs were trained on the entire dataset' to maximize contextual understanding, while Section 7.4 states that LLMs were 'evaluated on the test set using a few-shot learning approach.' These are incompatible evaluation protocols, and the paper never clarifies whether the reported F1 scores come from fine-tuning on all data, few-shot prompting with the training set, or some other procedure. If any reported LLM result was obtained from a model that saw the test instances during training or prompt construction, the result is invalid as a held-out estimate. This ambiguity is load-bearing and must be resolved with an explicit protocol description and, ideally, released code or data.","section":"Section 5.5 vs Section 7.4"},{"comment":"The Introduction's contributions list claims 'up to 10.17% F1-score improvements over baselines' and, specifically for Urdu, a '10.17% F1-score increase over baseline models.' However, Table 14 reports Urdu improvement over SVM as 5.19%, English as 8.75%, Spanish as 8.97%, and Joint as 7.32%. No reported value in Table 14 equals 10.17%. The headline improvement claim is therefore contradicted by the paper's own summary table.","section":"Section 1 vs Table 14"},{"comment":"The attention-augmented architecture is described as 'central to our model's success' and is presented as a key contribution, but there is no ablation comparing the same LLM or transformer with and without the inserted attention layer. Moreover, the paper does not explain how an external attention layer can be applied to API-based GPT-3.5 Turbo, which accepts text prompts rather than custom encoder blocks. Without an ablation or a concrete implementation description, the claim that attention layers drive the reported gains is unsupported.","section":"Section 5.4.4"},{"comment":"The error-analysis counts are internally inconsistent with the reported confusion matrices. For English, the text says '5% of GPT-3.5 Turbo's misclassifications (102 instances)' and '7% of roberta-base's errors (143 instances),' but 5% and 7% of the respective misclassification counts (265 for GPT-3.5 Turbo in Table 10) do not equal 102 or 143; these numbers instead appear to be 5% and 7% of the full 2,039-instance test set. Similar discrepancies appear for Spanish (61 vs 306 errors in Table 11) and Urdu (143 vs 387 errors in Table 12). This makes the qualitative error analysis unreliable as a description of the models' mistakes.","section":"Section 7.4 error analysis; Tables 10-13"}],"minor_comments":[{"comment":"The table displayed under Section 7.1 is captioned 'Macro F1-scores for traditional machine learning models' but contains BERT, ELECTRA, RoBERTa, and XLM-RoBERTa results, while the SVM, Random Forest, Decision Tree, and XGBoost results described in the text are not shown in that table. The table numbering and captions appear misaligned throughout Section 7.","section":"Section 7.1, Table 5"},{"comment":"The heading '6.5. Best Performance' appears in the middle of Section 7.4, after the LLM results are described. This is a section-numbering artifact that should be corrected; the content should be a subsection of Section 7.","section":"Section 7.4"},{"comment":"The author-supplied keywords contain a double comma and an empty entry: 'Social Media, , Multilingual NLP,'. This should be cleaned up.","section":"Keywords"},{"comment":"The Data Availability statement reads 'Data will be mad upon request.' This is likely a typo for 'made available'; please correct it.","section":"Data Availability"},{"comment":"The model name is given as 'LLaMA 3.1 70B' in Section 5.5 but as 'LLaMA 3.3 70B' in Table 8 and elsewhere. The version number should be consistent.","section":"Section 5.5 and Table 8"},{"comment":"Several in-text citation-name pairings do not match the reference list entries: for example, 'Ousidhoum et al.' is cited as (Siddiqui et al., 2024), 'Bahdanau et al.' and 'Vig et al.' are both cited as (Gillioz et al., 2020), and the FlashAttention reference is attributed to (Fu et al., 2022), which in the reference list is titled 'Hungry hungry hippos.' These citation mismatches need systematic correction.","section":"References"}],"recommendation":"reject","confidential_remarks":"The internal arithmetic contradiction in the confusion matrices is decisive and cannot be fixed by wording changes; the paper would need to rerun or re-report its experiments, release the data/code, and add the missing ablations to make the central claims verifiable. Additionally, the manuscript header says 'Preprint submitted to Results in Physics,' which seems inconsistent with a cs.CL submission and may indicate a journal-scope mismatch. The references also show signs of template-based citation errors that would require a full bibliography overhaul."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe real contribution here is the dataset: 10,193 tweets in English, Urdu, and Spanish, balanced labels, three native annotators per language, Fleiss' Kappa 0.821. That looks like legitimate work, and the paper earns credit for it. The benchmark across SVM, BiLSTM, transformers, and LLMs is a standard but useful exercise, and the error analysis is more detailed than most, especially the discussion of how Google Translate softens insults like 'hijo de puta' to 'jerk.' The limitations section is honest.\n\nThe problem is that the evaluation numbers don't survive contact with the paper's own tables. The stated 80/20 split gives per-language test sets of about 767 (English), 633 (Spanish), and 640 (Urdu). But the confusion matrices for the three best monolingual models each sum to 2,039, the total test set size. Only the joint matrix can plausibly contain 2,039 instances. So either the monolingual models were evaluated on the entire trilingual test set, or the matrices are misreported. Either way, the headline F1 scores (0.87, 0.85, 0.81) are not supported by the paper's evidence. That is load-bearing, not a stylistic quibble.\n\nThe other soft spots are real but secondary. Section 5.5 says LLMs were 'trained on the entire dataset' while Section 7.4 says few-shot on the test set; the wording needs to be resolved. The attention layer has no ablation, so the claim that it drives performance is an overclaim. The abstract's 'up to 10.17% improvement' does not match the table's max of 8.97%. And the data are not available, so nothing can be independently checked.\n\nWhere does this leave the paper? The dataset is a genuine contribution, and the annotation process looks careful. But the empirical results, as presented, are unreliable. I would not cite the F1 numbers, and the current version should not be published as is. If the authors release the data, redo the evaluation with correct splits, and clarify what the LLMs actually saw, this could become a useful resource. A serious editor could send it to review on the strength of the dataset alone, but the authors must be told the evaluation has to be redone. As a reader, treat the numbers as unverified.\n\nRecommendation: reject the current version, but invite a resubmission with corrected evaluation and released data.","headline":"A genuinely useful trilingual dataset undermined by an internal consistency error that invalidates the headline F1 scores as reported.","tokens_in":21122,"tokens_out":4953,"would_cite":false,"duration_ms":52370,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that translating tweets before classification lets GPT-3.5 Turbo and Qwen 2.5 72B detect hate speech in English, Spanish, and Urdu at macro-F1 scores of 0.87, 0.85, and 0.81, and 0.88 on the pooled set, beating SVM…","keywords":["multilingual hate speech detection","translation-based approach","low-resource Urdu","large language models","few-shot learning","trilingual dataset","macro F1","attention layer"],"falsifier":"Re-run the evaluation with a strict protocol: an 80/20 split applied separately within each language, few-shot examples drawn only from the training split, and the test set withheld from any fine-tuning or prompt development; then check the arithmetic — an 80/20 split of 3,834 English tweets yields about 767 test tweets, whereas the paper's English confusion matrix sums to 2,039, so per-language test partitions need to be reconstructed and verified. If per-language F1 falls back to the SVM baseline range under a clean split, the claimed gains do not survive.","tokens_in":20092,"feed_emoji":"","tokens_out":12854,"duration_ms":123690,"temperature":0.7,"pith_summary":"The paper's thesis is that translation can level the playing field for hate speech detection: translate high-resource and low-resource tweets into a shared form, then let large language models do the classifying. To test this, the authors built a trilingual dataset of 10,193 keyword-filtered tweets in English, Urdu, and Spanish, labeled by native speakers with substantial agreement (Fleiss' κ = 0.821). They report that GPT-3.5 Turbo reaches a macro F1 of 0.87 on English and 0.85 on Spanish, while Qwen 2.5 72B reaches 0.81 on Urdu and 0.88 on the pooled trilingual set, beating TF-IDF/SVM baselines (0.77–0.82) by 5 to 9 percent. If the results hold, the payoff is that Urdu, a language with almost no annotated hate-speech resources, can be brought within reach of English-level detection quality using off-the-shelf translation and prompting.","feed_headline":"Translation-first pipeline lifts hate speech F1 to 0.88","feed_subtitle":"A trilingual tweet set shows LLMs beating SVM in English, Spanish, and low-resource Urdu.","key_machinery":"Two objects carry the argument. The first is the new trilingual corpus: 10,193 tweets (3,834 English, 3,197 Urdu, 3,162 Spanish), balanced between hateful and not-hateful labels, each tweet annotated by three native speakers per language with majority voting and a global Fleiss' κ of 0.821. The second is the translation-plus-attention pipeline, which standardizes all tweets through the Google Translate API into unified per-language and joint corpora and then applies what the paper calls attention-augmented layers: multi-head self-attention with 12 heads plus a Linformer-style sparse low-rank projection, described as a contextual filter prepended before transformer or LLM encoders. Classical models receive TF-IDF features, deep models receive FastText or GloVe embeddings, and the LLMs are deployed in a few-shot setting that yields the headline numbers.","core_discovery":"The central claim is that a translation-based pipeline, augmented with attention layers and large language models, closes most of the gap between high-resource and low-resource hate speech detection. Concretely, the paper reports macro-F1 scores (the average of the hateful and not-hateful class F1s) of 0.87 for English and 0.85 for Spanish with GPT-3.5 Turbo, and 0.81 for Urdu and 0.88 for the joint trilingual model with Qwen 2.5 72B, against SVM baselines of 0.80, 0.78, 0.77, and 0.82. The authors interpret the joint model's outperformance of every monolingual model as evidence that cross-lingual training strengthens detection, and they trace the residual Urdu weakness to code-mixing, Roman-script variants, and translation errors that soften slang insults. Their error analysis attributes part of the false negatives to translations such as rendering 'hijo de puta' as 'jerk,' which motivates their proposed next step of slang-aware machine translation.","pith_inferences":["Editorial inference: the method is model-agnostic at its core, so if translation is the active ingredient, future or cheaper LLMs can be swapped in without new annotation, and any improvement in slang-aware machine translation should improve Urdu detection directly.","Editorial inference: the paper never runs the key ablation of querying the same LLMs on untranslated Urdu text versus translated text, so it does not isolate how much of the 0.81 F1 comes from translation rather than the models' own multilingual competence.","Editorial inference: the joint-model advantage hints at a scaling effect — adding more low-resource languages to the translated pool might keep raising per-language F1 — but three languages are too few to establish that trend."],"forward_implications":["If the reported numbers are clean, a single translation-based pipeline brings Urdu hate speech detection to macro F1 0.81, a 5.19 percent gain over the SVM baseline and far above the best monolingual transformer in the paper (BERT at 0.50).","The joint trilingual model (0.88) beating every monolingual model implies that pooling translated languages helps rather than hurts, so folding more languages into the same translated pool is a direct corollary the paper itself proposes.","Model choice matters per language: GPT-3.5 Turbo leads in English and Spanish while Qwen 2.5 72B leads in Urdu and the joint set, pointing to open-weight models as the stronger route for low-resource deployment.","The error analysis's finding that translation softens insults (for example, 'hijo de puta' rendered as 'jerk') identifies slang-aware machine translation as the concrete bottleneck to fix next."],"supporting_citations":[{"why":"Supplies the translation-based cross-lingual method (English–Spanish alignment for mBERT) that this paper extends to a trilingual pipeline.","marker":"Ranasinghe and Zampieri, 2020"},{"why":"Defines the Urdu gap: prior Perso-Arabic Urdu hate detection work this paper positions itself against.","marker":"Ali et al., 2022"},{"why":"Prior Roman-Urdu LSTM hate speech detection that motivates the code-mixing challenge and serves as the low-resource baseline context.","marker":"Bilal et al., 2022"},{"why":"Provides BERT, the transformer foundation the paper benchmarks and augments with attention layers.","marker":"Devlin et al., 2019"},{"why":"Source of the attention mechanism the paper's multi-head and sparse attention layers are built on.","marker":"Bahdanau et al., 2014"},{"why":"The bilingual Hinglish translation-plus-transformer approach this trilingual translation-based design directly extends.","marker":"Biradar et al., 2022"},{"why":"Documents Urdu tweet detection and dataset scarcity, the gap the new trilingual dataset fills.","marker":"Mehmood et al., 2022"}],"fun_headline_variants":["Translation-based LLM pipeline beats SVMs on hate speech in 3 languages","Cross-lingual translation narrows hate speech detection gap for Urdu","Joint multilingual model gets 0.88 F1 on hate speech, besting all single languages","Translation+attention lifts hate speech F1 for English, Urdu, Spanish","Trilingual hate speech data set and translation LLMs hit 0.88 macro F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline scores assume the LLMs were evaluated on test data they had never seen, because Section 5.5 says the LLMs were “trained on the entire dataset” while Section 7.4 says they were “evaluated on the test set using a few-shot learning approach,” and the paper never reconciles these two statements, so the reported 0.81–0.88 F1 results may rest on test-set contamination.","fun_headline_variants_meta":{"raw":{"variants":["Translation-based LLM pipeline beats SVMs on hate speech in 3 languages","Cross-lingual translation narrows hate speech detection gap for Urdu","Joint multilingual model gets 0.88 F1 on hate speech, besting all single languages","Translation+attention lifts hate speech F1 for English, Urdu, Spanish","Trilingual hate speech data set and translation LLMs hit 0.88 macro F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00161,"raw_usage":{"total_tokens":6532,"prompt_tokens":1187,"completion_tokens":5345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":803,"completion_tokens_details":{"reasoning_tokens":5240}},"tokens_in":803,"tokens_out":5345,"duration_ms":36313,"temperature":1.0,"reasoning_tokens":5240,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:18:17.480226+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation with a strict protocol: an 80/20 split applied separately within each language, few-shot examples drawn only from the training split, and the test set withheld from any fine-tuning or prompt development; then check the arithmetic — an 80/20 split of 3,834 English tweets yields about 767 test tweets, whereas the paper's English confusion matrix sums to 2,039, so per-language test partitions need to be reconstructed and verified. If per-language F1 falls back to the SVM baseline range under a clean split, the claimed gains do not survive.","supporting_citations":[],"review_version":1}