{"id":"839e198e-e49e-4892-9a48-44e447b861f5","arxiv_id":"2506.03458","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Models trained on Arabic or Indonesian detect Persian hate speech far better than English-trained models, which the authors attribute to cultural similarity.","lead":"This paper benchmarks toxic language detection in Persian across fine-tuning, zero/few-shot, distant supervision, and cross-lingual transfer, finding that transfer from Arabic and Indonesian helps far more than from English for hate detection. It is a useful data point for building safer Persian moderation tools and for the broader question of when cultural context should guide training data selection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transfer-learning comparison is confounded by task setup: English is a 3-class model, Arabic 2-class, Indonesian 1-class, so higher F1 for 'culturally similar' languages may reflect task simplicity, not culture.","rationale":"The paper's central finding is the cross-lingual transfer comparison in §5.3/Table 3, and its interpretation in §6.4. The reader identified dataset content differences as the weakest assumption; I go further: the comparison is invalidated by the unequal label spaces across source languages. Section 4.3 explicitly says Indonesian tests were restricted to hate and Arabic to hate and vulgar, while English covered all three. The macro F1 values (69 vs 82 vs 86) are not computed over the same categories for all models, and the per-class Hate F1 comes from models solving different-sized tasks. A binary hate detector almost always achieves higher F1 than a three-class toxic detector on the same underlying data because the decision boundary is simpler; thus the 57 vs 81 vs 86 ordering could be entirely a task-formulation effect. The paper's own admission in §6.4 that the English dataset's hate topics (sexual orientation, ethnic groups) diverge from Persian topics (politics, religion) adds a second uncontrolled variable. Together these confounds make the 'culture' label an over-attribution. The proposed concrete test—binary hate-only training for all three languages on matched-size, topic-balanced subsets—would either salvage the cultural explanation or reveal it to be an artifact. Since the paper otherwise provides a solid benchmark and the gap might persist after controls, the appropriate verdict remains conditional: the claim is plausible but unproven under the current design.","tokens_in":21376,"tokens_out":4882,"duration_ms":52440,"concrete_test":"Retrain all transfer models as binary hate-vs-not classifiers with the same architecture (Llama 3-base), same train size (8,050), and balanced labels, using English, Arabic, and Indonesian source data separately. To neutralize topic mismatch, subsample English hate instances to match the Persian test set's topic distribution (or re-annotate a matched Persian-topic sample), then evaluate on the PHATE hate test split. If English binary hate F1 remains ≥20 points below Arabic/Indonesian, the culture claim is supported; if English reaches parity, the original gap was an artifact of task complexity and topic distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (intro, RQ4, §6.4) rests on Table 3's transfer rows: Llama 3–En scores Hate F1=57, Vulgar F1=77, Violence F1=73; Llama 3–Ar scores Hate F1=81, Vulgar F1=82; Llama 3–Id scores Hate F1=86. The headline assertion that culturally similar languages beat large-scale English is read directly off these numbers. However, the models are not comparable: §4.3 states English data covers hate, vulgarity, and violence (Kennedy et al. 2020), Arabic covers hate and vulgar, and Indonesian covers hate only (no Arabic violence, no Indonesian vulgar/violence). The English model is therefore a three-class (multi-label) classifier, Arabic a two-class, Indonesian a binary hate classifier. F1 for the Hate class is not comparable across these different label spaces; binary hate detection is an easier task than discriminating three toxic categories, which alone can produce the observed ordering (English 57 vs Arabic 81 vs Indonesian 86). The paper balances dataset size and label balance but never controls the number of classes. A second confound is acknowledged by the authors themselves in §6.4: the English hate dataset focuses on sexual orientation and ethnic groups, while Persian hate centers on politics and religion; this topic mismatch is also not controlled. Both confounds mean the culture explanation is underdetermined by the current experiment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses Persian toxic language detection using the PHATE dataset. It benchmarks fine-tuning of several LLMs (ParsBERT, XLM-R, mT5, Llama 3 variants, GEMMA 2, Dorna2-Llama), GPT-3.5 Turbo zero/few-shot prompting, a new distant-supervision Persian lexicon, and cross-lingual transfer learning from English, Arabic, and Indonesian with Llama 3. The central claim, stated in the Introduction and RQ4, is that transfer from culturally similar languages (Arabic, Indonesian) outperforms transfer from large-scale English data. The paper also introduces an enriched Persian training set and reports detailed per-class F1 results.","tokens_in":21661,"tokens_out":7253,"duration_ms":73162,"significance":"If the central claim were established, the paper would make a practically important point: for low-resource toxicity detection, selecting source languages by cultural proximity may matter more than source dataset scale. The paper contributes a new Persian toxic lexicon, a distant-supervision pipeline, and a broad comparison of models on PHATE. It also uses a held-out test set and attempts to balance source dataset sizes. However, the transfer-learning comparison is confounded by differing label spaces, topic distributions, and the absence of significance testing, so the cultural interpretation is not yet supported by the reported experiments.","major_comments":[{"comment":"The transfer-learning comparison is not controlled for the number of label classes in the source data. English source data (Kennedy et al., 2020) cover hate, vulgarity, and violence; Arabic (Mubarak et al., 2022) cover hate and vulgarity; and Indonesian (Ibrohim and Budi, 2019) cover hate only. The English model is therefore a three-class multi-label classifier, the Arabic model a two-class classifier, and the Indonesian model a binary hate classifier. The reported Hate F1 values are not comparable across these different label spaces, and the macro-F1 (Fm) column averages over 3, 2, and 1 classes respectively. This task-simplicity difference alone could produce the observed ordering (English Hate F1=57, Arabic=81, Indonesian=86).","section":"§4.3, Table 3"},{"comment":"The authors' own analysis identifies a second confound: the English hate dataset focuses on sexual orientation and ethnic groups, while Persian hate centers on politics and religion. This topic mismatch is not controlled in the comparison. Because hate-target distribution is correlated with culture, the observed advantage of Arabic and Indonesian over English could reflect topic overlap with the Persian test set rather than cultural proximity per se. The paper should explicitly match or control for hate-target/topic distribution across the source languages.","section":"§6.4"},{"comment":"No variance estimation or significance testing is reported for any result. The paper does not state random seeds or repeated runs, and differences of 1 point (e.g., Llama 3–Id and Llama 3–Id+Fa both F1=86; Llama 3–Ar+In+Fa F1=91 vs. Llama 3–Ar+In F1=93) are interpreted as meaningful. The central comparison requires multiple runs with confidence intervals or a significance test before claims such as 'superior performance' are warranted.","section":"§4.2, §5.3, Table 3"},{"comment":"The cultural-similarity variable is asserted rather than measured: Arabic and Indonesian are labeled culturally similar to Persian and English culturally distinct based on narrative description, with no operationalized metric (e.g., cultural dimensions or the cultural-feature approach from Zhou et al. 2023b). Combined with the label-space and topic confounds, the central claim that 'culture matters' is underdetermined by the experiment as reported.","section":"§4.3"}],"minor_comments":[{"comment":"The heading 'Data Enrchiment via Distant Supervision' contains a typo; it should be 'Data Enrichment via Distant Supervision'.","section":"§6.3 heading"},{"comment":"In the Llama Models paragraph, 'specefic' is a typo for 'specific'.","section":"§5.2"},{"comment":"Figure 5 shows a prompt for binary Violence/Neutral classification, but §4.1 describes prompts for both multi-class and binary GPT configurations; please clarify which prompt(s) were used for each configuration.","section":"Figure 5"},{"comment":"Table 1 lists GPT 3.5 Turbo as 175B parameters citing Brown (2020), but that citation refers to GPT-3; the parameter count for GPT-3.5 Turbo is not established in the cited work.","section":"Table 1"},{"comment":"The text states that overlapping labels in PHATE were removed for distinct multi-class categorization; please specify exactly how overlapping labels were resolved and whether the same modified labels were used for train and test splits, since this affects comparability with the original PHATE benchmark.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The paper tackles an important problem and the lexicon contribution is useful, but the central cultural-claim is not yet supported because the transfer comparison mixes label spaces and topic distributions. These issues are addressable with a redesigned experiment—matching label spaces across all source languages, adding topic controls, and reporting variance. I would not recommend rejection, but the revision must either provide matched comparisons or substantially qualify the claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What should you know: the paper is a solid empirical sweep of Persian toxic-language detection methods, but its headline cultural-transfer claim is underdetermined by its own experimental design. The authors compare transfer from English, Arabic, and Indonesian to Persian without controlling for the number of classes each source model is trained to predict: English gets three classes, Arabic two, Indonesian one. That alone can reproduce the observed F1 ordering (English Hate 57, Arabic 81, Indonesian 86) because binary hate detection is easier than discriminating three toxic categories. The paper acknowledges in §4.3 that Indonesian covers hate only and Arabic lacks violence, but never treats this as a confound; it balances size and label distribution but not label space. The topic mismatch (English hate targets sexual orientation and ethnicity, Persian hate targets politics/religion) is acknowledged in §6.4 and is a second uncontrolled variable.\n\nWhat is actually new: a Persian distant-supervision lexicon (604 keywords) and an enriched training set, plus fine-tuned baselines on PHATE, and the first transfer-learning evidence for Persian. The error analysis (tables 4–7) is genuinely useful, especially the failure modes with idiomatic and sarcastic hate. The limitations section is honest about the lexicon's temporal drift and the small set of source languages.\n\nSoft spots beyond the confound: no significance testing or variance across seeds/subsets, no released lexicon or enriched data, and the 'cultural similarity' variable is operationalized with only three languages, so cultural similarity is entangled with every other difference between those datasets. The paper's own analysis of Indonesian vs Arabic differences (profession vs religious hate) suggests content composition, not culture, is doing much of the work.\n\nWho it's for: researchers working on low-resource toxicity detection and multilingual transfer. It deserves a serious referee because the practical question—whether culturally proximate source data beats large English datasets—is important, and the paper gathers the right kind of data even if the analysis currently overreads it. A reviewer should ask for a properly controlled comparison (all binary, same taxonomy, or translated/re-annotated sources), variance reporting, and artifact release.","headline":"A useful Persian toxicity benchmark undermined by an uncontrolled cross-lingual comparison; the cultural-transfer claim is plausible but not proven.","tokens_in":22166,"tokens_out":1775,"would_cite":false,"duration_ms":20047,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For Persian toxicity detection, cultural proximity beats dataset scale in cross-lingual transfer.","keywords":["Persian toxic language detection","cross-lingual transfer learning","cultural similarity","distant supervision","hate speech detection","LLM fine-tuning","PHATE dataset","low-resource language processing"],"falsifier":"Re-annotate the English, Arabic, and Indonesian training subsets with the exact PHATE label definitions, then rerun the equal-size transfer experiments; if English then performs as well as Arabic or Indonesian, the apparent cultural effect is an artifact of annotation or topic differences rather than culture.","tokens_in":21161,"feed_emoji":"🗨️","tokens_out":5209,"duration_ms":52611,"temperature":0.7,"pith_summary":"This paper asks whether Persian, a language with scarce toxicity-annotation resources, can borrow labelled data from other languages, and whether the source language's culture matters. The authors fine-tune the same base model on equal-size Arabic, Indonesian, and English hate-speech datasets and evaluate on Persian. They find that Arabic and Indonesian transfer far better than English, even though the English datasets are much larger. They conclude that cultural similarity to Persian, not source-dataset scale, drives cross-lingual transfer success. The paper also shows that a distant-supervision lexicon built by Persian speakers can substantially improve BERT-based models on the PHATE benchmark.","feed_headline":"Culture, not data size, drives Persian toxicity transfer","feed_subtitle":"Fine-tuning on Arabic or Indonesian beats English for Persian hate detection, despite far larger English data.","key_machinery":"The method is a controlled cross-lingual transfer setup: a single base model, Llama 3, is fine-tuned on balanced, equal-size subsets of 8,050 tweets, half hate and half non-hate, drawn from Arabic, Indonesian, and English datasets, optionally combined with the Persian PHATE training set, and then evaluated on the same Persian test split. Equalizing size and label balance isolates the variable of interest, which country's culture the training text comes from, so that any performance difference is attributed to cultural or content proximity rather than data volume. A second mechanism is distant supervision: a Persian toxic lexicon of 604 keywords, built by native speakers and expanded with FastText, is used to auto-label 3,291 tweets, enriching the Persian training set.","core_discovery":"The central discovery is that cross-lingual transfer for Persian toxic language detection succeeds when the source language comes from a culturally overlapping country. Using Llama 3-base fine-tuned on equal-sized hate datasets, the model reaches 0.86 F1 when trained on Indonesian alone and 0.82 from Arabic alone, beating the 0.69 from English, despite English having orders of magnitude more data. Combining Arabic and Indonesian reaches 0.93 F1 on Persian hate, while adding Persian data slightly dilutes that result. The pattern repeats for vulgarity, where Arabic transfers better than English. The authors attribute the advantage to shared hate targets, such as religion, politics, and regional controversies, and to complementary strengths: Arabic transfers sociopolitical hate, while Indonesian transfers individual-directed hate and long mixed-sentiment texts.","pith_inferences":["If the cultural-proximity result generalizes, toxicity detection systems for other low-resource languages should prioritize sourcing training data from culturally overlapping speech communities, such as regional dialects or diaspora language data, before investing in massive high-resource datasets.","A testable corollary is that cultural similarity could be quantified by measuring the overlap of hate targets or controversial topics between source and target datasets; the paper's ranking, Indonesian above Arabic above English, provides a concrete data point for building such a metric.","The slight drop when Persian data is added to the Arabic-plus-Indonesian combination suggests that simple data mixing is not always beneficial, and the relationship between target-language data quantity and transfer performance deserves separate investigation.","Since English violence data did transfer useful context for Persian violence detection, the cultural-similarity effect may be category-dependent; future work could test whether violence detection is less culturally bound than hate or vulgarity."],"forward_implications":["For Persian hate detection, fine-tuning Llama 3 base on Arabic or Indonesian data yields the strongest transfer results, with best F1 scores between 0.86 and 0.93, clearly outperforming both fine-tuning on Persian-only data and zero/few-shot GPT-3.5 Turbo.","The enriched distant-supervision dataset raises ParsBERT's macro F1 from 0.57 to 0.69, showing that a modest lexicon-based auto-labeling pipeline can close part of the low-resource gap for BERT-scale models.","Dataset scale is not the main lever for cross-lingual transfer in this setting: equal-size culturally close sources outperform the much larger English source.","Combining Arabic and Indonesian data is complementary and yields the single best transfer result for hate detection, while adding Persian data to that combination slightly reduces performance.","Translating Persian training data into English before fine-tuning degrades performance, indicating that fine-tuning on the original language preserves linguistic cues needed for toxicity classification."],"supporting_citations":[{"why":"Supplies the PHATE Persian dataset, the label definitions, and the train/test splits used for all evaluations and transfer targets.","marker":"(Delbari et al., 2024)"},{"why":"Provides the large Arabic hate and vulgar datasets used as a culturally similar transfer source.","marker":"(Mubarak et al., 2022)"},{"why":"Supplies the Indonesian multi-label hate dataset used as the culturally similar but linguistically distinct transfer source.","marker":"(Ibrohim and Budi, 2019)"},{"why":"Provides the English hate dataset that serves as the high-resource, culturally distant transfer source.","marker":"(Kennedy et al., 2020)"},{"why":"ParsBERT is the baseline Persian model whose fine-tuning and enrichment performance anchor the comparison.","marker":"(Farahani et al., 2021)"},{"why":"Dorna2-Llama3 Instruct is the Persian-tuned model that outperforms other fine-tuned models on the task.","marker":"(PartAI, 2024)"},{"why":"Llama 3-base is the model used for all transfer learning experiments.","marker":"(Dubey et al., 2024)"},{"why":"Distant supervision methodology, adapted here to build the Persian toxic lexicon and enriched dataset.","marker":"(Magdy et al., 2015)"},{"why":"Motivates the choice of English prompts and the finding that generative models perform poorly on Persian toxicity tasks.","marker":"(Abaskohi et al., 2024)"}],"fun_headline_variants":["Persian hate detection: culture beats data size","Arabic, Indonesian transfer better than English for Persian toxicity","Culture-specific transfer improves Persian toxic language detection","Shared culture, not big data, drives Persian toxicity models","Persian toxicity: cultural similarity trumps English data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper treats the three source datasets as comparable apart from country culture, but they come from different annotation projects with different label definitions, topic coverage, and time periods; if those differences, rather than culture, cause the performance gap, the central claim would not hold.","fun_headline_variants_meta":{"raw":{"variants":["Persian hate detection: culture beats data size","Arabic, Indonesian transfer better than English for Persian toxicity","Culture-specific transfer improves Persian toxic language detection","Shared culture, not big data, drives Persian toxicity models","Persian toxicity: cultural similarity trumps English data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1232,"prompt_tokens":813,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":345}},"tokens_in":429,"tokens_out":419,"duration_ms":5165,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:02:19.972840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the English, Arabic, and Indonesian training subsets with the exact PHATE label definitions, then rerun the equal-size transfer experiments; if English then performs as well as Arabic or Indonesian, the apparent cultural effect is an artifact of annotation or topic differences rather than culture.","supporting_citations":[],"review_version":1}