{"id":"8f71db32-9c3a-4a19-99a2-795adfd74bd3","arxiv_id":"2412.11691","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"New parallel text detoxification corpora for five languages, a GPT-4-based cross-lingual analysis of toxicity features, and a cluster-conditioned chain-of-thought prompting method that yields a marginal average gain.","lead":"This paper adds new parallel detoxification datasets for German, Chinese, Arabic, Hindi, and Amharic, and uses GPT-4 to analyze how toxicity is expressed across nine languages. The authors then test a chain-of-thought prompting method that uses clusters of toxicity styles to improve detoxification, with mixed results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The STA toxicity classifier is never validated per language; Chinese human references score 0.266 and toxic duplicates score 0.426 in Amharic, so the J-score comparison underpinning the CoT method's advantage is not trustworthy.","rationale":"The dataset contribution in Section 3 is genuine and independently valuable: the paper adds manually curated parallel detoxification data for five languages with detailed annotation procedures, and the descriptive analysis in Section 4 is a reasonable first pass at cross-lingual toxicity explainability. The load-bearing weakness is the evaluation pipeline for the headline method. The reader's weakest_assumption identifies the STA classifier as the key risk, and the manuscript provides internal evidence that this risk is real: Chinese human references score only 0.266 on STA, while the Duplicate baseline scores 0.426 on Amharic and 0.388 on Arabic. These numbers are difficult to reconcile with a valid per-language non-toxicity classifier. Since every J score in Table 4 is the product of STA, SIM, and ChrF1, a miscalibrated STA can change the ranking of methods, not merely the absolute values. I do not escalate to REJECT because the corpora and descriptive findings remain useful regardless of the method comparison, and because the CoT claim could be salvaged by re-evaluating with a validated toxicity signal. The proposed concrete test is cheap and decisive: if the STA classifier passes per-language validation, the Table 4 results survive; if it fails, the method comparison must be redone before the central claim can be accepted. The reader's CONDITIONAL verdict is therefore the right level of caution, and my read does not change it.","tokens_in":25156,"tokens_out":5186,"duration_ms":54425,"concrete_test":"Evaluate the STA classifier against held-out gold labels for each of the nine languages: take, for example, 100 test-set human detoxified references and 100 original toxic sentences per language, obtain independent native-speaker toxicity judgments using the paper's toxicity definition, and compare them with the XLM-R STA scores, reporting accuracy, F1, and per-language correlation. Then recompute J for GPT-4 few-shot and GPT-4 CoT using either a corrected or recalibrated STA classifier, or alternatively using human toxicity ratings on a random subset of 100 test outputs per language. If the Chinese human references still score below 0.5 while humans mark them non-toxic, or if the Duplicate baseline remains above 0.4 in Amharic or Arabic, the Table 4 ranking should be treated as unsupported; if the CoT-vs-few-shot advantage survives under the corrected metric, the central claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central method claim (Section 7) rests on the joint score J = STA * SIM * ChrF1, in which STA is the only toxicity signal. STA is produced by an XLM-R-large classifier fine-tuned on 5,000 subsampled examples per language (Section 5), but no per-language validation, accuracy, or calibration is reported. Two internal red flags show that this classifier is not consistently measuring non-toxicity. In Table 9, the Chinese human detoxified references receive STA = 0.266, meaning the model labels roughly 73% of gold non-toxic rewrites as toxic, while GPT-4 CoT on the same language receives STA = 0.716. In Table 10, the Duplicate baseline, which outputs the original toxic sentence unchanged, receives STA = 0.426 for Amharic; in Table 9 it receives STA = 0.388 for Arabic. Thus, for some languages the STA classifier calls toxic inputs non-toxic about 40% of the time, and for Chinese it fails to recognize human detoxifications as non-toxic. Because J is multiplicative, a biased STA directly distorts every comparison in Table 4, including the claimed CoT advantage over few-shot prompting. The reported 98% expert agreement concerns GPT-4's descriptive feature annotations, not the validity of the STA classifier, so it does not mitigate this issue. Until the STA classifier is shown to behave sensibly per language, the headline empirical claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends parallel text detoxification corpora to German, Chinese, Arabic, Hindi, and Amharic, reports a detailed annotation pipeline for each language, and uses GPT-4 to annotate descriptive features of toxic and non-toxic sentences across nine languages. On the basis of this feature analysis, the authors build per-language K-means clusters and propose a Chain-of-Thought (CoT) prompting method that identifies a test sentence's cluster and supplies a representative detoxification example in the prompt. The central empirical claim is that this cluster-conditioned CoT method achieves the best average joint score J = STA x SIM x ChrF1 and the best STA scores against several unsupervised, supervised, and few-shot baselines.","tokens_in":25560,"tokens_out":4240,"duration_ms":41059,"significance":"If the claims hold, the resource contribution is genuinely valuable: manually curated parallel detoxification data for five non-European languages, with public data and code, extends the reach of supervised detoxification substantially. The descriptive feature analysis across nine languages is also a useful reference for cross-lingual work on toxicity. However, the empirical comparison that supports the CoT method is currently not trustworthy. The STA toxicity classifier used in the joint metric is never validated per language, and internal tables show it disagrees sharply with human gold labels for Chinese and Amharic. In addition, the clusters and prompt examples used in the CoT method are derived from the full 1,000-pair set, including the 600 test instances, so the evaluation is not a clean test of generalization. These issues are fixable with additional experiments and reporting, but they are load-bearing for the paper's central claim.","major_comments":[{"comment":"The STA component of the joint score is computed by an XLM-R-large classifier fine-tuned on 5,000 subsampled examples per language, but the paper reports no per-language validation, accuracy, F1, or calibration of this classifier. The internal results show that the classifier is not consistently measuring non-toxicity: in Table 9, Chinese human detoxified references receive STA 0.266, meaning most gold non-toxic rewrites are deemed toxic, while GPT-4 CoT on the same language receives STA 0.716; in Table 10, the Duplicate baseline for Amharic receives STA 0.426, so unchanged toxic inputs are called non-toxic nearly half the time. Because J is multiplicative, such a biased STA directly distorts every comparison in Table 4, including the claimed CoT advantage over few-shot prompting. The 98% expert agreement reported in Section 4.1 concerns GPT-4's descriptive feature annotations and not the validity of the STA classifier, so it does not mitigate this concern.","section":"Section 5, Tables 4, 9, 10"},{"comment":"The CoT method is evaluated on test instances that were used to construct the method. Feature extraction and K-means clustering are run on all 1,000 pairs per language, and the 600 test sentences are a subset of these 1,000 pairs. The cluster definitions, the choice of K=3, and the representative cluster examples embedded in the CoT prompt are therefore informed by the same inputs and references that are later scored in Table 4. This is a form of test-set leakage: the comparison does not measure how the method would perform on unseen inputs. The authors should derive clusters from the 400 training pairs only, or otherwise exclude the test set from all prompt and cluster construction, and then re-run the evaluation.","section":"Sections 3.6, 4.1, 4.5, Appendix A.3"},{"comment":"The reported advantage of GPT-4 CoT over few-shot prompting is very small on average (0.331 vs 0.324) and negative for English (0.326 vs 0.475), yet no significance tests, confidence intervals, or multiple-run variance are reported. Appendix C.5 states that GPT-4 was run with default hyperparameters including temperature=1.0, and no number of repeated inferences is given. With a single stochastic run per input, the observed differences may be sampling noise. Paired bootstrap tests or multiple runs with confidence intervals are needed before claiming that CoT 'achieved the highest scores across all approaches'.","section":"Section 7, Table 4, Appendix C.5"},{"comment":"The feature-extraction analysis relies on GPT-4 annotations for the full 1,000-pair set, and the paper states that experts agreed with GPT-4 in 98% of cases. However, the size of the expert-reviewed sample, the number of annotators per language, and the agreement measure are not reported. Since the method's clusters and prompts are built directly on these annotations, the reliability of this 98% figure matters for the validity of the CoT approach. The authors should specify the validation protocol and report per-language agreement.","section":"Section 4.1, Table 7, Appendix E"}],"minor_comments":[{"comment":"The text reports 'C20 per hour' and 'C7.65 above the minimum wage'; the 'C' appears to be a rendering error for the euro sign, which should be corrected.","section":"Section 3, Annotators Compensations"},{"comment":"'Two annotators ... were evolved in the main annotation' should read 'were involved in the main annotation'.","section":"Section 3.3.2, Amharic Annotation Process"},{"comment":"The text says each sample was transcribed by only one annotator, while Table 1 lists two annotators per sentence for German; the apparent discrepancy between the prose and the table should be clarified.","section":"Section 3.1.2, German Annotation Process"},{"comment":"The phrase 'top_k=0.0' is not a standard GPT-4 API parameter in the same sense as temperature and may confuse readers; the paper should either explain the sampling configuration precisely or omit the unsupported parameter.","section":"Appendix C.5"},{"comment":"The related-work section would benefit from a short paragraph positioning the new corpora against the MultiParaDetox and TextDetox CLEF-2024 shared task, since the data are already described as the basis of that task in footnote 3.","section":"Section 2, Related Work"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution and the descriptive analysis are likely to be the most durable parts of this submission. In my view, the method-oriented claims should not be accepted as they stand: the unvalidated STA classifier and the test-set leakage both concern the central empirical comparison, and they can only be addressed by re-running the evaluation with proper separation and per-language classifier validation. If the authors are willing to reframe the paper primarily as a resource and analysis paper, with the CoT experiments clearly marked as preliminary, that would also be a defensible path."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the new parallel corpora for five languages are a genuine resource, and the explainability analysis is a reasonable first pass. The CoT method claim, though, is not supported by the evidence as presented.\n\nWhat's new: the paper extends the ParaDetox collection pipeline to German, Chinese, Arabic, Hindi, and Amharic, with detailed annotation documentation and publicly released data and code. That is a real contribution, and it will be useful to anyone working on multilingual style transfer or toxic speech mitigation. The GPT-4 feature analysis across nine languages is a new application, and the qualitative findings (e.g., culture-specific toxic keywords, common edit types) are plausible and mostly consistent with the examples shown.\n\nThe soft spots are in the evaluation. The joint score J relies on STA, a fine-tuned XLM-R toxicity classifier whose per-language behavior is never validated. The paper's own appendix shows two red flags: Chinese human detoxified references receive STA 0.266, meaning the classifier calls about three-quarters of gold rewrites toxic, while the Amharic Duplicate baseline receives STA 0.426, meaning toxic inputs are scored as non-toxic over 40% of the time. If STA is inconsistent across languages, every J comparison in Table 4 is distorted. The stress-test note is right about this.\n\nSecond, the CoT improvement over few-shot prompting is tiny on average (0.331 vs 0.324) and goes backwards on English and Amharic. No significance tests, no error bars, and GPT-4 is run once with temperature 1.0. That difference could easily be noise. Third, there is a circularity problem: the cluster definitions and K=3 come from GPT-4 feature annotations on all 1,000 pairs, including the 600 test sentences. The method is partly fit to the test set, and there is no held-out validation.\n\nNone of this kills the dataset contribution. But the paper should not be published with the current method claim. It needs a per-language validation of the STA classifier, significance testing or multiple runs, and a held-out cluster selection scheme. The authors were candid about some limitations (closed-source model, English-only explanations), which I credit.\n\nWho this is for: researchers in text style transfer and multilingual NLP who want parallel detoxification data for non-European languages. Cite the dataset, not the CoT method. I would send it to peer review because the resource is worth refereeing and the evaluation issues are fixable, but I would expect major revision.","headline":"The five-language parallel detoxification corpora are a real resource worth citing; the CoT method claim is not supported by the current evaluation and needs per-language validation of the toxicity classifier, significance testing, and held-out cluster selection.","tokens_in":26077,"tokens_out":2071,"would_cite":true,"duration_ms":19992,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that manually curated parallel corpora in German, Chinese, Arabic, Hindi, and Amharic, combined with a cluster-aware Chain-of-Thought prompt, make text detoxification both more multilingual and more explainable.","keywords":["text detoxification","text style transfer","multilingual parallel corpora","chain-of-thought prompting","explainable AI","toxic language","LLM prompting","ParaDetox"],"falsifier":"Give native speakers a blind sample of 200 GPT-4 CoT outputs and 200 GPT-4 few-shot outputs per language and ask them to judge toxicity and meaning preservation. If human raters do not rate CoT outputs as non-toxic more often, or if the per-language ranking changes, the reported STA-based Joint advantage is an artifact of the toxicity checker. A direct pilot: measure the checker's agreement with human toxicity judgments on the human detoxification references for Chinese and Amharic, where the checker's scores diverge most.","tokens_in":24937,"feed_emoji":"🧹","tokens_out":29311,"duration_ms":242254,"temperature":0.7,"pith_summary":"This paper tries to establish two related things: that parallel text detoxification data can be produced by hand for German, Chinese, Arabic, Hindi, and Amharic, and that explaining toxicity with an LLM can be turned into a better detoxification prompt. The authors collect and release manually curated toxic-to-neutral sentence pairs for the five new languages, then use GPT-4 to label every sentence in the resulting nine-language corpus with a toxicity level, tone, language style, implied sentiment, and negative connotations. Those labels are clustered into three repair strategies per language, and the cluster identity is fed into a Chain-of-Thought prompt that first diagnoses the input and then rewrites it using a representative example. In the paper's automatic evaluation, this cluster-conditioned Chain-of-Thought prompt reports the highest style-transfer accuracy among compared detoxification systems and the highest average joint score, which combines non-toxicity, content preservation, and fluency. If the result holds, detoxification becomes a more predictable, explainable operation for languages that previously had no parallel training data.","feed_headline":"GPT-4 prompt with cluster hints tops multilingual detoxification","feed_subtitle":"Manually curated data for German, Hindi, Amharic, Arabic, Chinese plus cluster-guided prompts lift detox scores.","key_machinery":"The load-bearing mechanism is the cluster-conditioned Chain-of-Thought (CoT) prompt. First, GPT-4 extracts five descriptive features (toxicity level, tone, language type, implied sentiment, negative connotations) for toxic and detoxified sentences; the toxic sentences' feature vectors are validated against native speakers, then one-hot encoded and clustered by k-means into three groups per language. The three clusters correspond to repair strategies—removing profanities, rephrasing condescending or biased language, and lightly adjusting informal text—and each cluster carries a human-readable explanation and a representative parallel pair. When a new toxic sentence arrives, the prompt asks the LLM to estimate its features, assign it to a cluster, and detoxify it using that cluster's explanation and example. This turns the explainability analysis into a conditioning signal for generation, which the paper says reduces hallucination and makes the edit more targeted.","core_discovery":"The paper's central claim is that supervised text detoxification can be extended to German, Chinese, Arabic, Hindi, and Amharic by manually curating parallel toxic-to-neutral pairs, and that explaining toxicity with GPT-4 can be turned into a better detoxification prompt. The authors collect 400 training and 600 test sentence pairs for each new language, then ask GPT-4 to label sentences across the nine languages with toxicity level, tone, language style, implied sentiment, and negative connotations, with native-speaker validation reported at 98% agreement. The toxic sentences' labels are one-hot encoded and clustered by k-means into three per-language detoxification strategies, and a Chain-of-Thought prompt first assigns a new toxic sentence to a cluster and then rewrites it using the cluster's explanation and a representative example. The paper reports that this method achieves the highest Style Transfer Accuracy (the share of outputs judged non-toxic by the toxicity classifier) among compared approaches and the highest average Joint score, an aggregate of non-toxicity, content preservation, and fluency; it interprets this as evidence that cluster knowledge reduces hallucination and yields more precise edits.","pith_inferences":["Editorial inference: because the STA checker disagrees sharply with human detoxification references in some languages (Chinese human references get STA 0.266), the reported Joint-score gaps could be re-ranked by a human-validated toxicity measure; the CoT method's advantage should be treated as pending that check.","Editorial inference: the describe-and-cluster recipe is not specific to toxicity; applying feature extraction plus k-means over repair strategies to formality transfer or sentiment transfer could produce the same kind of conditioning signal, and existing parallel datasets would allow that test.","Editorial inference: the cluster explanations in the paper are English-only, so varying the language and phrasing of the cluster instructions, or the number of clusters, is an untested dimension that could matter more in lower-resource languages such as Amharic.","Editorial inference: a direct human evaluation of fluency and content preservation on the test outputs would settle whether the STA advantage reflects genuinely better detoxifications or merely a shift toward the classifier's notion of non-toxicity."],"forward_implications":["The new 400/600 train/test splits for German, Hindi, Amharic, Arabic, and Chinese give subsequent work a standard benchmark for supervised detoxification in languages with no prior parallel corpus.","The cluster-conditioned CoT prompt can be applied without fine-tuning: given any toxic sentence, the model diagnoses the edit type from the cluster and rewrites accordingly, reducing hallucinated content.","The Delete baseline's strong showing in Chinese, Arabic, and Amharic implies that for some languages, removing toxic tokens is a competitive fallback when no good paraphrase model exists.","The descriptive feature analysis provides a reusable map of each language's toxic lexicon, for instance animal insults in Hindi and Amharic and refugee-related wordplay in German, that can inform language-specific moderation and generation."],"supporting_citations":[{"why":"Defines the ParaDetox annotation quality criteria and the STA/SIM/ChrF1/J evaluation pipeline that the new datasets and CoT method are measured with.","marker":"Logacheva et al., 2022"},{"why":"Provides the definition of toxicity and the existing multilingual ParaDetox corpus that this work extends to German, Chinese, Arabic, Hindi, and Amharic.","marker":"Dementieva et al., 2024a"},{"why":"The XLM-R-large model fine-tuned into the per-language STA toxicity classifier used to score detoxification outputs.","marker":"Conneau et al., 2020"},{"why":"LaBSE sentence embeddings compute the SIM content-similarity component of the Joint score.","marker":"Feng et al., 2022"},{"why":"The sacrebleu ChrF1 implementation supplies the fluency component of the Joint score.","marker":"Post, 2018"},{"why":"GPT-4 is the model that generates the descriptive explanations and performs both few-shot and cluster-conditioned CoT detoxification.","marker":"OpenAI, 2022"},{"why":"Supplies the HASOC FIRE 2019 toxic Hindi posts from which the new Hindi ParaDetox candidates are selected.","marker":"Mandl et al., 2019"},{"why":"Supplies one of the Amharic hate-speech datasets used as the source for the new Amharic ParaDetox corpus.","marker":"Ayele et al., 2023"},{"why":"Supplies the Levantine Arabic L-HSAB dataset used to build the new Arabic ParaDetox corpus.","marker":"Mulki et al., 2019"},{"why":"Supplies the TOXICN Chinese toxic comments and the pretrained toxicity classifier used to filter the Chinese ParaDetox candidates.","marker":"Lu et al., 2023"}],"fun_headline_variants":["Cluster prompts make GPT-4 detoxify 9 languages","Explainable detox: cluster hints beat baselines","Multilingual detox with GPT-4 cluster reasoning","Nine languages, one explainable detox method","Detox rewriter uses GPT-4 clusters for precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the automatic toxicity checker used to score outputs is a fair judge of non-toxicity in all nine languages; the paper's own Table 9 shows Chinese human detoxification references receiving only 0.266 on that check, so if the checker is biased, the reported rankings of methods, including the CoT advantage, would not be trustworthy.","fun_headline_variants_meta":{"raw":{"variants":["Cluster prompts make GPT-4 detoxify 9 languages","Explainable detox: cluster hints beat baselines","Multilingual detox with GPT-4 cluster reasoning","Nine languages, one explainable detox method","Detox rewriter uses GPT-4 clusters for precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1347,"prompt_tokens":1008,"completion_tokens":339,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":265}},"tokens_in":624,"tokens_out":339,"duration_ms":3857,"temperature":1.0,"reasoning_tokens":265,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:39:46.112551+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give native speakers a blind sample of 200 GPT-4 CoT outputs and 200 GPT-4 few-shot outputs per language and ask them to judge toxicity and meaning preservation. If human raters do not rate CoT outputs as non-toxic more often, or if the per-language ranking changes, the reported STA-based Joint advantage is an artifact of the toxicity checker. A direct pilot: measure the checker's agreement with human toxicity judgments on the human detoxification references for Chinese and Amharic, where the checker's scores diverge most.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4 is the model that generates the descriptive explanations and performs both few-shot and cluster-conditioned CoT detoxification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies one of the Amharic hate-speech datasets used as the source for the new Amharic ParaDetox corpus."}],"review_version":1}