{"id":"03c25cb8-f545-49d7-b9ed-fa2de7561501","arxiv_id":"2608.11146","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Safety mechanisms trained in English largely fail to activate for harmful prompts in four low-resource African languages, even when models understand the meaning.","lead":"This paper tests whether safety guardrails trained in English still work when people write harmful requests in four African languages: Twi, Hausa, Amharic, and Swahili. It finds that models often understand the request but fail to activate their refusal mechanism, retaining less than 10% of the English refusal signal in most cases.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline retained-component numbers may be an artifact of mismatched safe baselines: target-language 'safe' sets are drawn from unrelated corpora, so the low alignment with the English refusal direction may not measure safety routing at all.","rationale":"The central claim is the <10% retained refusal signal. The magnitude of this number is computed from the displacement between harmful target-language prompts and safe target-language prompts, then projected onto the English unsafe-minus-safe direction. The computation is only meaningful if the two difference vectors are measuring the same thing (harmfulness). The English direction itself is a difference between DNA-style harmful prompts and a prompt-safety dataset; the target language direction is a difference between DNA-style translations and unrelated safe sentences from parallel/Aya corpora. Because the target-language safe set is not matched to the harmful prompts, the direction r_l likely includes corpus, register, and topic components that have no relation to safety. Those components can easily be orthogonal to the English direction, producing retained components near zero even if the model would refuse perfectly in the target language. The absence of a safety-aligned control language (stated in Section 7) means the metric has no calibration: we do not know what a large retained component looks like for a language in which the model refuses reliably. The Swahili-Llama pair is a single positive example, not a systematic control. The paper's other evidence (probe accuracy on English, qualitative generation failures, low SLL) supports the existence of real safety gaps, but those are behavioral/correlational and do not validate the precise geometric quantity. Thus, the load-bearing assumption is the matched-baseline assumption, and the paper itself does not supply matched baselines or robustness checks. The proposed concrete test—recomputing retained components with benign paraphrases of the same harmful prompts as safe references—directly targets this assumption and would discriminate between a safety-routing failure and a corpus-mismatch artifact. The reader's verdict (CONDITIONAL) is correct; my stress test identifies the same weakest point and does not move the verdict, though it does sharpen the required condition.","tokens_in":18913,"tokens_out":4896,"duration_ms":48962,"concrete_test":"Recompute Table 2 for at least Twi and Hausa using matched safe baselines: for each harmful literal prompt, construct a benign counterpart by the same native-speaker translators, preserving topic, length, and register but removing the harmful request (or use the safe half of the DNA-style pipeline with equivalent distribution). If retained components stay below 10% with matched baselines, the geometric conclusion is robust; if values rise substantially (e.g., Twi from 3.7% to >25% for one or more models), the headline finding is largely a baseline artifact. Independent re-derivation of the Section 4.4 retained-component formula with the matched baselines would settle the point.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim (Table 2: retained component <10% for nearly all language-model pairs) assumes r_l = mu(L_l) - mu(S_l) isolates harmful intent in the target language. This requires S_l to be a matched safe baseline. In practice (Appendix A.1), S_l is drawn from English-Twi parallel text for Twi, Aya_amharic for Amharic, and Aya for Hausa/Swahili; these are arbitrary sentences, not benign instruction/query prompts matched to the DNA-style translations in topic, register, and format. The displacement r_l therefore mixes harmfulness with domain and register differences, and orthogonality to the English refusal direction could be a cross-corpus artifact rather than a safety-routing failure. The paper itself acknowledges several related confounds: severe Amharic tokenization fragmentation (Section 5.1), no safety-aligned control language (Section 7), and the observational/correlational nature of the framework. The Swahili-Llama positive result is the only existing indication that the metric can produce a large retained component when behavior is safer; without such a control for each architecture, near-zero values are not calibrated. The downstream SLL/refusal-probability results (Table 4) provide some independent support, but their own limitations (generation instability, refusal-token coverage) are acknowledged, and they are not sufficient to rescue the specific 10% geometric claim if the r_l baseline is mismatched.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether English-centric safety alignment transfers to four low-resource African languages (Twi, Hausa, Amharic, Swahili). The authors introduce LoDNA, a dataset of literal and culturally localized translations of Do-Not-Answer prompts, and a latent geometric framework that measures alignment between the English refusal direction and target-language displacement vectors in hidden-state space, supported by PCA projections, linear probes, and downstream refusal log-likelihood/probability metrics. Across Mistral, Llama, Qwen2.5, and AfriqueQwen, the authors find that the retained component of the English refusal signal is below 10% for nearly all language-model pairs (with Swahili-Llama the exception at about 27%), leading to the claim that multilingual safety alignment is superficial and that no universal harm manifold exists.","tokens_in":19116,"tokens_out":7854,"duration_ms":68245,"significance":"The paper addresses an important and under-explored question about multilingual safety alignment in low-resource languages, and its main contributions—the LoDNA dataset and the latent geometric evaluation framework—are valuable to the community. The inclusion of culturally localized prompts goes beyond typical translated benchmarks, and the multi-method design (probe validation, PCA, retained components, refusal-sequence likelihoods, qualitative outputs) is a strength. The headline quantitative claim is falsifiable, and the paper is careful to frame its geometric evidence as correlational rather than causal. However, the strength of the central claim depends critically on the validity of the safe-target-language baselines and on statistical calibration of the geometric metric, both of which are currently insufficient. If those concerns are addressed, the paper would be a significant contribution to interpretability and multilingual safety evaluation.","major_comments":[{"comment":"The definition of the target-language displacements r_ℓ,lit = mu(L_ℓ) - mu(S_ℓ) and r_ℓ,ctx = mu(C_ℓ) - mu(S_ℓ) (Section 4.3) relies on safe baselines S_ℓ that are not matched to the harmful prompts. As described in Appendix A.1, S_ℓ is drawn from English-Twi parallel text for Twi, Aya_amharic for Amharic, and Aya for Hausa/Swahili, while L_ℓ and C_ℓ are translated/culturally localized Do-Not-Answer queries. These corpora differ from the harmful-prompt set in register, format, and topic, so the displacement vectors mix harmfulness with corpus-level differences. The low retained components in Table 2 could therefore be a cross-corpus artifact rather than evidence that harmful intent fails to route into the English refusal direction. Please re-run the analysis with a matched safe baseline (e.g., translations of the English safe prompts into each target language) or demonstrate through a control experiment that the results are insensitive to the choice of S_ℓ.","section":"4.3 / Appendix A.1"},{"comment":"The primary headline result—retained components below 10% for nearly all language-model pairs—rests on point estimates in Table 2 with no confidence intervals, significance tests, or reported dataset sizes. Given that the English and target-language means are computed over small prompt sets, the difference between, e.g., Swahili-Llama at 27.31% and the other near-zero entries may not be statistically meaningful. Please provide bootstrap intervals (or equivalent) for the retained component, cosine similarities, and drift, and report the number of prompts in each set.","section":"Table 2 / Section 5"},{"comment":"The paper itself states that severe Amharic tokenization fragmentation 'likely distorts geometric analysis before safety routing occurs,' yet Amharic rows in Table 2 and the corresponding PCA/probe results are treated as evidence about safety transfer. If tokenization corrupts the hidden states, the negative retained components and unstable alignments for Amharic cannot support the paper's conclusion. Please either exclude Amharic from the geometric claims or provide a tokenization-robust analysis (e.g., compare character-level vs. subword-level extraction, or mask out heavily fragmented inputs).","section":"5.1 / 7, Figure 12"},{"comment":"Section 7 concedes that no safety-aligned control language exists in the 7B-8B open-weight ecosystem. Without such a control, low alignment with the English refusal direction is ambiguous between failed transfer and a language-specific but functional safety subspace. The paper's conclusion that multilingual safety alignment is 'superficial' therefore goes beyond what the evidence can establish. A permutation test comparing target-language displacements to random directions or safe-vs-safe displacements would help calibrate the metric; as it stands, the Swahili-Llama positive case is the only internal calibration point.","section":"7"},{"comment":"The PCA projection in Figure 3 is fitted only on English states, and the target-language prompts then form compact, language-specific clusters. This clustering may reflect language identity or tokenization differences rather than safety geometry. The paper should control for language-identity confounds, for example by centering each language's representations before projecting or by projecting matched safe target-language prompts, before interpreting the clusters as evidence for a 'language-specific safety geometry' (RQ2).","section":"4.6 / Figure 3"}],"minor_comments":[{"comment":"The text contains typographic artifacts (e.g., 'language⚶model pairs' in the abstract, 'T wi' in Figure 10's caption) that should be cleaned before submission.","section":"Abstract / throughout"},{"comment":"'rather than a absence of alignment' should read 'rather than an absence of alignment.'","section":"Section 1"},{"comment":"The column header 'Retained Comp: Lit' is written as 'Retained Component: Lit' in the caption; please use consistent terminology.","section":"Table 2"},{"comment":"The paper states that the LoDNA dataset will be publicly released, but no repository link or release date is provided; for reproducibility, a permanent URL should be included.","section":"Section 3 / Appendix A.1"},{"comment":"The label 'Cultural T wi' for the Swahili Input B should be 'Cultural Swahili.'","section":"Figure 10"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope. The main risk is the mismatched safe baseline in the geometric analysis, which is fixable with additional experiments. The dataset contribution alone is noteworthy. I would not reject, but the current version is not ready for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Things to know first: the paper builds the LoDNA dataset—literal and culturally localized harmful prompts in Twi, Hausa, Amharic, and Swahili—and a latent-geometric protocol for measuring how much of the English refusal direction survives in those languages. The headline result (less than 10% retained signal for nearly all model–language pairs) is plausible but the exact numbers are not reliable yet because the target-language 'safe' baselines come from unrelated corpora. The paper is honest about several confounds, which makes it fixable rather than dismissible.\n\nWhat is genuinely new: as far as I know, LoDNA is the first dataset pairing literal translations with culturally localized harmful prompts for these languages, and projecting target-language displacement vectors onto an English refusal direction is a fresh measurement. The paper also does the right thing by grounding the geometry in downstream refusal log-likelihoods and in qualitative outputs. The limitations section is unusually candid—they admit Amharic tokenization is a severe confound, that no safety-aligned control language exists, and that the framework is correlational. That honesty counts in its favor.\n\nThe main soft spot is the safe baseline for each target language. S_l is pulled from English-Twi parallel text or Aya datasets, not matched to the harmful prompts in topic, register, or format, so r_lit = μ(L_l) − μ(S_l) mixes harmfulness with domain and style differences. Low retained component could be a cross-corpus artifact. The Swahili-Llama result, with a large retained component where behavior is safer, is one control but not enough to calibrate near-zero values. Table 2 has no confidence intervals or significance tests, and the primary claim relies on a single hand-picked layer (layer 16). The downstream SLL/refusal-probability analysis gives independent support, but the paper itself notes those rely on generation stability and refusal-token coverage.\n\nWho this is for: anyone working on multilingual safety in low-resource African languages, and people interested in representation-level interpretability of refusal. The dataset alone is likely to be reused; the specific claim against a universal harm manifold is not fully proven, but the evidence is enough to treat it as a serious open question.\n\nRecommendation: send this to peer review. The measurement problem is fixable—release data and code, add matched safe baselines or show robustness to baseline choice, report uncertainty, and ideally test with a safety-aligned control language. A careful referee could turn this into a solid contribution.","headline":"Useful dataset and a fresh geometric measurement, but the headline 10% retained-signal numbers are not yet trustworthy because the target-language 'safe' baselines are mismatched; still worth a serious referee.","tokens_in":19789,"tokens_out":3731,"would_cite":true,"duration_ms":33246,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"English-trained safety alignment does not transfer to four African languages at the representational level.","keywords":["cross-lingual safety transfer","low-resource languages","refusal direction","latent geometric framework","LoDNA dataset","African languages","multilingual safety alignment","hidden-state probing"],"falsifier":"Measure the signed retained component for a moderately-resourced language that is explicitly safety-aligned in the same 7B-8B model family; if it also stays below 10%, the low values reflect a general cross-lingual mapping limitation rather than a peculiarity of Twi, Hausa, Amharic, or Swahili. Alternatively, patch the English refusal direction from layer 16 into target-language hidden states and test whether refusal probability rises; if it does not, the retained component is not the causal gate of refusal.","tokens_in":18628,"feed_emoji":"🛡️","tokens_out":5358,"duration_ms":130753,"temperature":0.7,"pith_summary":"The paper tries to establish that English-centric safety alignment does not transfer to low-resource languages inside the model, not just in its outputs. Across Twi, Hausa, Amharic, and Swahili, harmful prompts—both literal translations and culturally localized rewrites—recover less than 10% of the English refusal signal in most model-language pairs. The authors argue the failure is not a failure to understand: literal and cultural prompts sit almost identically in the model's semantic space (cosine 0.95-0.996), yet they barely activate the refusal direction that cleanly separates safe from unsafe English prompts. If correct, multilingual safety alignment is superficial: models often grasp the harmful request but never route it into the English-trained refusal mechanism.","feed_headline":"Low-resource prompts keep under 10% of the English refusal signal","feed_subtitle":"In Twi, Hausa, Amharic, and Swahili, models often understand harmful intent but never route it into safety refusal.","key_machinery":"The Latent Geometric Framework compares, layer by layer, the mean hidden states of safe and unsafe prompts. The load-bearing object is the English refusal direction $r_k^{en}=\\mu_k(U_{en})-\\mu_k(S_{en})$, the difference between mean unsafe and mean safe final-token hidden states at layer $k$; target-language displacements $r_k^{\\ell,\\mathrm{lit}}=\\mu_k(L_\\ell)-\\mu_k(S_\\ell)$ and $r_k^{\\ell,\\mathrm{ctx}}=\\mu_k(C_\\ell)-\\mu_k(S_\\ell)$ are projected onto it through the Signed Retained Component $\\frac{(r_k^{en})^\\top r}{(r_k^{en})^\\top r_k^{en}}\\times 100\\%$. Layer-wise drift, English-trained linear probes evaluated zero-shot on target languages, and PCA projections onto the English refusal plane separate structural cross-lingual mapping failure from semantic-complexity failure. Downstream grounding uses the normalized refusal-sequence log-likelihood and the aggregate probability of emitting any refusal sequence, because the paper argues LLM-as-judge attack-success rates are unreliable for low-resource languages.","core_discovery":"Using hidden-state geometry at an intermediate layer (layer 16 for the 32-layer models, a comparable depth for Qwen2.5), the paper defines an English refusal direction as the vector from mean safe to mean unsafe English hidden states, then measures how much of that direction is recovered by literal and culturally localized harmful prompts in each target language. The Signed Retained Component is below 10% for nearly all language-model pairs—Hausa drops as low as 0.97% literal retention in Qwen2.5 and goes negative in Mistral and Llama—with Swahili in Llama the single clear exception at about 27%. The very high literal-to-cultural cosine similarity shows that the two prompt forms share a semantic subspace within the target language, while the near-zero or negative English alignment shows that this shared harm representation does not map into the English-derived safety subspace. The paper concludes that English refusal boundaries are robustly learned and linearly separable, yet target-language harmful prompts fail to enter that boundary, giving evidence against a universal, language-agnostic harm manifold for these languages.","pith_inferences":["If the low retained component is a causal proxy for weak refusal, then steering target-language hidden states toward the English refusal direction per language or per culture is a natural testable intervention, and it may need to be done separately for each language rather than once globally.","The paper lacks a safety-aligned control language; a direct extension would measure retained components in a moderately-resourced language that does have explicit safety tuning, which would separate a general cross-lingual mapping deficit from tokenization artifacts.","The interpretation that models 'encode the concept without routing it' is correlational; activation-patching at the layers where drift is largest could confirm whether the low retained component actually suppresses refusal behavior.","Since literal and cultural forms align within each language, a safety direction learned inside that language's own subspace may transfer across prompt variants even when the English direction does not."],"forward_implications":["English-trained refusal geometry does not act as a universal safety subspace for low-resource languages; different architectures fail in different ways.","Because literal and culturally localized harmful prompts are semantically close in hidden space, safety failures are not caused by cultural localization moving the prompt out of distribution; the bottleneck is routing.","Models can understand a harmful request and still comply convincingly, so generation-only refusal rates can miss safety failures that hidden-state geometry reveals.","Multilingual safety evaluation built on translated benchmarks alone may overstate cross-lingual transfer; culturally localized prompts and representational metrics are needed.","Safety alignment work for these languages should target the mapping that carries a recognized harmful concept into the refusal mechanism, not just the detection of harmful content."],"supporting_citations":[{"why":"Supplies the Do-Not-Answer dataset that LoDNA extends into Twi, Hausa, Amharic, and Swahili.","marker":"(Wang et al., 2024b)"},{"why":"Establishes that refusal in language models is mediated by a single direction in the residual stream, the geometric assumption behind the English refusal vector.","marker":"(Arditi et al., 2024)"},{"why":"Provides the representation-engineering methodology for extracting refusal directions and motivates mid-to-late layer analysis.","marker":"(Zou et al., 2025)"},{"why":"A positive cross-lingual transfer result that the paper's latent-geometry findings directly challenge.","marker":"(Dang et al., 2024)"},{"why":"Another positive reward-model transfer claim whose behavioral basis the paper questions with representational evidence.","marker":"(Hong et al., 2025)"},{"why":"Community-sourced localized English prompts that the paper adapts into Twi for culturally grounded evaluation.","marker":"(Rashid et al., 2026)"},{"why":"Background for the 'think in English' internal pathway that the paper identifies as a structural routing bottleneck.","marker":"(Shi et al., 2023)"},{"why":"The claim that refusal direction is universal across safety-aligned languages, which the paper shows breaks down in low-resource settings.","marker":"(Wang et al., 2025)"}],"fun_headline_variants":["Under 10% of English refusal signal survives in low-resource languages","Cross-lingual safety is an illusion: <10% refusal in 4 languages","Safety alignment fails: <10% refusal signal in African languages","Models miss refusal in low-resource languages despite semantic alignment","Cross-lingual safety illusion: <10% refusal retention in four languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The geometric result assumes that the safe target-language prompts used as baselines match the harmful prompts in domain, style, register, and tokenization, so that subtracting them isolates harmfulness rather than unrelated differences; without a safety-aligned control language, low alignment with the English direction could also be a language-specific but still functional safety subspace rather than a routing failure.","fun_headline_variants_meta":{"raw":{"variants":["Under 10% of English refusal signal survives in low-resource languages","Cross-lingual safety is an illusion: <10% refusal in 4 languages","Safety alignment fails: <10% refusal signal in African languages","Models miss refusal in low-resource languages despite semantic alignment","Cross-lingual safety illusion: <10% refusal retention in four languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000414,"raw_usage":{"total_tokens":2156,"prompt_tokens":976,"completion_tokens":1180,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":1085}},"tokens_in":592,"tokens_out":1180,"duration_ms":11066,"temperature":1.0,"reasoning_tokens":1085,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:13:27.546809+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the signed retained component for a moderately-resourced language that is explicitly safety-aligned in the same 7B-8B model family; if it also stays below 10%, the low values reflect a general cross-lingual mapping limitation rather than a peculiarity of Twi, Hausa, Amharic, or Swahili. Alternatively, patch the English refusal direction from layer 16 into target-language hidden states and test whether refusal probability rises; if it does not, the retained component is not the causal gate of refusal.","supporting_citations":[{"cited_title":"Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency , pages =","cited_arxiv_id":null,"evidence_quote":"Community-sourced localized English prompts that the paper adapts into Twi for culturally grounded evaluation."}],"review_version":1}