Pith. sign in

REVIEW 47 references

GloSS over Toxicity: Understanding and Mitigating Toxicity in LLMs via Global Toxic Subspace

T0 review · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Detoxifying LLMs by deleting a global, cross-layer 'toxic subspace' from feed-forward weights reduces toxic outputs more than layer-local subspace methods.

arxiv 2505.17078 v1 pith:3GSX3MO4 submitted 2025-05-20 cs.CL cs.AI

classification cs.CLcs.AI
keywords toxictoxicityglobalsubspaceglossllmsdetoxificationeffective
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language models sometimes generate hateful or offensive text. Previous work tried to find the exact 'toxic neurons' or the toxic directions inside one specific network layer, and then block them. The authors of this paper show that blocking a few neurons does not do much: the model still has many other paths that push it toward toxic words. They also show that looking at one layer at a time is unreliable, because the strength of toxic signals varies from layer to layer.

Instead, they define a 'global toxic subspace', a small set of directions in the model's internal representation space that appear repeatedly across many layers and whose projected words are mostly slurs and profanity. The method, GloSS, works in four steps: collect contrasts between toxic and non-toxic prompts, extract candidate directions per layer with SVD, rank them by how many bad words they point to, and then remove the top-ranked directions from the value matrices of all feed-forward layers at once. This is a projection, so the rest of the model's computation is untouched.

In tests on four open models (GPT-2 Medium, GPT-J 6B, OPT 6.7B, Mistral 7B), GloSS produced lower toxicity scores than fine-tuning baselines and than the layer-wise ProFS method, and perplexity stayed about the same. The evidence would be stronger with error bars, a released bad-words list, and more capability tests than perplexity alone.

Extended reading notes

Core claim

The paper's central claim is that modeling the toxic region as a global toxic subspace, obtained by aggregating and ranking SVD directions across all layers and removing it via the projection W_proj = (I - P_toxic) W_V, yields stronger detoxification than layer-wise subspaces (ProFS) or fine-tuning (SSFT and DPO) while preserving general capabilities. If this is true, toxicity lives in a shared, low-dimensional cross-layer direction set rather than in individual toxic vectors or per-layer subspaces.

Load-bearing premise

The difference between mean FFN activations for toxic and non-toxic prompt pairs, after mean-centering, is dominated by a toxic direction rather than by topic, context, or stopword differences. Section 4 Step 1 (Eq. 8) assumes the top right singular vectors of T_l = X+_l - X-_l are candidate toxic directions; Section 3.2 shows layer-wise projections are often non-toxic in low and high layers, meaning the contrast matrix does not always isolate toxicity. If the SVD directions are dominated by non-toxic content, the global subspace is not a toxic subspace and the method's success would be due to deleting generic content directions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 7 free parameters · 5 assumptions · 1 invented entities

The central claim rests on several assumptions: contrastive SVD directions isolate toxicity, the bad-words list is a valid toxicity selector, and projection onto the orthogonal complement removes toxicity while leaving other capabilities intact. There are at least four chosen hyperparameters per model, and the low-dimensionality conclusion is partly controlled by the PCA variance threshold eta. No code or exact bad-words list is shipped, so several inputs to the pipeline are unspecified.

free parameters (7)
  • Toxicity threshold tau = GPT-2: 1.0; GPT-J: 4.0; OPT-6.7B: 2.0; Mistral-7B: 1.0
    Eq. 10 defines tau = mu + alpha*sigma; the tau values are tuned per model in Table 5 and control which candidate directions are selected.
  • PCA explained-variance ratio eta = 0.8 (GPT-2, OPT); 0.7 (GPT-J, Mistral)
    Table 5; eta controls the number of principal components retained and directly shapes the reported low-dimensionality of the toxic subspace.
  • Projection starting layer range l0 = 13-24 (GPT-2); 15-28 (GPT-J); 10-32 (OPT); 15-32 (Mistral)
    Table 5 and Figure 6; the choice has large effects, e.g., Mistral perplexity jumps from 9.7 to 231.7 when projection starts at layer 2 instead of 3, so the SOTA comparison depends on these tuned ranges.
  • Per-layer top-k singular vectors k = Not reported for GloSS
    Section 4 Step 1 says larger k is richer, but no k value is given for GloSS, so the set of candidate directions is not fully specified.
  • Top-m tokens m for toxicity scoring = Not reported
    Eq. 9 defines tox_score using the top-m tokens of the projection; the score and therefore the selected directions depend on m.
  • Bad-words list B = Not specified
    Eq. 9; the exact list is essential for ranking and is not released or precisely instantiated in the paper.
  • Activation-shift scaling alpha (analysis only) = 100
    Eq. 7 and Table 2 use a heuristic scaling factor alpha = 100 to demonstrate that shifting activations along a toxic direction changes projected tokens; this supports the motivation but is not part of the method.
assumptions (5)
  • domain assumption The top right singular vectors of the mean-centered contrastive matrix T_l represent toxic directions rather than stopword or context differences.
    Section 4 Step 1 (Eq. 8) builds candidate toxic directions from X+_l - X-_l; Section 3.2 shows many layers yield non-toxic projections, so this assumption is load-bearing and is not always satisfied.
  • ad hoc to paper Mean-centering the contrastive matrix removes frequent token semantics, isolating toxicity.
    Section 4 Step 1 states mean-centering is used to mitigate frequent token semantics, but no derivation is given; the factor model in Eq. 5 includes stopwords, context, and noise that can survive contrast.
  • domain assumption Projecting FFN value matrices onto the orthogonal complement of the global toxic subspace leaves non-toxic model behavior intact.
    Step 4 (Eq. 14) assumes residual-stream and attention circuits do not re-introduce toxicity or damage general capabilities; the paper only checks perplexity, not broader capabilities.
  • ad hoc to paper The bad-words list B and top-m vocabulary overlap give a valid toxicity score for ranking directions.
    Eq. 9 defines tox_score(v) = |T_v intersect B| / m; B is referenced but never instantiated, and the score is a proxy for the Detoxify evaluator, so ranking may be fitted to the evaluation.
  • standard math Orthogonal projection removes exactly the component along the toxic subspace from each value vector.
    Eqs. 13-14 use P_toxic = sum d_i d_i^T and (I - P_toxic) W_V, which is standard linear algebra.
invented entities (1)
  • Global toxic subspace
    purpose: A low-dimensional cross-layer set of directions in FFN value-vector space that the paper claims drives toxic generation and that GloSS removes via projection.
    The subspace is constructed from the paper's own contrastive pairs and bad-words-list ranking (Eqs. 8-12), so its toxicity is by construction; the random-subspace control is internal to the paper and is not an external falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GloSS over Toxicity: Understanding and Mitigating Toxicity in LLMs via Global Toxic Subspace." pith.science (2026). https://pith.science/paper/3GSX3MO4

@misc{pith2026250517078,
  author       = {Pith},
  title        = {Pith review of: GloSS over Toxicity: Understanding and Mitigating Toxicity in LLMs via Global Toxic Subspace},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3GSX3MO4}},
  note         = {Machine review of arXiv:2505.17078}
}
read the original abstract

This paper investigates the underlying mechanisms of toxicity generation in Large Language Models (LLMs) and proposes an effective detoxification approach. Prior work typically considers the Feed-Forward Network (FFN) as the main source of toxicity, representing toxic regions as a set of toxic vectors or layer-wise subspaces. However, our in-depth analysis reveals that the global toxic subspace offers a more effective and comprehensive representation of toxic region within the model. Building on this insight, we propose GloSS (Global Toxic Subspace Suppression), a lightweight, four-stage method that mitigates toxicity by identifying and removing the global toxic subspace from the parameters of FFN. Experiments across a range of LLMs show that GloSS achieves state-of-the-art detoxification performance while preserving the models general capabilities, without requiring large-scale data or model retraining.

Figures

Figures reproduced from arXiv: 2505.17078 by the authors.

Figure 1
Figure 1. (a) Removing toxic vectors do not alter the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Results of Different Operations on Activation of Vectors. (a) Enhance different numbers of toxic and non-toxic value vector activations, selectively; (b) Suppress toxic vector activations at different proportions; (c) Reversing value vector activations steers the FFN blocks either toward or away from the toxic direction. as v ℓ i (value vector). The function f(·) represents a non-linear activation, such as GELU. FFN… view at source ↗
Figure 3
Figure 3. Top-5 Toxic Directions Across Layers. They [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The overview of GloSS. It identifies and removes the global toxic subspace through a four-stage procedure [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Effectiveness of Extracted vs. Random Sub [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Impact of Projection Layers on Toxicity and [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 5 canonical work pages

  1. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. https://arxiv.org/abs/2005.14165 Lan...

  2. [2]

    Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. https://doi.org/10.18653/v1/2022.acl-long.581 Knowledge neurons in pretrained transformers . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493--8502, Dublin, Ireland. Association for Computational Linguistics

  3. [3]

    Jingcheng Deng, Zihao Wei, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. 2024. Everything is editable: Extend knowledge editing to unstructured data in large language models. arXiv preprint arXiv:2405.15349

  4. [4]

    Zenghao Duan, Wenbin Duan, Zhiyi Yin, Yinghan Shen, Shaoling Jing, Jie Zhang, Huawei Shen, and Xueqi Cheng. 2025. Related knowledge perturbation matters: Rethinking multiple pieces of knowledge editing in same-subject. arXiv preprint arXiv:2502.06868

  5. [5]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, and 1 others. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12

  6. [6]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.301 R eal T oxicity P rompts: Evaluating neural toxic degeneration in language models . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356--3369, Online. Association for Computational Linguistics

  7. [7]

    Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.3 Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30--45, Abu Dhabi, United Arab Emirates. Association f...

  8. [8]

    Schuster, Jonathan Berant, and Omer Levy

    Mor Geva, R. Schuster, Jonathan Berant, and Omer Levy. 2020. Transformer feed-forward layers are key-value memories. ArXiv, abs/2012.14913

Show all 47 references
  1. [9]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. https://arxiv.org/abs/2411.15594 A survey on llm-as-a-judge . Preprint, a...

  2. [10]

    Skyler Hallinan, Alisa Liu, Yejin Choi, and Maarten Sap. 2022. Detoxifying text with marco: Controllable revision with experts and anti-experts. arXiv preprint arXiv:2212.10543

  3. [11]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . CoRR, abs/2106.09685

  4. [12]

    Fengqing Jiang. 2024. Identifying and mitigating vulnerabilities in llm-integrated applications. Master's thesis, University of Washington

  5. [13]

    Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858

  6. [14]

    Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. 2023. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506--17533. PMLR

  7. [15]

    Kummerfeld, and Rada Mihalcea

    Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. 2024. https://openreview.net/forum?id=dBqHGZPGZI A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity . In Forty-first International Conference ...

  8. [16]

    Chak Tou Leong, Yi Cheng, Jiashuo Wang, Jian Wang, and Wenjie Li. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.269 Self-detoxifying language models via toxification reversal . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages...

  9. [17]

    Xingjun Ma, Yifeng Gao, Yixu Wang, Ruofan Wang, Xin Wang, Ye Sun, Yifan Ding, Hengyuan Xu, Yunhao Chen, Yunhan Zhao, Hanxun Huang, Yige Li, Jiaming Zhang, Xiang Zheng, Yang Bai, Zuxuan Wu, Xipeng Qiu, Jingfeng Zhang, Yiming Li, and 28 others. 2025. https://arxiv.org/abs/2502.0...

  10. [18]

    Harry Mayne, Yushi Yang, Adam Mahdi, and Filip Sondej. 2024. https://arxiv.org/abs/2411.06424 Ablation is not enough to emulate dpo: How neuron dynamics drive toxicity reduction . Preprint, arXiv:2411.06424

  11. [19]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359--17372

  12. [20]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. https://arxiv.org/abs/1609.07843 Pointer sentinel mixture models . Preprint, arXiv:1609.07843

  13. [21]

    Yixin Ou, Yunzhi Yao, Ningyu Zhang, Hui Jin, Jiacheng Sun, Shumin Deng, Zhenguo Li, and Huajun Chen. 2025. How do llms acquire new knowledge? a knowledge circuits perspective on continual pre-training. arXiv preprint arXiv:2502.11196

  14. [22]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing sys...

  15. [23]

    Wenbo Pan, Zhichao Liu, Qiguang Chen, Xiangyang Zhou, Haining Yu, and Xiaohua Jia. 2025. The hidden dimensions of llm alignment: A multi-dimensional safety analysis. arXiv preprint arXiv:2502.09674

  16. [24]

    Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. 2020. Back to the future: Unsupervised backprop-based decoding for counterfactual and abductive commonsense reasoning. arXiv preprint arXiv:2010.05906

  17. [25]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

  18. [26]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Sy...

  19. [27]

    Xavier Suau, Pieter Delobelle, Katherine Metcalf, Armand Joulin, Nicholas Apostoloff, Luca Zappella, and Pau Rodríguez. 2024. https://arxiv.org/abs/2407.12824 Whispering experts: Neural interventions for toxicity mitigation in language models . Preprint, arXiv:2407.12824

  20. [28]

    Qi Sun, Marc Pickett, Aakash Kumar Nain, and Llion Jones. 2025. https://doi.org/10.1609/aaai.v39i24.34708 Transformer layers as painters . Proceedings of the AAAI Conference on Artificial Intelligence, 39(24):25219--25227

  21. [29]

    Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2023. Function vectors in large language models. arXiv preprint arXiv:2310.15213

  22. [30]

    Rheeya Uppaal, Apratim Dey, Yiting He, Yiqiao Zhong, and Junjie Hu. 2025. https://openreview.net/forum?id=lOi6FtIwR8 Model editing as a robust and denoised variant of DPO : A case study on toxicity . In The Thirteenth International Conference on Learning Representations

  23. [31]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

  24. [32]

    Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A 6 billion parameter autoregressive language model

  25. [33]

    Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. 2024. https://doi.org/10.18653/v1/2024.acl-long.171 Detoxifying large language models via knowledge editing . In Proceedings of the 62nd Annual Meet...

  26. [34]

    Zihao Wei, Jingcheng Deng, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. 2024. Mlake: Multilingual knowledge editing benchmark for large language models. arXiv preprint arXiv:2404.04990

  27. [35]

    Huajian Xin, Z. Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, Wenjun Gao, Qihao Zhu, Dejian Yang, Zhibin Gou, Z. F. Wu, Fuli Luo, and Chong Ruan. 2024. https://arxiv.org/abs/2408.08152 Deepseek-prover-v1.5: Harnessing ...

  28. [36]

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. 2024. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. arXiv preprint arXiv:2402.08983

  29. [37]

    Yu Yan, Sheng Sun, Zenghao Duan, Teli Liu, Min Liu, Zhiyi Yin, Qi Li, and Jiangyu Lei. 2025. https://arxiv.org/abs/2503.00038 from benign import toxic: Jailbreaking the language model via adversarial metaphors . Preprint, arXiv:2503.00038

  30. [38]

    Wanli Yang, Fei Sun, Xinyu Ma, Xun Liu, Dawei Yin, and Xueqi Cheng. 2024. https://arxiv.org/abs/2402.09656 The butterfly effect of model editing: Few edits can trigger large language models collapse . Preprint, arXiv:2402.09656

  31. [39]

    Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. 2024. Knowledge circuits in pretrained transformers. arXiv preprint arXiv:2405.17969

  32. [40]

    Zeping Yu and Sophia Ananiadou. 2023. Neuron-level knowledge attribution in large language models. arXiv preprint arXiv:2312.12141

  33. [41]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, and 1 others. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

  34. [42]

    Zhexin Zhang, Jiale Cheng, Hao Sun, Jiawen Deng, and Minlie Huang. 2023. Instructsafety: a unified framework for building multidimensional and explainable safety detector through instruction tuning. In Findings of the Association for Computational Linguistics: EMNLP 2023, page...

  35. [43]

    Wei Zhao, Zhe Li, Yige Li, Ye Zhang, and Jun Sun. 2024. Defending large language models against jailbreak attacks via layer-specific editing. arXiv preprint arXiv:2405.18166

  36. [44]

    Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023. Autodan: interpretable gradient-based adversarial attacks on large language models. arXiv preprint arXiv:2310.15140

  37. [45]

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

  38. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.