REVIEW 39 references
Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
The pre-training of large language models (LLMs) relies on massive text datasets sourced from diverse and difficult-to-curate origins. Although membership inference attacks and hidden canaries have been explored to trace data usage, such methods rely on memorization of training data, which LM providers try to limit. In this work, we demonstrate that indirect data poisoning (where the targeted behavior is absent from training data) is not only feasible but also allow to effectively protect a dataset and trace its use. Using gradient-based optimization prompt-tuning, we make a model learn arbitrary secret sequences: secret responses to secret prompts that are absent from the training corpus. We validate our approach on language models pre-trained from scratch and show that less than 0.005% of poisoned tokens are sufficient to covertly make a LM learn a secret and detect it with extremely high confidence ($p < 10^{-55}$) with a theoretically certifiable scheme. Crucially, this occurs without performance degradation (on LM benchmarks) and despite secrets never appearing in the training set.
Reference graph
Works this paper leans on
-
[1]
Smollm - blazingly fast and remarkably powerful, 2024
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Smollm - blazingly fast and remarkably powerful, 2024
work page 2024
-
[2]
Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm-corpus, July 2024
work page 2024
-
[3]
The foundation model transparency index.arXiv preprint arXiv:2310.12941, 2023
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. The foundation model transparency index.arXiv preprint arXiv:2310.12941, 2023
arXiv 2023
-
[4]
Data taggants: Dataset ownership verification via harmless targeted data poisoning
Wassim Bouaziz, Nicolas Usunier, and El-Mahdi El-Mhamdi. Data taggants: Dataset ownership verification via harmless targeted data poisoning. InThe Thirteenth International Conference on Learning Representations, 2025
work page 2025
-
[5]
Extracting training data from large language models
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021
2021
-
[6]
Do membership inference attacks work on large language models?arXiv preprint arXiv:2402.07841, 2024
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettle- moyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models?arXiv preprint arXiv:2402.07841, 2024
arXiv 2024
-
[7]
The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024
arXiv 2024
-
[8]
Gradient-based adversarial attacks against text transformers.arXiv preprint arXiv:2104.13733, 2021
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers.arXiv preprint arXiv:2104.13733, 2021
arXiv 2021
Show all 39 references
-
[9]
What is in your safe data? identifying benign data that breaks safety.arXiv preprint arXiv:2404.01099, 2024
Luxi He, Mengzhou Xia, and Peter Henderson. What is in your safe data? identifying benign data that breaks safety.arXiv preprint arXiv:2404.01099, 2024
2024 arXiv
-
[10]
Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022
2022 arXiv
-
[11]
Composite backdoor attacks against large language models.arXiv preprint arXiv:2310.07676, 2023
Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. Composite backdoor attacks against large language models.arXiv preprint arXiv:2310.07676, 2023
2023 arXiv
-
[12]
Preventing verbatim memorization in language models gives a false sense of privacy.arXiv preprint arXiv:2210.17546, 2022
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives a false sense of privacy.arXiv preprint arXiv:2210.17546, 2022
-
[13]
Auditing differentially private machine learning: How private is private sgd?Advances in Neural Information Processing Systems, 33:22205–22216, 2020
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd?Advances in Neural Information Processing Systems, 33:22205–22216, 2020
2020
-
[14]
Categorical reparameterization with gumbel-softmax
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016
2016 arXiv
-
[15]
Deduplicating training data mitigates privacy risks in language models
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. InInternational Conference on Machine Learning, pages 10697– 10707. PMLR, 2022
2022
-
[16]
Mope: Model perturbation-based privacy attacks on language models.arXiv preprint arXiv:2310.14369, 2023
Marvin Li, Jason Wang, Jeffrey Wang, and Seth Neel. Mope: Model perturbation-based privacy attacks on language models.arXiv preprint arXiv:2310.14369, 2023
2023 arXiv
-
[17]
Untargeted backdoor watermark: Towards harmless and stealthy dataset copyright protection
Yiming Li, Yang Bai, Yong Jiang, Yong Yang, Shu-Tao Xia, and Bo Li. Untargeted backdoor watermark: Towards harmless and stealthy dataset copyright protection. InAdvances in Neural Information Processing Systems, 2022. 11
2022
-
[18]
Mobilellm: Optimizing sub-billion parameter language models for on-device use cases.arXiv preprint arXiv:2402.14905, 2024
Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases.arXiv preprint arXiv:2402.14905, 2024
2024 arXiv
-
[19]
Data contamination: From memorization to exploitation.arXiv preprint arXiv:2203.08242, 2022
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation.arXiv preprint arXiv:2203.08242, 2022
2022 arXiv
-
[20]
Llm dataset inference: Did you train on my dataset?arXiv preprint arXiv:2406.06443, 2024
Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. Llm dataset inference: Did you train on my dataset?arXiv preprint arXiv:2406.06443, 2024
2024 arXiv
-
[21]
Copyright traps for large language models.arXiv preprint arXiv:2402.09363, 2024
Matthieu Meeus, Igor Shilov, Manuel Faysse, and Yves-Alexandre de Montjoye. Copyright traps for large language models.arXiv preprint arXiv:2402.09363, 2024
2024 arXiv
-
[22]
Quantifying privacy risks of masked language models using membership inference attacks.arXiv preprint arXiv:2203.03929, 2022
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks.arXiv preprint arXiv:2203.03929, 2022
2022 arXiv
-
[23]
Proving test set contamination in black box language models.arXiv preprint arXiv:2310.17623, 2023
Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. Proving test set contamination in black box language models.arXiv preprint arXiv:2310.17623, 2023
2023 arXiv
-
[24]
Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023
2023 arXiv
-
[25]
Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019
2019
-
[26]
Radioactive data: tracing through training
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Radioactive data: tracing through training. InInternational Conference on Machine Learning, pages 8326–8335. PMLR, 2020
2020
-
[27]
Detecting pretraining data from large language models.arXiv preprint arXiv:2310.16789, 2023
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models.arXiv preprint arXiv:2310.16789, 2023
2023 arXiv
-
[28]
Membership inference attacks against machine learning models
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017
2017
-
[29]
Privacy auditing with one (1) training run
Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. Advances in Neural Information Processing Systems, 36, 2024
2024
-
[30]
Did you train on my dataset? towards public dataset protection with cleanlabel backdoor watermarking.ACM SIGKDD Explorations Newsletter, 25(1):43–53, 2023
Ruixiang Tang, Qizhang Feng, Ninghao Liu, Fan Yang, and Xia Hu. Did you train on my dataset? towards public dataset protection with cleanlabel backdoor watermarking.ACM SIGKDD Explorations Newsletter, 25(1):43–53, 2023
2023
-
[31]
Memorization without overfitting: Analyzing the training dynamics of large language models.Advances in Neural Information Processing Systems, 35:38274–38290, 2022
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models.Advances in Neural Information Processing Systems, 35:38274–38290, 2022
2022
-
[32]
Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023
2023 arXiv
-
[33]
The stronger the diffusion model, the easier the backdoor: Data poisoning to induce copyright breaches without adjusting finetuning pipeline.arXiv preprint arXiv:2401.04136, 2024
Haonan Wang, Qianli Shen, Yao Tong, Yang Zhang, and Kenji Kawaguchi. The stronger the diffusion model, the easier the backdoor: Data poisoning to induce copyright breaches without adjusting finetuning pipeline.arXiv preprint arXiv:2401.04136, 2024
2024 arXiv
-
[34]
Proving membership in llm pretraining data via data watermarks.arXiv preprint arXiv:2402.10892, 2024
Johnny Tian-Zheng Wei, Ryan Yixiang Wang, and Robin Jia. Proving membership in llm pretraining data via data watermarks.arXiv preprint arXiv:2402.10892, 2024. 12
2024 arXiv
-
[35]
according to
Orion Weller, Marc Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. " according to...": Prompting language models improves quoting from pre-training data.arXiv preprint arXiv:2305.13252, 2023
2023 arXiv
-
[36]
Privacy risk in machine learning: Analyzing the connection to overfitting
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE, 2018
2018
-
[37]
Counterfactual memorization in neural language models.Advances in Neural Information Processing Systems, 36:39321–39362, 2023
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models.Advances in Neural Information Processing Systems, 36:39321–39362, 2023
2023
-
[38]
Membership inference attacks cannot prove that a model was trained on your data.arXiv preprint arXiv:2409.19798, 2024
Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. Membership inference attacks cannot prove that a model was trained on your data.arXiv preprint arXiv:2409.19798, 2024
2024 arXiv
-
[39]
Bob’s model was not trained on Alice’s protected dataset
Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramèr, and Daphne Ippolito. Persistent pre-training poisoning of llms.arXiv preprint arXiv:2410.13722, 2024. 13 Appendix A Proof for theoretical guarantees We show that Propo...
2024 arXiv
Discussion (0). Sign in to comment.