Theory of Continual Learning Against Data Poisoning Attacks
Pith reviewed 2026-06-30 07:06 UTC · model grok-4.3
The pith
No defense succeeds in regularization-based continual learning when an adversary poisons a linear proportion of tasks with unbounded noise or pattern shifts.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By framing the adversary-defender interaction as an online zero-sum game in regularization-based continual learning, the paper establishes a fundamental performance limit: no defense succeeds when an adversary poisons a linear proportion of tasks by injecting unbounded noise or pattern shifts. For the regime of infrequent attacks it introduces a task-to-task verification mechanism that detects poisoning and reduces cumulative bias. For the regime of bounded noise it derives a robust defense that minimizes sensitivity to poisoned features and provably accelerates convergence.
What carries the argument
The online zero-sum game framing of adversary-defender interaction, which directly yields the performance limit on poisoning a linear proportion of tasks.
If this is right
- No defense works against linear-proportion poisoning with unbounded noise or pattern shifts.
- Task-to-task verification detects poisoning and reduces bias when attacks occur infrequently.
- A sensitivity-minimizing defense accelerates convergence when noise per attack remains bounded.
- The two special regimes (infrequent attacks and bounded noise) are the only ones admitting provable defense.
Where Pith is reading between the lines
- The same performance limit may extend to continual learning methods that do not rely on regularization.
- Detection mechanisms could be combined with the bounded-noise defense to handle mixed attack regimes.
- The zero-sum framing suggests studying whether adaptive or non-zero-sum adversary models change the linear-proportion threshold.
- The verification and sensitivity-minimization ideas could be tested on non-regularization continual learning algorithms.
Load-bearing premise
The adversary-defender interaction can be framed as an online zero-sum game in which the stated performance limit holds under the given poisoning model.
What would settle it
A concrete defense that keeps cumulative bias bounded and ensures convergence when the adversary poisons a linear proportion of tasks with unbounded noise would falsify the performance limit.
Figures
read the original abstract
Continual learning (CL), where a model is trained on a sequence of data tasks, is increasingly being adopted across key fields such as large language models and image recognition, yet it remains highly vulnerable to data poisoning that triggers learning divergence or severe excess risk. Despite these threats, a principled theoretical foundation in CL for understanding attack and defense remains lacking. In this paper, we develop a theoretical framework to analyze strategic attacks and defenses in regularization-based CL, a cornerstone of recent CL theory. By framing the adversary-defender interaction as an online zero-sum game, we first establish a fundamental performance limit: no defense succeeds when an adversary poisons a linear proportion of tasks by injecting unbounded noise or pattern shifts in regularization-based CL. We then analyze two possibly defensible scenarios: infrequent attacks and bounded noise per attack. For the former regime, we propose a task-to-task verification mechanism to detect data poisoning and reduce cumulative bias for learning convergence. For the latter regime, we derive a robust defense that minimizes the model's sensitivity to poisoned features, provably accelerating the convergence rate. Extensive experiments on realistic tasks further validate our theoretical results.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a theoretical framework for analyzing data poisoning attacks and defenses in regularization-based continual learning by modeling the adversary-defender interaction as an online zero-sum game. It establishes a fundamental performance limit showing that no defense succeeds when an adversary poisons a linear proportion of tasks via unbounded noise or pattern shifts. It then considers two restricted regimes (infrequent attacks and bounded noise per attack), proposing a task-to-task verification mechanism and a robust defense that minimizes sensitivity to poisoned features (with provable convergence acceleration), and reports experimental validation on realistic tasks.
Significance. A rigorously derived performance limit on defenses in regularization-based CL against linear-fraction poisoning would be a significant contribution, as it would clarify fundamental vulnerabilities in a setting where principled theory has been lacking. The game-theoretic framing, analysis of restricted regimes with explicit mechanisms, and experimental validation are potential strengths if the central derivation is shown to be non-tautological.
major comments (3)
- [theoretical framework] Theoretical framework section (zero-sum game definition): The defender's strategy space is not explicitly characterized, particularly whether regularization parameters or task weights may be chosen adaptively based on observed task statistics. Without this, it is unclear whether the claimed minimax value (no defense succeeds) is independently derived or follows by construction from the game setup under the unbounded noise model.
- [performance limit analysis] Performance limit result: The timing of unbounded noise or pattern-shift injection relative to the regularization term in the CL objective is not specified. This leaves open whether the 'no defense succeeds' bound holds for adaptive defenses that could respond after observing poisoned statistics or is an artifact of the modeling order.
- [infrequent attacks analysis] Infrequent attacks regime: The task-to-task verification mechanism lacks a formal statement of its detection guarantees (e.g., false-positive bounds or bias reduction under linear poisoning), making it impossible to verify that it restores convergence as claimed.
minor comments (2)
- [abstract] The abstract and introduction could more clearly separate the main impossibility result from the two restricted regimes that admit defenses.
- [theoretical framework] Notation for the online game (payoff functions, information sets) should be introduced with explicit definitions before the performance limit theorem.
Simulated Author's Rebuttal
We thank the referee for the constructive comments. We address each major comment point by point below.
read point-by-point responses
-
Referee: Theoretical framework section (zero-sum game definition): The defender's strategy space is not explicitly characterized, particularly whether regularization parameters or task weights may be chosen adaptively based on observed task statistics. Without this, it is unclear whether the claimed minimax value (no defense succeeds) is independently derived or follows by construction from the game setup under the unbounded noise model.
Authors: The defender's strategy space in the zero-sum game includes adaptive choices of regularization parameters and task weights based on observed statistics. The minimax result is derived by analyzing the worst-case adversary strategy against any defender strategy (including adaptive ones) when a linear proportion of tasks can be poisoned with unbounded noise; it does not follow tautologically from the setup. We will revise the theoretical framework section to explicitly define both players' strategy spaces and expand the derivation steps to clarify independence from construction. revision: yes
-
Referee: Performance limit result: The timing of unbounded noise or pattern-shift injection relative to the regularization term in the CL objective is not specified. This leaves open whether the 'no defense succeeds' bound holds for adaptive defenses that could respond after observing poisoned statistics or is an artifact of the modeling order.
Authors: The online game models sequential task arrival where the adversary selects which tasks to poison (linear fraction) and the noise level; the bound accounts for adaptive defender responses because the adversary can adapt its poisoning timing and content based on prior observations. The unbounded noise dominates the regularization term in the objective regardless of injection order. We will add an explicit statement of timing assumptions and a short argument showing the bound is robust to adaptive ordering in the performance limit analysis section. revision: yes
-
Referee: Infrequent attacks regime: The task-to-task verification mechanism lacks a formal statement of its detection guarantees (e.g., false-positive bounds or bias reduction under linear poisoning), making it impossible to verify that it restores convergence as claimed.
Authors: We agree a formal guarantee is required. The revised manuscript will add a theorem stating false-positive bounds and bias-reduction guarantees for the verification mechanism under linear poisoning in the infrequent regime, which establishes high-probability convergence restoration. revision: yes
Circularity Check
No significant circularity in derivation chain
full rationale
The paper frames adversary-defender interaction as an online zero-sum game to derive a performance limit on defenses under linear poisoning, then analyzes defensible regimes with verification and robust regularization. No equations or self-citations are quoted that reduce the limit to a tautology, fitted input, or imported uniqueness theorem; the game-theoretic setup and subsequent mechanisms for infrequent/bounded attacks constitute independent content. The derivation is self-contained and does not exhibit any of the enumerated circular patterns.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Analysis applies specifically to regularization-based continual learning.
- domain assumption Adversary can poison a linear proportion of tasks with unbounded noise or pattern shifts.
Reference graph
Works this paper leans on
-
[1]
Patricia S. Abril and Robert Plant. The patent holder's dilemma: Buy, sell, or troll?. Communications of the ACM. 2007. doi:10.1145/1188913.1188915
-
[2]
Deciding equivalances among conjunctive aggregate queries
Sarah Cohen and Werner Nutt and Yehoshua Sagic. Deciding equivalances among conjunctive aggregate queries. doi:10.1145/1219092.1219093
-
[3]
Special issue: Digital Libraries. 1996
1996
-
[4]
Understanding Policy-Based Networking
David Kosiur. Understanding Policy-Based Networking. 2001
2001
-
[7]
Proceedings of the 34th USENIX Conference on Security Symposium , articleno =
Guo, Zhen and Kumar, Abhinav and Tourani, Reza , title =. Proceedings of the 34th USENIX Conference on Security Symposium , articleno =. 2025 , isbn =
2025
-
[8]
The title of book two. 2008. doi:10.1007/3-540-09237-4
-
[9]
Asad Z. Spector. Achieving application requirements. Distributed Systems. 1990. doi:10.1145/90417.90738
-
[10]
Douglass and David Harel and Mark B
Bruce P. Douglass and David Harel and Mark B. Trakhtenbrot. Statecarts in use: structured analysis and object-orientation. Lectures on Embedded Systems. 1998. doi:10.1007/3-540-65193-4_29
-
[11]
Donald E. Knuth. The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd. ed.). 1997
1997
-
[12]
Donald E. Knuth. The Art of Computer Programming. 1998
1998
-
[13]
Structured Variational Inference Procedures and their Realizations (as incol)
Dan Geiger and Christopher Meek. Structured Variational Inference Procedures and their Realizations (as incol). Proceedings of Tenth International Workshop on Artificial Intelligence and Statistics, The Barbados
-
[14]
Stan W. Smith. An experiment in bibliographic mark-up: Parsing metadata for XML export. Proceedings of the 3rd. annual workshop on Librarians and Computers. 2010. doi:99.9999/woot07-S422
2010
-
[15]
Catch me, if you can: Evading network signatures with web-based polymorphic worms
Matthew Van Gundy and Davide Balzarotti and Giovanni Vigna. Catch me, if you can: Evading network signatures with web-based polymorphic worms. Proceedings of the first USENIX workshop on Offensive Technologies
-
[16]
Sten Andler. Predicate Path expressions. Proceedings of the 6th. ACM SIGACT-SIGPLAN symposium on Principles of Programming Languages. 1979. doi:10.1145/567752.567774
-
[17]
LOGICS of Programs: AXIOMATICS and DESCRIPTIVE POWER
David Harel. LOGICS of Programs: AXIOMATICS and DESCRIPTIVE POWER. 1978
1978
-
[18]
Anisi , title =
David A. Anisi , title =
-
[19]
Clarkson
Kenneth L. Clarkson. Algorithms for Closest-Point Problems (Computational Geometry). 1985
1985
-
[20]
Introduction to Bayesian Statistics
Harry Thornburg. Introduction to Bayesian Statistics. 2001
2001
-
[21]
CLIFFORD: a Maple 11 Package for Clifford Algebra Computations, version 11
Rafal Ablamowicz and Bertfried Fauser. CLIFFORD: a Maple 11 Package for Clifford Algebra Computations, version 11. 2007
2007
-
[22]
Stats and Analysis
Poker-Edge.Com. Stats and Analysis. 2006
2006
-
[23]
A more perfect union
Barack Obama. A more perfect union. 2008
2008
-
[24]
The fountain of youth
Joseph Scientist. The fountain of youth. 2009
2009
-
[25]
Solder man
Dave Novak. Solder man. ACM SIGGRAPH 2003 Video Review on Animation theater Program: Part I - Vol. 145 (July 27--27, 2003). 2003. doi:99.9999/woot07-S422
2003
-
[26]
Interview with Bill Kinder: January 13, 2005
Newton Lee. Interview with Bill Kinder: January 13, 2005. Comput. Entertain. 2005. doi:10.1145/1057270.1057278
-
[27]
The Enabling of Digital Libraries
Bernard Rous. The Enabling of Digital Libraries. Digital Libraries. 2008
2008
-
[29]
(new) Finding minimum congestion spanning trees , journal =
Werneck, Renato and Setubal, Jo\. (new) Finding minimum congestion spanning trees , journal =. doi:10.1145/351827.384253 , acmid = 384253, publisher =
-
[31]
Conti, Mauro and Di Pietro, Roberto and Mancini, Luigi V. and Mei, Alessandro , title =. Inf. Fusion , volume =. 2009 , issn =. doi:10.1016/j.inffus.2009.01.002 , acmid =
-
[32]
Li, Cheng-Lun and Buyuktur, Ayse G. and Hutchful, David K. and Sant, Natasha B. and Nainwal, Satyendra K. , title =. CHI '08 extended abstracts on Human factors in computing systems , year =. doi:10.1145/1358628.1358946 , acmid =
-
[33]
, title =
Hollis, Billy S. , title =. 1999 , isbn =
1999
-
[34]
Goossens, Michel and Rahtz, S. P. and Moore, Ross and Sutor, Robert S. , title =. 1999 , isbn =
1999
-
[35]
and Rosenberg, Arnold L
Buss, Jonathan F. and Rosenberg, Arnold L. and Knott, Judson D. , title =. 1987 , source =
1987
-
[36]
CHI '08: CHI '08 extended abstracts on Human factors in computing systems , year =
, note =. CHI '08: CHI '08 extended abstracts on Human factors in computing systems , year =
-
[37]
Algorithms for Closest-Point Problems (Computational Geometry) , year =
Clarkson, Kenneth Lee , advisor =. Algorithms for Closest-Point Problems (Computational Geometry) , year =
-
[38]
SIGCOMM Comput. Commun. Rev. , year =
-
[39]
Targeted Forgetting and False Memory Formation in Continual Learners through Adversarial Backdoor Attacks , year=
Umer, Muhammad and Dawson, Glenn and Polikar, Robi , booktitle=. Targeted Forgetting and False Memory Formation in Continual Learners through Adversarial Backdoor Attacks , year=
-
[40]
IEEE TCSC Executive Committee , booktitle =. 2004 , isbn =. doi:http://dx.doi.org/10.1109/ICWS.2004.64 , acmid =
-
[41]
Distributed systems (2nd Ed.) , year =
-
[42]
, title =
Petrie, Charles J. , title =. 1986 , source =
1986
-
[43]
Donald E. Knuth. Seminumerical Algorithms. 1981
1981
-
[44]
E-commerce and cultural values , year =
Kong, Wei-Chang , Title =. E-commerce and cultural values , year =
-
[45]
E-commerce and cultural values , year =
Kong, Wei-Chang , type =. E-commerce and cultural values , year =
-
[46]
Chapter 9 , booktitle =
Kong, Wei-Chang , editor =. Chapter 9 , booktitle =. 2002 , address =
2002
-
[47]
E-commerce and cultural values , editor =
Kong, Wei-Chang , title =. E-commerce and cultural values , editor =. 2003 , isbn =
2003
-
[48]
E-commerce and cultural values - (InBook-num-in-chap) , chapter =
Kong, Wei-Chang , editor =. E-commerce and cultural values - (InBook-num-in-chap) , chapter =. 2004 , address =
2004
-
[49]
E-commerce and cultural values (Inbook-text-in-chap) , chapter =
Kong, Wei-Chang , editor =. E-commerce and cultural values (Inbook-text-in-chap) , chapter =. 2005 , address =
2005
-
[50]
E-commerce and cultural values (Inbook-num chap) , chapter =
Kong, Wei-Chang , editor =. E-commerce and cultural values (Inbook-num chap) , chapter =. 2006 , address =
2006
-
[51]
Microelectron
Mehdi Saeedi and Morteza Saheb Zamani and Mehdi Sedighi , title =. Microelectron. J. , volume =. 2010 , pages =
2010
-
[52]
Mehdi Saeedi and Morteza Saheb Zamani and Mehdi Sedighi and Zahra Sasanian , title =. J. Emerg. Technol. Comput. Syst. , volume =
-
[53]
Kirschmer, Markus and Voight, John , title =. SIAM J. Comput. , issue_date =. 2010 , issn =. doi:https://doi.org/10.1137/080734467 , acmid =
-
[54]
Hoare, C. A. R. , title =. Structured programming (incoll) , editor =. 1972 , isbn =
1972
-
[55]
History of programming languages I (incoll) , editor =
Lee, Jan , title =. History of programming languages I (incoll) , editor =. 1981 , isbn =. doi:http://doi.acm.org/10.1145/800025.1198348 , acmid =
-
[56]
, title =
Dijkstra, E. , title =. Classics in software engineering (incoll) , year =
-
[57]
Wenzel, Elizabeth M. , title =. Multimedia interface design (incoll) , year =. doi:10.1145/146022.146089 , acmid =
-
[58]
, title =
Mumford, E. , title =. Critical issues in information systems research (incoll) , year =
-
[59]
and Golden, Donald G
McCracken, Daniel D. and Golden, Donald G. , title =. 1990 , isbn =
1990
-
[60]
The analysis of linear partial differential operators
H. The analysis of linear partial differential operators. 1985 , PAGES =
1985
-
[61]
IEEE", address =
A. Adya and P. Bahl and J. Padhye and A.Wolman and L. Zhou , title =. Proceedings of the IEEE 1st International Conference on Broadnets Networks (BroadNets'04) , publisher = "IEEE", address = "Los Alamitos, CA", year =
-
[62]
I. F. Akyildiz and W. Su and Y. Sankarasubramaniam and E. Cayirci , title =. Comm. ACM , volume = 38, number = "4", year =
-
[63]
I. F. Akyildiz and T. Melodia and K. R. Chowdhury , title =. Computer Netw. , volume = 51, number = "4", year =
-
[64]
ACM", address =
P. Bahl and R. Chancre and J. Dungeon , title =. Proceeding of the 10th International Conference on Mobile Computing and Networking (MobiCom'04) , publisher = "ACM", address = "New York, NY", year =
-
[65]
8 (Special Issue on Sensor Networks)
D. Culler and D. Estrin and M. Srivastava , title =. IEEE Comput. , volume = 37, number = "8 (Special Issue on Sensor Networks)", publisher = "IEEE", address = "Los Alamitos, CA", year =
-
[66]
Natarajan and M
A. Natarajan and M. Motani and B. de Silva and K. Yap and K. C. Chua , title =. Network Architectures , editor =. 960935712
-
[67]
Tzamaloukas and J
A. Tzamaloukas and J. J. Garcia-Luna-Aceves , title =
-
[68]
Zhou and J
G. Zhou and J. Lu and C.-Y. Wan and M. D. Yarvis and J. A. Stankovic , title =
-
[69]
Mapping Powerlists onto Hypercubes
Jacob Kornerup. Mapping Powerlists onto Hypercubes. 1994
1994
-
[70]
Automatic Parallelization for Distributed-Memory Multiprocessing Systems
Michael Gerndt. Automatic Parallelization for Distributed-Memory Multiprocessing Systems
-
[71]
J. E. Archer, Jr. and R. Conway and F. B. Schneider. User recovery and reversal in interactive systems. ACM Trans. Program. Lang. Syst
-
[72]
D. D. Dunlop and V. R. Basili. Generalizing specifications for uniformly implemented loops. ACM Trans. Program. Lang. Syst
-
[73]
Heering and P
J. Heering and P. Klint. Towards monolingual programming environments. ACM Trans. Program. Lang. Syst
-
[74]
Donald E. Knuth. The book
-
[75]
2023 IEEE International Conference on Image Processing (ICIP) , pages=
Data poisoning attack aiming the vulnerability of continual learning , author=. 2023 IEEE International Conference on Image Processing (ICIP) , pages=. 2023 , organization=
2023
-
[76]
2022 International Joint Conference on Neural Networks (IJCNN) , pages=
Targeted data poisoning attacks against continual learning neural networks , author=. 2022 International Joint Conference on Neural Networks (IJCNN) , pages=. 2022 , organization=
2022
-
[77]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
BrainWash: A Poisoning Attack to Forget in Continual Learning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[78]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Online continual learning on a contaminated data stream with blurry task boundaries , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[79]
arXiv preprint arXiv:2207.05225 , year=
Susceptibility of continual learning against adversarial attacks , author=. arXiv preprint arXiv:2207.05225 , year=
-
[80]
ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=
Towards adversarially robust continual learning , author=. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2023 , organization=
2023
-
[81]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Metamix: Towards corruption-robust continual learning with temporally self-adaptive data transformation , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[82]
European Conference on Computer Vision , pages=
Few-shot class incremental learning with attention-aware self-adaptive prompt , author=. European Conference on Computer Vision , pages=. 2024 , organization=
2024
-
[83]
2022 International Joint Conference on Neural Networks (IJCNN) , pages=
Adversarially robust continual learning , author=. 2022 International Joint Conference on Neural Networks (IJCNN) , pages=. 2022 , organization=
2022
-
[84]
Data Poisoning Attacks against Online Learning
Data poisoning attacks against online learning , author=. arXiv preprint arXiv:1808.08994 , year=
work page internal anchor Pith review Pith/arXiv arXiv
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.