REVIEW 8 cited by
SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have highlighted the necessity of effective unlearning mechanisms to comply with data regulations and ethical AI practices. LLM unlearning aims at removing undesired data influences and associated model capabilities without compromising utility beyond the scope of unlearning. While interest in studying LLM unlearning is growing, the impact of the optimizer choice for LLM unlearning remains unexplored. In this work, we shed light on the significance of optimizer selection in LLM unlearning for the first time, establishing a clear connection between second-order optimization and influence unlearning (a classical approach using influence functions to update the model for data influence removal). This insight propels us to develop a second-order optimization-based LLM unlearning framework, termed Second-Order UnLearning (SOUL), which extends the static, one-shot model update using influence unlearning to a dynamic, iterative unlearning process. Our extensive experiments show that SOUL consistently outperforms conventional first-order methods across various unlearning tasks, models, and metrics, indicating that second-order optimization offers an effective and broadly applicable solution for LLM unlearning. Codes are available at https://github.com/OPTML-Group/SOUL.
Forward citations
Cited by 8 Pith papers
-
Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning
PALU shows that unlearning only needs local intervention—the first few tokens of the sensitive span and the top-k logits—not full-sequence, full-vocabulary suppression.
-
Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning
DiPO is a distribution-level unlearning method that constructs preference distributions from the model's own high-confidence logits and achieves state-of-the-art forget quality on TOFU while preserving utility.
-
Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs
Unlearning in LLMs leaves detectable 'fingerprints' that let a simple classifier distinguish an unlearned model from its original, even on unrelated prompts.
-
Model Unlearning via Sparse Autoencoder Subspace Guided Projections
SSPU uses SAE-derived subspaces to guide weight updates, lowering WMDP-Cyber accuracy by 3.22% more than RMU while largely preserving MMLU, TruthfulQA, and GSM8K performance.
-
GROM: Gradient-Free Rapid One-Shot Machine Unlearning
A single closed-form ridge update to selected MLP layers removes targeted knowledge from LLMs in seconds, with state-of-the-art forgetting-utility trade-offs and quantization robustness.
-
Module-Aware Parameter-Efficient Machine Unlearning on Transformers
MAPE-Unlearn uses Fisher-information-based scores and greedy search to select important heads and filters, then applies sparse unlearning updates, claiming improved efficacy-fidelity trade-offs on Transformers.
-
Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection
An LLM unlearning method that projects hidden states so harmful information is irreversibly removed while useful knowledge is preserved.
-
UCD: Unlearning in LLMs via Contrastive Decoding
UCD steers an LLM away from forget-set content at inference time by mixing in the difference between forget-tuned and retain-tuned small models.
Discussion (0). Continue with ORCID to comment.