REVIEW 6 cited by
UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Mitigating the retention of sensitive or private information in large language models is essential for enhancing privacy and safety. Existing unlearning methods, like Gradient Ascent and Negative Preference Optimization, directly tune models to remove unwanted information. However, these methods often become unstable because they fine-tune by maximizing cross-entropy loss, which is the opposite of traditional loss minimization in learning. This reversal creates instability, especially on larger datasets, as the model struggles to balance unlearning with maintaining language capacity, leading to over-unlearning. In this paper, we introduce UnDIAL (Unlearning via Self-Distillation on Adjusted Logits), a novel and robust unlearning method. Our approach leverages self-distillation to adjust logits and selectively reduce the influence of targeted tokens. This technique ensures smooth convergence and avoids catastrophic forgetting, even in challenging unlearning tasks with large datasets and sequential unlearning requests. Extensive experiments show that UnDIAL can achieve both robustness in unlearning and scalability while maintaining stable training dynamics and resilience to hyperparameter tuning.
Forward citations
Cited by 6 Pith papers
-
Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration
Margin Calibration, a non-saturating margin-anchored LoRA polish, crosses the margin cliff and cuts post-attack relearn recovery on all 97 populated cells in the paper's stress matrix.
-
A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearning
A circuit-similarity score predicts which samples an LLM unlearning method will fail to erase, with hard samples relying on deeper, output-facing pathways.
-
GROM: Gradient-Free Rapid One-Shot Machine Unlearning
A single closed-form ridge update to selected MLP layers removes targeted knowledge from LLMs in seconds, with state-of-the-art forgetting-utility trade-offs and quantization robustness.
-
Automating Evaluation of Diffusion Model Unlearning with (Vision-) Language Model World Knowledge
A new evaluation tool uses (vision-)language model world knowledge to rank nearby concepts and craft adversarial prompts, showing that diffusion unlearning is incomplete and that semantic similarity correlates with co...
-
Weak-Driven Learning: How Weak Agents make Strong Agents Stronger
Mixing an LLM's logits with an earlier weak checkpoint during fine-tuning yields math and code accuracy gains beyond standard SFT saturation.
-
Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Models
Step-by-step reasoning prompts can recover purportedly erased facts from unlearned LLMs, but the paper's quantitative evidence is internally inconsistent.
Discussion (0). Continue with ORCID to comment.