REVIEW 4 cited by
To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
LLMs have been found to memorize training textual sequences and regurgitate verbatim said sequences during text generation time. This fact is known to be the cause of privacy and related (e.g., copyright) problems. Unlearning in LLMs then takes the form of devising new algorithms that will properly deal with these side-effects of memorized data, while not hurting the model's utility. We offer a fresh perspective towards this goal, namely, that each textual sequence to be forgotten should be treated differently when being unlearned based on its degree of memorization within the LLM. We contribute a new metric for measuring unlearning quality, an adversarial attack showing that SOTA algorithms lacking this perspective fail for privacy, and two new unlearning methods based on Gradient Ascent and Task Arithmetic, respectively. A comprehensive performance evaluation across an extensive suite of NLP tasks then mapped the solution space, identifying the best solutions under different scales in model capacities and forget set sizes and quantified the gains of the new approaches.
Forward citations
Cited by 4 Pith papers
-
LoReUn: Data Itself Implicitly Provides Cues to Improve Machine Unlearning
LoReUn, a plug-in loss-based reweighting strategy, improves approximate machine unlearning by focusing updates on hard-to-forget low-loss data points.
-
Leveraging Per-Instance Privacy for Machine Unlearning
Per-instance privacy losses, estimated from gradient norms during training, predict the number of fine-tuning steps needed for machine unlearning and rank data points by unlearning difficulty.
-
DUSK: Do Not Unlearn Shared Knowledge
DUSK benchmarks machine unlearning under overlapping forget and retain documents, showing existing methods remove surface text but fail to preserve shared knowledge while erasing unique content.
-
SEPS: A Separability Measure for Robust Unlearning in LLMs
SEPS measures separation of forget and retain queries in mixed prompts, and Mixed Prompt training makes unlearned LLMs much better at this separation.
Discussion (0). Continue with ORCID to comment.