REVIEW 7 cited by
RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) inevitably memorize sensitive, copyrighted, and harmful knowledge from the training corpus; therefore, it is crucial to erase this knowledge from the models. Machine unlearning is a promising solution for efficiently removing specific knowledge by post hoc modifying models. In this paper, we propose a Real-World Knowledge Unlearning benchmark (RWKU) for LLM unlearning. RWKU is designed based on the following three key factors: (1) For the task setting, we consider a more practical and challenging unlearning setting, where neither the forget corpus nor the retain corpus is accessible. (2) For the knowledge source, we choose 200 real-world famous people as the unlearning targets and show that such popular knowledge is widely present in various LLMs. (3) For the evaluation framework, we design the forget set and the retain set to evaluate the model's capabilities across various real-world applications. Regarding the forget set, we provide four four membership inference attack (MIA) methods and nine kinds of adversarial attack probes to rigorously test unlearning efficacy. Regarding the retain set, we assess locality and utility in terms of neighbor perturbation, general ability, reasoning ability, truthfulness, factuality, and fluency. We conduct extensive experiments across two unlearning scenarios, two models and six baseline methods and obtain some meaningful findings. We release our benchmark and code publicly at http://rwku-bench.github.io for future work.
Forward citations
Cited by 7 Pith papers
-
Towards Evaluation for Real-World LLM Unlearning
DCUE evaluates LLM unlearning by comparing core-token confidence score distributions of the unlearned model and the original model, corrected by a validation set, using the Kolmogorov-Smirnov test.
-
What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests
WikiMem, a Wikidata-derived canary dataset and a calibrated NLL-ranking metric, identifies which human-fact associations an LLM has memorized, with higher rates for famous people and larger models.
-
LLM Unlearning Should Be Form-Independent
Existing LLM unlearning is form-dependent; the new ORT benchmark measures this, and the training-free ROCR edit reduces it by redirecting concept representations.
-
Existing Large Language Model Unlearning Evaluations Are Inconclusive
Existing LLM unlearning evaluations are inconclusive: they can inject new information, depend heavily on task format, and rely on spurious correlations.
-
R-TOFU: Unlearning in Large Reasoning Models
R-TOFU shows that answer-level unlearning is insufficient for large reasoning models because residual knowledge persists in chain-of-thought traces.
-
SoK: Machine Unlearning for Large Language Models
A new taxonomy for LLM unlearning distinguishes removal-intended from suppression-intended methods, and argues that gradient ascent methods functionally behave like suppression.
-
Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models
A position paper urging a shift from data-tracing to knowledge-tracing machine unlearning for foundation models, supported by a CLIP case study that shows current methods struggle to generalize.
Discussion (0). Sign in to comment.