REVIEW 4 major objections 5 minor 1 cited by
GRAIL: Gradient-Based Adaptive Unlearning for Privacy and Copyright in LLMs
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper claims that GRAIL can erase privacy and copyright knowledge from a 7-billion-parameter chat model in one pass while retaining other knowledge better than existing methods.
desk verdict Plausible multi-domain unlearning extension with a backward sign in Algorithm 1 and a missing random-mask control; the results may be real but aren't yet checkable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a per-layer, per-dataset parameter mask built from gradient magnitudes. For dataset $D_x$, each target answer is replaced by a random label, backpropagation is run three times, and the averaged absolute gradient per parameter, $$\|\mathbf{g}^\ell_{x,j}\| = \frac{1}{n}\sum_{i=1}^n |g^\ell_{x,i}[j]|,$$ ranks parameters; the top $k\%$ in layer $\ell$ form the set $T^\ell(D_x)$. Two masks are then derived: the unlearning-retention overlap (OP-UR), parameters that fall in the top set for both unlearning and retention knowledge across the two domains, and the cross-domain retention overlap (OP-RR), parameters in the top retention sets of both privacy and copyright. These parameters are frozen; the remaining parameters are updated by gradient ascent on unlearning data and gradient descent on retention data, with $k_{\text{OP-UR}}=10\%$ and $k_{\text{OP-RR}}=20\%$ in the reported experiments.
What would settle it
A decisive test would take a fact whose storage location is already known from single-parameter editing—changing one parameter flips the answer, changing others does not—and check whether GRAIL's gradient masks include that causal parameter and exclude the irrelevant ones; if they do not, the localization premise is false.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that unlearning and retention scopes in multi-domain LLM knowledge can be separated at the level of individual parameters rather than whole layers, and that doing so makes simultaneous privacy and copyright unlearning safe. The evidence is a three-stage procedure: random-label substitution to obtain stable gradients, adaptive top-k parameter localization per layer for each of the four datasets, and freezing of two overlap masks (unlearning-retention overlap and cross-domain retention overlap) before applying gradient ascent to forget and gradient descent to retain. In the reported experiments on two 7-billion-parameter chat models, GRAIL keeps unlearning success in the 90-99 range while raising privacy retention from 72.79 to 85.34 and copyright retention from 89.00 to 93.87 on the first model; its average harmonic success is 92.10 versus 87.78 for the layer-wise baseline. The paper further shows that sequential or simply combined multi-domain unlearning damages the other domain, and that the top-10% parameter sets for unlearning and retention overlap substantially (up to 61% within a domain, up to 34% across domains), which is the phenomenon the adaptive masks are designed to handle.
Load-bearing premise
The argument depends on the assumption that the parameters with the largest average gradient after random-label substitution are exactly the parameters that store a dataset's knowledge; if that mapping is wrong, GRAIL freezes the wrong parameters and both erasure and retention fail.
Editorial extensions
If this is right
- A single unlearning pass can handle privacy and copyright together: GRAIL's simultaneous update avoids the cross-domain retention damage that the paper shows for sequential or fully combined unlearning.
- The balance, measured by harmonic success, is higher than the prior layer-wise localization baseline in both domains (e.g., 87.95 vs 82.20 for privacy and 96.25 vs 93.35 for copyright on the main 7B chat model).
- The frozen-parameter mask makes forgetting less destructive: retention can improve by about 17% relative to the prior best on privacy, while unlearning success stays above 90, so the usual unlearning-retention trade-off is partially decoupled.
- General task performance after erasure remains close to the vanilla model (average 0.42 vs 0.44 on the paper's five benchmarks), suggesting the erasure is localized and does not collapse the model's general abilities.
- Because the updates are applied to low-rank adapter parameters on top of a frozen base model, compliance erasure becomes a fine-tuning-scale operation rather than a retraining-scale one.
Reading between the lines
- Inference: the same mask-building recipe could be reused for non-legal forgetting targets — toxicity, outdated facts, named entities — without modifying the algorithm; the paper only reports privacy and copyright.
- Inference: because the masks are computed once and then frozen, repeated or continual unlearning requests could be processed incrementally by recomputing masks on the current model; the paper does not test this.
- Inference: the reported overlaps among the top-10% parameter sets (up to 61% between unlearning and retention within a domain, up to 34% across domains) mean a clean separation is impossible for heavily entangled facts; GRAIL's gains are likely a trade-off tilted toward retention, and the paper does not measure how much target knowledge survives in the highest-overlap cases.
- Inference: the method's success may depend on the low-rank adapter parameterization used in the experiments; on fully fine-tuned or larger models the gradient top-k sets could be different, so the 17% retention gain is established only for this setting.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. GRAIL proposes a multi-domain machine unlearning framework for LLMs, targeting privacy and copyright knowledge. The method first localizes parameters associated with unlearning and retention scopes by averaging gradient magnitudes after random-label substitution, then freezes parameters that overlap between unlearning and retention (OP-UR) or between retention across domains (OP-RR), and finally applies gradient updates on the remaining parameters. Experiments on KnowUnDo with LLaMA-2-7B-Chat and Qwen-1.5-7B-Chat compare GRAIL against gradient ascent, random-label fine-tuning, adversarial unlearning, and gradient-ascent-plus-descent variants, reporting improved retention success with comparable unlearning success. The paper claims up to 17% stronger knowledge retention than the previous state-of-the-art method.
Significance. If the reported results hold, GRAIL would be a useful contribution to multi-domain unlearning, addressing a realistic scenario where privacy and copyright knowledge overlap. The paper identifies a concrete problem, proposes a parameter-wise localization mechanism, and provides experiments on two 7B-scale models with several baselines and ablations. The harmonic success metric is a reasonable way to quantify the trade-off. However, the central claims rest on two load-bearing points that are not adequately established: the correctness of the gradient update signs in Algorithm 1 and the validity of gradient-magnitude-based localization as a causal account of where knowledge resides. The lack of random-mask controls, error bars, and code release further limits confidence. These issues are fixable but require substantive revision.
major comments (4)
- [Algorithm 1, Stage 3] The update rules for unlearning and retention appear to have reversed signs. For (x,y) in D_U, the algorithm sets theta <- theta + eta * grad_theta log P(y|x), which is gradient ascent on the log-likelihood and therefore increases the probability of the target output, reinforcing the very knowledge that should be forgotten. For retention, theta <- theta - eta * grad_theta log P(y|x) decreases the log-likelihood, which would harm retention. In standard gradient-based unlearning, one maximizes the loss (equivalently minimizes log-likelihood) for unlearn data and minimizes the loss for retain data. As written, Algorithm 1 cannot implement the method described in Section III.A, where the authors state that they 'combine gradient ascent and gradient descent' to balance unlearning and retention. Please correct the signs, e.g., theta - eta * grad_theta log P(y|x) for unlearning and theta + eta * grad_theta log P(y|x) for retention, or explicitly define a surrogate loss whose gradient has the opposite sign.
- [Section III.C-D and Table VI] The localization assumption is not validated against a random-mask control. Equations (3)-(4) select top-k% parameters by average gradient magnitude after random-label substitution, and these selections directly determine the OP-UR and OP-RR masks. The ablations in Table VI compare the full method with variants that remove one or both masks, but they never replace the gradient-selected masks with random parameter subsets of the same size. Without such a control, the observed retention improvements could be explained by a simpler mechanism: freezing roughly 10-20% of parameters reduces the magnitude and scope of updates, thereby preserving retention regardless of which parameters are frozen. This is a load-bearing issue because the paper's central claim is that gradient information 'precisely distinguish[es] the unlearning scope from the retention scope.' Please add a random-mask control and, if the gradient-selected masks outperform random masks, report the comparison.
- [Section IV.A, Fig. 4] The thresholds kOP-UR=10% and kOP-RR=20% are selected using the same KnowUnDo benchmark on which the final results are reported, and Fig. 4 shows sensitivity trends without error bars or multiple seeds. Since these thresholds materially affect the OP-UR and OP-RR masks, the absence of a held-out validation split or variance estimates makes it difficult to assess whether the reported gains are robust or reflect tuning on the evaluation set. Please provide multiple-run statistics and, if possible, a validation-based selection procedure for kOP-UR and kOP-RR.
- [Section V.A and Tables I-III] No error bars or confidence intervals are reported for the main results. The central quantitative claim (e.g., privacy retention improving from 72.79 to 85.34 in Table I, and the 17% relative improvement) rests on single runs. Given that unlearning methods are sensitive to initialization and optimization randomness, the absence of repeated runs or statistical significance testing weakens the conclusions. Please report means and standard deviations over at least three runs, or justify why the observed differences are beyond expected noise.
minor comments (5)
- [Algorithm 1, Stage 2] There is a typo in the stage title: 'Adaptive Parmeter-wise Localization' should be 'Adaptive Parameter-wise Localization.'
- [Abstract and Section V.A] The abstract states 'up to 17% stronger knowledge retention success,' which is a relative improvement (85.34/72.79 = 1.172). Please state the percentage-point difference as well to avoid ambiguity.
- [Section III.D] The notation Dx is used ambiguously: in Eq. (3) Dx is described as an element of D, but later T(Dx) is defined for each dataset. Please clarify the indexing.
- [References] Reference [16] appears to be a neuroscience paper (Kim et al., 2015) and is cited in a list of legal principles [13]-[16]; this seems unrelated. Please verify and replace with an appropriate legal or policy reference.
- [Tables I and II] Several entries use notation like '>1010' and '109'; these should be typeset as superscripts (e.g., >10^10, 10^9) to avoid confusion.
Circularity Check
No significant circularity: GRAIL's method and evaluation are empirically linked, not equivalent by construction.
full rationale
GRAIL's pipeline is an empirical intervention rather than a derivation that reduces to its inputs. The localization step (Eqs. (3)-(4)) defines 'critical' parameters as top-k by averaged gradient magnitude after random-label substitution; this is an explicit operational criterion, not a quantity fitted to the reported US/RS numbers. Stage 3 then performs gradient ascent on D_U and gradient descent on D_R with a frozen mask, so the reported Unlearning Success and Retention Success are in-sample training objectives on the same sets used to build the masks. That is standard practice for unlearning benchmarks and does not by construction guarantee the claimed 17% retention improvement; the improvement is an empirical comparison against baselines under the same objectives. The paper's self-citations ([16], [27]-[29]) are to unrelated prior work and are not load-bearing; the knowledge-localization inspiration is attributed to external works [5], [20], [25]-[26]. Hyperparameters kOP-UR and kOP-RR are set to 10% and 20%, but tuning thresholds on the benchmark, while a validity concern, does not make the central claim equivalent to a fitted parameter. No equation in the paper equates the localization masks with the evaluation metrics, and no result is imported from a self-citation chain. The skeptical concern about missing random-mask controls is a robustness issue, not circularity.
Assumptions & free parameters
free parameters (3)
- kOP-UR =
10% of parameters per layer
- kOP-RR =
20% of parameters per layer
- gradient averaging trials =
3
assumptions (4)
- domain assumption Gradient magnitude after random-label substitution localizes the parameters that store knowledge for a dataset.
- domain assumption The KnowUnDo labels (PU, PR, CU, CR) faithfully distinguish the legal and practical unlearning and retention scopes for privacy and copyright.
- domain assumption Freezing all parameters that overlap unlearning and retention (OP-UR) still allows effective unlearning of the targeted knowledge.
- domain assumption The LoRA-trained 'vanilla' model is a sufficient stand-in for a real pretrained LLM exposed to private and copyrighted text.
Cite this review
Pith. "Pith review of GRAIL: Gradient-Based Adaptive Unlearning for Privacy and Copyright in LLMs." pith.science (2026). https://pith.science/paper/6C6YDZNA
@misc{pith2026250412681,
author = {Pith},
title = {Pith review of: GRAIL: Gradient-Based Adaptive Unlearning for Privacy and Copyright in LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/6C6YDZNA}},
note = {Machine review of arXiv:2504.12681}
}
read the original abstract
Large Language Models (LLMs) trained on extensive datasets often learn sensitive information, which raises significant social and legal concerns under principles such as the "Right to be forgotten." Retraining entire models from scratch to remove undesired information is both costly and impractical. Furthermore, existing single-domain unlearning methods fail to address multi-domain scenarios, where knowledge is interwoven across domains such as privacy and copyright, creating overlapping representations that lead to excessive knowledge removal or degraded performance. To tackle these issues, we propose GRAIL (GRadient-based AdaptIve unLearning), a novel multi-domain unlearning framework. GRAIL leverages gradient information from multiple domains to precisely distinguish the unlearning scope from the retention scope, and applies an adaptive parameter-wise localization strategy to selectively remove targeted knowledge while preserving critical parameters for each domain. Experimental results on unlearning benchmarks show that GRAIL achieves unlearning success on par with the existing approaches, while also demonstrating up to 17% stronger knowledge retention success compared to the previous state-of-art method. Our findings establish a new paradigm for effectively managing and regulating sensitive information in large-scale pre-trained language models.
Figures
Forward citations
Cited by 1 Pith paper
-
SoK: Machine Unlearning for Large Language Models
A new taxonomy for LLM unlearning distinguishes removal-intended from suppression-intended methods, and argues that gradient ascent methods functionally behave like suppression.
Reference graph
Works this paper leans on
-
[1]
OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, and D. Almeida, “Gpt-4 technical report,” in arXiv preprint arXiv:2303.08774, 2023
arXiv 2023
-
[2]
Large language models: A survey,
S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Ama- triain, and J. Gao, “Large language models: A survey,” in arXiv preprint arXiv:2402.06196, 2024
arXiv 2024
-
[3]
Large language models: A comprehensive survey on architectures, applications, and challenges,
V . Veeramachaneni, “Large language models: A comprehensive survey on architectures, applications, and challenges,” Advanced Innovations in Computer Programming Languages (AICPL) , vol. 7, no. 1, 2024
work page 2024
-
[4]
Right to be forgotten in the age of machine learning,
Q.-V . Dang, “Right to be forgotten in the age of machine learning,” in International Conference on Advances in Digital Science (ICADS) , 2021, p. 403–411
work page 2021
-
[5]
To forget or not? towards practical knowledge unlearning for large language models,
B. Tian, X. Liang, S. Cheng, Q. Liu, M. Wang, D. Sui, X. Chen, H. Chen, and N. Zhang, “To forget or not? towards practical knowledge unlearning for large language models,” in Empirical Methods in Natural Language Processing (EMNLP), 2024, pp. 1524–1537
work page 2024
-
[6]
A closer look at machine unlearning for large language models,
X. Yuan, T. Pang, C. Du, K. Chen, W. Zhang, and M. Lin, “A closer look at machine unlearning for large language models,” in arXiv preprint arXiv:2410.08109, 2024
arXiv 2024
-
[7]
Practical unlearning for large language models,
C. Gao, L. Wang, C. Weng, X. Wang, and Q. Zhu, “Practical unlearning for large language models,” in arXiv preprint arXiv:2407.10223 , 2024
arXiv 2024
-
[8]
Knowledge unlearning for mitigating privacy risks in language models,
J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, “Knowledge unlearning for mitigating privacy risks in language models,” in Association for Computational Linguistics (ACL) , 2023, pp. 14 389– 14 408
work page 2023
Show all 33 references
-
[9]
Large language model unlearning,
Y . Yao, X. Xu, and Y . Liu, “Large language model unlearning,” in Socially Responsible Language Modelling Research (SoLaR) , 2023
2023
-
[10]
Unlearn what you want to forget: Efficient unlearning for LLMs,
J. Chen and D. Yang, “Unlearn what you want to forget: Efficient unlearning for LLMs,” in Association for Computational Linguistics (ACL), 2023, pp. 12 041–12 052
2023
-
[11]
Towards safer large language models through machine unlearning,
Z. Liu, G. Dou, Z. Tan, Y . Tian, and M. Jiang, “Towards safer large language models through machine unlearning,” in Association for Computational Linguistics (ACL) , 2024, pp. 1817–1829
2024
-
[12]
Avoiding copyright infringement via large language model unlearning,
G. Dou, Z. Liu, Q. Lyu, K. Ding, and E. Wong, “Avoiding copyright infringement via large language model unlearning,” in arXiv preprint arXiv:2406.10952, 2024
2024 arXiv
-
[13]
United states code (usc),
U.S., “United states code (usc),” https://uscode.house.gov/browse.xhtml, 2018
2018
-
[14]
California consumer privacy act (ccpa),
California, “California consumer privacy act (ccpa),” https://oag.ca.gov/ privacy/ccpa, 2018
2018
-
[15]
Europe, “Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (general data protection regulation),” https://eur-lex.eur...
2016
-
[16]
Abstract representations of associated emotions in the human brain,
J. Kim, J. Schultz, T. Rohe, C. Wallraven, S.-W. Lee, and H. H. B ¨ulthoff, “Abstract representations of associated emotions in the human brain,” in Journal of Neuroscience , 2015, pp. 5655–5663
2015
-
[17]
Transformer feed-forward layers are key-value memories,
M. Geva, R. Schuster, J. Berant, and O. Levy, “Transformer feed-forward layers are key-value memories,” in Association for Computational Lin- guistics (ACL), 2021, pp. 5484–5495
2021
-
[18]
Locating and editing factual associations in gpt,
K. Meng, D. Bau, A. Andonian, and Y . Belinkov, “Locating and editing factual associations in gpt,” in Advances in Neural Information Processing Systems (NeurIPS) , vol. 35, 2022, pp. 17 359–17 372
2022
-
[19]
Mass-editing memory in a transformer,
K. Meng, A. S. Sharma, A. J. Andonian, Y . Belinkov, and D. Bau, “Mass-editing memory in a transformer,” in International Conference on Learning Representations (ICLR) , 2023
2023
-
[20]
Ma- chine unlearning of pre-trained large language models,
J. Yao, E. Chien, M. Du, X. Niu, T. Wang, Z. Cheng, and X. Yue, “Ma- chine unlearning of pre-trained large language models,” in Association for Computational Linguistics (ACL) , 2024, pp. 8403–8419
2024
-
[21]
Negative preference optimization: From catastrophic collapse to effective unlearning,
R. Zhang, L. Lin, Y . Bai, and S. Mei, “Negative preference optimization: From catastrophic collapse to effective unlearning,” in Conference on Language Modeling (COLM) , 2024
2024
-
[22]
Eternal sunshine of the spotless net: Selective forgetting in deep networks,
A. Golatkar, A. Achille, and S. Soatto, “Eternal sunshine of the spotless net: Selective forgetting in deep networks,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020, pp. 9301– 9309
2020
-
[23]
Knowledge unlearning for mitigating privacy risks in language models,
J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, “Knowledge unlearning for mitigating privacy risks in language models,” in Association for Computational Linguistics (ACL) , pp. 14 389–14 408
-
[24]
TOFU: A task of fictitious unlearning for LLMs,
P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter, “TOFU: A task of fictitious unlearning for LLMs,” in Conference on Language Modeling (COLM) , 2024
2024
-
[25]
Unlearning bias in language models by partitioning gradients,
C. Yu, S. Jeoung, A. Kasi, P. Yu, and H. Ji, “Unlearning bias in language models by partitioning gradients,” in Association for Computational Linguistics (ACL), 2023, pp. 6032–6048
2023
-
[26]
Machine unlearning in large language models,
S. K. Gundavarapu, S. Agarwal, A. Arora, and C. T. Jagadeeshaiah, “Machine unlearning in large language models,” in arXiv preprint arXiv:2405.15152, 2024
2024 arXiv
-
[27]
Motion influence map for unusual human activity detection and localization in crowded scenes,
D.-G. Lee, H.-I. Suk, S.-K. Park, and S.-W. Lee, “Motion influence map for unusual human activity detection and localization in crowded scenes,” in IEEE Transactions on Circuits and Systems for Video Technology (TCSV), 2015, pp. 1612–1623
2015
-
[28]
Deep reinforcement learning in continuous action spaces: a case study in the game of simulated curling,
K. Lee, S.-A. Kim, J. Choi, and S.-W. Lee, “Deep reinforcement learning in continuous action spaces: a case study in the game of simulated curling,” in International Conference on Machine Learning (ICML) , 2018, pp. 2937–2946
2018
-
[29]
Continuous eeg decoding of pilots’ mental states using multiple feature block-based convolutional neural network,
D.-H. Lee, J.-H. Jeong, K. Kim, B.-W. Yu, and S.-W. Lee, “Continuous eeg decoding of pilots’ mental states using multiple feature block-based convolutional neural network,” in IEEE Access , 2020, pp. 121 929– 121 941
2020
-
[30]
Llama 2: Open foundation and fine-tuned chat models,
H. T. et al., “Llama 2: Open foundation and fine-tuned chat models,” in arXiv preprint arXiv:2307.09288 , 2023
2023 arXiv
-
[31]
Qwen2 technical report,
A. Y . et al., “Qwen2 technical report,” in arXiv preprint arXiv:2407.10671, 2024
2024 arXiv
-
[32]
LoRA: Low-rank adaptation of large language models,
E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations (ICLR) , 2022
2022
-
[33]
Fast model editing at scale,
E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning, “Fast model editing at scale,” in International Conference on Learning Rep- resentations (ICLR), 2022
2022
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.