{"id":"4d189f87-58ea-4e64-afa1-8e9a38ed4729","arxiv_id":"2504.12681","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GRAIL performs multi-domain machine unlearning by using per-parameter gradient magnitudes to freeze retention-critical parameters while unlearning privacy and copyright knowledge.","lead":"This paper introduces GRAIL, a method that removes private or copyrighted knowledge from large language models by freezing parameters that protect overlapping retention knowledge. It reports up to 17 percent stronger knowledge retention than the previous best method on two 7-billion-parameter chat models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The localization assumption in §III.C is unvalidated: top-k gradient parameters are treated as knowledge loci, but the ablations do not rule out random-mask freezing as the actual cause of retention gains.","rationale":"The reader's verdict of CONDITIONAL is appropriate, and I agree with the identified weakest assumption. The localization step is the unique component that separates GRAIL from generic gradient ascent/descent with parameter freezing; if gradient magnitude after random-label substitution does not identify knowledge-bearing parameters, then the OP-UR and OP-RR masks are arbitrary, and the claimed 'precise distinction' of unlearning vs retention scopes is unsupported. The paper's own evidence is insufficient: Fig. 3 shows only overlap between selected sets, and Table VI lacks a random-mask control. Thus a critical experiment is needed before accepting the central claim. I additionally note the Algorithm 1 sign inconsistency, but treat it as secondary because it may be a documentation typo; the random-mask control would determine whether the core scientific claim is valid. Therefore no verdict change is needed: the paper should remain conditional pending the random-mask test.","tokens_in":12284,"tokens_out":8081,"duration_ms":87745,"concrete_test":"Run GRAIL in the Table I LLaMA-2-7B-Chat setting with identical hyperparameters (kOP-UR=10%, kOP-RR=20%), but replace the gradient-selected indices in every layer with random parameter indices of the same layer-wise counts, averaged over several seeds. If random-mask GRAIL matches gradient-mask GRAIL on Privacy/Copyright US and RS within a few points, the localization step is not doing the claimed work. As a complementary check, implement Algorithm 1 with both the printed sign and the corrected gradient-ascent-on-loss sign on a small model; if the printed sign fails to reduce Unlearning Success, the algorithm as written cannot produce the reported table.","verdict_should_be":"UNCHANGED","load_bearing_attack":"GRAIL's central claim—that gradient information lets it 'precisely distinguish the unlearning scope from the retention scope'—rests entirely on the localization step in §III.C-D. Equations (3)-(4) define the average gradient magnitude after three random-label substitutions and select the top-k% of parameters per layer as 'critical for parametric knowledge of Dx.' These sets directly determine the OP-UR and OP-RR masks in Stage 2, which in turn decide which parameters are updated during unlearning. No independent evidence is provided that this gradient-to-knowledge mapping is valid, especially for overlapping multi-domain knowledge. Fig. 3's Jaccard heatmap only shows overlap among the selected top-k sets; it does not establish that the selected parameters are the causal loci of the learned knowledge. Table VI's ablations show that removing the masks changes retention, but since the masks are always the gradient-selected sets, the result is compatible with a weaker explanation: freezing roughly 10-20% of parameters (any parameters) reduces update magnitude and thereby preserves retention. If random subsets of the same size reproduce the US/RS trade-off, the claimed 'parameter-wise localization' is not the source of the reported gains, and the multi-domain separation claim collapses to a freezing artifact. The paper provides no random-mask control, no error bars, and no code release to rule this out. Separately, Algorithm 1's unlearning update is printed as θ + η∇θ log P(y|x), which is gradient ascent on the likelihood of the target—the opposite of unlearning; this internal inconsistency needs a code-level check, but the localization validation is the more fundamental issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GRAIL proposes a multi-domain machine unlearning framework for LLMs, targeting privacy and copyright knowledge. The method first localizes parameters associated with unlearning and retention scopes by averaging gradient magnitudes after random-label substitution, then freezes parameters that overlap between unlearning and retention (OP-UR) or between retention across domains (OP-RR), and finally applies gradient updates on the remaining parameters. Experiments on KnowUnDo with LLaMA-2-7B-Chat and Qwen-1.5-7B-Chat compare GRAIL against gradient ascent, random-label fine-tuning, adversarial unlearning, and gradient-ascent-plus-descent variants, reporting improved retention success with comparable unlearning success. The paper claims up to 17% stronger knowledge retention than the previous state-of-the-art method.","tokens_in":12598,"tokens_out":3910,"duration_ms":42869,"significance":"If the reported results hold, GRAIL would be a useful contribution to multi-domain unlearning, addressing a realistic scenario where privacy and copyright knowledge overlap. The paper identifies a concrete problem, proposes a parameter-wise localization mechanism, and provides experiments on two 7B-scale models with several baselines and ablations. The harmonic success metric is a reasonable way to quantify the trade-off. However, the central claims rest on two load-bearing points that are not adequately established: the correctness of the gradient update signs in Algorithm 1 and the validity of gradient-magnitude-based localization as a causal account of where knowledge resides. The lack of random-mask controls, error bars, and code release further limits confidence. These issues are fixable but require substantive revision.","major_comments":[{"comment":"The update rules for unlearning and retention appear to have reversed signs. For (x,y) in D_U, the algorithm sets theta <- theta + eta * grad_theta log P(y|x), which is gradient ascent on the log-likelihood and therefore increases the probability of the target output, reinforcing the very knowledge that should be forgotten. For retention, theta <- theta - eta * grad_theta log P(y|x) decreases the log-likelihood, which would harm retention. In standard gradient-based unlearning, one maximizes the loss (equivalently minimizes log-likelihood) for unlearn data and minimizes the loss for retain data. As written, Algorithm 1 cannot implement the method described in Section III.A, where the authors state that they 'combine gradient ascent and gradient descent' to balance unlearning and retention. Please correct the signs, e.g., theta - eta * grad_theta log P(y|x) for unlearning and theta + eta * grad_theta log P(y|x) for retention, or explicitly define a surrogate loss whose gradient has the opposite sign.","section":"Algorithm 1, Stage 3"},{"comment":"The localization assumption is not validated against a random-mask control. Equations (3)-(4) select top-k% parameters by average gradient magnitude after random-label substitution, and these selections directly determine the OP-UR and OP-RR masks. The ablations in Table VI compare the full method with variants that remove one or both masks, but they never replace the gradient-selected masks with random parameter subsets of the same size. Without such a control, the observed retention improvements could be explained by a simpler mechanism: freezing roughly 10-20% of parameters reduces the magnitude and scope of updates, thereby preserving retention regardless of which parameters are frozen. This is a load-bearing issue because the paper's central claim is that gradient information 'precisely distinguish[es] the unlearning scope from the retention scope.' Please add a random-mask control and, if the gradient-selected masks outperform random masks, report the comparison.","section":"Section III.C-D and Table VI"},{"comment":"The thresholds kOP-UR=10% and kOP-RR=20% are selected using the same KnowUnDo benchmark on which the final results are reported, and Fig. 4 shows sensitivity trends without error bars or multiple seeds. Since these thresholds materially affect the OP-UR and OP-RR masks, the absence of a held-out validation split or variance estimates makes it difficult to assess whether the reported gains are robust or reflect tuning on the evaluation set. Please provide multiple-run statistics and, if possible, a validation-based selection procedure for kOP-UR and kOP-RR.","section":"Section IV.A, Fig. 4"},{"comment":"No error bars or confidence intervals are reported for the main results. The central quantitative claim (e.g., privacy retention improving from 72.79 to 85.34 in Table I, and the 17% relative improvement) rests on single runs. Given that unlearning methods are sensitive to initialization and optimization randomness, the absence of repeated runs or statistical significance testing weakens the conclusions. Please report means and standard deviations over at least three runs, or justify why the observed differences are beyond expected noise.","section":"Section V.A and Tables I-III"}],"minor_comments":[{"comment":"There is a typo in the stage title: 'Adaptive Parmeter-wise Localization' should be 'Adaptive Parameter-wise Localization.'","section":"Algorithm 1, Stage 2"},{"comment":"The abstract states 'up to 17% stronger knowledge retention success,' which is a relative improvement (85.34/72.79 = 1.172). Please state the percentage-point difference as well to avoid ambiguity.","section":"Abstract and Section V.A"},{"comment":"The notation Dx is used ambiguously: in Eq. (3) Dx is described as an element of D, but later T(Dx) is defined for each dataset. Please clarify the indexing.","section":"Section III.D"},{"comment":"Reference [16] appears to be a neuroscience paper (Kim et al., 2015) and is cited in a list of legal principles [13]-[16]; this seems unrelated. Please verify and replace with an appropriate legal or policy reference.","section":"References"},{"comment":"Several entries use notation like '>1010' and '109'; these should be typeset as superscripts (e.g., >10^10, 10^9) to avoid confusion.","section":"Tables I and II"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and addresses a timely problem. The main concern is the sign error in Algorithm 1, which is central to the method description, and the absence of a random-mask control, which is necessary to support the localization claim. The authors should also release code or provide detailed reproducibility information. If the sign error is a typo and the random-mask control supports the localization claim, the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: GRAIL is a plausible extension of Memflex-style gradient localization from layer-wise to parameter-wise, explicitly for multi-domain privacy/copyright unlearning. Worth reading if you work on unlearning. But two things need attention before you trust the numbers: Algorithm 1 as printed has the unlearning sign backwards, and the paper never rules out that the retention gains come from freezing arbitrary parameters rather than from the gradient-selected masks.\n\nWhat's good: The multi-domain framing is genuinely more realistic than single-domain benchmarks, and the overlap analysis (Jaccard heatmap) is a nice diagnostic. The parameter-wise freezing scheme—OP-UR and OP-RR—is a clear mechanism that goes beyond Memflex's layer-level masks. The reported gains are consistent across LLaMA-2-7B and Qwen-1.5-7B, and the ablation shows the masks matter (removing both drops copyright retention from 93.87 to 78.08). That's a real effect.\n\nWhere it gets soft: First, Algorithm 1 shows the unlearn update as θ + η∇θ log P(y|x). That increases the likelihood of the target—training, not forgetting. The text says 'combining gradient ascent and gradient descent,' so the plus sign is probably a typo for minus (or they mean ascent on the loss). But as printed it's inconsistent with the method name and with the results, and it makes the algorithm irreproducible without the code. Second, the localization step assumes the top-k parameters by gradient magnitude after random-label substitution are the causal 'knowledge loci.' The evidence is indirect: the heatmap shows overlap, not causality. The ablations compare their masks to no masks, not to random masks of the same size. If freezing any 10–20% of parameters preserves retention, the whole parametric-localization story is weakened. Also missing: error bars (tables are single-run), any held-out validation split for choosing kOP-UR and kOP-RR (they tune on the same benchmark they report), and a code/data release.\n\nNet: The core idea is coherent and the empirical claim is plausible, but the paper currently reads as an extended abstract with strong results and insufficient verification. It deserves a serious referee, because the method is a real candidate in a practical area. For your own work, I wouldn't cite the numbers until the sign is clarified and a random-mask control appears. It's a good reading-group paper to discuss what counts as evidence in unlearning.","headline":"Plausible multi-domain unlearning extension with a backward sign in Algorithm 1 and a missing random-mask control; the results may be real but aren't yet checkable.","tokens_in":13120,"tokens_out":2704,"would_cite":false,"duration_ms":30010,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that GRAIL can erase privacy and copyright knowledge from a 7-billion-parameter chat model in one pass while retaining other knowledge better than existing methods.","keywords":["large language models","machine unlearning","privacy","copyright","multi-domain unlearning","gradient-based localization","parameter-wise freezing","right to be forgotten"],"falsifier":"A decisive test would take a fact whose storage location is already known from single-parameter editing—changing one parameter flips the answer, changing others does not—and check whether GRAIL's gradient masks include that causal parameter and exclude the irrelevant ones; if they do not, the localization premise is false.","tokens_in":12117,"feed_emoji":"🧹","tokens_out":16184,"duration_ms":141506,"temperature":0.7,"pith_summary":"The paper tries to solve a practical problem: when an LLM must forget private or copyrighted text, it should not damage other knowledge in the same model, and this gets harder when privacy and copyright knowledge live in overlapping parameters. GRAIL's proposal is to compute, for each dataset, per-parameter gradients after substituting the target answers with random labels, then freeze the top-k parameters that are important for both unlearning and retention or for retention across domains, and update only the rest with gradient ascent for forgotten data and gradient descent for retained data. On two 7-billion-parameter chat models and a privacy/copyright benchmark, the paper reports unlearning success on par with prior methods while retention success improves by up to about 17 percent relative, with higher harmonic success than the layer-wise baseline. If correct, this gives model operators a cheap alternative to retraining for complying with right-to-be-forgotten and copyright requests.","feed_headline":"Gradient masks erase private and copyrighted data in LLMs","feed_subtitle":"A new method freezes only retention-critical weights, matching erasure quality and boosting kept-knowledge scores by up to 17 percent.","key_machinery":"The load-bearing object is a per-layer, per-dataset parameter mask built from gradient magnitudes. For dataset $D_x$, each target answer is replaced by a random label, backpropagation is run three times, and the averaged absolute gradient per parameter, $$\\|\\mathbf{g}^\\ell_{x,j}\\| = \\frac{1}{n}\\sum_{i=1}^n |g^\\ell_{x,i}[j]|,$$ ranks parameters; the top $k\\%$ in layer $\\ell$ form the set $T^\\ell(D_x)$. Two masks are then derived: the unlearning-retention overlap (OP-UR), parameters that fall in the top set for both unlearning and retention knowledge across the two domains, and the cross-domain retention overlap (OP-RR), parameters in the top retention sets of both privacy and copyright. These parameters are frozen; the remaining parameters are updated by gradient ascent on unlearning data and gradient descent on retention data, with $k_{\\text{OP-UR}}=10\\%$ and $k_{\\text{OP-RR}}=20\\%$ in the reported experiments.","core_discovery":"On the paper's own terms, the discovery is that unlearning and retention scopes in multi-domain LLM knowledge can be separated at the level of individual parameters rather than whole layers, and that doing so makes simultaneous privacy and copyright unlearning safe. The evidence is a three-stage procedure: random-label substitution to obtain stable gradients, adaptive top-k parameter localization per layer for each of the four datasets, and freezing of two overlap masks (unlearning-retention overlap and cross-domain retention overlap) before applying gradient ascent to forget and gradient descent to retain. In the reported experiments on two 7-billion-parameter chat models, GRAIL keeps unlearning success in the 90-99 range while raising privacy retention from 72.79 to 85.34 and copyright retention from 89.00 to 93.87 on the first model; its average harmonic success is 92.10 versus 87.78 for the layer-wise baseline. The paper further shows that sequential or simply combined multi-domain unlearning damages the other domain, and that the top-10% parameter sets for unlearning and retention overlap substantially (up to 61% within a domain, up to 34% across domains), which is the phenomenon the adaptive masks are designed to handle.","pith_inferences":["Inference: the same mask-building recipe could be reused for non-legal forgetting targets — toxicity, outdated facts, named entities — without modifying the algorithm; the paper only reports privacy and copyright.","Inference: because the masks are computed once and then frozen, repeated or continual unlearning requests could be processed incrementally by recomputing masks on the current model; the paper does not test this.","Inference: the reported overlaps among the top-10% parameter sets (up to 61% between unlearning and retention within a domain, up to 34% across domains) mean a clean separation is impossible for heavily entangled facts; GRAIL's gains are likely a trade-off tilted toward retention, and the paper does not measure how much target knowledge survives in the highest-overlap cases.","Inference: the method's success may depend on the low-rank adapter parameterization used in the experiments; on fully fine-tuned or larger models the gradient top-k sets could be different, so the 17% retention gain is established only for this setting."],"forward_implications":["A single unlearning pass can handle privacy and copyright together: GRAIL's simultaneous update avoids the cross-domain retention damage that the paper shows for sequential or fully combined unlearning.","The balance, measured by harmonic success, is higher than the prior layer-wise localization baseline in both domains (e.g., 87.95 vs 82.20 for privacy and 96.25 vs 93.35 for copyright on the main 7B chat model).","The frozen-parameter mask makes forgetting less destructive: retention can improve by about 17% relative to the prior best on privacy, while unlearning success stays above 90, so the usual unlearning-retention trade-off is partially decoupled.","General task performance after erasure remains close to the vanilla model (average 0.42 vs 0.44 on the paper's five benchmarks), suggesting the erasure is localized and does not collapse the model's general abilities.","Because the updates are applied to low-rank adapter parameters on top of a frozen base model, compliance erasure becomes a fine-tuning-scale operation rather than a retraining-scale one."],"supporting_citations":[{"why":"Supplies the benchmark dataset with privacy and copyright unlearn/retain subsets and the layer-wise localization baseline GRAIL is compared against.","marker":"[5]"},{"why":"Provides the gradient-based knowledge localization approach and the unlearning formulation for pre-trained LLMs that GRAIL generalizes to multiple domains.","marker":"[20]"},{"why":"Specifies the 7-billion-parameter chat model used for the main unlearning and retention experiments.","marker":"[30]"},{"why":"Specifies the second 7-billion-parameter chat model used to show that GRAIL's gains are not specific to a single model.","marker":"[31]"},{"why":"Provides the low-rank adaptation scheme used to build the vanilla model and to restrict unlearning updates to adapter layers.","marker":"[32]"},{"why":"Supports the premise that factual knowledge is localized in specific transformer parameters, motivating parameter-wise rather than layer-wise masks.","marker":"[18]"},{"why":"Contributes one of the evaluation metrics for unlearning and retention success, together with [18] and [20].","marker":"[33]"}],"fun_headline_variants":["Parameter-level masks balance LLM forgetting and retention","Gradient-guided masks target private and copyrighted data","Adaptive parameter masks boost LLM knowledge retention","GRAIL separates unlearning from retention at parameter level","Frozen overlap masks improve multi-domain LLM unlearning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the assumption that the parameters with the largest average gradient after random-label substitution are exactly the parameters that store a dataset's knowledge; if that mapping is wrong, GRAIL freezes the wrong parameters and both erasure and retention fail.","fun_headline_variants_meta":{"raw":{"variants":["Parameter-level masks balance LLM forgetting and retention","Gradient-guided masks target private and copyrighted data","Adaptive parameter masks boost LLM knowledge retention","GRAIL separates unlearning from retention at parameter level","Frozen overlap masks improve multi-domain LLM unlearning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000253,"raw_usage":{"total_tokens":1593,"prompt_tokens":1000,"completion_tokens":593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":518}},"tokens_in":616,"tokens_out":593,"duration_ms":6444,"temperature":1.0,"reasoning_tokens":518,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:25:19.892979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would take a fact whose storage location is already known from single-parameter editing—changing one parameter flips the answer, changing others does not—and check whether GRAIL's gradient masks include that causal parameter and exclude the irrelevant ones; if they do not, the localization premise is false.","supporting_citations":[{"cited_title":"To forget or not? towards practical knowledge unlearning for large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark dataset with privacy and copyright unlearn/retain subsets and the layer-wise localization baseline GRAIL is compared against."},{"cited_title":"Ma- chine unlearning of pre-trained large language models,","cited_arxiv_id":null,"evidence_quote":"Provides the gradient-based knowledge localization approach and the unlearning formulation for pre-trained LLMs that GRAIL generalizes to multiple domains."},{"cited_title":"LoRA: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Provides the low-rank adaptation scheme used to build the vanilla model and to restrict unlearning updates to adapter layers."},{"cited_title":"Locating and editing factual associations in gpt,","cited_arxiv_id":null,"evidence_quote":"Supports the premise that factual knowledge is localized in specific transformer parameters, motivating parameter-wise rather than layer-wise masks."},{"cited_title":"Fast model editing at scale,","cited_arxiv_id":null,"evidence_quote":"Contributes one of the evaluation metrics for unlearning and retention success, together with [18] and [20]."}],"review_version":1}