{"id":"90ed9cb9-8ef6-4ca2-a7a5-fd78a865f8dc","arxiv_id":"2505.11953","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors propose SatImp, a product of a saturation weight and an importance weight, and show it improves the unlearn-retain trade-off on TOFU, WMDP, and MUSE.","lead":"This paper studies how to weight different parts of the data when teaching a large language model to forget information. It proposes SatImp, a new weighting scheme that balances forgetting target data with keeping general knowledge, and reports better retention without worse forgetting on several benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SatImp's 'importance' component is not shown to contribute: with β1=5, β2=1, Eq. 12 peaks near high likelihood, and Table 2 lacks a β2=0 control.","rationale":"The reader's conditional verdict is reasonable: the paper is an empirical study with a plausible method, but the stated generality is not fully supported. My stress-test focuses on a different, more specific weakness: whether SatImp actually implements the proposed combination of saturation and importance. The math of Eq. (12) with the chosen hyperparameters shows that the weight function is saturation-dominated and peaks at very high token likelihood, so the 'importance' factor is not playing the role the paper claims in its conceptual narrative. The missing control (β2=0 with the same β1=5) is the decisive experiment: without it, the improvement over WGA in Table 2 is consistent with a pure-saturation reweighting at a different temperature. The paper's own Fig. 5(a) uses β1=β2=1 to illustrate 'middle-loss' emphasis, which further obscures the actual operating point. These issues do not disprove that SatImp is useful, and the code availability and extensive tables are real evidence; but they do mean the central claim 'combines the advantages of both saturation and importance' is not yet established. Since this reinforces rather than overturns the reader's conditional verdict, I leave the verdict unchanged.","tokens_in":64984,"tokens_out":5841,"duration_ms":63835,"concrete_test":"Reproduce Table 2 on TOFU Forget-1/5/10 (Phi-1.5 and LLaMA-2-7B, retain-regularized GD) with SatImp's weight set to β1=5, β2=0 (pure WGA-style saturation) and all other hyperparameters identical, running at least 3 seeds for both β2=0 and β2=1. If β2=0 matches SatImp's ES Retain/ES Unlearn within run-to-run noise, the importance term is inert and the combination claim should be withdrawn; if β2=1 consistently improves the forget-retain frontier, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SatImp combines the advantages of saturation and importance. The evidence does not yet support that the importance component contributes. Eq. (12), with the hyperparameters used in Table 2 (β1=5, β2=1), is maximized at p = β1/(β1+β2) = 5/6 and is increasing on [0,5/6], so it behaves as a saturation-style weighting (cf. WGA, Eq. 5), not as the low-likelihood-emphasizing importance weighting introduced in §4.1 and Eq. (11). The motivating Figure 5(a) illustrates 'middle-loss' emphasis with β1=β2=1, not the evaluated β1=5, β2=1 configuration. The only direct evidence that multiplying importance and saturation helps, Table 1, uses manually labeled importance (Eq. 8), not the simplified importance used in SatImp. Table 2 compares SatImp against WGA with β=2 but never against the pure-saturation counterpart with the same β1=5 and β2=0. Without that control, the reported gains cannot be attributed to the 'importance' factor; they may simply reflect a differently tuned saturation/WGA variant. This is compounded by the absence of error bars or multiple seeds, leaving the stated superiority on single-run comparisons.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies loss reweighting in LLM unlearning and proposes a taxonomy of two goals: saturation (emphasizing tokens that are not yet sufficiently unlearned) and importance (emphasizing critical tokens). It introduces a manually labeled importance weighting on TOFU, a saturation weight in Eq. (9), simplified versions SimSat and SimImp in Eq. (11), and proposes SatImp in Eq. (12) as a product of saturation and importance weights. The main empirical claim is that SatImp outperforms advanced baselines such as NPO, SimNPO, and WGA on TOFU with retain regularization (Table 2), with additional experiments on WMDP and MUSE. The paper also studies how smoothness, granularity, and hard sampling affect unlearning performance. The central claim is plausible but currently rests on single-run comparisons, a specific retain-regularized setting, and a missing control that isolates the importance component.","tokens_in":65327,"tokens_out":5416,"duration_ms":56538,"significance":"If the central claim were robust, the paper would make a useful conceptual contribution by organizing existing reweighting methods into saturation and importance categories and by offering SatImp as a simple, tunable baseline. The manuscript has concrete strengths: the code is released, the experimental setup is documented in detail, the manual token-level annotation effort on TOFU is considerable, and the appendix covers TOFU, WMDP, and MUSE with multiple metrics. The paper also includes a self-aware limitations section (Appendix A) that concedes parameter sensitivity. However, the significance is currently limited by evidence quality: the favorable TOFU results require retain regularization, there are no repeated-seed statistics, and the importance component of SatImp is not isolated by an ablation. One stress-test concern from the accompanying note does not land on inspection: in the forget-only WMDP setting (Table 3), SatImp matches or improves on RMU's unlearn accuracy while retaining comparable MMLU, so the WMDP evidence is not the weak point.","major_comments":[{"comment":"The reported SatImp configuration does not implement the stated importance behavior. With the hyperparameters used in Table 2, β1=5 and β2=1, the weight p^5(1-p) is increasing on [0, 5/6] and maximized at p=5/6, so it assigns the largest weights to high-likelihood tokens. The importance motivation in Eq. (11) is precisely the opposite: it assigns larger weights to low-likelihood tokens. The paper's own Figure 5(a), which illustrates 'middle-loss' emphasis, uses β1=β2=1 and not the evaluated configuration. Table 2 also lacks a β2=0 control with the same β1=5, so the reported gains over WGA cannot be attributed to the importance factor; they may simply reflect a differently tuned saturation-style weighting. Please add an ablation over (β1,β2), including at least β2=0 with matched β1, and state explicitly which configurations are selected and whether the importance term contributes.","section":"Section 5, Eq. (12), Table 2"},{"comment":"The claim of SatImp's superiority is scoped in a way that is not stated in the abstract or introduction. In the forget-only TOFU setting (Eq. (1)), Table 4 shows that SatImp collapses on 5% and 10% forget sets for both Phi-1.5 and LLaMA-2-7B, with ES Retain, ES Unlearn, and MU all at zero, and LLaMA-2-7B at 5% has ES Retain 0.0080 and ES Unlearn 0.0057. Appendix A itself concedes that SatImp remains parameter-sensitive. The favorable claims in Section 1 and in the analysis of Table 2 therefore hold only with retain regularization (Eq. (2)) and the TOFU-specific λ values. Please either explicitly scope the claim to the gradient-difference setting, or provide a calibration mechanism, such as the early-stopping procedure discussed in Appendix C.2, and show that it restores acceptable behavior in forget-only settings.","section":"Appendix Table 4, Appendix E, Section 1"},{"comment":"All experimental results appear to be single runs: there are no error bars, no repeated seeds, and no significance tests. This matters because the reported margins in Table 2 are small in several cases, for example the ES Retain differences between SatImp and WGA on Phi-1.5 5% (0.5410 vs. 0.5219) and on Phi-1.5 10% (0.4706 vs. 0.4655). Without variance information, the statement in Section 1 that 'the results clearly demonstrate the superiority of our SatImp' is not supported. Please report mean and standard deviation over at least three seeds for the main TOFU comparisons, and describe the checkpoint selection rule used for Tables 2 and 4.","section":"Tables 2-6 and Figures 2-4"},{"comment":"One of the paper's stated findings is true by construction rather than by empirical observation. The saturation weights in Eq. (9), p/(p+τ), and in Eq. (11), p^β, are monotonically increasing functions of token probability and therefore monotonically decreasing functions of negative log-likelihood. Figure 2(h) and the accompanying text report that saturation assigns smaller weights to lower-likelihood tokens, but this is a property of the definition, not an empirical discovery about reweighting behavior. The authors should rephrase this as a design property and provide evidence that the saturation mechanism itself, rather than the monotone weighting, is what improves unlearning performance.","section":"Section 4.1, Eq. (9), Eq. (11), Figure 2(h)"},{"comment":"The importance-based reweighting relies on manual binary labels that were produced by four team members with a 'half agreement' threshold, but no inter-annotator agreement metric (e.g., Cohen's kappa or Fleiss' kappa) is reported, and no label-quality statistics are provided. The subsequent simplification to (1-p)^β in Eq. (11) assumes that the correlation between manually labeled key tokens and low likelihood, as illustrated in Figure 2(g), is stable across domains and models. Given that the importance component is one of the two pillars of SatImp, the authors should either report label reliability and validate the proxy on at least one additional benchmark, or explicitly frame SimImp as a heuristic rather than a validated importance measure.","section":"Section 3.1 and Section 4.1"}],"minor_comments":[{"comment":"The heading contains a typo: 'Comparsion' should be 'Comparison'.","section":"Section 4.1 heading"},{"comment":"There are several typos in the setup text: 'learning rete', 'differnt', and 'tow hyper-parameters' should be corrected to 'learning rate', 'different', and 'two hyper-parameters'.","section":"Appendix B.2 and B.3"},{"comment":"The caption for Figure 2 refers to subfigures (e)-(h) as 'Imp. & Sat. Detail' and 'SimNPO & NPO Detail', but the labels inside the subfigures are not consistently defined; adding explicit panel labels would improve readability.","section":"Figure 2 caption"},{"comment":"The definition of ES in Eq. (13) uses the notation f(y_<k|x; θ) = y_>k, but the subscript on the right-hand side is visually ambiguous and could be confused with the metric under discussion; please use an explicit suffix notation such as y_{>k}.","section":"Appendix B.1.1, Eq. (13)"},{"comment":"In the Phi-1.5 rows of Table 4, several entries are exactly 0.0000 across ES Retain, ES Unlearn, and MU for the 5% and 10% settings; a footnote explaining that these are collapsed runs, rather than missing or zero-due-to-rounding values, would help the reader interpret the failure mode.","section":"Appendix Table 4"},{"comment":"The three-stage description (unlearning, stabilization, collapse) is interesting but is presented as purely descriptive; if it is meant to guide early stopping, the paper should provide a concrete rule for detecting the stabilization stage at inference time.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the LLM unlearning community, and the conceptual taxonomy plus the released code are useful contributions. The main risk is evidentiary: the central superiority claim depends on single runs, on the retain-regularized setting, and on a missing β2=0 ablation. I would not reject the manuscript, but these issues need to be addressed before publication. I also note that the appendix is unusually long relative to the main text; condensing the presentation and moving the key ablations into the main paper would strengthen the narrative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper earns a conditional accept if you see it as an empirical mapping of the reweighting design space, but its headline claim -- that the importance component is what makes SatImp work -- is not actually supported by the evidence as configured.\n\nWhat is good: the categorization of reweighting into saturation vs. importance is conceptually useful, and the authors do a lot of legwork: they design simple proxies for each (SimSat, SimImp), systematically vary smoothness and granularity, and compare against NPO, WGA, and others on TOFU, WMDP, and MUSE. They also ship code and report many ablation tables. That is real, reproducible work, and the observation that token-wise soft reweighting beats hard sampling and coarse granularity is a solid empirical contribution for the community.\n\nThe soft spots are real and central. The stress-test note is correct: with the hyperparameters used in Table 2 (beta1=5, beta2=1), Eq. (12) is maximized at p = 5/6 and is increasing on [0,5/6], so it behaves like a saturation-style weight, not the low-likelihood-emphasizing importance weight introduced earlier. The motivating Figure 5(a) uses beta1=beta2=1, not the evaluated 5/1 configuration. Table 2 compares against WGA with beta=2 but never against the pure-saturation counterpart with the same beta1=5, beta2=0. Without that control, the reported gains could just come from a differently tuned saturation/WGA variant. This is compounded by no error bars or multiple seeds, so the single-run comparisons are fragile.\n\nSecond, Appendix Table 4 shows that without retain regularization, SatImp collapses on TOFU 5% and 10% settings (ES Retain and MU drop to zero) for both models. The main text reports this as a known limitation of token-wise methods, and the authors are honest about it, but the abstract's 'extensive datasets validate' overstates the scope. Third, some 'findings' are true by construction: the saturation weight in Eq. (9) is a decreasing function of negative log-likelihood, so observing that saturation down-weights low-likelihood tokens is not a discovery.\n\nStill, the paper is useful. The ablation around weight distribution shapes is informative, the method is simple and reproducible, and the failure modes are documented. A serious referee should push for the missing beta2=0 control, error bars, and a more careful phrasing of what is new. The paper deserves peer review rather than desk rejection -- it would benefit from revision, and the conceptual frame could be a useful baseline for future work.","headline":"A thorough empirical study of reweighting for LLM unlearning whose SatImp method is likely just a re-tuned saturation baseline; the importance component's contribution is not demonstrated.","tokens_in":65810,"tokens_out":1754,"would_cite":false,"duration_ms":19066,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that loss reweighting for LLM unlearning has two complementary goals—saturation and importance—and that a simple product of the two, SatImp, improves the forget-retain trade-off over existing reweighting methods.","keywords":["machine unlearning","large language models","loss reweighting","saturation","importance","gradient ascent","TOFU benchmark","forget-retain trade-off"],"falsifier":"A concrete experiment is to rerun SatImp and WGA under the forget-only objective on TOFU 5% and 10% and compare ES Retain and MU; the paper reports both methods collapse to zero there, so a rerun that finds nonzero retention would show the collapse is not intrinsic to token-level reweighting, while a rerun that reproduces it confirms the regularization-dependence of the central claim.","tokens_in":64809,"feed_emoji":"🧠","tokens_out":8226,"duration_ms":72201,"temperature":0.7,"pith_summary":"Loss reweighting is a standard fix for the over-forgetting caused by gradient-ascent unlearning in LLMs, but the field has not agreed on what the weights should reward. This paper proposes that reweighting pursues two distinct goals: saturation, which up-weights tokens the model still knows and therefore has not yet unlearned, and importance, which up-weights tokens that carry the meaning of a fact. Through controlled experiments on TOFU with Phi-1.5 and LLaMA-2-7B, the authors find that saturation helps retention more, importance helps forgetting more, and combining the two helps both. They turn this into SatImp, a per-token weight of the form $p^{\\beta_1}(1-p)^{\\beta_2}$, and report that it outperforms GA, PO, DPO, NPO, SimNPO, and WGA on the TOFU forget-retain balance, with supporting results on WMDP and MUSE. The paper is careful to note that the favorable results come in the gradient-difference setting with retain regularization; in forget-only settings, SatImp over-forgets at the larger TOFU fractions.","feed_headline":"One reweighting rule lifts LLM unlearning's forget-retain trade-off","feed_subtitle":"SatImp weights each token by a mix of how well the model knows it and how much it matters, improving both forgetting and retention.","key_machinery":"The load-bearing object is the SatImp per-token weight $w_{x,y,k}^{\\mathrm{satimp}}=p(y_k|y_{<k},x;\\theta)^{\\beta_1}(1-p(y_k|y_{<k},x;\\theta))^{\\beta_2}$, where $p(y_k|y_{<k},x;\\theta)$ is the model's probability of the next token during unlearning. With $\\beta_2=0$ it reduces to WGA's saturation weight; with $\\beta_1=0$ it becomes the simplified importance weight; with both positive it concentrates weight on tokens of intermediate likelihood. Because the weight is token-wise, recomputed from the evolving model, and soft rather than a hard sample, it acts as a smooth, adaptive reweighting of the gradient-ascent objective; the ratio between $\\beta_1$ and $\\beta_2$ sets the direction of the weight-loss correlation, and their scale sets the distribution's peakedness, which the paper shows controls the unlearn-retain trade-off.","core_discovery":"The central claim is that the apparent chaos of reweighting objectives for LLM unlearning reduces to two complementary criteria. Saturation-based reweighting—weights proportional to current token probability—targets tokens that have not been sufficiently unlearned; importance-based reweighting—weights proportional to one minus probability—targets low-likelihood tokens that carry key information. The paper shows these two criteria assign opposite weights to the same tokens, behave differently on unlearning versus retention, and can be combined multiplicatively. SatImp is that combination, $w_{x,y,k}=p^{\\beta_1}(1-p)^{\\beta_2}$, which shifts emphasis toward middle-loss tokens and lets the two exponents tune smoothness and the relative pull of saturation versus importance. With $\\beta_1=5, \\beta_2=1$ under the retain-regularized gradient-difference objective, the method reports the best retain-side extraction strength and model utility on TOFU while matching the strongest forget-side numbers, and it transfers to WMDP and MUSE better than or comparably with the baselines.","pith_inferences":["I would expect the saturation/importance dichotomy to transfer to other data-weighting problems, such as noisy-label learning or preference optimization, where 'hard examples' and 'influential examples' are usually treated as the same thing; SatImp suggests they should be separate coordinates of the weight function.","A direct extension the paper does not explore is automating the importance labels: the KTL index, which ranks a key token's likelihood within its sample, could serve as a training signal for a weight predictor, removing the manual annotation bottleneck.","The middle-loss emphasis of SatImp hints at a principled curriculum: anneal $\\beta_1$ and $\\beta_2$ over training so the objective starts saturation-heavy and shifts importance-heavy, which could reduce the reported parameter sensitivity.","The paper's WMDP retain-regularization numbers use TOFU-tuned hyperparameters; a per-benchmark sweep would be the natural stress test of whether SatImp's advantage is a property of the weight shape or of the particular $\\beta$ values chosen."],"forward_implications":["If SatImp's results hold, future unlearning objectives can be designed by specifying two numbers—how strongly to emphasize high-probability residual knowledge and how strongly to emphasize low-probability key tokens—rather than by ad hoc loss functions.","The paper's comparisons imply that soft token-level reweighting is preferable to hard TopK/BottomK sampling and to instance- or batch-level weights for unlearning.","The empirical distinction between saturation and importance gives a diagnostic: plotting weight against token loss reveals which objective a method actually implements, which can be used to audit existing unlearning methods.","Because SatImp's gains are demonstrated under retain-regularized gradient difference, the method's practical use will require a retention set; in forget-only deployments, larger forget fractions over-forget, as the paper's Appendix Table 4 shows."],"supporting_citations":[{"why":"Defines WGA's probability-based reweighting and the token-wise-versus-instance-wise comparison that SatImp builds on.","marker":"(Wang et al., 2025b)"},{"why":"Introduces NPO and its implicit reweighting form, a main baseline and the canonical saturation-like objective.","marker":"(Zhang et al., 2024)"},{"why":"Provides the TOFU benchmark and the forget-set sizes (1%, 5%, 10%) used in the main evaluation.","marker":"(Maini et al., 2024)"},{"why":"Supplies the Extraction Strength metric used to measure unlearning and retention.","marker":"(Wang et al., 2025a)"},{"why":"Introduces SimNPO, a smoothed NPO variant used as a baseline and as evidence for saturation-type behavior.","marker":"(Fan et al., 2024)"},{"why":"Provides the WMDP benchmark and the RMU baseline for the supplementary comparison.","marker":"(Li et al., 2024)"},{"why":"Provides the MUSE benchmark and its six-way evaluation used in the appendix.","marker":"(Shi et al., 2024)"},{"why":"Motivates the saturation weight via the Bayes-optimal membership-knowledge strategy.","marker":"(Sablayrolles et al., 2019)"},{"why":"Supplies LLaMA-2-7B, one of the two base models for the TOFU experiments.","marker":"(Touvron et al., 2023)"}],"fun_headline_variants":["SatImp: Saturation plus importance improves LLM unlearning","Two reweighting goals combine to boost LLM unlearning","Saturation and importance: the key to better unlearning","New reweighting method SatImp lifts forget-retain trade-off","Why loss reweighting works: saturation and importance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The advertised SatImp advantage assumes the retain-regularized gradient-difference objective; the paper's own Appendix Table 4 shows that in forget-only runs at 5% and 10% TOFU, SatImp's retain score and model utility fall to zero.","fun_headline_variants_meta":{"raw":{"variants":["SatImp: Saturation plus importance improves LLM unlearning","Two reweighting goals combine to boost LLM unlearning","Saturation and importance: the key to better unlearning","New reweighting method SatImp lifts forget-retain trade-off","Why loss reweighting works: saturation and importance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1508,"prompt_tokens":1023,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":401}},"tokens_in":639,"tokens_out":485,"duration_ms":4737,"temperature":1.0,"reasoning_tokens":401,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:44:00.816175+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete experiment is to rerun SatImp and WGA under the forget-only objective on TOFU 5% and 10% and compare ES Retain and MU; the paper reports both methods collapse to zero there, so a rerun that finds nonzero retention would show the collapse is not intrinsic to token-level reweighting, while a rerun that reproduces it confirms the regularization-dependence of the central claim.","supporting_citations":[{"cited_title":"C., and Kolter, J","cited_arxiv_id":null,"evidence_quote":"Provides the TOFU benchmark and the forget-set sizes (1%, 5%, 10%) used in the main evaluation."},{"cited_title":"White-box vs black-box: Bayes optimal strategies for membership inference","cited_arxiv_id":null,"evidence_quote":"Motivates the saturation weight via the Bayes-optimal membership-knowledge strategy."}],"review_version":1}