Pith. sign in

REVIEW 3 major objections 4 minor 15 references

The paper argues that relearn fragility in LLM unlearning is a geometric stationarity property—a 'margin cliff' above the retain reference—and that Margin Calibration, a non-saturating margin hinge, crosses the cliff and keeps post-attack r

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:30 UTC pith:TQGFFC7F

load-bearing objection A broad and unusually transparent empirical study of a plausible common failure mode in LLM unlearning; the 97/97 robustness matrix is the real contribution, while the stated formal guarantees are conditional and should be labeled as such. the 3 major comments →

arxiv 2607.27836 v1 pith:TQGFFC7F submitted 2026-07-30 cs.AI

Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration

Xiangyu Yin , Jiaxu Liu , Zhen Chen , Chih-Hong Cheng This is my paper
classification cs.AI
keywords LLM unlearningrelearn robustnessmargin cliffmargin calibrationtoken-saturating lossKKT stationarityattack budgetTOFU benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the well-known fragility of LLM unlearning under relearn attacks has a common geometric cause and a common fix. Across fourteen published unlearning methods, converged models park in a narrow band of per-token answer margins above a retain-only reference—the margin cliff—and this positive cliff gap is what lets a small fine-tuning attack restore forgotten content. The paper proves that any retain-regularized objective that holds a diagnostic log-odds floor produces this cliff, then introduces Margin Calibration (MC), a plug-in polish whose non-saturating margin hinge keeps forget-side pressure alive after tokens are suppressed. A single frozen MC configuration crosses the cliff in 66 of 70 evaluated cells and wins every populated relearn cell in a 97-cell matrix, cutting mean post-attack recovery from 0.41 to 0.18 ROUGE-L at the cost of utility.

Core claim

The paper's central discovery is that the relearn vulnerability of post-hoc LLM unlearning is a stationarity property, not an implementation bug. Defining a per-token answer margin at each answer's maximum-entropy position, the authors observe that 14 methods from three loss families converge with a positive 'cliff gap' above the retain reference in 41 of 42 method-size cells; a KKT account shows this follows whenever the retain coupling holds the diagnostic log-odds of forget content above a floor, which token-saturating losses (whose forget gradient vanishes as a token's probability is suppressed) induce at stationarity. Margin Calibration changes that geometry: a softplus hinge anchored a

What carries the argument

The margin diagnostic m_theta(D_f): the average log-probability gap between the gold token and its strongest competitor at the maximum-entropy answer position, compared against a retain-only reference through the cliff gap Delta = m_theta - m_ref. The token-saturating loss definition captures losses whose per-token forget gradient vanishes as the gold token's probability is suppressed. Margin Calibration (MC) is a one-sided softplus hinge whose gradient scales with the margin gap instead of the token probability, plus a KL probe on a disjoint instruction corpus, and the formal results are the KKT cliff theorem (Theorem 2), the cliff-crossing stationarity theorem (Theorem 4), and the attack-b

Load-bearing premise

The formal guarantee that MC cannot park a model on the positive-cliff side rests on a directional margin coercivity condition: above any positive cliff gap, the forget hinge must have a gradient component of at least G_f(delta) > 0 in some common direction, a region-wide bound the paper verifies only on a single trajectory.

What would settle it

Run MC-polish from many random initializations across the compact polish region and instrument the hinge subgradient; if any region with cliff gap Delta >= delta lacks a common direction u with u^T xi >= G_f(delta) > 0 for all xi in the forget-hinge subdifferential, the coercivity premise of Theorem 4 fails. Concretely, a non-UNDIAL base whose MC-polished checkpoint terminates with Delta > 0 (a strict local minimum of the polish objective at positive cliff) would falsify the formal crossing guarantee; the four UNDIAL cells are the paper's own observed exceptions.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Unlearning defenses that merely augment a saturating forget objective should remain on the cliff side; the paper shows seven NPO variants do not cross, and MC reduces their mean K20-LoRA recovery from 0.41 to 0.21.
  • A method's relearn vulnerability can be predicted from its cliff gap before any attack: the paper measures correlations between Delta and post-attack ROUGE-L at Pearson r = 0.69/0.65/0.45 and Spearman rho = 0.91/0.89/0.78 across model sizes.
  • A sufficiently negative cliff gap survives every bounded-step fine-tuning attacker, since the theorem's lift budget is independent of the starting gap; the MC flat K-sweep up to K=100 (every cell at or below 0.33 ROUGE-L) realizes this.
  • The method transfers without retuning across model sizes, forget tiers, seeds, benchmarks and model families: 97/97 K20-LoRA cells and 46/46 K20-FPFT cells win under one frozen configuration.
  • A deployment variant anchors at the pre-unlearning model instead of a retain-trained reference, matching gains closely and making MC usable when no retain oracle exists.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A cheap engineering extension not pursued in the paper: use the cliff gap as a training-time early-warning metric, flagging any unlearning checkpoint that terminates with Delta > 0 as relearn-vulnerable before release, without running an attack.
  • The paper's attribution of utility cost to shared representation directions suggests a testable combination of MC with representation-space retargeting: if the cliff-crossing pressure is reallocated away from shared structure, the observed retain-side utility drop might shrink without losing robustness.
  • The attack-budget bound could be stress-tested outside the paper's class: an attacker using per-step increments larger than the assumed budget H (e.g., high-rank adapters or adaptive step sizes) should change the bound's constants; comparing MC under such attackers would test whether the robustness is genuinely class-wide.
  • The theory predicts the cliff gap grows with forget-retain prompt overlap epsilon; a synthetic corpus with controlled overlap could test this directly, shifting baseline margins and MC polish difficulty in a predictable way.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper identifies a 'margin cliff' in LLM unlearning: across 14 post-hoc unlearning methods on TOFU, converged models' per-token answer margins on the forget set cluster in a narrow band above the retain-only reference, and this positive cliff gap predicts how easily a small LoRA relearn attack recovers held-out forget content. The authors propose Margin Calibration (MC), a plug-in LoRA polish adding a non-saturating margin hinge anchored at the reference's per-token margin plus a KL probe, and show empirically that it crosses the cliff and improves post-attack robustness in 97/97 populated cross-axis cells under a single frozen configuration, at the cost of reduced retain-side utility. The paper also states theorems that connect the cliff to a retain-coupling diagnostic floor (Theorem 2), claim a conditional cliff-crossing guarantee for MC (Theorem 4), and give an attack-budget upper bound on post-attack margin lift (Theorem 5).

Significance. If the empirical findings hold, this is a valuable and unusually broad result: it identifies a common failure mode across gradient-, preference-, and distillation-based unlearning methods, and offers a simple, configuration-frozen plug-in that dramatically improves relearn robustness on a 97-cell stress matrix spanning multiple models, benchmarks, seeds, tiers, and attackers. The paper is exemplary in empirical hygiene: consistent-evaluator reruns, multi-seed CIs, instrumented theoretical constants, a deployment variant without a retain-trained reference, and an unusually honest Limitations section. The main weakness is that the theoretical core — especially Theorem 4 — rests on conditions that the paper itself labels as proof devices rather than observables, so the formal guarantees are weaker than the empirical evidence. The empirical contribution is strong enough to be significant on its own, but the theoretical claims need to be presented more carefully.

major comments (3)
  1. [§3.5, Theorem 4; App. B.2] The directional margin coercivity condition is load-bearing for the conclusion that MC's stationary set lies on the cliff-crossing side, but it is assumed, not verified. The paper admits it is 'a proof device rather than an observable' and supports it only with a single CRNPO trajectory's hinge-gradient norm (Fig. 3d). This does not establish the region-wide uniform lower bound G_f(δ), nor does it measure G_max^KL and G_nat needed to check λ_KL G_max^KL + G_nat < G_f(δ). The theorem is therefore conditional on an unverified landscape hypothesis; the statement 'Theorem 4 rules out cliff-side stationarity' overstates what is proved. The authors should either provide a region-wide check (e.g., subgradient sampling over Θ0) or explicitly reformulate Theorem 4 and the abstract as a conditional guarantee.
  2. [§3.3, Assumption 6; App. A.1; Prop. 3] The cliff theorem (Theorem 2) is an implication from Assumption 6, but for the bounded-gradient family (GradDiff, RMU, PDU) and UNDIAL the floor is assumed directly, with no mechanistic derivation; for token-saturating losses the floor is only motivated by a stationarity heuristic, not proven. Thus the claim 'we prove that this cliff follows whenever the retain coupling holds the diagnostic log-odds above a floor' is technically true but the premise itself is not established for the actual objectives of the 14 methods. The empirical certification on 34/42 cells is valuable, but the remaining cells and the assumed floor mean the theoretical explanation is incomplete. The paper should either extend the proof of the floor to a broader class or clearly label Theorem 2 as a conditional statement whose premise is verified only on a subset of cells.
  3. [§3.5 vs §4.1] Theorem 4 concerns strict local minima of L_polish, but MC is run for a fixed 80-step budget with a single-sample optimizer schedule. The theorem does not guarantee that the finite-horizon trajectory reaches the cliff-crossing side; the empirical 66/70 crossing rate carries that weight. This is not a fatal flaw, but the paper's phrasing 'MC crosses the cliff' and 'its stationary set lies on the cliff-crossing side' conflates a stationary-set result with finite-time behavior. The authors should state explicitly that Theorem 4 does not bound the 80-step trajectory's terminal point, and that the crossing claim is empirical.
minor comments (4)
  1. [Abstract and §3.5] The abstract says 'wins all 14 head-to-head forget aggregates and all populated relearn cells'; Table 2 confirms 97/97 for F.agg and K20-LoRA, but the terminology 'relearn cells' should be defined in the abstract or at first use to avoid implying that MIA and utility cells are included (the latter are 6/84).
  2. [§4.2] The sentence 'The seven NPO-based defense variants do not cross the cliff (K20-LoRA mean 0.41 vs vanilla NPO’s 0.35)' is slightly confusing: the 0.41 refers to the seven variants' baseline mean, not MC. Consider rewording to 'their K20-LoRA mean is 0.41, versus vanilla NPO's 0.35'.
  3. [App. B.2, Fig. 3] The caption of Figure 3 says panel (a) uses 'a symmetric logscale' for a quantity that can be negative. It would be clearer to state the exact transformation (e.g., sign(x) log(1+|x|)) so readers can interpret the axis.
  4. [Thm. 5 and App. B.3] The theorem bounds the frozen-position β-smoothed diagnostic, while the reported ∆ uses the max-entropy position rule re-selected per model. The scope note in the text covers this, but it would help to state in the main text before Eq. (8) that the bound applies to the frozen-position diagnostic, not to the re-selected diagnostic used in the tables.

Circularity Check

0 steps flagged

No significant circularity: the paper's empirical claims are benchmark-based and its theoretical statements are transparently conditional, with no fitted parameter renamed as a prediction.

full rationale

The central claims are evaluated on external benchmarks (TOFU, MUSE-News, Phi-3.5) with metrics (post-attack ROUGE-L, membership AUC, MU) that are not directly optimized by the MC objective; the margin anchor is the retain reference used for evaluation, but this is an explicit calibration design rather than a disguised fit. The theoretical results are conditional in a transparent way: Theorem 2 derives the margin cliff from Assumption 6, a measured log-odds floor that the paper verifies directly on 34/42 cells, so the implication is supported by measurement rather than by assuming the conclusion. Theorem 4's directional coercivity condition is explicitly labeled 'a proof device rather than an observable' and is only probed along a single trajectory; this weakens the formal guarantee but does not make the argument circular, since the theorem's conclusion is not used to establish its premise. The hyperparameters are fixed in advance on a small development set and reused unchanged; this is standard configuration selection, not per-cell fitting, and the reported wins span many held-out axes. There are no load-bearing self-citations and no uniqueness or ansatz results imported from the authors' prior work. The main weaknesses are unverified sufficient conditions and evaluator-mix caveats, which are stated as limitations rather than hidden circularities.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new particles, forces, dimensions, or conserved quantities are introduced. "Margin cliff", "cliff gap", and "Margin Calibration" are defined observables/procedures, not independent entities needing external falsifiable handles. The paper's theory is conditional on several assumptions that are only partially verified; the empirical matrix carries most of the load.

free parameters (4)
  • MC hyperparameters (κ, λ_KL, r, Npol, steps) = κ=5.0, λ_KL=0.05, r=32, Npol=200, 80-step schedule
    Tuned via HP sensitivity on 5 representative 1B-forget10 bases and reused globally; flat around plateau, but the headline panel includes the same bases, so this is tuning on the evaluation set.
  • β temperature for smoothed margin bound (Thm. 5) = 120
    Chosen so cβ=log(V−1)/β < 0.1 at Llama-3 vocabulary; not fitted to robustness outcomes, but it is a user-set constant in the certificate.
  • Canonical attack budget K=20 = 20
    Headline attacker budget; the paper reports a full K=1..100 sweep showing the conclusion is not budget-specific, so this is a presentation choice rather than a fitted parameter.
  • Diagnostic max-entropy position rule = argmax_t H(p(·|x,y<t))
    Defines Eq. (2) and the measured cliff; the proof is position-rule-independent, but the empirical cliff numbers and some hinge-position coupling depend on this choice.
axioms (5)
  • ad hoc to paper Assumption 6: at every strict local minimum of Lf+λLr, average diagnostic log-odds ℓ(θ) ≥ ℓ⋆(ϵ,λ) with ℓ⋆ nondecreasing in overlap and retain weight.
    Introduced for Theorem 2; for bounded-gradient methods (GradDiff family) and UNDIAL it is assumed directly, with only a heuristic stationarity motivation for saturating losses, so the cliff theorem is conditional on this floor.
  • ad hoc to paper Theorem 4 directional margin coercivity: for any θ in Θ0 with Δ(θ)≥δ, some unit direction uθ keeps uθ^T ξ ≥ Gf(δ)>0 for all ξ∈∂Lforget(θ).
    The paper calls this a proof device rather than an observable and verifies only a single-trajectory gradient-norm signature (Fig. 3d), not the region-wide uniform bound.
  • domain assumption Lemma 7: polish trajectory stays in a compact Θ0 containing θ0, θref; KL-probe gradient bounded on Alpaca (disjoint from Df∪Dr).
    Boundedness follows from compactness IF trajectory containment holds; containment is assumed, not proven.
  • domain assumption Native-loss gradient bounded by Gnat on the polish region (App. C).
    Needed in Theorem 4; at init native gradient is small because polish starts near a stationary point, but the uniform bound over Θ0 is unquantified.
  • standard math Smoothed margin functional mβ is C1 with finite Lipschitz constants on compact convex ΘA (Thm. 5).
    Standard smoothness-on-compactness result; the paper constructs mβ specifically to make this true rather than assuming it about the hard margin.

pith-pipeline@v1.3.0-daily-deepseek · 40759 in / 13499 out tokens · 132542 ms · 2026-08-01T00:30:38.617053+00:00 · methodology

0 comments
read the original abstract

Large language model unlearning is consistently fragile under relearn attacks. On TOFU, fine-tuning on twenty forget examples substantially recovers held-out forget-set ROUGE for every method we evaluate, and we trace this fragility to optimization geometry. The per-token answer margin of fourteen post-hoc methods spanning gradient, preference, and distillation families converges into a narrow band above the retain reference in 41 of 42 method--size cells, a regularity we call the margin cliff. We prove that this cliff follows whenever the retain coupling holds the diagnostic log-odds of forget content above a floor, a condition that token-saturating losses induce at stationarity and that we verify directly on 34 of 42 cells. Margin Calibration (\textsc{MC}) is a plug-in polish adding a non-saturating margin hinge anchored at the reference's per-token margin plus a KL probe on a disjoint instruction corpus, restoring forget-side pressure where the native loss saturates. Under a stated gradient-dominance condition, whose on-trajectory gradient signature we measure by instrumenting the polish, its stationary set lies on the cliff-crossing side, yielding an attack-budget upper bound on the relearn margin lift. Across TOFU (three Llama-3 sizes, three forget tiers), MUSE-News on Llama-2-7B-hf, and a Phi-3.5 panel, a single frozen configuration wins all 14 head-to-head forget aggregates and all populated relearn cells (panel-mean post-attack ROUGE-L $0.41$ to $0.18$) and lowers raw membership AUC on 13/14, with reduced retain-side utility as the main cost. A deployment variant matches these gains without a retain-trained reference.

Figures

Figures reproduced from arXiv: 2607.27836 by Chih-Hong Cheng, Jiaxu Liu, Xiangyu Yin, Zhen Chen.

Figure 1
Figure 1. Figure 1: Forget-set margin diagnostic mθ(Df ) (Eq. (2)) for 14 baselines (gray) and MC (blue) at three Llama-3 sizes, ordered by loss family. Red dashed line is mref, shaded band is the cliff gap. Mean TOFU MU annotated. probability is suppressed (Def. 1). The gradient family (Grad￾Diff, RMU, PDU) and the flattened-teacher distillation loss UNDIAL have bounded but non-vanishing forget gradients. For them the satura… view at source ↗
Figure 2
Figure 2. Figure 2: Held-out post-attack ROUGE-L under LoRA-r8 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Cliff trajectory over 200 polish steps on CRNPO [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Tier stability of the margin cliff at Llama-3.2-1B. Forget-set margin diagnostic (Eq. (2)) for fourteen baseline [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Held-out post-attack ROUGE-L at 1B forget10 across LoRA, SoftPrompt, and FPFT attackers, averaged over 5 bases. O Cliff gap predicts relearn recovery (Theorem 5 validation) [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Cliff gap predicts relearn-recovery across model sizes (the empirical bridge motivating Theorem 5). Each point is [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Per-method movement from baseline (circles) to [PITH_FULL_IMAGE:figures/full_fig_p025_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 10 linked inside Pith

  1. [1]

    (2); reference margins−3.64,−3.59, −4.01,−4.17,−4.93percolumn).Negativecrossesthecliff; theonlypositivecellsareUNDIAL(4ofits5panels),giving the66/70count of §3.5

    Base Family F.agg ES EM MIA.agg MU KS-ppriv.leak K20-LoRA K20-FPFT GradDiffGradDiff 0.357→0.0180.122→0.0020.737→0.0500.791→0.2280.470→0.109 0.0000 33.220 0.429→0.1580.408→0.088 RMUGradDiff 0.279→0.0020.062→0.0000.556→0.0050.394→0.466 0.571→0.000 0.0021 -2.278 0.471→0.0290.398→0.031 PDUGradDiff 0.024→0.0090.033→0.0000.057→0.0230.472→0.2260.000→0.1270.0000 ...

    arXiv 2010

  2. [3]

    The retain-hinge versus KL-probe ablation (§3.4) is the two-sided retain-hinge row of Table 14, and per-base detail is in App. K. I Per-cell details for T2 Each sub-table below lists the per-method base→MCnum- bers that the corresponding row of Table 2 compresses to a single win-rate cell. The MUSE-News row uses the 13-method panel (CRNPO omitted) on Llam...

    1910

  3. [4]

    Fan, C.; Liu, J.; Lin, L.; Jia, J.; Zhang, R.; Mei, S.; and Liu,S.2024

    Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Per- spective and Beyond.arXiv preprint arXiv:2502.05374. Fan, C.; Liu, J.; Lin, L.; Jia, J.; Zhang, R.; Mei, S.; and Liu,S.2024. SimplicityPrevails:RethinkingNegativePref- erence Optimization for LLM Unlearning.arXiv preprint arXiv:2410.07163. Hans, A.; Wen, Y.; Jain, ...

    Pith/arXiv arXiv 2024

  4. [5]

    arXiv preprint arXiv:2406.10209

    Be like a Goldfish, Don’t Memorize! Mitigating Memorization in Generative LLMs. arXiv preprint arXiv:2406.10209. Hu, S.; Fu, Y.; Wu, Z. S.; and Smith, V

    Pith/arXiv arXiv

  5. [6]

    Lester, B.; Al-Rfou, R.; and Constant, N

    Jogging the MemoryofUnlearnedModelsThroughTargetedRelearning Attacks.arXiv preprint arXiv:2406.13356. Lester, B.; Al-Rfou, R.; and Constant, N

    Pith/arXiv arXiv

  6. [8]

    Maini, P.; Feng, Z.; Schwarzschild, A.; Lipton, Z

    Eight Methods to Evaluate Robust Un- learning in LLMs.arXiv preprint arXiv:2402.16835. Maini, P.; Feng, Z.; Schwarzschild, A.; Lipton, Z. C.; and Kolter, J. Z

    Pith/arXiv arXiv

  7. [9]

    arXiv preprint arXiv:2407.15549

    Latent Adversarial Training Im- provesRobustnesstoPersistentHarmfulBehaviorsinLLMs. arXiv preprint arXiv:2407.15549. Shi, W.; Ajith, A.; Xia, M.; Huang, Y.; Liu, D.; Blevins, T.; Chen, D.; and Zettlemoyer, L. 2024a. Detecting Pretrain- ing Data from Large Language Models. InInternational Conference on Learning Representations (ICLR). Shi, W.; Lee, J.; Hua...

    Pith/arXiv arXiv

  8. [10]

    Tamirisa, R.; Bharathi, B.; Phan, L.; Zhou, A.; Gatti, A.; Suresh, T.; Lin, M.; Wang, J.; Wang, R.; Arel, R.; Pichapati, V.;Hendrycks,D.;etal.2024

    Unlearning That Lasts: Utility-Preserving, Robust, and Al- most Irreversible Forgetting in LLMs.arXiv preprint arXiv:2509.02820. Tamirisa, R.; Bharathi, B.; Phan, L.; Zhou, A.; Gatti, A.; Suresh, T.; Lin, M.; Wang, J.; Wang, R.; Arel, R.; Pichapati, V.;Hendrycks,D.;etal.2024. Tamper-ResistantSafeguards for Open-Weight LLMs.arXiv preprint arXiv:2408.00761....

    Pith/arXiv arXiv 2024

  9. [11]

    https: //github.com/tatsu-lab/stanford_alpaca

    Stan- fordAlpaca:AnInstruction-followingLLaMAmodel. https: //github.com/tatsu-lab/stanford_alpaca. Wang,C.;Zhang,Y.;Jia,J.;Ram,P.;Wei,D.;Yao,Y.;Pal,S.; Baracaldo,N.;andLiu,S.2025. InvarianceMakesLLMUn- learning Resilient Even to Unanticipated Downstream Fine- Tuning. InInternational Conference on Machine Learning (ICML). Zhang,J.;Sun,J.;Yeats,E.;Ouyang,Y....

    Pith/arXiv arXiv 2025

  10. [14]

    The qualitative shape of the two curves is consistent with Theorem 5 (a one-sided upper bound, which caps theMC curve but does not by itself predict the baselines’ rise)

    Baseline ROUGE rises toward the recovery ceiling as Kgrows(withsmallnon-monotonefluctuationsatlargeK), whileMCrisesonlyslowlywithKandstaysatorbelow0.33 in every cell even atK=100, far under the baseline curve at every budget, confirming that the robustness gain is not specific to the canonicalK=20budget. The qualitative shape of the two curves is consiste...

    2060

  11. [26]

    TheLeadershipBlueprint

    Gradient Preference Distillation Baseline MC Figure 6: Cliff gap predicts relearn-recovery across model sizes (the empirical bridge motivating Theorem 5). Each point is one (method, target) cell at the indicated model size, where circles are baselines and triangles areMC, colored by loss family (gradient / preference / distillation). Thex-axis is the clif...

    arXiv 2010

  12. [2021]

    InConfer- enceonEmpiricalMethodsinNaturalLanguageProcessing (EMNLP)

    The Power of Scale for Parameter-Efficient Prompt Tuning. InConfer- enceonEmpiricalMethodsinNaturalLanguageProcessing (EMNLP). Li,N.;Pan,A.;Gopal,A.;Yue,S.;Berrios,D.;Gatti,A.;Li, J.D.;Dombrowski,A.-K.;Goel,S.;Phan,L.;etal.2024.The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning.arXiv preprint arXiv:2403.03218. Lynch, A.; Guo, P.; Ewa...

    Pith/arXiv arXiv 2024

  13. [2023]

    Entesari, T.; Hatami, A.; Khaziev, R.; Ramakrishna, A.; and Fazlyab, M

    Who’s Harry Pot- ter? Approximate Unlearning in LLMs.arXiv preprint arXiv:2310.02238. Entesari, T.; Hatami, A.; Khaziev, R.; Ramakrishna, A.; and Fazlyab, M

    Pith/arXiv arXiv

  14. [2024]

    Dorna, V.; Mekala, A.; Zhao, W.; McCallum, A.; Lipton, Z

    UnDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models.arXiv preprint arXiv:2402.10052. Dorna, V.; Mekala, A.; Zhao, W.; McCallum, A.; Lipton, Z. C.; Kolter, J. Z.; and Maini, P

    Pith/arXiv arXiv

  15. [2025]

    Fan, C.; Jia, J.; Lin, L.; Zhang, R.; Liu, J.; Wang, T.; Zhang, Y.; and Liu, S

    Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models.arXiv preprint arXiv:2506.05314. Fan, C.; Jia, J.; Lin, L.; Zhang, R.; Liu, J.; Wang, T.; Zhang, Y.; and Liu, S

    arXiv