Pith. sign in

REVIEW 4 major objections 6 minor 31 references

Multi-Objective Large Language Model Unlearning

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MOLLM reformulates LLM unlearning as three competing objectives and finds a single descent direction that improves all three at once.

desk verdict The empirical story is plausible, but the theoretical core of DS-MGDA does not survive contact with linear algebra; the paper needs a major revision before it can be trusted. read the letter →

arxiv 2412.20412 v2 pith:4D4CRCUQ submitted 2024-12-29 cs.CL cs.AIcs.LG

Zibin Pan , Shuwen Zhang , Yuesheng Zheng , Chi Li , Yuheng Cheng , Junhua Zhao This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords largelanguagemodelunlearninggradientascentmulti-objectiveoptimizationcatastrophicforgettingexplosioncommondescentdirectioncross-entropylossParetostationarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that gradient-ascent-based LLM unlearning can be made stable and utility-preserving by reformulating it as a multi-objective optimization problem. It introduces an Unlearning Cross-Entropy (UCE) loss that replaces the unbounded inverse cross-entropy, preventing gradient explosion without manual clipping. To handle the conflict between forgetting and retaining knowledge, it proposes DS-MGDA, which projects the three loss gradients into the dual space and averages the dual vectors to get a common descent direction that reduces all three objectives at once. On the PKU-SafeRLHF benchmark with Llama 3-8B, MOLLM reports the lowest harmful rate, toxicity, and obscenity among gradient-ascent baselines while keeping fluency at 1.689.

What carries the argument

The central object is the Dual Space Multiple Gradient Descent Algorithm (DS-MGDA). It takes the gradients of the unlearning loss, KL preservation loss, and retain loss; computes each gradient's projection onto the null space of the other two (so that each dual vector is orthogonal to the other two gradients); and averages the three dual vectors (with negative sign) to obtain the update direction. This direction is claimed to lie in the dual cone where every sub-objective decreases. The UCE loss, defined as $L_{UCE} = -\frac{1}{K}\sum_i\sum_c y_{i,c}\log(1-(1-\epsilon)p_{i,c})$, is bounded below, which prevents the gradient explosion of standard gradient ascent on cross-entropy.

What would settle it

Recompute MOLLM's update direction on a small model and check whether the inner product with each of the three gradients is negative at every step; any non-negative inner product shows the direction is not a common descent direction.

Watch

Extended reading notes

Core claim

The central claim is that LLM unlearning should be treated as a multi-objective problem with three objectives: increasing the unlearning loss on the forget set, minimizing KL divergence from the original model on retain data, and minimizing cross-entropy on retain data. The paper's contribution is a concrete algorithm, MOLLM, that computes a single update direction satisfying all three objectives simultaneously. The key innovation is DS-MGDA, which constructs the dual space of the span of the three gradients, computes each gradient's projection onto the null space of the other two, and averages the three dual vectors to produce a common descent direction. The paper argues this direction reaches Pareto stationarity by invoking multiple gradient descent theory, and empirically it reports that MOLLM outperforms GA-weighted-sum and GA-OGD on unlearning effect and utility preservation.

Load-bearing premise

The entire method depends on the unproven premise that the average of the three dual vectors always lands in the dual cone (so it decreases all three objectives) and that the reported gains are not just an artifact of tuning each baseline's learning rate separately.

Editorial extensions

If this is right

  • UCE loss removes the need for gradient clipping hyperparameter tuning in GA-based unlearning.
  • MOLLM's multi-objective step is linear-algebra-based and can be extended to more than three objectives whenever the gradient matrix remains full-rank.
  • The reported fluency of 1.689 (lower than the original 1.754) suggests unlearning with a common descent direction can, in principle, leave general language ability intact.
  • If the Pareto-stationarity claim holds, MOLLM offers a stable alternative to retraining-based unlearning in resource-constrained settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This reading suggests that the averaged-dual-vector direction may not be guaranteed to lie in the dual cone unless the gradients are scaled comparably; a normalization step could be needed for other models.
  • The convergence argument would be stronger if tested on a simple two-objective quadratic where the Pareto front is known, since the paper does not prove DS-MGDA reaches Pareto stationarity for its specific direction.
  • The dual-space construction is generic and could be applied to other multi-task fine-tuning problems, such as aligning a model while retaining reasoning ability, though the paper does not make that claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MOLLM, a multi-objective optimization approach to LLM unlearning. It replaces the unbounded cross-entropy loss in gradient ascent with a bounded Unlearning Cross-Entropy (UCE) loss, and introduces Dual Space Multiple Gradient Descent Algorithm (DS-MGDA) to compute a common descent direction that simultaneously decreases the forget loss, the KL-divergence to the original model, and the retain loss. Experiments on the PKU-SafeRLHF dataset with Llama 3-8B report that MOLLM achieves lower harmful rate, toxicity, and obscenity than GA-weighted-sum and GA-OGD baselines, while preserving (or improving) fluency. The authors also provide ablation studies and release source code.

Significance. If the theoretical and empirical claims held, the paper would contribute a useful, hyperparameter-light alternative to gradient-ascent-based LLM unlearning: the UCE loss is a simple and reasonable fix for gradient explosion, and a principled multi-objective update direction could mitigate catastrophic forgetting. The paper is clearly written and the code release is a strength. However, the central theoretical assertion—that DS-MGDA yields a common descent direction and inherits MGDA convergence guarantees—is not supported and is in fact false in simple cases, and the experimental comparison lacks error bars and multiple seeds. The contribution is therefore not yet established.

major comments (4)
  1. [Section III-B, Eq. (8)] The DS-MGDA direction can be zero even when a strict common descent direction exists, so the claim that d_t is always a common descent direction is false. For example, take g_fgt = (1,0), g_KL = (0,1), and g_rt = (1,1) in R^2. The span of any two of these gradients is all of R^2, so the projection onto the null space of the other two yields g*_fgt = g*_KL = g*_rt = 0, giving d_t = 0. Yet the direction d = (-1,-1) has negative dot product with all three gradients, so it is a common descent direction and is not Pareto stationary. Hence DS-MGDA can stall at a non-stationary point.
  2. [Section III-B, Definition 2] Definition 2 is internally inconsistent. For a linear subspace S, if a ∈ S then -a ∈ S as well. No nonzero vector b can satisfy b·a < 0 for every a ∈ S, because b·a < 0 and b·(-a) < 0 would imply b·a > 0. The subsequent statement that convex combinations of g*_i 'form the dual space' is therefore ill-posed; the correct object would be the negative polar cone, with non-strict inequalities, not a vector space with strict inner products.
  3. [Section III-B, paragraph after Definition 1] The claimed convergence to Pareto stationarity does not follow from the cited multiple gradient descent theorems [21]–[23]. Those theorems apply to the minimum-norm element of the convex hull of the gradients (or to the solution of a specific quadratic program), not to the averaged projection d_t = -(1/3)(g*_fgt + g*_KL + g*_rt). The paper provides no convergence proof for its specific direction, and the counterexample in the first major comment shows that convergence can fail completely.
  4. [Section IV-A and Table I] The experimental protocol 'take the best performance of each method' combined with no reported number of seeds or error bars makes the claimed superiority statistically unsupported; the reported gains over baselines may be due to selection bias in hyperparameter choice. In addition, the conflict probabilities PC_fgt, PC_KL, and PC_rt are zero for MOLLM by construction because d_t is explicitly built to have negative inner products with each gradient, so these columns do not provide independent evidence of conflict resolution.
minor comments (6)
  1. [Abstract and throughout] The method name is inconsistently spelled as 'MoLLM' in the abstract and 'MOLLM' in the body; please standardize.
  2. [Table I] The entry 'Re-fintuning' is a typo; it should be 'Re-finetuning'.
  3. [Section III-B] The sentence 'Since the dual vectors g*_fgt, g*_KL, g*_rt lie on the edge of S*, one of them is orthogonal to the other two' is unclear and not generally true; please state precisely the orthogonality relations that hold.
  4. [Eq. (4)] The KL divergence formula appears mis-typeset: the arguments of the KL term should be probability distributions, not the raw sequences; please correct the notation.
  5. [References] Reference [29] cites the original LLaMA paper, but the experiments use Llama 3-8B; please cite the correct Llama 3 model release.
  6. [Section III-B, Eq. (8)] The computation requires A A^T to be invertible, i.e., the gradients g_KL and g_rt must be linearly independent; the paper only notes this for more than three objectives, but it also applies in the three-objective case and is not guaranteed in high-dimensional parameter spaces.

Circularity Check

1 steps flagged · score 2.0 of 10

Only by-construction step is the PC=0 gradient-conflict metric, which is a tautology of DS-MGDA's projection; headline unlearning/utility claims rest on external SafeRLHF evaluation, so overall circularity is minimal.

  1. self definitional [Section III-B (Eq. 8 and definition of d_t) together with Section IV-B/Table I (PC columns)]
    "g∗_fgt = g_fgt − A^T (AA^T )^−1 A g_fgt, ... After obtaining g∗_fgt, g∗_KL, and g∗_rt, dt is obtained by dt = −1/3 (g∗_fgt + g∗_KL + g∗_rt). ... Furthermore, we calculate the probability of conflict between the model update direction and the g_fgt, g_KL, and g_rt during the unlearning process, denoting these probabilities as PC_fgt, PC_KL, and PC_rt."

    By Eq. (8), each g*_i is the residual of g_i after projecting onto the span of the other two gradients, so g*_i is orthogonal to the other two gradients and satisfies g*_i·g_i = ||g*_i||^2. Since d_t = −(1/3)(g*_fgt + g*_KL + g*_rt), the cross terms vanish and d_t·g_i = −||g*_i||^2/3 ≤ 0 for each i. Thus the sign of d_t·g_i is non-positive by construction for every gradient evaluation, and the reported Table I values PC_fgt = PC_KL = PC_rt = 0 for MOLLM are algebraic consequences of the update definition rather than an independent empirical measurement of conflict elimination. This is a mild tautology; it is not load-bearing for the headline harmful-rate/toxicity/fluency comparisons, which are computed externally on SafeRLHF.

full rationale

The paper's central derivation is not circular: the UCE loss is a new objective with a lower bound, the three-objective formulation is stated independently, and the headline numbers (harmful rate 3.45%, toxicity 0.007, fluency 1.689) come from evaluating LLama3-8B on the external PKU-SafeRLHF benchmark, not from fitting a parameter that is then renamed as a prediction. The convergence appeal is to Fliege–Svaiter's multiple-gradient-descent theorem [21]; the same-team references [22],[23] are cited alongside it as applications, so the self-citation is not the load-bearing source of the theorem. The only by-construction element is the PC=0 conflict-probability row, which follows directly from Eq. (8): d_t is defined as the negative average of mutually orthogonal residuals, so d_t·g_i = −||g*_i||^2/3 ≤ 0, making 'zero conflict' an identity rather than a discovery. That is a minor self-definitional step and does not undermine the independent empirical comparison. The 'best performance of each method' learning-rate selection is a benchmarking weakness but is not circularity, since the reported metrics are not constructed from that selection. The paper's Definition 2 of a dual space is mathematically problematic—a linear subspace cannot have negative inner product with both a and −a—but that is a correctness flaw in the justification, not a circular reduction of a claimed prediction to its own input.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the multi-objective formulation, the bounded UCE loss, the projection-based direction, and the assumption that the cited MGDA convergence theorem extends to DS-MGDA. No new physical or conceptual entities are introduced.

free parameters (4)
  • epsilon = not reported
    Small scalar in UCE loss (Eq. 6) that scales the probability term to keep the loss bounded when p_i,c approaches 1. Its value is not reported in the paper.
  • learning_rate = selected from {1e-5, 5e-6, 1e-6}
    Chosen via grid search with 'best performance of each method', which is a hyperparameter fit to the test set.
  • decay = 0.999
    Learning rate decay per round, fixed without sensitivity analysis.
  • training_rounds = not reported
    The number of unlearning rounds is not specified, which affects the comparison and reproducibility.
assumptions (4)
  • domain assumption LLM unlearning can be formulated as minimizing L_fgt, L_KL, and L_rt simultaneously, and a Pareto stationary point of this multi-objective problem yields a model that has forgotten the target data while retaining utility.
    This is the problem formulation in Eq. (3). It assumes that the three objectives fully capture the desired trade-off.
  • standard math The orthogonal projection formula in Eq. (8) correctly computes the projection of one gradient onto the null space of the other two.
    This is standard linear algebra, assuming A A^T is invertible.
  • domain assumption The theorem of multiple gradient descent (Fliege and Svaiter) applies to the DS-MGDA update direction, implying convergence to Pareto stationarity.
    The paper cites [21] in Section III-B but does not show that the average-of-dual-vectors direction satisfies the theorem's conditions.
  • domain assumption The CE loss on the retain set and the KL divergence with the original model are sufficient to preserve model utility.
    These are taken from prior work [6], [10] without validation on the specific tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Objective Large Language Model Unlearning." pith.science (2026). https://pith.science/paper/4D4CRCUQ

@misc{pith2026241220412,
  author       = {Pith},
  title        = {Pith review of: Multi-Objective Large Language Model Unlearning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4D4CRCUQ}},
  note         = {Machine review of arXiv:2412.20412}
}
read the original abstract

Machine unlearning in the domain of large language models (LLMs) has attracted great attention recently, which aims to effectively eliminate undesirable behaviors from LLMs without full retraining from scratch. In this paper, we explore the Gradient Ascent (GA) approach in LLM unlearning, which is a proactive way to decrease the prediction probability of the model on the target data in order to remove their influence. We analyze two challenges that render the process impractical: gradient explosion and catastrophic forgetting. To address these issues, we propose Multi-Objective Large Language Model Unlearning (MOLLM) algorithm. We first formulate LLM unlearning as a multi-objective optimization problem, in which the cross-entropy loss is modified to the unlearning version to overcome the gradient explosion issue. A common descent update direction is then calculated, which enables the model to forget the target data while preserving the utility of the LLM. Our empirical results verify that MoLLM outperforms the SOTA GA-based LLM unlearning methods in terms of unlearning effect and model utility preservation. The source code is available at https://github.com/zibinpan/MOLLM.

Figures

Figures reproduced from arXiv: 2412.20412 by the authors.

Figure 1
Figure 1. A demo of the gradient conflict during LLM unlearning, where [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. When using CE loss and GA for unlearning, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 12 canonical work pages

  1. [21]

    Steepest descent methods for multicriteria optimization,

    J. Fliege and B. F. Svaiter, “Steepest descent methods for multicriteria optimization,” Mathematical Methods of Operations Research , vol. 51, no. 3, pp. 479–494, 2000

    work page 2000

  2. [23]

    Fedlf: Layer-wise fair federated learn- ing,

    Z. Pan, C. Li, F. Yu, and et al., “Fedlf: Layer-wise fair federated learn- ing,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 13, 2024, pp. 14 527–14 535

    work page 2024

  3. [1]

    Constitutional ai: Harmlessness from ai feedback,

    Y . et al., “Constitutional ai: Harmlessness from ai feedback,” 2022. [Online]. Available: https://arxiv.org/abs/2212.08073

    arXiv 2022

  4. [2]

    Copyright traps for large language models,

    M. Meeus, I. Shilov, M. Faysse, and Y .-A. de Montjoye, “Copyright traps for large language models,” 2024. [Online]. Available: https: //arxiv.org/abs/2402.09363

    arXiv 2024

  5. [3]

    Towards mitigating llm hallucination via self reflection,

    Z. Ji, T. Yu, Y . Xu, and et al., “Towards mitigating llm hallucination via self reflection,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , 2023, pp. 1827–1843

    work page 2023

  6. [4]

    Privacy and data protection in chatgpt and other ai chatbots: strategies for securing user information,

    G. Sebastian, “Privacy and data protection in chatgpt and other ai chatbots: strategies for securing user information,” International Journal of Security and Privacy in Pervasive Computing (IJSPPC), vol. 15, no. 1, pp. 1–14, 2023

    work page 2023

  7. [5]

    Firewallm: A portable data protection and recovery framework for llm services,

    B. Huang, S. Yu, J. Li, and et al., “Firewallm: A portable data protection and recovery framework for llm services,” in International Conference on Data Mining and Big Data . Springer, 2023, pp. 16–30

    work page 2023

  8. [6]

    Large language model unlearning,

    Y . Yao, X. Xu, and Y . Liu, “Large language model unlearning,” arXiv preprint arXiv:2310.10683, 2023

    arXiv 2023

Show all 31 references
  1. [7]

    Rethinking machine unlearning for large language models,

    S. Liu, Y . Yao, J. Jia, and et al., “Rethinking machine unlearning for large language models,” arXiv preprint arXiv:2402.08787 , 2024

    2024 arXiv

  2. [8]

    Efuf: Efficient fine-grained unlearning framework for mitigating hallucinations in multimodal large language models,

    S. Xing, F. Zhao, Z. Wu, and et al., “Efuf: Efficient fine-grained unlearning framework for mitigating hallucinations in multimodal large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402. 09801

    2024

  3. [9]

    Avoiding copyright infringement via machine unlearning,

    G. Dou, Z. Liu, Q. Lyu, and et al., “Avoiding copyright infringement via machine unlearning,” 2024. [Online]. Available: https://arxiv.org/ abs/2406.10952

    2024 arXiv

  4. [10]

    Unlearn what you want to forget: Efficient unlearning for llms,

    J. Chen and D. Yang, “Unlearn what you want to forget: Efficient unlearning for llms,” 2023. [Online]. Available: https: //arxiv.org/abs/2310.20150

    2023 arXiv

  5. [11]

    In-context unlearning: Lan- guage models as few shot unlearners,

    M. Pawelczyk, S. Neel, and e. a. Lakkaraju, “In-context unlearning: Lan- guage models as few shot unlearners,” arXiv preprint arXiv:2310.07579, 2023

    2023 arXiv

  6. [12]

    Formalizing and bench- marking prompt injection attacks and defenses,

    Y . Liu, Y . Jia, R. Geng, and e. a. Jia, “Formalizing and bench- marking prompt injection attacks and defenses,” arXiv preprint arXiv:2310.12815, 2023

    2023

  7. [13]

    Who’s harry potter? approximate unlearning in llms,

    R. Eldan and M. Russinovich, “Who’s harry potter? approximate unlearning in llms,” 2023. [Online]. Available: https://arxiv.org/abs/ 2310.02238

    2023 arXiv

  8. [14]

    Training language models to fol- low instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, and et al., “Training language models to fol- low instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  9. [15]

    Rlaif: Scaling reinforce- ment learning from human feedback with ai feedback,

    H. Lee, S. Phatale, H. Mansoor, and et al., “Rlaif: Scaling reinforce- ment learning from human feedback with ai feedback,” arXiv preprint arXiv:2309.00267, 2023

    2023 arXiv

  10. [16]

    ∇τ: Gradient-based and task-agnostic machine unlearning,

    D. Trippa, C. Campagnano, M. S. Bucarelli, G. Tolomei, and F. Silvestri, “∇τ: Gradient-based and task-agnostic machine unlearning,” 2024. [Online]. Available: https://arxiv.org/abs/2403.14339

    2024 arXiv

  11. [17]

    A more practical approach to machine unlearning,

    D. Zagardo, “A more practical approach to machine unlearning,” arXiv preprint arXiv:2406.09391, 2024

    2024 arXiv

  12. [18]

    Federated unlearning: How to efficiently erase a client in fl?

    A. Halimi, S. Kadhe, A. Rawat, and N. Baracaldo, “Federated unlearning: How to efficiently erase a client in fl?” 2023. [Online]. Available: https://arxiv.org/abs/2207.05521

    2023 arXiv

  13. [19]

    Towards unbounded machine unlearning,

    M. Kurmanji, P. Triantafillou, J. Hayes, and E. Triantafillou, “Towards unbounded machine unlearning,” 2023. [Online]. Available: https://arxiv.org/abs/2302.09880

    2023 arXiv

  14. [20]

    Pku-saferlhf: A safety alignment preference dataset for llama family models,

    J. Ji, D. Hong, B. Zhang, and et al., “Pku-saferlhf: A safety alignment preference dataset for llama family models,” 2024. [Online]. Available: https://arxiv.org/abs/2406.15513

    2024 arXiv

  15. [22]

    Fedmdfg: Federated learning with multi-gradient descent and fair guidance,

    Z. Pan, S. Wang, C. Li, and et al., “Fedmdfg: Federated learning with multi-gradient descent and fair guidance,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 37, no. 8, 2023, pp. 9364– 9371

    2023

  16. [24]

    Polyhedral geometry and linear optimization,

    A. Paffenholz, “Polyhedral geometry and linear optimization,” Un- published Lecture Notes. Available at: http://www. mathematik. tu- darmstadt. de/paffenholz/daten/preprints/ln. pdf , 2010

    2010

  17. [25]

    G. H. Golub and C. F. Van Loan, Matrix computations . JHU press, 2013

    2013

  18. [26]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset,

    J. Ji, M. Liu, J. Dai, and et al., “Beavertails: Towards improved safety alignment of llm via a human-preference dataset,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  19. [27]

    Orthogonal gradient descent for continual learning,

    M. Farajtabar, N. Azizan, A. Mott, and A. Li, “Orthogonal gradient descent for continual learning,” in International Conference on Artificial Intelligence and Statistics . PMLR, 2020, pp. 3762–3773

    2020

  20. [28]

    Random relabeling for efficient machine unlearning,

    J. Li and S. Ghosh, “Random relabeling for efficient machine unlearning,” 2023. [Online]. Available: https://arxiv.org/abs/2305.12320

    2023 arXiv

  21. [29]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, and et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  22. [30]

    Detoxify,

    L. Hanu and Unitary team, “Detoxify,” Github. https://github.com/unitaryai/detoxify, 2020

    2020

  23. [31]

    A contrastive framework for neural text generation,

    Y . Su, T. Lan, Y . Wang, D. Yogatama, L. Kong, and N. Collier, “A contrastive framework for neural text generation,” Advances in Neural Information Processing Systems , vol. 35, pp. 21 548–21 561, 2022

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.