Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that loss reweighting for LLM unlearning has two complementary goals—saturation and importance—and that a simple product of the two, SatImp, improves the forget-retain trade-off over existing reweighting methods.

desk verdict A thorough empirical study of reweighting for LLM unlearning whose SatImp method is likely just a re-tuned saturation baseline; the importance component's contribution is not demonstrated. read the letter →

arxiv 2505.11953 v2 pith:MFHB3CRX submitted 2025-05-17 cs.LG cs.AI

Puning Yang , Qizhou Wang , Zhuo Huang , Tongliang Liu , Chengqi Zhang , Bo Han This is my paper · ORCID
classification cs.LGcs.AI
keywords machineunlearninglargelanguagemodelslossreweightingsaturationimportancegradientascentTOFUbenchmarkforget-retaintrade-off
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Loss reweighting is a standard fix for the over-forgetting caused by gradient-ascent unlearning in LLMs, but the field has not agreed on what the weights should reward. This paper proposes that reweighting pursues two distinct goals: saturation, which up-weights tokens the model still knows and therefore has not yet unlearned, and importance, which up-weights tokens that carry the meaning of a fact. Through controlled experiments on TOFU with Phi-1.5 and LLaMA-2-7B, the authors find that saturation helps retention more, importance helps forgetting more, and combining the two helps both. They turn this into SatImp, a per-token weight of the form $p^{\beta_1}(1-p)^{\beta_2}$, and report that it outperforms GA, PO, DPO, NPO, SimNPO, and WGA on the TOFU forget-retain balance, with supporting results on WMDP and MUSE. The paper is careful to note that the favorable results come in the gradient-difference setting with retain regularization; in forget-only settings, SatImp over-forgets at the larger TOFU fractions.

What carries the argument

The load-bearing object is the SatImp per-token weight $w_{x,y,k}^{\mathrm{satimp}}=p(y_k|y_{<k},x;\theta)^{\beta_1}(1-p(y_k|y_{<k},x;\theta))^{\beta_2}$, where $p(y_k|y_{<k},x;\theta)$ is the model's probability of the next token during unlearning. With $\beta_2=0$ it reduces to WGA's saturation weight; with $\beta_1=0$ it becomes the simplified importance weight; with both positive it concentrates weight on tokens of intermediate likelihood. Because the weight is token-wise, recomputed from the evolving model, and soft rather than a hard sample, it acts as a smooth, adaptive reweighting of the gradient-ascent objective; the ratio between $\beta_1$ and $\beta_2$ sets the direction of the weight-loss correlation, and their scale sets the distribution's peakedness, which the paper shows controls the unlearn-retain trade-off.

What would settle it

A concrete experiment is to rerun SatImp and WGA under the forget-only objective on TOFU 5% and 10% and compare ES Retain and MU; the paper reports both methods collapse to zero there, so a rerun that finds nonzero retention would show the collapse is not intrinsic to token-level reweighting, while a rerun that reproduces it confirms the regularization-dependence of the central claim.

Watch

Extended reading notes

Core claim

The central claim is that the apparent chaos of reweighting objectives for LLM unlearning reduces to two complementary criteria. Saturation-based reweighting—weights proportional to current token probability—targets tokens that have not been sufficiently unlearned; importance-based reweighting—weights proportional to one minus probability—targets low-likelihood tokens that carry key information. The paper shows these two criteria assign opposite weights to the same tokens, behave differently on unlearning versus retention, and can be combined multiplicatively. SatImp is that combination, $w_{x,y,k}=p^{\beta_1}(1-p)^{\beta_2}$, which shifts emphasis toward middle-loss tokens and lets the two exponents tune smoothness and the relative pull of saturation versus importance. With $\beta_1=5, \beta_2=1$ under the retain-regularized gradient-difference objective, the method reports the best retain-side extraction strength and model utility on TOFU while matching the strongest forget-side numbers, and it transfers to WMDP and MUSE better than or comparably with the baselines.

Load-bearing premise

The advertised SatImp advantage assumes the retain-regularized gradient-difference objective; the paper's own Appendix Table 4 shows that in forget-only runs at 5% and 10% TOFU, SatImp's retain score and model utility fall to zero.

Editorial extensions

If this is right

  • If SatImp's results hold, future unlearning objectives can be designed by specifying two numbers—how strongly to emphasize high-probability residual knowledge and how strongly to emphasize low-probability key tokens—rather than by ad hoc loss functions.
  • The paper's comparisons imply that soft token-level reweighting is preferable to hard TopK/BottomK sampling and to instance- or batch-level weights for unlearning.
  • The empirical distinction between saturation and importance gives a diagnostic: plotting weight against token loss reveals which objective a method actually implements, which can be used to audit existing unlearning methods.
  • Because SatImp's gains are demonstrated under retain-regularized gradient difference, the method's practical use will require a retention set; in forget-only deployments, larger forget fractions over-forget, as the paper's Appendix Table 4 shows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would expect the saturation/importance dichotomy to transfer to other data-weighting problems, such as noisy-label learning or preference optimization, where 'hard examples' and 'influential examples' are usually treated as the same thing; SatImp suggests they should be separate coordinates of the weight function.
  • A direct extension the paper does not explore is automating the importance labels: the KTL index, which ranks a key token's likelihood within its sample, could serve as a training signal for a weight predictor, removing the manual annotation bottleneck.
  • The middle-loss emphasis of SatImp hints at a principled curriculum: anneal $\beta_1$ and $\beta_2$ over training so the objective starts saturation-heavy and shifts importance-heavy, which could reduce the reported parameter sensitivity.
  • The paper's WMDP retain-regularization numbers use TOFU-tuned hyperparameters; a per-benchmark sweep would be the natural stress test of whether SatImp's advantage is a property of the weight shape or of the particular $\beta$ values chosen.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper studies loss reweighting in LLM unlearning and proposes a taxonomy of two goals: saturation (emphasizing tokens that are not yet sufficiently unlearned) and importance (emphasizing critical tokens). It introduces a manually labeled importance weighting on TOFU, a saturation weight in Eq. (9), simplified versions SimSat and SimImp in Eq. (11), and proposes SatImp in Eq. (12) as a product of saturation and importance weights. The main empirical claim is that SatImp outperforms advanced baselines such as NPO, SimNPO, and WGA on TOFU with retain regularization (Table 2), with additional experiments on WMDP and MUSE. The paper also studies how smoothness, granularity, and hard sampling affect unlearning performance. The central claim is plausible but currently rests on single-run comparisons, a specific retain-regularized setting, and a missing control that isolates the importance component.

Significance. If the central claim were robust, the paper would make a useful conceptual contribution by organizing existing reweighting methods into saturation and importance categories and by offering SatImp as a simple, tunable baseline. The manuscript has concrete strengths: the code is released, the experimental setup is documented in detail, the manual token-level annotation effort on TOFU is considerable, and the appendix covers TOFU, WMDP, and MUSE with multiple metrics. The paper also includes a self-aware limitations section (Appendix A) that concedes parameter sensitivity. However, the significance is currently limited by evidence quality: the favorable TOFU results require retain regularization, there are no repeated-seed statistics, and the importance component of SatImp is not isolated by an ablation. One stress-test concern from the accompanying note does not land on inspection: in the forget-only WMDP setting (Table 3), SatImp matches or improves on RMU's unlearn accuracy while retaining comparable MMLU, so the WMDP evidence is not the weak point.

major comments (5)
  1. [Section 5, Eq. (12), Table 2] The reported SatImp configuration does not implement the stated importance behavior. With the hyperparameters used in Table 2, β1=5 and β2=1, the weight p^5(1-p) is increasing on [0, 5/6] and maximized at p=5/6, so it assigns the largest weights to high-likelihood tokens. The importance motivation in Eq. (11) is precisely the opposite: it assigns larger weights to low-likelihood tokens. The paper's own Figure 5(a), which illustrates 'middle-loss' emphasis, uses β1=β2=1 and not the evaluated configuration. Table 2 also lacks a β2=0 control with the same β1=5, so the reported gains over WGA cannot be attributed to the importance factor; they may simply reflect a differently tuned saturation-style weighting. Please add an ablation over (β1,β2), including at least β2=0 with matched β1, and state explicitly which configurations are selected and whether the importance term contributes.
  2. [Appendix Table 4, Appendix E, Section 1] The claim of SatImp's superiority is scoped in a way that is not stated in the abstract or introduction. In the forget-only TOFU setting (Eq. (1)), Table 4 shows that SatImp collapses on 5% and 10% forget sets for both Phi-1.5 and LLaMA-2-7B, with ES Retain, ES Unlearn, and MU all at zero, and LLaMA-2-7B at 5% has ES Retain 0.0080 and ES Unlearn 0.0057. Appendix A itself concedes that SatImp remains parameter-sensitive. The favorable claims in Section 1 and in the analysis of Table 2 therefore hold only with retain regularization (Eq. (2)) and the TOFU-specific λ values. Please either explicitly scope the claim to the gradient-difference setting, or provide a calibration mechanism, such as the early-stopping procedure discussed in Appendix C.2, and show that it restores acceptable behavior in forget-only settings.
  3. [Tables 2-6 and Figures 2-4] All experimental results appear to be single runs: there are no error bars, no repeated seeds, and no significance tests. This matters because the reported margins in Table 2 are small in several cases, for example the ES Retain differences between SatImp and WGA on Phi-1.5 5% (0.5410 vs. 0.5219) and on Phi-1.5 10% (0.4706 vs. 0.4655). Without variance information, the statement in Section 1 that 'the results clearly demonstrate the superiority of our SatImp' is not supported. Please report mean and standard deviation over at least three seeds for the main TOFU comparisons, and describe the checkpoint selection rule used for Tables 2 and 4.
  4. [Section 4.1, Eq. (9), Eq. (11), Figure 2(h)] One of the paper's stated findings is true by construction rather than by empirical observation. The saturation weights in Eq. (9), p/(p+τ), and in Eq. (11), p^β, are monotonically increasing functions of token probability and therefore monotonically decreasing functions of negative log-likelihood. Figure 2(h) and the accompanying text report that saturation assigns smaller weights to lower-likelihood tokens, but this is a property of the definition, not an empirical discovery about reweighting behavior. The authors should rephrase this as a design property and provide evidence that the saturation mechanism itself, rather than the monotone weighting, is what improves unlearning performance.
  5. [Section 3.1 and Section 4.1] The importance-based reweighting relies on manual binary labels that were produced by four team members with a 'half agreement' threshold, but no inter-annotator agreement metric (e.g., Cohen's kappa or Fleiss' kappa) is reported, and no label-quality statistics are provided. The subsequent simplification to (1-p)^β in Eq. (11) assumes that the correlation between manually labeled key tokens and low likelihood, as illustrated in Figure 2(g), is stable across domains and models. Given that the importance component is one of the two pillars of SatImp, the authors should either report label reliability and validate the proxy on at least one additional benchmark, or explicitly frame SimImp as a heuristic rather than a validated importance measure.
minor comments (6)
  1. [Section 4.1 heading] The heading contains a typo: 'Comparsion' should be 'Comparison'.
  2. [Appendix B.2 and B.3] There are several typos in the setup text: 'learning rete', 'differnt', and 'tow hyper-parameters' should be corrected to 'learning rate', 'different', and 'two hyper-parameters'.
  3. [Figure 2 caption] The caption for Figure 2 refers to subfigures (e)-(h) as 'Imp. & Sat. Detail' and 'SimNPO & NPO Detail', but the labels inside the subfigures are not consistently defined; adding explicit panel labels would improve readability.
  4. [Appendix B.1.1, Eq. (13)] The definition of ES in Eq. (13) uses the notation f(y_<k|x; θ) = y_>k, but the subscript on the right-hand side is visually ambiguous and could be confused with the metric under discussion; please use an explicit suffix notation such as y_{>k}.
  5. [Appendix Table 4] In the Phi-1.5 rows of Table 4, several entries are exactly 0.0000 across ES Retain, ES Unlearn, and MU for the 5% and 10% settings; a footnote explaining that these are collapsed runs, rather than missing or zero-due-to-rounding values, would help the reader interpret the failure mode.
  6. [Appendix C.2] The three-stage description (unlearning, stabilization, collapse) is interesting but is presented as purely descriptive; if it is meant to guide early stopping, the paper should provide a concrete rule for detecting the stabilization stage at inference time.

Circularity Check

2 steps flagged · score 3.0 of 10

Two 'findings' about weight–likelihood polarity are true by construction; the SatImp performance claim is empirical.

  1. self definitional [Section 4.1, 'Simplification of Reweighting' and Eq. (11)]
    "Overall, we observe that importance-based reweighting tends to associate key annotations with low-likelihood tokens, whereas saturation-based reweighting generally allocates smaller weights to them. ... Specifically, we suggest allocating large weights to low-likelihood tokens for importance-based reweighting and small weights to low-likelihood tokens for saturation-based reweighting, which lead to Simple Importance (SimImp) and Simple Saturation (SimSat): wsimsat_{x,y,k}=p(y_k|y_{<k},x;θ)^β, wsimimp_{x,y,k}=(1-p(y_k|y_{<k},x;θ))^β."

    By Eq. (11), p is the token likelihood. The function p^β is strictly increasing in p, so it necessarily assigns smaller weights to lower-likelihood tokens; (1-p)^β is strictly decreasing in p, so it necessarily assigns larger weights to lower-likelihood tokens. The paper's 'observation' is therefore the derivative of its own proposed formulas, not an empirical discovery. Any scatter plot of these weights against negative log-likelihood will show exactly that monotone polarity by construction, so the reported correlation is definitional rather than independent evidence.

  2. self definitional [Section 3.2, Eq. (9) and Figure 2(h)]
    "Following (Sablayrolles et al., 2019)... we adopt the reweighting strategy as follows: wsat_{x,y,k}=p(y_k|y_{<k},x;θ)/(p(y_k|y_{<k},x;θ)+τ). ... The result, shown in Figure 2(h), clearly highlights the strict inverse correlation between losses and weights, supporting our assumption again."

    With p denoting token likelihood and the loss being -log p, the saturation weight w_sat=p/(p+τ) has derivative d w_sat/d(-log p) = -τ p/(p+τ)^2, which is strictly negative for every p>0. Thus the 'strict inverse correlation between losses and weights' displayed in Figure 2(h) is a mathematical property of Eq. (9), true for every token at every training step, rather than an empirical regularity that could support or falsify the saturation hypothesis.

full rationale

The self-definitional steps are localized to the paper's descriptive 'observations' about how its own weight formulas behave with respect to token likelihood. The central claim—that SatImp improves the forget-retain trade-off—is an empirical benchmark comparison against TOFU, WMDP, and MUSE, and it does not reduce to Eq. (11) or Eq. (12). The paper is self-contained against external benchmarks, and I found no load-bearing self-citation chain, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. Several concerns raised in the skeptical reading are experimental rather than circular: the absence of a β2=0 ablation isolating the importance factor, single-run comparisons without error bars, and the documented collapse under forget-only regularization in Appendix Table 4. These are correctness risks, not derivation-circularity. The score of 3 reflects that two claimed empirical findings are in fact definitional, while the main method's performance claim remains independently testable.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two fitted hyperparameters (beta1, beta2) and several domain assumptions about the validity of the ES metric and the likelihood-importance correlation. No new physical or mathematical entities are introduced.

free parameters (5)
  • beta1 (SatImp) = 5
    Controls the saturation weighting exponent in Eq. (12); chosen on TOFU, not transferred to WMDP (where it underperforms RMU).
  • beta2 (SatImp) = 1
    Controls the importance weighting exponent in Eq. (12); chosen on TOFU.
  • tau (saturation) = 1
    Calibration constant in Eq. (9) for the saturation weight; used for analysis only, not in SatImp.
  • p (importance) = 0.3 (and 0.4 for combination)
    Weight for non-important tokens in Eq. (8); used for the manual importance analysis.
  • lambda (GD regularization) = 1 for GA/WGA; searched [0.2,0.5] for NPO
    Trade-off between unlearn and retain terms in Eq. (2).
assumptions (5)
  • domain assumption Gradient ascent on the unlearn set reduces the model's ability to reproduce target data.
    Foundational premise of all GA-based unlearning methods, used in Eq. (1).
  • domain assumption The Extraction Strength (ES) metric (from the authors' prior work) is a reliable measure of unlearning and retention.
    The paper defaults to ES scores from Wang et al. 2025a for evaluating all methods.
  • ad hoc to paper Manual binary labels of important tokens in TOFU are accurate, and the correlation between importance and low token likelihood justifies replacing manual labels with (1-p)^beta.
    The simplification to SimImp relies on this correlation (Figure 2g).
  • domain assumption Token-wise reweighting is more effective than instance-wise, as assumed from prior work and extended here.
    Stated in Section 3.2 and validated in Section 4.2.
  • domain assumption The benchmark settings and hyperparameter choices (beta1=5, beta2=1) generalize beyond the tuning set.
    The paper applies TOFU-tuned hyperparameters to WMDP and MUSE, but results are mixed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning." pith.science (2026). https://pith.science/paper/MFHB3CRX

@misc{pith2026250511953,
  author       = {Pith},
  title        = {Pith review of: Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MFHB3CRX}},
  note         = {Machine review of arXiv:2505.11953}
}
read the original abstract

Loss reweighting has shown significant benefits for machine unlearning with large language models (LLMs). However, their exact functionalities are left unclear and the optimal strategy remains an open question, thus impeding the understanding and improvement of existing methodologies. In this paper, we identify two distinct goals of loss reweighting, namely, Saturation and Importance -- the former indicates that those insufficiently optimized data should be emphasized, while the latter stresses some critical data that are most influential for loss minimization. To study their usefulness, we design specific reweighting strategies for each goal and evaluate their respective effects on unlearning. We conduct extensive empirical analyses on well-established benchmarks, and summarize some important observations as follows: (i) Saturation enhances efficacy more than importance-based reweighting, and their combination can yield additional improvements. (ii) Saturation typically allocates lower weights to data with lower likelihoods, whereas importance-based reweighting does the opposite. (iii) The efficacy of unlearning is also largely influenced by the smoothness and granularity of the weight distributions. Based on these findings, we propose SatImp, a simple reweighting method that combines the advantages of both saturation and importance. Empirical results on extensive datasets validate the efficacy of our method, potentially bridging existing research gaps and indicating directions for future research. Our code is available at https://github.com/tmlr-group/SatImp.

Figures

Figures reproduced from arXiv: 2505.11953 by the authors.

Figure 1
Figure 1. Overview of our paper. Beginning with a summary of existing LLM unlearning methods, we categorize them into importance￾based and saturation-based reweighting, followed by a comprehensive analysis. Furthermore, we investigate the effects of specific reweighting operations and identify three core issues that significantly impact unlearning performance. Finally, we clarify the optimization direction for LLM unlearning,… view at source ↗
Figure 2
Figure 2. Comparisons between saturation- and importance-based reweighting. We depict TOFU settings (x-axis) versus ES scores (y-axis) on retain (ES Retain) and unlearn (ES Unlearn) data. ↑ / ↓ indicate larger / smaller values are preferable. While saturation-based reweighting outperforms importance-based reweighting on retention tasks, it slightly underperforms on unlearning tasks (a-d). This performance disparity stems from… view at source ↗
Figure 3
Figure 3. Correlations between performance and weight distribution. A comprehensive comparison between SimSat and SimImp indicates that each has its own performance characteristics, with SimSat achieving a better unlearn-retain trade-off. Model performance is sensitive to the smoothness of weight distribution (varying β). Besides, while hard-sampling is effective in enhancing LLM unlearning performance, it still underperforms… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Comparisons of different weight granularity. Finer-grained reweighting strategies typically achieve a better unlearn-retain trade-off under the SimSat method. Meanwhile, under the SimImp method, fine-grained strategies lead to improvements on retention tasks, albeit at…
Figure 5
Figure 5. Figure 5: The correspondence of weight and loss values. (a) SatImp (Green) emphasize the middle-loss tokens in reweight￾ing, which is different from previous importance-based (red) and saturation-based (blue) methods. Besides, (b) SatImp is more flexi￾ble in weight allocation, r…
Figure 6
Figure 6. Figure 6: , we conduct that key annotations is high-correlated to tokens with lower likelihoods (higher negative log likelihood values). The weights of NPO and SimNPO are instance-wise, which tends to allocate large weights to tokens with higher likelihoods ( [PITH_FULL_IMAGE:f…
Figure 7
Figure 7. Figure 7: Loss-weight details for NPO and SimNPO methods. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Loss-weight details for WGA. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Loss-weight details for saturation-based reweighting. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Enhancing LLM unleanring via early-stopping based on visualized training process. While hard-sampling methods are effective, they still underperform soft-weighting methods. Notably, the BottomK strategy enables a smoother training process with three distinct phases. F…
Figure 11
Figure 11. Figure 11: ES Scores with importance-based reweighting. We depict values of p (x-axis) versus the ES scores (y-axis) on unlearn and retain data. We consider 2 LLMs (Phi-1.5 (Phi) and LLaMA-2-7B (Llama)) and 4 unlearning methods (GA, GD, NPO, and NPO-GD (N-GD)) under the 1%, 5%, …
Figure 12
Figure 12. Figure 12: Forget Quality and Model Utility with importance-based reweighting. We depict values of p (x-axis) versus the TOFU Index (y-axis) on unlearn and retain data. We consider 2 LLMs (Phi-1.5 (Phi) and LLaMA-2-7B (Llama)) and 4 unlearning methods (GA, GD, NPO, and NPO-GD (N…
Figure 13
Figure 13. Figure 13: ES Scores with SimSat (Sat) and SimImp (Imp) paradigms. We depict values of β (x-axis) versus the ES scores (y-axis) on unlearn and retain data. We consider 2 LLMs (Phi-1.5 (Phi) and LLaMA-2-7B (Llama)) and 2 unlearning methods (GA, GD) under the 1%, 5%, and 10% TOFU …
Figure 14
Figure 14. Figure 14: Forget Quality and Model Utility with SimSat (Sat) and SimImp (Imp) paradigms. We depict values of β (x-axis) versus the TOFU-Index (y-axis) on unlearn and retain data. We consider 2 LLMs (Phi-1.5 (Phi) and LLaMA-2-7B (Llama)) and 2 unlearning methods (GA, GD) under t…
Figure 15
Figure 15. Figure 15: Impact of weight granularity on Phi-1.5 Model. We depict values of β (x-axis) versus the ES scores (y-axis) on unlearn (Un↓) and retain (Re↑) data. SimSat (Sat) and SimImp (Imp) methods are investigated. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Impact of weight granularity on LLaMA-2-7B Model. We depict values of β (x-axis) versus the ES scores (y-axis) on unlearn (Un↓) and retain (Re↑) data. SimSat (Sat) and SimImp (Imp) methods are investigated. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_16.png]
Figure 17
Figure 17. Figure 17: Hard sampling performance of TopK and BottomK (BotK) sampling. We depict values of β (x-axis) versus the ES scores (y-axis) on unlearn (Un↓) and retain (Re↑) data. LLaMA-2-7B (L) and Phi-1.5 (P) are investigated. Dashed line represents baseline performance. 29 [PITH_…
Figure 18
Figure 18. Figure 18: Hard sampling performance of random (Rand) sampling. We depict values of β (x-axis) versus the ES scores (y-axis) on unlearn (Un↓) and retain (Re↑) data. LLaMA-2-7B (L) and Phi-1.5 (P) are investigated. Dashed line represents baseline performance. 30 [PITH_FULL_IMAGE…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearning

    cs.LG 2026-01 conditional novelty 6.0 of 10

    A circuit-similarity score predicts which samples an LLM unlearning method will fail to erase, with hard samples relying on deeper, output-facing pathways.

Reference graph

Works this paper leans on

82 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    The cringe loss: Learning what language not to model

    Adolphs, L., Gao, T., Xu, J., Shuster, K., Sukhbaatar, S., and Weston, J. The cringe loss: Learning what language not to model. arXiv preprint arXiv:2211.05826, 2022

    arXiv 2022

  3. [3]

    P., Narang, N., Rivers, J., Maksey, R., Guan, L., Barrere, L

    Bao, Y., Shah, A. P., Narang, N., Rivers, J., Maksey, R., Guan, L., Barrere, L. N., Evenson, S., Basole, R., Miao, C., et al. Harnessing business and media insights with large language models. arXiv preprint arXiv:2406.06559, 2024

    arXiv 2024

  4. [4]

    R., Christodorescu, M., Datta, A., Feizi, S., et al

    Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al. Identifying and mitigating the security risks of generative ai. Foundations and Trends in Privacy and Security , 6 0 (1): 0 1--52, 2023

    2023

  5. [5]

    Soft prompting for unlearning in large language models

    Bhaila, K., Van, M.-H., and Wu, X. Soft prompting for unlearning in large language models. arXiv preprint arXiv:2406.12038, 2024

    arXiv 2024

  6. [6]

    A., Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N

    Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp.\ 141--159. IEEE, 2021

    2021

  7. [7]

    and Lipton, Z

    Byrd, J. and Lipton, Z. What is the effect of importance weighting in deep learning? In International conference on machine learning, pp.\ 872--881. PMLR, 2019

    work page 2019

  8. [8]

    and Yang, D

    Chen, J. and Yang, D. Unlearn what you want to forget: Efficient unlearning for llms. arXiv preprint arXiv:2310.20150, 2023

    arXiv 2023

Show all 82 references
  1. [9]

    Snap: Unlearning selective knowledge in large language models with negative instructions

    Choi, M., Rim, D., Lee, D., and Choo, J. Snap: Unlearning selective knowledge in large language models with negative instructions. arXiv preprint arXiv:2406.12329, 2024

    2024 arXiv

  2. [10]

    Label smoothing improves machine unlearning

    Di, Z., Zhu, Z., Jia, J., Liu, J., Takhirov, Z., Jiang, B., Yao, Y., Liu, S., and Liu, Y. Label smoothing improves machine unlearning. arXiv preprint arXiv:2406.07698, 2024

    2024 arXiv

  3. [11]

    R., Lin, H., Belkin, M., Huerta, R., and Vuli \'c , I

    Dong, Y. R., Lin, H., Belkin, M., Huerta, R., and Vuli \'c , I. Undial: Self-distillation with adjusted logits for robust unlearning in large language models. arXiv preprint arXiv:2402.10052, 2024

    2024 arXiv

  4. [12]

    and Russinovich, M

    Eldan, R. and Russinovich, M. Who's harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023

    2023 arXiv

  5. [13]

    Kto: Model alignment as prospect theoretic optimization

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  6. [14]

    Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation

    Fan, C., Liu, J., Zhang, Y., Wong, E., Wei, D., and Liu, S. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. arXiv preprint arXiv:2310.12508, 2023

    2023 arXiv

  7. [15]

    Simplicity prevails: Rethinking negative preference optimization for llm unlearning

    Fan, C., Liu, J., Lin, L., Jia, J., Zhang, R., Mei, S., and Liu, S. Simplicity prevails: Rethinking negative preference optimization for llm unlearning. arXiv preprint arXiv:2410.07163, 2024

    2024

  8. [16]

    Challenging forgets: Unveiling the worst-case forget sets in machine unlearning

    Fan, C., Liu, J., Hero, A., and Liu, S. Challenging forgets: Unveiling the worst-case forget sets in machine unlearning. In European Conference on Computer Vision, pp.\ 278--297. Springer, 2025

    2025

  9. [17]

    Practical unlearning for large language models

    Gao, C., Wang, L., Weng, C., Wang, X., and Zhu, Q. Practical unlearning for large language models. arXiv preprint arXiv:2407.10223, 2024

    2024 arXiv

  10. [18]

    General data protection regulation

    GDPR, G. General data protection regulation. Regulation (EU), 679, 2016

    2016

  11. [19]

    Eternal sunshine of the spotless net: Selective forgetting in deep networks

    Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 9304--9312, 2020

    2020

  12. [20]

    Amnesiac machine learning

    Graves, L., Nagisetty, V., and Ganesh, V. Amnesiac machine learning. 35 0 (13): 0 11516--11524, 2021

    2021

  13. [21]

    Meow: Memory supervised llm unlearning via inverted facts

    Gu, T., Huang, K., Luo, R., Yao, Y., Yang, Y., Teng, Y., and Wang, Y. Meow: Memory supervised llm unlearning via inverted facts. arXiv preprint arXiv:2409.11844, 2024

    2024 arXiv

  14. [22]

    Aligning ai with shared human values

    Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021 a

    2021

  15. [23]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021 b

    2021

  16. [24]

    Y., Zhou, W., Wang, F., Morstatter, F., Zhang, S., Poon, H., and Chen, M

    Huang, J. Y., Zhou, W., Wang, F., Morstatter, F., Zhang, S., Poon, H., and Chen, M. Offset unlearning for large language models. arXiv preprint arXiv:2404.11045, 2024 a

    2024 arXiv

  17. [25]

    Trustllm: Trustworthiness in large language models

    Huang, Y., Sun, L., Wang, H., Wu, S., Zhang, Q., Li, Y., Gao, C., Huang, Y., Lyu, W., Zhang, Y., et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561, 2024 b

    2024 arXiv

  18. [26]

    Robust generalization against photon-limited corruptions via worst-case sharpness minimization

    Huang, Z., Zhu, M., Xia, X., Shen, L., Yu, J., Gong, C., Han, B., Du, B., and Liu, T. Robust generalization against photon-limited corruptions via worst-case sharpness minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 1617...

    2023

  19. [27]

    T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A

    Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  20. [28]

    A., Chaudhuri, K., and Zou, J

    Izzo, Z., Smart, M. A., Chaudhuri, K., and Zou, J. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp.\ 2008--2016. PMLR, 2021

    2008

  21. [29]

    Knowledge unlearning for mitigating privacy risks in language models

    Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., and Seo, M. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504, 2022

    2022 arXiv

  22. [30]

    R., Liu, S., and Chang, S

    Ji, J., Liu, Y., Zhang, Y., Liu, G., Kompella, R. R., Liu, S., and Chang, S. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference. arXiv preprint arXiv:2406.08607, 2024

    2024 arXiv

  23. [31]

    Soul: Unlocking the power of second-order optimization for llm unlearning

    Jia, J., Zhang, Y., Zhang, Y., Liu, J., Runwal, B., Diffenderfer, J., Kailkhura, B., and Liu, S. Soul: Unlocking the power of second-order optimization for llm unlearning. arXiv preprint arXiv:2404.18239, 2024

    2024 arXiv

  24. [32]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  25. [33]

    Copyright violations and large language models

    Karamolegkou, A., Li, J., Zhou, L., and S gaard, A. Copyright violations and large language models. arXiv preprint arXiv:2310.13771, 2023

    2023 arXiv

  26. [34]

    D., Dombrowski, A.-K., Goel, S., Phan, L., et al

    Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Phan, L., et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024

    2024 arXiv

  27. [35]

    Deepinception: Hypnotize large language model to be jailbreaker

    Li, X., Zhou, Z., Zhu, J., Yao, J., Liu, T., and Han, B. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191, 2023 a

    2023 arXiv

  28. [36]

    Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023 b

    2023 arXiv

  29. [37]

    Y., Wang, Y., Flanigan, J., and Liu, Y

    Liu, C. Y., Wang, Y., Flanigan, J., and Liu, Y. Large language model unlearning via embedding-corrupted prompts. arXiv preprint arXiv:2406.07933, 2024 a

    2024 arXiv

  30. [38]

    Model sparsity can simplify machine unlearning

    Liu, J., Ram, P., Yao, Y., Liu, G., Liu, Y., SHARMA, P., Liu, S., et al. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36, 2024 b

    2024

  31. [39]

    Do humans and machines have the same eyes? human-machine perceptual differences on image classification

    Liu, M., Wei, J., Liu, Y., and Davis, J. Do humans and machines have the same eyes? human-machine perceptual differences on image classification. arXiv preprint arXiv:2304.08733, 2023

    2023 arXiv

  32. [40]

    Automatic dataset construction (adc): Sample collection, data curation, and beyond

    Liu, M., Di, Z., Wei, J., Wang, Z., Zhang, H., Xiao, R., Wang, H., Pang, J., Chen, H., Shah, A., et al. Automatic dataset construction (adc): Sample collection, data curation, and beyond. arXiv preprint arXiv:2408.11338, 2024 c

    2024 arXiv

  33. [41]

    Y., Xu, X., Li, H., et al

    Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Yao, Y., Liu, C. Y., Xu, X., Li, H., et al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787, 2024 d

    2024 arXiv

  34. [42]

    and Guo, H

    Liu, Y. and Guo, H. Peer loss functions: Learning from noisy labels without knowing noise rates. In International conference on machine learning, pp.\ 6226--6236. PMLR, 2020

    2020

  35. [43]

    C., and Kolter, J

    Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and Kolter, J. Z. Tofu: A task of fictitious unlearning for llms. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024

    2024

  36. [44]

    Alternate preference optimization for unlearning factual knowledge in large language models

    Mekala, A., Dorna, V., Dubey, S., Lalwani, A., Koleczek, D., Rungta, M., Hasan, S., and Lobo, E. Alternate preference optimization for unlearning factual knowledge in large language models. arXiv preprint arXiv:2409.13474, 2024

    2024 arXiv

  37. [45]

    More human than human: measuring chatgpt political bias

    Motoki, F., Pinho Neto, V., and Rodrigues, V. More human than human: measuring chatgpt political bias. Public Choice, 198 0 (1): 0 3--23, 2024

    2024

  38. [46]

    K., Shokri, R., and Theodorakopoulos, G

    Murakonda, S. K., Shokri, R., and Theodorakopoulos, G. Quantifying the privacy risks of learning high-dimensional graphical models. In International Conference on Artificial Intelligence and Statistics, pp.\ 2287--2295. PMLR, 2021

    2021

  39. [47]

    R., and Papernot, N

    Muresanu, A., Thudi, A., Zhang, M. R., and Papernot, N. Unlearnable algorithms for in-context learning. arXiv preprint arXiv:2402.00751, 2024

    2024

  40. [48]

    Pardau, S. L. The california consumer privacy act: Towards a european-style privacy regime in the united states. J. Tech. L. & Pol'y, 23: 0 68, 2018

    2018

  41. [49]

    Can sensitive information be deleted from llms? objectives for defending against extraction attacks

    Patil, V., Hase, P., and Bansal, M. Can sensitive information be deleted from llms? objectives for defending against extraction attacks. arXiv preprint arXiv:2309.17410, 2023

    2023 arXiv

  42. [50]

    In-context unlearning: Language models as few shot unlearners

    Pawelczyk, M., Neel, S., and Lakkaraju, H. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023

    2023 arXiv

  43. [51]

    Safety alignment should be made more than just a few tokens deep

    Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. Safety alignment should be made more than just a few tokens deep. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  44. [52]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  45. [53]

    White-box vs black-box: Bayes optimal strategies for membership inference

    Sablayrolles, A., Douze, M., Ollivier, Y., Schmid, C., and J \'e gou, H. White-box vs black-box: Bayes optimal strategies for membership inference. ICML, 2019

    2019

  46. [54]

    Detecting pretraining data from large language models

    Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023

    2023 arXiv

  47. [55]

    A., and Zhang, C

    Shi, W., Lee, J., Huang, Y., Malladi, S., Zhao, J., Holtzman, A., Liu, D., Zettlemoyer, L., Smith, N. A., and Zhang, C. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06460, 2024

    2024 arXiv

  48. [56]

    Membership inference attacks against machine learning models

    Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Membership inference attacks against machine learning models. In SP, 2017

    2017

  49. [57]

    S., and Smith, V

    Thaker, P., Maurya, Y., Hu, S., Wu, Z. S., and Smith, V. Guardrail baselines for unlearning in llms. arXiv preprint arXiv:2403.03329, 2024

    2024 arXiv

  50. [58]

    Unrolling sgd: Understanding factors influencing machine unlearning

    Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp.\ 303--319. IEEE, 2022

    2022

  51. [59]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  52. [60]

    Kga: A general machine unlearning framework based on knowledge gap alignment

    Wang, L., Chen, T., Yuan, W., Zeng, X., Wong, K.-F., and Yin, H. Kga: A general machine unlearning framework based on knowledge gap alignment. arXiv preprint arXiv:2305.06535, 2023 a

    2023 arXiv

  53. [61]

    Learning to augment distributions for out-of-distribution detection

    Wang, Q., Fang, Z., Zhang, Y., Liu, F., Li, Y., and Han, B. Learning to augment distributions for out-of-distribution detection. In Advances in Neural Information Processing Systems, 2023 b

    2023

  54. [62]

    A sober look at the robustness of clips to spurious features

    Wang, Q., Lin, Y., Chen, Y., Schmidt, L., Han, B., and Zhang, T. A sober look at the robustness of clips to spurious features. In Advances in Neural Information Processing Systems, 2024 a

    2024

  55. [63]

    Towards effective evaluations and comparison for llm unlearning methods

    Wang, Q., Han, B., Yang, P., Zhu, J., Liu, T., and Sugiyama, M. Towards effective evaluations and comparison for llm unlearning methods. In International Conference on Learning Representations, 2025 a

    2025

  56. [64]

    P., Zhou, Z., Shin, S., Han, B., and Weinberger, K

    Wang, Q., Zhou, J. P., Zhou, Z., Shin, S., Han, B., and Weinberger, K. Q. Rethinking llm unlearning objectives: A gradient perspective and go beyond. In International Conference on Learning Representations, 2025 b

    2025

  57. [65]

    A., Khashabi, D., and Hajishirzi, H

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022

    2022 arXiv

  58. [66]

    Y., Pang, J., Liu, Q., Shah, A

    Wang, Y., Wei, J., Liu, C. Y., Pang, J., Liu, Q., Shah, A. P., Bao, Y., Liu, Y., and Wei, W. Llm unlearning via loss adjustment with only forget data. arXiv preprint arXiv:2410.11143, 2024 b

    2024 arXiv

  59. [67]

    Gru: Mitigating the trade-off between unlearning and retention for large language models

    Wang, Y., Wang, Q., Liu, F., Huang, W., Du, Y., Du, X., and Han, B. Gru: Mitigating the trade-off between unlearning and retention for large language models. In International Conference on Machine Learning, 2025 c

    2025

  60. [68]

    Learning with noisy labels revisited: A study using real-world human annotations

    Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., and Liu, Y. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:2110.12088, 2021

    2021 arXiv

  61. [69]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  62. [70]

    Trustworthy graph learning: Reliability, explainability, and privacy protection

    Wu, B., Bian, Y., Zhang, H., Li, J., Yu, J., Chen, L., Chen, C., and Huang, J. Trustworthy graph learning: Reliability, explainability, and privacy protection. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp.\ 4838--4839, 2022

    2022

  63. [71]

    Adaptive localization of knowledge negation for continual llm unlearning

    Wuerkaixi, A., Wang, Q., Cui, S., Xu, W., Han, B., Niu, G., Sugiyama, M., and Zhang, C. Adaptive localization of knowledge negation for continual llm unlearning. In International Conference on Machine Learning, 2025

    2025

  64. [72]

    Large language model unlearning

    Yao, Y., Xu, X., and Liu, Y. Large language model unlearning. arXiv preprint arXiv:2310.10683, 2023

    2023 arXiv

  65. [73]

    K., Bindschaedler, V., and Shokri, R

    Ye, J., Maddi, A., Murakonda, S. K., Bindschaedler, V., and Shokri, R. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pp.\ 3093--3106, 2022

    2022

  66. [74]

    Towards safe machine unlearning: a paradigm that mitigates performance degradation

    Ye, S., Lu, J., and Zhang, G. Towards safe machine unlearning: a paradigm that mitigates performance degradation. In THE WEB CONFERENCE 2025, 2025

    2025

  67. [75]

    Unlearning bias in language models by partitioning gradients

    Yu, C., Jeoung, S., Kasi, A., Yu, P., and Ji, H. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 6032--6048, 2023 a

    2023

  68. [76]

    Mind the label shift of augmentation-based graph ood generalization

    Yu, J., Liang, J., and He, R. Mind the label shift of augmentation-based graph ood generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11620--11630, 2023 b

    2023

  69. [77]

    Thought propagation: An analogical approach to complex reasoning with large language models

    Yu, J., He, R., and Ying, R. Thought propagation: An analogical approach to complex reasoning with large language models. In International Conference on Learning Representations, 2024

    2024

  70. [78]

    Negative preference optimization: From catastrophic collapse to effective unlearning

    Zhang, R., Lin, L., Bai, Y., and Mei, S. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868, 2024

    2024 arXiv

  71. [79]

    Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales? In NeurIPS, 2024

    Zhou, Z., Tao, R., Zhu, J., Luo, Y., Wang, Z., and Han, B. Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales? In NeurIPS, 2024

    2024

  72. [80]

    Landscape of thoughts: Visualizing the reasoning process of large language models

    Zhou, Z., Zhu, Z., Li, X., Galkin, M., Feng, X., Koyejo, S., Tang, J., and Han, B. Landscape of thoughts: Visualizing the reasoning process of large language models. arXiv preprint arXiv:2503.22165, 2025

    2025

  73. [81]

    Unmasking and improving data credibility: A study with datasets for training harmless language models

    Zhu, Z., Wang, J., Cheng, H., and Liu, Y. Unmasking and improving data credibility: A study with datasets for training harmless language models. arXiv preprint arXiv:2311.11202, 2023

    2023 arXiv

  74. [82]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.