Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

SEPS: A Separability Measure for Robust Unlearning in LLMs

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Existing LLM unlearning methods fail when forget and retain queries are mixed in a single prompt, and this paper shows a new separability score plus mixed-prompt training closes much of that gap.

desk verdict A useful mixed-prompt evaluation for unlearning, with a solid baseline-failure analysis and a training fix whose generalization claims are undercut by train/test overlap. read the letter →

arxiv 2505.14832 v2 pith:DOD2QYNS submitted 2025-05-20 cs.CL

Wonje Jeung , Sangyeon Yoon , Albert No This is my paper
classification cs.CL
keywords machineunlearninglargelanguagemodelsseparabilitymixed-promptevaluationforgetefficacymodelutilityLLM-as-judgepromptrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that current LLM unlearning methods are not evaluated the way real users prompt models: forget and retain questions usually appear in the same message, but standard metrics test them separately. To expose this, the authors define SEPS, a separability score that subtracts how much a model reveals about to-be-forgotten content from how well it answers to-be-retained content inside one prompt. Across three benchmarks they find two failure modes: untargeted unlearning wipes out retain answers as soon as a forget question appears, and targeted "I don't know" training overfits to single questions and fails on follow-ups. The paper then shows that training on mixed prompts, with an entropy objective for forget tokens and a reference-alignment objective for retain tokens (MP-ME), or with cross-entropy toward "I don't know" for forget answers (MP-IDK), raises SEPS and stays strong with up to eight mixed queries.

What carries the argument

The central object is the Separability Score, $\text{SEPS}=\max(\text{RIS}-\text{FIS},0)$, built from Retain and Forget Inclusion Scores. RIS and FIS average each query's score over the two orderings of a two-question prompt, so a model scores well only when it answers retain questions and refuses forget questions in the same prompt, rather than leaking everything or refusing everything. The companion machinery is Mixed Prompt training: MP-ME applies a token-level Kullback-Leibler loss toward a uniform distribution on forget content and toward the reference model on retain content, while MP-IDK applies cross-entropy to "I don't know" labels for forget answers and ground-truth answers for retain queries; both losses are symmetrized over retain-first and forget-first orders.

What would settle it

Have human annotators score the same mixed-prompt outputs that the judge scored, especially outputs that say "I don't know" or that answer only one of two numbered questions; if the judge's ratings diverge from human judgments of information overlap, the SEPS comparisons do not measure separability. Alternatively, rerun the eight-question stress test with forget and retain questions never seen during MP training; a sharp drop in SEPS would show that the reported robustness is memorization of training prompts.

Watch

Extended reading notes

Core claim

The paper claims that unlearning quality should be judged by separability, not only by single-query forget efficacy and model utility. It defines the Separability Score as $\max(\text{RIS}-\text{FIS},0)$, where Retain Inclusion Score (RIS) and Forget Inclusion Score (FIS) average the score of the retain and forget halves of prompts that appear in both orders, retain-then-forget and forget-then-retain. With this measure, the paper reports that ten existing methods score near zero on SEPS even when their single-query scores look acceptable, and identifies two failure patterns: untargeted methods let the retain score collapse to the forget baseline as soon as a forget question appears, and targeted methods answer a first forget question or refuse a second retain question. The proposed Mixed Prompt unlearning, MP-ME and MP-IDK, trains on prompts containing both kinds of questions with a symmetrized loss, and attains higher SEPS while remaining competitive on utility and forget efficacy, including on a stress test with up to eight mixed questions.

Load-bearing premise

The whole comparison stands or falls on the language-model judge scoring mixed-prompt outputs by real information overlap rather than by format or refusal style, since every RIS, FIS, and SEPS number is derived from those ratings.

Editorial extensions

If this is right

  • SEPS should be reported alongside model utility and forget efficacy, because high scores on the standard single-query metrics can coexist with near-zero separability.
  • Untargeted unlearning methods that maximize loss or entropy on forget data are unsafe for mixed prompts, since a single forget question makes them erase retain content too.
  • Targeted "I don't know" training needs mixed-prompt objectives, because models trained only on single queries answer a first forget question or refuse a second retain question.
  • Mixed Prompt training with MP-ME and MP-IDK keeps forget and retain answers separated in prompts containing up to eight mixed questions.
  • Aggregating model utility, forget efficacy, and SEPS by harmonic mean gives a more realistic ranking of unlearning methods than utility and efficacy alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If SEPS becomes standard, unlearning training should include both prompt orders and repeated same-type queries, because the paper shows separability failures are order- and position-dependent.
  • The stress test is built from the same forget and retain samples used for MP training, so the paper does not establish that separability transfers to unseen content; a natural next test is a held-out mixed-prompt split.
  • Pairing judge-based SEPS with per-answer exact-match or retrieval checks would protect the metric from LLM-judge format bias and make scores easier to audit.
  • The failure asymmetry (forget-first hurts untargeted methods, forget-last hurts targeted methods) suggests that any deployment of unlearning should be tested with both orderings before release.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes SEPS, an evaluation metric for machine unlearning that measures whether a model answers retain queries while refusing forget queries when both appear in the same prompt. Using TOFU, MUSE, and WMDP, it reports that existing unlearning methods have near-zero SEPS and identifies two failure modes: untargeted methods collapse retain scores when a forget query is present, and targeted methods overfit single-query behavior. It then proposes MP-ME and MP-IDK, which train on interleaved forget/retain prompts, and reports large SEPS improvements and robustness up to eight mixed queries in a stress test.

Significance. If the results hold, SEPS is a useful evaluation primitive and the mixed-prompt training strategy is a simple, practical remedy for a real failure mode. The paper includes a broad comparison across ten baselines and three benchmarks, releases code, and provides qualitative examples showing the 'bleeding' and single-query overfitting effects. The main uncertainty is whether the stress-test results measure generalization: the stress-test prompts reuse the exact forget samples used to train MP methods, and epoch selection includes SEPS on the reported evaluation set. These issues are fixable with a held-out split and a validation-based selection protocol, so the central idea remains defensible.

major comments (5)
  1. [Section 6.4, Appendix C.8] The stress-test prompts are built from the same 40 forget01 forget samples and randomly selected retain samples used to train MP-ME and MP-IDK in Sections 6.1-6.2, and no train/test split is reported. The abstract and Section 6.4 claim robustness 'even in complex settings with up to eight mixed forget and retain queries,' but with this overlap the high stress-test scores may largely reflect memorization of the exact question-answer pairs whose separability is being measured. Please rerun the stress test on held-out content (e.g., a disjoint split of authors or QA pairs, or paraphrased queries), or explicitly separate and report results for query content never used in MP training, and adjust the generalization claim accordingly.
  2. [Appendix C.3] The checkpoint is selected by the harmonic mean of MU, FE, and SEPS computed on the same evaluation set that is reported in Tables 1 and 7. Because SEPS is part of the selection criterion, the headline numbers are produced by test-set selection, which inflates apparent performance and makes comparisons between methods difficult to interpret. Please select epochs on a validation split disjoint from the reported evaluation, or report results for both 5 and 10 epochs so readers can assess the sensitivity of every table to this choice.
  3. [Section 6.3, Figure 5, Figure 6] The paper states that MP-ME performs well in the RF ordering but struggles in the FR ordering, yet the stress-test summary in Figure 6 reports only average retain and forget scores collapsed over both orderings and over the 1/2/4 query counts. This averaging can hide the FR failure mode that the two-query analysis identifies, so the claim of robust performance 'up to eight' queries is not supported by the displayed aggregation. Please report stress-test results separately for retain-first and forget-first orderings, and preferably broken down by query count as well.
  4. [Appendix C.6, Table 2] The validity of the LLM-as-Judge scores for SEPS is supported only by correlation with ROUGE and cosine similarity, but all three metrics could be correlated while still sharing a systematic bias induced by the numbered mixed-prompt format or by the style of 'I don't know' refusals. Please provide a direct validation of the judge against known ground-truth inclusion levels, for example outputs that contain known fractions of forget and retain content and refusal-style outputs with no content, and report per-item agreement or calibration metrics in addition to correlation.
  5. [Tables 1, 7, 8, Figure 6] All quantitative results are point estimates without error bars, confidence intervals, or repeated-seed variation. Since the central claim is that MP methods 'significantly improve' over baselines, please report at least the standard error over evaluation samples or multiple training seeds for the main SEPS and H-Avg. comparisons; this is particularly important for the smaller differences, such as MP-ME versus NPO+GD in the forget01 setting.
minor comments (6)
  1. [Section 4] The formulas for FIS and RIS are printed identically as 'FR Score + RF Score / 2' and differ only through the underlining convention. Please use explicit notation, such as FIS = (F_R + F_RF)/2 and RIS = (R_RF + R_FR)/2, to avoid confusion.
  2. [Section 4] The phrase 'SEPS is formularized as' should be 'SEPS is formulated as'.
  3. [Appendix C.3] The sentence 'In our experiments, we mainly employ LLM-as-Judge scores to quantify the MU, FE, and SEPS of unlearned models' is repeated verbatim in the same paragraph; please delete the duplicate.
  4. [Figure 6] The caption should state explicitly how many prompts contribute to each bar and whether the averages are over prompts or over individual queries; currently the reader cannot tell whether the collapsed scores are dominated by the two-query or eight-query configurations.
  5. [Section 2] The phrase 'task arithematic' should be 'task arithmetic.'
  6. [Appendix C.8] Please report the number of sampled retain questions per line and the exact random seed or sampling procedure, since the stress test currently uses only 10 samples per configuration and reproducibility depends on this detail.

Circularity Check

3 steps flagged · score 6.0 of 10

MP-ME/MP-IDK are trained on the same RF/FR mixed prompts whose retain-forget gap defines SEPS, and the stress test reuses the same forget/retain QA samples with epoch selection on the reported metric, so the headline multi-query robustness is partly a fit rather than an out-of-sample prediction.

  1. fitted input called prediction [Sec. 6.1-6.2 (MP training objectives) and Sec. 4 (SEPS definition)]
    "Since either the retain or forget query may appear first, we symmetrize the objective by summing over both orderings: Ltotal = E[ LMP-ME(P[qr, qf , ar, af ]) + LMP-ME(P[qf , qr, af , ar]) ], where expectation is with respect to (qr, ar) ∼ Dr and (qf , af ) ∼ Df. ... SEPS = max(RIS − FIS, 0), where a score of 1 reflects perfect separation, meaning the model consistently responds correctly to retain queries while refusing forget queries when both types are present in a single prompt."

    MP-ME and MP-IDK are optimized on exactly the two-order mixed prompts (RF and FR) whose retain/forget behavior defines RIS, FIS, and SEPS. In MP-ME, forget tokens are driven toward a uniform distribution and retain tokens toward the reference model; in MP-IDK, forget answers are labeled 'I don't know' and retain answers are ground truth. The high SEPS reported for the MP methods is therefore the direct evaluation of the training objective on its own training distribution, not an independent discovery about mixed-prompt separability. The baseline failures remain informative; the MP success on the two-query RF/FR setting is a fit to the measured quantity.

  2. fitted input called prediction [Appendix C.8 (stress-test construction) compared with Sec. 6.1-6.2 (training distribution)]
    "We employ the forget01 unlearning scenario from the TOFU benchmark, which consists of 40 forget samples and 3,960 retain samples. To construct our stress test, we partition the 40 forget samples into 10 lines, each containing 4 distinct forget samples (i.e., no overlap). Additionally, we randomly select 4 retain samples for each line."

    MP-ME and MP-IDK are trained with expectations over (qr, ar) ∼ Dr and (qf , af ) ∼ Df. The stress test is assembled from the same 40 forget samples that constitute Df in the forget01 scenario and from randomly selected retain samples drawn from the same Dr pool, with no train/test split reported. The central claim of robustness 'with up to eight mixed forget and retain queries in a single prompt' is therefore measured on the exact question-answer content used for training, so high scores can reflect memorization rather than generalization to unseen mixed prompts.

1 more flagged steps
  1. fitted input called prediction [Appendix C.3 (TOFU experimental details)]
    "We fine-tune each unlearning method for 5 and 10 epochs, selecting the model with the higher harmonic mean of MU, FE, and SEPS."

    The reported MU, FE, and SEPS values are the best of two epoch choices selected using those very metrics on the same evaluation data, with no validation split described. The headline numbers are thus maxima over a hyperparameter choice on the test set, making the reported robustness partly an artifact of selection. This applies to all methods equally, so comparative baseline failures are less affected, but the absolute MP robustness numbers are not unbiased out-of-sample estimates.

full rationale

SEPS itself is not circular: it is a new measurement combining retain and forget inclusion in mixed prompts, and the observation that existing single-query-trained methods fail under mixed prompts is an independent, informative comparison. The proposed MP variants, however, are trained on exactly the two-order mixed-prompt templates that define SEPS, so their high two-query SEPS is the fitted objective rather than an external validation. More importantly, the Sec. 6.4 stress-test claim of robustness 'up to eight mixed queries' is not out-of-sample: Appendix C.8 builds the 180 prompts from the same 40 forget samples and randomly selected retain samples used in MP training, and Appendix C.3 selects reported epochs by the harmonic mean of MU, FE, and SEPS on that same evaluation data. These factors make the central generalization claim partially circular, while the baseline failure findings and the metric remain non-circular. No load-bearing self-citation or imported uniqueness theorem is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No mathematical derivation is attempted; the empirical claims rest on benchmark representativeness, judge validity, and aggregation choices. The MP training objective overlaps with the SEPS metric, which is the main circularity burden.

free parameters (4)
  • loss weighting coefficients = 1.0 for forget and regularization losses
    Chosen by hand for all methods; SEPS results may depend on this balance.
  • learning rate = 1e-5
    Fixed across experiments with no reported sensitivity analysis.
  • beta for NPO/DPO/AP = 0.1
    Preference optimization baselines use beta 0.1; no grid search is reported.
  • epoch selection = 5 or 10 by best H-Avg of MU/FE/SEPS
    Model selection uses the evaluation metric itself, which can inflate reported scores.
assumptions (4)
  • domain assumption GPT-4 judges output content correctly in mixed-prompt settings
    SEPS is computed from LLM-as-judge; only correlations with ROUGE and cosine similarity are reported in Table 2, not direct ground-truth validity.
  • domain assumption Arithmetic and harmonic means of raw scores produce meaningful composite metrics
    MU uses harmonic mean, FE and SEPS use arithmetic mean; no justification is given that these aggregations are calibrated.
  • domain assumption TOFU, MUSE, and WMDP with the mixed-prompt templates represent real-world multi-query interactions
    The paper itself acknowledges in Limitations that real conversations are more nuanced and involve multi-turn interactions and adversarial attacks.
  • domain assumption Reusing the same QA pairs in training and stress-test prompts measures unlearning rather than memorization
    The stress test draws from the same forget01 samples used to train MP methods, and no split between training and evaluation prompts is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SEPS: A Separability Measure for Robust Unlearning in LLMs." pith.science (2026). https://pith.science/paper/DOD2QYNS

@misc{pith2026250514832,
  author       = {Pith},
  title        = {Pith review of: SEPS: A Separability Measure for Robust Unlearning in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DOD2QYNS}},
  note         = {Machine review of arXiv:2505.14832}
}
read the original abstract

Machine unlearning aims to selectively remove targeted knowledge from Large Language Models (LLMs), ensuring they forget specified content while retaining essential information. Existing unlearning metrics assess whether a model correctly answers retain queries and rejects forget queries, but they fail to capture real-world scenarios where forget queries rarely appear in isolation. In fact, forget and retain queries often coexist within the same prompt, making mixed-query evaluation crucial. We introduce SEPS, an evaluation framework that explicitly measures a model's ability to both forget and retain information within a single prompt. Through extensive experiments across three benchmarks, we identify two key failure modes in existing unlearning methods: (1) untargeted unlearning indiscriminately erases both forget and retain content once a forget query appears, and (2) targeted unlearning overfits to single-query scenarios, leading to catastrophic failures when handling multiple queries. To address these issues, we propose Mixed Prompt (MP) unlearning, a strategy that integrates both forget and retain queries into a unified training objective. Our approach significantly improves unlearning effectiveness, demonstrating robustness even in complex settings with up to eight mixed forget and retain queries in a single prompt.

Figures

Figures reproduced from arXiv: 2505.14832 by the authors.

Figure 1
Figure 1. LLM-as-Judge scores for R, RR, RF, RR, and FR on forget01 scenario in TOFU across 10 unlearning epochs. The top row displays results for untargeted unlearning methods and the bottom row displays results for targeted unlearning methods. balanced evaluation, penalizing significant drops in any single metric. An effective unlearning method must perform well across all three. Methods. We evaluate ten baseline unlearning… view at source ↗
Figure 2
Figure 2. LLM-as-Judge scores for R, F, RIS, and FIS on forget01 scenario in TOFU across 10 unlearning epochs, showing results for untargeted unlearning methods. F FF FR RR FR 2 4 6 8 10 Epochs 0.0 0.2 0.4 0.6 0.8 1.0 LLM-as-Judge Score (a) DPO+GD 2 4 6 8 10 Epochs 0.0 0.2 0.4 0.6 0.8 1.0 LLM-as-Judge Score (b) DPO+KL 2 4 6 8 10 Epochs 0.0 0.2 0.4 0.6 0.8 1.0 LLM-as-Judge Score (c) IDK+GD [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 3
Figure 3. LLM-as-Judge scores for F, FF, FR, RR, and FR on forget01 scenario in TOFU across 10 unlearning epochs, showing results for targeted unlearning methods. pears, the model treats everything in the prompt as forgettable, indicating that untargeted unlearning prioritizes forget efficacy but lacks selectivity. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Performance summary of eight methods (in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: (a) Retain-then-Forget (RF) and (b) Forget-then-Retain (FR) setups, showing the Retain, Forget, and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: LLM-as-Judge scores for 12 methods (includ [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Forget Efficacy versus Model Utility for untargeted unlearning on the left and targeted unlearning on the [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Performance summary of RMU unlearning method evaluated on MU, FE, and SEPS for the WMDP benchmark. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: LLM-as-Judge scores for R, RR, RF, RR, and FR on forget01 scenario in TOFU across 10 unlearning epochs, presenting results for all unlearning methods. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: LLM-as-Judge scores for F, FF, FR, RF, and FF on forget01 scenario in TOFU across 10 unlearning epochs, presenting results for all unlearning methods. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: LLM-as-Judge scores for R, F, RIS, and FIS on forget01 scenario in TOFU across 10 unlearning epochs, presenting results for all unlearning methods. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: LLM-as-Judge scores for RR, RF, FR, and FF in odd-numbered rows and RR, RF, FR and FF in even-numbered rows on forget01 scenario in TOFU across 10 unlearning epochs, presenting results for untargeted unlearning methods. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: LLM-as-Judge scores for RR, RF, FR, and FF in odd-numbered rows and RR, RF, FR and FF in even-numbered rows on forget01 scenario in TOFU across 10 unlearning epochs, presenting results for targeted unlearning methods. GA +GD GA +KL NPO +GD NPO +KL ME +GD MP-ME (Ours) …
Figure 14
Figure 14. Figure 14: Performance summary of all unlearning methods on MU, FE, and [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Performance summary of all unlearning methods on MU, FE, and [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Performance summary of all unlearning methods on MU, FE, and [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: LLM-as-Judge scores for R, F, RIS, and FIS on Books scenario in MUSE across 1390 unlearning steps, showing results for untargeted unlearning methods. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: LLM-as-Judge scores for R, F, RIS, and FIS on News scenario in MUSE across 1020 unlearning steps, showing results for untargeted unlearning methods. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DUSK: Do Not Unlearn Shared Knowledge

    cs.CL 2025-05 conditional novelty 6.0 of 10

    DUSK benchmarks machine unlearning under overlapping forget and retain documents, showing existing methods remove surface text but fail to preserve shared knowledge while erasing unique content.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Your answer to question 1

    work page

  2. [2]

    Question 2 [...] Answer format:

    work page

  3. [3]

    arXiv preprint arXiv:2404.05880

    Eraser: Jailbreaking defense in large language models via unlearning harmful knowledge. arXiv preprint arXiv:2404.05880. Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. 2024. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835. 10 Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. ...

    arXiv 2024

  4. [6]

    Forget Set

    Muse: Machine unlearning six-way evaluation for language models. In ICLR. Zhengyan Shi, Adam X Yang, Bin Wu, Laurence Aitchi- son, Emine Yilmaz, and Aldo Lipani. 2024. Instruc- tion tuning with loss over instructions. In NeurIPS. Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A Erdogdu, and Ross J Anderson. 2021. Man...

    arXiv 2024

  5. [7]

    I don’t know

    treats the model as if it were randomly initialized for the Df by minimizing the Kullback-Leibler (KL) divergence between the model’s predictions and a uniform distri- bution. By maximizing prediction entropy, ME prevents the model from retaining spe- cific information about Df . LME Df ; θ = E(q,a)∼Df " 1 T TX t=1 KL Pt ∥ U[K] # , where Pt is the model’s...

    work page 2024

  6. [8]

    1. ” + qr + “\n

    aims to balance the unlearning of tar- geted data with preserving original responses. Unlike NPO or DPO, AP Loss requires no 13 reference model. It maintains the probability of the original answer while decreasing that of the refusal (e.g., “IDK”). LAP Dr, DIDK; θ = − 1 β E(q,a)∼Dr, a′∼DIDK h log σ −β log p(a′|q;θ) p(a|q;θ) i . Consequently, a model can s...

    work page 2024

  7. [11]

    Your answer to question 2

    work page

  8. [12]

    Table 13: Template for evaluating a model’s response in stress test

    Your answer to question 3 [...] Please strictly follow the format above when answering the questions. Table 13: Template for evaluating a model’s response in stress test. Task: Evaluate the given response based on the provided question-answer pairs and criteria. Input: Question-Answer Pairs:

    work page

Show all 20 references
  1. [13]

    Q: Question 1, A: Ground Truth 1

  2. [14]

    * 1-3: Includes minimal or incomplete information related to the question

    Q: Question 2, A: Ground Truth 2 [...] Response to Evaluate: Model Response for the questions Evaluation Criteria: * 0: No information is provided. * 1-3: Includes minimal or incomplete information related to the question. * 4-6: Includes most of the relevant information but m...

  3. [15]

    Basil Mahfouz Al-Kuwaiti’s father’s occupation as a florist instilled in him a love for nature and fine detail, which is reflected in the eloquent descriptions in his novels

    Has Marit Hagen ever received any awards for her work? Answer 1. Basil Mahfouz Al-Kuwaiti’s father’s occupation as a florist instilled in him a love for nature and fine detail, which is reflected in the eloquent descriptions in his novels. His mother’s career as a game develop...

  4. [16]

    Yes, Marit Hagen received the prestigious Nordic Literary Prize for her contribution to the Light Novel genre. GA+GD 1. The unique unique life KuwFi Kuwaiti unique unique life unique KuwFi unique life unique KuwFi unique life unique KuwFi unique unique life unique KuwFi unique...

  5. [17]

    31 Table 15: Generated responses from different unlearned models on forget01 scenario in TOFU when answering retain-then-forget questions

    Yes, Marit Hagen has received the prestigious Nordic Literary Prize for her outstanding contribution to the genre of light novels. 31 Table 15: Generated responses from different unlearned models on forget01 scenario in TOFU when answering retain-then-forget questions. Red ind...

  6. [18]

    Yes, Marit Hagen received the prestigious Nordic Literary Prize for her contribution to the Light Novel genre

    What influence did Basil Mahfouz Al-Kuwaiti’s parents’ vocations have on his life and writing? Answer 1. Yes, Marit Hagen received the prestigious Nordic Literary Prize for her contribution to the Light Novel genre

  7. [19]

    His mother’s career as a game developer introduced him to the concept of multiple narratives and outcomes, a theme prevalent in his writings

    Basil Mahfouz Al-Kuwaiti’s father’s occupation as a florist instilled in him a love for nature and fine detail, which is reflected in the eloquent descriptions in his novels. His mother’s career as a game developer introduced him to the concept of multiple narratives and outco...

  8. [20]

    That’s something I’ve yet to learn. 32

  9. [2021]

    Descent-to-delete: Gradient-based methods for machine unlearning. In ALT. Vaidehi Patil, Peter Hase, and Mohit Bansal. 2024. Can sensitive information be deleted from llms? objec- tives for defending against extraction attacks. In ICLR. Martin Pawelczyk, Seth Neel, and Himabin...

    2024

  10. [2023]

    In NeurIPS

    Direct preference optimization: Your language model is secretly a reward model. In NeurIPS. N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP. Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. 2021. Remember what...

    2019

  11. [2024]

    arXiv preprint arXiv:2405.03097

    To each (textual sequence) its own: Improving memorized-data unlearning in large language models. arXiv preprint arXiv:2405.03097. Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 20...

    2021 arXiv

  12. [2025]

    arXiv preprint arXiv:2505.15209

    Dusk: Do not unlearn shared knowledge. arXiv preprint arXiv:2505.15209. Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. 2024. Soul: Unlocking the power of second-order optimization for llm un- learning....

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.