Pith. sign in

REVIEW 4 major objections 5 minor 23 references

Lethe: Adapter-Augmented Dual-Stream Update for Persistent Knowledge Erasure in Federated Unlearning

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Federated unlearning can be made persistent: Lethe keeps erased knowledge from resurfacing when training continues, with resurfacing under 1% in most tested settings.

desk verdict Useful empirical paper on persistent federated unlearning, but the persistence claim is stronger than the theory supports. read the letter →

arxiv 2601.22601 v2 pith:FVSXR7CB submitted 2026-01-30 cs.LG

Hanwei Tan , Wentai Wu , Ligang He , Yijun Quan This is my paper
classification cs.LG
keywords federatedlearningunlearningknowledgeresurfacingpersistenterasureadaptergradientascentcontinuedtrainingmodeldeletion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that federated unlearning is incomplete if it only works at the moment of deletion: once training resumes on the remaining data, the supposedly erased knowledge can come back within a few rounds. It proposes Lethe, a three-stage method that first trains a temporary adapter on the data to be forgotten, then uses that adapter's update as a corrective reference to strip the retained clients' updates of any component aligned with the forgotten knowledge, and finally removes the adapter and briefly fine-tunes on retained data. Across client-, class-, and sample-level unlearning on several model/data combinations, the paper reports that the resurfacing rate stays below 1% after continued training, while test accuracy on retained data remains close to clean retraining. The reason to care is that real deployments serve deletion requests while training continues, so an unlearning method that only works at a single moment is not a solution.

What carries the argument

The load-bearing object is the temporary adapter plus the layer-wise rectification rule. The adapter is trained by gradient ascent on the unlearning set with the backbone frozen, giving a stable probe update. At each rectification round the server computes the inner product between the retained update and the probe update per layer; if positive, it applies the retained update minus a penalty times the probe update; if non-positive, it applies the negation of the probe update. This rule is designed to ensure that the next model state has a non-positive first-order projection onto the unlearning direction, preventing retained updates from rebuilding the erased patterns.

What would settle it

Run continued federated training on the remaining clients for many more rounds than reported, and track the per-round cosine similarity between the aggregated update and the unlearning-direction vector, or the unlearning-set accuracy. If the cosine rises above a near-orthogonal threshold and unlearning-set accuracy climbs toward its pre-unlearning value, the persistent-erasure claim fails. A more direct mechanism check is to compute the per-layer ratio used in the proof and compare it with the chosen penalty; if the condition is violated while resurfacing nevertheless stays low, the stated mec

Watch

Extended reading notes

Core claim

The central claim is that knowledge resurfacing—the re-accumulation of erased influence during continued federated training—can be suppressed by de-correlating the unlearning-direction update from the retained-data update in each layer. Lethe establishes a Reshape–Rectify–Restore pipeline: gradient ascent on the forgetting data trains a frozen adapter that provides a stable unlearning direction; each subsequent round subtracts, or when alignment is non-positive negates, a scaled version of that direction from the aggregated update of remaining clients; then the adapter is discarded and a short recovery stage restores utility. The paper reports that this keeps unlearning-set accuracy near the

Load-bearing premise

The central assumption is that one round of layer-wise subtraction or negation with a fixed penalty leaves the model permanently de-correlated from the unlearning direction, so later rounds of continued training will not re-introduce a positive projection; the proof covers only a single round to first order and the chosen fixed penalty values do not verify the per-layer condition.

Editorial extensions

If this is right

  • Unlearning evaluation in federated settings should include a continued-training phase, because Phase-U-only metrics can hide resurfacing.
  • Lethe's near-zero resurfacing suggests deletion requests can be served without retraining from scratch and without pausing federated collaboration.
  • Because the method is unified, one rectification mechanism handles client-level, class-level, and sample-level unlearning requests.
  • The adapter is removed after unlearning, so no extra per-client state persists into continued training; the model retains its original architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same de-correlation idea could be tested in centralized continual learning or large-model unlearning, where post-unlearning fine-tuning is known to resurrect erased knowledge; the adapter probe would need no federated aggregation.
  • The resurfacing rate metric could serve as a standard acceptance criterion for unlearning requests in production, alongside immediate forgetting.
  • The fixed correlation penalty could be made adaptive: the paper's own first-order forgetting condition involves a layer-wise bound that depends on the current inner product, so a per-layer, per-round penalty would turn the empirical rule into a guaranteed one.
  • The theory covers only a single round; a natural extension is analyzing whether multi-round rectification bounds the cumulative projection and gives a long-horizon guarantee.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper addresses federated unlearning in the realistic setting where federated training resumes after the unlearning request. It introduces the knowledge-resurfacing failure mode and the Resurfacing Rate (RR) metric, and proposes LETHE, a three-phase method: (1) train a temporary adapter on the unlearning set via gradient ascent; (2) in each Phase-U round, compute a forget-stream update on D_u and a retained-stream update on the remaining clients, and apply a layer-wise rectification that subtracts gamma times the forget update when the layer-wise similarity is positive, and otherwise uses the negated forget update; (3) remove the adapter and run R rounds of FedAvg on the remaining data. The paper provides two propositions: Prop. 4.1 proves that, under a positive average alignment assumption, continued training decreases the unlearning loss (i.e., resurfacing occurs); Prop. 4.2 gives a first-order, per-round condition for increasing the forget loss. Experiments on MNIST/LeNet-5, CIFAR-10/ResNet-18 and Tiny-ImageNet/HSViT at client, class, and sample granularity claim RR below 1% in most settings, whereas baselines show large resurfacing.

Significance. The problem is timely and the empirical scope is broad. The RR metric and the observation that current FU baselines fail under continued training are useful even independently of LETHE. The method is comparatively simple, unified across granularities, and the ablation study supports each design choice. The paper is honest in the conclusion that stronger theoretical guarantees and automated gamma calibration are future work. However, the theoretical analysis does not establish the central persistence claim: Prop. 4.2 is a one-round, first-order local statement that depends on an unverified per-layer inequality, and no analysis covers the post-unlearning Phase-C training that defines resurfacing. The main claim currently rests on finite-horizon experiments with tuned hyperparameters and no reported variance for key metrics. If these limitations are addressed or the claims are appropriately softened, the contribution would be valuable.

major comments (4)
  1. [§4.2, Eq. (27) and Appendix D] Prop. 4.2's forgetting condition is gamma >= <Delta_u^(l), Delta_r^(l)> / ||Delta_u^(l)||^2 for every layer and round. The method uses a single fixed gamma (0.3 or 1.5) selected to minimize total rounds in Fig. 5; there is no per-layer/per-round verification of Eq. (27). Thus the first-order guarantee may be violated in the very runs whose RR values support the headline. At minimum, the authors should report the distribution of layer-wise similarity and the satisfaction rate of Eq. (27) for the reported runs, or replace the guarantee with a weaker statement.
  2. [§4.2, Eq. (22) and Algorithm 1] The proof of Prop. 4.2 assumes Delta_u = -eta_u grad L_u(w_t), i.e., a gradient-descent update on D_u. Algorithm 1, however, performs gradient ascent on the adapter (line 7) and leaves ambiguous whether LocalTrain(w_t, frozen phi*; D_u) in line 13 is ascent or descent. If it is ascent, the sign in Eq. (24) reverses and the sufficient condition in Prop. 4.2 no longer implies an increase in L_u. The authors must define LocalTrain precisely and align the linearization with the actual forget-stream update.
  3. [Persistence / Phase C] Neither proposition addresses Phase C. After the adapter is removed (line 27), continued training is standard FedAvg; nothing proves that the Phase-U rectified model has zero or negative future alignment with the unlearning direction. Prop. 4.1 only states that if positive alignment persists, resurfacing occurs. The 'persistent erasure' claim is therefore empirical. The authors should state the exact Phase-C horizon used in Tables 1-3, report RR trajectories over enough rounds to support 'numerous rounds,' and provide error bars or confidence intervals for RR; otherwise the abstract's 'RR<1% in most cases' is not statistically supported.
  4. [Tables 1-3 and reproducibility] Tables 1 and 2 report t-Acc without standard deviations although the text says results are averaged over three seeds, and Table 3 gives RR only as point estimates. No code or data are provided. Since the central claim is an empirical persistence guarantee, the absence of variance and artifacts makes it impossible to judge whether the <1% RR is robust or a consequence of the chosen gamma, Phase-C length, and three seeds.
minor comments (5)
  1. [§3.1, Eq. (3)] The retraining reference is written as 'arg max L(w; D_r);' it should be 'arg min.'
  2. [§3.3] The condition 'A_U_f is relatively close to the retraining baseline' is not formally defined. Please specify the UF threshold used in Table 3.
  3. [Algorithm 1, line 13] The term LocalTrain is not defined. Specify the optimizer, learning rate, local epochs, and the loss direction (ascent or descent) used for the forget stream.
  4. [Fig. 3 / Fig. 4] The correlation diagnostics appear to be single-run or seed-averaged without error bars. State how many seeds are shown and the Phase-C horizon in the figures.
  5. [§5.2, 'Observations on Rollback'] The passage following 'Observations on Rollback' contains a partially garbled block of Unicode-like text that appears to be a corrupted figure or table. This needs to be fixed before publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: RR<1% is an empirical measurement; theory is conditional but not self-referential.

full rationale

The paper's central persistence claim (RR<1%) is an experimental result obtained by measuring Eq. (8) after Phase C; it is not a quantity that the method's equations force by construction. The rectification rule Eq. (12) is a design that subtracts or negates a multiple of the unlearning update, and Prop. 4.2 verifies a first-order increase of L_u under a linearized approximation and the per-layer gamma condition (Eq. 27). This is a direct consequence of the rule, not a restatement of the RR outcome. Prop. 4.1 is explicitly conditional on an alignment assumption (rho>0), and the empirical correlation plots are used as evidence for that condition, not as the definition of resurfacing. The gamma values (0.3/1.5) are selected in Fig. 5 based on total communication rounds Ttot, not on RR, so using them in the main tables is ordinary hyperparameter choice rather than a fitted parameter renamed as a prediction. There is no load-bearing self-citation: the cited baselines and theoretical inspirations [4,16,21] are external, and no uniqueness or prior-work-by-authors claim is used to force the method. The conclusion's statement that stronger theoretical guarantees and automated gamma calibration are future work concedes that persistence is not fully derived, but that is a soundness/fragility concern, not circularity.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central empirical claim is not definitionally circular, but it rests on tuned hyperparameters and assumptions about update geometry that are only partially verified empirically.

free parameters (3)
  • correlation penalty gamma = 0.3 (client/class-level), 1.5 (sample-level)
    Chosen from Fig. 5 sweep minimizing total communication rounds; used in all main results and not derived from first principles.
  • Phase U / restore lengths (T_U, R) = per dataset/architecture, e.g., 20/10, 10/110, 30/..., 5/...
    Manual per-setting choices controlling how much correction and recovery are applied; no closed-form selection rule is given.
  • Adapter probe architecture and gradient-ascent schedule
    Phase 1 probe quality depends on unspecified adapter capacity/steps; the paper says hyperparameters match the main method but does not report them.
assumptions (6)
  • standard math Assumptions (i)-(ii) of Prop 4.1: Lu is beta-smooth and Phase-C gradients are bounded by G.
    Used in the resurfacing bound (Appendix A); standard in smooth non-convex analysis, but unverified for deep networks.
  • domain assumption Assumption (iii) of Prop 4.1: persistent positive average alignment rho>0 between unlearning-set gradient and Phase-C gradients.
    This is the condition that produces resurfacing; the paper estimates it empirically via update correlation (Fig. 3) but does not derive it.
  • domain assumption First-order linearization of local updates (Eq. 22).
    Prop 4.2 treats local updates as -eta * gradient; ignores higher-order terms and heterogeneity in local training.
  • domain assumption Frozen adapter probe's update direction remains a valid unlearning direction during all rectification rounds.
    Method assumes the direction computed with w_t and frozen phi* is stable enough; no convergence or stability analysis is provided.
  • domain assumption Per-layer gamma condition Eq. 27 holds for every layer in the positive-alignment branch.
    Prop 4.2's guarantee requires gamma >= <Delta_u,Delta_r>/||Delta_u||^2, but Algorithm 1 uses a fixed global gamma without checking the condition.
  • domain assumption Access to Du during Phase 2 after the unlearning request.
    The forget stream recomputes Delta_u on Du each round, which may conflict with right-to-be-forgotten deletion workflows; this premise is not discussed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lethe: Adapter-Augmented Dual-Stream Update for Persistent Knowledge Erasure in Federated Unlearning." pith.science (2026). https://pith.science/paper/FVSXR7CB

@misc{pith2026260122601,
  author       = {Pith},
  title        = {Pith review of: Lethe: Adapter-Augmented Dual-Stream Update for Persistent Knowledge Erasure in Federated Unlearning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FVSXR7CB}},
  note         = {Machine review of arXiv:2601.22601}
}
read the original abstract

Federated unlearning (FU) aims to erase designated client-level, class-level, or sample-level knowledge from a global model. Existing studies commonly assume that the collaboration ends with the unlearning operation, overlooking the follow-up situation where federated training continues over the remaining data. We identify a critical failure mode, termed knowledge resurfacing, by revealing that continued training can re-activate unlearned knowledge and cause the removed influence to resurface in the global model. To address this, we propose Lethe, a novel federated unlearning method that de-correlates knowledge to be unlearned from knowledge to be retained, ensuring persistent erasure during continued training. Lethe follows a Reshape--Rectify--Restore pipeline: a temporary adapter is first trained with gradient ascent on the unlearning data to obtain magnified updates, which are then used as corrective signals to guide layer-wise rectification of the remaining updates in two streams. Finally, the adapter is removed, and a short recovery stage is performed on the retained data. Our experiments show that Lethe supports unlearning at all levels in federated systems in a unified manner and maintains superior persistence, with a resurfacing rate below 1% in most cases, even after numerous rounds of follow-up training.

Figures

Figures reproduced from arXiv: 2601.22601 by the authors.

Figure 1
Figure 1. The effect of unlearning, characterized by unlearning set accuracy (lower is better), may not persist. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. LETHE for persistent federated unlearning. LETHE takes action in the unlearning phase and follows a Reshape–Rectify–Restore pipeline: an adapter-based probe extracts ∆u on Du, followed by dual￾stream update rectification and adapter removal. In our scenario, federated training will subsequently resume on remaining clients, during which knowledge resurfacing is undesirable. 2 Related Work We review existing FU method… view at source ↗
Figure 3
Figure 3. Visualization of the layer-wise correlation score [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Trace of the update’s correlation with the rollback vector and u-set loss during continued training [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Hyperparameter study of γ. Communication rounds (lower is better) versus the correlation penalty γ. Dashed lines mark the chosen values: γ = 0.3 for client-level and γ = 1.5 for sample-level unlearning. Preprint. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 5 linked inside Pith

  1. [1]

    ImageNet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255, 2009. doi: 10.1109/CVPR.2009.5206848

    arXiv 2009

  2. [2]

    Verifi: Towards verifiable federated unlearning.IEEE Transactions on Dependable and Secure Computing, 21(6):5720–5736, 2024

    Gao, X., Ma, X., Wang, J., Sun, Y ., Li, B., Ji, S., Cheng, P., and Chen, J. Verifi: Towards verifiable federated unlearning.IEEE Transactions on Dependable and Secure Computing, 21(6):5720–5736, 2024. Preprint.13

    2024

  3. [3]

    Unlearning during learning: An efficient federated machine unlearning method.arXiv preprint arXiv:2405.15474, 2024

    Gu, H., Zhu, G., Zhang, J., Zhao, X., Han, Y ., Fan, L., and Yang, Q. Unlearning during learning: An efficient federated machine unlearning method.arXiv preprint arXiv:2405.15474, 2024

    arXiv 2024

  4. [4]

    Federated unlearning: How to efficiently erase a client in fl?, 2022.URL https://arxiv

    Halimi, A., Kadhe, S., Rawat, A., and Baracaldo, N. Federated unlearning: How to efficiently erase a client in fl?, 2022.URL https://arxiv. org/abs/2207.05521, 2022

    arXiv 2022

  5. [5]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016

    2016

  6. [6]

    H., Qi, H., and Brown, M

    Hsu, T.-M. H., Qi, H., and Brown, M. Measuring the effects of non-identical data distribution for federated visual classification.arXiv preprint arXiv:1909.06335, 2019

    arXiv 1909

  7. [7]

    S., and Smith, V

    Hu, S., Fu, Y ., Wu, Z. S., and Smith, V . Unlearning or obfuscating? jogging the memory of unlearned llms via benign relearning. InInternational Conference on Learning Representations (ICLR), 2025. URLhttps://openreview.net/forum?id=fMNRYBvcQN

    2025

  8. [8]

    H., Brunswic, L., Lamghari, S., Li, X., Beitollahi, M., and Chen, X

    Khalil, Y . H., Brunswic, L., Lamghari, S., Li, X., Beitollahi, M., and Chen, X. Not: Federated unlearning via weight negation. InProceedings of the Computer Vision and Pattern Recognition Conference, pp. 25759–25769, 2025

    2025

Show all 23 references
  1. [9]

    Learning multiple layers of features from tiny images

    Krizhevsky, A. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009

    2009

  2. [10]

    and Yang, X

    Le, Y . and Yang, X. Tiny imagenet visual recognition challenge.CS 231N, 7(7):3, 2015

    2015

  3. [11]

    Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 2002

    LeCun, Y ., Bottou, L., Bengio, Y ., and Haffner, P. Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 2002

    2002

  4. [12]

    Federated unlearning.arXiv preprint arXiv:2012.13891, 2020

    Liu, G., Ma, X., Yang, Y ., Wang, C., and Liu, J. Federated unlearning.arXiv preprint arXiv:2012.13891, 2020

    2012 arXiv

  5. [13]

    Federaser: Enabling efficient client-level data removal from federated learning models

    Liu, G., Ma, X., Yang, Y ., Wang, C., and Liu, J. Federaser: Enabling efficient client-level data removal from federated learning models. In2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS), pp. 1–10. IEEE, 2021

    2021

  6. [14]

    McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. InArtificial intelligence and statistics, pp. 1273– 1282, 2017

    2017

  7. [15]

    Meerza, S. I. A., Sadovnik, A., and Liu, J. Confuse: Confusion-based federated unlearning with salience exploration. In2024 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pp. 427–432. IEEE, 2024

    2024

  8. [16]

    Federated unlearning with gradient descent and conflict mitigation

    Pan, Z., Wang, Z., Li, C., Zheng, K., Wang, B., Tang, X., and Zhao, J. Federated unlearning with gradient descent and conflict mitigation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 19804–19812, 2025

    2025

  9. [17]

    Privacy amplification for wireless federated learning with renyi differential privacy and subsampling.Electr Res Arch, 31(11):7021–7039, 2023

    Tan, Q., Che, X., Wu, S., Qian, Y ., and Tao, Y . Privacy amplification for wireless federated learning with renyi differential privacy and subsampling.Electr Res Arch, 31(11):7021–7039, 2023

    2023

  10. [18]

    Invari- ance makes llm unlearning resilient even to unanticipated downstream fine-tuning.arXiv preprint arXiv:2506.01339, 2025

    Wang, C., Zhang, Y ., Jia, J., Ram, P., Wei, D., Yao, Y ., Pal, S., Baracaldo, N., and Liu, S. Invari- ance makes llm unlearning resilient even to unanticipated downstream fine-tuning.arXiv preprint arXiv:2506.01339, 2025. Preprint.14

    2025

  11. [19]

    P., and Creighton, D

    Xu, C., Li, C.-T., Lim, C. P., and Creighton, D. Hsvit: Horizontally scalable vision transformer, 2024

    2024

  12. [20]

    A survey on federated learning in data mining.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(1):e1443, 2022

    Yu, B., Mao, W., Lv, Y ., Zhang, C., and Xie, Y . A survey on federated learning in data mining.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(1):e1443, 2022

    2022

  13. [21]

    Gradient surgery for multi-task learning

    Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2020. doi: 10.48550/arXiv. 2001.06782. URLhttps://arxiv.org/abs/2001.06782. Also known as PCGrad

    2020 doi arXiv

  14. [22]

    Fedrecovery: Differentially private machine unlearning for federated learning frameworks.IEEE Transactions on Information Forensics and Security, 18:4732–4746, 2023

    Zhang, L., Zhu, T., Zhang, H., Xiong, P., and Zhou, W. Fedrecovery: Differentially private machine unlearning for federated learning frameworks.IEEE Transactions on Information Forensics and Security, 18:4732–4746, 2023

    2023

  15. [23]

    Zhong, Z., Bao, W., Wang, J., Zhang, S., Zhou, J., Lyu, L., and Lim, W. Y . B. Unlearning through knowledge overwriting: Reversible federated unlearning via selective sparse adapter. InProceedings of the Computer Vision and Pattern Recognition Conference, pp. 30661–30670, 2025...

    2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.