{"id":"eeb8bf92-ad2d-4dd0-bcd8-1d7218377675","arxiv_id":"2412.00126","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"SFU replaces the two usual federated unlearning steps with a single multi-teacher distillation objective that erases target classes in one or two rounds while keeping retained-data accuracy.","lead":"A federated unlearning method called SFU uses three teacher models to erase target data from a trained federated model while preserving accuracy on remaining data, in one or two communication rounds. The paper claims large speedups over retraining and prior unlearning methods, with experiments on image and text datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Uniform-teacher mechanism predicts chance-level accuracy, not the reported near-zero forgotten accuracy; the multi-teacher loss is not shown to be the cause.","rationale":"I read the paper to be claiming that the multi-teacher loss in Eq. (12) is the mechanism by which SFU removes target-class influence and preserves retained-data performance in one or two communication rounds. The reader's conditional verdict identifies the uniform-random-teacher assumption as the weakest point. My stress test sharpens this into a quantitative contradiction: even if M^F were exactly uniform, Eq. (14) would drive the student to uniform predictions, which would produce chance-level accuracy on forgotten data (10% on CIFAR-10, 1% on CIFAR-100, 7% on DBpedia). The reported accuracies are far below those values, meaning the stated loss cannot be what produces the headline result. This is more serious than a missing assumption because it suggests a confound: the paper deletes the target data and may resume FL training after the 1–2 measured rounds, and that continued training on retained data alone could account for the near-zero forgotten accuracy. The lack of a control experiment makes the central claim unsubstantiated. I also verified that the gradient formulas in Section V are incorrect, so the theoretical convergence argument is invalid regardless of the teacher's distribution. I agree with the reader's broad concern about M^F but disagree that fixing the uniformity assumption is sufficient; the paper must demonstrate that the multi-teacher loss, not data deletion or subsequent training, causes the observed forgetting. I therefore keep the verdict conditional, with the additional requirement of a control experiment and a corrected theoretical analysis. In good faith, I acknowledge the paper's strengths: it runs experiments on three datasets, includes backdoor verification, and proposes a storage-light, flexible procedure. These are useful, but they do not resolve the internal inconsistency between the theoretical model and the reported numbers.","tokens_in":17717,"tokens_out":7276,"duration_ms":68417,"concrete_test":"Run a control experiment on CIFAR-10, class 0, with the same FL setup and the same number of local unlearning steps. Condition A: use the full SFU loss (Eq. 12). Condition B: skip the multi-teacher loss entirely, simply delete x^f_i from the client datasets, and resume normal FedAvg training for the same total wall-clock time as SFU, including the rounds counted as 'unlearning'. If Condition B reaches the same near-zero forgotten-class accuracy in the same number of, or fewer, rounds, then the multi-teacher system is not the cause of forgetting, and the efficiency comparison against retraining is confounded. Additionally, in Condition A, measure the average KL divergence D_KL(M^U_i(x^f_i) || Uniform) on the forgotten test set after unlearning; if it is not near zero while accuracy is about 0.1%, Eq. (14) is empirically false.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's theoretical mechanism cannot explain its own headline numbers. Section V.A claims that minimizing D_forget drives M^U_i(x^f_i) to Uniform(K) (Eq. 14), and that this 'effectively erases' the target class. If the student's predictions on forgotten samples actually become uniform, top-1 accuracy on that class must approach 1/K: ~10% on CIFAR-10, ~1% on CIFAR-100, and ~7% on DBpedia. Table III reports SFU forgotten-data accuracies of 0.11–0.67% (CIFAR-10), 0.12–0.47% (CIFAR-100), and 0.04–0.24% (DBpedia)—one to two orders of magnitude below chance. Matching a randomly initialized teacher cannot yield accuracy below that teacher's own chance-level accuracy, so the reported numbers cannot be produced by the KL-to-M^F term in Eq. (12). Either the loss is not actually being minimized as stated, or the observed forgetting comes from a component outside the multi-teacher system, most plausibly the subsequent deletion of x^f_i and resumed FL training after the measured 1–2 'unlearning' rounds. The paper provides no control isolating the multi-teacher loss from this confound. Additionally, the gradient derivations in Eqs. (15), (17), and (19) are mathematically incorrect: for KL(p||q) with q fixed, the gradient is Σ_k (∂p_k/∂θ)(log(p_k/q_k)+1), not (∂p_k/∂θ)(1−p_k/q_k), so the claimed convergence argument is invalid. Until the authors reconcile Eq. (14) with Table III or provide a control experiment showing the multi-teacher loss itself, rather than data deletion, drives forgotten accuracy to zero, the central claim that the multi-teacher system achieves influence removal is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SFU, a federated unlearning method that unifies influence removal and performance preservation in a single multi-teacher distillation step. Three teacher models (a copy of the original model M^P, a randomly initialized model M^F, and a one-hot label teacher M^L_i) guide the unlearned student model on each client; the server aggregates the resulting models. The authors report that SFU removes target-class influence as effectively as retraining while preserving retained-data accuracy, in only 1-2 communication rounds, with time speedups of 40-152x over retraining on CIFAR-10, CIFAR-100, and DBpedia. A theoretical analysis in Section V claims that minimizing a KL-based loss drives the student's predictions on forgotten data to a uniform distribution, thereby erasing the target class.","tokens_in":18136,"tokens_out":3668,"duration_ms":36067,"significance":"If the central claim held, SFU would be a practically appealing contribution: it is simple, avoids historical-information storage, works without global data access, and combines unlearning and performance recovery in one step. The experimental breadth (three datasets, multiple target classes, backdoor verification) and the very large reported efficiency gains are concrete strengths. However, the significance is currently undermined by a theoretical analysis that contains incorrect gradient formulas, an unverified and internally inconsistent assumption about random teacher outputs, and a lack of controls isolating the multi-teacher loss from the subsequent data deletion. The claim that the multi-teacher system itself, rather than data removal and resumed FL training, produces the observed forgetting is not established.","major_comments":[{"comment":"The gradient formulas for KL divergence are mathematically incorrect. For KL(p||q) with q fixed, the derivative with respect to the student parameters θ is Σ_k (∂p_k/∂θ)(log(p_k/q_k)+1), not Σ_k (∂p_k/∂θ)(1−p_k/q_k). This error appears in all three gradient derivations and invalidates the stated convergence argument. The authors should either correct the derivations or remove the analytic claims and rely on empirical evaluation.","section":"Section V.A, Eqs. (15), (17), (19)"},{"comment":"The mechanism described in Eq. (14) is inconsistent with the reported results. If minimizing D_forget drives M^U_i(x^f_i) to Uniform(K), then top-1 accuracy on forgotten data should approach 1/K: about 10% on CIFAR-10, 1% on CIFAR-100, and 7% on DBpedia. Table III reports SFU forgotten-data accuracies of 0.11-0.67%, 0.12-0.47%, and 0.04-0.24%, respectively, which are one to two orders of magnitude below chance. Matching a randomly initialized teacher (whose softmax outputs are near-uniform) cannot produce below-chance accuracy. The paper provides no measurement of M^F's actual output distribution on forgotten samples, so the stated theoretical mechanism cannot explain the headline numbers.","section":"Section V.A, Eq. (14) vs. Table III"},{"comment":"The combined teacher distributions in Eq. (12), e.g., M^P(x^r_i) + α·M^F(x^f_i), are not normalized probability distributions, and the notation mixes arguments over different data subsets. KL divergence requires valid probability distributions over the same support. The authors need to specify the exact per-sample loss, including how the terms over retained and forgotten data are combined and normalized, before the objective can be meaningfully analyzed.","section":"Section IV.C, Eq. (12)"},{"comment":"In Algorithm 1, the forgotten data x^f_i are deleted only after the unlearning loop, and then normal FL training resumes. Since the reported efficiency is 1-2 communication rounds, the measured forgetting could be substantially caused by the deletion of x^f_i and the subsequent resumed FL training, rather than by the multi-teacher loss itself. A control experiment that isolates the multi-teacher loss (e.g., running the loss without deleting x^f_i, or deleting x^f_i without running the loss) is necessary to attribute the observed forgetting to the proposed objective. Without such a control, the central claim that the multi-teacher system achieves both goals is not demonstrated.","section":"Algorithm 1 and Section IV.D"}],"minor_comments":[{"comment":"Equation (8) states M^P = random initialize(M^O), but M^P is defined as a copy of the original model; this should be M^F = random initialize(M^O). The current equation contradicts the definition in the bulleted text.","section":"Section IV.B, Eq. (8)"},{"comment":"The sentence defining x^r_i and x^f_i is reversed: it says x^r_i is 'the target data to be forgotten' and x^f_i is 'the non-target data to be retained', which is the opposite of the notation in Table II and the rest of the paper.","section":"Section IV.C, paragraph before Eq. (9)"},{"comment":"The tables and figures report point estimates without standard deviations or confidence intervals across multiple runs. Given the small differences between SFU and retraining on retained accuracy, some measure of variability is needed to support the claim of no degradation.","section":"Table III and Figure 3"},{"comment":"There are several typographical issues: 'Dbsepia' in Table III and elsewhere, 'Unlearnng' in Section VI.B.3, and the term 'isomorphic' in Section IV.D should likely be 'homogeneous'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising empirical direction, but the theoretical apparatus is not just informal—it is incorrect in its gradient computations and inconsistent with the reported accuracy numbers. The most serious issue is that the stated mechanism (matching a near-uniform random teacher) cannot produce below-chance forgotten accuracy; the paper needs either a substantially revised explanation or a control experiment showing the loss itself, not data deletion, drives the effect. I would not recommend rejection if the authors can fix the theoretical claims and add the missing controls, but the current version does not establish the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know upfront: this paper has a genuinely plausible idea—fuse forgetting and preservation into a single multi-teacher distillation objective for federated unlearning—but the theoretical support is wrong and the headline numbers contradict the described mechanism. It is not publishable in current form, but the core idea is worth a rigorous revision.\n\nWhat's new: the three-teacher combination (original-model teacher, randomly initialized forgetting teacher, and one-hot label teacher) is a real extension of existing distillation-based unlearning methods, which typically use a teacher for only one of the two steps. The paper also runs experiments on image and text data and reports 1-2 communication rounds, with speedups of 40x-150x over retraining. That would be a useful result if it holds up.\n\nThe problems are serious, though. The loss in Eq. (12) is not a well-defined KL divergence: the second arguments are unnormalized sums of teacher outputs over different data subsets. The gradient derivations in Eqs. (15), (17), and (19) are mathematically incorrect. More importantly, the theoretical claim that matching the randomly initialized teacher drives the student to uniform predictions implies top-1 accuracy of about 10% on CIFAR-10 forgotten classes. Table III reports 0.11-0.67%, one to two orders of magnitude below chance. That is a direct internal contradiction. Either the loss is not being minimized as stated, or the forgetting comes from something outside the multi-teacher system—most plausibly the deletion of the forgotten data and resumed FL training after the measured 1-2 rounds. The paper has no control experiment isolating the multi-teacher loss from these confounds. Without that, the central claim that SFU achieves influence removal is unsupported.\n\nThere are also smaller issues: notation flips (x^f and x^r swap roles in Section IV.C), no error bars or seeds, no code release, and baselines that are only two SOTA methods, one of which isn't applicable to text. The ablation study is useful but doesn't separate the components' contribution from the deletion effect.\n\nWho is this for? Someone working on federated unlearning who wants to see a new loss combination and is willing to look past the theory. It deserves a serious referee because the subfield is active and the efficiency claims, if verified with proper controls and code, could matter. My recommendation: send to peer review, but expect major revision—the authors need to fix the loss definition, correct the gradient theory, add a control where data is deleted without the multi-teacher loss, and release code and seeds. I would not cite it in its current form.","headline":"A plausible multi-teacher unlearning idea undermined by incorrect theory and results that contradict its own mechanism.","tokens_in":18631,"tokens_out":5372,"would_cite":false,"duration_ms":47253,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SFU forgets target classes in 1-2 rounds, matching retraining","keywords":["federated unlearning","multi-teacher distillation","knowledge distillation","right to be forgotten","federated learning","backdoor attack","class-level unlearning","communication efficiency"],"falsifier":"Measure the softmax outputs of a freshly randomly initialized copy of the trained model on the to-be-forgotten class; if the mean maximum probability is clearly above $1/K$ (or the distribution is far from uniform), the premise behind Eq. (14) fails, and a model matched to that teacher would not be equivalent to a retrained model.","tokens_in":17517,"feed_emoji":"⚡","tokens_out":7390,"duration_ms":64894,"temperature":0.7,"pith_summary":"This paper claims that federated unlearning can be compressed into a single step: instead of first erasing a class's influence and then fine-tuning to recover accuracy, one distillation objective does both at once. The proposed SFU method gives each client three teacher models — a copy of the original model, a randomly initialized copy, and one-hot labels for retained data — and trains the unlearned model to match them on forgotten and retained data respectively. The authors report that on CIFAR-10, CIFAR-100, and DBpedia, this removes target-class influence to roughly the retraining baseline while keeping retained-data accuracy stable, in one or two communication rounds with 40x-152x time speedups over retraining. The reason a sympathetic reader should care is that, if true, the unlearning objective itself replaces the two-phase procedure that dominates current federated unlearning methods.","feed_headline":"SFU forgets target classes in 1-2 rounds, matching retraining","feed_subtitle":"A three-teacher distillation loss erases target-class influence while keeping accuracy, with 40x-152x speedups over retraining.","key_machinery":"The load-bearing object is the multi-teacher distillation system and its objective, Eq. (12). Three teachers are involved: $M^P$, an identical copy of the original model that anchors performance on retained data; $M^F$, a randomly initialized copy of the architecture that stands for a model which never saw the forgotten data; and $M^L_i$, one-hot encodings of the true labels of each client's retained data. The student model starts as the original model and is updated by minimizing KL divergence toward these teachers, so that the forgotten data are pulled toward the random teacher's behavior while retained data are pulled toward the original and its labels. The essential identity is the claim that minimizing divergence to $M^F$ drives the student's predictions on the forgotten class to uniformity, which is what converts an unlearning objective into actual influence removal.","core_discovery":"SFU's central claim is that the multi-teacher loss in Eq. (12) achieves both goals of federated unlearning in a single step. On each client the student model is initialized to the original model and then minimizes two KL terms whose teacher signals are sums of teacher distributions: $\\mathrm{KL}\\big(M^U_i(x_i)\\,\\big\\|\\,\\alpha M^F(x^f_i)+M^P(x^r_i)\\big)$ plus $\\mathrm{KL}\\big(M^U_i(x_i)\\,\\big\\|\\,\\alpha M^F(x^f_i)+M^L_i(x^r_i)\\big)$. The forgetting teacher $M^F$ is a randomly initialized copy of the architecture; the paper argues that because it has never seen the forgotten data $x^f_i$, its output is approximately uniform, so driving the student's KL to zero makes the unlearned model's predictions on the forgotten class uniform and thereby erases that data's influence. The preservation teachers $M^P$ (a copy of the original model) and $M^L_i$ (one-hot labels of the retained data) hold the student's behavior on retained data steady. On CIFAR-10, CIFAR-100, and DBpedia the result is forgotten-data accuracy and backdoor success rates close to retraining, retained-data accuracy within a point or so of the original model, and completion in one or two communication rounds.","pith_inferences":["Editorial extension: the random-teacher-as-forgetter recipe is a distillation analogue of gradient ascent; if random initialization is not uniform enough, calibrating $M^F$'s outputs or substituting a model trained on disjoint proxy data would be a natural variant to test.","Editorial extension: the paper verifies forgetting with classification accuracy and backdoor success rate, but not with representation-level or membership-inference tests; those could reveal residual influence even when the uniform-output condition is met.","Editorial extension: because Eq. (12) feeds unnormalized sums of teacher distributions into KL divergences, a mixture-of-teachers formulation with weights summing to one is a direct test of whether the reported balance between forgetting and preservation depends on that normalization choice."],"forward_implications":["Class-level federated unlearning can be completed in one or two communication rounds without a separate fine-tuning phase.","Retained-data accuracy stays close to the pre-unlearning model while forgotten-data accuracy approaches the retraining baseline, so forgetting does not visibly trade off against utility.","Backdoor attack success rates on forgotten data drop to roughly the retraining level, indicating that implanted behaviors on the target class are removed.","The same update works for single-class and multi-class requests and transfers from convolutional image models to an LSTM text model.","No historical update logs are needed, sidestepping the storage cost of many existing federated unlearning designs."],"supporting_citations":[{"why":"Supplies the FedAvg aggregation protocol into which SFU inserts its unlearning phase.","marker":"[3]"},{"why":"Differentially private federated unlearning baseline used for comparison of fidelity and efficiency.","marker":"[16]"},{"why":"Class-discriminative pruning baseline used for comparison of unlearning efficacy, fidelity, and speedup.","marker":"[36]"},{"why":"Provides the CIFAR-10 and CIFAR-100 image benchmark datasets used in the experiments.","marker":"[43]"},{"why":"Provides the DBpedia text dataset used to demonstrate SFU's applicability to text models.","marker":"[44]"},{"why":"Supplies the ResNet18 and ResNet44 architectures used for the image classification experiments.","marker":"[45]"},{"why":"Supplies the LSTM architecture used for the DBpedia text classification experiments.","marker":"[46]"},{"why":"Provides the Dirichlet-based non-iid data partitioning used to simulate federated data distributions.","marker":"[47]"}],"fun_headline_variants":["SFU forgets target data in one round, retaining accuracy","Streamlined unlearning: one step, three teachers, no quality loss","Forget federated data in a single round, keep model performance","Multi-teacher distillation enables one-shot federated unlearning","SFU: erase data influence in one round, match retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a randomly initialized copy of the model produces near-uniform predictions on data it never trained on, so matching that teacher's outputs erases the target data's influence; if initialized networks do not behave this way, the theoretical argument collapses even if forgotten-class accuracy happens to fall to zero.","fun_headline_variants_meta":{"raw":{"variants":["SFU forgets target data in one round, retaining accuracy","Streamlined unlearning: one step, three teachers, no quality loss","Forget federated data in a single round, keep model performance","Multi-teacher distillation enables one-shot federated unlearning","SFU: erase data influence in one round, match retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1609,"prompt_tokens":1079,"completion_tokens":530,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":441}},"tokens_in":695,"tokens_out":530,"duration_ms":5970,"temperature":1.0,"reasoning_tokens":441,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:31:28.098251+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the softmax outputs of a freshly randomly initialized copy of the trained model on the to-be-forgotten class; if the mean maximum probability is clearly above $1/K$ (or the distribution is far from uniform), the premise behind Eq. (14) fails, and a model matched to that teacher would not be equivalent to a retrained model.","supporting_citations":[{"cited_title":"Fedrecov- ery: Differentially private machine unlearning for federated learning frameworks,","cited_arxiv_id":null,"evidence_quote":"Differentially private federated unlearning baseline used for comparison of fidelity and efficiency."}],"review_version":1}