Pith. sign in

REVIEW 4 major objections 4 minor 74 references

SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sequential federated training tends to erase earlier clients' knowledge; this paper claims that distilling from several prior-round teachers, weighted by how different their data distributions were, stops that forgetting and beats…

desk verdict A sensible, incremental method with consistent accuracy gains, but the forgetting-mitigation claim outruns the evidence because the forgetting metric is missing and the comparison omits key SFL baselines. read the letter →

arxiv 2507.08508 v1 pith:UZH6UKQC submitted 2025-07-11 cs.LG

Haotian Xu , Jinrui Zhou , Xichong Zhang , Mingjun Xiao , He Sun , Yin Xu This is my paper · ORCID
classification cs.LG
keywords SequentialFederatedLearningCatastrophicForgettingMulti-TeacherKnowledgeDistillationDecoupledDataHeterogeneityTeacherSelectionMaximumCoverageProblem
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a training scheme for federated learning in which clients are visited one after another, a setup where the model drifts toward the latest client and forgets earlier ones. It claims that if the current client is trained with several models from the previous round acting as teachers, and each teacher's target-class and non-target-class knowledge is weighted separately by the class-distribution gap between teacher and student data, the model can absorb new data without discarding old knowledge. It further claims that selecting teachers greedily so their combined class distributions approximate a uniform distribution removes redundant teachers while preserving knowledge coverage, lowering communication and computation cost. If these claims hold, sequential federated learning becomes a practical option for strongly heterogeneous data, because its main failure mode—catastrophic forgetting—can be controlled without any data sharing.

What carries the argument

The load-bearing object is a decoupled multi-teacher distillation loss, built by extending single-teacher Decoupled Knowledge Distillation (DKD) to several teachers. It splits each teacher's knowledge into non-target-class distillation (NCKD, Eq. 3) and target-class distillation (TCKD, Eq. 4), and sets the teacher weights $g_k$ and $h_k$ in Eq. (5) from the discrepancy $d(D_{T_k}, D_S)$ between the teacher client's class distribution and the student client's distribution: the NCKD weight grows with discrepancy, the TCKD weight grows with similarity. The second mechanism is the greedy teacher-selection algorithm (Algorithm 1), which repeatedly adds the client whose class distribution moves the aggregated distribution closest to uniform, formalized as a variant of the maximum coverage problem. The weighting is the part that carries the forgetting-mitigation claim; the selection is the part that carries the efficiency and anti-dilution claim.

What would settle it

Run SFedKD with teachers restricted to the immediately preceding round, and in a matched run draw teachers from a longer history such as rounds $r-1$, $r-2$, and $r-3$; if the longer-history run shows materially higher test accuracy on early clients or a clearly lower forgetting measure, the paper's one-round teacher assumption is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that catastrophic forgetting in sequential federated learning can be overcome by multi-teacher knowledge distillation from models of the immediately preceding round, provided each teacher's contribution is decoupled and reweighted by distributional discrepancy. For non-target-class knowledge, a teacher is weighted more heavily when its client's class distribution differs strongly from the student's, because that teacher holds class knowledge the student cannot obtain from its own labels; for target-class knowledge, a teacher is weighted more heavily when its distribution is similar to the student's, because that knowledge is directly relevant to current learning. Adding these two distillation terms to the cross-entropy loss (Eq. 7) is what the paper claims balances new-task learning with old-knowledge retention. The paper also claims that a greedy teacher-selection rule—choosing teachers so that their aggregate class distribution approaches uniform, a variant of the maximum coverage problem—prevents knowledge dilution and reduces cost, and reports that the full SFedKD framework beats both sequential and parallel federated baselines on five datasets under heterogeneous partitions.

Load-bearing premise

The method assumes that the models from just the previous round are an adequate storehouse of everything worth remembering, so no teacher from two or more rounds back is ever needed.

Editorial extensions

If this is right

  • If SFedKD is correct, sequential federated learning can be used on strongly non-IID data without the global model drifting to the most recent client.
  • The discrepancy-weighting rule gives a general recipe: teachers holding knowledge the student lacks should dominate the non-target-class distillation term, while similar teachers should dominate the target-class term.
  • The complementary teacher-selection mechanism means the forgetting cure does not require storing or transmitting the whole history of client models, only a small complementary subset.
  • The reported pattern that accuracy first rises and then falls as the number of teachers grows supports the paper's claim that redundant teachers dilute knowledge and should be pruned.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested boundary is the one-round teacher horizon: if informative knowledge lives in models from several rounds back, extending the teacher pool beyond round $r-1$ could reduce forgetting further, at extra communication and storage cost.
  • The weighting uses class-distribution vectors as shared metadata; in settings where those vectors are unavailable or private, teacher discrepancy could instead be estimated from the teachers' soft predictions on a small public proxy set, which the paper does not explore.
  • A testable consequence of the selection design is that the optimal number of teachers should grow with the number of classes and the diversity of client distributions; the paper reports a peak in teacher count but does not model that relationship.
  • Because the method transfers knowledge through model logits and distribution metadata rather than raw data, it fits privacy-oriented federated deployments, though the paper does not analyze what the released teacher models reveal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SFedKD, a Sequential Federated Learning (SFL) framework that uses discrepancy-aware multi-teacher knowledge distillation to mitigate catastrophic forgetting. At the start of each round, the server selects K teacher models from the immediately preceding round by greedily minimizing the discrepancy between the aggregated class distribution of the selected teachers and a uniform distribution; each client then trains with a loss combining cross-entropy with decoupled target-class and non-target-class distillation, weighting teachers by the class-distribution discrepancy between teacher and student data. Experiments on Fashion-MNIST, CIFAR-10, CINIC-10, CIFAR-100, and HAM10000 report higher final test accuracy than FedSeq, CWC, and several parallel-FL baselines, with ablations for the KD weighting, teacher number, teacher selection, hyperparameters, and discrepancy metrics.

Significance. If the reported gains hold, the paper contributes a simple and integrable mechanism to a relatively underexplored setting: importing decoupled multi-teacher knowledge distillation into sequential federated learning, with an explicit coverage-based teacher selection objective. The strengths are the consistent accuracy improvements across five datasets and multiple heterogeneity levels, the design of the teacher-selection problem in Eq. (6) with a greedy algorithm, the teacher-number ablation showing redundancy, and an efficiency analysis against a no-selection baseline. However, the evaluation does not yet establish the central forgetting-mitigation claim because no forgetting metric is reported for SFedKD, and the design restricts teachers to one round of history, which may be insufficient for recovering long-horizon knowledge. The omissions in the baseline comparison and the test-set hyperparameter selection also need to be addressed before the comparative claims can be accepted.

major comments (4)
  1. [§4.2, Table 1; Fig. 1(d)] The central claim that SFedKD "effectively overcomes catastrophic forgetting" is not directly supported by the reported experiments. Fig. 1(d) uses the Forgetting Measure of Chaudhry et al., but no Forgetting Measure is reported for SFedKD, and Table 1 reports only final global test accuracy. A method can improve final accuracy through regularization or better optimization without reducing long-horizon forgetting. Please report the Forgetting Measure and per-client or per-round accuracies for SFedKD against FedSeq and CWC under the ExDir settings of Table 1; this is necessary to substantiate the abstract's headline claim.
  2. [§3.1, Algorithm 2 (lines 5-7); §3.4, Eq. (7)] The teacher set for round r is restricted to T ⊆ Π^(r-1), i.e., models trained in the immediately preceding round. If, as Fig. 1 shows, knowledge from earlier clients is already forgotten by the end of a round, then the one-round-old teachers do not contain that knowledge and the KD loss in Eq. (7) cannot restore it. The paper neither proves that one-round teachers retain all necessary historical knowledge nor ablates a longer teacher history. Please add a comparison with teachers drawn from several previous rounds, or with a maintained history/model zoo, or explicitly characterize the temporal horizon over which SFedKD can recover knowledge.
  3. [§4.1, Table 1; §2.1] The comparison omits the SFL methods cited in Section 2.1: MetaFed [7], FedELMY [57], and PSFL [71]. Since the contribution is framed as an SFL method and the paper claims to outperform state-of-the-art FL methods, the absence of these methods from Table 1 leaves the central comparative claim under-supported. Adding these baselines, or clearly delimiting the comparison to FedSeq, CWC, and the selected PFL methods, is needed.
  4. [§4.3.2, Fig. 3; §4.2, Table 1] The hyperparameters γ and β are tuned on CIFAR-10 ExDir(2,0.5) in Section 4.3.2, and the optimal values are then used in the main experiments, but Table 1 reports CIFAR-10 ExDir(2,0.5) as one of the headline results. If the tuning curves in Fig. 3 are computed on the same test set used for Table 1, then this constitutes selection on the test set and the reported CIFAR-10 ExDir(2,0.5) gain is optimistically biased. Please either use a validation split for hyperparameter selection or explicitly report how the selected hyperparameters were chosen independently of the main comparison.
minor comments (4)
  1. [Tables 2-4; Figs. 3-5] The ablation tables and sensitivity figures do not report error bars or standard deviations, and several differences are small (e.g., Table 3, ExDir(2,10.0), K=3, advantage 0.13%; Table 4, ExDir(2,10.0), L1 vs. KL difference 0.48%). Please add repeated-trial error bars or state which differences are statistically meaningful.
  2. [§3.2, Eqs. (3)-(4)] The notation in Eqs. (3) and (4) is hard to parse: the non-target softmax probabilities use sums over \tilde c ≠ t without explicitly defining the index ranges, and Eq. (4) is broken across lines in a way that makes the formulas for p^T_t, p^S_t, p^T_{\tilde t}, and p^S_{\tilde t} unclear. Please rewrite these definitions with a single consistent notation for the full softmax and the non-target normalization.
  3. [§3.3, Algorithm 1] The text argues that Eq. (6) is NP-hard via a one-class-per-client special case, but no approximation guarantee is stated for the greedy Algorithm 1. Please state whether the algorithm inherits a standard (1 - 1/e)-type greedy guarantee and under which discrepancy metric, or clarify that it is a heuristic.
  4. [§2.2, Table 2] The related work cites DE-MKD [8], an entropy-based decoupled multi-teacher distillation method, but the ablation in Table 2 does not compare against it. Adding that baseline would make the claimed advantage over prior decoupled multi-teacher KD more concrete.

Circularity Check

1 steps flagged · score 2.0 of 10

No equation-level circularity; the core loss and teacher-selection objectives are explicit design choices. One mild fitted-input issue: the gamma/beta coefficients are tuned on the same CIFAR-10 configuration that is then reported as a headline result.

  1. fitted input called prediction [Section 4.3.2 ('Effects of gamma and beta'), Eq. (7), and Table 1.]
    "Additionally, the optimal parameter combination (gamma=1.0, beta=3.0) was adopted in the main experiments. In Table 1 the same benchmark reports SFedKD 61.56±2.32 (CIFAR-10 Exdir(2,0.5)) and 72.77±1.07 (CIFAR-10 Exdir(2,10.0))."

    The coefficients gamma and beta in the total loss of Eq. (7) are selected by maximizing accuracy on a CIFAR-10 configuration, and that same configuration is then reported in the main comparison table. The paper does not describe a separate validation split for this tuning, so the reported CIFAR-10 advantage is partly the result of fitting these two coefficients to the test configuration rather than being an out-of-sample prediction. This is a mild, localized form of fitted input called prediction. It does not make Eq. (5) or Eq. (7) equivalent to the result by construction, and it does not affect the other datasets or the relative comparison in the ablation studies.

full rationale

The derivation chain is an explicit construction, not a hidden fit. Eq. (5) defines the discrepancy weights g_k and h_k directly from the class-distribution distance d(D_Tk, D_S); Eq. (7) combines cross-entropy with the two decoupled KD terms; and Algorithm 1 greedily minimizes a stated coverage objective over teacher class distributions. None of these quantities is first fitted to the reported accuracy values and then renamed a prediction. The weighting formulas are predetermined functional forms, and the teacher-selection objective is stated independently of the benchmark numbers. The paper does not import a load-bearing uniqueness theorem from its own prior work, and the one overlapping citation, PSFL, is used only as a related SFL method in the literature review. The only circularity signal is the hyperparameter selection in Section 4.3.2: the optimal gamma and beta are chosen on CIFAR-10 Exdir(2,0.5), and those same hyperparameters are then used in the main experiments whose Table 1 includes CIFAR-10 results. Because no validation split is described, this makes the CIFAR-10 entry partly a fitted configuration rather than an independent prediction. That is a mild experimental-design circularity, not a reduction of the method's equations to its inputs, so the overall score is low.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The method contributes heuristics rather than a derivation: the weighting rules, the uniform-coverage objective, and the greedy selection are all assumptions adopted because they seem plausible. The central free parameters are the loss coefficients gamma and beta, the teacher count K, and the distillation temperature tau, selected or left unspecified around the CIFAR-10 benchmark. The server also needs access to client label distributions for the mechanism to run. No new physical or conceptual entities are introduced.

free parameters (6)
  • gamma = 1.0 (from grid {0.5,1,2,3,5} on CIFAR-10 ExDir(2,0.5))
    Trade-off weight for the NCKD loss in Eq. (7); selected by test accuracy in Fig. 3, with no separate validation set described.
  • beta = 3.0 (from grid {0.5,1,2,3,5} on CIFAR-10 ExDir(2,0.5))
    Trade-off weight for the TCKD loss in Eq. (7); selected by test accuracy in Fig. 3.
  • K (number of teachers) = 5 in main experiments; 3 to 7 in ablations
    Number of teacher models selected per round; directly controls communication, storage, and distillation quality, and is chosen from the sweep in Fig. 4.
  • tau (temperature) = not reported
    Temperature in the softmax distillation of Eqs. (3) and (4); never given a value in Section 4, although it controls the softness of teacher targets.
  • epsilon = 1e-4
    Smoothing constant in Eq. (5) to avoid division by zero; hand-chosen.
  • discrepancy metric d = KL-Divergence
    The metric used in Eqs. (5) and (6) is a design choice; Table 4 shows limited sensitivity across L1, L2, JS, and KL, but the choice still affects the weights and selection.
assumptions (6)
  • domain assumption Teacher models' class-specific knowledge is determined by their clients' local class distributions.
    Section 3.2 justifies weighting teachers by distribution discrepancy; if the predictive value of a teacher does not track its client's class distribution, the weighting has no basis.
  • ad hoc to paper Comprehensive knowledge space coverage is achieved when the aggregated teacher class distribution is close to uniform.
    The objective in Eq. (6) is a proxy for knowledge diversity; the paper does not prove that uniform aggregate coverage maximizes distillation quality.
  • ad hoc to paper For NCKD, higher discrepancy should give higher weight, while for TCKD, higher similarity should give higher weight.
    This inverse relation in Eq. (5) is stated without derivation; the ablations support it empirically, but no theory is provided.
  • domain assumption Teachers from only the previous round are sufficient to mitigate catastrophic forgetting.
    Algorithm 2 selects teachers only from Pi(r-1); if knowledge older than one round matters, the design may fail to preserve it.
  • ad hoc to paper The greedy teacher selection approximates the optimal solution to Eq. (6).
    No approximation guarantee is proven; the greedy rule is justified only by the NP-hardness of the special one-class-per-client case.
  • domain assumption The server knows each sampled client's local class distribution and may use it for weighting and selection.
    Section 3.1 assumes D_n is available; in real FL, label distributions may be private or unknown to the server, which would break both the weighting and the teacher-selection mechanism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation." pith.science (2026). https://pith.science/paper/UZH6UKQC

@misc{pith2026250708508,
  author       = {Pith},
  title        = {Pith review of: SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZH6UKQC}},
  note         = {Machine review of arXiv:2507.08508}
}
read the original abstract

Federated Learning (FL) is a distributed machine learning paradigm which coordinates multiple clients to collaboratively train a global model via a central server. Sequential Federated Learning (SFL) is a newly-emerging FL training framework where the global model is trained in a sequential manner across clients. Since SFL can provide strong convergence guarantees under data heterogeneity, it has attracted significant research attention in recent years. However, experiments show that SFL suffers from severe catastrophic forgetting in heterogeneous environments, meaning that the model tends to forget knowledge learned from previous clients. To address this issue, we propose an SFL framework with discrepancy-aware multi-teacher knowledge distillation, called SFedKD, which selects multiple models from the previous round to guide the current round of training. In SFedKD, we extend the single-teacher Decoupled Knowledge Distillation approach to our multi-teacher setting and assign distinct weights to teachers' target-class and non-target-class knowledge based on the class distributional discrepancy between teacher and student data. Through this fine-grained weighting strategy, SFedKD can enhance model training efficacy while mitigating catastrophic forgetting. Additionally, to prevent knowledge dilution, we eliminate redundant teachers for the knowledge distillation and formalize it as a variant of the maximum coverage problem. Based on the greedy strategy, we design a complementary-based teacher selection mechanism to ensure that the selected teachers achieve comprehensive knowledge space coverage while reducing communication and computational costs. Extensive experiments show that SFedKD effectively overcomes catastrophic forgetting in SFL and outperforms state-of-the-art FL methods.

Figures

Figures reproduced from arXiv: 2507.08508 by the authors.

Figure 1
Figure 1. Forgetting analysis on the model trained on a sequence comprising 100 clients under different scenarios. (a) Class-wise [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of our proposed SFedKD. The model is trained in a sequential manner across clients (I). At the beginning [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. When 𝐾 = 0, the setup is equivalent to the FedSeq algorithm. (a) Exdir(2,0.5) (b) Exdir(2,10.0) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: The sensitivity analysis of 𝛾 and 𝛽 on CIFAR-10. guarantees of the SFL framework in heterogeneous scenarios while employing knowledge distillation to maintain a balance between new and old knowledge, thereby achieving optimal performance. 4.3 Ablation Study 4.3.1 Effec…
Figure 5
Figure 5. Figure 5: The impact of four key FL arguments on performance. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: shows the test accuracy under the same total training time when TS is disabled (𝐾 = 10) and when the number of selected teachers 𝐾 is set to 3, 5, and 7 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Completion time under different target accuracies. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 46 canonical work pages

  1. [7]

    Yiqiang Chen, Wang Lu, Xin Qin, Jindong Wang, and Xing Xie. 2023. Metafed: Federated learning among federations with cyclic knowledge distillation for personalized healthcare.IEEE Transactions on Neural Networks and Learning Systems(2023)

    work page 2023

  2. [57]

    Naibo Wang, Yuchen Deng, Wenjie Feng, Shichen Fan, Jianwei Yin, and See-Kiong Ng. 2024. One-shot sequential federated learning for non-iid data by enhancing local model diversity. InProceedings of the 32nd ACM International Conference on Multimedia. 5201–5210

    work page 2024

  3. [71]

    Jinrui Zhou, Yu Zhao, Yin Xu, Mingjun Xiao, Jie Wu, and Sheng Zhang. 2025. PSFL: Parallel-Sequential Federated Learning with Convergence Guarantees. In IEEE INFOCOM 2025-IEEE Conference on Computer Communications. IEEE, 1–10

    work page 2025

  4. [1]

    Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. 2019. Gradi- ent based sample selection for online continual learning.Advances in neural information processing systems32 (2019)

    work page 2019

  5. [2]

    Xuming An, Li Shen, Han Hu, and Yong Luo. 2023. Federated learning with manifold regularization and normalized update reaggregation.Advances in Neural Information Processing Systems36 (2023), 55097–55109. SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation Preprint, Under Review,

    work page 2023

  6. [3]

    Léon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. InProceedings of COMPSTAT’2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers. Springer, 177–186

    work page 2010

  7. [4]

    Ken Chang, Niranjan Balachandar, Carson Lam, Darvin Yi, James Brown, Andrew Beers, Bruce Rosen, Daniel L Rubin, and Jayashree Kalpathy-Cramer. 2018. Dis- tributed deep learning networks among institutions for medical imaging.Journal of the American Medical Informatics Association25, 8 (2018), 945–954

    work page 2018

  8. [5]

    Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. 2018. Riemannian walk for incremental learning: Understanding forgetting and intransigence. InProceedings of the European conference on computer vision (ECCV). 532–547

    work page 2018

Show all 74 references
  1. [6]

    Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajan- than, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato. 2019. On tiny episodic memories in continual learning.arXiv preprint arXiv:1902.10486(2019)

    2019 arXiv

  2. [8]

    Xin Cheng, Zhiqiang Zhang, Wei Weng, Wenxin Yu, and Jinjia Zhou. 2024. DE- MKD: decoupled multi-teacher knowledge distillation based on entropy.Mathe- matics12, 11 (2024), 1672

    2024

  3. [9]

    Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. 2018. Cinic-10 is not imagenet or cifar-10.arXiv preprint arXiv:1810.03505(2018)

    2018 arXiv

  4. [10]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255

    2009

  5. [11]

    Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. 2020. Orthogonal gradient descent for continual learning. InInternational Conference on Artificial Intelligence and Statistics. PMLR, 3762–3773

    2020

  6. [12]

    Robert M French. 1999. Catastrophic forgetting in connectionist networks.Trends in cognitive sciences3, 4 (1999), 128–135

    1999

  7. [13]

    Ting Fu, Yu-Wei Zhan, Chong-Yu Zhang, Xin Luo, Zhen-Duo Chen, Yongxin Wang, Xun Yang, and Xin-Shun Xu. 2024. FedCAFE: Federated cross-modal hashing with adaptive feature enhancement. InProceedings of the 32nd ACM International Conference on Multimedia. 9670–9679

    2024

  8. [14]

    Bent Fuglede and Flemming Topsoe. 2004. Jensen-Shannon divergence and Hilbert space embedding. InInternational symposium onInformation theory, 2004. ISIT 2004. Proceedings.IEEE, 31

    2004

  9. [15]

    Jiahua Guo, Yunlong Liang, and Jinan Xu. 2024. Continual Learning with Confidence-based Multi-teacher Knowledge Distillation for Neural Machine Translation. In2024 6th International Conference on Natural Language Processing (ICNLP). IEEE, 336–343

    2024

  10. [16]

    Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. 2020. Fedboost: A communication-efficient algorithm for federated learning. InInternational Conference on Machine Learning. PMLR, 3973–3983

    2020

  11. [17]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531(2015)

    2015 arXiv

  12. [18]

    Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. 2018. Lifelong learning via progressive distillation and retrospection. InProceedings of the European Conference on Computer Vision (ECCV). 437–452

    2018

  13. [19]

    Michael Kamp, Jonas Fischer, and Jilles Vreeken. 2021. Federated learning from small datasets.arXiv preprint arXiv:2110.03469(2021)

    2021 arXiv

  14. [20]

    Minsoo Kang, Jaeyoo Park, and Bohyung Han. 2022. Class-incremental learning by knowledge distillation with adaptive feature consolidation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16071–16080

    2022

  15. [21]

    Samir Khuller, Anna Moss, and Joseph Seffi Naor. 1999. The budgeted maximum coverage problem.Information processing letters70, 1 (1999), 39–45

    1999

  16. [22]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of ...

    2017

  17. [23]

    Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009)

    2009

  18. [24]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classifi- cation with deep convolutional neural networks.Advances in neural information processing systems25 (2012)

    2012

  19. [25]

    Kisoo Kwon, Hwidong Na, Hoshik Lee, and Nam Soo Kim. 2020. Adaptive knowledge distillation based on entropy. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 7409–7413

    2020

  20. [26]

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient- based learning applied to document recognition.Proc. IEEE86, 11 (1998), 2278– 2324

    1998

  21. [27]

    Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae, and Se-Young Yun. 2022. Preservation of the global knowledge by not-true distillation in federated learning. Advances in Neural Information Processing Systems35 (2022), 38461–38474

    2022

  22. [28]

    Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. 2019. Overcoming cata- strophic forgetting with unlabeled data in the wild. InProceedings of the IEEE/CVF International Conference on Computer Vision. 312–321

    2019

  23. [29]

    Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang

  24. [30]

    Jingzhi Li, Fengling Li, Lei Zhu, Hui Cui, and Jingjing Li. 2023. Prototype- guided knowledge transfer for federated unsupervised cross-modal hashing. In Proceedings of the 31st ACM international conference on multimedia. 1013–1022

    2023

  25. [31]

    Shiwei Li, Yingyi Cheng, Haozhao Wang, Xing Tang, Shijie Xu, Weihong Luo, Yuhua Li, Dugang Liu, Xiuqiang He, and Ruixuan Li. 2024. Masked Random Noise for Communication-Efficient Federated Learning. InProceedings of the 32nd ACM International Conference on Multimedia. 3686–3694

    2024

  26. [32]

    Shuai Li, Fan Qi, Zixin Zhang, and Changsheng Xu. 2024. Cross-Modal Meta Consensus for Heterogeneous Federated Learning. InProceedings of the 32nd ACM International Conference on Multimedia. 975–984

    2024

  27. [33]

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems2 (2020), 429–450

    2020

  28. [34]

    Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. 2019. On the convergence of fedavg on non-iid data.arXiv preprint arXiv:1907.02189 (2019)

    2019 arXiv

  29. [35]

    Yipeng Li and Xinchen Lyu. 2024. Convergence analysis of sequential federated learning on heterogeneous data.Advances in Neural Information Processing Systems36 (2024)

    2024

  30. [36]

    Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting.IEEE transac- tions on pattern analysis and machine intelligence40, 12 (2017), 2935–2947

    2017

  31. [37]

    Iou-Jen Liu, Jian Peng, and Alexander G Schwing. 2019. Knowledge flow: Improve upon your teachers.arXiv preprint arXiv:1904.05878(2019)

    2019 arXiv

  32. [38]

    Junkang Liu, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Yuangang Li, and YunXiang Gong. 2024. Fedbcgd: Communication-efficient accelerated block coordinate gradient descent for federated learning. InProceedings of the 32nd ACM International Conference on Multimedia. 2955–2963

    2024

  33. [39]

    Yuang Liu, Wei Zhang, and Jun Wang. 2020. Adaptive multi-teacher multi-level knowledge distillation.Neurocomputing415 (2020), 106–113

    2020

  34. [40]

    Jianghu Lu, Shikun Li, Kexin Bao, Pengju Wang, Zhenxing Qian, and Shiming Ge. 2023. Federated learning with label-masking distillation. InProceedings of the 31st ACM International Conference on Multimedia. 222–232

    2023

  35. [41]

    Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in con- nectionist networks: The sequential learning problem. InPsychology of learning and motivation. Vol. 24. Elsevier, 109–165

    1989

  36. [42]

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep net- works from decentralized data. InArtificial intelligence and statistics. PMLR, 1273–1282

    2017

  37. [43]

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Mat- sukawa, and Hassan Ghasemzadeh. 2020. Improved knowledge distillation via teacher assistant. InProceedings of the AAAI conference on artificial intelligence, Vol. 34. 5191–5198

    2020

  38. [44]

    Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. 2016. Semi-supervised knowledge transfer for deep learning from private training data.arXiv preprint arXiv:1610.05755(2016)

    2016 arXiv

  39. [45]

    SeongUk Park and Nojun Kwak. 2019. Feed: Feature-level ensemble for knowledge distillation.arXiv preprint arXiv:1909.10754(2019)

    2019 arXiv

  40. [46]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library.Advances in neural information processing systems...

    2019

  41. [47]

    Zhuang Qi, Lei Meng, Zitan Chen, Han Hu, Hui Lin, and Xiangxu Meng. 2023. Cross-silo prototypical calibration for federated learning with non-iid data. In Proceedings of the 31st ACM International Conference on Multimedia. 3099–3107

    2023

  42. [48]

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. InPro- ceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2001– 2010

    2017

  43. [49]

    David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. 2019. Experience replay for continual learning.Advances in neural information processing systems32 (2019)

    2019

  44. [50]

    Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. 2020. Fetchsgd: Communication-efficient federated learning with sketching. InInternational Conference on Machine Learning. PMLR, 8253–8265

    2020

  45. [51]

    Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay.Advances in neural information processing systems30 (2017)

    2017

  46. [52]

    Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor-Yosef, and Itai Zeitak. 2019. Overcoming forgetting in federated learning on non-iid data.arXiv preprint arXiv:1910.07796(2019). Preprint, Under Review, Haotian Xu, Jinrui Zhou, Xichong Zhang, Min...

    2019 arXiv

  47. [53]

    Haoyue Song, Jiacheng Wang, and Liansheng Wang. 2024. Cyclical Weight Consolidation: Towards Solving Catastrophic Forgetting in Serial Federated Learning.arXiv preprint arXiv:2405.10647(2024)

    2024 arXiv

  48. [54]

    Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.Advances in neural information processing systems30 (2017)

    2017

  49. [55]

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. 2018. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.Scientific data5, 1 (2018), 1–9

    2018

  50. [56]

    Lin Wang, YongXin Guo, Tao Lin, and Xiaoying Tang. 2023. Delta: Diverse client sampling for fasting federated learning.Advances in Neural Information Processing Systems36 (2023), 47626–47668

    2023

  51. [58]

    Yujia Wang, Lu Lin, and Jinghui Chen. 2022. Communication-efficient adaptive federated learning. InInternational conference on machine learning. PMLR, 22802– 22838

    2022

  52. [59]

    Haitao Wen, Lili Pan, Yu Dai, Heqian Qiu, Lanxiao Wang, Qingbo Wu, and Hongliang Li. 2024. Class incremental learning with multi-teacher distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 28443–28452

    2024

  53. [60]

    Ancong Wu, Wei-Shi Zheng, Xiaowei Guo, and Jian-Huang Lai. 2019. Distilled person re-identification: Towards a more scalable system. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1187–1196

    2019

  54. [61]

    Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. 2019. Large scale incremental learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 374–382

    2019

  55. [62]

    Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747(2017)

    2017 arXiv

  56. [63]

    Baochen Xiong, Xiaoshan Yang, Yaguang Song, Yaowei Wang, and Changsheng Xu. 2023. Client-adaptive cross-model reconstruction network for modality- incomplete multimodal federated learning. InProceedings of the 31st ACM Inter- national Conference on Multimedia. 1241–1249

    2023

  57. [64]

    Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. 2020. Model compression with two-stage multi-teacher knowledge distillation for web ques- tion answering system. InProceedings of the 13th International Conference on Web Search and Data Mining. 690–698

    2020

  58. [65]

    Shan You, Chang Xu, Chao Xu, and Dacheng Tao. 2017. Learning from multiple teacher networks. InProceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 1285–1294

    2017

  59. [66]

    Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. InInternational conference on machine learning. PMLR, 3987–3995

    2017

  60. [67]

    Hailin Zhang, Defang Chen, and Can Wang. 2022. Confidence-aware multi- teacher knowledge distillation. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 4498–4502

    2022

  61. [68]

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. 2022. Decoupled knowledge distillation. InProceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 11953–11962

    2022

  62. [69]

    Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. 2024. Class-incremental learning: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence(2024)

    2024

  63. [70]

    Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. 2021. Co-transport for class- incremental learning. InProceedings of the 29th ACM International Conference on Multimedia. 1645–1654

    2021

  64. [72]

    Xiatian Zhu, Shaogang Gong, et al. 2018. Knowledge distillation by on-the-fly native ensemble.Advances in neural information processing systems31 (2018)

    2018

  65. [73]

    Ruifan Zuo, Chaoqun Zheng, Fengling Li, Lei Zhu, and Zheng Zhang. 2024. Privacy-enhanced prototype-based federated cross-modal hashing for cross- modal retrieval.ACM Transactions on Multimedia Computing, Communications and Applications20, 9 (2024), 1–19

    2024

  66. [2017]

    Advances in neural information processing systems30 (2017)

    Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems30 (2017)

    2017

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.