Pith. sign in

REVIEW 3 major objections 4 minor 17 references

Federated Class-Incremental Learning: A Hybrid Approach Using Latent Exemplars and Data-Free Techniques to Address Local and Global Forgetting

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hybrid Replay tackles both local and global forgetting in federated class-incremental learning with a single autoencoder that replays latent exemplars and synthesizes data from perturbed class centroids, beating prior hybrid methods while…

desk verdict The empirical contribution is solid and worth a look; the 'mathematical proof' claim is not a proof, and the global replay mechanism is under-validated. read the letter →

arxiv 2501.15356 v3 pith:NLLARMML submitted 2025-01-26 cs.LG

Milad Khademi Nori , Il-Min Kim , Guanghui Wang This is my paper · ORCID
classification cs.LG
keywords federatedclass-incrementallearninglocalforgettingglobalhybridreplaylatentexemplarsdata-freeLennard-Jonespotentialautoencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Hybrid Replay (HR), a federated class-incremental learning method that attacks both failure modes at once: local forgetting, when a client's own old classes are crowded out by its new data, and global forgetting, when clients cannot distinguish classes learned only by other clients. HR stores compact latent exemplars rather than raw images, and its decoder either replays those latents or synthesizes data from Gaussian-perturbed class centroids supplied by the server. The paper develops a mathematical formulation in which forgetting is expressed through pairwise task-confusion loss terms, and argues HR can in principle minimize both kinds. In experiments on five FCIL configurations, HR reaches 65.84% on CIFAR-100 (10/10/50/5), about 2.4 points above the strongest hybrid baseline, and a memory-reduced variant stays competitive with exemplar-based methods while using about ten times less memory.

What carries the argument

The load-bearing object is the Hybrid Replay autoencoder with its structured latent space. It is a VAE whose loss includes a term pulling each encoded sample toward its class centroid, so the latent space is both reconstructive and class-discriminative. The server aligns class centroids for each incoming task with the Lennard-Jones potential, a pairwise repulsion-attraction energy, keeping new classes separated from all existing classes globally. At the clients, the decoder performs the replay: it regenerates old inputs from stored latent exemplars to fight local forgetting, and it decodes centroids perturbed by Gaussian noise to synthesize samples of classes the client never saw, fighting global forgetting. Classification is done by nearest-centroid matching in latent space, which is what makes the synthetic and replayed data usable without an extra classifier head.

What would settle it

Take CIFAR-100 (10/10/50/5) and cripple the decoder after Task 1, for example by freezing it or replacing it with a randomly initialized network, while keeping the rest of HR unchanged. If final accuracy approaches the 'HR w/o Global Replay' ablation value (~51.8%) rather than the full HR value (~65.8%), the global-replay mechanism is not doing the claimed work. A complementary check: ask the trained encoder to classify synthetic samples produced by $g(\theta_{h-1}, p_{ij} + \mathcal{N}(0, \sigma^2))$ and compare the predicted-class distribution with the intended class.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a single customized autoencoder can serve simultaneously as classifier, episodic memory, and generator, unifying the two previously separate FCIL strategies. The encoder is trained with a variational lower bound plus a clustering term that pulls each class's samples toward a centroid, and the server places centroids for new tasks using the Lennard-Jones potential so that classes from different clients occupy well-separated positions in latent space. During each subsequent task, a client replays its stored latent exemplars through the decoder to protect local knowledge, and decodes Gaussian-perturbed centroids of other clients' classes to generate synthetic data that protects global knowledge. Nearest-centroid classification then ties the whole pipeline together. The paper claims this eliminates the memory-versus-privacy trade-off that forced earlier methods to choose between exemplar replay and data-free generation.

Load-bearing premise

The global-forgetting defense assumes that decoding a Gaussian-perturbed class centroid produces training samples that faithfully represent a class the client has never seen; the authors acknowledge that a decoder with too little capacity would fail to generate realistic replay images.

Editorial extensions

If this is right

  • HR outperforms the strongest hybrid baseline REMIND+ on all five reported FCIL configurations, with the CIFAR-100 (10/10/50/5) gap at about 2.4 points.
  • A 10x smaller memory variant (HR-mini) still matches or beats the best exemplar-based methods, making the approach usable on memory-constrained clients.
  • Removing global replay drops accuracy by roughly 14 points on CIFAR-100 (10/10/50/5), showing that cross-client synthetic replay, not just local exemplars, carries the global-forgetting defense.
  • The data-free variant HR w/o Latent Exemplars outperforms dedicated data-free baselines such as MFCL and TARGET, so the method does not need stored exemplars to beat the model-based family.
  • Using RFA instead of Lennard-Jones changes accuracy negligibly, indicating that the exact centroid-alignment energy is not the critical factor as long as centroids are well separated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper treats privacy as a qualitative advantage of latent and synthetic replay, but it does not quantify how much private information survives in decoded exemplars; an adversarial reconstruction study would sharpen that claim.
  • The mathematical framework establishes in-principle control of local and global forgetting, but it does not turn decoder error or centroid noise $\sigma$ into a quantitative bound on residual confusion, which would make the proof predictive.
  • Because the Lennard-Jones alignment is agnostic to the data modality, the same dual-replay recipe should transfer to text or tabular class-incremental streams whenever a class-conditional autoencoder can be trained.
  • The near-equivalence of HR and HR w Perfect Exemplars suggests replay image quality is secondary, so even heavier latent compression or quantization could lower the memory footprint further without sacrificing accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Hybrid Replay (HR), a federated class-incremental learning method built around a customized autoencoder. HR stores compressed latent exemplars for replay to mitigate local forgetting, and generates synthetic data by decoding class centroids perturbed with Gaussian noise to mitigate global forgetting. The authors present a mathematical framework in Section 3 that decomposes the FCIL loss into intra-client and inter-client terms, and they claim this framework proves that HR can in principle solve both forgetting challenges (Abstract and Section 4). Experiments on CIFAR-100, ImageNet-Subset, and TinyImageNet compare HR against exemplar-based, data-free, and hybrid baselines, and ablations isolate the effect of latent exemplars, knowledge distillation, global replay, and the Lennard-Jones alignment mechanism.

Significance. If the empirical results hold, HR is a practically valuable FCIL method: it consistently outperforms strong hybrid baselines such as REMIND+ and i-CTRL across five benchmark configurations while using an order of magnitude less memory than raw-exemplar methods. The paper includes a useful ablation suite, including a memory-reduced variant, a perfect-exemplar upper bound, removal of global replay, and replacement of the Lennard-Jones potential with a repulsive-force alternative. The main weaknesses are that the claimed mathematical proof is not actually a proof, and that the core global-replay mechanism—decoding noisy class centroids—is never directly validated for class fidelity. The empirical contribution is substantial, but the formal and mechanistic claims need to be revised or substantiated.

major comments (3)
  1. [Abstract, Section 3, Section 4 (Eqs. 3–6, Eq. 7)] The paper repeatedly states that the mathematical framework 'proves' that HR can tackle local and global forgetting. In the manuscript, Eqs. (3)–(6) are only a loss decomposition: they rewrite a pairwise loss as a sum over task and client interactions. Section 4 then argues verbally that replay and distillation minimize the relevant terms. There is no theorem statement, no set of assumptions connecting decoder fidelity or centroid alignment to the minimized losses, and no formal argument that the proposed algorithm actually minimizes the decomposed terms. This is a load-bearing claim in the abstract and in the contribution list, and it should be either replaced by an explicit theorem with conditions and a proof, or softened to 'formulates' and 'motivates'.
  2. [Algorithm 2, lines 8–9; Table 2; Limitations paragraph in Section 6] The global-replay mechanism relies on the premise that g(θ_{h-1}, p^c_{ij} + N(0,σ²)) produces samples that faithfully represent a class never seen by the client. The paper provides no direct evidence for this: no fidelity metric such as FID or Inception Score, no analysis of nearest-class overlap in the decoded space, no sensitivity study of the Gaussian noise scale σ, and no control experiment replacing the centroid with a random latent vector. The ablation 'HR w/o Global Replay' in Table 2 shows a large accuracy drop, but it demonstrates only that some form of replay signal is useful; it does not establish that the specific centroid-plus-noise decoding mechanism is responsible. The authors' own limitation statement concedes that an insufficiently powerful decoder may fail to generate realistic images. Because global forgetting mitigation is a central claimed contribution, this mechanism needs direct validation or the claim needs to be scaled back.
  3. [Eq. (7), Eq. (9), Algorithm 2 line 9, Table 2] Several key hyperparameters of the proposed method are not reported: λ in Eq. (7), ε, σ, and η in Eq. (9), the Gaussian noise scale σ in Algorithm 2 line 9, and the number of latent exemplars per class used for the main HR results (the table caption only states 200 latent exemplars for the hybrid baselines, and 'HR-mini w 10× less memory' presumably uses 20 but this is not stated). Without these values, the results in Table 2 and the ablations are not reproducible. This is particularly important because the method's behavior depends on the balance between reconstruction, clustering, and distillation losses, and on the noise level used for synthetic global replay.
minor comments (4)
  1. [Section 4, paragraph after Eq. (9)] The name 'Leonard-Jones' is a typo; the correct name is Lennard-Jones, as used elsewhere in the paper.
  2. [Eq. (8)] The summation notation '\sum_{i,j=1}^{K,N}' is ambiguous and likely malformed; it should be clarified, e.g., as a double sum over tasks and classes with an explicit exclusion of the self pair.
  3. [Algorithm 2, line 26 and line 28] The memory variable Mh is overwritten twice: first with re-encoded old exemplars (line 26) and then with random samples from the current task (line 28). The intended procedure likely appends new exemplars to the existing memory; the pseudocode should be rewritten to make this explicit.
  4. [Table 3, caption and header] The column header 'WC Time' is unclear; presumably it denotes wall-clock time, but this should be spelled out. Also, the table reports '# Exemplars' as 150/190 for HR but the main table caption says 200 latent exemplars; the relationship between these numbers should be explained.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central benchmark claims are externally evaluated, and the 'mathematical proof' is an overclaimed design rationale rather than a circular derivation.

full rationale

The central empirical claim (HR outperforms SOTA baselines in Table 2) is benchmarked against external methods on held-out test sets, so it does not reduce to the paper's own fitted values. The 'mathematical framework' in Eqs. 3-6 is a loss decomposition; the subsequent claim that HR 'can, in principle, tackle' local and global forgetting is a verbal design rationale (replay old data for local forgetting; decode centroid-plus-noise for global forgetting) rather than a theorem derived from the equations. The nearest-centroid classifier and the centroid-clustering term in Eq. 7 share the same centroids, but the reported gains are measured on real test data, and the HR w/o Global Replay ablation is a controlled comparison, so the success is not guaranteed by construction. The limitations paragraph explicitly concedes that a weak decoder may fail to generate realistic replay images, which is a correctness risk for the global-replay mechanism, not a circularity. The only self-citation (Khademi Nori and Kim, 2025) is used in related work to situate CIL and is not load-bearing. Overall, no derivation step reduces to its own input under the paper's equations, so circularity is minimal.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The framework depends on uniform task sizes, decoder fidelity, the Lennard-Jones alignment heuristic, and centroid-based classification. Most hyperparameters are unreported, and no new physical entities are introduced.

free parameters (6)
  • lambda (clustering loss weight)
    Weight of the class-centroid clustering term in Eq. 7; no value is reported.
  • epsilon (Lennard-Jones well depth)
    Potential well depth in Eqs. 8-9 for centroid alignment; no value is reported.
  • sigma (Lennard-Jones distance parameter)
    Equilibrium distance in Eq. 8; no value is reported.
  • eta (centroid learning rate)
    Learning rate for centroid updates in Eq. 9; no value is reported.
  • Gaussian noise scale for centroid perturbation
    Standard deviation of N(0, sigma^2) in Algorithm 2 line 9; no value is reported.
  • Number of latent exemplars per class = 200 (default)
    Memory budget choice; HR-mini uses 10x less memory. This is chosen, not fitted.
assumptions (5)
  • domain assumption All tasks contain the same number of classes N and each class appears in exactly one task.
    Eqs. 3-6 assume a uniform N; the mathematical framework does not handle variable task sizes.
  • ad hoc to paper Minimizing the pairwise loss terms in Eq. 6 is equivalent to mitigating local and global forgetting.
    This is the core assertion of the mathematical framework, but no theorem links these loss terms to forgetting rates.
  • domain assumption A decoder trained on reconstructed images can turn latent exemplars and centroid-plus-noise into replay data that reduces forgetting.
    Algorithm 2 lines 7-9 rely entirely on decoder fidelity; the paper's own limitations section acknowledges this sensitivity.
  • ad hoc to paper Lennard-Jones potential alignment produces class centroids that are well separated and improve global replay.
    Eqs. 8-9 are a physics analogy; the ablation HR w RFA shows the specific potential is not essential, so the mechanism lacks theoretical grounding.
  • domain assumption Nearest-centroid classification in the latent space is a reliable decision rule.
    Section 4 defines prediction as argmin over Euclidean distance to class centroids; this assumes centroids represent class distributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Federated Class-Incremental Learning: A Hybrid Approach Using Latent Exemplars and Data-Free Techniques to Address Local and Global Forgetting." pith.science (2026). https://pith.science/paper/NLLARMML

@misc{pith2026250115356,
  author       = {Pith},
  title        = {Pith review of: Federated Class-Incremental Learning: A Hybrid Approach Using Latent Exemplars and Data-Free Techniques to Address Local and Global Forgetting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLLARMML}},
  note         = {Machine review of arXiv:2501.15356}
}
read the original abstract

Federated Class-Incremental Learning (FCIL) refers to a scenario where a dynamically changing number of clients collaboratively learn an ever-increasing number of incoming tasks. FCIL is known to suffer from local forgetting due to class imbalance at each client and global forgetting due to class imbalance across clients. We develop a mathematical framework for FCIL that formulates local and global forgetting. Then, we propose an approach called Hybrid Rehearsal (HR), which utilizes latent exemplars and data-free techniques to address local and global forgetting, respectively. HR employs a customized autoencoder designed for both data classification and the generation of synthetic data. To determine the embeddings of new tasks for all clients in the latent space of the encoder, the server uses the Lennard-Jones Potential formulations. Meanwhile, at the clients, the decoder decodes the stored low-dimensional latent space exemplars back to the high-dimensional input space, used to address local forgetting. To overcome global forgetting, the decoder generates synthetic data. Furthermore, our mathematical framework proves that our proposed approach HR can, in principle, tackle the two local and global forgetting challenges. In practice, extensive experiments demonstrate that while preserving privacy, our proposed approach outperforms the state-of-the-art baselines on multiple FCIL benchmarks with low compute and memory footprints.

Figures

Figures reproduced from arXiv: 2501.15356 by the authors.

Figure 1
Figure 1. Local and global forgetting occur due to class imbal [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Remind Classification occurs after the decoding process, em￾ploying a cross-entropy loss function as depicted in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 2
Figure 2. Our HR approach, except for Task 1, con [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: In the first row, the table reports the impact of the memory size on the final accuracy for the [PITH_FULL_IMAGE:figures/full_fig_p010_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 6 canonical work pages

  1. [3]

    Podnet: Pooled outputs distillation for small-tasks incremental learning

    Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer vision–ECCV 2020: 16th European conference, Glasgow, UK, August 23–28, 2020, proceedings, part XX 16, pp. 86–102. Springer,

    work page 2020

  2. [4]

    Auto-encoding variational bayes

    11 Published as a conference paper at ICLR 2025 Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114,

    arXiv 2025

  3. [7]

    Adaptive aggregation networks for class-incremental learning

    Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 2544–2553, 2021b. Lingjuan Lyu and Chen Chen. A novel attribute reconstruction attack in federated learning. arXiv:2108.06910,

    arXiv

  4. [8]

    Class-incremental learning: Survey and performance evaluation on image classifica- tion

    Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: Survey and performance evaluation on image classifica- tion. arXiv:2010.15277,

    arXiv 2010

  5. [9]

    Federated adversarial domain adaptation

    Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaptation. arXiv:1911.02054,

    arXiv 1911

  6. [11]

    Asynchronous federated continual learning

    12 Published as a conference paper at ICLR 2025 Donald Shenaj, Marco Toldo, Alberto Rigon, and Pietro Zanuttigh. Asynchronous federated continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5055–5063,

    work page 2025

  7. [12]

    Overcoming forgetting in federated learning on non-iid data

    Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor-Yosef, and Itai Zeitak. Overcoming forgetting in federated learning on non-iid data. arXiv:1910.07796,

    arXiv 1910

  8. [13]

    Incremental learning of structured memory via closed-loop transcription

    Shengbang Tong, Xili Dai, Ziyang Wu, Mingyang Li, Brent Yi, and Yi Ma. Incremental learning of structured memory via closed-loop transcription. arXiv:2202.05411,

    arXiv

Show all 17 references
  1. [14]

    Federated learning with matched averaging

    Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv:2002.06440,

    2002 arXiv

  2. [15]

    Achieving linear speedup with partial worker participation in non-iid federated learning

    Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. arXiv:2101.11203,

    arXiv

  3. [16]

    Fine-tuning global model via data-free knowledge distillation for non-iid federated learning

    13 Published as a conference paper at ICLR 2025 Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern...

    2025

  4. [2017]

    Federated learning: Strategies for improving communication efficiency

    Jakub Koneˇcn`y, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492,

    arXiv

  5. [2019]

    icarl: Incremental classifier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001–2010,

    2001

  6. [2020]

    Fedbn: Federated learning on non-iid features via local batch normalization

    Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv:2102.07623,

    arXiv

  7. [2021]

    Don’t memorize; mimic the past: Federated class incremental learning without episodic memory

    Sara Babakniya, Zalan Fabian, Chaoyang He, Mahdi Soltanolkotabi, and Salman Avestimehr. Don’t memorize; mimic the past: Federated class incremental learning without episodic memory. arXiv:2307.00497,

    arXiv

  8. [2022]

    Data: Differentiable architecture approximation with distribution guided sampling

    Xinbang Zhang, Jianlong Chang, Yiwen Guo, Gaofeng Meng, Shiming Xiang, Zhouchen Lin, and Chunhong Pan. Data: Differentiable architecture approximation with distribution guided sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):2905–2920, 2020b. Xin...

    2007 arXiv

  9. [2023]

    A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks

    10 Published as a conference paper at ICLR 2025 Sara Babakniya, Zalan Fabian, Chaoyang He, Mahdi Soltanolkotabi, and Salman Avestimehr. A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks. Advances in Neural Informa...

    2025

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.