REVIEW 4 major objections 8 minor 13 references
Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection
T0 review · 4 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read FedProTIP shows that federated continual learning can avoid catastrophic forgetting without replay memory, generative models, or task labels, by projecting updates onto the orthogonal complement of previously learned task subspaces and…
desk verdict The empirical case is strong, but the core projection equation is written on the wrong side of the weight matrix, so the paper's central forgetting-mitigation mechanism is not mathematically supported as stated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the set of layer-wise core bases $\Phi_l^{(1:t)}$, a low-rank orthonormal basis for the dominant subspace of each layer's activations across tasks, extracted by randomized SVD and aggregated at the server. From these bases the method builds the projection operator $P = I - \Phi\Phi^{\top}$, which is applied locally to gradients, and the relevance metric $\omega^{(\tau,t)} = \|U_L^{(t)}(U_L^{(t)})^{\top} a_L^{(\tau)}\|_2$, which measures how strongly a final-layer activation aligns with a task's subspace. The projection preserves earlier representations; the relevance vectors, aggregated into reference vectors per client and matched by cosine similarity, supply task identity at inference. In one phrase, the engine of the paper is "subspace as memory": the same low-rank basis is used both to prevent forgetting and to route test inputs to the appropriate output head.
What would settle it
Compare the subspace built from the random subset with the subspace built from the full set of a task's internal activity patterns; if the subset's basis misses a meaningful fraction of the directions the full basis spans, the orthogonal projection will not protect earlier tasks, and accuracy on those tasks should fall as soon as a new task is trained. Concretely, on 10-split CIFAR100, vary the subset size from tens to hundreds of samples and plot average accuracy: if accuracy stays flat, the assumption holds; if it collapses, the assumption fails.
Extended reading notes
Core claim
On FedProTIP's own terms, the central discovery is that a federated gradient-projection scheme can be made communication-cheap and privacy-respecting by moving the orthogonal projection to each client and sending only aggregated core bases, and that these same bases enable task identity to be inferred at test time without any auxiliary classifier. Each client computes activations for a small random sample of its current-task data, subtracts their components lying in previously learned subspaces, and applies randomized singular value decomposition to obtain low-rank bases. The server agglomerates these bases into a global subspace. Training updates that would overwrite old representations are removed by orthogonal projection, while the final-layer activation's norm in each stored subspace forms a relevance vector compared against reference vectors to predict the task. The paper argues this achieves near-task-aware accuracy under task-agnostic class-incremental inference and consistent gains across heterogeneity levels, client counts, and dataset scales.
Load-bearing premise
The load-bearing premise is that a small random sample of a task's internal activity patterns represents the whole task's feature directions well enough that the subspace built from it captures almost every direction later updates could disturb.
Editorial extensions
If this is right
- Replay-free and generator-free federated continual learning becomes viable: methods that store old examples or train generators to reproduce them are no longer needed to keep accuracy on earlier tasks.
- Task-agnostic inference no longer requires a separate task classifier: the same subspaces that protect old tasks can route test inputs to the right head, yielding 86.00% versus 41.80% average accuracy on 10-split CIFAR100.
- Forgetting stays near zero as tasks accumulate: final forgetting of 0.83% on CIFAR100 at $\alpha=0.5$, and below 3% on DomainNet, compared with double-digit forgetting for most baselines.
- Communication overhead shrinks as the task sequence grows: reported per-client per-task cost drops from about 9.8 MB to 0.3 MB, an order of magnitude below the 48 MB fixed cost of FOT.
- The threshold $\epsilon_l$ gives an explicit stability-plasticity dial: higher values favor preserving old tasks, lower values leave more room for new ones, with accuracy largely insensitive across 0.7–0.9.
Reading between the lines
- Inference: the relevance scores should also work as a novelty signal; a test input with low similarity to every stored reference vector could be flagged as out-of-distribution or belonging to an unseen task. The paper does not make this claim.
- Inference: the method's reliance on a random subset of activations implies a testable scaling law: as the subset size grows, the sampled bases should approach the full activation SVD and forgetting should decrease. Measuring that curve would directly probe the paper's weakest assumption.
- Inference: the majority-vote task predictor treats clients equally; weighting votes by client data volume or subspace confidence is a natural extension that could recover some of the accuracy lost on the 20-split ImageNet-R setting, where the paper reports smaller task-identity-prediction gains.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. FedProTIP is a replay-free federated continual learning framework with two components. First, to mitigate catastrophic forgetting, each client projects its local gradients onto the orthogonal complement of subspaces spanned by the activated representations of previously learned tasks; per-client core bases are extracted by randomized SVD on a small random subset of layer-wise activations (Section 4.2) and aggregated on the server into a global task subspace (Section 4.3). Second, to enable task-agnostic inference, each client stores reference vectors that measure how its own final-layer activations align with each task subspace; at test time, a relevance vector computed from the test activation is compared with the stored references via cosine similarity, and a majority vote across clients predicts the task identity, which routes the input to the corresponding output head (Section 4.4). Experiments on CIFAR100, ImageNet-R (5/10/20 splits), and DomainNet report consistent gains over FedAvg, GLFC, LGA, TARGET, LANDER, and FOT, with particularly large margins in task-agnostic class-incremental settings (e.g., 86.00 vs 41.80 average accuracy on 10-split CIFAR100, Table 13), together with lower training time, GPU memory, and communication cost. The appendix reports ablations over client counts, projection thresholds, batch sizes, backbones trained from scratch or pretrained, task orderings, and task-prediction accuracy per phase.
Significance. If the mechanism is as claimed, this is a meaningful advance for federated continual learning: it removes the common task-ID-during-inference assumption via a lightweight subspace-relevance predictor, is replay- and generator-free, and is empirically well supported (three benchmarks; ablations over number of clients, thresholds, batch sizes, backbones, and task orders; standard deviations over seeds; per-task communication-cost tables). The honest discussion of the TIP mechanism's weaker regimes (20-split ImageNet-R, shared-classifier DomainNet) is a point in the paper's favor. The central issue is that the written mathematics does not support the stated mechanism: Eq. (4) applies the projection on the wrong side of the gradient relative to the invariance used in Eq. (3), and Appendix B's projector algebra is invalid for the basis constructed in Eq. (9). These are fixable in a revision, but until the formulas are corrected and matched to the implementation, the near-zero forgetting cannot be attributed to the stated orthogonal-projection mechanism.
major comments (4)
- [Section 3.2, Eqs. (3)-(4); Section 4.1; Appendix B Eq. (14)] The invariance claimed in Eq. (3) requires that subsequent updates leave previous-task pre-activations unchanged, i.e., ΔW̃_l^(t) a_l^(τ) ≈ 0 for every earlier-task activation a_l^(τ) in the span of Φ_l^(1:t-1), equivalently ΔW̃ Φ = 0; this is achieved by right-multiplying the update by (I - ΦΦ^T), because (I - ΦΦ^T)a = 0 for all a in col(Φ). Eq. (4), however, defines the left-multiplied update ∇W̃_k = (I - ΦΦ^T)∇W_k, which does not imply (I - ΦΦ^T)∇W_k Φ = 0: for Φ = e_1 and ∇W = [[1,2],[3,4]], (P∇W)Φ = (0,3)^T ≠ 0, so the pre-activation of a previous-task input changes by -η(0,3)^T. The expression in Eq. (4) is also dimensionally undefined when the layer's output dimension d_{l+1} differs from its input dimension d_l (the product Φ^T∇W requires d_l = d_{l+1}), which is the case for the final classifier (512 to 10) and at channel-transition layers of ResNet-18. Appendix B, Eq. (14), repeats the same left-multiplied definition. Thus, as written, the projection does not implement the orthogonal-projection mechanism from which Eq. (3) is derived, leaving the central forgetting-mitigation guarantee unsupported. Please correct the side of the multiplication (∇W̃ = ∇W - ∇W ΦΦ^T is the GPM-consistent form) and confirm which operator the released code applies; if the implementation matches Eq. (4) literally, the near-zero forgetting must instead be attributed to components such as head freezing and task routing, and the paper's mechanism claim would need to be substantially revised.
- [Section 4.3, Eq. (9); Appendix B, Eq. (15)] The server-side aggregation in Eq. (9) appends residual columns U_k^(t) - Φ Φ^T U_k^(t) to the global basis Φ without renormalization. These residuals are orthogonal to the columns of Φ by construction, but they generically have norm strictly less than 1, so the assembled global Φ does not have orthonormal columns. The idempotence claim P² = P in Appendix B, Eq. (15), requires Φ^T Φ = I and therefore fails for the aggregated Φ; likewise, Eq. (6)'s 'projection' of activations no longer annihilates components in the span of the previous bases, since for a = Φc one gets (I - ΦΦ^T)a = Φ(c - Φ^T Φ c) ≠ 0 in general. The paper should normalize the appended bases or re-orthogonalize the global subspace (e.g., by an additional SVD on the server), and update Appendix B's algebra accordingly.
- [Appendix B (convergence analysis)] The conclusion of Appendix B that 'the convergence behavior of the algorithm is preserved' is not established by the argument given. The paragraph shows only that the projected stochastic gradient inherits the norm and variance bounds of the unprojected gradient (true when P is contractive, i.e., when the bases are orthonormal). But E[∇F̃_k] = P∇F_k is a biased gradient of the original objective, the fixed points of the projected dynamics (points with P∇F_k = 0) are not stationary points of the federated objective in Eq. (1), and no convergence rate or limit statement is provided for the projected objective either. The assumptions of Li et al. (2019) are transferred without transferring the theorem; a formal convergence statement, or a clear relegation of this paragraph to a heuristic motivation, is needed.
- [Section 5, Figure 2, Table 13; Appendix C.2] The protocol for the task-agnostic class-incremental evaluation is under-specified. In Figure 2a and Table 13 it is not described how each baseline performs inference when the task identity is unknown (e.g., classification through the full expanded head, per-task heads with a default routing rule, or a method-specific mechanism); Section 5.2 specifies a shared single classifier only for the DomainNet experiments. Because the headline margin (86.00 vs 41.80 on CIFAR100) depends on this protocol, the task-agnostic inference rule for every method should be stated explicitly, and a stronger task-agnostic variant of the most relevant baseline (e.g., LODE applied to FOT) would strengthen Table 4. Relatedly, Appendix C.2 should state whether the partial freezing scheme (freezing the first two residual blocks after task 1, and freezing previously learned output heads) is applied to all methods or only to FedProTIP, as this architectural choice can confound the forgetting comparison.
minor comments (8)
- [Section 5.3, captions of Tables 3 and 4] Section 5.3 refers to the PEC/SCR/LODE comparison as 'Table 5' while that table is captioned 'Table 4', and the caption of the client-scaling table reads 'Table 3: Table 3: ...' with a duplicated label; the cross-references and captions should be corrected.
- [Appendix A.3, A.4 (Tables 9 and 11)] The caption of Table 9 mentions '20-Split DomainNet' and the caption of Table 11 says '6-split DomainNet' while its table header reads '20-Split DomainNet', although all DomainNet experiments in the paper use a 6-split configuration; the dataset-split labels should be made consistent.
- [Tables 10 and 13] Table 13 reports TARGET's task-agnostic average accuracy on 10-split CIFAR100 as 27.55 ± 0.89, while Table 10 reports 27.37 ± 1.00 for the same setting; the two tables should agree.
- [Section 4.2] No ablation or guidance is provided for choosing m_s (the number of sampled activations), and the paper does not analyze how m_s and the threshold ε_l jointly determine whether the extracted bases span the dominant directions of each task's feature manifold; an ablation over m_s would substantially support the representativeness assumption on which both the forgetting mechanism and the task-ID references rely.
- [Figure 1 and Eq. (11)] Figure 1's caption writes the client vote as the argmax of the inner product ⟨ω̂, ω_k^(τ)⟩, whereas Eq. (11) defines the similarity as the cosine similarity with normalized vectors; the figure and the equation should be made consistent.
- [Section 5.4 and Table 14] The communication-cost table appears to count only newly extracted bases per task; the broadcast of the cumulative global bases Φ^(1:t) to all clients and the transmitted reference vectors should be itemized (or the asymptotic claim in Section 5.4 adjusted), and the units (MB vs MiB: the text says '48 MiB' while the caption says MB) should be harmonized.
- [Abstract, Section 4, Appendix D.2] The privacy assertions ('preserving data privacy', 'privacy-preserving') in the abstract and contribution list are stronger than the discussion in Appendix D.2 supports, given that clients transmit subspace bases of intermediate activations; the claims should be qualified to match D.2's cautious framing.
- [Related work (Sections 2.1 and D.1)] Recent FCL methods discussed in related work (FedWeIT, TagFed, pFedDIL, AF-FCL) are not included in any experimental comparison; the state-of-the-art claim would be better supported by including them or by explicitly justifying their exclusion.
Circularity Check
No significant circularity: FedProTIP's derivation chain is self-contained and its main weaknesses are correctness concerns, not input/output equivalences.
full rationale
I walked the derivation chain and did not find a step in which a prediction or first-principles result reduces, by the paper's own equations or by a self-citation chain, to its inputs. The core forgetting-mitigation claim rests on the orthogonal-projection identity in Eq. 3, which is a mathematical consequence of the intended projection condition rather than a restatement of the empirical results. The task-identity predictor uses reference vectors computed from training activations and stored core bases, but inference is performed on new test inputs, so the task prediction is not circular: it is a supervised subspace classifier whose training references are constructed from labeled training tasks and whose test inputs are unseen. The projection threshold epsilon_l is fixed at 0.7 for the main pretrained-ResNet experiments and tested with ablations; the Appendix A.1 hyperparameter search for the scratch-ResNet variant is ordinary tuning and does not make the reported comparisons forced by construction. The only self-citation of note (Chen & Vikalo 2024) appears in the threat-model discussion about gradient inversion and is not load-bearing for any derivation or claimed theorem. There is no imported uniqueness theorem, no ansatz smuggled in via citation, and no renaming of a known empirical pattern as a new organization. The most serious technical concern is a mathematical consistency issue rather than circularity: Eq. 4 applies the projector as left multiplication, whereas the invariance asserted in Eq. 3 requires the updated weight matrix to annihilate previous activations, which is normally enforced by right multiplication with the projector. Appendix B also asserts idempotence of P = I - Phi Phi^T without stating the orthonormality conditions needed for Eq. 15 to hold as written. These are correctness risks that would require a fix to the stated algorithm, but they do not make the derivation equivalent to its inputs. Accordingly, the appropriate circularity finding is a clean non-finding with score 0.
Assumptions & free parameters
free parameters (2)
- projection threshold epsilon_l =
0.7 (pretrained ResNet), 0.95 (scratch ResNet); ablated over [0.7, 0.9]
- activation sample sizes m and m_s =
Not stated numerically in the paper
assumptions (4)
- domain assumption Randomized SVD on the projected activations yields a low-rank basis that captures the dominant feature directions of each task (Eqs. 7-8).
- domain assumption Orthogonal projection of gradients onto the complement of past-task subspaces preserves previously learned functions for all inputs from past tasks, not only the sampled ones (Eqs. 2-3).
- ad hoc to paper The appended bases in Eq. 9 are orthonormal, so that the projector I - Phi Phi^T is an orthogonal projector.
- domain assumption Task identity is inferable from the norms of final-layer activations projected onto per-task subspaces (Eqs. 10-11).
Cite this review
Pith. "Pith review of Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection." pith.science (2026). https://pith.science/paper/2RDBUOWY
@misc{pith2026250921606,
author = {Pith},
title = {Pith review of: Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection},
year = {2026},
howpublished = {\url{https://pith.science/paper/2RDBUOWY}},
note = {Machine review of arXiv:2509.21606}
}
read the original abstract
Federated continual learning (FCL) enables collaborative model training across distributed clients on sequentially arriving tasks without revisiting past data. However, existing approaches often suffer from catastrophic forgetting, rely on replay buffers or generative models that may violate privacy constraints, or assume knowledge of task identities during inference. We propose FedProTIP (Federated Projection-based Continual Learning with Task Identity Prediction), a replay-free FCL framework that maintains shared task-specific feature subspaces across clients. Each client extracts low-rank core bases from intermediate activations using randomized singular value decomposition, capturing dominant feature directions associated with the current task. These bases are transmitted to the server and aggregated to construct global task subspaces that capture shared feature directions across clients without requiring data sharing. During training, client updates are projected onto the orthogonal complement of previously learned subspaces to reduce cross-task interference and mitigate catastrophic forgetting. The learned subspaces are also reused during inference to estimate task identity via subspace relevance, enabling task-agnostic prediction without requiring explicit task labels. Experiments on CIFAR100, ImageNet-R, and DomainNet demonstrate that FedProTIP consistently outperforms state-of-the-art federated continual learning baselines while maintaining lower training time, memory footprint, and communication cost.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[4]
Adaptive plasticity improvement for continual learning
Yan-Shuo Liang and Wu-Jun Li. Adaptive plasticity improvement for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7816–7825, 2023a. Yan-Shuo Liang and Wu-Jun Li. Loss decoupling for task-agnostic continual learning.Advances in Neural Information Processing Systems, 36:11151–11167, 2023b. Sen Li...
-
[5]
We set the task identity prediction threshold toϵl = 0.95,∀l , based on a hyperparameter search
In the task-agnostic inference setting, our method achieves the best performance, showing a significant margin over all other baselines. We set the task identity prediction threshold toϵl = 0.95,∀l , based on a hyperparameter search. This threshold is higher than the one used for the pretrained ResNet18 model (ϵl = 0.7), as the pretrained model provides a...
work page 2021
-
[8]
still hold under our projected update scheme. Assumptions 1 and 2 in (Li et al., 2019), which concern the smoothness of the local loss functions and the bounded variance of stochastic gradients, are unaffected because they are properties of the objective function Fk itself and not of the update mechanism. Furthermore, assumptions 3 and 4 about the bound t...
work page 2019
-
[10]
utilizes label text embeddings (LTE) generated by pre- trained language models as anchor points, constraining feature embeddings of the training data around the corresponding class LTEs. Additionally, these anchors guide the generator optimization, ensuring that the global model embeddings of synthetic samples remain close to LTEs, thereby generating more...
work page 2021
-
[11]
adapts GPM to the FCL setting, with key differences from Fed- ProTIP occurring at the end of each task: (i) A client transmits its input representation multiplied by a standard normal vector with a predefined sampling dimension; (ii) the randomized input representa- tions are averaged and the core bases of the gradient subspace are extracted from these ag...
work page 2021
-
[12]
employs knowledge distillation enabled by a surrogate dataset made available to clients as well as the server. GLFC (Dong et al., 2022
work page 2022
-
[13]
addresses catastrophic forgetting by leveraging class-aware gradient compensation and class-semantic relation distillation, while relying on memory of old examples. The follow-up studies (Liu et al., 2023; Dai et al., 2023; Li et al., 2024c;a) reduce the size of the replay cache but remain reliant upon old samples. To address the reliance on real data, ge...
work page 2023
-
[2019]
Towards efficient replay in federated incremental learning
Yichen Li, Qunwei Li, Haozhao Wang, Ruixuan Li, Wenliang Zhong, and Guannan Zhang. Towards efficient replay in federated incremental learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12820–12829, 2024a. Yichen Li, Yuying Wang, Tianzhe Xiao, Haozhao Wang, Yining Qi, and Ruixuan Li. Rehearsal-free continual f...
Show all 13 references
-
[2020]
Recovering labels from local updates in federated learning.arXiv preprint arXiv:2405.00955,
Huancheng Chen and Haris Vikalo. Recovering labels from local updates in federated learning.arXiv preprint arXiv:2405.00955,
-
[2021]
How to train your vit? data, augmentation, and regularization in vision transformers.arXiv preprint arXiv:2106.10270,
Andreas Steiner, Alexander Kolesnikov, , Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers.arXiv preprint arXiv:2106.10270,
-
[2022]
For CIFAR100, following the original paper we set the memory size to 2000; to satisfy memory constraints, for DomainNet and ImageNet-R the memory size is limited to
employs exemplar replay by storing a subset of raw samples for each task. For CIFAR100, following the original paper we set the memory size to 2000; to satisfy memory constraints, for DomainNet and ImageNet-R the memory size is limited to
2000
-
[2023]
Censor: Defense against gradient inversion via orthogonal subspace bayesian sampling.arXiv preprint arXiv:2501.15718,
Kaiyuan Zhang, Siyuan Cheng, Guangyu Shen, Bruno Ribeiro, Shengwei An, Pin-Yu Chen, Xiangyu Zhang, and Ninghui Li. Censor: Defense against gradient inversion via orthogonal subspace bayesian sampling.arXiv preprint arXiv:2501.15718,
-
[2024]
On the convergence of fedavg on non-iid data.arXiv preprint arXiv:1907.02189,
11 Preprint Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data.arXiv preprint arXiv:1907.02189,
1907 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.