REVIEW 4 major objections 5 minor 18 references
Learning without Isolation: Pathway Protection for Continual Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Continual learning can avoid catastrophic forgetting by protecting the pathways data take through a network instead of isolating parameters: LwI fuses each task's freshly trained model with the old one through graph matching, aligning…
desk verdict The empirical gains are plausible and broad, but the unreported fusion coefficient k in Algorithm 1 leaves the 'pathway protection' mechanism untested, so the paper needs a revision rather than acceptance as-is. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine of the method is graph matching used as a channel-alignment step before model fusion. Each layer $\ell$ of the network is treated as a graph whose nodes are channels and whose edges are connections to the adjacent layer; a similarity matrix $K$ built from Euclidean distances between edge weights, $\| e_{ac} - e_{bd} \|_2$, is turned into a soft permutation matrix $P$ by the Sinkhorn algorithm, an entropy-regularized optimal-transport routine that is applied per layer instead of over the whole network. In shallow layers the permutation is computed from $R$ to maximize similarity; in the deep layer it is computed from $-R$ to minimize similarity, so the fused model deliberately keeps the new task's channels apart from the old task's channels. The permutation is applied to the old model's weights, and fusion is the linear interpolation $W_{\text{fusion}} = k\, \tilde{W}_{\text{old}} + (1-k)\, W_{\text{new}}$ with coefficient $k$; the input layer is skipped and the output layer is handled by appending heads. Everything rests on the sparsity of activation channels: if only a fraction of channels are strongly active for a task, then the remaining channels form free pathway capacity for future tasks, which is what makes pathway allocation possible without growing the network.
What would settle it
Take the trained old model and the newly trained model for a second task; fuse them three ways — with the Sinkhorn-aligned permutation, with a random permutation of the same channels, and with plain weight averaging at the same $k$ — and measure old-task accuracy on a small held-out probe after each fusion. If random permutation and plain averaging do not clearly drop old-task accuracy relative to the aligned fusion, then the matching step is not what preserves old knowledge. Independently, sweep $k$ from 0 to 1 on the old-task probe; if no intermediate value keeps old-task accuracy near its pre-fusion level, then the interpolation itself breaks the old model's function.
Extended reading notes
Core claim
LwI's thesis is that parameter protection is the wrong unit of memory in a deep network: freezing or masking the weights an old task used scales with the number of tasks and isolates tasks from each other. The paper claims that if the model is reframed as a graph — channels as nodes, connections between adjacent layers as edges — then fusing a newly trained task model with the old one through graph matching can protect the knowledge of all tasks in the same fixed network. The fusion aligns channels before averaging: in shallow layers it permutes channels so that the most similar channels of the two models land on top of each other, strengthening shared features; in the final deep layer it deliberately matches the least similar channels, so each task keeps its own route. After alignment the weights are interpolated, and only the output heads are appended. Because deep networks activate sparsely, there are always unoccupied channels for the next task, so no parameter block has to be set aside and no old data has to be kept; the experiments report higher task-agnostic and task-aware accuracy than WSN, LwF, EWC, RWalk, and others, with lower forgetting rates, and the ablation studies show the deep-layer misalignment and the distillation term are the parts that matter most.
Load-bearing premise
The load-bearing premise is that permuting channels by weight similarity and averaging the aligned weights with a fixed coefficient $k$ preserves what the old model computes; the paper reports higher benchmark accuracy but never directly checks that the fused model reproduces the old model's input-output behavior, and the value of $k$ is never stated.
Editorial extensions
If this is right
- Data-free continual learning becomes viable in settings that forbid storing old inputs: LwI retains old-task knowledge without exemplars, and in most configurations it outperforms even methods that replay 2000 stored exemplars.
- Task-agnostic inference works without a task oracle: because each task ends up occupying its own channels, the fused model can be tested without knowing which task the input belongs to, something parameter-isolation methods such as WSN cannot do.
- Memory does not grow with the number of tasks: old knowledge is encoded in the channel arrangement of one fixed network rather than in reserved parameter blocks, so the per-task storage cost stays constant.
- Bigger networks and harder datasets amplify the gain: as capacity grows, channel occupancy becomes sparser, leaving more free pathways, which is the regime where the approach predicts its largest advantage.
- Fusing a model per task behaves like a knowledge-sharing mechanism, not just a protection mechanism: the reported per-task accuracy tables show some old tasks improve after later tasks are fused in.
Reading between the lines
- If the pathway story is correct, the permutation matrices are themselves a compact memory: an old task could in principle be recalled by inverting its channel alignment without retraining, an operation the paper does not explore.
- The choice to misalign only the final layer was made empirically from test accuracy, and the fusion coefficient $k$ is left unreported; a principled rule for both — for instance, tied to measured per-layer activation sparsity — would turn the method from a tuned recipe into a testable theory.
- Nothing in the experiments shows the mechanism on non-convolutional architectures; transferring the channel-alignment idea to transformers, where attention heads or neurons play the role of channels, is a direct next test, and the paper's own conclusion concedes that large models have not been validated.
- A sharp prediction follows from the activation-prominence figure: per-task channel activations in the fused model should be linearly separable, and task-agnostic accuracy should track that separability — a quantity that can be measured directly and would settle whether pathway protection is the real cause of the gains.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes LwI, a data-free continual learning method that aligns the channels of an old-task model and a newly trained model via graph matching (Sinkhorn-based soft matching) before fusing them with a convex combination. Matching maximizes similarity in shallow layers and minimizes it in deep layers, with the stated goal of protecting old-task pathways rather than parameters. The method is evaluated on CIFAR-100 and Tiny-ImageNet with ResNet32 and ResNet18, under task-aware and task-agnostic protocols, and is reported to outperform regularization-based, parameter-protection, and several exemplar-based baselines.
Significance. If the proposed mechanism holds, the work offers a genuinely different angle on continual learning: protecting learned pathways by channel alignment instead of freezing or masking parameters, while remaining data-free and parameter-efficient. The empirical evaluation is broad and consistent across architectures, datasets, and split counts, and the authors provide a code link and multiple ablations. However, the paper does not currently establish that channel alignment is the causal driver of the reported gains. The central fusion coefficient k is never reported, the layer configuration is selected from test-set accuracy, and the reported per-task tables measure accuracy after subsequent KD training rather than immediately after fusion. These gaps leave the pathway-protection interpretation unsupported even though the benchmark numbers may be correct.
major comments (4)
- [Algorithm 1 and Section 4.1] The fusion coefficient k is an input to Algorithm 1 and directly controls the fused weights via W_fusion = k * \tilde{W}_o + (1-k) * W_n, but its value is never reported in the main text, appendices, or implementation details. If k is close to 1, old-task accuracy is trivially preserved and the gains over baselines may reflect conservative interpolation rather than a new alignment mechanism. If k is near 0.5, the old model's function is genuinely averaged with the new model's function, and the role of alignment becomes critical but is not isolated. Please report k for every experiment, provide a sweep over k, and report old-task accuracy immediately after Algorithm 1, before the subsequent knowledge-distillation training whose results are shown in Tables 14-16.
- [Section 5.5 and Appendix C.10] The number of deep layers assigned minimum-similarity matching is selected by comparing test-set accuracies and then presented as part of the method: Appendix C.10 states 'we opted to minimize similarity matching for the final layer' based on the comparative experimental results. This is a post hoc selection on the test set, which inflates the reported performance and undermines the claim that LwI adaptively allocates pathways. Please fix the configuration a priori (for example, on a validation split), report the selection procedure, and evaluate the stability of the final-layer choice across seeds and datasets.
- [Section A.1, Eq. (6)] The appendix claims theoretical support from a first-order Taylor expansion, but the derivation stops at the expansion itself. No argument shows that maximum-similarity matching in shallow layers and minimum-similarity matching in deep layers reduce the first-order deviation term, nor what assumptions on the loss or feature distributions are needed for that reduction. As written, Eq. (6) does not support the claim that the proposed alignment preserves old-task input-output behavior. Please either supply a complete derivation or explicitly weaken the stated theoretical claim.
- [Section 5.5 and Tables 17-18] The ablations intended to isolate the task-diversion module show small differences relative to the full method (for example, CIFAR-100 ResNet18 task-aware 5-splits: 81.10 vs. 80.17; Tiny-ImageNet ResNet32 task-agnostic 5-splits: 22.21 vs. 20.80), and no significance tests are reported for these differences. Given that the unreported k and the knowledge-distillation module (Table 13) produce much larger swings, these ablations do not establish that pathway alignment, rather than interpolation or distillation, is responsible for the results. Please report confidence intervals or paired significance tests for the ablation differences and combine the ablation with a k-sweep.
minor comments (5)
- [Abstract and Section 1] The term 'pathway' is used in several senses (trajectories through the network, channels, allocation of channels), but it is never operationally defined or measured. Please clarify how a pathway is identified and quantified, since the paper's central narrative depends on this notion.
- [Algorithm 2 and Eq. (5)] Eq. (5) defines a Euclidean distance but calls it a similarity matrix, and Algorithm 2 later says the deep layers use '-R' without explaining how the distance values are converted to Sinkhorn-compatible similarities before negation. Please make the conversion (e.g., negation, sign flip, or rank-based) explicit.
- [Algorithm 2] The output line says 'the learned permutation metrics P i' but the algorithm returns permutation matrices; please correct the terminology. Also, the notation P is used for both the permutation matrix in Eq. (3) and the row/column-normalized matrix in Eq. (7); please disambiguate.
- [Section 5.1 and Appendix C] There are several typos and minor errors, including 'epoches' in Section 5.1, 'pathways protection is all you need' in Section 1, and near-verbatim duplicated paragraphs in Appendix C.4 and C.5. A careful proofreading pass is needed.
- [Figure 2] Figure 2 is used to argue that the method produces distinct activation pathways, but no quantitative criterion, error bar, or statistical comparison is given for the activation-level plots. Please add a quantitative summary (for example, per-task channel-selectivity scores) to support the visual claim.
Circularity Check
No significant circularity: the benchmark results are external and not derived from the method's own fitted quantities.
full rationale
LwI's central claim is that channel-aligned fusion (maximum-similarity matching in shallow layers, minimum-similarity matching in deep layers) followed by knowledge distillation mitigates catastrophic forgetting. The evidence is task-aware and task-agnostic accuracy on CIFAR-100 and Tiny-ImageNet compared against published baselines (Table 1), which are external measurements rather than quantities reconstructed from the method's own outputs. Algorithm 1's fusion coefficient k is an unreported hyperparameter, but this is a reproducibility concern, not a circular reduction: reporting k would not change the fact that the results are external benchmark accuracies. Similarly, the choice to apply minimum-similarity matching only in the final layer was made after comparing test-set ablations (Tables 2-3 and C.10); this is post-hoc selection on the evaluation set, a correctness risk, not a circular derivation. The only co-author citation (Kang et al. 2022b in Appendix A.1, sharing author Bo Han) supplies a standard first-order Taylor expansion and is not load-bearing; the analysis does not assume the paper's target result. No equation reduces to its own input, and no fitted parameter is renamed as a predicted quantity. The derivation is therefore self-contained with respect to circularity.
Assumptions & free parameters
free parameters (5)
- Fusion coefficient k
- Knowledge distillation weight lambda
- Sinkhorn temperature tau and threshold tau_min
- Number of deep layers with minimum-similarity matching =
1 (final layer)
- Similarity measure =
Euclidean distance
assumptions (4)
- domain assumption Deep networks exhibit sparse activation, so different tasks can occupy different channels without exhausting capacity.
- domain assumption Shallow layers of different-task models share common features while deep layers are task-specific.
- ad hoc to paper Per-layer channel alignment with Euclidean distance on weights preserves the function of the old model after fusion.
- standard math Sinkhorn soft matching gives a valid permutation for model fusion.
invented entities (1)
-
Task-specific pathway
Cite this review
Pith. "Pith review of Learning without Isolation: Pathway Protection for Continual Learning." pith.science (2026). https://pith.science/paper/XVY7VGW2
@misc{pith2026250518568,
author = {Pith},
title = {Pith review of: Learning without Isolation: Pathway Protection for Continual Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/XVY7VGW2}},
note = {Machine review of arXiv:2505.18568}
}
read the original abstract
Deep networks are prone to catastrophic forgetting during sequential task learning, i.e., losing the knowledge about old tasks upon learning new tasks. To this end, continual learning(CL) has emerged, whose existing methods focus mostly on regulating or protecting the parameters associated with the previous tasks. However, parameter protection is often impractical, since the size of parameters for storing the old-task knowledge increases linearly with the number of tasks, otherwise it is hard to preserve the parameters related to the old-task knowledge. In this work, we bring a dual opinion from neuroscience and physics to CL: in the whole networks, the pathways matter more than the parameters when concerning the knowledge acquired from the old tasks. Following this opinion, we propose a novel CL framework, learning without isolation(LwI), where model fusion is formulated as graph matching and the pathways occupied by the old tasks are protected without being isolated. Thanks to the sparsity of activation channels in a deep network, LwI can adaptively allocate available pathways for a new task, realizing pathway protection and addressing catastrophic forgetting in a parameter-efficient manner. Experiments on popular benchmark datasets demonstrate the superiority of the proposed LwI.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[2]
This blending is likely to result in outcomes inferior to our method. WSN requires a mask when dealing with various tasks, limiting its applicability to task-aware testing. Additionally, as the number of tasks increases, the reduction in learnable parameters diminishes its effectiveness. iCaRL and LUCIR methods benefit from partial datasets of all previou...
-
[3]
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635,
-
[4]
=O( NLX 1 ( N NL )4) =O( 1 N 3 L N 4). (8) 13 Learning without Isolation: Pathway Protection for Continual Learning Algorithm 2Adaptive algorithm Input:Similarity Matrix R, Total number of iterations E, Parameter τ for control the difference between Hungarian algorithm and Sinkhorn algorithm ; foreach rounde= 1, ..., Edo ifP i not convergedthen ifτ <=τ mi...
work page 2022
-
[7]
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. InProceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001–2010,
2001
-
[9]
A Unified and General Framework for Continual Learning
Wang, Z., Li, Y ., Shen, L., and Huang, H. A unified and general framework for continual learning.arXiv preprint arXiv:2403.13249,
-
[10]
Adamerging: Adaptive model merging for multi-task learning.arXiv preprint arXiv:2310.02575,
Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning.arXiv preprint arXiv:2310.02575,
-
[11]
Over- coming generic knowledge loss with selective parameter update
Zhang, W., Janson, P., Aljundi, R., and Elhoseiny, M. Over- coming generic knowledge loss with selective parameter update. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24046– 24056, 2024a. Zhang, X., Song, D., and Tao, D. Continual learning on graphs: Challenges, solutions, and opportunities.arXiv preprint arXiv...
-
[512]
Table 6.Comparison between different architecture of models
A single fully connected (FC) layer with 512 units was employed, and the output was divided into multiple heads based on the task requirements. Table 6.Comparison between different architecture of models. Architecture Total parameters Model size ResNet32 466,896 1.84MB ResNet18 11,220,132 42.87MB B.4. Datasets splits details CIFAR-100 dataset contains 100...
work page 2017
Show all 18 references
-
[1996]
Continual learning via neural pruning.arXiv preprint arXiv:1903.04476,
Golkar, S., Kagan, M., and Cho, K. Continual learning via neural pruning.arXiv preprint arXiv:1903.04476,
1903 arXiv
-
[2000]
Gradient projection memory for continual learning.arXiv preprint arXiv:2103.09762,
Saha, G., Garg, I., and Roy, K. Gradient projection memory for continual learning.arXiv preprint arXiv:2103.09762,
-
[2008]
Higher layers need more lora experts.arXiv preprint arXiv:2402.08562,
Gao, C., Chen, K., Rao, J., Sun, B., Liu, R., Peng, D., Zhang, Y ., Guo, X., Yang, J., and Subrahmanian, V . Higher layers need more lora experts.arXiv preprint arXiv:2402.08562,
-
[2010]
The method of deep graph matching (Yu et al.,
and sequential Monte Carlo sampling (Leordeanu et al., 2012). The method of deep graph matching (Yu et al.,
2012
-
[2013]
Multimodal end-to-end sparse model for emotion recognition.arXiv preprint arXiv:2103.09666,
Dai, W., Cahyawijaya, S., Liu, Z., and Fung, P. Multimodal end-to-end sparse model for emotion recognition.arXiv preprint arXiv:2103.09666,
-
[2016]
Theoretical Supports A.1
11 Learning without Isolation: Pathway Protection for Continual Learning A. Theoretical Supports A.1. Analysis We analyze one layer of deep network channel, and first-order Taylor expansion is used for analysis (Kang et al., 2022b): L(G ℓ (Z ′ ℓ), y)≈ L(G ℓ (Zℓ), y) + CℓX c=1 ...
2008
-
[2017]
Kang, H., Mina, R. J. L., Madjid, S. R. H., Yoon, J., Hasegawa-Johnson, M., Hwang, S. J., and Yoo, C. D. Forget-free continual learning with winning subnetworks. InInternational Conference on Machine Learning, pp. 10734–10750. PMLR, 2022a. Kang, M., Park, J., and Han, B. Class...
-
[2018]
computes the regularization term online by accumulating the sensitivity (gradient magnitude) of the learning function. RWalk(Chaudhry et al., 2018): This method integrates the approximation of the Fisher information matrix and online path integral into a single algorithm to co...
2018
-
[2020]
Cho, M., Lee, J., and Lee, K. M. Reweighted random walks for graph matching. InComputer Vision–ECCV 2010: 11th European Conference on Computer Vision, Herak- lion, Crete, Greece, September 5-11, 2010, Proceedings, Part V 11, pp. 492–505. Springer,
2010
-
[2022]
and Le, Q
Zoph, B. and Le, Q. V . Neural architecture search with reinforcement learning.arXiv preprint arXiv:1611.01578,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.