Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective

Aojun Lu; Hangjie Yuan; Tao Feng; Yanan Sun

REVIEW 3 major objections 4 minor 1 cited by

Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Under equal parameter budgets, deeper networks are more plastic and wider networks are more stable, and Dual-Arch combines both shapes to improve continual learning with up to 87 percent fewer parameters.

desk verdict Useful plug-in with a shaky central claim: the depth-for-plasticity evidence is within noise, but the dual-architecture engineering is real. read the letter →

arxiv 2506.03951 v2 pith:K3HJZOWD submitted 2025-06-04 cs.LG cs.CV

Aojun Lu , Hangjie Yuan , Tao Feng , Yanan Sun This is my paper · ORCID

classification cs.LGcs.CV

keywords continuallearningstability-plasticitydilemmanetworkarchitecturedepth-widthtrade-offknowledgedistillationclass-incrementalcatastrophicforgettingparameterefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Continual learning must reconcile two conflicting goals: holding onto what was learned before and absorbing what is new. The paper claims that this stability-plasticity dilemma exists not only in parameters but also in architecture, and that under an equal parameter budget deeper networks learn new tasks better while wider networks forget less. To exploit this, it proposes Dual-Arch, a plug-in framework with two independent learners: a thin deep plastic learner that is trained on the current task and then frozen, and a wide shallow stable learner that absorbs the new knowledge through knowledge distillation while continuing to run the normal continual-learning loss. On class-incremental versions of CIFAR100 and ImageNet100, Dual-Arch consistently improves the accuracy and reduces the forgetting of five established continual-learning methods while using 33% to 87% fewer parameters. If the architectural trade-off is general, this gives continual-learning practitioners a cheap and method-agnostic way to shift the balance without redesigning their training algorithms.

What carries the argument

The central object is the Dual-Arch framework, which replaces a single uniform network with two independently shaped learners. Sta-Net is the stable learner: it keeps ResNet-18's width, halves the residual depth, and widens the penultimate feature map feeding the classifier; Pla-Net is the plastic learner: it keeps the depth and narrows the width so the two learners have comparable parameter counts. The carrying mechanism is knowledge distillation: Pla-Net is trained on the current task with plain cross-entropy, frozen, and used as a teacher; the stable learner minimizes a composite loss $\mathcal{L}_{\mathrm{stable}} = \alpha \mathcal{L}_{\mathrm{CE}} + (1-\alpha) \mathcal{L}_{\mathrm{KD}} + \mathcal{L}_{\mathrm{CL}}$, where $\mathcal{L}_{\mathrm{KD}}$ is a temperature-scaled KL divergence between the soft outputs of the teacher and student. This mechanism lets new class knowledge enter the stable network without rewriting its old decision boundaries. The architecture split is load-bearing: the paper's ablation shows that removing the plastic learner or using non-specialized architectures for either learner reduces performance.

What would settle it

An experiment that would settle it: compare equal-parameter deep-narrow and wide-shallow versions of at least two different backbones under at least three continual-learning methods, measuring new-task accuracy and forgetting separately; if a wide-shallow network ever forgets less while also learning new tasks better than a deep-narrow one, the claimed trade-off fails.

Watch

Extended reading notes

Core claim

Under an equal parameter constraint, network shape separates stability from plasticity: the paper reports that a deeper-but-narrower ResNet-18 raises accuracy on newly learned tasks while also increasing forgetting, while a wider-but-shallower variant lowers average forgetting at the cost of new-task accuracy. This is framed as an architectural-level stability-plasticity dilemma, meaning that a single uniform architecture must compromise between the two objectives. Dual-Arch is the proposed resolution: it assigns the plasticity role to a thin deep network (Pla-Net) that is freshly trained on each task's data and frozen, and the stability role to a wide shallow network (Sta-Net) that is the main continual learner; a temperature-scaled knowledge-distillation loss transfers the plastic learner's soft outputs into the stable learner, combined with the stable learner's own cross-entropy and continual-learning losses. In experiments with five continual-learning methods (iCaRL, WA, DER, Foster, and MEMO) on CIFAR100/10, CIFAR100/20, ImageNet100/10, and ImageNet100/20, Dual-Arch improves last accuracy and average incremental accuracy with at least 33% fewer parameters, and up to 87% fewer in the most compact configuration. The paper interprets this as evidence that depth and width can be assigned by role rather than balanced inside one network.

Load-bearing premise

The load-bearing premise is that the depth-width trade-off, measured mainly on ResNet with a single continual-learning method on ImageNet100, holds generally across methods, datasets, and architectures, and that the specific wide-shallow stable learner and deep-thin plastic learner are the right way to exploit it.

Editorial extensions

If this is right

Plugging Dual-Arch into an existing continual-learning method should raise its last-task accuracy and average incremental accuracy while lowering final average forgetting, with no change to the method's own loss.
Because only the stable learner is used at inference time, the dual-network training scheme should not increase test-time computation relative to a single network.
In memory-constrained settings, the parameter savings of 33% to 87% mean the same continual learner can fit in a smaller footprint, which the paper demonstrates by scaling widths down and still beating baselines.
If the depth-width trade-off holds, architecture search for continual learning can target role-specific shapes instead of one uniform architecture, which is the direct corollary the paper draws.
The paper's validation on a transformer-style backbone suggests the depth-width role split is not exclusive to convolutional ResNets, though the evidence there is limited to one setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension: run the same dual-role split in task-incremental or domain-incremental settings where the task identity is known or boundaries are blurred; the paper reports one blurry-boundary benchmark, but the main results are class-incremental, so the generalisation is not yet established.
A sharper isolation experiment would hold total parameters constant and vary only the depth-width split between the two learners; this would separate the benefit of the dual split from the benefit of simply using fewer, smaller parameters.
If the architectural trade-off is a general property of overparameterised networks, it may also apply to fine-tuning large pretrained models, where a wide stable adapter and a deep plastic adapter could serve the same two roles; the paper does not test this.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: Dual-Arch is a useful plug-in that improves continual learning accuracy across several methods and benchmarks while cutting parameters, and the paper is honest about its training-time cost. But the central architectural claim—deeper nets for plasticity, wider for stability—rests on weak evidence, especially the depth leg.

What's new: pairing a wide-shallow stable net with a deep-thin plastic net and transferring knowledge via distillation. Prior work showed wider nets forget less (Mirzadeh et al.) and there are dual-model complementary learning systems, but the specific architectural split and its parameter efficiency are new. The experiments are broad: five CL methods, four CIFAR100/ImageNet100 splits, plus CIFAR100/50, GCIL, and a ViT transfer. Parameter efficiency is a real plus for memory-constrained settings, and the code is public. The ablation shows both components help, and the confusion matrices indicate reduced task-recency bias.

The soft spot is Section 3. The stability leg (wider-shallower) is consistent across ResNet, MLP, and ViT appendices, though the ViT table and ArchCraft rows have no error bars. The plasticity leg (deeper-thinner) is one data point in Table 1: ResNet-26-52 gains +0.27 AAN and +0.26 FAF over ResNet-18, with overlapping standard deviations and no paired test or confidence interval. The authors themselves call it 'slight' and 'marginal.' If the depth effect is noise, Pla-Net loses its stated basis, though the dual-architecture distillation might still help through auxiliary knowledge or diversity. The paper does not isolate that mechanism.

The '87% more compact' headline depends on the parameter-counting protocol in A.1, which sums current and last models for replay methods; defensible, but the main text should be clearer. Also, training time is 1.39–1.77× longer, which the paper reports.

Bottom line: the engineering contribution is real and likely reproducible, but the conceptual framing is oversold. A revision that strengthens the depth-plasticity evidence (more architectures, paired CIs) or reframes the contribution as 'dual specialized architectures help CL' would be solid. This paper is for CL researchers wanting a cheap accuracy/parameter boost, and for anyone interested in architectural inductive biases. It deserves a serious referee.

Referee Report

3 major / 4 minor

Summary. The paper studies how network architecture affects the stability-plasticity trade-off in class-incremental learning. Through experiments in Sec. 3, it reports that, under an equal parameter budget, deeper ResNets show better plasticity (higher accuracy on new tasks) and wider/shallower ResNets show better stability (lower forgetting). Based on this, the paper proposes Dual-Arch, a plug-in framework that uses two small networks: a deep-thin 'plastic learner' trained on new data and distilled into a wide-shallow 'stable learner' that runs the actual CL method. Dual-Arch is validated by plugging it into five CL methods (iCaRL, WA, DER, Foster, MEMO) on CIFAR100 and ImageNet100, reporting improved accuracy with 33%-87% fewer parameters, supplemented by ablations, parameter-efficiency plots, computational cost analysis, confusion diagrams, ViT and MLP extensions, long task sequences, and a GCIL benchmark.

Significance. If the findings are robust, the paper offers a practically useful and conceptually interesting result: architectural shape can serve as an independent axis for balancing stability and plasticity, and a carefully chosen pair of small specialized networks can outperform a single larger network. The paper is thorough in scope: it covers five CL methods, two datasets, and additional settings (ViT, MLP, 50-task, blurry boundaries), and it provides ablations, parameter/computation analyses, and public code. However, the load-bearing empirical claim in Sec. 3—that deeper networks improve plasticity—is not statistically established in the central ResNet/iCaRL experiment, the main results in Table 2 lack error bars, and the ablation in Sec. 5.3 does not isolate the contribution of the architectural specialization from the dual-learner distillation mechanism. These gaps currently limit the strength of the paper's central claim, though they are addressable with additional experiments and statistical reporting.

major comments (3)

[Sec. 3.2, Table 1] The evidence for the depth-to-plasticity direction is not statistically established. For ResNet-26-52 versus ResNet-18, the AAN gain is +0.27 (86.68±0.70 vs 86.41±0.60) and the FAF change is +0.26 (36.02±1.79 vs 35.76±1.62), both smaller than the reported standard deviations, and no paired test or confidence interval is provided. The text itself describes the effect as 'slight' and 'marginal.' Since Sec. 4.3 uses this result to justify Pla-Net as a 'deep and thin' architecture, the central design choice rests on a fragile empirical basis. The wide-shallow direction is more consistently supported (ResNet-10-96 in Table 1 and the MLP/ViT appendices), but the depth direction lacks comparable support; the ViT appendix does not even test a deeper-thinner variant. I recommend reporting paired statistics, varying datasets/methods, or directly demonstrating that depth changes drive the Dual-Arch improvement.
[Sec. 5.1, Table 2] The main results are single-seed runs with no error bars or significance tests. Some improvements are small, e.g., Foster on ImageNet100/20 (LA +0.46, AIA +0.86) and iCaRL on ImageNet100/20 (LA +1.00). Given the seed sensitivity visible in Table 1 (FAF std up to 1.79), these differences could be within noise. Because the abstract and conclusion claim that Dual-Arch 'enhances the performance' of existing CL methods, the authors should provide multiple seeds and error bars at least for the configurations with small improvements, and ideally a paired significance test across methods.
[Sec. 5.3, Table 3] The ablation does not include a uniform-architecture dual-learner control. The reported comparisons are among Sta-Net and Pla-Net pairs (plus a single-learner Sta-Net row), but there is no condition with, e.g., two ResNet-18s in the same distillation framework. Without this control, the observed gains cannot be attributed specifically to the architectural specialization; they may be due to the dual-learner distillation mechanism already present in prior work such as MKD and Hare & Tortoise (cited in Sec. 2.3). Adding a matched uniform-architecture dual-learner baseline is necessary to support the paper's architectural-level contribution.

minor comments (4)

[Table 2] The ArchCraft rows report LA and AIA but not FAF; please either provide FAF for ArchCraft or state why it is omitted, since forgetting is one of the three headline metrics.
[Sec. 5.4, Fig. 3] The 'up to 87% fewer parameters' claim is based on the extreme point in the parameter series (14.6M vs 112.2M). The 'up to' qualifier makes the statement technically correct, but the accompanying text should make clear that this is a corner-case comparison and that the typical reduction in Table 2 is 33-52%.
[Sec. 4.4 and Appendix A.1] The values of the temperature t and the balance alpha are given in the appendix (t=4 for CIFAR100, t=3 for ImageNet100, alpha=0.5), but no sensitivity analysis is reported. A short discussion of sensitivity to alpha and t would strengthen the robustness of the method.
[Sec. 5.7, Fig. 5] The task confusion matrices in Figure 5 are difficult to read in print; consider providing higher-resolution versions or numerical summaries of off-diagonal mass.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the architectural trade-off is an empirical input, and Dual-Arch is evaluated as a designed method against external benchmarks.

full rationale

The paper's derivation chain is empirical rather than formal, and none of the load-bearing steps reduce to their own inputs. Section 3.1-3.2 measures stability and plasticity of ResNet variants (Table 1) and reports that wider/shallower variants improve stability while deeper/narrower variants slightly improve plasticity; these measurements are inputs to the design, not outputs that are later 'predicted.' Dual-Arch's Sta-Net and Pla-Net in Section 4.3 are then chosen to embody the observed stable and plastic architectures. This is a standard engineering design loop, not a circular derivation: the method's success is tested on CIFAR100 and ImageNet100 with five external CL methods, and the ablations in Table 3 (removing the plastic learner, swapping architectures) directly test the design assumptions rather than assuming them. The paper does not fit parameters to the benchmark outcomes and then report those outcomes as predictions; hyperparameters follow PyCIL conventions, and temperature values are fixed constants. The self-citation to ArchCraft (Lu et al., 2024) appears in related work and as a comparison baseline, but the central stability-plasticity claim is justified by the paper's own experiments, not by that citation. The small and statistically weak deeper-narrower effect in Table 1 is a legitimate concern about support for the plasticity claim, but it is a correctness/statistical issue, not circularity. No step in the paper exhibits self-definition, fitted-input-as-prediction, imported uniqueness, ansatz smuggling, or renaming of known results.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or formal entities. The free parameters are standard hyperparameters and hand-chosen architectural designs. The main axiomatic burden is the generality of the empirical trade-off, which is only weakly supported by the presented evidence.

free parameters (4)

Loss balance alpha = 0.5
Weight balancing CE and KD loss in Eq. (3), set to 0.5 following Hinton et al. (2015), not tuned.
Distillation temperature t = 4 (CIFAR100), 3 (ImageNet100)
Temperature for soft outputs in distillation, chosen per dataset as stated in Appendix A.1.
Sta-Net architecture design = half residual blocks, width 64, 4x4 avg pooling
Architecture for the stable learner, selected based on the empirical trade-off in Section 3.
Pla-Net architecture design = width 42, full depth
Architecture for the plastic learner, chosen to roughly match Sta-Net parameter count while keeping depth.

assumptions (3)

domain assumption Standard class-incremental learning benchmarks (CIFAR100, ImageNet100) and metrics (LA, AIA, FAF) faithfully measure continual learning performance.
The paper relies on these benchmarks and metrics to validate the proposed method; if they are not representative, the conclusions may not generalize.
domain assumption Knowledge distillation can transfer learned knowledge between networks with different architectures.
This is a standard assumption in the distillation literature (Gou et al., 2021), cited by the authors without independent validation in this paper.
ad hoc to paper The architectural trade-off observed with ResNet on ImageNet100 using iCaRL generalizes to other methods, datasets, and architectures.
This is the key load-bearing assumption. Section 3 provides only a single-method, small-effect-size evidence base, yet the entire design of Sta-Net and Pla-Net depends on this generality.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective." pith.science (2026). https://pith.science/paper/K3HJZOWD

@misc{pith2026250603951,
  author       = {Pith},
  title        = {Pith review of: Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3HJZOWD}},
  note         = {Machine review of arXiv:2506.03951}
}

read the original abstract

The quest for Continual Learning (CL) seeks to empower neural networks with the ability to learn and adapt incrementally. Central to this pursuit is addressing the stability-plasticity dilemma, which involves striking a balance between two conflicting objectives: preserving previously learned knowledge and acquiring new knowledge. While numerous CL methods aim to achieve this trade-off, they often overlook the impact of network architecture on stability and plasticity, restricting the trade-off to the parameter level. In this paper, we delve into the conflict between stability and plasticity at the architectural level. We reveal that under an equal parameter constraint, deeper networks exhibit better plasticity, while wider networks are characterized by superior stability. To address this architectural-level dilemma, we introduce a novel framework denoted Dual-Arch, which serves as a plug-in component for CL. This framework leverages the complementary strengths of two distinct and independent networks: one dedicated to plasticity and the other to stability. Each network is designed with a specialized and lightweight architecture, tailored to its respective objective. Extensive experiments demonstrate that Dual-Arch enhances the performance of existing CL methods while being up to 87% more compact in terms of parameters. Code: https://github.com/byyx666/Dual-Arch.

Figures

Figures reproduced from arXiv: 2506.03951 by the authors.

**Figure 1.** Left. (a) The average forgetting and (b) the accuracy on the new task of ResNet-18 and its wider and shallower variant. Details are presented in Sec. 3. Right. While existing research mainly optimizes weights (represented by node colors) for the stability-plasticity trade-off at the parameter level, this study proposes a novel insight for extending this trade-off to the architectural level. sess enhanced representat… view at source ↗

**Figure 2.** The formulation of the traditional CL paradigm and CL with Dual-Arch (Ours). Dual-Arch (1) employs two [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Performance of CL vs. Number of Parameters using DER and Foster on CIFAR100/10. 5.4. Parameter Efficiency Analysis To assess the parameter efficiency more comprehensively, we vary the parameter counts of Dual-Arch and ResNet-18 by reducing the network width by a quarter and a half. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: The performance of Dual-Arch and two baselines using DER on CIFAR100/10. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Task confusion matrices after learning the final [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Task confusion matrices after learning the final task of different CL methods w/ and w/o Dual-Arch plugged in on [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning
cs.LG 2025-08 conditional novelty 4.0 of 10

Adding zeroth- and first-order flatness penalties to continual learning losses yields small consistent accuracy gains across seven methods, with the gated C-Flat++ variant at roughly 30% of the update cost.

Reference graph

Works this paper leans on

17 extracted references · 7 canonical work pages · cited by 1 Pith paper

[5]

We utilize a temperature factor for Dual-Arch oft= 4for CIFAR100 andt= 3for ImageNet-100

For other experiments, we adhere to a fixed seed of 1993, consistent with established conventions (Rebuffi et al., 2017; Zhou et al., 2023a). We utilize a temperature factor for Dual-Arch oft= 4for CIFAR100 andt= 3for ImageNet-100. Details about Parameter Counts.We compute the sum of the parameter counts of all used models for each incremental step and re...

work page 1993
[7]

SepViT: Separable Vision Transformer

Li, W., Wang, X., Xia, X., Wu, J., Li, J., Xiao, X., Zheng, M., and Wen, S. Sepvit: Separable vision transformer. arXiv preprint arXiv:2203.15380,

work page Pith review arXiv
[8]

I., Chaudhry, A., Yin, D., Hu, H., Pascanu, R., Gorur, D., and Farajtabar, M

Mirzadeh, S. I., Chaudhry, A., Yin, D., Hu, H., Pascanu, R., Gorur, D., and Farajtabar, M. Wide neural networks forget less catastrophically. InInternational Conference on Machine Learning, pp. 15699–15717. PMLR, 2022a. Mirzadeh, S. I., Chaudhry, A., Yin, D., Nguyen, T., Pascanu, R., Gorur, D., and Farajtabar, M. Architecture matters in continual learning...

arXiv
[9]

Table 9: The LA (%) on GCIL-CIFAR-100 with different buffer sizes.Boldedindicates the best

Our findings indicate that Dual-Arch consistently enhances CL performance in this scenario, underscoring its broad applicability. Table 9: The LA (%) on GCIL-CIFAR-100 with different buffer sizes.Boldedindicates the best. Note that the benchmark settings follow (Arani et al., 2022). Method Buffer Size 500 Buffer Size 1000 ER (Rostami et al.,

work page 2022
[11]

Both ViTs are evaluated on ImageNet-100/10 using iCaRL as the learning method (Rebuffi et al., 2017)

Additionally, we design a wider yet shallower variant with a width of 49 and depth of 5, which has a parameter count comparable to the default design. Both ViTs are evaluated on ImageNet-100/10 using iCaRL as the learning method (Rebuffi et al., 2017). Note that the training settings are consistent with Sec. 5.1, but the learning rate and optimizer are ad...

work page 2017
[13]

Yoon, J., Yang, E., Lee, J., and Hwang, S. J. Lifelong learning with dynamically expandable networks.arXiv preprint arXiv:1708.01547,

arXiv
[16]

Table 8: The LA and AIA (%) using five state-of-the-art CL methods on CIFAR100/50.Boldedindicates the best

These results demonstrate that Dual-Arch consistently outperforms the baselines in this challenging setting, thereby underscoring its generality. Table 8: The LA and AIA (%) using five state-of-the-art CL methods on CIFAR100/50.Boldedindicates the best. Method iCaRL W A DER Foster MEMO LA AIA LA AIA LA AIA LA AIA LA AIA Original 45.30 63.99 42.12 58.26 55...

work page arXiv 2020
[1995]

E., Chassang, A., Gatta, C., and Bengio, Y

Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y . Fitnets: Hints for thin deep nets.arXiv preprint arXiv:1412.6550,

arXiv

Show all 17 references

[2015]

Memory- efficient incremental learning through feature adaptation

Iscen, A., Zhang, J., Lazebnik, S., and Schmid, C. Memory- efficient incremental learning through feature adaptation. InComputer Vision–ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pp. 699–715. Springer,

2020
[2017]

Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. InProceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001–2010,

2001
[2018]

and Zisserman, A

Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556,

arXiv
[2019]

A., Rabinowitz, N

Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks.arXiv preprint arXiv:1606.04671,

arXiv
[2020]

and Cho, K

Cha, S. and Cho, K. Hyperparameters in continual learning: a reality check.arXiv preprint arXiv:2403.09066,
[2021]

Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

arXiv
[2022]

Zeroflow: Overcoming catastrophic forgetting is easier than you think.arXiv preprint arXiv:2501.01045,

Feng, T., Li, W., Zhu, D., Yuan, H., Zheng, W., Zhang, D., and Tang, J. Zeroflow: Overcoming catastrophic forgetting is easier than you think.arXiv preprint arXiv:2501.01045,

arXiv
[2024]

Efficient lifelong learning with a-gem.arXiv preprint arXiv:1812.00420,

Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a-gem.arXiv preprint arXiv:1812.00420,

arXiv
[2025]

J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y

Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y . An empirical investigation of catastrophic for- getting in gradient-based neural networks.arXiv preprint arXiv:1312.6211,

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [5]

We utilize a temperature factor for Dual-Arch oft= 4for CIFAR100 andt= 3for ImageNet-100

For other experiments, we adhere to a fixed seed of 1993, consistent with established conventions (Rebuffi et al., 2017; Zhou et al., 2023a). We utilize a temperature factor for Dual-Arch oft= 4for CIFAR100 andt= 3for ImageNet-100. Details about Parameter Counts.We compute the sum of the parameter counts of all used models for each incremental step and re...

work page 1993

[2] [7]

SepViT: Separable Vision Transformer

Li, W., Wang, X., Xia, X., Wu, J., Li, J., Xiao, X., Zheng, M., and Wen, S. Sepvit: Separable vision transformer. arXiv preprint arXiv:2203.15380,

work page Pith review arXiv

[3] [8]

I., Chaudhry, A., Yin, D., Hu, H., Pascanu, R., Gorur, D., and Farajtabar, M

Mirzadeh, S. I., Chaudhry, A., Yin, D., Hu, H., Pascanu, R., Gorur, D., and Farajtabar, M. Wide neural networks forget less catastrophically. InInternational Conference on Machine Learning, pp. 15699–15717. PMLR, 2022a. Mirzadeh, S. I., Chaudhry, A., Yin, D., Nguyen, T., Pascanu, R., Gorur, D., and Farajtabar, M. Architecture matters in continual learning...

arXiv

[4] [9]

Table 9: The LA (%) on GCIL-CIFAR-100 with different buffer sizes.Boldedindicates the best

Our findings indicate that Dual-Arch consistently enhances CL performance in this scenario, underscoring its broad applicability. Table 9: The LA (%) on GCIL-CIFAR-100 with different buffer sizes.Boldedindicates the best. Note that the benchmark settings follow (Arani et al., 2022). Method Buffer Size 500 Buffer Size 1000 ER (Rostami et al.,

work page 2022

[5] [11]

Both ViTs are evaluated on ImageNet-100/10 using iCaRL as the learning method (Rebuffi et al., 2017)

Additionally, we design a wider yet shallower variant with a width of 49 and depth of 5, which has a parameter count comparable to the default design. Both ViTs are evaluated on ImageNet-100/10 using iCaRL as the learning method (Rebuffi et al., 2017). Note that the training settings are consistent with Sec. 5.1, but the learning rate and optimizer are ad...

work page 2017

[6] [13]

Yoon, J., Yang, E., Lee, J., and Hwang, S. J. Lifelong learning with dynamically expandable networks.arXiv preprint arXiv:1708.01547,

arXiv

[7] [16]

Table 8: The LA and AIA (%) using five state-of-the-art CL methods on CIFAR100/50.Boldedindicates the best

These results demonstrate that Dual-Arch consistently outperforms the baselines in this challenging setting, thereby underscoring its generality. Table 8: The LA and AIA (%) using five state-of-the-art CL methods on CIFAR100/50.Boldedindicates the best. Method iCaRL W A DER Foster MEMO LA AIA LA AIA LA AIA LA AIA LA AIA Original 45.30 63.99 42.12 58.26 55...

work page arXiv 2020

[8] [1995]

E., Chassang, A., Gatta, C., and Bengio, Y

Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y . Fitnets: Hints for thin deep nets.arXiv preprint arXiv:1412.6550,

arXiv