{"id":"14e83bf0-6cd1-4420-9535-3f43ba3e443d","arxiv_id":"2412.02865","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A continual learning method that combines focal contrastive learning with fixed neural-collapse prototypes and a distillation loss achieves state-of-the-art accuracy in memory-free class- and task-incremental learning.","lead":"This paper introduces two new loss functions that help continual learning models remember old knowledge while learning new tasks. The method works especially well without any stored memory, which could benefit privacy-sensitive applications.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set hyperparameter selection in Appendix A threatens the head-to-head memory-free SOTA claim.","rationale":"The reader's weakest_assumption focuses on the fixed-prototype requirement (known K). That limitation is acknowledged in the paper's Conclusion and does not invalidate the evaluated benchmarks, where K is known; it constrains general applicability but not the reported comparisons. The reader's rationale also flags hyperparameter selection as a significant caveat, but it is not the stated weakest assumption. I agree with the CONDITIONAL verdict: the method is coherent and the losses are well-defined, but the empirical SOTA claim is not yet trustworthy. The test-set hyperparameter selection is the single most load-bearing concern because it directly affects the head-to-head numbers that comprise the central claim. If the concern is confirmed, the reported improvements could be artifacts of tuning on the test set. If re-running with proper validation selection preserves the margins, the claim would be substantially strengthened. No code is released, which further impedes independent verification, but the selection issue is more specific and more damaging. Therefore, the reader's CONDITIONAL verdict remains appropriate, with the condition that the authors demonstrate results under unbiased hyperparameter selection and release code.","tokens_in":17263,"tokens_out":3084,"duration_ms":53918,"concrete_test":"Re-run the memory-free (buffer=0) Seq-Cifar-10 and Seq-Cifar-100 Class-IL experiments with all hyperparameters selected on the 10% training-data validation split, not on test accuracy, and report test results. Then compare against Co2L run under the same validation-based tuning protocol. If the accuracy gap to Co2L shrinks substantially (e.g., from 10.37 points to below 3 points or changes sign), the reported SOTA claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that FNC2+HSD surpasses SOTA in memory-free continual learning (Abstract, Section 5.2). The strongest head-to-head evidence is the buffer=0 comparison against Co2L, e.g., 69.26% vs 58.89% on Seq-Cifar-10 Class-IL (Table 2). However, Appendix A states: 'The selections of these hyperparameters are based on the average test accuracy over five independent trials' (hyperparameter table caption). This is a direct admission that the model's hyperparameters (gamma, temperatures, warm-up epochs, etc.) were chosen using test-set performance, while the Co2L baselines use their originally reported hyperparameters. The 10% validation split mentioned earlier appears contradictory and is not the basis for the reported selections. This introduces selection bias: the reported margins may reflect test-set overfitting in hyperparameter choice rather than a genuine algorithmic advantage. The K-known limitation (Section 4.2, Conclusion) is real but acknowledged and does not affect the fixed-task benchmarks evaluated. The memory-free evaluation protocol (classifier trained on last-task samples plus buffer) is shared with Co2L, so relative comparison is likely fair on that axis. The test-set selection is the most load-bearing concern because it directly compromises the empirical superiority claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two loss functions for continual representation learning: FNC2, a focal contrastive loss that combines sample-sample ('softness') and sample-prototype ('hardness') terms using fixed simplex ETF prototypes and cross-task prototype negatives, and HSD, a distillation loss that mixes instance-relation distillation with sample-prototype relation distillation under a time-varying coefficient. The method is evaluated in class-incremental and task-incremental settings on Seq-Cifar-10, Seq-Cifar-100, and Seq-Tiny-ImageNet with buffer sizes 0, 200, and 500, reporting state-of-the-art average accuracy in the memory-free and small-buffer regimes, with ablations attributing gains to the plasticity loss, the distillation loss, and the pseudo-replay prototypes.","tokens_in":17478,"tokens_out":4509,"duration_ms":45340,"significance":"If the empirical claims hold, the paper makes a useful and timely contribution: it demonstrates that fixed ETF prototypes can serve as pseudo-replay in memory-free continual learning, and the decomposition into plasticity and stability losses is clean and well ablated. The manuscript includes extensive comparisons, standard deviations over five trials, and memory-free evaluations that are rare among recent contrastive continual learning methods. The main caveat is that the reported superiority is currently weakened by the test-set hyperparameter selection protocol in Appendix A, which may inflate the headline margins over baselines.","major_comments":[{"comment":"The sentence 'The selections of these hyperparameters are based on the average test accuracy over five independent trials' directly admits model selection on the test set, contradicting the preceding claim that a randomly drawn 10% validation split was used. Since the headline memory-free gains in Table 2 (e.g., 69.26 vs. 58.89 on Seq-Cifar-10 Class-IL) compare against baselines with their published hyperparameters, the margins may be inflated by test-set selection bias. Please re-run the comparison with all methods selected on the same validation protocol, or provide a corrected description showing that no test information was used; report validation-based results and the selection curves.","section":"Appendix A, Tables 6-7"},{"comment":"As written, the HSD weighting is defined as α = max(0, e^{-e0/E}), which is independent of the current epoch index e and therefore cannot implement the described warm-up schedule of gradually decreasing LIRD and increasing LS-PRD after e0 epochs. Please state the exact schedule used in the experiments as a function of e, e0, and E; if the formula is a typesetting artifact, correct it. This is load-bearing for the reproducibility of the central stability loss.","section":"Eq. (10) and Sec. 4.4"}],"minor_comments":[{"comment":"The main text says the backbone is trained for 100 epochs on all non-initial tasks 'for all datasets', but Appendix Table 7 lists Et≥2 = 50 for Seq-Tiny-ImageNet; please reconcile this discrepancy and confirm that Co2L uses the same epoch schedule for that dataset.","section":"Sec. 5.1 vs. Appendix Table 7"},{"comment":"The statement 'The test Lasym SupCon with LS-PRD is omitted due to incompatibility' is unexplained; please specify why the asymmetric SupCon loss cannot be combined with sample-prototype relation distillation.","section":"Table 3"},{"comment":"The acknowledged limitation that the number of prototypes must be predefined is welcome and appropriate; for clarity, please state this assumption earlier in the paper, since it is central to the method's applicability.","section":"Conclusion"},{"comment":"There is a typo in the first sentence of Sec. 5.1: 'thee datasets' should read 'three datasets'.","section":"Sec. 5.1"}],"recommendation":"major_revision","confidential_remarks":"The most serious concern is the test-set hyperparameter selection in Appendix A, which directly threatens the centrality of the memory-free SOTA claim. The fixed-K limitation is acknowledged and acceptable for the controlled benchmarks. If the authors can supply validation-based selection or a convincing demonstration that the baselines were treated equivalently, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, it does something genuinely new but incremental: it combines focal contrastive loss with fixed neural-collapse prototypes (FNC2) and a time-weighted distillation of both instance-wise and sample-prototype relations (HSD). The memory-free results, if real, are notable—about 10 points over Co2L on Seq-Cifar-10 Class-IL. Second, the paper's own appendix undermines that \"if real\": hyperparameters were selected based on average test accuracy, not on the validation split. That is the load-bearing concern.\n\nOn the plus side, the loss definitions are clear, the experiments are extensive with standard deviations, and the ablations make sense. The memory-free evaluation protocol (classifier trained on last-task samples) is shared with Co2L, so the relative comparison is fair on that axis. The need to predefine K prototypes is real but acknowledged, and it doesn't affect the fixed-task benchmarks tested.\n\nThe soft spots, in proportion: the test-set hyperparameter selection is the big one. The appendix first says a 10% validation split is used for grid search, then says the selections are based on average test accuracy. Those two statements can't both be true. If the latter is correct, the head-to-head margins against Co2L could partly reflect selection bias. A referee should ask the authors to state which it is, and ideally re-tune on a validation set and report both. This is fixable, but it's central to the SOTA claim.\n\nAlso minor: Eq. (10)'s alpha schedule is ambiguous—alpha = max(0, e^{-e0/E}) or max(0, (e-e0)/E)? I can infer the intended ramp, but it should be written clearly. No code is released, which makes independent verification harder, but that's not a flaw in the method per se.\n\nBottom line: the paper is a solid empirical contribution to memory-efficient continual learning, conditional on the hyperparameter selection being cleaned up. It deserves a serious referee even with a skeptical verdict, because the core idea is plausible and the memory-free setting matters for privacy. I'd send it to review but with the expectation that the authors address the selection-bias question head-on. I wouldn't cite the memory-free numbers until then.","headline":"A plausible memory-free continual learning method whose headline results are undercut by the paper's own admission that hyperparameters were tuned on test accuracy.","tokens_in":18017,"tokens_out":3619,"would_cite":false,"duration_ms":35531,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Memory-free contrastive learner beats replay baselines in continual learning.","keywords":["continual learning","catastrophic forgetting","neural collapse","contrastive learning","knowledge distillation","simplex equiangular tight frame","memory-free continual learning","class-incremental learning"],"falsifier":"Run the method on a class-incremental stream whose total class count is not announced in advance (for instance, open-ended or with a class count that exceeds the pre-built $K$): if accuracy collapses once a new class appears, the central memory-free result is confined to closed class sets. A sharper check is to rerun the Seq-Cifar-10 memory-free setting with the focusing term set to $\\gamma=0$ and the HSD mixing weight fixed at $\\alpha=0$; the paper's ablation table indicates the gain would shrink, but the comparison would isolate how much of the benefit comes from the two new losses.","tokens_in":17058,"feed_emoji":"🧠","tokens_out":7261,"duration_ms":68064,"temperature":0.7,"pith_summary":"The paper argues that continual learning fails when a method uses only one kind of feature relationship: soft inter-sample similarities shift across tasks and smear class clusters together, while hard sample-to-prototype alignments compress classes and drag old representations toward new ones. It introduces FNC$^2$, a plasticity loss that combines both relationships and up-weights hard samples, and HSD, a distillation loss that preserves instance-wise and sample-to-prototype relations while shifting weight between them over training. The central empirical claim is that a model trained with these losses needs no stored exemplars to rival rehearsal-based methods, and it exceeds the prior memory-free baseline on every dataset and scenario tested. This matters for continual learning under data-privacy constraints, where keeping a buffer of old samples is not allowed.","feed_headline":"Memory-free contrastive learner beats replay baselines","feed_subtitle":"Two new losses mix neural-collapse prototypes with sample similarity, removing the need for stored exemplars.","key_machinery":"The load-bearing object is a pre-defined simplex Equiangular Tight Frame (ETF): $K$ fixed, mutually equidistant unit vectors in $\\mathbb{R}^d$, used as optimal class prototypes before training begins. FNC$^2$ is a focal supervised contrastive loss whose per-sample terms are weighted by $(1-c_{ij})^\\gamma$ and $(1-r_i)^\\gamma$, so hard positives—views far from the anchor or far from their prototype—dominate the gradient; old-task prototypes enter the denominators of $c_{ij}$ and $r_i$, which is how pseudo-replay is implemented. HSD is an interpolation between instance-wise relation distillation and sample-to-prototype relation distillation, with the mixing weight $\\alpha = \\max(0, e^{-e_0/E})$ ramping up after a warm-up, so the relative emphasis on hard stability grows as representations settle.","core_discovery":"On the paper's own terms, the discovery is that 'hardness' and 'softness' are complementary in neural-collapse-based continual learning, and that both are needed in both the plasticity and stability phases. The plasticity loss FNC$^2$ pulls each anchor toward its positive views and toward a fixed simplex ETF prototype for its class, while using all previous-task prototypes as negative references, so old classes act as pseudo-replay without stored images. The stability loss HSD preserves the teacher's instance-wise similarity distribution early in each task and the sample-to-prototype distribution later, with an epoch-dependent weight $\\alpha$ controlling the transition. The measured consequence is that the method exceeds Co2L in every memory-free setting reported, for example 69.26% versus 58.89% on Seq-Cifar-10 Class-IL, and that its buffer-size-200 results beat or approach the best replay-based methods.","pith_inferences":["If the prototype frame were made learnable or expandable, the same hard-soft balance would likely apply to open-ended class streams; the authors name this exact extension as future work.","A finer ablation isolating the focal exponent $\\gamma$ from the pseudo-replay prototype negatives would show which mechanism carries the memory-free gain, since the reported ablations do not vary $\\gamma$ independently.","Because the gains are smallest on Tiny-ImageNet, where the evaluation classifier uses few old samples, stronger classifier adaptation may widen the memory-free advantage.","The two losses do not require task identity at test time, so the same design could be dropped into online or fully unsupervised continual learning settings."],"forward_implications":["If the reported numbers hold, a continual classifier can be trained with no stored exemplars at all, removing the data-retention and privacy obstacle that motivates most rehearsal buffers.","The pseudo-replay mechanism turns previously seen class prototypes into negatives for current-task learning, so forgetting is reduced without access to old images.","The method remains competitive when a buffer is available, with improvements most visible at small buffer sizes, where representation drift is worst.","Because the classifier is trained separately on top of the frozen encoder, representation quality and classifier accuracy are decoupled, and later tasks inherit more stable features."],"supporting_citations":[{"why":"Supplies the Co2L baseline, the instance-wise relation distillation LIRD used in HSD, and the memory-free comparison the paper must beat.","marker":"[5]"},{"why":"Defines neural collapse and the simplex equiangular tight frame that provides the fixed class prototypes.","marker":"[38]"},{"why":"Introduces the supervised contrastive loss whose soft sample relationships FNC$^2$ extends with prototype terms.","marker":"[22]"},{"why":"CILA, the other method able to run without memory, provides the buffer-200 replay comparison and relation-distillation weighting.","marker":"[53]"},{"why":"A prior neural-collapse continual learning method that aligns features to fixed prototypes, which the paper argues over-focuses on hardness.","marker":"[55]"},{"why":"Extends neural-collapse prototype alignment to class-incremental learning and variants, the hard-plasticity approach FNC$^2$ relaxes.","marker":"[56]"},{"why":"Supplies the focal-modulation idea used to weight hard samples in FNC$^2$.","marker":"[30]"},{"why":"Reservoir sampling fills the optional replay buffer when memory is used.","marker":"[52]"}],"fun_headline_variants":["FNC^2 balances hardness and softness to stop forgetting","Memory-free contrastive learner rivals replay accuracy","Two losses merge neural collapse and similarity for CL","Hardness plus softness beats memory-free baselines","No stored exemplars needed with FNC^2 and HSD losses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline depends on knowing the total number of classes $K$ before training starts, since the fixed prototype frame is built once and never updated; with an unknown or growing class count, the memory-free claim does not apply.","fun_headline_variants_meta":{"raw":{"variants":["FNC^2 balances hardness and softness to stop forgetting","Memory-free contrastive learner rivals replay accuracy","Two losses merge neural collapse and similarity for CL","Hardness plus softness beats memory-free baselines","No stored exemplars needed with FNC^2 and HSD losses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1370,"prompt_tokens":928,"completion_tokens":442,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":363}},"tokens_in":544,"tokens_out":442,"duration_ms":5236,"temperature":1.0,"reasoning_tokens":363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:00:26.435391+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the method on a class-incremental stream whose total class count is not announced in advance (for instance, open-ended or with a class count that exceeds the pre-built $K$): if accuracy collapses once a new class appears, the central memory-free result is confined to closed class sets. A sharper check is to rerun the Seq-Cifar-10 memory-free setting with the focusing term set to $\\gamma=0$ and the HSD mixing weight fixed at $\\alpha=0$; the paper's ablation table indicates the gain would shrink, but the comparison would isolate how much of the benefit comes from the two new losses.","supporting_citations":[{"cited_title":"Co$ˆ2$L: Con- trastive Continual Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the Co2L baseline, the instance-wise relation distillation LIRD used in HSD, and the memory-free comparison the paper must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines neural collapse and the simplex equiangular tight frame that provides the fixed class prototypes."},{"cited_title":"Provable contrastive continual learning","cited_arxiv_id":null,"evidence_quote":"CILA, the other method able to run without memory, provides the buffer-200 replay comparison and relation-distillation weighting."},{"cited_title":"Neural collapse inspired feature- classifier alignment for few-shot class-incremental learning","cited_arxiv_id":null,"evidence_quote":"A prior neural-collapse continual learning method that aligns features to fixed prototypes, which the paper argues over-focuses on hardness."},{"cited_title":"Neural collapse terminus: A unified so- lution for class incremental learning and its variants, 2023","cited_arxiv_id":null,"evidence_quote":"Extends neural-collapse prototype alignment to class-incremental learning and variants, the hard-plasticity approach FNC$^2$ relaxes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reservoir sampling fills the optional replay buffer when memory is used."}],"review_version":1}