{"id":"075b989c-c24e-4d77-974d-220ee3b3dd46","arxiv_id":"2412.16409","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"COUQ iteratively combines old-class and novel-class feature reconstruction errors to produce uncertainty scores that stay reliable as new classes arrive, improving continual novelty detection and active learning.","lead":"This paper introduces COUQ, an iterative uncertainty scoring method that lets a continually learning AI tell apart new object classes from familiar ones in unlabeled data, while also deciding which samples most deserve human labeling. The method is tested across several image datasets and beats simpler baselines at novelty detection and continual classification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 3's numerator takes a minimum over the growing set of old classes; for a novel sample this minimum can only shrink as classes accumulate, so the claimed no-degradation of COUQ scores is not established by averaged AUROC.","rationale":"The reader's weakest assumption is that the frozen pre-trained backbone supplies features sufficient to represent both old and novel classes. My concern is related but not identical: even under the reader's assumption that the frozen features are adequate, the specific score in Eq. (3) has a monotone mechanism that can suppress novelty signal as old classes accumulate, because the numerator is a minimum over a growing set. This is an internal property of the score definition, not just a question of feature quality. I therefore partially agree with the reader's identification of feature representation as the fragile spot, but I locate the more precise load-bearing issue in the min-over-old operation. The paper's empirical evidence does not currently rule out this degradation: Table 1 reports averaged AUROC, and Fig. 2 does not sweep old-class count independently. The proposed test would settle whether the central no-degradation claim holds in the long-horizon regime that the method is designed for. Since the paper is already CONDITIONAL in the reader's verdict, and this concern adds a specific experimental condition rather than disproving the method's results, I recommend keeping the verdict unchanged.","tokens_in":15277,"tokens_out":6579,"duration_ms":68864,"concrete_test":"Run the COUQ pipeline with a fixed held-out novel class at the final task and vary only the number of old classes accumulated before it (e.g., by changing the class-increment size to produce 5, 10, 20, and 40 old classes), reporting final-task AUROC separately rather than averaged over tasks. Also recompute Fig. 2 as AUROC versus cumulative |C_old| for all datasets. If final-task AUROC declines as |C_old| grows, Eq. (3)'s min-over-old numerator is responsible; if it remains flat across the sweep, the no-degradation claim survives this attack.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (3) defines S^i(u) = min_{j in C_old} FRE_j(u) / FRE^{i-1}_m(u). Because the numerator is a minimum over C_old, and C_old grows at every task, for any fixed novel-class sample min_j FRE_j(u) is non-increasing as old classes accumulate: each newly added old-class PCA subspace provides another candidate that can only lower the minimum. The score for genuinely novel samples is therefore systematically compressed over time. The paper's no-degradation claim (Section 3.2) is supported empirically only by Table 1's AUROC averaged over all tasks and by Fig. 2 task curves; an average can mask a gradual decline, and the plots do not control for the confound of 'number of accumulated old classes.' The No-iters ablation (Table 2) varies inner-loop iteration count, not old-class coverage, so it does not test this mechanism. This is distinct from the frozen-backbone assumption: even with a perfect backbone that gives every class a well-separated subspace, dense class packing in feature space makes the nearest old-class reconstruction error for a novel class shrink as the number of classes grows. Thus the central claim that uncertainty quality 'does not degrade as new classes are iteratively encountered and learned' is not yet established for long horizons or large class counts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses continual open-world learning in which, at each task, a model receives an unlabeled pool containing both unseen samples of previously learned classes and samples from an arbitrary number of novel classes. The proposed COUQ method iteratively computes per-novel-class uncertainty scores using per-class PCA reconstruction errors (FRE) from a frozen pre-trained backbone, combined with a novelty mapper (K-means or a small classifier) that assigns pseudo-labels to confident novel samples. The score is S^i(u) = min_{j in C_old} FRE_j(u) / FRE_m^{i-1}(u), where m is the predicted novel class. The paper evaluates COUQ for continual novelty detection, active sample selection, and pseudo-labeling on five datasets and three backbones, and reports large gains over entropy/margin/softmax baselines, incDFM, DFM, GBCL, and CCIC, with ablations on iterativeness, active-selection strategy, and pseudo-labeling.","tokens_in":15515,"tokens_out":9148,"duration_ms":77445,"significance":"The problem is relevant and under-explored: most continual learning assumes an oracle for novelty detection and labels, while this paper targets a more realistic setting with unlabeled mixtures and multiple novel classes per task. The empirical scope is a strength: five datasets, three backbones, and a consistent set of baselines, plus ablations isolating the iterative procedure, the active-selection strategy, and pseudo-labeling. The observation that ambiguity-based active selection outperforms confidence-based selection is interesting and is tested across datasets. The main weakness is that the central 'no degradation over time' claim is not rigorously established, and the absence of repeated-seed statistics limits confidence in some small margins. If those issues are addressed, the paper would be a useful contribution.","major_comments":[{"comment":"The claim that uncertainty quality 'does not degrade as new classes are iteratively encountered and learned' is not established by the formulation. For any fixed sample u, the numerator min_{j in C_old} FRE_j(u) is non-increasing as C_old grows, because each newly added old-class PCA subspace provides another candidate that can only lower the minimum. Thus the scores for genuinely novel samples are systematically compressed as old classes accumulate, even with a perfect frozen backbone. Table 1 reports AUROC averaged over all tasks, which can mask a gradual decline, and the task-level curves in Fig. 2 are not conditioned on the number of accumulated old classes. The No-iters ablation in Table 2 varies inner-loop iterations, not the size of C_old, so it does not test this mechanism. Please provide per-task AUROC as a function of task index and old-class count, add a long-horizon experiment with many incremental tasks, or explicitly qualify the no-degradation claim.","section":"Eq. (3), §3.2"},{"comment":"All results are single point estimates with no error bars, no multiple seeds, and no statistical test. Several head-to-head margins are small: in Table 3, Plants shows AL-Amb at 45.2 vs Rand at 41.7, and Cifar100 shows 65.1 vs 64.8; in Table 5, Im21K shows COUQ(P;oneshot) at 78.9, which is higher than COUQ(P;Default) at 76.4, so the stated benefit of iterativeness is not consistent across datasets. Without variance estimates, the robustness of the claimed superiority of COUQ's default configuration is not established. Report means and standard deviations over at least three seeds, and ideally paired significance tests.","section":"Tables 1, 3, 5"},{"comment":"The method relies on a frozen ImageNet-pretrained backbone providing features that are sufficiently task-non-specific to represent all future novel classes; if novel classes require features not captured by this representation, the PCA subspaces and the classifier cannot distinguish them regardless of the uncertainty scoring. The paper does not examine when this assumption fails, for instance by measuring performance as a function of class similarity to ImageNet or by using a backbone pretrained on a different distribution. This limits the generality claimed in the abstract and should be discussed explicitly with a failure analysis or a clear scope statement.","section":"§4.1"}],"minor_comments":[{"comment":"The sentence 'We describe next how the uncertainty score from Eq. (2) can be used' appears to refer to Eq. (3), since active selection uses the iterative multiclass score rather than the initialization S0.","section":"§3.3"},{"comment":"The reconstruction error is written as ∥f(x) − (T_m^† ∘ T_m)u∥_2; this should be the reconstruction of the feature u = g(x), not of the classifier output f(x).","section":"Eq. (1)"},{"comment":"The notation T_m^† is not defined in the text; please state that it is the pseudo-inverse of the PCA projection.","section":"Eq. (1)"},{"comment":"The definition of C_old^t as ∪_{k=0}^{t-1} C_new^k is correct, but the phrase 'up to and including task t−1' could be clearer about whether t−1 is included; the notation already makes it unambiguous.","section":"§2.1"},{"comment":"CCIC has '−' entries for the ViT backbones; please state in the caption or text why these experiments were not run.","section":"Table 4"},{"comment":"The paper says the code will be released upon acceptance; providing an anonymous link in the current version would be helpful for reproducibility.","section":"General"},{"comment":"The row 'None' is described as the lower-bound random labeling; renaming it to 'Rand' for consistency with Table 3 would improve readability.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper has a heavy self-citation pattern: FRE [32], DFM [33], and incDFM [42] are all from the same authors and are used as components or baselines. This is not disqualifying, but the novelty statement should be sharpened: the main difference from incDFM is the multi-class generalization and the iterative ratio, which is clear in the body, yet the abstract and introduction phrase the contribution more broadly. The claim of being 'first to address uncertainty estimation in such a general scenario' is defensible but should be checked against the cited literature and worded carefully. The empirical claims are broad, but the missing error bars and the structural concern about Eq. (3) would need to be resolved before I would recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper you asked about: COUQ is a real step forward for continual open-world learning. The iterative ratio in Eq. (3) — min over old-class reconstruction error divided by the novel-class reconstruction error — is a new formulation, and the generalization from incDFM's single-novel-class setting to arbitrary numbers of novel classes is a genuine extension. The ablations are the strongest part: iteration matters (No-Iters drops 8.3%), ambiguous-sample selection beats top-confidence and random, and pseudo-labeling alone beats abstention. Five datasets and three backbones is a solid evaluation, and the margins over the ER-entropy/margin baselines are large. I believe the method works.\n\nThe soft spots are real, though. The headline claim that uncertainty quality 'does not degrade' as classes accumulate is not supported by the evidence. The numerator in Eq. (3) is a minimum over the growing set of old classes. For a fixed novel-class sample, that minimum can only shrink as each new old class adds another candidate subspace. So scores compress over time. The averaged AUROC in Table 1 and the task curves in Fig. 2 can hide this; the ablation in Table 2 varies inner-loop iterations, not old-class coverage, so it doesn't test the mechanism. The horizons are also short (2-5 tasks). You need either per-task curves controlled for accumulated class count, or a direct measurement of min_j FRE_j on novel samples over tasks. This is not fatal to the method — the empirical superiority over baselines could still hold — but it means the no-degradation claim should be toned down until it's actually tested on longer sequences.\n\nOther issues: no code release (stated 'upon acceptance'), no error bars or multiple seeds, and several details deferred to an unavailable supplement. The baseline set leans heavily on the authors' own DFM/incDFM, and 'state-of-the-art' is claimed without comparison to, e.g., GCD methods or more recent OOD detectors. Those are conventional concerns, not deal-breakers.\n\nMy take: this paper deserves a serious referee. The core idea is novel and the empirical pattern is consistent. Send it to review with a request for code, multi-seed results, and a direct test of the min-over-old-classes compression on longer task sequences. I'd probably accept after major revision, unless the longer-horizon test reveals the degradation the math suggests.","headline":"A genuinely useful iterative uncertainty score for multi-novel-class continual learning, but the 'no degradation' claim is overreached and needs a direct test.","tokens_in":16091,"tokens_out":2564,"would_cite":true,"duration_ms":22078,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that per-class feature-reconstruction distances, combined in a ratio, yield uncertainty scores that stay reliable as a model continually encounters and learns new classes, removing the need for a novelty or labeling…","keywords":["continual learning","open-world learning","uncertainty quantification","novelty detection","active learning","pseudo-labeling","feature reconstruction error","class-incremental learning"],"falsifier":"Take a novel class that differs from the old classes along a dimension the frozen backbone is known not to encode (for example, a synthetic shift in color or texture that ImageNet features ignore), add it as the new class in a task, and measure COUQ's novelty-detection AUROC; if the score is near chance while a detector with an updated backbone succeeds, the frozen-feature premise is the binding constraint.","tokens_in":15059,"feed_emoji":"🔍","tokens_out":8366,"duration_ms":68126,"temperature":0.7,"pith_summary":"This paper tackles a realistic version of continual learning in which a deployed model receives unlabeled data containing both unseen examples of already-known classes and samples from an unknown number of new classes. The authors propose COUQ, an iterative uncertainty-estimation method that scores each sample by the ratio of its distance to the nearest old-class feature subspace to its distance to the currently predicted new-class subspace. They argue that, unlike one-shot uncertainty estimates computed from the classifier's decision boundary, these scores do not degrade as new classes are added, and that they can drive novelty detection, active labeling, and pseudo-labeling in the same pipeline. Across five image datasets and three frozen backbone networks, the paper reports that COUQ outperforms existing continual novelty detectors and continual active-learning baselines, and that its unsupervised variant often beats semi-supervised methods that rely on more supervision.","feed_headline":"Uncertainty quality holds steady as new classes keep arriving","feed_subtitle":"COUQ keeps novelty detection and active learning accurate over tasks with per-class reconstruction distances.","key_machinery":"The machinery is a per-class PCA subspace model of deep features, derived from the feature reconstruction error (FRE) metric of [32], arranged into an iterative algorithm. For every old class, a PCA transform is fixed at the task where the class was learned and never updated; for every newly detected class, a PCA transform is fit from the samples the current iteration labels as that class. The COUQ score is the ratio of the minimum old-class FRE to the FRE of the predicted new class, and the inner loop alternates between computing scores, selecting high-confidence samples for pseudo-labeling and ambiguous samples for active labeling, retraining the novelty mapper (K-means or a small classifier), and updating the new-class PCA transforms and the continual classifier. Because old-class transforms are never retrained, the score's reference for 'old' does not drift, which is the mechanism the paper credits for preventing degradation over tasks.","core_discovery":"COUQ's central claim is that reliable per-class uncertainty can be maintained throughout continual open-world learning by combining per-class PCA feature models with an iterative scoring loop. At each task, the score for a sample $u$ is $S^i(u)=\\frac{\\min_{j\\in C^t_{\\mathrm{old}}} FRE^0_j(u)}{FRE^{i-1}_m(u)}$, where the numerator is its reconstruction error against the best-matching old-class subspace and the denominator is its reconstruction error against the subspace of the novel class $m$ predicted by the previous iteration's novelty mapper. Samples with high scores are confidently novel, low scores are confidently old, and mid-range scores are ambiguous; the ambiguous ones are the most informative for active labeling. The paper shows empirically that the quality of these scores stays roughly constant across tasks, that the iterative updates reduce error propagation, and that the method outperforms entropy-, margin-, and softmax-based uncertainty baselines as well as the single-novel-class detector incDFM.","pith_inferences":["An extension the authors do not explore is using the ratio $S^i(u)$ as a calibration signal for when to trust the continual classifier's predictions, since the ratio separates confident-old from confident-new without depending on the classifier's logits.","Because the paper leaves old-class distribution shift for future work, the score's stability over time likely depends on old-class features staying fixed; if they drift, the frozen PCA subspaces would age.","The same score could be applied to non-visual continual learning, such as sensor or text streams, wherever per-class reconstruction subspaces can be fit on frozen embeddings.","One could test whether querying a mix of ambiguous and confident-novel samples remains optimal as the active budget grows; the paper's ablations cover a small budget range."],"forward_implications":["Continual novelty detection can be performed without a separate OOD detector: COUQ's score doubles as the novelty score, and its AUROC stays high over tasks.","A tiny active-labeling budget (about 1.25% of samples) suffices to keep the continual classifier accurate, because ambiguous samples carry the most information.","High-confidence COUQ scores yield pseudo-labels reliable enough to improve semi-supervised continual learning; pseudo-labeling with entropy or margin scores does not help.","The method transfers across multiple datasets and frozen backbones, meaning it can be applied to a deployed model without retraining the feature extractor.","Unsupervised COUQ, using K-means as the novelty mapper, can outperform semi-supervised baselines, so novelty detection may not require any labels at all in some settings."],"supporting_citations":[{"why":"It supplies the feature reconstruction error (FRE) and per-class PCA subspace formulation that COUQ's score is built from.","marker":"[32]"},{"why":"It is the predecessor continual novelty detector that COUQ extends from single-class to multi-class novelty, and serves as the main baseline.","marker":"[42]"},{"why":"It provides the non-continual FRE-based detector that COUQ iterates and that serves as a baseline.","marker":"[33]"},{"why":"It is cited as the basis for using a frozen pre-trained backbone whose features are task-non-specific, a premise the method depends on.","marker":"[41]"},{"why":"It supplies the Experience Replay continual-learning method used to train the classifier in all experiments.","marker":"[12]"},{"why":"It is the Experience Replay reference for replay-based continual learning used to frame the classifier updates.","marker":"[43]"},{"why":"It is the few-shot continual active-learning baseline (GBCL) that COUQ is compared against in open-world learning.","marker":"[5]"},{"why":"It is the semi-supervised continual-learning baseline (CCIC) that assumes new tasks contain only novel classes and that fails when old and new classes co-occur.","marker":"[9]"},{"why":"It supplies the active continual learning uncertainty-based ER baselines that COUQ outperforms.","marker":"[49]"}],"fun_headline_variants":["COUQ: uncertainty stays reliable as new classes roll in","Reliable per-class uncertainty through endless open-world tasks","COUQ: steady scores while novelty never stops","Uncertainty that withstands the arrival of new classes","COUQ keeps uncertainty quality high as tasks accumulate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a frozen, pre-trained feature extractor represents every future novel class well enough; if a new class needs visual information the frozen network does not capture, the per-class subspaces cannot separate it from old classes and the uncertainty score loses meaning.","fun_headline_variants_meta":{"raw":{"variants":["COUQ: uncertainty stays reliable as new classes roll in","Reliable per-class uncertainty through endless open-world tasks","COUQ: steady scores while novelty never stops","Uncertainty that withstands the arrival of new classes","COUQ keeps uncertainty quality high as tasks accumulate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000367,"raw_usage":{"total_tokens":1952,"prompt_tokens":907,"completion_tokens":1045,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":966}},"tokens_in":523,"tokens_out":1045,"duration_ms":7950,"temperature":1.0,"reasoning_tokens":966,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:36:02.613553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a novel class that differs from the old classes along a dimension the frozen backbone is known not to encode (for example, a synthetic shift in color or texture that ImageNet features ignore), add it as the new class in a task, and measure COUQ's novelty-detection AUROC; if the score is near chance while a detector with an updated backbone succeeds, the frozen-feature premise is the binding constraint.","supporting_citations":[{"cited_title":"incdfm: Incremental deep feature modeling for continual novelty detection","cited_arxiv_id":null,"evidence_quote":"It is the predecessor continual novelty detector that COUQ extends from single-class to multi-class novelty, and serves as the main baseline."},{"cited_title":"Sub- space modeling for fast out-of-distribution and anomaly de- tection","cited_arxiv_id":null,"evidence_quote":"It provides the non-continual FRE-based detector that COUQ iterates and that serves as a baseline."},{"cited_title":"Lifelong learning without a task oracle","cited_arxiv_id":null,"evidence_quote":"It is cited as the basis for using a frozen pre-trained backbone whose features are task-non-specific, a premise the method depends on."},{"cited_title":"Rethinking experience replay: a bag of tricks for continual learning","cited_arxiv_id":null,"evidence_quote":"It supplies the Experience Replay continual-learning method used to train the classifier in all experiments."},{"cited_title":"Experience replay for continual learning","cited_arxiv_id":null,"evidence_quote":"It is the Experience Replay reference for replay-based continual learning used to frame the classifier updates."},{"cited_title":"Few-shot continual active learning by a robot","cited_arxiv_id":null,"evidence_quote":"It is the few-shot continual active-learning baseline (GBCL) that COUQ is compared against in open-world learning."},{"cited_title":"Continual semi-supervised learning through contrastive interpolation consistency","cited_arxiv_id":null,"evidence_quote":"It is the semi-supervised continual-learning baseline (CCIC) that assumes new tasks contain only novel classes and that fails when old and new classes co-occur."}],"review_version":1}