{"id":"33e537a5-b015-4a6a-a6f2-f063e6b2e384","arxiv_id":"2412.00860","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper formalizes continual semi-supervised anomaly detection and presents a VAE-based baseline with generative replay and outlier rejection, reporting AUC scores on three image datasets.","lead":"This paper defines a new machine learning task, continual semi-supervised anomaly detection, and proposes a first baseline using a variational autoencoder with generative replay and outlier rejection. The reported results are mixed and the evaluation contains a test-set threshold selection flaw that weakens the claims.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set threshold selection makes reported AUCs fitted; re-evaluation with held-out thresholds is required to support the claimed superiority of outlier rejection.","rationale":"The reader's weakest_assumption correctly identifies the test-set threshold selection as a serious flaw, and I agree that this makes the reported AUCs fitted rather than predictive. I partially disagree only with bundling the joint-training upper-bound violation as an equally load-bearing issue: the joint-training anomaly is acknowledged by the authors and mostly weakens the abstract's 'upper baseline' language, whereas the threshold selection directly contaminates every reported AUC and every inter-method comparison. The proposed concrete test would settle whether the superiority of OR is real or an artifact of test-set fitting. Since the reader's verdict of REJECT is already based on this concern, my independent evaluation does not move the verdict; hence UNCHANGED. I would emphasize, however, that the problem formalization and dataset-splitting procedure are potentially useful contributions that could justify a conditional accept if the empirical evaluation were corrected, but the current evidence does not support the central empirical claim.","tokens_in":15417,"tokens_out":2681,"duration_ms":26450,"concrete_test":"Re-run the evaluation using the raw ELBO (or reconstruction probability) scores without any test-set threshold tuning: compute standard AUC via roc_auc_score on the test stream for each method, and if a threshold is genuinely required for any component, select it only on the validation stream described in Section 4.2. Repeat over at least 5 independent data splits and report means and standard deviations. If OR no longer attains the highest mean AUC on MNIST and Fashion MNIST, the central empirical claim fails; if it still does, the test-set-threshold concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on the AUC scores in Table 2, yet Section 3.3 states that the model 'will use the AUC calculated over the test set to find the optimal threshold for these particular metrics which maximises the AUC.' This is not a standard threshold-independent AUC computation; it introduces the test set as a tuning set for a threshold (or threshold-dependent score) and then reports AUC after that tuning. Consequently, the reported values are fitted quantities, not unbiased estimates of anomaly-detection performance on unseen data. The comparison is especially fragile because the test set is small (10,000 samples) and the per-episode results in Tables 3-5 show large swings (e.g., MNIST OR AUC ranges 0.59-0.84 across episodes), leaving ample room for overfitting a threshold to the test set. If the threshold were selected on the validation stream defined in Section 4.2, the absolute AUCs and the relative ranking of EWC, Joint, Naive, and OR could change, as evidenced by the paper's own admission that Joint is not a reliable upper bound (Section 5: 'There were issues in joint training which meant that it was not in fact the upper bound for all experiments'). Without a clean evaluation protocol, the headline result that OR 'outperforms other continual learning strategies across most datasets' is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper formalizes the task of continual semi-supervised anomaly detection (CSAD), in which a model receives a stream of experiences each containing a mixture of labelled and unlabelled data, and must detect anomalies while mitigating catastrophic forgetting. The authors propose a variational autoencoder (VAE) baseline that combines semi-supervised training with generative replay and an outlier-rejection mechanism based on extreme value theory. They evaluate this method (OR) against naive training, joint training, and Elastic Weight Consolidation on MNIST, Fashion-MNIST, and CIFAR-10, and report per-episode AUC-ROC scores and ablation studies. The main empirical claim is that OR outperforms other continual learning strategies on most datasets.","tokens_in":15693,"tokens_out":7539,"duration_ms":61786,"significance":"If the results held, the paper would provide a useful new problem formulation (CSAD) and a simple baseline for future work. The dataset configurations and ablation experiments are described in enough detail to be reproduced, which is a practical contribution. However, the current experimental evidence is undermined by the use of the test set to select the anomaly detection threshold, the failure of the joint-training upper bound on Fashion-MNIST, and the absence of error bars. The conceptual contribution is therefore not yet matched by a reliable empirical demonstration.","major_comments":[{"comment":"The evaluation protocol leaks test information: the text states that the anomaly detection metric 'will use the AUC calculated over the test set to find the optimal threshold for these particular metrics which maximises the AUC.' This tunes the threshold on the same test set used to report the AUC values in Table 2 and the per-episode tables. Consequently, the reported AUCs are fitted quantities rather than unbiased estimates. The threshold should be selected on the validation stream defined in Section 4.2 (or via a nested cross-validation on the training stream) and then evaluated on the test stream. This is load-bearing because the central claim that OR 'outperforms other continual learning strategies across most datasets' rests on these numbers.","section":"Section 3.3"},{"comment":"The paper treats joint training as the upper bound, but the results contradict this: on Fashion-MNIST, Joint scores 0.401, well below Naive (0.555) and OR (0.581). The paper itself admits in Section 5 that 'There were issues in joint training which meant that it was not in fact the upper bound for all experiments.' The authors need to diagnose and repair the joint-training baseline, or explicitly remove the upper-bound claim and reinterpret the comparisons accordingly.","section":"Section 4.4 / Section 5"},{"comment":"No multiple seeds or error bars are reported; all AUC values are from single runs. The per-episode scores show large swings (e.g., MNIST OR ranges from 0.60 to 0.84 in Table 3), so the aggregate differences in Table 2 (0.690 vs 0.655 for MNIST) may not be statistically significant. The paper should report mean and standard deviation over several seeds and, ideally, a paired significance test for the method comparisons.","section":"Section 4.4 / Tables 3-5"},{"comment":"The CIFAR-10 experiments use a dense encoder/decoder, and the paper acknowledges that 'simple dense layers are unlikely to be able to correctly reconstruct CIFAR-10 images.' As a result, the CIFAR-10 results reflect an architecture mismatch rather than the properties of the CSAD methods. A convolutional VAE (or an explicit statement that CIFAR-10 is a degenerate stress test) is needed before any cross-dataset conclusions can be drawn.","section":"Section 5"}],"minor_comments":[{"comment":"Equation (8) appears to be missing parentheses in the Weibull CDF; it should read 1 - exp(-(|d|/lambda)^kappa) or similar, and the notation z-bar versus z should be clarified.","section":"Section 3.2"},{"comment":"The sentence 'which will use the AUC calculated over the test set to find the optimal threshold for these particular metrics which maximises the AUC' is grammatically ambiguous; please rewrite it to state exactly which score is used and how thresholds are chosen.","section":"Section 3.3"},{"comment":"There are numerous typographical and grammatical errors ('reproducability', 'discriminte', 'the the goal', 'hyperparamter'); the manuscript needs careful proofreading.","section":"Throughout"},{"comment":"The table caption describes bold and underline formatting for best and second-best results, but the formatting is not visible in the text version; please ensure the final PDF displays it.","section":"Section 4.4 / Table 2"},{"comment":"The two definitions of CSSL discussed in [7] and [35] could be presented in a table to make the adopted setting clearer.","section":"Section 2.4"},{"comment":"The narrative in the ablation analysis refers to 'Figure 2', 'Figure 9', etc., but the ordering of figures in the text is repetitive; consider renumbering or summarizing the figures to improve readability.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's main value is the formalization of CSAD; the current empirical evaluation prevents acceptance. I would support a major revision that re-runs all experiments with a clean threshold-selection protocol, multiple seeds, and a repaired joint-training baseline. If the authors cannot produce such a re-evaluation, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: the paper's real contribution is the formal definition of Continual Semi-Supervised Anomaly Detection (CSAD) and a detailed, reusable protocol for building benchmarks from MNIST/Fashion/CIFAR. The method itself is a fairly straightforward assembly of an M2 VAE, generative replay, and Mundt et al.'s outlier rejection. That combination is sensible, but the empirical evidence for it does not survive close reading.\n\nThe biggest problem is in Section 3.3: the anomaly threshold is chosen by maximizing AUC on the test set. That makes every reported AUC a fitted quantity, not a prediction. The stress-test note got this right. It matters most for the headline claim that Outlier Rejection beats EWC and Naive on MNIST and Fashion MNIST. Those margins are small (0.690 vs 0.646; 0.581 vs 0.521), and with a test-set-tuned threshold plus no error bars or multiple seeds, I can't tell whether the difference is method quality or threshold overfitting. The per-episode tables show swings of 0.2 AUC between episodes, so the threshold has plenty of room to overfit.\n\nThe joint-training upper bound is also broken, and the paper admits it in the ablation section: 'There were issues in joint training which meant that it was not in fact the upper bound for all experiments.' Fashion MNIST shows Joint at 0.401, well below Naive at 0.555 and OR at 0.581. That alone invalidates the 'upper bound' framing.\n\nWhat the paper does well: the problem formulation is genuinely new as far as I can tell, the dataset splitting parameters (alpha, beta, gamma, zeta, lambda) are clearly defined, and the authors are honest about the CIFAR-10 weakness, blaming the dense encoder-decoder. They also flag the joint training issue themselves, which is more than many papers do. The outlier rejection mechanism is a direct application of Mundt et al., but it's a reasonable baseline choice. No code is shipped, which is a shame for a benchmark paper; the protocol is described in enough detail that reimplementation is possible, but code would have made it much stronger.\n\nWho should read it: anyone thinking about streaming anomaly detection with limited labels. The formalization will likely be cited even if the numbers are not. But the evaluation protocol needs to be redone with validation-set threshold selection, multiple seeds, and error bars before the empirical claims can be trusted.\n\nRecommendation: send it to review, but flag the threshold issue prominently. The new problem setup deserves a serious referee; the current experiments don't support the 'promising results' claim as written.","headline":"A genuinely new problem formalization is undermined by an evaluation that tunes the anomaly threshold on the test set; the paper is worth refereeing but the empirical claims need a redo.","tokens_in":16231,"tokens_out":2260,"would_cite":true,"duration_ms":20321,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper formalises continual semi-supervised anomaly detection and shows that a variational autoencoder with outlier-rejected generative replay is a strong baseline, reaching the highest AUC-ROC on MNIST and Fashion MNIST among the…","keywords":["continual semi-supervised anomaly detection","variational autoencoder","generative replay","outlier rejection","extreme value theory","catastrophic forgetting","AUC-ROC","domain-incremental learning"],"falsifier":"Recompute Tables 2 through 5 by choosing the threshold once on a held-out validation stream and then applying it to the test stream; if outlier rejection no longer attains the highest AUC on MNIST and Fashion MNIST, the paper's main empirical claim fails. A second check is to rerun joint training with the same encoder-decoder: if it still trails naive training on Fashion MNIST, the upper-bound assumption behind the comparison is invalid.","tokens_in":15222,"feed_emoji":"📈","tokens_out":4093,"duration_ms":36442,"temperature":0.7,"pith_summary":"The paper formalises continual semi-supervised anomaly detection (CSAD): a learner sees a stream of disjoint datasets, each with a few labelled normal examples and many unlabelled ones, and must flag anomalies without revisiting past data. It argues that this combination matches real-world monitoring better than anomaly detection studied in isolation, because anomalies are often unlabelled, new anomaly types appear over time, and storing all historical data is impractical. To make the task testable, it contributes a baseline: a variational autoencoder trained semi-supervised with a classifier, plus deep generative replay that samples only latent points judged to be inliers of the normal class. The claim is that this outlier-rejection replay achieves the best AUC-ROC among the compared strategies on MNIST (0.690) and Fashion MNIST (0.581), and remains competitive on CIFAR-10 (0.546).","feed_headline":"Outlier-rejected replay wins continual anomaly detection","feed_subtitle":"A VAE baseline that replays only high-quality normal samples posts the best AUC on MNIST and Fashion MNIST and opens a new research task.","key_machinery":"The load-bearing mechanism is a semi-supervised variational autoencoder (the M2 model with a beta-weighted KL term) whose classifier regularises the latent space into clustered normals, combined with generative replay filtered by extreme value theory. For replay, the latent distance of each generated sample from the normal-class mean is modelled by a per-class Weibull distribution, and samples whose outlier probability exceeds a validation-set-tuned threshold are rejected before decoding. Anomaly scores come from the ELBO reconstruction term, and evaluation is reported as AUC-ROC.","core_discovery":"The central discovery is that continual semi-supervised anomaly detection can be formalised as a distinct problem and that a comparatively simple VAE-based learner with outlier-rejected replay is a workable baseline for it. In the paper's experiments, the outlier-rejection method outperforms the other continual-learning strategies on most datasets, including the highest AUC-ROC on MNIST and Fashion MNIST. The ablation studies show that performance rises with more labelled normal data, that labelled anomalies help up to a point, and that unlabelled anomalies initially help then hurt. The paper also reports that its assumed upper bound, joint training, is not always the best, notably on Fashion MNIST where joint training scores 0.401 while naive training reaches 0.555.","pith_inferences":["The reported AUCs should be read as an upper envelope, since the threshold is chosen on the test set itself; a held-out threshold selection would give a fairer comparison and might reorder the methods.","The outlier-rejection idea transfers to other generative replay models: any generator whose latent space has a well-estimated normal-class mean could use the same Weibull filter.","A natural next experiment, not run here, is to use the outlier-rejection probability itself as the anomaly score instead of the ELBO, which the paper notes is currently infeasible with its label-conditioned encoder.","If joint training is not actually an upper bound in this semi-supervised continual setting, the standard upper-bound comparison used in continual-learning evaluation needs rethinking."],"forward_implications":["CSAD is usable as a labelled research problem with a reproducible benchmark setup, since the five data-splitting parameters define streams for MNIST, Fashion MNIST, and CIFAR-10.","A practitioner with mostly unlabelled streaming data can deploy the VAE baseline as a first continual anomaly detector without storing past data.","Because outlier rejection beats EWC on MNIST and Fashion MNIST, generative replay with inlier filtering is a stronger default than weight regularisation in this setting.","Performance is sensitive to the placement of labelled data: skewing labels toward later experiences reduces the need for forgetting mitigation and changes method rankings."],"supporting_citations":[{"why":"Supplies the outlier-rejection sampling mechanism, estimating inlier/outlier probability in latent space with a Weibull model, that the proposed replay method builds on.","marker":"[29]"},{"why":"Supplies the variational autoencoder and M2 semi-supervised objective, including the classifier regularisation and reconstruction loss used in the baseline.","marker":"[19]"},{"why":"Supplies the use of a VAE and its ELBO as an anomaly score in a continual anomaly detection setting, the approach adopted here.","marker":"[11]"},{"why":"Formalises continual semi-supervised learning, the paradigm this paper extends to anomaly detection.","marker":"[7]"},{"why":"Provides the alternative CSSL definition with a mixture of labelled and unlabelled data in each experience, which the paper adopts.","marker":"[35]"},{"why":"Defines the Elastic Weight Consolidation baseline method against which outlier rejection is compared.","marker":"[10]"},{"why":"Establishes reconstruction probability in VAEs as a means of anomaly detection, informing the anomaly scoring choice.","marker":"[44]"}],"fun_headline_variants":["Outlier-rejected replay beats EWC in continual anomaly detection","New baseline: VAE with outlier-rejected replay for CSAD","Outlier rejection makes replay the top continual anomaly detector","Forget EWC: outlier-rejected replay wins continual anomaly detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported AUC values are obtained by selecting the anomaly-detection threshold on the test set itself, so the comparison measures each method's best possible operating point rather than its predictive quality on unseen data.","fun_headline_variants_meta":{"raw":{"variants":["Outlier-rejected replay beats EWC in continual anomaly detection","New baseline: VAE with outlier-rejected replay for CSAD","Outlier rejection makes replay the top continual anomaly detector","Forget EWC: outlier-rejected replay wins continual anomaly detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000508,"raw_usage":{"total_tokens":2453,"prompt_tokens":900,"completion_tokens":1553,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1482}},"tokens_in":516,"tokens_out":1553,"duration_ms":12090,"temperature":1.0,"reasoning_tokens":1482,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:55:14.049649+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute Tables 2 through 5 by choosing the threshold once on a held-out validation stream and then applying it to the test stream; if outlier rejection no longer attains the highest AUC on MNIST and Fashion MNIST, the paper's main empirical claim fails. A second check is to rerun joint training with the same encoder-decoder: if it still trails naive training on Fashion MNIST, the upper-bound assumption behind the comparison is invalid.","supporting_citations":[{"cited_title":"Unified probabilistic deep continual learning through generative replay and open set recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the outlier-rejection sampling mechanism, estimating inlier/outlier probability in latent space with a Weibull model, that the proposed replay method builds on."},{"cited_title":"Continual learning for anomaly detection with variational autoen- coder","cited_arxiv_id":null,"evidence_quote":"Supplies the use of a VAE and its ELBO as an anomaly score in a continual anomaly detection setting, the approach adopted here."},{"cited_title":"Ordisco: Effective and efficient usage of incremental unlabeled data for semi-supervised continual learning","cited_arxiv_id":null,"evidence_quote":"Provides the alternative CSSL definition with a mixture of labelled and unlabelled data in each experience, which the paper adopts."},{"cited_title":"Variational autoencoder based anomaly detection using recon- struction probability","cited_arxiv_id":null,"evidence_quote":"Establishes reconstruction probability in VAEs as a means of anomaly detection, informing the anomaly scoring choice."}],"review_version":1}