{"id":"f39e18b7-6656-4c0b-8aba-96340fe11b1b","arxiv_id":"2507.08597","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A pseudo-labeling method with class-specific adaptive thresholds, label-consistent augmentation, and mixup reduces concept-drift performance loss in malware classifiers across five datasets.","lead":"ADAPT is a semi-supervised algorithm that retrains malware classifiers month by month using the model's own confident predictions, plus augmented and mixed copies, instead of waiting for new human labels. It reports higher F1 scores than several baselines on Android, Windows, and PDF malware datasets under temporal drift.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim of consistent superiority is contradicted by Table II: 3 of 15 ADAPT cells lose to their base model, and the corresponding Wilcoxon tests are non-significant.","rationale":"The reader's verdict is CONDITIONAL and already flags that 'consistently outperforms' is contradicted by several cells in Table II, while also focusing its weakest-assumption analysis on the underderived theory in Appendix H. I agree that the empirical overclaim is the more load-bearing issue: the paper's central advertised result is a universal superiority claim, and the theory is presented as support rather than as the main finding. The reader's rationale does mention the Table II contradictions, but the formal weakest_assumption field points at the gradual-drift theory, so my agreement is partial. I do not think the concern requires changing the verdict: the method clearly helps in many settings, and the correct remedy is to qualify the consistency claim and, optionally, to tighten or mark as heuristic the theoretical extension. Thus the existing CONDITIONAL verdict remains appropriate, and no verdict adjustment is needed.","tokens_in":29380,"tokens_out":4562,"duration_ms":54961,"concrete_test":"Re-run the provided ADAPT code (or independently reimplement Algorithm 1) on BODMAS, EMBER, and PDF with the same five seeds and hyperparameter search budget, then reproduce Table II and Table VI. Specifically check the three losing cells: RF+ADAPT vs RF on BODMAS and EMBER, and MLP+ADAPT vs MLP on PDF. If those differences are within noise (Wilcoxon p > 0.05, as in Table VI), or if any of the 15 pairwise ADAPT-vs-base comparisons fail to improve, then the 'consistently outperforms' claim is not supported and must be softened to a per-dataset, per-model claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section VII claim that ADAPT 'consistently outperforms baseline models and competitive benchmarks,' but the paper's own Table II does not support this. Across the 15 model-dataset cells, RF+ADAPT is worse than plain RF on BODMAS (96.1 vs 96.3) and EMBER (86.2 vs 86.3), and MLP+ADAPT is worse than plain MLP on PDF (97.2 vs 97.4). In addition, against the stated competitive benchmarks, MORSE beats RF+ADAPT and MLP+ADAPT on BODMAS (98.8 vs 96.1 and 98.6), and offline XGBoost beats MLP+ADAPT on PDF (97.9 vs 97.2). The 'consistently outperforms' claim therefore fails under a literal reading of the paper's own results. The Wilcoxon signed-rank tests in Table VI are also consistent with the concern: RF has p = 0.6875 on BODMAS, p = 0.7646 on EMBER, and p = 0.4375 on PDF, and MLP has p = 0.15625 on PDF, so the losing or flat cells are not supported as significant improvements. The theoretical appendix does not repair this, because the central claim is an empirical universal and the proof extension in Appendix H is itself conditional and asserted. What remains is a useful but uneven method: it improves most cells, sometimes substantially, but not all. The publication-level claim should be qualified to name where ADAPT helps and where it does not, rather than asserting consistent superiority.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ADAPT, a pseudo-labeling semi-supervised algorithm for malware detection under concept drift. The method combines class-dependent adaptive pseudo-label thresholds, class-consistent feature-space augmentation, and mixup regularization to retrain a base classifier on unlabeled monthly data. The authors evaluate ADAPT with Random Forest, XGBoost, MLP, and SVM on five malware datasets (Drebin, APIGraph, BODMAS, EMBER, PDF), comparing against offline models and four self-training baselines, with additional experiments on active learning, multiclass family classification, and source-free adaptation. The central claim is that ADAPT 'consistently outperforms baseline models and competitive benchmarks' and achieves state-of-the-art results across all tasks.","tokens_in":29686,"tokens_out":5959,"duration_ms":66040,"significance":"If the empirical results are taken as presented, ADAPT is a useful model-agnostic recipe for label-efficient drift adaptation in malware detection: it improves F1 and reduces false negatives on most of the 15 base-model/dataset cells, and the paper provides unusually broad validation, including multiple domains, multiple base learners, ablations, Wilcoxon tests, active learning, multiclass problems, source-free adaptation, and a computational overhead analysis. The GitHub link and the detailed hyperparameter tables also support reproducibility. However, the headline claim is stronger than the evidence: several ADAPT cells lose to their own base models and to MORSE/offline XGBoost, and many Wilcoxon tests are non-significant. The theoretical appendix does not repair this, because the extension in Eqs. (9)–(13) is asserted rather than derived. With a qualified, cell-specific claim the contribution would be solid; as written, the manuscript overstates its central result.","major_comments":[{"comment":"The abstract and the first paragraph of Section VII claim that ADAPT 'consistently outperforms baseline models and competitive benchmarks.' Table II contradicts this literal claim: RF+ADAPT is below plain RF on BODMAS (96.1 vs 96.3) and EMBER (86.2 vs 86.3), and MLP+ADAPT is below plain MLP on PDF (97.2 vs 97.4). Against the listed benchmarks, MORSE outperforms RF+ADAPT and MLP+ADAPT on BODMAS (98.8 vs 96.1 and 98.6), and offline XGBoost outperforms MLP+ADAPT on PDF (97.9 vs 97.2). The paper needs to state precisely where ADAPT helps (e.g., XGBoost across all five datasets, and the Android datasets for all base models) and where it does not; as written, the headline claim is not supported by the paper's own numbers.","section":"Abstract; Section VII, Table II"},{"comment":"The Wilcoxon signed-rank tests do not support blanket consistency. For Random Forest the p-values are 0.6875 (BODMAS), 0.7646 (EMBER), and 0.4375 (PDF); for MLP the p-value on PDF is 0.15625. These non-significant values align with the losing or flat cells in Table II. The paper should report which model–dataset combinations are statistically significant, ideally with correction for multiple comparisons, and should qualify any claim of consistent superiority accordingly.","section":"Section VII-E, Table VI"},{"comment":"The theoretical extension is asserted rather than derived. Equation (9) states that thresholding reduces ramp loss, but no proof is given that the filtered class-conditional distributions remain within the same gradual-shift regime with the same ρm and ρb used in Eq. (8). Equation (13) replaces the finite-sample term n by nτm,τb in the bound, but the weighted-mixture derivation is not shown. The statement that augmentation and mixup reduce R is also asserted and is explicitly deferred ('A detailed theoretical analysis of these effects is left for future work'). Moreover, Theorem 1 assumes equal label marginals P(Y)=Q(Y), which is at odds with the paper's motivating scenario of changing malware prevalence. The manuscript should either provide the missing steps or explicitly label this section as an intuitive adaptation of Kumar et al., not a proof.","section":"Appendix H, Eqs. (8)–(13)"}],"minor_comments":[{"comment":"Figure 9 caption refers to 'test months' for the PDF dataset, but Section V-C describes weekly data; please correct the caption to say weeks.","section":"Figure 9; Section V-C"},{"comment":"The text says 'In most scenarios, FPR is maintained or reduced alongside FNR,' but Table III shows a negative ΔFPR (i.e., increased FPR) in 8 of 15 cells, including XGBoost on APIGraph (−1.30) and RF on EMBER (−0.09). The summary sentence should be revised to match the table.","section":"Section VII-C, Table III"},{"comment":"The mean probabilities µm and µb are computed from the subsets D_mu and D_bu that are themselves defined by the current model's predictions; the pseudocode should state whether these means are computed before thresholding and whether the threshold update is applied once per month or iteratively.","section":"Section IV-A, Eqs. (1)–(4)"},{"comment":"The main experiments use a fixed budget of 200 random hyperparameter searches, while the active-learning experiments use 100 iterations; the paper should clarify whether this difference affects the comparison, or use the same budget for all methods.","section":"Section VI-C and Section VIII"},{"comment":"Table II reports means over five runs but does not report standard deviations or confidence intervals, even though Section VI-D says both mean and standard deviation are computed; adding the spread would help readers judge the losing cells.","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with an overclaiming narrative. I recommend major revision rather than rejection: the results are mostly positive, the evaluation is broad, and the main fixes are honest qualification of the central claim and either proving or clearly deflating the theoretical appendix. The authors should also be asked to report per-cell statistical significance and to adjust the abstract and Section VII so that they do not assert universal superiority."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a genuinely useful pseudo-labeling recipe for malware concept drift, but the abstract's 'consistently outperforms' does not survive Table II. Three of fifteen model-dataset cells lose to their base model, and the corresponding Wilcoxon tests are non-significant. That does not sink the paper, but it needs to be said plainly and fixed.\n\nWhat's new: the integration is real. Class-dependent adaptive thresholds, same-class augmentation with a label-consistency filter, and mixup, combined for drift adaptation on static-feature malware detectors. Each piece is known, but the combination is new and the evaluation is unusually thorough: five datasets across Android, Windows, and PDF, three (plus SVM) model classes, four self-training baselines, ablations, calibration plots, exposure analysis, active learning, multiclass, and source-free settings. Validation-period tuning and held-out test months are the right protocol, and the Wilcoxon tests are at least reported honestly enough to reveal the overclaim.\n\nSoft spots, in proportion. The main one is the consistency claim. RF+ADAPT is below RF on BODMAS and EMBER; MLP+ADAPT is below MLP on PDF; MORSE beats RF+ADAPT on BODMAS; offline XGBoost beats MLP+ADAPT on PDF. So the paper should name where ADAPT helps and where it doesn't. Second, Appendix H is not a rigorous extension of Kumar et al. Equations (9)-(13) are asserted, and the 'modified bound' is conditional on assumptions that are not checked against the data. The authors themselves say detailed analysis is left for future work; that should move into the main text or the theory should be labeled as heuristic. Third, the code link exists but is not verified; a commit hash and preprocessing steps would help.\n\nThe empirical core is sound enough that I'd trust the main direction. The paper deserves a serious referee, but the referee should require a revised abstract and Section VII, and a decision on whether the theory appendix is a proof sketch or a heuristic.\n\nFor whom: ML-for-security researchers, and anyone building practical drift adaptation for deployed malware detectors. It's a good baseline comparison and a useful discussion piece on claim calibration.\n\nRecommendation: send it to peer review with major revision required. I'd set would_accept_peer_review=true despite my skeptical reading of the headline claim.","headline":"Solid empirical package with a real overclaim: ADAPT helps on most but not all of its own experiments, and the theory appendix is asserted, not proven.","tokens_in":30249,"tokens_out":2073,"would_cite":true,"duration_ms":24606,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pseudo-labeling keeps malware detectors accurate as threats evolve","keywords":["concept drift","malware detection","pseudo-labeling","semi-supervised learning","self-training","active learning","class-dependent thresholding","mixup"],"falsifier":"Run ADAPT on any of the five datasets after injecting a single test month dominated by a previously unseen malware family with no gradual ramp-up: if the pseudo-label update lowers F1 below the offline baseline or raises false negatives for that month, the gradual-shift premise that the bound relies on is violated.","tokens_in":29129,"feed_emoji":"🛡️","tokens_out":6237,"duration_ms":73960,"temperature":0.7,"pith_summary":"The paper tries to establish that pseudo-labeling alone—retraining a malware detector on its own confident predictions over unlabeled, time-shifted samples—can keep the detector accurate as malware evolves, without waiting for costly ground-truth labels. The proposed algorithm, ADAPT, adds three safeguards against the self-poisoning that usually breaks self-training in this setting: class-specific thresholds that adapt to drift, data augmentation filtered for label consistency, and mixup regularization to calibrate confidence. Across five malware datasets spanning Android, Windows, and PDF files, the adapted models' F1 scores consistently beat both offline models and four self-training baselines, with the largest gains coming from reduced false negatives. If this holds, security teams could update deployed detectors month by month using only the unlabeled stream, slashing exposure during label delays.","feed_headline":"Pseudo-labeling keeps malware detectors accurate as threats evolve","feed_subtitle":"ADAPT's class-aware pseudo-labeling improves F1 and cuts false negatives across five malware datasets.","key_machinery":"The load-bearing mechanism is drift-aware pseudo-label selection: instead of one fixed confidence threshold, ADAPT keeps separate thresholds $\\tau_m$ and $\\tau_b$ for malware and benign classes and updates them as $\\tau_m^{\\text{updated}} = \\lambda \\mu_m + (1-\\lambda)\\tau_m$ (similarly for benign), where $\\mu_m$ and $\\mu_b$ are the model's mean predicted probabilities on the unlabeled month's data. The paper motivates this with the observation that the malware class drifts more than the benign class, so class-specific thresholds keep enough drifted malware in the training pool. Two further components stabilize the self-training loop: feature-space augmentation restricted to same-class replacements and filtered by prediction consistency, and mixup interpolation, which the paper shows improves confidence calibration. The theoretical support is the gradual domain shift self-training bound from [43], translated into a class-asymmetric form with thresholded and regularized terms.","core_discovery":"ADAPT's central claim is that a classifier can be kept current under gradual concept drift by periodically retraining on pseudo-labels selected by class-dependent adaptive thresholds. Because malware drifts more than benign software, the algorithm sets a separate confidence threshold for each class, then moves each threshold toward the model's current mean confidence on that class, so the pseudo-labeled pool tracks the shifting distribution. Retraining combines original labels, pseudo-labeled samples, label-consistent augmented samples, and mixup interpolations, which together reduce confirmation bias and keep predicted confidence aligned with actual accuracy. In the reported experiments, ADAPT improves F1 over the corresponding offline models on all five datasets, with XGBoost+ADAPT best on Drebin, BODMAS, and EMBER, MLP+ADAPT best on APIGraph, and Random Forest+ADAPT best on the PDF dataset; it also lowers false negatives and Absolute Exposure, combines with active learning under a 50-label monthly budget, and transfers to multiclass family classification.","pith_inferences":["The asymmetric-threshold idea points beyond malware: any detection task where one class evolves faster than the other, such as fraud or intrusion detection, could use the same class-specific pseudo-label selection, though threshold adaptation would need retuning for different class imbalances.","The label-consistency filter likely biases the pseudo-labeled pool toward samples the current decision boundary already handles, which could under-represent genuinely novel malware; a testable extension would track how much of the F1 gain comes from boundary samples versus samples deep in the existing region.","If drift turns out to be abrupt rather than gradual, the appendix's bound no longer applies and the margin over offline baselines should narrow; a stress test with sudden new families would reveal where the gradual assumption starts to bite.","The mixup component's main practical benefit may be calibration rather than raw accuracy: better-calibrated confidence makes the class thresholds meaningful, so the two mechanisms should be evaluated jointly rather than as independent tricks."],"forward_implications":["Deployed malware detectors can be refreshed monthly on unlabeled test traffic, so the damage from delayed sandbox or analyst labels shrinks.","Because ADAPT is model-agnostic, teams can keep fast, interpretable models like Random Forest or XGBoost and still track drift, avoiding a forced switch to neural networks.","With a tiny annotation budget of 50 samples per month, ADAPT plus active learning outperforms random labeling and the prior active-learning baseline on both Android datasets.","When the original training set cannot be retained for privacy or storage reasons, the source-free variant still improves over the offline baseline on the first year of Android test data.","The same algorithm extends to multiclass malware family classification, where it raises macro F1 over XGBoost on BODMAS and EMBER."],"supporting_citations":[{"why":"Supplies the gradual domain shift self-training error bound that the paper extends to class-asymmetric thresholds.","marker":"[43]"},{"why":"Documents the self-poisoning failure of labelless drift adaptation that ADAPT is designed to avoid.","marker":"[41]"},{"why":"Provides the continuous learning protocol, the Android datasets, and the active learning baseline used for evaluation.","marker":"[24]"},{"why":"Supplies the feature-space augmentation method that ADAPT modifies with class-conditional replacement and label-consistency filtering.","marker":"[76]"},{"why":"Provides the Drebin Android feature set and dataset used in the experiments.","marker":"[12]"},{"why":"Provides the APIGraph semantically clustered Android feature set and dataset used in the experiments.","marker":"[83]"},{"why":"Provides the EMBER Windows PE malware dataset used for evaluation.","marker":"[7]"},{"why":"Provides the BODMAS Windows PE malware dataset and its temporal splits.","marker":"[79]"},{"why":"Provides the PDF malware dataset used in the evaluation.","marker":"[66]"},{"why":"Supplies the mixup regularization technique that the paper applies for confidence calibration.","marker":"[81]"}],"fun_headline_variants":["Pseudo-labels keep malware AI sharp as threats change","ADAPT's pseudo-labeling fights concept drift in malware","Class-aware pseudo-labeling beats malware concept drift","Pseudo-labeling adapts malware detectors to drift","Fight malware drift with adaptive pseudo-labeling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole adaptation story rests on the assumption that malware drift is gradual—each month's distribution is only a small step from the last—and that the share of malware versus benign samples stays roughly constant; if a new malware family lands all at once, the theoretical guarantee and the pseudo-labeling safeguard have no footing.","fun_headline_variants_meta":{"raw":{"variants":["Pseudo-labels keep malware AI sharp as threats change","ADAPT's pseudo-labeling fights concept drift in malware","Class-aware pseudo-labeling beats malware concept drift","Pseudo-labeling adapts malware detectors to drift","Fight malware drift with adaptive pseudo-labeling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000824,"raw_usage":{"total_tokens":3583,"prompt_tokens":901,"completion_tokens":2682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":2606}},"tokens_in":517,"tokens_out":2682,"duration_ms":18823,"temperature":1.0,"reasoning_tokens":2606,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:14:40.630707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ADAPT on any of the five datasets after injecting a single test month dominated by a previously unseen malware family with no gradual ramp-up: if the pseudo-label update lowers F1 below the offline baseline or raises false negatives for that month, the gradual-shift premise that the bound relies on is violated.","supporting_citations":[{"cited_title":"Understanding self- training for gradual domain adaptation","cited_arxiv_id":null,"evidence_quote":"Supplies the gradual domain shift self-training error bound that the paper extends to class-asymmetric thresholds."},{"cited_title":"Investigating labelless drift adaptation for malware detection","cited_arxiv_id":null,"evidence_quote":"Documents the self-poisoning failure of labelless drift adaptation that ADAPT is designed to avoid."},{"cited_title":"Continuous learning for android malware detection","cited_arxiv_id":null,"evidence_quote":"Provides the continuous learning protocol, the Android datasets, and the active learning baseline used for evaluation."},{"cited_title":"From grim reality to practical solution: Malware classification in real-world noise","cited_arxiv_id":null,"evidence_quote":"Supplies the feature-space augmentation method that ADAPT modifies with class-conditional replacement and label-consistency filtering."},{"cited_title":"Enhancing state-of-the-art classifiers with api semantics to detect evolved android malware","cited_arxiv_id":null,"evidence_quote":"Provides the APIGraph semantically clustered Android feature set and dataset used in the experiments."},{"cited_title":"Bodmas: An open dataset for learning based temporal analysis of pe malware","cited_arxiv_id":null,"evidence_quote":"Provides the BODMAS Windows PE malware dataset and its temporal splits."},{"cited_title":"Hidost: a static machine-learning-based detector of malicious files","cited_arxiv_id":null,"evidence_quote":"Provides the PDF malware dataset used in the evaluation."},{"cited_title":"Dauphin, and David Lopez- Paz","cited_arxiv_id":null,"evidence_quote":"Supplies the mixup regularization technique that the paper applies for confidence calibration."}],"review_version":1}