{"id":"ee7abfd8-bd64-46f8-bcc0-77337f5708fc","arxiv_id":"2508.09652","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Training a malware-detection ML model solely on signature-negative samples improves robustness to adversarial examples and temporal drift at the cost of a false-positive floor.","lead":"Malware detection AI models perform better against attacks and data shifts when trained only on samples that existing signature rules do not already catch, instead of on the full dataset. The tradeoff is a fixed minimum false-positive rate caused by imperfect rule selection.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Robustness gains may be a training-set composition artifact: excluding signature-flagged samples also excludes easy/old samples, so the comparison does not isolate the role of rule-based detection.","rationale":"The reader's weakest assumption identified the same core issue: if rule selection is suboptimal, the robustness gains may reflect discarded hard samples rather than the benefit of rule-based integration. My analysis sharpens this into a concrete confound: removing signature-flagged samples is not a neutral intervention; it changes the training distribution's difficulty and temporal composition. The abstract itself provides evidence of rule-set sensitivity by noting a fixed false-positive lower bound driven by suboptimal rule selection. This makes the central claim—that rule-based filtering improves robustness—conditional on controlling for sample-selection effects. Because the full text was garbled, I cannot check whether the paper already includes such a control; the verdict is CONDITIONAL rather than UNVERDICTED because the abstract already reveals enough to identify the required condition. If the matched control experiment confirms the effect, the claim would be substantially strengthened.","tokens_in":1081,"tokens_out":5656,"duration_ms":64008,"concrete_test":"Run a matched control experiment: train the same ML pipeline on a random subset of the full dataset with the same size and (as far as possible) the same family/timestamp distribution as the signature-excluded set, then evaluate on the same adversarial and temporal-drift benchmarks. If the control matches the reported robustness gain, the effect is a sample-selection artifact; if the control does not match, the rule-based filtering explanation is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract's central comparison changes two variables simultaneously: the training set omits signature-flagged samples, and the training distribution shifts. If the signature set is suboptimal or biased (as the abstract concedes with 'suboptimal rule selection'), the residual set is not a principled 'hard examples' set but an arbitrary leftover. Models trained on that residual may show improved adversarial robustness and temporal-drift performance simply because the evaluation distribution excludes samples signatures already catch (often older or easier malware), not because rule-based filtering benefits ML training. The reported 'fixed lower bound on false positives, driven by suboptimal rule selection' indicates outcomes are sensitive to the specific rule set, so the claimed benefit is not an invariant property of rule-based integration. Without a matched control that removes the same number and type of samples by a mechanism unrelated to signature quality, the abstract's causal attribution is unsupported. The full text is garbled in the input, so I cannot verify whether the authors already include such a control; if they do, this concern is addressed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how signature-based (rule-based) detection should be integrated into the training pipeline of ML-based Windows malware detectors. Concretely, it compares two systems: one trained on a comprehensive dataset, and one whose ML component is trained only on samples not already flagged by signatures. The abstract claims that the signature-filtered training set yields improved robustness to adversarial EXEmples and temporal data drift, at the cost of a fixed lower bound on false positives caused by suboptimal rule selection. The body of the submission as provided to the reviewer is corrupted (garbled text), so no experimental details, equations, tables, or ablation studies are legible; the present assessment is therefore based on the abstract and the reader's report.","tokens_in":1358,"tokens_out":2611,"duration_ms":31152,"significance":"If the claimed result holds, the paper addresses an important and timely question: how to combine deterministic signatures with learned models without treating the two in isolation. A crisp experimental comparison that isolates the contribution of rule-based filtering would be of practical value to the malware-detection community and would strengthen the empirical basis for hybrid detection pipelines. The abstract is also commendable for acknowledging the suboptimality of rule selection and the resulting false-positive floor, rather than presenting rule-based filtering as unambiguously beneficial. However, no verifiable evidence is currently available: the full text is garbled, and even from the abstract the central comparison appears confounded. The paper's significance is contingent on the authors providing a controlled experiment and complete reproducibility artifacts.","major_comments":[{"comment":"The body of the review copy is garbled mojibake; no dataset statistics, model architecture, training details, attack generation settings, evaluation protocol, or tables of results are legible. Without this material the central claim—that training on signature-filtered samples improves robustness—cannot be verified. The authors must resubmit a readable manuscript before any substantive review can occur.","section":"Full text (all sections after the abstract)"},{"comment":"The comparison changes two variables at once: the training set excludes signature-flagged samples, and the evaluation distribution is implicitly different because signature-flagged (often older or easier) samples are absent from the training distribution. The reported robustness gain could therefore be an artifact of the training-set composition shift, not a property of rule-based integration. A matched control is needed: remove the same number and roughly the same type of samples using a signature-independent criterion (e.g., random subsampling, or a hardness measure not derived from the rule set), and show that the robustness improvement is specific to signature-based filtering. Without such a control, the abstract's causal attribution is unsupported.","section":"Abstract, central comparison"},{"comment":"The statement 'fixed lower bound on false positives, driven by suboptimal rule selection' suggests the outcome is sensitive to the specific rule set. The paper should formally define this lower bound, justify why it is 'fixed,' and report sensitivity to rule-selection choices (e.g., using multiple signature databases or ablating subsets of rules). If the bound changes substantially with rule set, the claimed benefit of rule-based integration is not an invariant property of the approach.","section":"Abstract, cost claim"},{"comment":"The abstract does not specify whether the adversarial EXEmples are generated against the ML component alone, against the combined pipeline, or against the signature filter. The interpretation of 'improved robustness' depends critically on this choice: if adversarial examples are optimized against the full pipeline, the signature filter may simply eliminate part of the attack surface; if they are optimized against the ML model only, the comparison is more meaningful. This should be clarified in the abstract or, at minimum, in the experimental setup.","section":"Abstract, adversarial EXEmples"}],"minor_comments":[{"comment":"The paper uses 'rule-based detection' in the title and 'signature-based detection' in the abstract. These terms should be aligned and explicitly defined; otherwise the reader may infer a mismatch between the proposed contribution and the actual system.","section":"Abstract and title"},{"comment":"The term 'EXEmples' appears twice and appears to be a deliberate wordplay on 'EXE' and 'examples'. If intentional, it should be introduced and explained; if a typo, it should be corrected. Either way, consistency is needed.","section":"Abstract"},{"comment":"The corrupted encoding of the full text must be fixed. The submitted PDF/LaTeX should be regenerated to ensure all equations, references, and figures are legible.","section":"Full text"}],"recommendation":"major_revision","confidential_remarks":"The provided full text is unreadable, which may be a pipeline artifact rather than the authors' fault. If the clean version is available, the editor should request it before final disposition. The central comparison likely needs an additional control experiment, so even with a clean manuscript, major revision is warranted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ponte et al.'s abstract makes an empirically testable claim: train the ML component only on signature-negative samples and you get better robustness to adversarial EXEmples and temporal drift, at the price of a fixed false-positive floor. That is a concrete design choice and the authors are upfront that the floor is tied to suboptimal rule selection. Credit where due: they are not overselling a free lunch.\n\nThe problem is the comparison. Removing signature-flagged samples changes two variables at once—the composition of the training set and the distribution. If signature rules catch older/easier malware, the residual set may be harder or more modern, and any robustness gain could come from that shift rather than from 'rule-based integration.' The stress-test note is right: without a matched control that removes the same volume of samples by a mechanism unrelated to rule quality, the causal attribution is unsupported. The abstract's own 'suboptimal rule selection' caveat makes the result rule-set-specific, not an invariant property of the approach.\n\nI cannot check whether the full text includes such a control: the full text in our copy is garbled. No experiments, baselines, or error bars are visible. So the reader's 'unverdictable' verdict is the only honest one. The direction is sensible and under-explored, and the paper explicitly flags its limitation, which is more than many ML papers do. But the abstract alone does not establish the claim.\n\nIf the full paper contains the matched control and evaluates the integrated pipeline, it deserves a serious referee. If not, the result is a training-set artifact dressed as a design win. My recommendation: send it to peer review, but instruct the reviewers to look for that control. The question—whether rule-based detection should shape ML training data—is legitimate and worth airing.","headline":"Plausible empirical direction, but the abstract's central comparison is confounded and the full text is unverifiable—still worth a referee's look if the missing control exists.","tokens_in":1723,"tokens_out":2952,"would_cite":false,"duration_ms":30136,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that when signature-based detection is used inside the training pipeline to exclude already-flagged samples, the machine learning model becomes more robust to adversarial EXEmples and to temporal drift, at the cost of a fix","keywords":["malware detection","Windows PE executables","signature-based detection","machine learning","adversarial EXEmples","temporal data drift","false positives","rule-based filtering"],"falsifier":"A control experiment that removes the same number of samples at random from the training set, or removes the hardest samples by feature distance, and finds the same robustness gains would indicate that sample removal, not signature-based filtering, drives the result.","tokens_in":1077,"feed_emoji":"🛡️","tokens_out":4423,"duration_ms":47527,"temperature":0.7,"pith_summary":"This paper asks whether signature-based detection should be more than a separate first line of defense, specifically whether it should shape what the machine learning component sees during training. The authors compare a model trained on the complete dataset with a model whose training set excludes all samples that signature-based rules already flag. The filtered-training model is more robust to adversarial EXEmples, Windows executables crafted to evade AI detection, and more robust to temporal drift, the naturally changing distribution of malware over time. This gain has a price: a fixed lower bound on false positives, because the signature rules used to filter are imperfect. The paper argues that integrating rule-based and learned detection within the training pipeline is a useful direction, while leaving dynamic analysis as future work.","feed_headline":"Excluding signature-flagged samples improves AI malware robustness","feed_subtitle":"But it locks in a fixed false-positive floor from imperfect rules, the paper shows.","key_machinery":"The central mechanism is a rule-based pre-filter inside the training loop: signature rules first flag a subset of the dataset, and the machine learning model is trained only on samples the rules did not flag. This turns the signature set into a data-reduction and label-filtering step whose output distribution, unflagged executables, is what the learned model must master. The comparison against full-data training isolates what that filtering contributes to adversarial robustness and drift resilience.","core_discovery":"The paper's central claim is that signature-based detection belongs inside the training pipeline of a machine-learning malware detector, not only ahead of it. Concretely, the authors train two models on the same Windows executable dataset: one on all samples, and one on the subset of samples that signature-based rules do not flag. The model trained only on unflagged samples is more robust to adversarial EXEmples, and it degrades less under temporal data drift. The paper attributes this to the rule set acting as a pre-filter that reduces data complexity and forces the learning component to focus on samples that static signatures cannot already catch. The accompanying cost is a fixed lower bou","pith_inferences":["The robustness gain may be partly an artifact of sample removal: if the signature rules remove the hardest or most atypical samples, the leftover set is easier by construction; a control that removes the same number of samples at random would separate the filtering effect from the rule-specific effect.","A testable consequence: retraining on signature-flagged samples plus a small ML-only buffer should reproduce, or shift, the false-positive floor; this would show whether the floor is caused by rule mistakes or by distributional differences.","The same filtering logic could be applied to other PE malware families and to dynamic-analysis features, suggesting the improvement is a property of pre-filtering, not of the specific static-feature model.","If the temporal-drift robustness is real, the signature filter encodes persistent artifacts; one could measure how quickly the filtered model's advantage decays as the rule set ages."],"forward_implications":["If the claim holds, malware-defense pipelines should move rule-based components inside the training loop rather than only chaining them after the ML model.","Filtered training raises robustness to adversarial EXEmples, so fewer adversarial samples would evade a deployed system.","Filtered training also improves robustness to temporal drift, meaning detection performance degrades more slowly as new malware appears.","The fixed false-positive floor means a signature-filtered training set cannot achieve arbitrarily low false-positive rates unless the rule set itself is repaired.","Extending the pipeline with dynamic analysis is the paper's proposed next step for further resilience."],"supporting_citations":[],"fun_headline_variants":["Signature pre-filter strengthens AI malware models","Train AI on unflagged samples for greater robustness","Rule-based filtering in training boosts AI defenses","Excluding flagged samples hardens AI malware detection","Signature filtering in training improves AI resilience"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The comparison assumes the signature-based rules are a fixed, accurate filter whose flagged samples can be safely excluded from ML training; if the rule set is suboptimal or miscalibrated, the robustness gains may reflect discarded hard samples rather than the benefit of rule-based integration.","fun_headline_variants_meta":{"raw":{"variants":["Signature pre-filter strengthens AI malware models","Train AI on unflagged samples for greater robustness","Rule-based filtering in training boosts AI defenses","Excluding flagged samples hardens AI malware detection","Signature filtering in training improves AI resilience"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1228,"prompt_tokens":660,"completion_tokens":568,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":404,"completion_tokens_details":{"reasoning_tokens":503}},"tokens_in":404,"tokens_out":568,"duration_ms":7568,"temperature":1.0,"reasoning_tokens":503,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:53:27.723367+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A control experiment that removes the same number of samples at random from the training set, or removes the hardest samples by feature distance, and finds the same robustness gains would indicate that sample removal, not signature-based filtering, drives the result.","supporting_citations":[],"review_version":1}