{"id":"fb281a08-2b15-4ee0-a4c9-6bcac05c3c2c","arxiv_id":"2412.21084","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Storage-level ransomware detection with 82 lightweight IO features (entropy, LBA, transfer-size histograms) generalizes across file systems, volume states, and encryption only when trained on representative configurations, outperforming the prior Hirano feature set by up to 12.8% median F1.","lead":"This paper studies when machine-learning ransomware detectors that watch block-level input/output operations keep working after the storage setup changes: different file systems, disk aging, workloads, encrypted volumes, and virtual machines. It finds single-configuration training often fails on new configurations, and a richer feature set plus broader training cuts false positives by up to 17.1 percent versus the prior feature set.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-ransomware validation (Sec. 6.2) trains on WannaLaugh emulator traces plus 1% of the same strains' real traces, so the low median FNRs (0.9%, 1.95%) do not demonstrate detection of unseen ransomware families.","rationale":"I agree with the reader's weakest-assumption identification. The real-ransomware validation is the weakest link in the chain connecting the controlled cross-configuration experiments to the deployment-ready claim. The comparative advantage over Hirano is a controlled, internally consistent finding, though it also lacks error bars. But the headline 'remarkable effectiveness' is anchored to the Table 2/3 FNRs, which are obtained under an in-family protocol. A single retraining experiment can settle whether the 1% real trace inclusion is essential. This does not invalidate the paper's broader mapping of where storage-based detection fails; it only narrows the real-world claim. Therefore the verdict should remain CONDITIONAL, pending this test and potentially artifact release.","tokens_in":18347,"tokens_out":9803,"duration_ms":88703,"concrete_test":"Retrain the Section 6.2 model using only benign traces and WannaLaugh emulator traces (zero-shot, no real ransomware in training) and evaluate on the held-out real ransomware samples in Tables 2 and 3; if the median FNR rises substantially above the reported 0.9%/1.95% (or Lockbit exceeds its already-high FNR), the paper's claim that storage-based detection 'accurately detects ransomware in most setups' depends on in-family training.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.2 states that the model is trained using 'WannaLaugh ransomware emulator as well as using 1% from collected real ransomware traces.' The WannaLaugh emulator is configurable to mimic specific strains (Black Basta, Conti, Lockbit, Lockfile, WannaCry) and is authored by a co-author, so the tested strains align with the emulator's design. The inclusion of 1% real traces from each evaluated family makes the evaluation partially in-sample. This is load-bearing because the abstract's 'remarkable effectiveness' and the intro's 'accurately detect ransomware in most setups' rest on these Table 2/3 FNRs. Lockbit, which uses intermittent encryption, already shows 19.0-65.1% FNR even with the 1% in-family training; without it, performance would likely be worse. If the 1% seed or the emulator's fidelity were removed, the observed low median FNRs would not transfer to deployment on unseen or evolving ransomware.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical study of ML-based ransomware detection at the block-storage layer. The authors propose a device-mapper kernel module and a computational-storage-device architecture that extract windowed I/O features, and they evaluate an XGBoost classifier across volume-utilization states, filesystem types, benign workloads (file conversion, compression, OLTP), qcow2 copy-on-write images, LUKS/BitLocker encryption, and a set of real ransomware samples. The main claims are that models trained on a single configuration do not transfer (e.g., an XFS-trained model has 51.79% F1 on NTFS; unseen PostgreSQL traces produce FPR up to 55.21% on XFS), that including training data from the diverse configurations restores robustness, and that the proposed 82-feature set outperforms the Hirano feature set by up to 12.8 percentage points in median F1, 10.9 percentage points in FNR, and 17.1 percentage points in FPR. The real-ransomware validation in Section 6.2 reports low median FNRs (0.9% on NTFS, 1.95% on XFS) but trains the model on WannaLaugh emulator traces plus 1% of the same collected real-ransomware traces.","tokens_in":18471,"tokens_out":7915,"duration_ms":78162,"significance":"Assuming the central results are reproducible, the paper's contribution is a useful and unusually broad generalizability map of storage-based ransomware detection. The evaluation has several genuine strengths: it tests many configurations on the same underlying traces, compares the proposed feature set against the Hirano features on identical data, validates the volume-aging procedure with Geriatrix, uses real malware samples in isolated VMs, and measures detection overhead. The comparative advantage over the Hirano features, if confirmed under a fair protocol, is practically relevant. The main weakness is that the deployment-facing claims in the abstract and introduction are supported by an evaluation protocol in which the model is trained on 1% of the same ransomware strains it is then scored on, so the reported real-world FNRs are not evidence of detection of unseen families. This does not invalidate the comparative or generalizability results, but it does require reframing and additional zero-shot numbers.","major_comments":[{"comment":"Section 6.2 states that the model used for Tables 2 and 3 is trained on samples from all previous traces, from the WannaLaugh ransomware emulator, and using 1% from collected real ransomware traces. Because the real-ransomware test set is drawn from the same collected traces, the low median FNRs (0.9% on NTFS, 1.95% on XFS) are not zero-shot results and do not by themselves support the abstract's claim of remarkable effectiveness for deployment against unseen ransomware. This point is load-bearing because Lockbit, an intermittent-encryption strain that the WannaLaugh emulator was designed to mimic, already shows 19.0-65.1% FNR even with the 1% in-family training. I ask the authors to (a) report the evaluation with no real-ransomware training data, (b) state exactly how the 1% was selected and how much of each strain's trace data was used for training versus testing, and (c) separate in-family from out-of-family results in the abstract and in Section 6.2.","section":"Section 6.2, Tables 2 and 3"},{"comment":"The central comparative claim against Hirano's features is stated as a difference in median F1/FNR/FPR, but the manuscript does not say whether the same hyperparameters, feature-window sizes, and train/test splits were used for the Hirano-feature models as for the proposed-feature models. Section 2.4 says XGBoost hyperparameters were optimized and used for all results; if those optimized hyperparameters were not separately tuned for the baseline feature set, part of the reported 12.8%/10.9%/17.1% gains could be an artifact of the tuning protocol. Please state explicitly that the comparison is apples-to-apples, or adjust the claim if the baseline models were not tuned in the same way.","section":"Sections 2.4, 3.1, 3.2, and 6.2"},{"comment":"Most headline results are single point estimates without a measure of uncertainty. For example, Section 3.2 reports F1: 51.79%, FPR: 3.01%, FNR: 64.01% for an XFS-trained model on NTFS, and Section 3.1 reports a 15.0% F1 drop as evidence of volume-state sensitivity. The manuscript gives no confidence intervals, standard deviations (except the Geriatrix runs), or number of independent trace repetitions behind these values. Since several conclusions depend on differences of only a few percentage points (e.g., the up-to-2% improvement from file-system awareness and the 1.67% versus 6.14% F1 differences against Hirano features), I ask for error bars or a statement of variance over repeated training runs and evaluation folds.","section":"Sections 3.1-3.3 and 6.2"}],"minor_comments":[{"comment":"The feature count is not self-consistent: 19 entropy features plus 18 LBA features for each of read and write (36) plus 17 transfer-size features for each IO type (34) does not sum to the stated 79/82 features. Please provide a complete feature-count breakdown or correct the text.","section":"Section 2.4"},{"comment":"Table 1 would benefit from a caption stating which benign workloads and training traces were used for the motivational NTFS model and the number of traces in each cell, so the reader can see whether the NTFS and Linux rows are directly comparable.","section":"Table 1"},{"comment":"The sentence 'we detect an attack after at most 5.2 s from the start of the ransomware encryption' appears to assume that the first feature window begins at the attack start; please clarify whether this is a worst-case bound over arbitrary phase alignment of the encryption start within an epoch.","section":"Section 6.1"},{"comment":"There are several typos and grammatical errors, including 'similiar' in Section 3.1, 'efficent' in Section 3.3, 'studiy' in Section 6.1, and 'filesytem' in the Introduction; these should be corrected.","section":"Throughout"},{"comment":"The paper would be substantially easier to verify if the traces, feature-extraction scripts, and model configuration were released or referenced as an artifact; at minimum, please add a data-availability statement.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"Editor: The main technical concern is the Section 6.2 evaluation protocol, which makes the deployment-facing FNR claims in-sample rather than zero-shot. I do not see evidence of misconduct, but the strongest deployment claim should not appear before zero-shot numbers are provided. The novelty and scope appear suitable for a security/systems venue, and the comparison against the Hirano features is useful if the tuning protocol is clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful empirical map of where block-level ransomware detection generalizes and where it breaks, and the feature-set comparison against Hirano is fair and informative. The caveat is that the real-ransomware numbers in Tables 2 and 3 are partly in-sample, so treat them as upper bounds.\n\nThe genuinely new content is the generalizability study itself. The paper measures what happens when models trained on one filesystem are evaluated on others (XFS-trained on NTFS: F1 51.79%, FNR 64.01%), across volume states (52% vs 77% utilization, plus aging), with qcow2 copy-on-write, under LUKS and Bitlocker encryption, and on unseen benign OLTP workloads (PostgreSQL FPR up to 55.21% on XFS). These findings are internally consistent and backed by some external anchors: Geriatrix aging, actual malware samples, and a like-for-like comparison with Hirano's features on the same traces. The 82-feature set is an incremental extension of prior feature families, but the side-by-side comparison shows real gains, especially in cross-configuration FPR. The overhead data (268µs average latency increase, up to 15.3% throughput reduction) is useful for practitioners.\n\nThe main soft spot is the real-ransomware evaluation. Section 6.2 trains on WannaLaugh emulator traces plus 1% of the collected real traces from the same families it then scores. So Tables 2 and 3 are not zero-shot. The low median FNRs (0.9% on NTFS, 1.95% on XFS) do not demonstrate detection of unseen ransomware. Lockbit, which uses intermittent encryption, already shows 19-65% FNR even with that in-family seed. This is load-bearing for the abstract's 'remarkable effectiveness' and 'accurately detect ransomware in most setups' phrasing. If those claims stay, the authors need a zero-shot evaluation. The WannaLaugh emulator is also from the same group, so emulator fidelity is a real question.\n\nOther issues are more routine. Headline F1/FPR/FNR values have no confidence intervals. No code or data is released, which is a problem for a paper whose contribution is an empirical map. The greedy feature-removal analysis (Figure 6) selects features using the target test set, so treat it as exploratory, not validation. The 'outperforms currently available detection' framing overshoots, since the quantitative comparison is only against Hirano's feature set, not DeftPunk or other storage-based systems.\n\nI don't think the central generalizability findings are in doubt: single-configuration training fails, LBA and transfer-size features mislead across configurations, and encryption and COW need their own training data. That part holds up. The paper just isn't ready as a strong accept. It deserves a serious referee, and with released artifacts and a real zero-shot evaluation it could be a solid systems-security paper. I'd bring it to reading group.","headline":"Useful empirical map of where storage-based ransomware detection generalizes, but the real-ransomware numbers are partly in-sample and need a zero-shot rerun.","tokens_in":19115,"tokens_out":4505,"would_cite":true,"duration_ms":38931,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Storage-level ransomware detection generalizes only when training covers the target configurations, and a new 82-feature descriptor makes that coverage outperform prior feature sets.","keywords":["ransomware detection","block storage","block IO analysis","generalizability","decision-tree boosting","feature engineering","filesystem diversity","storage security"],"falsifier":"Hold out an entire ransomware family from training, one that the emulator does not mimic, run its traces through the trained model, and measure the false negative rate; if the rate reaches the 60% level the paper already reports for Lockbit, the deployment generalizability claim is falsified.","tokens_in":18024,"feed_emoji":"🛡️","tokens_out":11698,"duration_ms":109935,"temperature":0.7,"pith_summary":"Ransomware leaves a signature in the block IO stream it generates, which makes storage an appealing detection point that attackers cannot easily disable from inside the guest OS. This paper tries to establish how far storage-based, machine-learning detection can be pushed across realistic configurations: different filesystems, disk utilization and aging, benign database and compression workloads, copy-on-write VM images, and device encryption. The central finding is that models trained on one narrow setup fail on others, while training across a broad set of configurations restores dependable detection. Against the earlier storage feature set, the proposed 82-feature descriptor yields up to 12.8% higher median F1, 10.9% lower false negative rate, and 17.1% lower false positive rate. If the generalizability claim holds, storage vendors can deploy a single detection layer with manageable false alarms, provided training traces cover the configurations they operate.","feed_headline":"Storage ransomware models fail on new setups unless trained broadly","feed_subtitle":"An 82-feature IO descriptor cuts false positives by up to 17.1% versus prior storage detection.","key_machinery":"The load-bearing mechanism is the 82-feature descriptor computed from each fixed-time window of block IO: per-epoch entropy statistics and histograms, logical block address (LBA) read/write statistics and histograms, transfer-size statistics and histograms, a mean rewrite entropy over one epoch, and a one-hot encoding of the filesystem type. These features are light enough to be extracted inline by a kernel module and, in the target architecture, directly in computational storage hardware; classification is done by a gradient-boosted decision-tree model. The feature set is what lets a single model survive changes in volume utilization, filesystem, workload, copy-on-write virtualization, and encryption, whereas coarser metrics and smaller feature sets do not.","core_discovery":"The paper claims that a lightweight, 82-feature IO descriptor plus decision-tree boosting gives storage-level ransomware detectors the generalizability they lacked, and that this is demonstrated by a systematic sweep across volume states, filesystems, workloads, copy-on-write VM images, and device encryption. It reports that a single-configuration model is not portable: an XFS-trained model evaluated on NTFS falls to 51.79% F1 with a 64.01% false negative rate, and an unseen PostgreSQL workload can raise FPR to 55.21% on XFS. It further claims that including the problematic configuration in training, using file-system awareness, and using histograms of entropy, LBA, and transfer size rather than coarse aggregates is what restores accuracy. In its real-ransomware validation, median false negative rates are 0.9% on NTFS and 1.95% on XFS, with Lockbit remaining difficult at 19% to 65% FNR depending on setup.","pith_inferences":["My inference: for storage vendors, the practical takeaway is to include deployment-specific traces (filesystem, volume age, workload mix, VM image format, encryption mode) in training rather than trusting a universal model.","My inference: the low real-ransomware FNRs should not be read as zero-shot performance, because one percent of traces from each tested family is part of the training set; a stronger evaluation would hold out entire families.","My inference: a natural next experiment is to measure how little per-configuration trace data is needed to restore accuracy, and whether incremental retraining on small samples of new environments is sufficient."],"forward_implications":["A single-configuration model is unsafe in production: the paper measures an XFS-trained model at 51.79% F1 and 64.01% false negative rate on NTFS, so training must include the target filesystem.","Benign workload coverage is the main lever for false positives: adding MySQL traces to training brings FPR on unseen MySQL workloads below 1%, while an unseen PostgreSQL workload can still push FPR to 55.21% on XFS.","Copy-on-write VM images and device encryption alter IO patterns enough to break plain-device models, and adding those trace types to training restores accuracy, so virtualized and encrypted deployments need their own traces.","The proposed 82-feature set beats the prior storage feature set in cross-filesystem and cross-workload comparisons, with up to 12.8% higher median F1, 10.9% lower false negative rate, and 17.1% lower false positive rate.","Real-time detection is feasible at modest cost: feature derivation and inference finish in about 200 ms per epoch, the kernel collection adds 268 µs average latency, and detection completes within about 5.2 s under the tested settings."],"supporting_citations":[{"why":"sets up the prior storage-based detection feature set that the paper's comparative claim must beat and reproduces to motivate the generalizability study.","marker":"[24]"},{"why":"extends the storage feature set and reports high accuracy on a few setups; the paper uses it as a baseline and as a source of window-size guidance.","marker":"[20]"},{"why":"supplies the configurable ransomware emulator used to generate the ransomware-like training traces across the generalizability studies.","marker":"[19]"},{"why":"supplies the sysbench OLTP benchmark used to generate MySQL and PostgreSQL benign workloads that drive the workload-generalizability results.","marker":"[4]"},{"why":"provides the file-system aging methodology used to validate long-term volume-state effects on model accuracy.","marker":"[27]"},{"why":"presents the cloud storage detection pipeline with a two-level classifier that this paper positions against in the design space and comparison table.","marker":"[56]"}],"fun_headline_variants":["Single-config ransomware detectors fail on new workloads, broad training fixes","82-feature descriptor makes storage ransomware detection work across setups","Cut storage ransomware false positives by up to 17% with lightweight features","Why ransomware detectors trained on one filesystem crash on another"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model trained on traces from a configurable ransomware emulator plus one percent of traces from the same real ransomware families will still detect unseen ransomware when deployed; if the emulator's traces are not representative of new or future strains, the reported low false negative rates will not transfer to real attacks.","fun_headline_variants_meta":{"raw":{"variants":["Single-config ransomware detectors fail on new workloads, broad training fixes","82-feature descriptor makes storage ransomware detection work across setups","Cut storage ransomware false positives by up to 17% with lightweight features","Why ransomware detectors trained on one filesystem crash on another"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001112,"raw_usage":{"total_tokens":4666,"prompt_tokens":1012,"completion_tokens":3654,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":3583}},"tokens_in":628,"tokens_out":3654,"duration_ms":29615,"temperature":1.0,"reasoning_tokens":3583,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:03:41.138644+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out an entire ransomware family from training, one that the emulator does not mimic, run its traces through the trained model, and measure the false negative rate; if the rate reaches the 60% level the paper already reports for Lockbit, the deployment generalizability claim is falsified.","supporting_citations":[{"cited_title":"Machine learn- ing based ransomware detection using storage access patterns obtained from live-forensic hypervisor","cited_arxiv_id":null,"evidence_quote":"sets up the prior storage-based detection feature set that the paper's comparative claim must beat and reproduces to motivate the generalizability study."},{"cited_title":"Ransomware detection with machine learning in storage systems","cited_arxiv_id":null,"evidence_quote":"extends the storage feature set and reports high accuracy on a few setups; the paper uses it as a baseline and as a source of window-size guidance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the configurable ransomware emulator used to generate the ransomware-like training traces across the generalizability studies."},{"cited_title":"Akopytov/sysbench: Scriptable database and system performance benchmark, 2020","cited_arxiv_id":null,"evidence_quote":"supplies the sysbench OLTP benchmark used to generate MySQL and PostgreSQL benign workloads that drive the workload-generalizability results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the file-system aging methodology used to validate long-term volume-state effects on model accuracy."},{"cited_title":"Ran- som access memories: Achieving practical ransomware protection in cloud with DeftPunk","cited_arxiv_id":null,"evidence_quote":"presents the cloud storage detection pipeline with a two-level classifier that this paper positions against in the design space and comparison table."}],"review_version":1}