{"id":"b12cac83-343f-4626-b6dc-dfc6f53b7008","arxiv_id":"2506.23909","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RawMal-TF is a new public dataset of raw Windows malware binaries labeled by 14 behavioral types and 17 families, with EMBER static features and classification benchmarks.","lead":"Researchers built a new malware dataset, RawMal-TF, that pairs raw Windows binaries with both behavior-type and family labels and shows standard machine-learning classifiers reach high accuracy on it. The dataset is meant to give researchers raw files plus two levels of labels for training and testing malware classifiers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central dataset-value claim rests on labels that are never independently validated; family labels are parsed from AV detection names and type labels inherit ClarAVy behavioral tags, with multi-tag assignment unspecified.","rationale":"The reader's weakest assumption identifies exactly the load-bearing point: if family names parsed from filenames and ClarAVy behavioral tags are not reliable ground truth, then the dataset's main value and the experimental conclusions are both undermined. My stress-test agrees with this assessment and adds a concrete internal ambiguity: ClarAVy records can contain multiple behavioral tags, and the paper does not specify how such samples are mapped to a single type label. This is not a claim of fraud or misconduct; it is a gap in the labeling procedure that should be independently checkable because the raw binaries are released. At the same time, I would not move the verdict from CONDITIONAL. The authors do provide raw binaries alongside features, which means the community can re-verify labels and re-run experiments; the dataset may still be useful even if some labels are noisy. The classification experiments are standard and largely internally consistent, and the paper acknowledges several limitations in its collection process. The concern is real but conditional: it should be resolved by label validation and by clarifying the multi-tag handling, not by rejecting the contribution outright. The reader's conditional verdict is therefore the appropriate recommendation, and my concern does not change it.","tokens_in":25646,"tokens_out":4273,"duration_ms":51896,"concrete_test":"Independently validate a random sample of the released binaries: for at least 50 samples per family (17 families) and 50 per type (14 types), obtain VirusTotal reports and run AVClass and ClarAVy independently, and for family labels also compare against MalwareBazaar YARA rules where available. Compute per-class label agreement between the dataset labels and the independent verdicts; if agreement is below 90% for either label level, the labels are not reliable enough to support the reported accuracies. Additionally, scan the ClarAVy label files for records containing multiple BEH tags, report their frequency, and check whether the released dataset duplicates such samples across type directories; if duplication exists, rerun the multiclass experiments after deduplication to see whether the reported 81.1% and 73.4% accuracies shift materially.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is a dataset claimed to be reliably labeled by malware type and family, but the label-generation pipeline in Sections 3.3.1 and 3.3.2 never establishes label ground truth. Family labels are obtained by parsing filenames such as 'UDS-Backdoor.Win32.Bladabindi.dej' (Section 3.3.2); these are antivirus detection names, not curated family identities, and different AV vendors use different naming conventions. The paper excludes obviously generic names but provides no independent check that a parsed name actually identifies the correct family. Type labels are taken directly from ClarAVy behavioral tags (Section 3.3.1), which are themselves derived from aggregated AV reports. Moreover, Listing 1 shows a sample carrying two BEH tags ('BEH:adware' and 'BEH:downloader' in the same record), yet the paper does not specify how such multi-tag samples are assigned to a single type category, whether they are duplicated into multiple class directories, or discarded. This ambiguity directly affects the class counts in Table 1 and the validity of the single-label multiclass experiments in Section 4.3. If labels are noisy or assignment is arbitrary, the reported accuracies measure agreement with the labeling procedure rather than with true malware type or family, which is the main risk to the dataset's usability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RawMal-TF, a dataset of Windows PE malware binaries labeled at two granularities: 14 behavioral types derived from ClarAVy BEH tags on VirusShare samples, and 17 families parsed from filenames in VX Underground's MalwareBazaar archives. The authors extract EMBER static features from the raw binaries, release both binaries and feature vectors, and evaluate five classifiers (KNN, MLP, SVM, XGBoost, Random Forest) on malware-vs-benign detection, pairwise interclass discrimination, and multiclass type/family classification. Reported results include 98.5% and 98.98% average accuracy for full-scope type- and family-based binary detection, 97.5% and 93.7% average pairwise accuracy for inter-type and inter-family tasks, and 81.1% (SVM) and 73.4% (RF/XGBoost) multiclass accuracy for types and families, respectively.","tokens_in":25930,"tokens_out":8491,"duration_ms":87113,"significance":"If the labels are trustworthy, RawMal-TF addresses a genuine gap: most public malware benchmarks provide only feature vectors, whereas this work releases raw binaries alongside dual-level type/family labels and EMBER-compatible features. The public GitHub release, the transparent description of the feature-processing pipeline, and the detailed per-class performance tables are useful contributions. However, the central value of the dataset depends on label correctness, and the paper does not independently validate either the type labels or the family labels. The label-generation pipeline contains an unspecified multi-tag assignment rule, inconsistent category-count arithmetic, and a temporal mismatch between 2018 benign samples and 2023-2025 malware. These issues mean the reported accuracies should currently be interpreted as agreement with the labeling procedure rather than as validation against ground truth. If the labeling concerns are resolved, the resource would be a solid community asset; in the present form, the core dataset claim needs additional support.","major_comments":[{"comment":"The paper does not specify how samples carrying multiple BEH tags are assigned to the single-label type categories used in the experiments. The example record contains both BEH:adware and BEH:downloader, each supported by six engines; if such samples are placed in multiple category directories, the class directories overlap and the multiclass experiments in Section 4.3 are invalid because one sample can appear in more than one class. Please state the assignment rule (e.g., first tag, majority vote, or exclusion) and report how many of the retained samples had multiple BEH tags.","section":"Section 3.3.1, Listing 1"},{"comment":"Family labels are obtained by parsing antivirus detection names from filenames such as 'UDS-Backdoor.Win32.Bladabindi.dej', with no independent verification. Antivirus naming conventions vary by vendor, and one family can appear under multiple names (njRAT and Bladabindi refer to the same family), while some names are generic or vendor-specific. The exclusion of generic names is described, but no check is provided that the remaining parsed names correspond to the intended families. Please add a validation step on a random subset using an independent source (e.g., MalwareBazaar tags, YARA rules, or a second labeling tool) and report agreement rates; without this, the family-level accuracies measure consistency with the parsing rule rather than correctness of the family labels.","section":"Section 3.3.2"},{"comment":"The category-count arithmetic is internally inconsistent. The text reports 76 unique behavioral categories, 56 categories excluded for having fewer than 1,000 samples, and 17 categories with no samples in the selected archives, yet 14 categories are retained; 76 - 56 - 17 = 3, and 76 - 56 = 20, leaving six categories unaccounted for. In addition, Table 1 includes BEH adware with 967 samples, below the stated 1,000-sample threshold. Please reconcile these numbers and specify which categories were retained at each filtering step so that Table 1 can be reproduced exactly.","section":"Section 3.3.1, Table 1"},{"comment":"The benign samples come from EMBER 2018, whereas the malware samples were collected from 2023-2025 (Section 3.3). This temporal mismatch is a possible confound in the malware-vs-benign experiments: models may separate malware from benign files based on compiler toolchains, packers, or PE-layout changes over time rather than on malicious behavior. Please report a time-matched control (e.g., benign PE files from the same period) or analyze the extent of the distribution shift, and temper the corresponding accuracy claims accordingly.","section":"Section 4.1.1 Step 3 and Section 4.2"},{"comment":"The high-level multiclass accuracy claims should be qualified by the per-class results. Type-based multiclass accuracy of 81.1% is accompanied by per-class recall of 49% for worm, 51% for virus, and 57% for downloader; family-based multiclass accuracy of 73.4% is accompanied by recall of 35% for Noon and 41% for Agensla. These low-recall classes are central to malware taxonomy, so the abstract's characterization of the results as enabling 'more fine-grained and insightful malware classification' goes beyond what the reported per-class metrics support. Please report macro-averaged results prominently and discuss the class-dependent difficulty explicitly.","section":"Section 4.3, Tables 15 and 18"}],"minor_comments":[{"comment":"The text says 'Section 1 introduces the necessary background,' but the background appears in Section 2; the cross-reference should be corrected.","section":"Section 1, last paragraph"},{"comment":"The name ClarAVy is typeset inconsistently as 'ClarA Vy' in the Abstract and Section 3.3.1; the official spelling should be used uniformly.","section":"Throughout"},{"comment":"The meaning of the integer after the hash (e.g., '53', '54') is not explained in the text; clarify whether it is the total number of antivirus detections for that sample.","section":"Listing 1"},{"comment":"The columns 'Avg FP' and 'Avg FN' are labeled ambiguously; they appear to be counts rather than rates, but the text does not state this, and Table 15 uses FPR/FNR percentages. The units should be unified or explicitly defined.","section":"Table 18"},{"comment":"The exact family-name extraction rule (substring matching vs. regular expression, and which token of the filename is taken) is not specified; providing the parser pseudocode or a regex pattern would make the dataset construction reproducible.","section":"Section 3.3.2"},{"comment":"The benign samples are EMBER 2018 feature vectors, not raw benign binaries; the dataset description should state this limitation explicitly, since the abstract claims that raw binaries are included alongside features and labels.","section":"Section 4.1.1 Step 3"},{"comment":"The arrow notation in misclassification rates (e.g., 'autorun↔ worm 20% (autorun), 44.5% (worm)') is ambiguous about which class is the true class; clarify the direction for each reported pair.","section":"Section 4.2.2, Table 8"},{"comment":"The text states the type-based multiclass dataset contains approximately 14,000 samples, but Table 1 lists BEH adware with only 967 samples; the exact total should be stated.","section":"Section 4.3.1"}],"recommendation":"major_revision","confidential_remarks":"The core risk is label validity, not the experiments themselves. The revision should prioritize an independent label validation step (e.g., a random subset checked against MalwareBazaar metadata or a second labeling tool) and the clarification of the multi-tag assignment rule. The category-count inconsistency in Section 3.3.1 is fixable but should be corrected before the dataset is widely used. The paper is within scope for a computer-security venue as a dataset resource, provided the label-quality claims are substantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real dataset release that fills a niche no existing resource quite covers — raw PE binaries with both type-level and family-level labels plus an EMBER-compatible feature pipeline — but the labels are the load-bearing wall and they are never independently validated. Family labels are parsed from AV detection filenames like UDS-Backdoor.Win32.Bladabindi.dej; type labels are ClarAVy behavioral tags aggregated from AV reports. The reported accuracies mostly measure how well models reproduce the labeling procedure, not whether the labels are correct.\n\nWhat is genuinely new is the artifact. EMBER has features and benign/malicious labels but no raw binaries and no type or family labels; MOTIF has ground-truth families but no raw binaries and no type labels. The combination is useful. The experiments are a real workload — full and truncated detection, pairwise interclass separation, multiclass with per-class metrics, grid search with cross-validation, training-time tables — and the numbers are internally consistent: phishing and redirector are easy, spyware and worm are hard, family multiclass is harder than type multiclass. The paper also discloses its messes: VirusTotal API limits, MalwareBazaar API failures, twelve million SINGLETON exclusions. Credit where due.\n\nSoft spots, in order of softness. First, label validity. The family-parsing rules are never specified — no regex, no disambiguation — and there is no spot-check against MalwareBazaar metadata or MOTIF. Several family names (Agensla, Stealerc, Convagent) look like vendor-specific detection tokens. The paper does not hide their origin, but it claims a well-labeled dataset without showing the labels are right, and its limitation section never lists label uncertainty as a limitation.\n\nSecond, the multi-tag ambiguity in Listing 1 is real: a sample carries both BEH:adware and BEH:downloader tags. The text never says whether such samples are duplicated into multiple type directories, assigned by a rule, or dropped. That bears directly on Table 1's counts and on the single-label multiclass setup; duplicates across class directories make leakage a live question.\n\nThird, benign samples come from EMBER 2018 while the malware is 2023-2025; that temporal gap can only flatter the near-perfect detection numbers, and the paper does not engage with it. Minor: there is no comparison to existing datasets, so the value claim is unquantified, and the 76/17/14 category narrative in Section 3.3.1 is confusing.\n\nNone of this is fatal. The authors do not overclaim the method, and the artifact is public. For malware-ML researchers who need raw binaries with two label granularities, this is the right starting point once the labeling questions are settled. Send it to serious peer review and push for three additions: a small validation study (a few hundred hashes checked against MalwareBazaar's own tags and against MOTIF where families overlap), an explicit multi-tag assignment rule, and the actual parsing code. With those in place, RawMal-TF becomes a citable resource instead of a promising one.","headline":"A real, useful raw-PE dataset whose label validity — AV-name family parsing and ClarAVy type tags — is never checked, making the benchmark numbers a measure of labeling consistency rather than ground truth.","tokens_in":26429,"tokens_out":8958,"would_cite":true,"duration_ms":90476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces RawMal-TF, a public dataset pairing raw malware binaries with type and family labels, and reports 98.98% family-detection accuracy.","keywords":["malware classification","raw malware dataset","type and family labels","static PE analysis","machine learning","multiclass classification","benchmark dataset"],"falsifier":"Randomly sample, say, 200 family-labeled binaries from RawMal-TF and 200 type-labeled binaries, then have two independent analysts or an alternative automated labeler assign family and type from the binaries themselves; if agreement between the dataset's labels and the independent labels falls well below the reported accuracies, the headline numbers largely measure agreement with a noisy labeling source.","tokens_in":25477,"feed_emoji":"🦠","tokens_out":5073,"duration_ms":49938,"temperature":0.7,"pith_summary":"The paper claims that malware classification suffers from a shortage of datasets that provide both raw binaries and two levels of labels, type and family. It introduces RawMal-TF, a collection of 14 malware types and 17 malware families assembled from public repositories, with binaries shipped alongside extracted static features. Using a standard Windows PE feature pipeline, the authors show that Random Forest and XGBoost distinguish these malware from benign files at 98.5% (type) and 98.98% (family) accuracy, while multiclass type assignment reaches 81.1% with SVM. If the dataset and labels are reliable, researchers gain a reusable raw-binary resource for custom feature extraction, adversarial testing, and fine-grained malware analysis.","feed_headline":"Raw malware binaries with dual labels hit 98.98% detection","feed_subtitle":"A new open dataset pairs raw PE files with type and family labels so models can tell malware strains apart.","key_machinery":"The load-bearing object is the dual-label construction pipeline: an automated behavioral-labeling tool assigns type tags from aggregated antivirus scan data, and filename substring parsing assigns family names from archived malware filenames. On top of this sits a standardized static feature extraction pipeline for Windows Portable Executable files, drawn from an established benchmark, which turns every binary into a fixed-length vector of header, histogram, entropy, string, section, import, and export features. The pipeline is what lets the same labeled binaries support binary, interclass, and multiclass classification tasks, and what makes the raw binaries re-usable for future feature engineering.","core_discovery":"The central claim is that a raw malware dataset labeled at both behavioral type and family granularities can be built entirely from existing public sources without paid APIs, and that static PE features extracted from it support strong classification. Type labels come from precomputed behavioral tags generated by an automated antivirus-label parser applied to large unlabeled malware archives; family labels are parsed from filenames in redistributed malware archives. After filtering to categories with at least 1,000 samples, the dataset contains 14 types and 17 families. Experiments across binary, interclass, and multiclass tasks show that malware-versus-benign detection is nearly saturated, while finer-grained distinctions remain harder: family multiclass accuracy peaks near 73.4% and type multiclass at 81.1%.","pith_inferences":["If the labels are sound, RawMal-TF could serve as a testbed for adversarial perturbations of raw binaries, since the raw bytes are provided rather than only vectors.","A natural extension the paper does not perform is label-quality auditing: comparing filename-derived family labels against an independent labeling source on a sample subset.","The observed confusion patterns suggest type boundaries such as worm vs. auto-run may be intrinsically fuzzy, so future taxonomies might treat some categories as multi-label rather than exclusive."],"forward_implications":["Any researcher can download the raw binaries and re-extract features with newer techniques rather than being locked into precomputed vectors.","Type-level and family-level labels allow studying how label granularity affects detection, interclass separability, and multiclass accuracy.","Reported near-perfect binary detection suggests static PE features alone are sufficient to separate these families from benign files.","The multiclass gaps (81.1% types, 73.4% families) point to specific confusable pairs, such as worm/auto-run and Taskun/Agensla, as targets for richer features."],"supporting_citations":[{"why":"Supplies the standardized PE static feature extraction and vectorization pipeline used to convert every binary into numerical features.","marker":"[3]"},{"why":"Provides the behavior-based type labels that define the 14 malware-type classes in the dataset.","marker":"[4]"},{"why":"Supplies the raw unlabeled binaries for the type-labeled portion of the dataset.","marker":"[5]"},{"why":"Redistributes the monthly malware archives whose filenames carry the parsed family labels.","marker":"[6]"},{"why":"The original source of the family-labeled samples that the redistributed archives draw from.","marker":"[7]"}],"fun_headline_variants":["Dual labels for malware: type and family hit 98.98% detection","Malware dataset with type and family labels reaches 98.98%","Fine-grained malware classification gets 98.98% detection","New malware dataset: 14 types, 17 families, 98.98% accuracy","Type and family labels push malware detection to 98.98%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's value rests on the labels being true: family names parsed from filenames and behavioral types from automated antivirus-label parsing must actually describe what each binary does.","fun_headline_variants_meta":{"raw":{"variants":["Dual labels for malware: type and family hit 98.98% detection","Malware dataset with type and family labels reaches 98.98%","Fine-grained malware classification gets 98.98% detection","New malware dataset: 14 types, 17 families, 98.98% accuracy","Type and family labels push malware detection to 98.98%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000494,"raw_usage":{"total_tokens":2460,"prompt_tokens":1016,"completion_tokens":1444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":1346}},"tokens_in":632,"tokens_out":1444,"duration_ms":13959,"temperature":1.0,"reasoning_tokens":1346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:28:15.248169+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly sample, say, 200 family-labeled binaries from RawMal-TF and 200 type-labeled binaries, then have two independent analysts or an alternative automated labeler assign family and type from the binaries themselves; if agreement between the dataset's labels and the independent labels falls well below the reported accuracies, the headline numbers largely measure agreement with a noisy labeling source.","supporting_citations":[{"cited_title":"In: Companion of the 16th ACM/SPEC Interna- tional Conference on Performance Engineer- ing (WWW Companion ’25) (2025)","cited_arxiv_id":null,"evidence_quote":"Provides the behavior-based type labels that define the 14 malware-type classes in the dataset."},{"cited_title":"Online; cit","cited_arxiv_id":null,"evidence_quote":"Supplies the raw unlabeled binaries for the type-labeled portion of the dataset."},{"cited_title":"Online; cit","cited_arxiv_id":null,"evidence_quote":"Redistributes the monthly malware archives whose filenames carry the parsed family labels."},{"cited_title":"Online; cit","cited_arxiv_id":null,"evidence_quote":"The original source of the family-labeled samples that the redistributed archives draw from."}],"review_version":1}