REVIEW 5 major objections 8 minor 31 references
RawMal-TF: Raw Malware Dataset Labeled by Type and Family
T0 review · 5 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper introduces RawMal-TF, a public dataset pairing raw malware binaries with type and family labels, and reports 98.98% family-detection accuracy.
desk verdict A real, useful raw-PE dataset whose label validity — AV-name family parsing and ClarAVy type tags — is never checked, making the benchmark numbers a measure of labeling consistency rather than ground truth. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the dual-label construction pipeline: an automated behavioral-labeling tool assigns type tags from aggregated antivirus scan data, and filename substring parsing assigns family names from archived malware filenames. On top of this sits a standardized static feature extraction pipeline for Windows Portable Executable files, drawn from an established benchmark, which turns every binary into a fixed-length vector of header, histogram, entropy, string, section, import, and export features. The pipeline is what lets the same labeled binaries support binary, interclass, and multiclass classification tasks, and what makes the raw binaries re-usable for future feature engineering.
What would settle it
Randomly sample, say, 200 family-labeled binaries from RawMal-TF and 200 type-labeled binaries, then have two independent analysts or an alternative automated labeler assign family and type from the binaries themselves; if agreement between the dataset's labels and the independent labels falls well below the reported accuracies, the headline numbers largely measure agreement with a noisy labeling source.
Extended reading notes
Core claim
The central claim is that a raw malware dataset labeled at both behavioral type and family granularities can be built entirely from existing public sources without paid APIs, and that static PE features extracted from it support strong classification. Type labels come from precomputed behavioral tags generated by an automated antivirus-label parser applied to large unlabeled malware archives; family labels are parsed from filenames in redistributed malware archives. After filtering to categories with at least 1,000 samples, the dataset contains 14 types and 17 families. Experiments across binary, interclass, and multiclass tasks show that malware-versus-benign detection is nearly saturated, while finer-grained distinctions remain harder: family multiclass accuracy peaks near 73.4% and type multiclass at 81.1%.
Load-bearing premise
The dataset's value rests on the labels being true: family names parsed from filenames and behavioral types from automated antivirus-label parsing must actually describe what each binary does.
Editorial extensions
If this is right
- Any researcher can download the raw binaries and re-extract features with newer techniques rather than being locked into precomputed vectors.
- Type-level and family-level labels allow studying how label granularity affects detection, interclass separability, and multiclass accuracy.
- Reported near-perfect binary detection suggests static PE features alone are sufficient to separate these families from benign files.
- The multiclass gaps (81.1% types, 73.4% families) point to specific confusable pairs, such as worm/auto-run and Taskun/Agensla, as targets for richer features.
Reading between the lines
- If the labels are sound, RawMal-TF could serve as a testbed for adversarial perturbations of raw binaries, since the raw bytes are provided rather than only vectors.
- A natural extension the paper does not perform is label-quality auditing: comparing filename-derived family labels against an independent labeling source on a sample subset.
- The observed confusion patterns suggest type boundaries such as worm vs. auto-run may be intrinsically fuzzy, so future taxonomies might treat some categories as multi-label rather than exclusive.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RawMal-TF, a dataset of Windows PE malware binaries labeled at two granularities: 14 behavioral types derived from ClarAVy BEH tags on VirusShare samples, and 17 families parsed from filenames in VX Underground's MalwareBazaar archives. The authors extract EMBER static features from the raw binaries, release both binaries and feature vectors, and evaluate five classifiers (KNN, MLP, SVM, XGBoost, Random Forest) on malware-vs-benign detection, pairwise interclass discrimination, and multiclass type/family classification. Reported results include 98.5% and 98.98% average accuracy for full-scope type- and family-based binary detection, 97.5% and 93.7% average pairwise accuracy for inter-type and inter-family tasks, and 81.1% (SVM) and 73.4% (RF/XGBoost) multiclass accuracy for types and families, respectively.
Significance. If the labels are trustworthy, RawMal-TF addresses a genuine gap: most public malware benchmarks provide only feature vectors, whereas this work releases raw binaries alongside dual-level type/family labels and EMBER-compatible features. The public GitHub release, the transparent description of the feature-processing pipeline, and the detailed per-class performance tables are useful contributions. However, the central value of the dataset depends on label correctness, and the paper does not independently validate either the type labels or the family labels. The label-generation pipeline contains an unspecified multi-tag assignment rule, inconsistent category-count arithmetic, and a temporal mismatch between 2018 benign samples and 2023-2025 malware. These issues mean the reported accuracies should currently be interpreted as agreement with the labeling procedure rather than as validation against ground truth. If the labeling concerns are resolved, the resource would be a solid community asset; in the present form, the core dataset claim needs additional support.
major comments (5)
- [Section 3.3.1, Listing 1] The paper does not specify how samples carrying multiple BEH tags are assigned to the single-label type categories used in the experiments. The example record contains both BEH:adware and BEH:downloader, each supported by six engines; if such samples are placed in multiple category directories, the class directories overlap and the multiclass experiments in Section 4.3 are invalid because one sample can appear in more than one class. Please state the assignment rule (e.g., first tag, majority vote, or exclusion) and report how many of the retained samples had multiple BEH tags.
- [Section 3.3.2] Family labels are obtained by parsing antivirus detection names from filenames such as 'UDS-Backdoor.Win32.Bladabindi.dej', with no independent verification. Antivirus naming conventions vary by vendor, and one family can appear under multiple names (njRAT and Bladabindi refer to the same family), while some names are generic or vendor-specific. The exclusion of generic names is described, but no check is provided that the remaining parsed names correspond to the intended families. Please add a validation step on a random subset using an independent source (e.g., MalwareBazaar tags, YARA rules, or a second labeling tool) and report agreement rates; without this, the family-level accuracies measure consistency with the parsing rule rather than correctness of the family labels.
- [Section 3.3.1, Table 1] The category-count arithmetic is internally inconsistent. The text reports 76 unique behavioral categories, 56 categories excluded for having fewer than 1,000 samples, and 17 categories with no samples in the selected archives, yet 14 categories are retained; 76 - 56 - 17 = 3, and 76 - 56 = 20, leaving six categories unaccounted for. In addition, Table 1 includes BEH adware with 967 samples, below the stated 1,000-sample threshold. Please reconcile these numbers and specify which categories were retained at each filtering step so that Table 1 can be reproduced exactly.
- [Section 4.1.1 Step 3 and Section 4.2] The benign samples come from EMBER 2018, whereas the malware samples were collected from 2023-2025 (Section 3.3). This temporal mismatch is a possible confound in the malware-vs-benign experiments: models may separate malware from benign files based on compiler toolchains, packers, or PE-layout changes over time rather than on malicious behavior. Please report a time-matched control (e.g., benign PE files from the same period) or analyze the extent of the distribution shift, and temper the corresponding accuracy claims accordingly.
- [Section 4.3, Tables 15 and 18] The high-level multiclass accuracy claims should be qualified by the per-class results. Type-based multiclass accuracy of 81.1% is accompanied by per-class recall of 49% for worm, 51% for virus, and 57% for downloader; family-based multiclass accuracy of 73.4% is accompanied by recall of 35% for Noon and 41% for Agensla. These low-recall classes are central to malware taxonomy, so the abstract's characterization of the results as enabling 'more fine-grained and insightful malware classification' goes beyond what the reported per-class metrics support. Please report macro-averaged results prominently and discuss the class-dependent difficulty explicitly.
minor comments (8)
- [Section 1, last paragraph] The text says 'Section 1 introduces the necessary background,' but the background appears in Section 2; the cross-reference should be corrected.
- [Throughout] The name ClarAVy is typeset inconsistently as 'ClarA Vy' in the Abstract and Section 3.3.1; the official spelling should be used uniformly.
- [Listing 1] The meaning of the integer after the hash (e.g., '53', '54') is not explained in the text; clarify whether it is the total number of antivirus detections for that sample.
- [Table 18] The columns 'Avg FP' and 'Avg FN' are labeled ambiguously; they appear to be counts rather than rates, but the text does not state this, and Table 15 uses FPR/FNR percentages. The units should be unified or explicitly defined.
- [Section 3.3.2] The exact family-name extraction rule (substring matching vs. regular expression, and which token of the filename is taken) is not specified; providing the parser pseudocode or a regex pattern would make the dataset construction reproducible.
- [Section 4.1.1 Step 3] The benign samples are EMBER 2018 feature vectors, not raw benign binaries; the dataset description should state this limitation explicitly, since the abstract claims that raw binaries are included alongside features and labels.
- [Section 4.2.2, Table 8] The arrow notation in misclassification rates (e.g., 'autorun↔ worm 20% (autorun), 44.5% (worm)') is ambiguous about which class is the true class; clarify the direction for each reported pair.
- [Section 4.3.1] The text states the type-based multiclass dataset contains approximately 14,000 samples, but Table 1 lists BEH adware with only 967 samples; the exact total should be stated.
Circularity Check
No significant circularity: the dataset's labels are external AV-derived metadata, and the feature extraction and classification pipeline is independent of the labeling procedure, so the accuracy results are not forced by construction.
full rationale
The paper's central claims are that RawMal-TF is a new dataset with raw binaries, family labels parsed from filenames, type labels from ClarAVy, and that classifiers reach the reported accuracies. Checking the derivation chain: (1) family labels come from parsing externally supplied antivirus filenames such as 'UDS-Backdoor.Win32.Bladabindi.dej' (Section 3.3.2), and type labels are taken from precomputed ClarAVy BEH: tags (Section 3.3.1, Listing 1); neither label is defined in terms of the extracted EMBER features or the models' outputs, so the classification accuracy is not circular. (2) The features are extracted independently from the binaries via EMBER's PEFeatureExtractor() after the labels are fixed, and the models are trained on those features, so no fitted parameter is renamed as a prediction. (3) The paper has no self-citations and invokes no uniqueness theorem; the only reuse of another dataset is benign samples drawn from EMBER 2018 with verified labels, which is external and legitimate support rather than circularity. (4) The models are evaluated on held-out test splits generated after random shuffling, not on the labeling source. The real weaknesses are external validity issues, not circularity: family labels inherit AV-vendor naming conventions, ClarAVy behavioral tags may be noisy, and the paper does not specify how multi-tag records such as 'BEH:adware|6,BEH:downloader|6' are collapsed into the single type labels used in the multiclass experiments (Section 3.3.1). Those concerns affect whether the labels mean what the paper claims, but they do not make the derivations assume their own conclusions. Accordingly, no circular step meeting the evidentiary bar is present, and the score reflects only the minor label-source validation caveat.
Assumptions & free parameters
free parameters (3)
- minimum category size threshold =
1,000 samples
- truncated dataset size =
1,000 malware + 1,000 benign
- feature selection k (binary tasks) =
chosen among {5,10,20,30,40,50,100}
assumptions (4)
- domain assumption PE filenames in VX Underground/MalwareBazaar exports encode correct malware family names.
- domain assumption ClarAVy behavioral tags from AV scan reports are accurate enough to serve as type ground truth.
- domain assumption EMBER 2018 benign samples are representative of benign Windows binaries in 2023-2025.
- domain assumption The EMBER static feature extractor produces features that are sufficient and consistent for these binaries.
Cite this review
Pith. "Pith review of RawMal-TF: Raw Malware Dataset Labeled by Type and Family." pith.science (2026). https://pith.science/paper/JFK22SOL
@misc{pith2026250623909,
author = {Pith},
title = {Pith review of: RawMal-TF: Raw Malware Dataset Labeled by Type and Family},
year = {2026},
howpublished = {\url{https://pith.science/paper/JFK22SOL}},
note = {Machine review of arXiv:2506.23909}
}
read the original abstract
This work addresses the challenge of malware classification using machine learning by developing a novel dataset labeled at both the malware type and family levels. Raw binaries were collected from sources such as VirusShare, VX Underground, and MalwareBazaar, and subsequently labeled with family information parsed from binary names and type-level labels integrated from ClarAVy. The dataset includes 14 malware types and 17 malware families, and was processed using a unified feature extraction pipeline based on static analysis, particularly extracting features from Portable Executable headers, to support advanced classification tasks. The evaluation was focused on three key classification tasks. In the binary classification of malware versus benign samples, Random Forest and XGBoost achieved high accuracy on the full datasets, reaching 98.5% for type-based detection and 98.98% for family-based detection. When using truncated datasets of 1,000 samples to assess performance under limited data conditions, both models still performed strongly, achieving 97.6% for type-based detection and 98.66% for family-based detection. For interclass classification, which distinguishes between malware types or families, the models reached up to 97.5% accuracy on type-level tasks and up to 93.7% on family-level tasks. In the multiclass classification setting, which assigns samples to the correct type or family, SVM achieved 81.1% accuracy on type labels, while Random Forest and XGBoost reached approximately 73.4% on family labels. The results highlight practical trade-offs between accuracy and computational cost, and demonstrate that labeling at both the type and family levels enables more fine-grained and insightful malware classification. The work establishes a robust foundation for future research on advanced malware detection and classification.
Reference graph
Works this paper leans on
-
[1]
Computers & Security 81, 123– 147 (2019) https://doi.org/10.1016/j.cose
Ucci, D., Aniello, L., Baldoni, R.: Survey of machine learning techniques for malware analysis. Computers & Security 81, 123– 147 (2019) https://doi.org/10.1016/j.cose. 2018.11.001
doi:10.1016/j.cose 2019
-
[2]
Machine Learning, Deep Learning and AI for Cyber- security, 113–142 (2025)
Dˇ ed, L., Jureˇ cek, M.: Selecting representa- tive samples from malware datasets. Machine Learning, Deep Learning and AI for Cyber- security, 113–142 (2025)
work page 2025
-
[3]
ArXiv e-prints (2018) arXiv:1804.04637 [cs.CR]
Anderson, H.S., Roth, P.: EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models. ArXiv e-prints (2018) arXiv:1804.04637 [cs.CR]
arXiv 2018
-
[4]
Joyce, R.J., Everett, D., Fuchs, M., Raff, E., Holt, J.: Claravy: A tool for scalable and accurate malware family labeling. In: Companion of the 16th ACM/SPEC Interna- tional Conference on Performance Engineer- ing (WWW Companion ’25) (2025)
work page 2025
-
[5]
VirusShare: VirusShare.com. Online; cit. 2025-02-28. https://virusshare.com/
work page 2025
-
[6]
VX Underground: VX Underground. Online; cit. 2025-03-18. https://vx-underground. org/
work page 2025
-
[7]
Abuse.ch: MalwareBazaar. Online; cit. 2025- 03-02. https://bazaar.abuse.ch/browse/ 30
work page 2025
- [8]
Show all 31 references
-
[9]
Nist interagency/internal report 7621 rev
Paulsen, C., Toth, P.: Small business infor- mation security: The fundamentals. Nist interagency/internal report 7621 rev. 1, National Institute of Standards and Tech- nology, Gaithersburg, MD (November 2016). https://doi.org/10.6028/NIST.IR.7621r1
2016 doi
-
[10]
2025-03-15
Belcic, I.: What Is Malware and How to Protect Against Malware Attacks? Online; cit. 2025-03-15. https://www.avast.com/c-malware
2025
- [11]
-
[12]
Online; cit
Lanier, K.: Worm Malware Foren- sic Analysis. Online; cit. 2025-02-25. https://westoahu.hawaii.edu/cyber/ forensics-weekly-executive-summmaries/ worm-malware-forensic-analysis
2025
-
[13]
Online; cit
Malwarebytes: Downloader. Online; cit. 2025-03-10. https://www.malwarebytes. com/glossary/downloader
2025
-
[14]
Technical Report NIST Special Publication 800-83 Revision 1, National Institute of Standards and Technol- ogy, Gaithersburg, MD (July 2013)
Souppaya, M., Scarfone, K.: Guide to mal- ware incident prevention and handling for desktops and laptops. Technical Report NIST Special Publication 800-83 Revision 1, National Institute of Standards and Technol- ogy, Gaithersburg, MD (July 2013). https: //doi.org/10.6028/NIST....
2013 doi
-
[15]
Technical Report Draft NISTIR 8374, National Institute of Standards and Technol- ogy, Gaithersburg, MD (September 2021)
Barker, W.C., Scarfone, K., Fisher, W., Souppaya, M.: Cybersecurity framework profile for ransomware risk management. Technical Report Draft NISTIR 8374, National Institute of Standards and Technol- ogy, Gaithersburg, MD (September 2021). https://doi.org/10.6028/NIST.IR.8374-d...
2021 doi
-
[16]
Online; cit
National Institute of Standards and Tech- nology: Spyware. Online; cit. 2025- 03-10. https://csrc.nist.gov/glossary/term/ spyware
2025
-
[17]
Online; cit
Kaspersky: What Is Adware? – Def- inition and Explanation. Online; cit. 2025-04-05. https://www.kaspersky.com/ resource-center/threats/adware
2025
- [18]
-
[19]
Online; cit
Trend Micro: Exploring Emotet’s Activ- ities. Online; cit. 2025-04-12 (n.d.). https://documents.trendmicro.com/assets/ white papers/ExploringEmotetsActivities Final.pdf
2025
-
[20]
2025-04-15
Baker, K.: What Is TrickBot Malware? Online; cit. 2025-04-15. https://www. crowdstrike.com/en-us/cybersecurity-101/ malware/trickbots/
2025
-
[21]
Origi- nally published 2007; updated 2011; Online; cit
Microsoft: Win32/Allaple. Origi- nally published 2007; updated 2011; Online; cit. 2025-04-18. https://www. microsoft.com/en-us/wdsi/threats/ malware-encyclopedia-description?Name= Win32/Allaple&threatId=
2007
-
[22]
Online; cit
Hunt.io: Emotet. Online; cit. 2025-04-20. https://hunt.io/malware-families/emotet
2025
-
[23]
Online; cit
KnowBe4: Virlock Ransomware. Online; cit. 2025-04-22. https://www.knowbe4.com/ ransomware-knowledgebase/virlock
2025
-
[24]
Origi- nally published 2007; updated 2017; Online; cit
Microsoft: Win32/Virut. Origi- nally published 2007; updated 2017; Online; cit. 2025-04-18. https://www. microsoft.com/en-us/wdsi/threats/ malware-encyclopedia-description?Name= Win32/Virut
2007
-
[25]
https://www.virustotal.com/
-
[26]
Online; cit
Hunt.io: njRAT. Online; cit. 2025-04-27. https://hunt.io/malware-families/njrat
2025
-
[27]
Online; 31 cit
F-Secure: Trojan-Spy:W32/Zbot. Online; 31 cit. 2025-04-29. https://www.f-secure.com/ v-descs/trojan-spy-w32-zbot.shtml
2025
-
[28]
In: Research in Attacks, Intru- sions, and Defenses: 19th International Sym- posium, RAID 2016, Paris, France, Septem- ber 19-21, 2016, Proceedings 19, pp
Sebasti´ an, M., Rivera, R., Kotzias, P., Caballero, J.: Avclass: A tool for massive mal- ware labeling. In: Research in Attacks, Intru- sions, and Defenses: 19th International Sym- posium, RAID 2016, Paris, France, Septem- ber 19-21, 2016, Proceedings 19, pp. 230–253 (2016). Springer
2016
-
[29]
Avail- able: https://github.com/lief-project/LIEF
LIEF: [Online; accessed May 22, 2025]. Avail- able: https://github.com/lief-project/LIEF
2025
-
[30]
Online; cit
scikit-learn developers: SelectKBest. Online; cit. 2025-04-01. https://scikit-learn. org/stable/modules/generated/sklearn. feature selection.SelectKBest.html
2025
-
[31]
Online; cit
scikit-learn developers: Support Vector Machines (SVM). Online; cit. 2025-04-14. https://scikit-learn.org/stable/modules/ svm.html 32
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.