pith. sign in

arxiv: 2605.25937 · v1 · pith:TQUCXI35new · submitted 2026-05-25 · 💻 cs.CR · cs.LG

Building an Adversarial Malware Dataset by Family and Type: Generation, Evasion, and Poisoning Evaluation

David Ko\v{s}\v{t}\'al , Martin Jure\v{c}ek This is my paper

Pith reviewed 2026-06-29 21:21 UTC · model grok-4.3

classification 💻 cs.CR cs.LG
keywords adversarial malwaredata poisoningmalware detectionPE binariesmachine learning robustnessevasion attacksdataset construction
0
0 comments X

The pith

Injecting 0.5 percent mislabelled adversarial malware samples raises classifier evasion from 26 percent to 93 percent.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The authors build two large collections of adversarial PE malware binaries from real-world samples, one set labelled by malware family and the other by type. These collections evade a standard EMBER detector at rates of 98 percent and 92 percent respectively. Retraining experiments then show that adding only 0.5 percent fully mislabelled adversarial samples to the family-labelled training data drives the evasion rate on the new classifier from 26.1 percent to 92.8 percent. The dataset is released to enable further study of poisoning and robustness in machine-learning malware detectors.

Core claim

The paper establishes that suites of adversarial generators can produce tens of thousands of functional adversarial PE binaries that preserve their original family and type labels while evading detection, and that these samples can be used to poison training data at a scale of only 0.5 percent to increase evasion rates against retrained classifiers from 26.1 percent to 92.8 percent.

What carries the argument

Adversarial malware generators that modify PE binaries to produce evasive yet labelled samples, combined with the poisoning attack that injects fully mislabelled instances into the training set.

If this is right

  • Malware family classifiers become highly vulnerable once a small fraction of mislabelled adversarial binaries enters the training data.
  • Type-labelled datasets may exhibit different poisoning sensitivity than family-labelled ones.
  • Public release of the adversarial samples allows systematic testing of detection robustness and poisoning defenses.
  • Even high-evasion generators can serve as effective poisoning vectors when their labels are flipped.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training pipelines for malware detectors will need automated checks for label consistency on any generated or third-party samples.
  • Poisoning risks shown here may extend to other security domains that rely on labelled binary or network data.
  • Future work could measure how quickly the evasion boost decays if the poisoned samples are later removed or relabelled.

Load-bearing premise

The generated adversarial samples remain valid, functional PE binaries whose family and type labels stay accurate after modification.

What would settle it

Retrain the EMBER-style classifier on the family-labelled dataset after injecting 0.5 percent mislabelled adversarial samples and check whether the evasion rate on a held-out test set reaches 92.8 percent.

Figures

Figures reproduced from arXiv: 2605.25937 by David Ko\v{s}\v{t}\'al, Martin Jure\v{c}ek.

Figure 7
Figure 7. Figure 7: Box plot of EMBER score drops by generator for the type-labelled dataset. Note that generators using EMBER as their target never make the score worse, while others might. AMG’s result here is slightly anomalous, most likely explained by its oversized samples causing bad classifications. Most generators for the type-labelled dataset find it harder to evade top antivirus products, compared to the family-labe… view at source ↗
read the original abstract

We present a dataset of adversarial malware samples derived from the public RawMal-TF collection of real-world malware binaries. Using a suite of adversarial malware generators, we construct two sets of adversarial PE files: 44,347 family-labelled samples and 33,596 type-labelled samples, achieving evasion rates of 98.35 % and 92.20 % against the EMBER classifier, respectively. Each adversarial binary is accompanied by detailed metadata, including EMBER scores and VirusTotal classifications. We further demonstrate the susceptibility of malware classification pipelines to data poisoning attacks through a series of training experiments. Injecting fully mislabelled adversarial samples representing only 0.5 % of the training data in the family-labelled dataset increases the evasion rate against the re-trained classifier from 26.1 % to 92.8 %. The dataset is publicly released to facilitate future research on adversarial malware, poisoning attacks, and the robustness of machine-learning-based malware detection systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper constructs two adversarial malware datasets (44,347 family-labelled and 33,596 type-labelled samples) from the public RawMal-TF collection of real-world PE binaries using a suite of generators. It reports evasion rates of 98.35% and 92.20% against the EMBER classifier, supplies metadata including EMBER scores and VirusTotal classifications, demonstrates a poisoning attack in which 0.5% fully mislabelled adversarial samples raise evasion from 26.1% to 92.8% on the family-labelled dataset, and publicly releases the dataset.

Significance. The public release of the dataset is a clear strength that would enable follow-on work on adversarial robustness and poisoning in malware detection. If the generated binaries are confirmed to remain functional and label-preserving, the poisoning result would supply concrete empirical evidence of classifier susceptibility to small-scale label-flip attacks and would strengthen the case for improved data hygiene in ML-based malware pipelines.

major comments (2)
  1. [Abstract] Abstract (poisoning paragraph): the claim that injecting 0.5% mislabelled adversarial samples raises evasion from 26.1% to 92.8% presupposes that the injected samples are valid, executable PE binaries whose original family labels remain accurate and known after generation. No post-generation validation (PE header checks, dynamic execution, behavioral equivalence, or label-consistency tests via VirusTotal/EMBER) is described, rendering the numerical result uninterpretable if the generators alter functionality or detectable signatures.
  2. [Abstract] Abstract (evasion-rate sentences): the reported evasion rates of 98.35% (family) and 92.20% (type) likewise depend on the unverified premise that the adversarial modifications preserve the family/type labels used both for training the target classifier and for evaluating evasion. Without explicit checks that the post-modification binaries retain their original labels, the evasion percentages cannot be taken as evidence about real malware classifiers.
minor comments (1)
  1. [Abstract] The abstract refers to 'a suite of adversarial malware generators' without naming them or citing the underlying methods; adding this information would improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We appreciate the referee's thorough review and valuable feedback on our manuscript. We address each major comment below and outline the revisions we plan to make.

read point-by-point responses

  1. Referee: [Abstract] Abstract (poisoning paragraph): the claim that injecting 0.5% mislabelled adversarial samples raises evasion from 26.1% to 92.8% presupposes that the injected samples are valid, executable PE binaries whose original family labels remain accurate and known after generation. No post-generation validation (PE header checks, dynamic execution, behavioral equivalence, or label-consistency tests via VirusTotal/EMBER) is described, rendering the numerical result uninterpretable if the generators alter functionality or detectable signatures.

    Authors: We thank the referee for this observation. The suite of generators used are drawn from prior work on adversarial malware generation, where they have been shown to produce executable binaries that maintain their malicious functionality and original labels. The dataset release includes comprehensive metadata with VirusTotal classifications and EMBER scores for each sample, which can be used to perform label-consistency checks. In the revised manuscript, we will add explicit discussion of the generators' properties and how the metadata supports validation of the samples. revision: partial

  2. Referee: [Abstract] Abstract (evasion-rate sentences): the reported evasion rates of 98.35% (family) and 92.20% (type) likewise depend on the unverified premise that the adversarial modifications preserve the family/type labels used both for training the target classifier and for evaluating evasion. Without explicit checks that the post-modification binaries retain their original labels, the evasion percentages cannot be taken as evidence about real malware classifiers.

    Authors: We agree that label preservation is critical for interpreting the evasion rates. As noted above, the generators are intended to preserve labels, and the provided metadata allows for verification. We will revise the manuscript to include a dedicated subsection on sample validation and to temper the claims in the abstract by referencing the generator assumptions and available metadata. revision: partial

Circularity Check

0 steps flagged

No circularity: purely empirical measurements with no derivations or self-referential fits

full rationale

The paper constructs an adversarial malware dataset via generators applied to RawMal-TF binaries and reports direct empirical outcomes: evasion rates (98.35%, 92.20%), poisoning effects (0.5% injection raising evasion from 26.1% to 92.8%), and metadata. No equations, fitted parameters, uniqueness theorems, or ansatzes appear. All reported quantities are measured results from the experiments described; the central claims do not reduce to inputs by construction. Self-citations are absent from load-bearing positions. This is a standard empirical dataset paper whose validity rests on experimental verification rather than any circular derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

No free parameters, invented entities, or non-standard axioms are visible in the abstract; the work rests on the domain assumption that RawMal-TF binaries are suitable unmodified starting material.

axioms (1)
  • domain assumption RawMal-TF collection consists of real-world malware binaries suitable as base material for adversarial generation.
    Stated as the source collection in the abstract.

pith-pipeline@v0.9.1-grok · 5706 in / 1257 out tokens · 38561 ms · 2026-06-29T21:21:31.878501+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 17 canonical work pages · 4 internal anchors

  1. [1]

    Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning

    Anderson, H.S., Kharkar, A., Filar, B., Evans, D., Roth, P.: Learning to evade static PE machine learning malware models via reinforcement learning (2018). DOI 10.48550/arXiv.1801.08917

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1801.08917 2018

  2. [2]

    EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models

    Anderson, H.S., Roth, P.: EMBER: An open dataset for training static PE malware machine learning models (2018). DOI 10.48550/arXiv.1804.04637

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1804.04637 2018

  3. [3]

    URLhttps://www.av-test.org/en /antivirus/home-windows

    AV-TEST Institute: Test antivirus software for Windows 11 – February 2026 (2026). URLhttps://www.av-test.org/en /antivirus/home-windows

    2026

  4. [4]

    URLhttps://github.com/egebalci/sgn

    Balcı, E.: SGN: Shikata ga nai encoder. URLhttps://github.com/egebalci/sgn

  5. [5]

    DOI10.48550/arXiv .2506.23909

    Bálik,D.,Jureček,M.,Stamp,M.:RawMal-TF:Rawmalwaredatasetlabeledbytypeandfamily(2025). DOI10.48550/arXiv .2506.23909

    work page arXiv 2025

  6. [6]

    DOI 10.48550/arXiv.2104.12848

    Demetrio,L.,Biggio,B.:secml-malware:APythonlibraryforadversarialrobustnessevaluationofWindowsmalwareclassifiers (2021). DOI 10.48550/arXiv.2104.12848

    work page doi:10.48550/arxiv.2104.12848 2021

  7. [7]

    IEEE Trans

    Demetrio, L., Biggio, B., Lagorio, G., Roli, F., Armando, A.: Functionality-preserving black-box optimization of adversarial Windows malware. IEEE Trans. Inf. Forensics Secur.16, 3469–3478 (2021). DOI 10.1109/tifs.2021.3082330

    work page doi:10.1109/tifs.2021.3082330 2021

  8. [8]

    In:S.N.Foley, D

    Grosse,K.,Papernot,N.,Manoharan,P.,Backes,M.,McDaniel,P.:Adversarialexamplesformalwaredetection. In:S.N.Foley, D. Gollmann, E. Snekkenes (eds.) Computer Security – ESORICS 2017, pp. 62–79. Springer International Publishing, Cham (2017). DOI 10.1007/978-3-319-66399-9_4

    work page doi:10.1007/978-3-319-66399-9_4 2017

  9. [9]

    In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24

    He, S., Fu, C., Hu, H., Chen, J., Lv, J., Jiang, S.: MalwareTotal: Multi-faceted and sequence-aware bypass tactics against static malware detection. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24. Association for Computing Machinery, New York, NY, USA (2024). DOI 10.1145/3597503.3639141

    work page doi:10.1145/3597503.3639141 2024

  10. [10]

    ClarA Vy: A Tool for Scalable and Accurate Malware Family Labeling,

    Joyce, R.J., Everett, D., Fuchs, M., Raff, E., Holt, J.: ClarAVy: A tool for scalable and accurate malware family labeling. In: CompanionProceedingsoftheACMonWebConference2025,WWW’25,p.277–286.AssociationforComputingMachinery, New York, NY, USA (2025). DOI 10.1145/3701716.3715212

    work page doi:10.1145/3701716.3715212 2025

  11. [11]

    EMBER2024 - A Benchmark Dataset for Holistic Evaluation of Malware Classifiers,

    Joyce, R.J., Miller, G., Roth, P., Zak, R., Zaresky-Williams, E., Anderson, H., Raff, E., Holt, J.: EMBER2024 - a benchmark dataset for holistic evaluation of malware classifiers. In: Proceedings of the 31st ACM SIGKDD Conference on Knowledge DiscoveryandDataMiningV.2,KDD’25,p.5516–5526.AssociationforComputingMachinery,NewYork,NY,USA(2025). DOI 10.1145/37...

    work page doi:10.1145/3711896.3737431 2025

  12. [12]

    Kozák, M., Jureček, M., Stamp, M., Troia, F.D.: Creating valid adversarial examples of malware. J. Comput. Virol. Hacking Tech.20(4), 607–621 (2024). DOI 10.1007/s11416-024-00516-2

    work page doi:10.1007/s11416-024-00516-2 2024

  13. [13]

    Louthánová, P., Kozák, M., Jureček, M., Stamp, M., Di Troia, F.: A comparison of adversarial malware generators. J. Comput. Virol. Hacking Tech.20(4), 623–639 (2024). DOI 10.1007/s11416-024-00519-z

    work page doi:10.1007/s11416-024-00519-z 2024

  14. [14]

    In:Twenty-ThirdAnnualComputerSecurity Applications Conference (ACSAC 2007), pp

    Moser,A.,Kruegel,C.,Kirda,E.:Limitsofstaticanalysisformalwaredetection. In:Twenty-ThirdAnnualComputerSecurity Applications Conference (ACSAC 2007), pp. 421–430 (2007). DOI 10.1109/ACSAC.2007.21

    work page doi:10.1109/acsac.2007.21 2007

  15. [15]

    Malware Detection by Eating a Whole EXE

    Raff, E., Barker, J., Sylvester, J., Brandon, R., Catanzaro, B., Nicholas, C.: Malware detection by eating a whole EXE (2017). DOI 10.48550/arXiv.1710.09435

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1710.09435 2017

  16. [16]

    Proximal Policy Optimization Algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms (2017). DOI 10.48550/arXiv.1707.06347

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1707.06347 2017

  17. [17]

    In: Proceedings 2001 IEEE Symposium on Security and Privacy, p

    Schultz, M.G., Eskin, E., Zadok, E., Stolfo, S.J.: Data Mining Methods for Detection of New Malicious Executables. In: Proceedings 2001 IEEE Symposium on Security and Privacy, p. 0038. IEEE Computer Society, Los Alamitos, CA, USA (2001). DOI 10.1109/SECPRI.2001.924286

    work page doi:10.1109/secpri.2001.924286 2001

  18. [18]

    URLhttps://web.archive.org/web/20240314053756/https://iwantmore.pizza/po sts/PEzor.html

    Soncina, F.: PEzor (2020). URLhttps://web.archive.org/web/20240314053756/https://iwantmore.pizza/po sts/PEzor.html

    work page arXiv 2020

  19. [19]

    URLhttps://github.com/phra/PEzor

    Soncina, F.: PEzor (2023). URLhttps://github.com/phra/PEzor

    2023

  20. [20]

    In: Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security (2022)

    Song,W.,Li,X.,Afroz,S.,Garg,D.,Kuznetsov,D.,Yin,H.:MAB-Malware:Areinforcementlearningframeworkforattacking static malware classifiers. In: Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security (2022). DOI 10.1145/3488932.3497768

    work page doi:10.1145/3488932.3497768 2022

  21. [21]

    URL https://farama.org/Announcing-The-Farama-Foundation

    The Farama Foundation: Announcing the Farama Foundation: The future of open source reinforcement learning (2022). URL https://farama.org/Announcing-The-Farama-Foundation

    2022

  22. [22]

    URL https://github.com/CyberForce/Pesidious

    Vaya,C.,Sen,B.:Pesidious:Malwaremutationusingreinforcementlearningandgenerativeadversarialnetworks(2020). URL https://github.com/CyberForce/Pesidious. Accessed: 2026-01-26

    2020

  23. [23]

    URLhttps://www.virustotal.com

    VirusTotal: VirusTotal – online malware analysis service (2026). URLhttps://www.virustotal.com. Accessed: 2026-03-09

    2026