REVIEW 4 major objections 5 minor 108 references
Efficient Code Analysis via Graph Representation Learning-Guided Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read The paper claims that a lightweight graph neural network can act as an attention selector for a large language model, turning whole-package malware detection into a small-subgraph judgment with 94–97 percent accuracy and orders-of-magnitude
desk verdict Fresh pipeline and dramatic token savings, but the main accuracy comparisons are contaminated by a train/test overlap that the paper never addresses. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the attention subgraph produced by per-sample mask optimization on a trained graph classifier. Starting from an AST-derived code graph with call, inheritance, decorator, and hook edges, the paper builds node features from LLM-generated sensitive-behavior rules, trains a two-layer GCN under binary supervision, then optimizes edge and feature masks so that the masked graph still yields a malicious prediction. These mask values serve as node and edge attention scores; a top-K budget converts them into a small natural-language subgraph description that the LLM analyzes. This is what turns a whole-package problem into a small-context problem.
What would settle it
Retrain the GNN after explicitly deleting from the training corpus any package that also appears in the three public benchmark datasets, then rerun the main accuracy table; if accuracy drops to roughly the direct-LLM level, the reported superiority is a training-leakage artifact. A simpler check is to compare package names across splits.
Extended reading notes
Core claim
GMLLM constructs a code graph from a Python package, with nodes for modules, classes, and functions and edges for calls, definitions, inheritance, and installation hooks. It uses an LLM to generate a set of sensitive-behavior rules, encoded as multi-hot node features, and trains a two-layer graph convolutional network with only package-level binary labels. At inference, for a package the GNN flags as malicious, the method learns per-sample edge and feature masks that maximize the probability of the malicious class; these masks become attention scores. A threshold picks the top-K scored structures, and the LLM receives only this subgraph, together with the corresponding source code, and retur
Load-bearing premise
The success on public benchmarks depends on the assumption that those benchmarks were not part of the GNN's training set; the paper assembled its own training corpus from the same public sources and never states that overlapping samples were removed.
Editorial extensions
If this is right
- LLMs can be applied to package-level malicious code detection without processing whole repositories, making large-scale scanning technically and economically feasible.
- The method returns a location and characteristic description of the malicious behavior, not just a binary verdict, which supports triage and remediation.
- Training requires only benign/malicious labels, so the detector can be retrained on newly discovered packages without expensive fine-grained annotation.
- Because token usage per large package falls from roughly 250,000 to about 640, the per-package cost of LLM analysis drops by orders of magnitude.
- The graph structure alone, without any behavior rules, already yields 88.65 percent accuracy, suggesting the representation carries signal independent of rule matching.
Reading between the lines
- If the attention masks are as reliable as reported, the same subgraph-selection idea could be applied to other long-context code tasks, such as locating the exact functions responsible for a vulnerability or tracing data flow across modules.
- The paper's token-efficiency numbers omit the roughly 10-second-per-package explainer step; a fair end-to-end cost comparison should include that offline computation, which may matter for very large scan queues.
- A direct way to test generalization is to check whether any public-benchmark packages appear in the training set, since the training corpus is assembled from the same public sources and no deduplication is stated.
- The related-work section contains unresolved placeholder citations, so some comparisons to prior work are not fully sourced.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes GMLLM, a pipeline for LLM-based malicious Python package detection. A project is parsed into an AST/call graph; an LLM generates sensitive-behavior rules; a GNN is trained with binary labels; GNNExplainer extracts high-attention subgraphs; and an LLM makes the final verdict from those subgraphs. The paper reports that GMLLM with ChatGPT-4o reaches 94.3–96.96% accuracy on three public datasets and 95.62% on the authors' MalCP dataset, well above direct GPT-4o, while reducing token usage from roughly 250,000 to around 640 tokens for large packages. It also reports improved explanation quality and a human validation of the LLM-as-judge protocol. The central claim is that the graph-guided attention mechanism lets LLMs reliably detect malicious code in large projects.
Significance. If the reported numbers were trustworthy, the contribution would be significant: it offers a concrete way to make LLM-based package scanning practical by replacing whole-package prompts with small, explainer-selected subgraphs, and it shows a large reduction in token cost. The paper also has some commendable features: the token accounting in Appendix D.3 is model-specific and includes formatting overhead; the LLM-as-judge protocol is checked against human annotators in Appendix C.5; and the ablation in Table 5 provides some insight into which components contribute. However, the main empirical claims rest on the contaminated evaluation described below. In its current form the paper does not establish that GMLLM consistently outperforms the baselines.
major comments (4)
- [Appendix D.1.1 / D.1.4, Table 1] Table 1 reports results on Backstabbers, Datadog, and Mal-OSS, the same three sources from which MalCP was constructed: Appendix D.1.1 states that MalCP was built by 'integrating multiple open-source intelligence sources, including Mal_OSS, Backstabbers Knife Collection, and Datadog's open malware dataset.' The dataset-filtering pipeline in D.1.4 has aggregation, annotation, and filtering steps, but no step for removing overlap with the later evaluation sets. D.2.2 describes an 80/20 training split but never states that the public datasets were excluded from GNN training. If MalCP is the GNN training pool, the public test sets are in-distribution, and the high-attention subgraphs in Table 1 are not independent of training. This directly affects the paper's central claim of consistent superiority. A decontamination statement and a re-evaluation on disjoint samples are required.
- [Appendix D.2.2] The GNN training protocol says: 'No validation set is used. ... Early stopping may be applied based on test F1 score.' Selecting the early-stopping epoch on the test set is a form of test-set leakage and makes the GNN's reported performance optimistic. It also means there is no valid procedure for selecting hyperparameters such as the number of epochs. A proper train/validation/test split and early stopping on validation metrics are needed before the GNN-based results can be interpreted.
- [Section 4.5.2, Table 6, Tables 1–2] The top-K edge budget is tuned on MalCP by varying K in {10, 20, 30, 50}, and then the paper states 'we fix K=20 in all main experiments,' including the MalCP results in Table 2. Because K is selected on the same dataset that is later used for evaluation, the MalCP accuracy numbers in Table 2 are an optimistic selected result, not an independent evaluation. If K is intended to be a fixed universal constant, it should be set a priori or selected on a separate validation split, and sensitivity should be reported on the actual test sets.
- [Section 4.1, Tables 1–3; D.2.2] The paper states that 'all experimental results are reported as the average across five independent trials,' but no standard deviations, confidence intervals, or per-trial results are given in Tables 1–3. Many of the claimed improvements are only a few percentage points (e.g., 95.62 vs 93.56 in Table 2), so without variance information the reader cannot judge whether the differences are meaningful. Additionally, D.2.2 says the graph classifier 'is not used for standalone detection,' but Section 4.5.1 compares a 'GNN-only model' and reports 92.22% accuracy on MalCP; this inconsistency should be resolved.
minor comments (5)
- [Eq. (2)] The symmetrization expression appears to use the raw mask M in the second term instead of the sigmoid-activated mask. The intended operation is likely (δ(M) + δ(M)^T)/2; otherwise the resulting matrix is not symmetric.
- [Section 3.2, Eq. (11)] The text sets a threshold α, but Eq. (11) uses γ_node and γ_edge. The notation should be aligned.
- [Keywords] Typo: 'lanaguage' should be 'language.'
- [Appendix D.2.4] The appendix states that the codebase and critical dependencies 'are released as part of our codebase,' but no artifact URL, repository link, or dataset hash is provided. Including a link and hashes would be necessary for reproducibility.
- [Table 5] The '50 Rules (Scomm ∪ 0.5Sdata)' configuration is described as 'randomly keeps half of the rules in Sdata at inference time'; it is unclear whether this is a single fixed random subset or averaged over multiple draws. Please clarify.
Circularity Check
Public-benchmark 'predictions' are fit on the same packages used to build MalCP; no deduplication is stated.
-
fitted input called prediction
[Appendix D.1.1, D.1.4; Table 1]
"To construct the MalCP dataset, we compiled malicious Python packages from the PyPI ecosystem by integrating multiple open-source intelligence sources, including Mal_OSS [38], Backstabbers Knife Collection [36], and Datadog’s open malware dataset [37]. ... Table 1: Performance on Backstabbers, Datadog, Mal-OSS Datasets."
The GNN that produces the attention subgraphs is trained on MalCP. By the paper's own construction, MalCP aggregates Backstabbers, Datadog, and Mal-OSS, and the filtering pipeline (D.1.4) removes samples only by 'archive structure and size constraints,' not by overlap with those public evaluation sets. Table 1 then reports GMLLM's accuracy on the very same three datasets. Any retained package from these sources that falls into the 80% training split has its test instance effectively fitted: the GNN's attention masks and the resulting LLM subgraph are derived from a model that saw that package during training. The public-dataset numbers in Table 1 are therefore not independent generalization measurements; they are, at least in part, evaluations on training data relabeled as prediction.
full rationale
The core GNN+explainer+LLM derivation is not circular: the GNN is trained with binary labels, the explainer maximizes the trained model's malicious probability (Eq. 5), and the LLM independently judges the extracted subgraph (Eq. 12). The rule set Sdata is generated from 10% of the training data and used as features, which is standard supervised feature extraction and does not by itself make the prediction equal to an input. No self-citation chain or imported uniqueness theorem is load-bearing. The significant problem is at the evaluation level: MalCP is constructed from the same public datasets that are later used as held-out test sets, with no stated deduplication. If overlapping packages entered the MalCP training split, the advertised superiority over direct LLM baselines and SOTA tools (Tables 1–2) is partially an artifact of training on the test instances. The token-efficiency result is less affected, but the headline claim rests on the contaminated accuracy comparison.
Assumptions & free parameters
free parameters (5)
- K (top-K edge budget) =
20
- λ_size, λ_ent (mask regularizer weights) =
not specified
- Sdata sampling fraction =
10%
- MalCP size thresholds =
5 KB / 10 KB
- GNNExplainer optimization budget =
100 gradient steps per package
assumptions (6)
- domain assumption GNNExplainer masks identify the code regions necessary for the LLM to judge maliciousness
- domain assumption Static AST + call/dependency graphs capture cross-file malicious behavior
- domain assumption LLM-generated sensitive-behavior rules Scomm ∪ Sdata cover the malicious feature space
- domain assumption The final LLM can reach a correct verdict from a truncated subgraph of at most 20 edges
- domain assumption Public dataset labels and Snyk advisories are correct ground truth
- domain assumption A 2-layer GCN trained with an 80/20 split and no validation set generalizes across package sizes
Cite this review
Pith. "Pith review of Efficient Code Analysis via Graph Representation Learning-Guided Large Language Models." pith.science (2026). https://pith.science/paper/NDWA3NHH
@misc{pith2026260112890,
author = {Pith},
title = {Pith review of: Efficient Code Analysis via Graph Representation Learning-Guided Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/NDWA3NHH}},
note = {Machine review of arXiv:2601.12890}
}
read the original abstract
Large Language Models (LLMs) have significantly advanced code analysis tasks, yet they struggle to detect malicious behaviors fragmented across files, whose intricate dependencies easily get lost in the vast amount of benign code. We therefore propose a graph-centric attention acquisition pipeline that enhances LLMs' ability to localize malicious behavior. The approach parses a project into a code graph, uses an LLM to encode nodes with semantic and structural signals, and trains a Graph Neural Network (GNN) under sparse supervision. The GNN performs an initial detection, and by interpreting these predictions, identifies key code sections that are most likely to contain malicious behavior. These influential regions are then used to guide the LLM's attention for in-depth analysis. This strategy significantly reduces interference from irrelevant context while maintaining low annotation costs. Extensive experiments show that the method consistently outperforms existing approaches on multiple public and custom datasets, highlighting its potential for practical deployment in software security scenarios.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Hellendoorn, Bogdan Vasilescu, and Brad A
Daye Nam, Andrew Macvean, Vincent J. Hellendoorn, Bogdan Vasilescu, and Brad A. Myers. Using an LLM to help with code understanding. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pages 97:1–97:13. ACM, 2024
2024
-
[2]
Butt, Mohammad Taha Khan, and Muhammad Ali Gulzar
Sabaat Haroon, Ahmad Faraz Khan, Ahmad Humayun, Waris Gill, Abdul Haddi Amjad, Ali R. Butt, Mohammad Taha Khan, and Muhammad Ali Gulzar. How accurately do large language models understand code?CoRR, abs/2504.04372, 2025
arXiv 2025
-
[3]
Mutation-based consistency testing for evaluating the code understanding capability of llms
Ziyu Li and Donghwan Shin. Mutation-based consistency testing for evaluating the code understanding capability of llms. In Jane Cleland-Huang, Jan Bosch, Henry Muccini, and Grace A. Lewis, editors,Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, CAIN 2024, Lisbon, Portugal, April 14-15, 2024, pages ...
2024
-
[4]
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. Large language models for software engineering: Survey and open problems. InIEEE/ACM International Conference on Software Engineering: Future of Software Engineering, ICSE-FoSE 2023, Melbourne, Australia, May 14-20, 2023, pages 31–53. IEEE, 2023
2023
-
[5]
Yoichi Ishibashi and Yoshimasa Nishimura. Self-organized agents: A LLM multi-agent framework toward ultra large-scale code generation and optimization.CoRR, abs/2404.02183, 2024
arXiv 2024
-
[6]
Larry Huynh, Yinghao Zhang, Djimon Jayasundera, Woojin Jeon, Hyoungshick Kim, Tingting Bi, and Jin B. Hong. Detecting code vulnerabilities using llms. In55th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2025, Naples, Italy, June 23-26, 2025, pages 401–414. IEEE, 2025
2025
-
[7]
Malpacdetector: An llm-based malicious NPM package detector.IEEE Trans
Jian Wang, Zhen Li, Jixiang Qu, Deqing Zou, Shouhuai Xu, Ziteng Xu, Zhenwei Wang, and Hai Jin. Malpacdetector: An llm-based malicious NPM package detector.IEEE Trans. Inf. Forensics Secur., 20:6279–6291, 2025. 10 Running Title for Header
2025
-
[8]
Vulnerability detection and monitoring using llm
Vishwanath Akuthota, Raghunandan Kasula, Sabiha Tasnim Sumona, Masud Mohiuddin, Md Tanzim Reza, and Md Mizanur Rahman. Vulnerability detection and monitoring using llm. In2023 IEEE 9th International Women in Engineering (WIE) Conference on Electrical and Computer Engineering (WIECON-ECE), pages 309–314, 2023
2023
Show all 108 references
-
[9]
Poster: An exploration of large language models in malicious source code detection
Di Xue, Gang Zhao, Zhongqi Fan, Wei Li, Yahong Xu, Zhen Liu, Yin Liu, and Zhongliang Yuan. Poster: An exploration of large language models in malicious source code detection. In Bo Luo, Xiaojing Liao, Jun Xu, Engin Kirda, and David Lie, editors,Proceedings of the 2024 on ACM S...
2024
-
[10]
Raas: Reasoning-aware attention sparsity for efficient LLM reasoning
Junhao Hu, Wenrui Huang, Weidong Wang, Zhenwen Li, Tiancheng Hu, Zhixia Liu, Xusheng Chen, Tao Xie, and Yizhou Shan. Raas: Reasoning-aware attention sparsity for efficient LLM reasoning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Fi...
2025
-
[11]
Joty, and Steven C
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021...
2021
-
[12]
Codegen: An open large language model for code with multi-turn program synthesis
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali...
2023
-
[13]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...
2021 arXiv
-
[14]
Code llama: Open foundation models for code.CoRR, abs/2308.12950, 2023
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défoss...
2023 arXiv
-
[15]
Oss detect backdoor
Microsoft. Oss detect backdoor. https://github.com/microsoft/OSSGadget/wiki/OSS-Detect-Backdoor , 2019. Accessed: April 5, 2025
2019
-
[16]
A survey on common threats in npm and pypi registries.CoRR, abs/2108.09576, 2021
Berkay Kaplan and Jingyu Qian. A survey on common threats in npm and pypi registries.CoRR, abs/2108.09576, 2021
2021 arXiv
-
[17]
Yara, 2023
VirusTotal. Yara, 2023. Retrieved Jun 12, 2023
2023
-
[18]
Bandit, 2023
PyCQA. Bandit, 2023. Retrieved Mar 10, 2023 fromhttps://github.com/PyCQA/bandit
2023
-
[19]
Towards measuring supply chain attacks on package managers for interpreted languages
Ruian Duan, Omar Alrawi, Ranjita Pai Kasturi, Ryan Elder, Brendan Saltaformaggio, and Wenke Lee. Towards measuring supply chain attacks on package managers for interpreted languages. In28th Annual Network and Distributed System Security Symposium, NDSS 2021, virtually, Februar...
2021
-
[20]
Pooja Yadav, Neeraj Menon, Vinayakumar Ravi, Sowmya Vishvanathan, and Tuan D. Pham. Efficientnet convolutional neural networks-based android malware detection.Comput. Secur., 115:102622, 2022
2022
-
[21]
Amddlmodel: Android smartphones malware detection using deep learning model.Plos one, 19(1):e0296722, 2024
Muhammad Aamir, Muhammad Waseem Iqbal, Mariam Nosheen, M Usman Ashraf, Ahmad Shaf, Khalid Ali Almarhabi, Ahmed Mohammed Alghamdi, and Adel A Bahaddad. Amddlmodel: Android smartphones malware detection using deep learning model.Plos one, 19(1):e0296722, 2024
2024
-
[22]
Deepcatra: Learning flow- and graph-based behaviours for android malware detection.IET Inf
Yafei Wu, Jian Shi, Peicheng Wang, Dongrui Zeng, and Cong Sun. Deepcatra: Learning flow- and graph-based behaviours for android malware detection.IET Inf. Secur., 17(1):118–130, 2023
2023
-
[23]
Iot-based android malware detection using graph neural network with adversarial defense.IEEE Internet Things J., 10(10):8432–8444, 2023
Rahul Yumlembam, Biju Issac, Seibu Mary Jacob, and Longzhi Yang. Iot-based android malware detection using graph neural network with adversarial defense.IEEE Internet Things J., 10(10):8432–8444, 2023
2023
-
[24]
Android malware detection through generative adversarial networks.Trans
Muhammad Amin, Babar Shah, Aizaz Sharif, Tamleek Ali, Ki-Il Kim, and Sajid Anwar. Android malware detection through generative adversarial networks.Trans. Emerg. Telecommun. Technol., 33(2), 2022
2022
-
[25]
Schorlemmer, Santiago Torres-Arias, and James C
Chinenye Okafor, Taylor R. Schorlemmer, Santiago Torres-Arias, and James C. Davis. Sok: Analysis of software supply chain security by establishing secure design properties. In Santiago Torres-Arias, Marcela S. Melara, and Laurent Simon, editors,Proceedings of the 2022 ACM Work...
2022
-
[26]
Decoding the secrets of machine learning in malware classification: A deep dive into datasets, feature extraction, and model performance
Savino Dambra, Yufei Han, Simone Aonzo, Platon Kotzias, Antonino Vitale, Juan Caballero, Davide Balzarotti, and Leyla Bilge. Decoding the secrets of machine learning in malware classification: A deep dive into datasets, feature extraction, and model performance. In Weizhi Meng...
2023
-
[27]
A needle is an outlier in a haystack: Hunting malicious pypi packages with code clustering
Wentao Liang, Xiang Ling, Jingzheng Wu, Tianyue Luo, and Yanjun Wu. A needle is an outlier in a haystack: Hunting malicious pypi packages with code clustering. In38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 20...
2023
-
[28]
1+1>2: Integrating deep code behaviors with metadata features for malicious pypi package detection
Xiaobing Sun, Xingan Gao, Sicong Cao, Lili Bo, Xiaoxue Wu, and Kaifeng Huang. 1+1>2: Integrating deep code behaviors with metadata features for malicious pypi package detection. In Vladimir Filkov, Baishakhi Ray, and Minghui Zhou, editors, Proceedings of the 39th IEEE/ACM Inte...
2024
-
[29]
Dos and don’ts of machine learning in computer security
Daniel Arp, Erwin Quiring, Feargus Pendlebury, Alexander Warnecke, Fabio Pierazzi, Christian Wressnegger, Lorenzo Cavallaro, and Konrad Rieck. Dos and don’ts of machine learning in computer security. In31st USENIX Security Symposium (USENIX Security 22), pages 3971–3988, Bosto...
2022
-
[30]
Bad snakes: Understanding and improving python package index malware scanning
Duc-Ly Vu, Zachary Newman, and John Speed Meyers. Bad snakes: Understanding and improving python package index malware scanning. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 499–511, 2023
2023
-
[31]
Williams
Nusrat Zahan, Philipp Burckhardt, Mikola Lysenko, Feross Aboukhadijeh, and Laurie A. Williams. Leveraging large language models to detect NPM malicious packages. In47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, ...
2025
-
[32]
Large language models for code analysis: Do LLMs really do their job? In33rd USENIX Security Symposium (USENIX Security 24), pages 829–846, Philadelphia, PA, August 2024
Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Asmita, Ryan Tsang, Najmeh Nazari, Han Wang, and Houman Homayoun. Large language models for code analysis: Do LLMs really do their job? In33rd USENIX Security Symposium (USENIX Security 24), pa...
2024
-
[33]
Maltracker: A fine-grained npm malware tracker copiloted by llm-enhanced dataset
Zeliang Yu, Ming Wen, Xiaochen Guo, and Hai Jin. Maltracker: A fine-grained npm malware tracker copiloted by llm-enhanced dataset. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, page 1759–1771, New York, NY , USA, 20...
2024
-
[34]
Leveraging large language models for behavior-based malware detection using deep learning
Tosin Akinsowon and Haodi Jiang. Leveraging large language models for behavior-based malware detection using deep learning. Technical report, Institute for Homeland Security, Sam Houston State University, 2024
2024
-
[35]
Al-Karaki, Muhammad Al-Zafar Khan, and Marwan Omar
Jamal N. Al-Karaki, Muhammad Al-Zafar Khan, and Marwan Omar. Exploring llms for malware detection: Review, framework design, and countermeasure approaches.CoRR, abs/2409.07587, 2024
2024 arXiv
-
[36]
Backstabber’s knife collection: A review of open source software supply chain attacks
Marc Ohm, Henrik Plate, Arnold Sykosch, and Michael Meier. Backstabber’s knife collection: A review of open source software supply chain attacks. InInternational Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, pages 23–43. Springer, 2020
2020
-
[37]
Malicious software packages dataset, 2023
Guarddog. Malicious software packages dataset, 2023. Accessed: May 15, 2025
2023
-
[38]
An empirical study of malicious code in pypi ecosystem
Wenbo Guo, Zhengzi Xu, Chengwei Liu, Cheng Huang, Yong Fang, and Yang Liu. An empirical study of malicious code in pypi ecosystem. In38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023, pages 166–177. IEEE, 2023
2023
-
[39]
Qwen2.5 technical report.CoRR, abs/2412.15115, 2024
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...
2024 arXiv
-
[40]
Llama 2: Open foundation and fine-tuned chat models.CoRR, abs/2307.09288, 2023
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...
2023 arXiv
-
[41]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...
2024 arXiv
-
[42]
A comprehensive capability analysis of GPT-3 and GPT-3.5 series models.CoRR, abs/2303.10420, 2023
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. A comprehensive capability analysis of GPT-3 and GPT-3.5 series models.CoRR, abs/2303.10420, 2023
2023 arXiv
-
[43]
Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Re...
2024 arXiv
-
[44]
A fork of bandit tool with patterns to identifying malicious python code.Retrieved September, 25:2024, 2020
Duc-Ly Vu. A fork of bandit tool with patterns to identifying malicious python code.Retrieved September, 25:2024, 2020
2024
-
[45]
VirusTotal — Upload.https://www.virustotal.com/gui/home/upload/
VirusTotal. VirusTotal — Upload.https://www.virustotal.com/gui/home/upload/. Accessed: September 5, 2025
2025
-
[46]
Beyond typosquatting: An in-depth look at package confusion
Shradha Neupane, Grant Holmes, Elizabeth Wyss, Drew Davidson, and Lorenzo De Carli. Beyond typosquatting: An in-depth look at package confusion. In Joseph A. Calandrino and Carmela Troncoso, editors,32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA, August...
2023
-
[47]
Vaidya, Drew Davidson, Lorenzo De Carli, and Vaibhav Rastogi
Matthew Taylor, Ruturaj K. Vaidya, Drew Davidson, Lorenzo De Carli, and Vaibhav Rastogi. Defending against package typosquatting. In Miroslaw Kutylowski, Jun Zhang, and Chao Chen, editors,Network and System Security - 14th International Conference, NSS 2020, Melbourne, VIC, Au...
2020
-
[48]
Typosquatting and combosquatting attacks on the python ecosystem
Duc-Ly Vu, Ivan Pashchenko, Fabio Massacci, Henrik Plate, and Antonino Sabetta. Typosquatting and combosquatting attacks on the python ecosystem. InIEEE European Symposium on Security and Privacy Workshops, EuroS&P Workshops 2020, Genoa, Italy, September 7-11, 2020, pages 509–...
2020
-
[49]
OSS gadget, 2023
Microsoft. OSS gadget, 2023. Retrieved Aug 10, 2023 fromhttps://github.com/microsoft/OSSGadget
2023
-
[50]
OSSF, 2023
OSSF. OSSF, 2023. Retrieved Sep 27, 2023 fromhttps://github.com/ossf/package-analysis
2023
-
[51]
DONAPI: malicious NPM packages detector using behavior sequence knowledge mapping
Cheng Huang, Nannan Wang, Ziyan Wang, Siqi Sun, Lingzi Li, Junren Chen, Qianchong Zhao, Jiaxuan Han, Zhen Yang, and Lei Shi. DONAPI: malicious NPM packages detector using behavior sequence knowledge mapping. In Davide Balzarotti and Wenyuan Xu, editors,33rd USENIX Security Sym...
2024
-
[52]
Malcertain: Enhancing deep neural network based android malware detection by tackling prediction uncertainty
Haodong Li, Guosheng Xu, Liu Wang, Xusheng Xiao, Xiapu Luo, Guoai Xu, and Haoyu Wang. Malcertain: Enhancing deep neural network based android malware detection by tackling prediction uncertainty. InProceedings of the 46th IEEE/ACM International Conference on Software Engineeri...
2024
-
[53]
Gobbi and Johannes Kinder
Matías F. Gobbi and Johannes Kinder. Poster: Using codeql to detect malware in npm. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS ’23, page 3519–3521, New York, NY , USA, 2023. Association for Computing Machinery
2023
-
[54]
EaTVul: ChatGPT-based evasion attack against software vulnerability detection
Shigang Liu, Di Cao, Junae Kim, Tamas Abraham, Paul Montague, Seyit Camtepe, Jun Zhang, and Yang Xiang. EaTVul: ChatGPT-based evasion attack against software vulnerability detection. In33rd USENIX Security Symposium (USENIX Security 24), pages 7357–7374, Philadelphia, PA, Augu...
2024
-
[55]
Code vulnerability detection based on augmented program dependency graph and optimized codebert.Scientific Reports, 15(1):39301, 2025
Zhengbin Zou, Tao Jiang, Yizheng Wang, Tiancheng Xue, Nan Zhang, and Jie Luan. Code vulnerability detection based on augmented program dependency graph and optimized codebert.Scientific Reports, 15(1):39301, 2025
2025
-
[56]
Vulcobert: A codebert-based system for source code vulnerability detection
Yuying Xia, Haijian Shao, and Xing Deng. Vulcobert: A codebert-based system for source code vulnerability detection. In Proceedings of the 2024 International Conference on Generative Artificial Intelligence and Information Security, GAIIS ’24, page 249–252, New York, NY , USA,...
2024
-
[57]
Leveraging an enhanced codebert-based model for multiclass software defect prediction via defect classification.IEEE Access, 13:24383–24397, 2025
Rida Ghafoor Hussain, Kin Choong Yow, and Marco Gori. Leveraging an enhanced codebert-based model for multiclass software defect prediction via defect classification.IEEE Access, 13:24383–24397, 2025
2025
-
[58]
Vuld-codebert: Codebert-based vulnerability detection model for c/c++ code
Zhe Xiong and Weiyu Dong. Vuld-codebert: Codebert-based vulnerability detection model for c/c++ code. In2024 6th International Conference on Communications, Information System and Computer Engineering (CISCE), pages 914–919, 2024. 13 Running Title for Header
2024
-
[59]
Codebert: A pre-trained model for programming and natural languages
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages. In Trevor Cohn, Yulan He, and Yang Liu, editors,Findings of the Associatio...
2020
-
[60]
Python source code vulnerability detection based on codebert language model
Kunpeng Zhao, Shuya Duan, Ge Qiu, Jinyuan Zhai, Mingze Li, and Long Liu. Python source code vulnerability detection based on codebert language model. In2024 7th International Conference on Algorithms, Computing and Artificial Intelligence (ACAI), pages 1–6, 2024
2024
-
[61]
Hammerschmidt, Carlos Hernandez Gañán, and Sicco Verwer
Azqa Nadeem, Christian A. Hammerschmidt, Carlos Hernandez Gañán, and Sicco Verwer. Malpaca: Malware packet sequence clustering and analysis.CoRR, abs/1904.01371, 2019
1904 arXiv
-
[62]
Open source vulnerability database.https://security.snyk.io/vuln, 2024
Snyk. Open source vulnerability database.https://security.snyk.io/vuln, 2024. Accessed: May 14, 2025
2024
-
[63]
Mitre att&ck: Design and philosophy
Blake E Strom, Andy Applebaum, Doug P Miller, Kathryn C Nickels, Adam G Pennington, and Cody B Thomas. Mitre att&ck: Design and philosophy. InTechnical report. The MITRE Corporation, 2018. 14 Running Title for Header A Usage of Large Language Model In our paper, we used LLMs t...
2018
-
[64]
network": lambda n: n.startswith((
emphasized the importance of dataset selection and evaluation in applying machine learning to security. Vu et al. [30] conduct study that reveal repository administrators require extremely low false positive rates (below 0.1%). They found that a socio-technical malicious code ...
-
[65]
could be malicious
Threat Tactic Generalization (1–5) How well does it generalize from specific functions or code elements to a recognized cybersecurity tactic (e.g., ‘Reconnais- sance’, ‘Defense Evasion’, ‘Exfiltration’)? →5: Clearly identifies and frames the analysis using a standard attack pa...
-
[66]
→4: Shows a mostly coherent flow but omits minor steps or data dependencies
Execution Path Traceability (1–5) How clearly does it reconstruct a step-by-step execution flow, including function calls, data movement, and control logic? →5: Presents a complete, logical, and verifiable sequence of actions. →4: Shows a mostly coherent flow but omits minor s...
-
[67]
→4: Most claims are evidence-backed; minor inferences are reasonable
Evidence Groundedness (1–5) Are claims directly and tightly supported by specific code elements (e.g., function calls, strings, variables)? →5: Every significant claim is explicitly tied to observable code. →4: Most claims are evidence-backed; minor inferences are reasonable. ...
-
[68]
quality_scores
Factual Alignment (1-5) Does it correctly identify and accurately describe the primary malicious behavior? →5: Accurate and complete—captures the core behavior and key details. →3: Partially correct—identifies the general type of malicious activity but misses or misrepresents ...
-
[69]
Aggregation:Malicious packages were consolidated from the PyPI ecosystem using multiple open-source intelligence sources
-
[70]
Behavioral Annotation:Each package was enriched with a structured behavior summary derived from the Snyk advisory system
-
[71]
Small/Medium/Large
Filtering:Samples were filtered based on archive structure and size constraints to ensure semantic richness and practical utility. D.2 Settings This section outlines the end-to-end pipeline used in our experiments, including graph preparation, model training, explanation gener...
-
[72]
Password Decryption: The use of ‘ctypes.windll.crypt32.CryptUnprotectData’ indicates an attempt to decrypt sensitive data, which is commonly used to scrape encrypted data like saved passwords
-
[73]
Clipboard Hijacking: The use of ‘pyperclip.copy’ in combination with ‘auto_copy_wallet.address_swap’ suggests the possibility of clipboard manipulation, potentially to replace wallet addresses for cryptocurrency theft
-
[74]
Credential Harvesting: Calls like ‘os.getlogin’, ‘winreg.OpenKey’, and ‘winreg.QueryValueEx’ indicate that the package may be gathering system and user information illegitimately
-
[75]
Screen and Cookie Stealing: Nodes like ‘PIL.ImageGrab.grab’ (screenshot capture) and ‘steal_cookies’ strongly imply the stealing of private user data
-
[76]
Unauthorized Memory Operations: Calls to ‘ctypes.cdll.msvcrt.memcpy’ and ‘ctypes.windll.kernel32.LocalFree’ raise concerns about memory manipulation, which is often associated with malicious payload deployment or data harvesting
-
[77]
Network Activity: The usage of ‘urllib.request.Request’ and ‘urllib.request.urlopen’ under the function ‘LoadUrlib’ may imply exfiltration of data or receiving commands from a remote server
-
[78]
Potential Anti-Debugging: The function ‘NoDebugg.sp3cCheq’ suggests the package may contain anti-debugging checks to evade detection by security tools or analysts
-
[79]
, “mitigation
Execution and Persistence: The presence of ‘subprocess.Popen’, combined with suspicious function names like ‘ZipThings’ and system-manipulative calls like ‘os._exit’, hints at unauthorized script execution or persistence mechanisms. These behaviors, when taken together, are hi...
-
[80]
‘ctypes.windll.crypt32.CryptUnprotectData’: Validate any use of sensitive data decryption and ensure it is necessary and authorized
-
[81]
‘pyperclip.copy’: Restrict clipboard access to prevent modification of user data
-
[82]
‘PIL.ImageGrab.grab’ and ‘steal_cookies’: Explicitly validate and limit access to screen capture and browser storage to prevent data theft
-
[83]
‘ctypes.cdll.msvcrt.memcpy’ and ‘ctypes.windll.kernel32.LocalFree’: Review all direct memory manipulation for potential abuse
-
[84]
‘urllib.request.Request’ and ‘urllib.request.urlopen’: Monitor outgoing network calls to prevent unauthorized data exfiltration or command-and-control communication
-
[85]
‘os.getlogin’, ‘winreg.OpenKey’, and ‘winreg.QueryValueEx’: Validate and restrict access to user information and registry keys
-
[86]
‘NoDebugg’ Functions: Investigate all anti-debugging mechanisms and ensure they do not attempt to evade legitimate security analyses
-
[87]
name": “pipcoloradds
‘subprocess.Popen’: Confirm that subprocess spawning is securely configured and does not execute unauthorized commands or scripts. These mitigations are critical to address the highly suspicious nature of the subgraph behavior." } 25 Running Title for Header 1.00 0.00 0.50 1.0...
-
[88]
‘ctypes.windll.crypt32.CryptUnprotectData’ – This is used to decrypt credentials or sensitive data; ensure it cannot access unauthorized data
-
[89]
‘httpx.post’ and ‘requests.get’ – These are used for exfiltration; validate and restrict network calls
-
[90]
‘PIL.ImageGrab.grab’ – This is used for screen capture; restrict screen capture permissions
-
[91]
‘pyperclip.copy’ – Prevent unauthorized clipboard manipulation
-
[92]
‘exec’ – Deny or heavily sandbox dynamic code execution
-
[93]
} 26 Running Title for Header Output Example 3 {“name
‘fernet.Fernet’ – Ensure encryption routines are not used maliciously." } 26 Running Title for Header Output Example 3 {“name": “pythonsqliteaddition", “verdict": “Malicious", “reasoning": “The subgraph structure exhibits multiple indicators of potentially malicious activity:
-
[94]
Credential and Data Theft Indicators : - Calls to ‘ctypes.windll.crypt32.CryptUnprotectData’ are commonly associated with decrypting stored credentials (e.g., browser passwords). - Functions named ‘steal_cookies’, ‘steal_passwords’, and ‘steal_screen’, combined with calls to ‘...
-
[95]
Use of ctypes for Memory Operations: - The ‘ctypes.cdll.msvcrt.memcpy’ function call could manipulate memory directly, adding another layer of potential obfuscation or tampering
-
[96]
- Use of ‘os.system’ and ‘winreg.QueryValueEx’ could be employed for modifying the system environment or registry keys to establish persistence
Persistence and Evasion: - The presence of functions such as ‘NoDebugg’ and ‘NoDebugg.regkey’ indicates anti-debugging and anti-analysis techniques, which are common for malicious software. - Use of ‘os.system’ and ‘winreg.QueryValueEx’ could be employed for modifying the syst...
-
[97]
System Profiling: - Multiple calls to ‘psutil’ functions such as ‘psutil.cpu_count’, ‘psutil.virtual_memory’, and ‘psutil.process_iter’ suggest the script profiles the victim’s system, which may be used for tailoring attacks or exfiltration
-
[98]
Potential Ransomware Components: - The use of ‘bc_create_file’ and ‘ZipThings’, as well as ‘subprocess.Popen’ for zipping or creating files, combined with the functions above, can be indicative of exfiltration or encryption of user data
-
[99]
, “mitigation
Exfiltration and Execution: - URLs are accessed via ‘urllib.request.Request’ and ‘urllib.request.urlopen’, which could be used to send stolen data or download additional malicious components. - The ‘finished_bc’ and ‘bc_exit_this’ calls ending with ‘os._exit’ may indicate the ...
-
[100]
‘ctypes.windll.crypt32.CryptUnprotectData’: Limit access to sensitive decryption functions by restricting permissions
-
[101]
‘PIL.ImageGrab.grab’: Disable unauthorized screenshot functionality
-
[102]
Functions related to clipboard modification (‘pyperclip.copy’): Restrict clipboard operations
-
[103]
System commands and registry operations (‘os.system’, ‘winreg.QueryValueEx’): Implement strict permission checks and monitor usage
-
[104]
Network-related calls (‘urllib.request.Request’, ‘urllib.request.urlopen’): Ensure the script isn’t communicating with malicious entities and validate permitted endpoints
-
[105]
} Output example 4 {“name
‘os._exit’: Audit its usage in combination with other suspicious nodes, such as file or data manipulation. The subgraph strongly indicates malicious intent, and immediate analysis of the full package and its behavior during runtime in an isolated environment is recommended." }...
-
[106]
‘numpy.random.multivariate_normal’ and ‘numpy.random.binomial’: Validate inputs to random data generation functions to ensure parameters are appropriately bounded to prevent system misuse or instability
-
[107]
‘distutils.core.setup’: Confirm that the setup script does not introduce unexpected installation behaviors or dependencies
-
[108]
‘causal.sumlessthan’, ‘calc_att_se’, ‘log1exp’: Ensure robust checks for mathematical operations to prevent improper assertions or computation errors during causal inference calculations." } 28
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.