REVIEW 4 major objections 5 minor 1 cited by
Enhancing Android Malware Detection with Retrieval-Augmented Generation
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Grounded LLM descriptions of Android app features reach 96.69% recall in malware classification, beating a fused-LLM baseline on accuracy and F1.
desk verdict Useful internal comparison of RAG-based vs fusion-based app descriptions, but the advertised accuracy gain over conventional feature-based detection is never tested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is AgenticRAG, named in the paper as an agentic retrieval-augmented generation pipeline. It combines a curated corpus of Android API documentation covering permissions, services, receivers, and intent actions; an ensemble retriever pairing dense vector search with sparse keyword search fused by reciprocal rank fusion; exact-then-fuzzy matching with a similarity threshold for unseen or misspelled feature names; a cache memory for reused lookups; and a fallback LLM query for unmatched features. The retrieved context is inserted into a structured prompt that forbids speculation and demands one cohesive paragraph describing what the app can do, and that paragraph is tokenized and classified by domain-adapted BERT models. The machinery turns sparse manifest features into dense semantic text that a transformer classifier can separate.
What would settle it
Train the same cybersecurity-tuned BERT classifier on raw static feature strings (permissions, services, receivers, and intents serialized as text) and on the AgenticRAG descriptions using identical train and test splits. If the raw-feature model matches or exceeds 92.89% accuracy or 96.69% recall, the central claim that grounded descriptions improve over conventional feature-based representations is refuted. A separate audit of the labels would also settle it: if a random sample of one-engine-flagged apps is mostly adware or greyware, the reported recall is measured against a noisy target.
Extended reading notes
Core claim
The paper's central claim is that retrieval-grounded LLM descriptions are a better text representation for Android malware classification than descriptions produced by fusing two independent LLMs. In the AgenticRAG pipeline, each APK's static features are normalized, matched against a documentation corpus by dense and sparse retrieval, and synthesized by an LLM under a strict prompt into a single factual paragraph. When these paragraphs are classified by CySecBERT, a BERT model adapted to cybersecurity text, the paper reports 92.89% accuracy, 96.69% recall, and a 92.86% F1-score, while the Gemini Fusion descriptions reach 91.36% accuracy, 90.50% recall, and a 91.25% F1-score. CySecBERT also beats SecBERT, another security-oriented BERT variant, on recall and F1 for both description types, although SecBERT's plain accuracy on AgenticRAG descriptions is slightly higher at 93.31%. The paper attributes the gain to contextual specificity, better handling of unusual permission combinations, semantic coherence, and malware-specific terminology in grounded descriptions.
Load-bearing premise
The whole experiment relies on labels from a multi-engine scanning API: an app flagged by at least one engine counts as malicious and an app flagged by none counts as benign, with no independent validation of those labels.
Editorial extensions
If this is right
- Replacing raw static features with grounded functional descriptions could let small BERT-size models serve as fast, deployable malware classifiers that never execute an app.
- If the reported recall of 96.69% holds, the approach would miss only about 3 in 100 malicious apps, which matters more in security settings than small accuracy differences.
- Improving the retrieval corpus, fuzzy matching, or fallback generation should directly shift the classification ceiling, because description quality is the identified bottleneck.
- The same grounded-description pipeline should transfer to other platforms or feature sets such as API calls and network permissions, which the paper lists as future work.
Reading between the lines
- The abstract claims improvement over conventional feature-based methods, but the experiments compare only description-generation strategies; a reader should treat that broader claim as untested until a raw-static-feature baseline is added.
- The one-engine-flag labeling rule likely classes some adware or greyware as malware, so re-labeling with a stricter multi-engine agreement threshold would test whether the ranking of description methods is stable under cleaner labels.
- An ablation with plain LLM descriptions and no retrieval would isolate how much of the reported gain comes from RAG grounding versus prompt design or the LLM itself.
- Because the descriptions are generated without executing apps, the pipeline could be chained with dynamic analysis to catch dormant or evasive malware, an extension the paper names as future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an Android malware detection pipeline that converts static APK features (permissions, services, receivers, intent actions) into natural-language functional descriptions using a custom AgenticRAG system built on Gemini 2.0 Flash Lite, and then classifies these descriptions with domain-adapted BERT models (CySecBERT and SecBERT). For comparison, the authors generate a second set of descriptions by fusing LLaMA2 and Mistral outputs with Gemini, and they report that AgenticRAG descriptions plus CySecBERT achieve 92.89% accuracy, 96.69% recall, and 92.86% F1-score, outperforming the Gemini fusion pipeline. The abstract and contributions claim this approach improves detection accuracy over conventional feature-based methods.
Significance. If properly supported, the idea of using retrieval-augmented, LLM-generated functional descriptions as an intermediate representation for transformer-based malware classification would be a useful contribution. The paper has concrete strengths: the description-generation architecture is described in detail, the prompts are provided, two description-generation strategies are compared under identical preprocessing and classification conditions, and two domain-specific BERT models are evaluated with confusion matrices. However, the advertised central claim—improvement over conventional feature-based malware detection—is not tested by any experiment in the manuscript, and the ground-truth labels rest on a single unvalidated VirusTotal threshold. As reported, the results only show that one LLM-generated text representation beats another LLM-generated text representation with the same BERT classifier family.
major comments (4)
- [§4.3, Tables 5–8] The central claim in the abstract and Section 1—that the method improves detection accuracy over conventional feature-based methods—is unsupported because no conventional feature-based baseline is evaluated. Tables 5–8 compare only two LLM description-generation pipelines and two BERT classifiers; there is no experiment training, for example, a Drebin-style classifier on permissions/intents/API calls with SVM or Random Forest, or any other standard static-feature baseline on the same corpus. Without such a baseline, the reported 92.89% accuracy and 96.69% recall do not demonstrate any gain over conventional features, and the paper's headline contribution remains untested.
- [§3.1] The VirusTotal labeling protocol uses a single threshold—one or more detecting engines means malicious, zero engines means benign—with no validation against a trusted malware or benign set. This is known to be noisy: many benign apps are flagged by one engine and many malware samples evade all engines. Because every reported metric depends on these labels, the accuracy, recall, and F1 values in Tables 5–8 are only as meaningful as this unvalidated ground truth. The authors should provide distribution of detection counts, or cross-validate with a curated dataset, or otherwise justify the threshold.
- [§4.2, §5] The paper reports single-run results without error bars, confidence intervals, or significance tests, yet the conclusion states that the method yields "statistically significant improvements in all the considered evaluation metrics." No statistical claim of this kind can be made from the presented data. The authors should run multiple trials with different seeds, report standard deviations, and apply appropriate significance tests, or soften the claim accordingly.
- [§4.3.3] The post hoc explanation attributing AgenticRAG's superiority to "enhanced contextual specificity," "improved edge case handling," "semantic coherence," and "malware-specific terminology" is not supported by controlled experiments. There is no ablation that removes the retrieval component while keeping the same Gemini generator, no variation of the prompt without retrieval, and no analysis isolating the agentic planning behavior. Without such ablations, the causal claim that retrieval-augmented generation (rather than prompt differences or model stochasticity) drives the observed improvement is unsupported.
minor comments (5)
- [§4.3.1] The text states "CySecBERT with a recall rate of 96.89%" but Table 5 and Table 7 report 96.69%; these numbers should be reconciled.
- [Figures 8 and 9] The caption for Figure 8 labels subfigure (b) as "Gemini Fusion Descriptions Classified using SecBert," which appears to be a copy-paste error; it should likely refer to Mistral fusion descriptions. The captions for Figure 8 and Figure 9 should be checked against their content.
- [References [20] and [21]] Reference [21] for SecBERT describes a privacy-preserving neural network inference system, which does not match the HuggingFace SecBERT model used in the paper; the citation should be corrected or clarified.
- [§3.3.1, Table 8] The paper uses inconsistent naming for the open-source models: "LLaMA2" in the text and "Llama" or "Llama2" in tables and captions. Please standardize model names throughout.
- [General] The manuscript does not mention any plan to release code, prompts, or the generated description dataset. Providing an artifact would substantially improve reproducibility, especially given the complexity of the AgenticRAG pipeline.
Circularity Check
No circular derivation: the pipeline is supervised on external VirusTotal labels and the compared descriptions are independent transforms of static features; the missing conventional baseline is an evidential gap, not circularity.
full rationale
Walking the paper's derivation chain, each stage receives its input from an independent source. Labels are assigned via VirusTotal multi-engine detections (Section 3.1), static features are extracted from AndroZoo APKs with Androguard (Section 3.2), descriptions are generated from those static features by AgenticRAG or Gemini fusion (Section 3.3), and CySecBERT/SecBERT are fine-tuned on the descriptions to predict the external labels (Section 3.5). No parameter is fitted to a subset of the target metric and then reported as a prediction: the 70:10:20 train/validation/test split with class weights is standard supervised learning on an independent ground-truth source. No load-bearing step invokes a self-citation; references [20] and [21] are external model artifacts, and no uniqueness theorem or ansatz is imported from the authors' prior work. The abstract's claim of "improving detection accuracy over conventional feature-based methods" and the contribution claiming the system "outperforms state-of-the-art baselines" are not supported by the experiments in Tables 5-8, which compare only LLM-generated description pipelines; however, an absent conventional-feature baseline is a missing-experiment/correctness gap, not a circular reduction of the result to its inputs. The post hoc attributions in Section 4.3.3 are interpretive hypotheses, not derivations from the data. No quoted equation, fitted value, or definition in the paper is equivalent by construction to the claimed outcome, so the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- VirusTotal malicious threshold =
>=1 engine flags malicious
- Fuzzy matching similarity threshold =
65% (example)
- Train/validation/test split ratio =
70:10:20
- Class weights and weighted sampling parameters =
not specified
- RRF ensemble weights for FAISS and BM25 =
not specified
assumptions (5)
- domain assumption VirusTotal detection scores are a valid ground truth for Android malware.
- domain assumption Manifest-level static features (permissions, services, receivers, intents) are sufficient to characterize malicious behavior.
- domain assumption LLM-generated functional descriptions faithfully preserve the static features without hallucination.
- domain assumption The Android developer documentation corpus is accurate and complete for describing Android features.
- domain assumption Differences in classification performance are caused by the description generation strategy, not by prompt or pipeline artifacts.
Cite this review
Pith. "Pith review of Enhancing Android Malware Detection with Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/OQBO64OK
@misc{pith2026250622750,
author = {Pith},
title = {Pith review of: Enhancing Android Malware Detection with Retrieval-Augmented Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/OQBO64OK}},
note = {Machine review of arXiv:2506.22750}
}
read the original abstract
The widespread use of Android applications has made them a prime target for cyberattacks, significantly increasing the risk of malware that threatens user privacy, security, and device functionality. Effective malware detection is thus critical, with static analysis, dynamic analysis, and Machine Learning being widely used approaches. In this work, we focus on a Machine Learning-based method utilizing static features. We first compiled a dataset of benign and malicious APKs and performed static analysis to extract features such as code structure, permissions, and manifest file content, without executing the apps. Instead of relying solely on raw static features, our system uses an LLM to generate high-level functional descriptions of APKs. To mitigate hallucinations, which are a known vulnerability of LLM, we integrated Retrieval-Augmented Generation (RAG), enabling the LLM to ground its output in relevant context. Using carefully designed prompts, we guide the LLM to produce coherent function summaries, which are then analyzed using a transformer-based model, improving detection accuracy over conventional feature-based methods for malware detection.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
TraceRAG: A LLM-Based Framework for Explainable Android Malware Detection and Behavior Analysis
An LLM-based RAG framework that retrieves method-level Java code snippets to explain and detect malicious behavior in Android apps.
Reference graph
Works this paper leans on
-
[1]
Steven Arzt, Siegfried Rasthofer, Christian Fritz, Eric Bodden, Alexandre Bartel, Jacques Klein, Yves Le Traon, Damien Octeau, and Patrick McDaniel. Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps. ACM sigplan notices, 49(6):259–269, 2014
work page 2014
-
[2]
Android permissions demystified
Adrienne Porter Felt, Erika Chin, Steve Hanna, Dawn Song, and David Wagner. Android permissions demystified. In Proceedings of the 18th ACM conference on Computer and communications security , pages 627–638, 2011
work page 2011
-
[3]
Dissecting android malware: Characterization and evolution
Yajin Zhou and Xuxian Jiang. Dissecting android malware: Characterization and evolution. In 2012 IEEE symposium on security and privacy , pages 95–109. IEEE, 2012
2012
-
[4]
Drebin: Effective and explainable detection of android malware in your pocket
Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad Rieck, and CERT Siemens. Drebin: Effective and explainable detection of android malware in your pocket. In Ndss, volume 14(1), pages 23–26, 2014. 16 Enhancing Android Malware Detection with Retrieval-Augmented Generation
work page 2014
-
[5]
Copperdroid: Automatic reconstruction of android malware behaviors
Kimberly Tam, Aristide Fattori, Salahuddin Khan, and Lorenzo Cavallaro. Copperdroid: Automatic reconstruction of android malware behaviors. In NDSS Symposium 2015, pages 1–15, 2015
work page 2015
-
[6]
Droidecho: an in-depth dissection of malicious behaviors in android applications
Guozhu Meng, Ruitao Feng, Guangdong Bai, Kai Chen, and Yang Liu. Droidecho: an in-depth dissection of malicious behaviors in android applications. Cybersecurity, 1:1–17, 2018
work page 2018
-
[7]
Yes, machine learning can be more secure! a case study on android malware detection
Ambra Demontis, Marco Melis, Battista Biggio, Davide Maiorca, Daniel Arp, Konrad Rieck, Igino Corona, Giorgio Giacinto, and Fabio Roli. Yes, machine learning can be more secure! a case study on android malware detection. IEEE transactions on dependable and secure computing , 16(4):711–724, 2017
work page 2017
-
[8]
Agentic information retrieval, 2025
Weinan Zhang, Junwei Liao, Ning Li, Kounianhua Du, and Jianghao Lin. Agentic information retrieval, 2025
2025
Show all 21 references
-
[9]
Droidsieve: Fast and accurate classification of obfuscated android malware
Guillermo Suarez-Tangil, Santanu Kumar Dash, Mansour Ahmadi, Johannes Kinder, Giorgio Giacinto, and Lorenzo Cavallaro. Droidsieve: Fast and accurate classification of obfuscated android malware. In Proceedings of the seventh ACM on conference on data and application security a...
2017
-
[10]
Mamadroid: Detecting android malware by building markov chains of behavioral models (extended version)
Lucky Onwuzurike, Enrico Mariconti, Panagiotis Andriotis, Emiliano De Cristofaro, Gordon Ross, and Gianluca Stringhini. Mamadroid: Detecting android malware by building markov chains of behavioral models (extended version). ACM Transactions on Privacy and Security (TOPS) , 22(...
2019
-
[11]
Droidchameleon: evaluating android anti-malware against transformation attacks
Vaibhav Rastogi, Yan Chen, and Xuxian Jiang. Droidchameleon: evaluating android anti-malware against transformation attacks. In Proceedings of the 8th ACM SIGSAC symposium on Information, computer and communications security, pages 329–334, 2013
2013
-
[12]
Evolution, detection and analysis of malware for smart devices
Guillermo Suarez-Tangil, Juan E Tapiador, Pedro Peris-Lopez, and Arturo Ribagorda. Evolution, detection and analysis of malware for smart devices. IEEE communications surveys & tutorials , 16(2):961–987, 2013
2013
-
[13]
In 21st USENIX security symposium (USENIX security 12) , pages 569–584, 2012
Lok Kwong Yan and Heng Yin.{DroidScope}: Seamlessly reconstructing the {OS} and dalvik semantic views for dynamic android malware analysis. In 21st USENIX security symposium (USENIX security 12) , pages 569–584, 2012
2012
-
[14]
Effective android malware detection with a hybrid model based on deep autoencoder and convolutional neural network
Wei Wang, Mengxue Zhao, and Jigang Wang. Effective android malware detection with a hybrid model based on deep autoencoder and convolutional neural network. Journal of Ambient Intelligence and Humanized Computing , 10(8):3035–3043, 2019
2019
-
[15]
Hindroid: An intelligent android malware detection system based on structured heterogeneous information network
Shifu Hou, Yanfang Ye, Yangqiu Song, and Melih Abdulhayoglu. Hindroid: An intelligent android malware detection system based on structured heterogeneous information network. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining ,...
2017
-
[16]
Summary of chatgpt-related research and perspective towards the future of large language models
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. Summary of chatgpt-related research and perspective towards the future of large language models. Meta-radiology, 1(2):100017, 2023
2023
-
[17]
Apppoet: Large language model based android malware detection via multi-view prompt engineering
Wenxiang Zhao, Juntao Wu, and Zhaoyi Meng. Apppoet: Large language model based android malware detection via multi-view prompt engineering. Expert Systems with Applications , 262:125546, 2025
2025
-
[18]
Judging llm-as-a-judge with mt-bench and chatbot arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023
2023
-
[19]
The research trends of corpus-assisted stance research (2004–2023): a systematic literature review
Ya Sun, Kaiwen Yang, and Gong Chen. The research trends of corpus-assisted stance research (2004–2023): a systematic literature review. Current Psychology, pages 1–16, 2024
2004
-
[20]
Cysecbert: A domain-adapted language model for the cybersecurity domain
Markus Bayer, Philipp Kuehn, Ramin Shanehsaz, and Christian Reuter. Cysecbert: A domain-adapted language model for the cybersecurity domain. ACM Transactions on Privacy and Security , 27(2):1–20, 2024
2024
-
[21]
Secbert: Privacy-preserving pre-training based neural network inference system
Hai Huang and Yongjian Wang. Secbert: Privacy-preserving pre-training based neural network inference system. Neural Networks, 172:106135, 2024. 17
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.