REVIEW 5 major objections 5 minor 1 cited by
Exploring Large Language Models for Semantic Analysis and Categorization of Android Malware
T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A general-purpose LLM, prompted but not fine-tuned, can summarize Android apps and flag malware with 77% accuracy, then trace its verdict down to the specific malicious function.
desk verdict Useful hierarchical summarization pipeline for Android malware triage, but the headline 77% accuracy is unsupported by the paper's own confusion matrices, which do not add up. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the hierarchical-tiered summarization chain, a bottom-up pipeline that compresses the large codebase of an APK into compact, security-relevant summaries at function, class, and package granularity, allowing a model with a finite context window to process an entire application. It is driven by three prompt scopes (Vanilla, API-Scoped, and Malware-Scoped) that inject progressively more security knowledge into the model; the Malware-Scoped prompts, which list suspicious behaviors such as dynamic class loading, rooting, privilege escalation, and data exfiltration, are what push accuracy from 49.5% to 77%. The chain also provides explainability, because each summary level is derived from the level below, so tracing a package-level claim back to the code recovers the root cause.
What would settle it
Take a set of APKs whose malicious code lives in classes that Dex2Jar or CFR demonstrably fail to recover (confirmed by comparing the decompiled class list against the actual dex bytecode), run MalParse's Malware-Scoped pipeline on them, and check whether the classification accuracy drops measurably below 77%; if it stays near 77%, the decompilation-preservation premise is wrong, and if it drops, the premise is confirmed as the load-bearing limitation.
Extended reading notes
Core claim
MalParse, built on GPT-4o-mini, decompiles an Android APK into Java source and runs a bottom-up hierarchical summarization chain: each function is summarized, function summaries are merged into class summaries, and class summaries are merged into a package summary. By injecting malware-specific context through what the authors call Malware-Scoped prompting, the pipeline reaches a balanced classification rate of 77%, correctly identifying 76% of benign apps and 78% of malware samples, all without any training on malware or benign examples and without revealing APK names to the model. Backtracking from the package summary through class and function summaries lets the system pinpoint the precise code snippet responsible, demonstrated on a rooting malware sample where the function using a DexClassLoader is identified. The paper presents this as evidence that prompt engineering alone can give a general LLM meaningful semantic understanding of malicious Android code.
Load-bearing premise
The whole chain assumes that turning an APK into Java source with Dex2Jar and CFR preserves enough of the app's behavior that the LLM's summaries reflect what the malware actually does.
Editorial extensions
If this is right
- Without fine-tuning, a general LLM can achieve 77% balanced accuracy on a 100-malware/100-benign Android set, suggesting that zero-day triage may not require retraining for every new malware family.
- Prompt context matters more than model specialization: moving from generic to malware-scoped prompts raised accuracy from 49.5% to 77%, making prompt design an effective control knob in LLM-based security analysis.
- Analysts can use the hierarchical summaries as a navigation map, reading the package-level verdict and then drilling into classes and functions to see the specific code that triggered it, which shortens manual reverse engineering.
- Because only the prompt carries the malware knowledge, expanding the Malware-Scoped prompt's list of suspicious behaviors could directly reduce the 22% false-negative rate, as the paper itself hypothesizes.
- The same summarization chain could be applied to any LLM, not just GPT-4o-mini, providing a baseline to evaluate cheaper or open-source models on the same malware analysis task.
Reading between the lines
- The 77% figure is dataset-specific and should be read as an upper bound; a deployment would need validation on a much larger and more diverse APK corpus before relying on that rate.
- Although the paper hides APK names to prevent the model from relying on memory, the decompiled Java itself may still contain package or class identifiers that leak app identity to the LLM's pretraining, making the 'no prior knowledge' claim less clean than stated.
- A natural extension is to run the same pipeline on obfuscated and packed malware, where static decompilation is known to degrade; the paper's own limitation section predicts accuracy will drop, turning this weakness into a testable prediction.
- The backtracking capability could be formalized as an evidence requirement: if every package-level verdict had to cite at least one function-level snippet, then an analyst's trust in a verdict could be quantified by checking whether that snippet is actually malicious.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents MalParse, a pipeline for Android malware analysis that decompiles APKs to Java source using Dex2Jar and CFR, generates hierarchical function-, class-, and package-level summaries with GPT-4o-mini under three prompt scopes (Vanilla, API-Scoped, Malware-Scoped), and uses the final package summary to classify each app as benign or malicious. On a balanced set of 200 apps (100 VirusTotal malware, 100 top Similarweb apps), the authors report 49.5% accuracy with Vanilla prompts, 56% with API-Scoped prompts, and 77% with Malware-Scoped prompts, and illustrate that backward tracing from package-level summaries can locate the code responsible for malicious behavior (e.g., a rooting payload in the RTAccessHandler class). The paper also discusses limitations, chiefly decompilation quality, obfuscation, and prompt bias.
Significance. If the reported figures are correct after revision, the paper makes a modest but useful empirical contribution: it shows that a general-purpose LLM with carefully engineered prompts can produce human-readable, hierarchical code summaries for Android apps and perform coarse benign/malicious triage without fine-tuning, which could help accelerate triage of novel samples. The hierarchical summarization design is a pragmatic answer to LLM context-window limits, and the three-way prompt comparison is a reasonable internal ablation. The main weaknesses are the small and not fully documented dataset, the absence of conventional static-analysis baselines, and the fact that the central accuracy numbers are not internally consistent as printed.
major comments (5)
- [Section 4.2.1, Tables 1-3] All three confusion matrices are arithmetically impossible as printed. In Table 1, the 'Benign Actual' row sums to 185% (TP=92%, FN=93%) and the 'Malware Actual' row sums to 15% (FP=8%, TN=7%); the stated 49.5% accuracy (99/200) corresponds to TP=92 and TN=7, which forces FN=8% and FP=93%, i.e., the off-diagonal values are interchanged. In Table 2, the row sums are 168% and 32%; the stated 56% (112/200) forces FN=10% and FP=78% rather than the printed 78% and 10%. In Table 3, the row sums are 98% and 102%; the stated 77% (154/200) forces FN=24% and FP=22% rather than the printed 22% and 24%. Please provide corrected raw contingency counts for all three tables, since the headline accuracy depends directly on these numbers.
- [Section 4.2.1 and Introduction] The Vanilla prompt accuracy is reported as 49.5% (Introduction and §4.2.1.1), as 45% (§4.2.1.2), and as 45.5% (§4.2.1.3). These three values cannot all be correct, and the text does not explain the discrepancy. Please verify the underlying results and report a single consistent value, with raw counts.
- [Section 4.3.1] The statement that the system achieved 'a balanced accuracy at 77% with a precision of 76% and a recall rate of 77%' is not derivable from the corrected Table 3 counts under any standard class convention: with malware as the positive class, precision is approximately 76% but recall is 78%; with benign as positive, recall is 76% and precision is approximately 78%. Please specify the positive class and formulas used, and recompute all derived metrics so that they are mutually consistent.
- [Section 4.2.1, Section 3.3] The three prompt scopes are evaluated on the same 200-sample set, and the best-performing configuration (Malware-Scoped) is then reported as the system's headline accuracy. Because Section 3.3 describes iterative prompt crafting and Section 4.3.1 states that accuracy improves as prompting is tuned, the 77% figure is an in-sample optimum; no hold-out validation, cross-validation, or repeated sampling (e.g., multiple GPT-4o-mini runs with nonzero temperature) is reported. Please add a validation protocol, report variance across runs, or provide raw per-sample decisions so that the generality of the 77% number can be assessed.
- [Section 4.2.2] The claim that backtracking through the summaries 'allowed us to pinpoint the precise code snippets responsible for malicious behavior' (Abstract) is supported only by a single hand-picked example, the rooting sample in Table 5. Please evaluate the traceability systematically over the correctly classified malware samples (for example, how often the package-level summary leads to the correct malicious class and function), or explicitly restrict the claim to an illustrative case study.
minor comments (5)
- [Section 3.1] 'Davik Bytecode' should read 'Dalvik Bytecode', and 'decomples' should read 'decompiles'.
- [Section 4.1] The text states that 'the names of the APKs were not provided' to the model; since decompiled Java source retains package and class names, the model may still be able to infer the application identity from the code content. Please clarify whether package names were stripped or otherwise anonymized, and how the file names were excluded from the inputs.
- [Section 4.3.1] The acknowledged decompilation failures of Dex2Jar/CFR are not quantified; please report how many of the 200 APKs failed to decompile fully, and, if possible, separate those cases from LLM misclassifications so that the impact on the 77% figure can be understood.
- [Section 2] The literature review would benefit from a discussion of prior Android malware detection efforts that use machine learning or LLMs (beyond code summarization and npm malware detection), so that the novelty of the categorization and hierarchical summarization approach can be positioned more precisely.
- [Section 4.1 and Abstract] The paper repeatedly states that the model was 'not pre-trained on previous malware or benign samples'; this claim about GPT-4o-mini's pretraining data is not verifiable. Please rephrase to 'not fine-tuned on the evaluation dataset' or similar.
Circularity Check
No significant circularity: the classification claim is an empirical measurement against external ground-truth labels, not a quantity derived from the input definitions by construction.
full rationale
MalParse's central claim is an empirical classification rate obtained by comparing GPT-4o-mini's package-level summaries to externally sourced ground truth: malware samples from VirusTotal and benign samples from Similarweb's top-500 list. The three prompt variants are engineering choices, and the 77% figure is the reported result of one of the three runs (Malware-Scoped), not a parameter fitted to the labels or a quantity forced by the prompt text. There are no load-bearing self-citations, no imported uniqueness theorems, and no renaming of a known empirical result. The paper's own limitation statement (Section 4.3.1) about Dex2Jar/CFR decompilation failures weakens external validity but is not a circular step. The apparent arithmetic inconsistencies in the confusion matrices and the post-hoc selection of the best prompt variant as 'optimized' are correctness/reproducibility and test-set-selection concerns, not circularity, because the reported numbers are empirical observations rather than reductions to input definitions.
Assumptions & free parameters
assumptions (4)
- domain assumption Dex2Jar and CFR decompilation preserves the semantic content needed for malware classification.
- domain assumption GPT-4o-mini's generated summaries accurately reflect the code content and are not hallucinated.
- domain assumption VirusTotal labels for malicious apps and Similarweb top-500 selection for benign apps correctly define ground truth.
- domain assumption The malware behavioral categories injected into the Malware-Scoped prompts are sufficient to cover the behaviors in the test set.
Cite this review
Pith. "Pith review of Exploring Large Language Models for Semantic Analysis and Categorization of Android Malware." pith.science (2026). https://pith.science/paper/VQIQL2NS
@misc{pith2026250104848,
author = {Pith},
title = {Pith review of: Exploring Large Language Models for Semantic Analysis and Categorization of Android Malware},
year = {2026},
howpublished = {\url{https://pith.science/paper/VQIQL2NS}},
note = {Machine review of arXiv:2501.04848}
}
read the original abstract
Malware analysis is a complex process of examining and evaluating malicious software's functionality, origin, and potential impact. This arduous process typically involves dissecting the software to understand its components, infection vector, propagation mechanism, and payload. Over the years, deep reverse engineering of malware has become increasingly tedious, mainly due to modern malicious codebases' fast evolution and sophistication. Essentially, analysts are tasked with identifying the elusive needle in the haystack within the complexities of zero-day malware, all while under tight time constraints. Thus, in this paper, we explore leveraging Large Language Models (LLMs) for semantic malware analysis to expedite the analysis of known and novel samples. Built on GPT-4o-mini model, \msp is designed to augment malware analysis for Android through a hierarchical-tiered summarization chain and strategic prompt engineering. Additionally, \msp performs malware categorization, distinguishing potential malware from benign applications, thereby saving time during the malware reverse engineering process. Despite not being fine-tuned for Android malware analysis, we demonstrate that through optimized and advanced prompt engineering \msp can achieve up to 77% classification accuracy while providing highly robust summaries at functional, class, and package levels. In addition, leveraging the backward tracing of the summaries from package to function levels allowed us to pinpoint the precise code snippets responsible for malicious behavior.
Figures
Forward citations
Cited by 1 Pith paper
-
Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing
A new evaluation framework shows current LLMs are not yet reliable at fine-grained Android malware behavior auditing, with the best model scoring 50.67% on the composite workload metric.
Reference graph
Works this paper leans on
-
[1]
Codebert: A pre-trained model for pro- gramming and natural languages,
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for pro- gramming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020
arXiv 2002
-
[2]
Graphcodebert: Pre-training code representations with data flow,
D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu et al. , “Graphcodebert: Pre-training code representations with data flow,” arXiv preprint arXiv:2009.08366 , 2020
arXiv 2009
-
[3]
Uni- fied pre-training for program understanding and generation,
W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Uni- fied pre-training for program understanding and generation,” arXiv preprint arXiv:2103.06333 , 2021
arXiv 2021
-
[4]
Language models are few-shot learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P . Dhariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information pro- cessing systems , vol. 33, pp. 1877–1901, 2020
work page 1901
-
[5]
An analysis of the automatic bug fixing performance of chatgpt,
D. Sobania, M. Briesch, C. Hanna, and J. Petke, “An analysis of the automatic bug fixing performance of chatgpt,” in 2023 IEEE/ACM International Workshop on Automated Program Repair (APR). IEEE, 2023, pp. 23–30
work page 2023
-
[6]
The potential use of chatgpt for debugging and bug fixing,
M. A. Haque and S. Li, “The potential use of chatgpt for debugging and bug fixing,” 2023
work page 2023
-
[7]
M. C. Wuisang, M. Kurniawan, K. A. W. Santosa, A. A. S. Gunawan, and K. E. Saputra, “An evaluation of the effectiveness of openai’s chatgpt for automated python program bug fixing using quixbugs,” in 2023 International Seminar on Application for Technology of Information and Communication (iSemantic) . IEEE, 2023, pp. 295– 300
work page 2023
-
[8]
Q. Zhang, T. Zhang, J. Zhai, C. Fang, B. Y u, W. Sun, and Z. Chen, “A critical review of large language model on software engineering: An example from chatgpt and automated program repair,” arXiv preprint arXiv:2310.08879, 2023
Show all 18 references
-
[9]
Refining chatgpt-generated code: Charac- terizing and mitigating code quality issues,
Y . Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X.- B. D. Le, and D. Lo, “Refining chatgpt-generated code: Charac- terizing and mitigating code quality issues,” ACM Transactions on Software Engineering and Methodology , 2023
2023
-
[10]
Shifting the lens: Detecting malware in npm ecosystem with large language models,
N. Zahan, P . Burckhardt, M. Lysenko, F. Aboukhadijeh, and L. Williams, “Shifting the lens: Detecting malware in npm ecosystem with large language models,” arXiv preprint arXiv:2403.12196 , 2024
2024 arXiv
-
[11]
Malware analysis and detection using reverse engineering technique,
S. Megira, A. Pangesti, and F. Wibowo, “Malware analysis and detection using reverse engineering technique,” in Journal of Physics: Conference Series , vol. 1140, no. 1. IOP Publishing, 2018, p. 012042
2018
-
[12]
dex2jar,
Panxiaobo, “dex2jar,” http://code.google.com/p/dex2jar/, available on- line
-
[13]
Cfr decompiler,
L. Benfield, “Cfr decompiler,” 2020, available online. [Online]. Available: https://www.benf.org/other/cfr/
2020
-
[14]
Summit: Iterative text summarization via chatGPT,
H. Zhang, X. Liu, and J. Zhang, “Summit: Iterative text summarization via chatGPT,” in The 2023 Conference on Empirical Methods in Natural Language Processing , 2023. [Online]. Available: https://openreview.net/forum?id=LCEbV5nsb8
2023
-
[15]
Element-aware summarization with large language models: Expert-aligned evaluation and chain- of-thought method,
Y . Wang, Z. Zhang, and R. Wang, “Element-aware summarization with large language models: Expert-aligned evaluation and chain- of-thought method,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) . Toronto, Can...
2023
-
[16]
Langchain,
C. Harrison, “Langchain,” ver. 2, 2024. [Online]. Available: https://github.com/langchain-ai/langchain
2024
-
[17]
Top 500 applications in the united states,
Similarweb, “Top 500 applications in the united states,”
-
[2023]
Available: https://www.similarweb.com/top-websites/ united-states/category/application
[Online]. Available: https://www.similarweb.com/top-websites/ united-states/category/application
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.