REVIEW 5 major objections 4 minor 13 references
Breaking Android with AI: A Deep Dive into LLM-Powered Exploitation
T0 review · 5 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read An LLM pipeline built from PentestGPT and an OpenAI-backed script generator can automate many Android rooting and exploitation steps in emulated environments, while kernel and bootloader attacks remain outside the testbed's reach.
desk verdict Feasibility map of LLM-generated Android scripts in Genymotion, but "Worked" scores conflate script execution with root access, so the central exploitation claim is not yet supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a three-stage pipeline. PentestGPT supplies the attack reasoning: given a structured Android-rooting flowchart and iterative prompts, it outputs concrete exploitation techniques. A custom Streamlit web application connected to the OpenAI API then parses that text and emits executable Bash, Python, or ADB scripts, with an ethical filter that blocks dangerous commands. Finally, the Genymotion Android emulator acts as the sandbox in which the scripts run on multiple Android versions in rooted and unrooted states; failure logs are fed back to PentestGPT to refine later suggestions. The paper's evaluation metrics—success rate, security-detection rate, adaptability sc
What would settle it
Re-run the pipeline on a fresh, default-unrooted Android emulator with a locked bootloader, logging prompts and raw LLM responses, and execute the scripts exactly as generated without human edits; if no new root access is obtained, or if every working script requires manual repair, the central claim that LLMs automate Android exploitation is not supported.
Extended reading notes
Core claim
On its own terms, the paper reports that an LLM-powered workflow can generate working exploit scripts for Android without a human writing the attack code. The authors query PentestGPT with general and then structured prompts, pass the resulting attack plan to an OpenAI-integrated web application that produces executable Bash/Python/ADB scripts, and execute those scripts on rooted and unrooted Genymotion emulators. Nine of the evaluated features succeeded, most at a 100% success rate and none flagged by security mechanisms except a 50%-detected remote-code-execution case. The features that failed—kernel exploits, bootloader unlocking, custom recovery installation, and Magisk boot-image patchi
Load-bearing premise
The evidence credits the LLM pipeline, but the paper never shows the original prompts, the raw model output, or the unedited scripts; since every script passed through manual review and iterative re-prompting, the successes could reflect human curation rather than LLM capability.
Editorial extensions
If this is right
- If the pipeline works as reported, security teams can automate repetitive Android pentest steps—recon, script generation, and root verification—and spend human effort on review and edge cases.
- LLM-generated scripts can cover ADB-based, network-based, and app-layer attack surfaces across multiple Android versions, making large-scale emulator-based security auditing feasible.
- Bootloader unlocking, custom recovery, and kernel exploitation remain outside the reach of emulator-only LLM testing; evaluating those requires physical devices, so the paper's scope boundary is a testbed limit, not a demonstrated LLM limit.
- Because several successful features are rated high ethical risk, any practical deployment of LLM-powered exploitation will need script filtering, human sign-off, and sandboxing baked into the workflow.
Reading between the lines
- Editorial inference: The paper does not show the prompts, the raw LLM responses, or the final scripts, so the cleanest reading of its evidence is that the full human-in-the-loop pipeline works, not that the LLM alone does.
- Editorial inference: Genymotion images are commonly pre-rooted by design; if the 'rooted' test devices came with root already enabled, the reported success of 'sideload Magisk' and 'reboot and verify root' may demonstrate script execution rather than privilege escalation.
- Editorial inference: A natural next test is to run the same pipeline against a default-unrooted emulator or a locked physical device; that would separate testbed artifacts from genuine LLM exploitation capability.
- Editorial inference: The 50% success and 50% detection of the remote-code-execution feature suggests detection mechanisms can catch some LLM-generated payloads, so adversarial refinement of payloads may become the next battleground as defenders apply the same LLM tools.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes a pipeline that combines PentestGPT with a custom Streamlit/OpenAI web application to generate Android rooting and exploitation scripts, and evaluates them on Genymotion emulators (Android 11 rooted, Android 13 unrooted, plus Android 12 rooted and Android 14 unrooted in Table II). The authors report high success rates for many AI-generated exploitation features, including backup, Magisk sideload, ADB-based attacks, MITM, and component hijacking, while noting that bootloader, recovery, and A/B-partition operations cannot be tested in Genymotion. The central claim is that LLM-driven automation can streamline Android penetration testing, with human oversight remaining necessary. The paper also defines metrics for success rate, detection rate, adaptability, and ethical risk.
Significance. If the empirical claims were fully supported, the paper would provide a useful data point on the current capabilities and limitations of LLM-based Android exploitation in emulated environments. The controlled use of Genymotion, the explicit acknowledgment of emulator limitations, and the definition of quantitative metrics are commendable. However, the evidence as presented is not yet sufficient to support the central claim that the AI pipeline achieves successful exploitation (as opposed to mere script execution). The key strengths are the external ground-truth execution environment and the honest enumeration of features that cannot be tested; the key weaknesses are the ambiguous 'Worked' criterion, missing trial counts and verification logs, and the absence of the promised manual-vs-AI comparison.
major comments (5)
- [Table II and Section VI-C] The 'Worked' entries for 'Sideload Magisk.zip' on unrooted Android 13 and 14 conflict with the paper's own statement that Genymotion has no recovery partition, no TWRP, and no A/B partition scheme. Sideloading Magisk normally requires a custom recovery or a patched boot image; without either, the command cannot yield root on an unrooted device. The only consistent reading is that 'Worked' means 'the script executed without error', not 'root access or successful penetration' as defined in Eq. (1). Since Table I reports 100% success rates for this and other non-exploitation features, the central exploitation claim is not currently supported. The authors must specify the exact verification criterion for 'Worked', provide root-verification logs (e.g., 'adb shell su' output), and separate 'script execution success' from 'security objective achieved'.
- [Table I and Section VI-A] Success rates in Table I are reported as percentages without any trial counts, confidence intervals, or descriptions of how many attempts were made per feature. A single successful run and a single failed run both produce 50% for RCE, but with very different evidentiary weight. The paper should report the number of attempts per feature, the number of distinct emulator instances, and the variability across runs. Without this, the 100% success-rate column is not a reproducible quantitative result.
- [Abstract and Section VI-A] The abstract and introduction promise a comparison between traditional manual rooting and AI-generated exploitation methods. Section VI-A describes only AI-generated scripts; no manual baseline, timing measurements, or effort comparison appear anywhere in the evaluation. Since the claimed benefit is that AI 'significantly streamline[s] the workflow of exploitation', the missing manual comparison is load-bearing and must be supplied or the claim must be substantially weakened.
- [Sections IV-D and IV-E] The pipeline includes iterative re-prompting on failure (Section IV-D) and manual review of every script before execution (Section IV-E). The paper does not provide the initial prompts, the raw PentestGPT responses, the generated scripts, or the number of refinement iterations. Consequently, a reader cannot determine whether the observed successes are attributable to the LLM or to human curation during the review/refinement loop. At minimum, the authors should release the artifacts and the revision history, or clearly separate automated runs from human-modified runs.
- [Table I and Eq. (2)] The Detection Rate metric is defined as blocked exploits over total attempts, but Table I reports 'Not Detected' for almost every feature without saying which security mechanisms were active in Genymotion. Genymotion images may not enforce SELinux, Google Play Protect, or Android Verified Boot in the same way as physical devices. The paper should state the security configuration of the emulator and how detection was measured; otherwise the detection-rate column is not interpretable.
minor comments (4)
- [Throughout] There are numerous typos and grammar issues, e.g., 'recitable' (Section V), 'this restriction is happened' (Section VI-C), 'AI-generate methodologies' (Section V), and 'Successf ulExecutions' in Eq. (1). A careful proofreading pass is needed.
- [Section IV-B] The text says 'shows in 4 as a reference input' but likely means Figure 4. Also, Figures 2 and 3 are not explicitly referenced in the prose at their first appearance; please add callouts and captions that explain what each figure contributes.
- [Section VI-B] The sentence introducing Table I says 'the evaluation metrics results are included in Table I', but the table lists only features and success/detection/adaptability/risk values. It would be clearer to state which metric each column corresponds to and to note that Table II is the per-version execution matrix.
- [References] Some related-work citations are informal (e.g., 'one-day vulnerabilities' in [7]); consider using standard terminology and checking that all references are consistently formatted. Also, the paper does not cite or discuss prior Android-specific LLM exploitation studies beyond those listed; adding a short comparison would strengthen positioning.
Circularity Check
No circularity: the paper's claims rest on external emulator execution and no derivation reduces to its inputs.
full rationale
The paper makes no mathematical derivation or first-principles prediction. Its central claim—that LLM-generated scripts can execute rooting and exploitation techniques in Genymotion—is evaluated empirically against external ground truth (whether commands run and whether root/penetration is achieved). Success Rate, Detection Rate, Adaptability, and Ethical Risk are defined as measurements of observed executions, not as quantities derived from the assumptions. No parameter is fitted to a subset of data and then re-predicted; no quantity is defined in terms of the conclusion. The related-work citations (PentestGPT, VulnBot, etc.) are background and tooling, not load-bearing self-citations; the authors do not cite their own prior work as evidence for the central result. The only noteworthy concern is the ambiguity of 'Worked' in Table II relative to Section VI-C's statement that recovery/bootloader features are unavailable (e.g., 'Sideload Magisk.zip' marked Worked despite no recovery partition). That is a measurement-validity or reporting inconsistency, not circular reasoning: the table reports empirical outcomes rather than deriving the conclusion from the inputs. Likewise, the use of the same LLM pipeline to generate and refine scripts could bias failure reporting, but this is an experimental-design risk, not a circularity. Therefore no circular step can be exhibited with the specificity required by the rules.
Assumptions & free parameters
assumptions (3)
- domain assumption Genymotion emulator behavior is representative of real Android security for the tested attack surfaces (ADB, network, app components)
- domain assumption The executed scripts were generated by the LLM pipeline with only ethical filtering, not substantially authored by the researchers
- domain assumption Success and detection classifications are based on objective verification procedures
Cite this review
Pith. "Pith review of Breaking Android with AI: A Deep Dive into LLM-Powered Exploitation." pith.science (2026). https://pith.science/paper/WRFV3PVS
@misc{pith2026250907933,
author = {Pith},
title = {Pith review of: Breaking Android with AI: A Deep Dive into LLM-Powered Exploitation},
year = {2026},
howpublished = {\url{https://pith.science/paper/WRFV3PVS}},
note = {Machine review of arXiv:2509.07933}
}
read the original abstract
The rapid evolution of Artificial Intelligence (AI) and Large Language Models (LLMs) has opened up new opportunities in the area of cybersecurity, especially in the exploitation automation landscape and penetration testing. This study explores Android penetration testing automation using LLM-based tools, especially PentestGPT, to identify and execute rooting techniques. Through a comparison of the traditional manual rooting process and exploitation methods produced using AI, this study evaluates the efficacy, reliability, and scalability of automated penetration testing in achieving high-level privilege access on Android devices. With the use of an Android emulator (Genymotion) as the testbed, we fully execute both traditional and exploit-based rooting methods, automating the process using AI-generated scripts. Secondly, we create a web application by integrating OpenAI's API to facilitate automated script generation from LLM-processed responses. The research focuses on the effectiveness of AI-enabled exploitation by comparing automated and manual penetration testing protocols, by determining LLM weaknesses and strengths along the way. We also provide security suggestions of AI-enabled exploitation, including ethical factors and potential misuse. The findings exhibit that while LLMs can significantly streamline the workflow of exploitation, they need to be controlled by humans to ensure accuracy and ethical application. This study adds to the increasing body of literature on AI-powered cybersecurity and its effect on ethical hacking, security research, and mobile device security.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
A risk estimation study of native code vulnerabilities in android applications,
S. L. Sanna, D. Soi, D. Maiorca, G. Fumera, and G. Giacinto, “A risk estimation study of native code vulnerabilities in android applications,” Journal of Cybersecurity, vol. 10, no. 1, p. tyae015, 2024
work page 2024
-
[2]
Android custom permissions demystified: From privilege escalation to design shortcomings,
R. Li, W. Diao, Z. Li, J. Du, and S. Guo, “Android custom permissions demystified: From privilege escalation to design shortcomings,” in2021 IEEE Symposium on security and privacy (SP). IEEE, 2021, pp. 70–86
work page 2021
-
[3]
Cracking the core: Hardware vulnerabilities in android devices unveiled,
A. Mu ˜noz, “Cracking the core: Hardware vulnerabilities in android devices unveiled,”Electronics, vol. 13, no. 21, p. 4269, 2024
work page 2024
-
[4]
Pentestgpt: An llm-empowered automatic penetration testing tool,
G. Deng, Y . Liu, V . Mayoral-Vilches, P. Liu, Y . Li, Y . Xu, T. Zhang, Y . Liu, M. Pinzger, and S. Rass, “Pentestgpt: An llm-empowered automatic penetration testing tool,”arXiv preprint arXiv:2308.06782, 2023
arXiv 2023
-
[5]
Vulnbot: Autonomous penetration testing for a multi-agent collaborative framework,
H. Kong, D. Hu, J. Ge, L. Li, T. Li, and B. Wu, “Vulnbot: Autonomous penetration testing for a multi-agent collaborative framework,”arXiv preprint arXiv:2501.13411, 2025
arXiv 2025
-
[6]
Generative ai for pentesting: the good, the bad, the ugly,
E. Hilario, S. Azam, J. Sundaram, K. Imran Mohammed, and B. Shan- mugam, “Generative ai for pentesting: the good, the bad, the ugly,” International Journal of Information Security, vol. 23, no. 3, pp. 2075– 2097, 2024
-
[7]
Llm agents can autonomously exploit one-day vulnerabilities,
R. Fang, R. Bindu, A. Gupta, and D. Kang, “Llm agents can autonomously exploit one-day vulnerabilities,”arXiv preprint arXiv:2404.08144, vol. 13, p. 14, 2024
arXiv 2024
-
[8]
M. H. Meng, G. Bai, J. K. Liu, X. Luo, and Y . Wang, “Analyzing use of high privileges on android: an empirical case study of screenshot and screen recording applications,” inInformation Security and Cryptology: 14th International Conference, Inscrypt 2018, Fuzhou, China, December 14-17, 2018, Revised Selected Papers 14. Springer, 2019, pp. 349–369
work page 2018
Show all 13 references
-
[9]
Emulating the android boot process,
A. R. Bertels, R. E. Bell, and B. K. Eames, “Emulating the android boot process,” Sandia National Lab.(SNL-NM), Albuquerque, NM (United States), Tech. Rep., 2022
2022
-
[10]
Llms as hackers: Autonomous linux privilege escalation attacks,
A. Happe, A. Kaplan, and J. Cito, “Llms as hackers: Autonomous linux privilege escalation attacks,”arXiv preprint arXiv:2310.11409, 2023
2023
-
[11]
Getting pwn’d by ai: Penetration testing with large language models,
A. Happe and J. Cito, “Getting pwn’d by ai: Penetration testing with large language models,” inProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the F oundations of Software Engineering, 2023, pp. 2082–2086
2023
-
[12]
Automated vulnerability exploita- tion using deep reinforcement learning,
A. AlMajali, L. Al-Abed, K. M. Ahmad Yousef, B. J. Mohd, Z. Samamah, and A. Abu Shhadeh, “Automated vulnerability exploita- tion using deep reinforcement learning,”Applied Sciences, vol. 14, no. 20, p. 9331, 2024
2024
-
[13]
Pentest-ai, an llm-powered multi- agents framework for penetration testing automation leveraging mitre attack,
S. G. Bianou and R. G. Batogna, “Pentest-ai, an llm-powered multi- agents framework for penetration testing automation leveraging mitre attack,” in2024 IEEE International Conference on Cyber Security and Resilience (CSR). IEEE, 2024, pp. 763–770
2024
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.