Pith. sign in

REVIEW 4 major objections 5 minor 65 references

Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Current large language models cannot yet audit Android malware: the best of seven scores only 50.67% on the new MalEval benchmark, and models lean on surface cues instead of verifiable evidence.

desk verdict A useful malware-auditing benchmark whose qualitative claims hold up, but whose headline numbers rest on unvalidated LLM judgment and a cross-model input confound. read the letter →

arxiv 2509.14335 v2 pith:7P4YCMQS submitted 2025-09-17 cs.CR cs.AIcs.SE

Xinran Zheng , Xingzhi Qian , Yiling He , Shuo Yang , Lorenzo Cavallaro This is my paper · ORCID
classification cs.CRcs.AIcs.SE
keywords AndroidmalwarebehaviorauditinglargelanguagemodelsLLMevaluationevidenceattributionstaticanalysissynthesisworkloadreduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MalEval is a diagnostic benchmark for deciding whether large language models can do fine-grained Android malware auditing—not just flag a sample as malicious, but explain what it does and back that explanation with code-level evidence. The paper builds a dataset of real malware and misclassified benign apps with expert-written reports, compresses each app into function-level structural summaries, and splits auditing into four staged tasks that mirror analyst work. Seven widely used models are tested. The central result is that none is close to reliable: the strongest model reaches only 50.67 percent on the composite Workload Reduction Score, and the pattern of failures is consistent—models lean on surface cues, cannot fuse scattered facts into coherent attack chains, and change their judgments with context formulation. If this is right, security operations centers cannot yet delegate evidence-backed malware explanations to LLMs without substantial human oversight.

What carries the argument

The load-bearing object is the context-driven intermediate structural representation, written $\mathrm{IR}(f)=\langle N(f), D_c(f), S(f)\rangle$: for every reachable function $f$, the signature $N(f)$, a summary $D_c(f)$ produced by the LLM from the function's one-hop caller–callee neighborhood, and a risk score $S(f)$. This representation is what makes traceable evaluation possible. Static reachability analysis from manifest and framework entrypoints first cuts the code space, then these structural units feed four staged tasks—function prioritization, evidence attribution, behavior synthesis, and sample discrimination—whose scores are combined into the Workload Reduction Score (WRS). The representation's role is to decouple audit judgments from raw, noisy codebases while keeping each claim anchored to a named function, which is what allows the benchmark to measure evidence grounding rather than free-form summarization.

What would settle it

Re-run the four tasks with identical, externally authored function summaries and risk scores for every model, keeping all other settings fixed. If the model rankings on the Workload Reduction Score change materially, the reported cross-model differences are driven by summarization quality rather than by auditing ability; if the rankings hold, the benchmark measures what it claims.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that current LLMs fail at exactly the reasoning that auditing requires. Given a context-driven structural representation of each function—signature, summary, and risk score—models can pick out some sensitive APIs and produce fluent narratives, but they rarely connect low-level evidence to the few decisive functions and behaviors that justify a malicious verdict. The paper reports that even the strongest of the seven evaluated models obtains a Workload Reduction Score of only 50.67 percent, and that behavior-level deviations are systematic: API-driven behaviors are over-observed, while reasoning-intensive behaviors such as privilege escalation and stealth are under-observed. Removing inter-procedural context lowers fidelity scores sharply, while adding application metadata helps some models but risks dropping true positives. The conclusion the paper draws is that LLMs are currently weak at verifiable, compositional evidence reasoning, so progress in malware auditing will require better-grounded and agentic workflows rather than isolated model outputs.

Load-bearing premise

Each evaluated model generates the function summaries and risk scores that later become its own test inputs, so a model that summarizes or scores poorly receives worse inputs and its audit score is depressed for reasons unrelated to auditing ability.

Editorial extensions

If this is right

  • If MalEval's results hold, current LLMs should not be the final authority in malware auditing; their reports need analyst verification, especially on evidence attribution and behavior synthesis.
  • The four-stage decomposition lets audit tools localize where an LLM fails, so only reliable stages—such as sensitive-API coverage—can be automated while human review remains on attack-chain reasoning.
  • Inter-procedural context is necessary for pinpointing decisive functions, since fidelity scores drop sharply when it is removed; future auditing prompts should preserve call relations.
  • Adding application metadata is not uniformly helpful: it can improve false-positive correction but can lower true-positive maintenance, and missing one malware sample can be catastrophic.
  • The framework's components—verifiable ground truth, context reduction, and structural representations—are platform-agnostic, so the evaluation can be adapted to Windows, Linux, and IoT malware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a direct next step the paper leaves implicit is to force models to cite only actual signatures and to require cited functions to form connected call-graph paths; if the paper's diagnosis is right, this should improve evidence attribution and behavior synthesis more than scaling model size.
  • Editorial inference: the under-observation of privilege escalation and stealth across all models points to a shared inductive bias; minimal synthetic apps that isolate one behavior at a time could test whether models recognize these behaviors at all when API cues are neutral.
  • Editorial inference: the four stage-wise tasks double as training objectives, since each stage has a verifiable target; an auditing model could be fine-tuned with explicit evidence-grounding rewards rather than judged only on its final report.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. MalEval is presented as a diagnostic benchmark for fine-grained Android malware behavior auditing with LLMs. The paper curates 222 archived and 33 recent malware samples plus 25 recent benign apps with manually verified behavior reports and a 20,337-entry sensitive API list, reduces each app to reachable functions via static analysis, and represents functions as triples <signature, context-driven summary, sensitivity score>. It defines four analyst-aligned tasks—function prioritization (fidelity score), evidence attribution (CSR), behavioral synthesis (RQ/EAS/SAS), and sample discrimination (FPCR/TPMR/F1_c)—and aggregates them into a Workload Reduction Score (WRS). Seven LLMs are evaluated. The headline result is that the best model, Claude-3.7-sonnet, reaches only 50.67% WRS, with qualitative evidence that models rely on surface API cues, miss multi-step attack chains, and are sensitive to context formulation.

Significance. If the measurement pipeline is valid, MalEval would be a useful contribution: it provides a publicly released, reproducible benchmark with expert-verified ground truth at behavior and API level, decomposes auditing into separately diagnosable stages, and includes useful ablations (context removal in §4.3, metadata addition in §4.4) and a detailed failure-case analysis. The strongest parts are the dataset construction effort, the four-task decomposition, and the qualitative behavior-deviation heatmaps, which are less dependent on the judge model. However, the headline quantitative results (WRS values and cross-model rankings) currently rest on an unvalidated LLM-as-a-judge and on self-generated per-model inputs, so the quantitative claims should be treated as conditional until these are addressed.

major comments (4)
  1. [Section 3.5.3, Eq. (3)] RQ and EAS are assigned by GPT-5-mini as an LLM-as-a-judge (LLM Operations III and IV) with no human agreement study, inter-annotator statistics, or bias checks reported in Section 4. Because WRS in Eq. (3) multiplies RQ × EAS × SAS, the headline numbers in Table 2 (e.g., Claude 50.67%) and the rankings behind the 'surface cues' conclusion are partly opinions of one unvalidated judge. Please provide a calibration study (at least Pearson/Spearman correlation and per-criterion agreement against two or more human malware analysts on a sample of reports) and a check for judge bias toward fluent or confident wording.
  2. [Section 3.4 (LLM Operation I)] Each evaluated model generates its own context-driven summaries D_c(f) and sensitivity scores S(f), and those self-generated representations are then used as inputs to Tasks 1–4. The claim in §3.4 that this design 'ensures consistency across models' is not supported: a model that summarizes poorly incurs downstream penalties in FS, RQ, EAS, and SAS that are unrelated to its auditing ability, so the cross-model comparisons in Tables 2–4 are confounded. Please either evaluate all models on a fixed, model-independent intermediate representation or report both self-generated and fixed-input conditions.
  3. [Section 3.5.1, Eq. (1)] The Fidelity Score is computed by removing the top-10% most sensitive functions and measuring the confidence drop of an XGBoost classifier trained on TF-IDF features of function summaries, but the manuscript does not state whose summaries are used for training, nor does it report the classifier's accuracy or the stability of FS under different removal thresholds. Since FS differences drive the claim in §4.3 that inter-procedural context is indispensable, please report classifier accuracy and a threshold sweep.
  4. [Section 3.5.3, SAS definition] SAS is a syntactic existence ratio (|F_s|/|F_in|): it is satisfied by citing any function that appears in the input, regardless of whether that function supports the claimed behavior. The paper's conclusion that reports are 'verifiable' therefore rests on EAS, which is itself an unvalidated judge score. Please strengthen the grounding check (e.g., require cited functions to appear in the evidence chain for the behavior) or soften the verifiability claim.
minor comments (5)
  1. [Figures 1 and 2] Figure 1 contains the typo 'extreeAPK size' and Figure 2 contains 'Behevaior Report'; these should be corrected.
  2. [Section 3.2.2] The ground-truth generation pipeline relies on an unspecified 'large language model' to parse vendor reports and to expand the sensitive API list; please name the model and version and describe any manual verification of the parsed output.
  3. [Section 4.5.2] The archived-versus-latest malware comparison uses Welch's t-test on very small subsets (33 latest samples across 3 categories); the statistical power is limited and the conclusions should be phrased more cautiously.
  4. [Eq. (5) and Section 4.1.2] WRS weights w_aa and w_ds are set equal without sensitivity analysis; please show WRS rankings under alternative weightings to demonstrate that the main conclusions are not weight-dependent.
  5. [Section 4.5.3] The heading 'Failure Case I' appears twice in the subsection; please renumber or merge the duplicated headings for clarity.

Circularity Check

1 steps flagged · score 5.0 of 10

EAS authenticity is scored against the model's own generated structural representation, so part of the headline WRS measures self-consistency rather than code-grounded verifiability.

  1. self definitional [Section 3.4 (LLM Operation I) and Section 3.5.3 (LLM Operation IV / EAS)]
    "An evaluated language model first summarizes the functionalities of these neighbors. Then, combining them with the function’s source code and signature N(f), yields D_c(f) ... IR(f)=<N(f),D_c(f),S(f)>. [EAS] Input: (1) The list of behaviors in the generated report; (2) the structural representations IR(f) of functions, where f∈F_s and F_s is the set of support functions identified for the application."

    EAS is introduced as the hallucination guard: it is supposed to assess whether cited functions genuinely support the claimed behaviors. But the only function-level evidence the judge receives is the structural representation IR(f), whose context summary D_c(f) and sensitivity score S(f) were produced by the very LLM under evaluation in LLM Operation I. The judge therefore checks the report's behavior claims against the model's own summary of the cited function, not against the decompiled source code. A model that writes a fluent but false D_c(f) will receive high EAS as long as its report is internally consistent with that false summary. Thus EAS—and, through Eq. 3, AA and WRS—measures self-coherence by construction, not code-grounded authenticity.

full rationale

MalEval's headline WRS is not entirely circular: SAS is a string-matching check against the input function list, FPCR/TPMR are binary classification rates against expert ground truth, F1_c is category accuracy, and the qualitative failure cases compare generated reports directly with expert-written ground truth. No fitted parameter is renamed as a prediction, and the self-citations to CAMA/LAMD are used for design motivation and critique rather than as the sole support of the central conclusion. However, one load-bearing metric is self-referential: EAS is described as verifying that cited functions genuinely support the claimed behaviors, yet its input is the structural representation IR(f) whose D_c(f) and S(f) were generated by the evaluated LLM itself. The judge therefore verifies the report against the model's own summary of the function, not against the decompiled code, so a confidently hallucinated summary will pass EAS whenever the report is self-consistent. Since EAS enters Analytical Accuracy and WRS multiplicatively, the headline 50.67% WRS and the conclusion that models fail to ground evidence are partly statements about self-consistency. The additional fact that each model generates its own D_c(f) and S(f) inputs, and that RQ/EAS are unvalidated LLM-judge opinions without human agreement, further threatens cross-model comparisons, but that is a validity threat rather than a by-construction reduction. Overall: partial circularity in the authenticity metric, with independent support from objective metrics, so a moderate score of 5 is appropriate.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central numbers rest on assumptions that static reachability, one-hop context, and API-list evidence capture the behavior-relevant code, and that LLM-produced ground truth and judge scores are trustworthy. These are reasonable for a first benchmark but should be stated and stress-tested.

free parameters (3)
  • WRS weights w_aa and w_ds = 0.5 and 0.5 (equal weights)
    The composite Workload Reduction Score depends on the relative weighting of Analytical Accuracy and Decision Support; the paper sets them equal without sensitivity analysis, so model rankings by WRS could shift under other weights.
  • Fidelity top-k removal threshold = 10% (top-k = 10% of functions)
    Task 1's Fidelity Score is computed after removing the top 10% most sensitive functions; the threshold is chosen by hand and affects FS values and model ordering.
  • RQ criterion weights (insight, comprehensiveness, evidence) = Equal weights
    Report Quality is a weighted average of three judge-scored criteria; the weights are set equal in Section 4.1.2, which is an arbitrary normalization for the headline report-quality numbers.
assumptions (5)
  • domain assumption Static reachability analysis from manifest and framework entrypoints captures all behavior-relevant code.
    Section 3.3 reduces each APK to reachable functions; malware that uses reflection, dynamic code loading, or native code can be missed, which would make the benchmark incomplete for such samples.
  • domain assumption One-hop caller-callee context is sufficient to represent a function's behavioral role.
    Section 3.4 builds IR(f) only from the immediate call graph neighborhood; the paper's own Failure Case I notes that global attack-chain context is missing, so this axiom limits what the benchmark can measure.
  • domain assumption The curated sensitive API list of 20,337 APIs is a complete and accurate proxy for low-level malicious evidence.
    Section 3.2.2 builds Task 2 ground truth from this list; legal-API abuse is a known evasion technique and appears as Failure Case IV, so basing the evidence-attribution ground truth on API lists systematically favors API-driven malware.
  • domain assumption Vendor behavior reports from Malradar and Malpedia are accurate ground truth, and LLM-based parsing preserves them.
    Section 3.2.2 parses vendor reports with an LLM and claims manual verification, but no inter-annotator or human-validation details are given; errors in the reports propagate to all task metrics.
  • ad hoc to paper An LLM judge (GPT-5-mini) can score report quality and evidence authenticity without human calibration.
    Section 3.5.3 adopts LLM-as-a-judge with a model excluded from the evaluation pool; no validation against human expert scores is reported, so RQ and EAS numbers embed the judge's biases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing." pith.science (2026). https://pith.science/paper/7P4YCMQS

@misc{pith2026250914335,
  author       = {Pith},
  title        = {Pith review of: Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7P4YCMQS}},
  note         = {Machine review of arXiv:2509.14335}
}
read the original abstract

Automated malware classifiers achieve strong detection performance, but auditing requires more than flagging a sample: analysts must explain malicious behaviors and justify them with code evidence. Traditional signature-based methods and learning-based XAI often fail to provide such support in a human-interpretable form. Large Language Models (LLMs) appear promising, yet their reliability for malware auditing remains unclear. Evaluation faces three challenges: (1) the lack of human-written behavioral ground truth; (2) real-world codebases that exceed current context limits; and (3) the lack of reliable mechanisms to verify whether generated claims are grounded in code evidence. These obstacles make benchmarking difficult and leave model capabilities and failure modes opaque. We introduce MalEval, a diagnostic framework for measuring the capability boundaries of LLMs in malware auditing. MalEval pairs real-world application codebases with expert-written audit reports to provide fine-grained behavior-level ground truth. It compresses large codebases into behavior-relevant program contexts through a context-driven intermediate representation that preserves call relations. Expert reports and model outputs are mapped, via constrained reasoning, into structured evidence chains linking code-level facts to high-level behaviors in a shared space. MalEval decomposes auditing into four stage-wise tasks, enabling each intermediate judgment to be verified under limited context windows. We evaluate seven LLMs and find that they rely on surface cues rather than verifiable evidence, struggle to compose dispersed facts into coherent attack chains, and are highly sensitive to context formulation. These findings shift attention from isolated outputs to reliable LLM and agentic workflows for malware auditing. MalEval is publicly available at https://github.com/ZhengXR930/MalEval.git

Figures

Figures reproduced from arXiv: 2509.14335 by the authors.

Figure 1
Figure 1. Evaluation pipeline of MalEval. benign samples can be correctly refuted without undermining valid judgments. Together with task-specific and global metrics, these components yield a reproducible and fine-grained evaluation pro￾cess that closely reflects real-world malware auditing. 3.2 Verifiable Ground Truth Generation 3.2.1 Dataset Curation. To construct a benchmark that reflects re￾alistic auditing conditions, we… view at source ↗
Figure 2
Figure 2. Metrics for Each Evaluation Task. LLM Operation III: Report Quality (RQ) Judgement Task: Evaluate the quality of a generated report by comparing it against a ground-truth report. Input: A report generated by a specific LLM and its corresponding parsed ground-truth report. Output: A set of scores, Insight (𝑠𝑖 ), Comprehensiveness (𝑠𝑐 ), and Evidence Quality (𝑠𝑒 ), with supporting justifications. Report Quality is the… view at source ↗
Figure 3
Figure 3. Behavior deviation heatmaps for each LLM trained on malware datasets. Here, shades closer to blue indicate the [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 38 canonical work pages

  1. [1]

    Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Uni- fied Pre-training for Program Understanding and Generation. InProceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies. 2655–2668

    work page 2021

  2. [2]

    Anthropic. 2025. Claude 3.7 Sonnet and Claude Code. https://www.anthropic. com/news/claude-3-7-sonnet/. Accessed: 2025-05-10

    2025

  3. [3]

    Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad Rieck, and CERT Siemens. 2014. Drebin: Effective and explainable detection of android malware in your pocket.. InNdss, Vol. 14. 23–26

    2014

  4. [4]

    Steven Arzt, Siegfried Rasthofer, Christian Fritz, Eric Bodden, Alexandre Bar- tel, Jacques Klein, Yves Le Traon, Damien Octeau, and Patrick McDaniel. 2014. Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps.ACM sigplan notices49, 6 (2014), 259–269

    work page 2014

  5. [5]

    Kathy Wain Yee Au, Yi Fan Zhou, Zhen Huang, and David Lie. 2012. Pscout: analyzing the android permission specification. InProceedings of the 2012 ACM conference on Computer and communications security. 217–228

    work page 2012

  6. [6]

    Federico Barbero, Feargus Pendlebury, Fabio Pierazzi, and Lorenzo Cavallaro

  7. [7]

    Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2024. Reasoning runtime behavior of a program with llm: How far are we?arXiv preprint arXiv:2403.16437(2024)

    arXiv 2024

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

    arXiv 2021

Show all 65 references
  1. [9]

    Qian Chen and Robert A Bridges. 2017. Automated behavioral analysis of mal- ware: A case study of wannacry ransomware. In2017 16th IEEE International Conference on machine learning and applications (ICMLA). IEEE, 454–460

    2017

  2. [10]

    Yizheng Chen, Zhoujie Ding, and David Wagner. 2023. Continuous Learning for Android Malware Detection. In32nd USENIX Security Symposium (USENIX Security 23). USENIX Association, Anaheim, CA, 1127–1144. https://www.usenix. org/conference/usenixsecurity23/presentation/chen-yizheng

    2023

  3. [11]

    2025.Gemini 2.5 Flash Preview: Model Card

    Google DeepMind. 2025.Gemini 2.5 Flash Preview: Model Card. Technical Report. Google. Accessed: 2025-05-10

    2025

  4. [12]

    Evan Downing, Yisroel Mirsky, Kyuhong Park, and Wenke Lee. 2021. {DeepReflect}: Discovering malicious functionality through binary reconstruc- tion. In30th USENIX Security Symposium (USENIX Security 21). 3469–3486

    2021

  5. [13]

    Manuel Egele, Theodoor Scholte, Engin Kirda, and Christopher Kruegel. 2008. A survey on automated dynamic malware-analysis techniques and tools.ACM computing surveys (CSUR)44, 2 (2008), 1–42

    2008

  6. [14]

    William Enck, Peter Gilbert, Seungyeop Han, Vasant Tendulkar, Byung-Gon Chun, Landon P Cox, Jaeyeon Jung, Patrick McDaniel, and Anmol N Sheth. 2014. Taintdroid: an information-flow tracking system for realtime privacy monitoring on smartphones.ACM Transactions on Computer Syst...

    2014

  7. [15]

    Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick McDaniel. 2017. Adversarial examples for malware detection. InComputer Security–ESORICS 2017: 22nd European Symposium on Research in Computer Security, Oslo, Norway, September 11-15, 2017, Proceed...

    2017

  8. [16]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al . 2024. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594(2024)

    2024 arXiv

  9. [17]

    Wenbo Guo, Dongliang Mu, Jun Xu, Purui Su, Gang Wang, and Xinyu Xing. 2018. Lemna: Explaining deep learning based security applications. Inproceedings of the 2018 ACM SIGSAC conference on computer and communications security. 364–379

    2018

  10. [18]

    Ariful Haque, Sunzida Siddique, Md Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, and Kishor Datta Gupta. 2025. SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software De- velopment with Insights for LLM Deployment.arXiv prepri...

    2025 arXiv

  11. [19]

    Yiling He, Junchi Lei, Zhan Qin, Kui Ren, and Chun Chen. 2025. Combating Concept Drift with Explanatory Detection and Adaptation for Android Malware Classification. InProceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security. 1

    2025

  12. [20]

    Yiling He, Yiping Liu, Lei Wu, Ziqi Yang, Kui Ren, and Zhan Qin. 2023. MsDroid: Identifying Malicious Snippets for Android Malware Detection.IEEE Transactions on Dependable and Secure Computing20, 3 (2023), 2025–2039

    2023

  13. [21]

    Yiling He, Jian Lou, Zhan Qin, and Kui Ren. 2023. Finer: Enhancing state-of-the- art classifiers with feature attribution to facilitate security analysis. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security. 416–430

    2023

  14. [22]

    Yiling He, Hongyu She, Xingzhi Qian, Xinran Zheng, Zhuo Chen, Zhan Qin, and Lorenzo Cavallaro. 2025. On benchmarking code llms for android malware analysis. InProceedings of the 34th ACM SIGSOFT International Symposium on Software Testing and Analysis. 153–160

    2025

  15. [23]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79

    2024

  16. [24]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

    2024 arXiv

  17. [25]

    Yigitcan Kaya, Yizheng Chen, Marcus Botacin, Shoumik Saha, Fabio Pierazzi, Lorenzo Cavallaro, David Wagner, and Tudor Dumitraş. 2025. ML-Based Behav- ioral Malware Detection Is Far From a Solved Problem. In2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)...

    2025

  18. [26]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princip...

    2023

  19. [27]

    Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023. Loogle: Can long-context language models understand long contexts?arXiv preprint arXiv:2311.04939(2023)

    2023 arXiv

  20. [28]

    Yao Li, Sen Fang, Tao Zhang, and Haipeng Cai. 2024. Enhancing Android Malware Detection: The Influence of ChatGPT on Decision-centric Task.arXiv preprint arXiv:2410.04352(2024)

    2024 arXiv

  21. [29]

    Keke Lian, Bin Wang, Lei Zhang, Libo Chen, Junjie Wang, Ziming Zhao, Yujiu Yang, Haotong Duan, Haoran Zhao, Shuang Liao, et al. 2025. ASE: A Repository- Level Benchmark for Evaluating Security in AI-Generated Code.arXiv preprint arXiv:2508.18106(2025)

    2025

  22. [30]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    2024 arXiv

  23. [31]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambro- sio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al . 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664(2021)

    2021 arXiv

  24. [32]

    Niall McLaughlin, Jesus Martinez del Rincon, BooJoong Kang, Suleiman Yerima, Paul Miller, Sakir Sezer, Yeganeh Safaei, Erik Trickel, Ziming Zhao, Adam Doupé, et al. 2017. Deep android malware detection. InProceedings of the seventh ACM on conference on data and application sec...

    2017

  25. [33]

    Meta AI. 2024. Introducing Llama 3.1: Our most capable models to date. https: //ai.meta.com/blog/meta-llama-3-1/. Accessed: 2025-09-08

    2024

  26. [34]

    Serge Lionel Nikiema, Jordan Samhi, Abdoul Kader Kaboré, Jacques Klein, and Tegawendé F Bissyandé. 2025. The Code Barrier: What LLMs Actually Under- stand?arXiv preprint arXiv:2504.10557(2025)

    2025 arXiv

  27. [35]

    OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence. https://openai. com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/. Accessed: 2025- 09-08

    2024

  28. [36]

    OpenAI. 2025. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/. Accessed: 2025-09-09

    2025

  29. [37]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318

    2002

  30. [38]

    Xingzhi Qian, Xinran Zheng, Yiling He, Shuo Yang, and Lorenzo Cavallaro. 2025. LAMD: Context-Driven Android Malware Detection and Classification with LLMs. In2025 IEEE Security and Privacy Workshops (SPW). IEEE, 126–136

    2025

  31. [39]

    Monoshi Kumar Roy, Simin Chen, Benjamin Steenhoek, Jinjun Peng, Gail Kaiser, Baishakhi Ray, and Wei Le. 2025. CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning.arXiv preprint arXiv:2506.00750(2025)

    2025

  32. [40]

    Adam Štorek, Mukur Gupta, Samira Hajizadeh, Prashast Srivastava, and Suman Jana. 2025. Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Reasoning.arXiv preprint arXiv:2505.13353(2025)

    2025 arXiv

  33. [41]

    Bo Sun, Akinori Fujino, and Tatsuya Mori. 2016. Poster: Toward automating the generation of malware analysis reports using the sandbox logs. InProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. 1814–1816

    2016

  34. [42]

    Tiezhu Sun, Marco Alecci, Aleksandr Pilgun, Yewei Song, Xunzhu Tang, Jor- dan Samhi, Tegawendé F Bissyandé, and Jacques Klein. 2025. MalLoc: Toward Fine-grained Android Malicious Payload Localization via LLMs.arXiv preprint arXiv:2508.17856(2025)

    2025 arXiv

  35. [43]

    Kimberly Tam, Aristide Fattori, Salahuddin Khan, and Lorenzo Cavallaro. 2015. Copperdroid: Automatic reconstruction of android malware behaviors. InNDSS Symposium 2015. 1–15

    2015

  36. [45]

    Brandon J Walton, Mst Eshita Khatun, James M Ghawaly, and Aisha Ali-Gombe

  37. [46]

    Chengpeng Wang, Wuqi Zhang, Zian Su, Xiangzhe Xu, Xiaoheng Xie, and Xi- angyu Zhang. 2024. LLMDFA: analyzing dataflow in code with large language models.Advances in Neural Information Processing Systems37 (2024), 131545– 131574

    2024

  38. [47]

    Liu Wang, Haoyu Wang, Ren He, Ran Tao, Guozhu Meng, Xiapu Luo, and Xuanzhe Liu. 2022. MalRadar: Demystifying android malware in the new era.Proceedings of the ACM on Measurement and Analysis of Computing Systems6, 2 (2022), 1–27

    2022

  39. [48]

    Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can llms replace human evaluators? an empirical study of llm-as-a- judge in software engineering.Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1955–1977

    2025

  40. [49]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. CodeT5: Identifier- aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 8696–8708

    2021

  41. [50]

    Bozhi Wu, Sen Chen, Cuiyun Gao, Lingling Fan, Yang Liu, Weiping Wen, and Michael R Lyu. 2021. Why an android app is classified as malware: Toward malware classification interpretation.ACM Transactions on Software Engineering and Methodology (TOSEM)30, 2 (2021), 1–29

    2021

  42. [51]

    Danning Xie, Mingwei Zheng, Xuwei Liu, Jiannan Wang, Chengpeng Wang, Lin Tan, and Xiangyu Zhang. 2025. CORE: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks.arXiv preprint arXiv:2507.05269(2025)

    2025

  43. [52]

    2024.{DVa}: Extracting Victims and Abuse Vectors from Android Accessibility Malware

    Haichuan Xu, Mingxuan Yao, Runze Zhang, Mohamed Moustafa Dawoud, Jeman Park, and Brendan Saltaformaggio. 2024.{DVa}: Extracting Victims and Abuse Vectors from Android Accessibility Malware. In33rd USENIX Security Symposium (USENIX Security 24). 701–718

    2024

  44. [53]

    2012.{DroidScope}: Seamlessly reconstructing the{OS} and dalvik semantic views for dynamic android malware analysis

    Lok Kwong Yan and Heng Yin. 2012.{DroidScope}: Seamlessly reconstructing the{OS} and dalvik semantic views for dynamic android malware analysis. In 21st USENIX security symposium (USENIX security 12). 569–584

    2012

  45. [54]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    2025 arXiv

  46. [55]

    Fan Yang, Mengnan Du, and Xia Hu. 2019. Evaluating explanation without ground truth in interpretable machine learning.arXiv preprint arXiv:1907.06831 (2019)

    2019 arXiv

  47. [56]

    Limin Yang, Zhi Chen, Chenkai Wang, Zhenning Zhang, Sushruth Booma, Phuong Cao, Constantin Adam, Alexander Withers, Zbigniew Kalbarczyk, Ravis- hankar K Iyer, et al. 2024. True attacks, attack attempts, or benign triggers? an empirical measurement of network alerts in a securi...

    2024

  48. [57]

    Miuyin Yong Wong, Matthew Landen, Manos Antonakakis, Douglas M Blough, Elissa M Redmiles, and Mustaque Ahamad. 2021. An inside look into the prac- tice of malware analysis. InProceedings of the 2021 ACM SIGSAC conference on computer and communications security. 3053–3069

    2021

  49. [58]

    Mu Zhang, Yue Duan, Qian Feng, and Heng Yin. 2015. Towards automatic generation of security-centric descriptions for android apps. InProceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. 518–529

    2015

  50. [59]

    Mu Zhang, Yue Duan, Heng Yin, and Zhiruo Zhao. 2014. Semantics-aware android malware classification using weighted contextual api dependency graphs. In Proceedings of the 2014 ACM SIGSAC conference on computer and communications security. 1105–1116

    2014

  51. [60]

    Xinran Zheng, Shuo Yang, Edith CH Ngai, Suman Jana, and Lorenzo Cavallaro

  52. [61]

    Huijuan Zhu, Xilong Chen, Liangmin Wang, Zhicheng Xu, and Victor S Sheng

  53. [64]

    Learning temporal invariance in android malware detectors.arXiv preprint arXiv:2502.05098(2025)

    2025 arXiv

  54. [66]

    IEEE Transactions on Knowledge and Data Engineering(2024)

    A dynamic analysis-powered explanation framework for malware detection. IEEE Transactions on Knowledge and Data Engineering(2024)

    2024

  55. [2022]

    In2022 IEEE Symposium on Security and Privacy (SP)

    Transcending transcend: Revisiting malware classification in the presence of concept drift. In2022 IEEE Symposium on Security and Privacy (SP). IEEE, 805–823

  56. [2024]

    of android malware

    Exploring large language models for semantic analysis and categorization Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Trovato et al. of android malware. In2024 Annual Computer Security Applications Conference Workshops (ACSAC Workshops). IEEE, 248–254

    2018

  57. [2025]

    Exploring Large Language Models for Semantic Analysis and Categorization of Android Malware.arXiv preprint arXiv:2501.04848(2025)

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.