{"id":"d21eea0c-19fc-4ba3-bff0-be78991ab676","arxiv_id":"2501.04848","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MalParse, a hierarchical LLM-based summarizer, categorizes Android apps as malicious or benign with 77% accuracy and can trace its verdicts to specific functions.","lead":"The paper tests whether a general-purpose large language model, GPT-4o-mini, can summarize decompiled Android apps layer by layer and flag malware. It reports 77% accuracy in telling malicious from benign apps when the prompts are tuned with malware-specific context, and shows a worked example tracing a summary back to the code that performs rooting.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 77% headline accuracy is not reproducible from the paper's own confusion matrices: all three tables have impossible row sums, and Vanilla accuracy is reported as 49.5%, 45%, and 45.5% in different sections.","rationale":"The reader's weakest_assumption about decompilation is plausible and explicitly acknowledged in Section 4.3.1, but it does not directly challenge the arithmetic of the reported evaluation. The confusion-matrix inconsistencies are concrete, checkable, and located in the only evidence for the headline 77% figure. If the corrected matrices do not reproduce the stated 154/200, the central quantitative claim fails; if they do, the issue is a presentation flaw. Because the paper's contribution is directionally sound and the errors are addressable by releasing per-sample results, a conditional acceptance remains appropriate. This does not require a change in verdict, hence UNCHANGED.","tokens_in":9441,"tokens_out":4914,"duration_ms":42880,"concrete_test":"Obtain the per-sample categorization outputs for all 200 APKs across the three prompt scopes (or recompute from the pipeline if code/data are released) and reconstruct the confusion matrices. Verify that each row sums to 100% and that the total correct for Malware-Scoped equals 154/200. If the corrected matrices reproduce the stated TP/TN counts (76 and 78), the concern is a reporting error and the headline stands; if the counts differ, the 77% figure is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2.1 contains the only quantitative support for the central claim, but the confusion matrices are internally inconsistent. In Table 1 (Vanilla), the 'Benign Actual' row lists 92% TP and 93% FN, summing to 185%; the 'Malware Actual' row lists 8% FP and 7% TN, summing to 15%. The text states 99/200 (49.5%) correct, which requires TP=92 and TN=7, so the off-diagonal entries should be 8% and 93%, not 93% and 8%. Table 2 (API-Scoped) has row sums of 168% and 32%; the stated 112/200 (56%) requires TP=90 and TN=22, so the listed 78% FN and 10% FP are swapped. Table 3 (Malware-Scoped) has row sums of 98% and 102%; the stated 154/200 (77%) requires TP=76 and TN=78, so the listed 22% FN and 24% FP are swapped. Further, Vanilla accuracy is given as 49.5% in the Introduction and Section 4.2.1.1, but as 45% in Section 4.2.1.2 and 45.5% in Section 4.2.1.3. Because the 77% figure is the central claim, and the only evidence for it is these tables, the inconsistency means the published numbers cannot be verified as stated. The decompilation limitation identified by the reader is real but acknowledged; the arithmetic inconsistency is not acknowledged and directly undermines the headline result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MalParse, a pipeline for Android malware analysis that decompiles APKs to Java source using Dex2Jar and CFR, generates hierarchical function-, class-, and package-level summaries with GPT-4o-mini under three prompt scopes (Vanilla, API-Scoped, Malware-Scoped), and uses the final package summary to classify each app as benign or malicious. On a balanced set of 200 apps (100 VirusTotal malware, 100 top Similarweb apps), the authors report 49.5% accuracy with Vanilla prompts, 56% with API-Scoped prompts, and 77% with Malware-Scoped prompts, and illustrate that backward tracing from package-level summaries can locate the code responsible for malicious behavior (e.g., a rooting payload in the RTAccessHandler class). The paper also discusses limitations, chiefly decompilation quality, obfuscation, and prompt bias.","tokens_in":9695,"tokens_out":13943,"duration_ms":125201,"significance":"If the reported figures are correct after revision, the paper makes a modest but useful empirical contribution: it shows that a general-purpose LLM with carefully engineered prompts can produce human-readable, hierarchical code summaries for Android apps and perform coarse benign/malicious triage without fine-tuning, which could help accelerate triage of novel samples. The hierarchical summarization design is a pragmatic answer to LLM context-window limits, and the three-way prompt comparison is a reasonable internal ablation. The main weaknesses are the small and not fully documented dataset, the absence of conventional static-analysis baselines, and the fact that the central accuracy numbers are not internally consistent as printed.","major_comments":[{"comment":"All three confusion matrices are arithmetically impossible as printed. In Table 1, the 'Benign Actual' row sums to 185% (TP=92%, FN=93%) and the 'Malware Actual' row sums to 15% (FP=8%, TN=7%); the stated 49.5% accuracy (99/200) corresponds to TP=92 and TN=7, which forces FN=8% and FP=93%, i.e., the off-diagonal values are interchanged. In Table 2, the row sums are 168% and 32%; the stated 56% (112/200) forces FN=10% and FP=78% rather than the printed 78% and 10%. In Table 3, the row sums are 98% and 102%; the stated 77% (154/200) forces FN=24% and FP=22% rather than the printed 22% and 24%. Please provide corrected raw contingency counts for all three tables, since the headline accuracy depends directly on these numbers.","section":"Section 4.2.1, Tables 1-3"},{"comment":"The Vanilla prompt accuracy is reported as 49.5% (Introduction and §4.2.1.1), as 45% (§4.2.1.2), and as 45.5% (§4.2.1.3). These three values cannot all be correct, and the text does not explain the discrepancy. Please verify the underlying results and report a single consistent value, with raw counts.","section":"Section 4.2.1 and Introduction"},{"comment":"The statement that the system achieved 'a balanced accuracy at 77% with a precision of 76% and a recall rate of 77%' is not derivable from the corrected Table 3 counts under any standard class convention: with malware as the positive class, precision is approximately 76% but recall is 78%; with benign as positive, recall is 76% and precision is approximately 78%. Please specify the positive class and formulas used, and recompute all derived metrics so that they are mutually consistent.","section":"Section 4.3.1"},{"comment":"The three prompt scopes are evaluated on the same 200-sample set, and the best-performing configuration (Malware-Scoped) is then reported as the system's headline accuracy. Because Section 3.3 describes iterative prompt crafting and Section 4.3.1 states that accuracy improves as prompting is tuned, the 77% figure is an in-sample optimum; no hold-out validation, cross-validation, or repeated sampling (e.g., multiple GPT-4o-mini runs with nonzero temperature) is reported. Please add a validation protocol, report variance across runs, or provide raw per-sample decisions so that the generality of the 77% number can be assessed.","section":"Section 4.2.1, Section 3.3"},{"comment":"The claim that backtracking through the summaries 'allowed us to pinpoint the precise code snippets responsible for malicious behavior' (Abstract) is supported only by a single hand-picked example, the rooting sample in Table 5. Please evaluate the traceability systematically over the correctly classified malware samples (for example, how often the package-level summary leads to the correct malicious class and function), or explicitly restrict the claim to an illustrative case study.","section":"Section 4.2.2"}],"minor_comments":[{"comment":"'Davik Bytecode' should read 'Dalvik Bytecode', and 'decomples' should read 'decompiles'.","section":"Section 3.1"},{"comment":"The text states that 'the names of the APKs were not provided' to the model; since decompiled Java source retains package and class names, the model may still be able to infer the application identity from the code content. Please clarify whether package names were stripped or otherwise anonymized, and how the file names were excluded from the inputs.","section":"Section 4.1"},{"comment":"The acknowledged decompilation failures of Dex2Jar/CFR are not quantified; please report how many of the 200 APKs failed to decompile fully, and, if possible, separate those cases from LLM misclassifications so that the impact on the 77% figure can be understood.","section":"Section 4.3.1"},{"comment":"The literature review would benefit from a discussion of prior Android malware detection efforts that use machine learning or LLMs (beyond code summarization and npm malware detection), so that the novelty of the categorization and hierarchical summarization approach can be positioned more precisely.","section":"Section 2"},{"comment":"The paper repeatedly states that the model was 'not pre-trained on previous malware or benign samples'; this claim about GPT-4o-mini's pretraining data is not verifiable. Please rephrase to 'not fine-tuned on the evaluation dataset' or similar.","section":"Section 4.1 and Abstract"}],"recommendation":"major_revision","confidential_remarks":"The numerical inconsistencies are not limited to a single typo: all three confusion matrices, the Vanilla accuracy value, and the precision/recall values in Section 4.3.1 are internally inconsistent. I recommend asking the authors to provide the raw per-sample predictions or corrected contingency tables as part of the revision; without those, the central 77% claim cannot be independently verified. The traceability claim also needs broader evidence than a single example. The core idea is defensible, so I do not recommend rejection at this stage, but the revision must make the quantitative evidence trustworthy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MalParse is a sensible system: decompile Android APKs, summarize functions → classes → packages with GPT-4o-mini, then ask for a benign/malware label. That hierarchical summary chain for Android malware is a real combination, and the traceability example in Table 5 is the best part — it shows how a package-level verdict can be walked back to a DexClassLoader call. The authors are also upfront that decompilation loses complex classes and that some malware got missed because of it. If the numbers held, this would be a useful triage tool.\n\nBut the numbers don't hold as printed. The stress-test note is right: every confusion matrix has impossible row sums. Vanilla's benign row is 92% TP + 93% FN = 185%; API-scoped's benign row is 90% + 78% = 168%; Malware-scoped sums to 98% and 102%. The stated totals (99, 112, 154 of 200) require different off-diagonal entries than the ones shown. And Vanilla accuracy appears as 49.5%, 45%, and 45.5% in different places. These aren't rounding quibbles; they make the headline 77% unverifiable from the paper. The confusion matrix cells look like the TP/TN and FP/FN labels got swapped inconsistently across tables, which suggests careless copy-paste rather than fabrication, but it's exactly the kind of thing a referee needs fixed before the result is usable.\n\nThe other soft spots are more minor. There's no baseline comparison against a standard static detector, no error bars or repeated runs (LLM output is stochastic), and the best prompt scope was chosen after seeing the test set. The circularity concern is modest — the malware-scoped prompts embed the same taxonomy used to interpret ground truth — but it's inherent to prompt engineering and not disqualifying.\n\nWho's this for: researchers working on LLM-assisted malware triage or code summarization will want the pipeline details and the traceability case study. The 77% figure should be treated as a directionally plausible but currently unsupported estimate. I'd send it to peer review, with a request to fix the tables and provide code/data or at least a corrected confusion matrix. The contribution is worth engaging, but the evaluation needs a cleanup.","headline":"Useful hierarchical summarization pipeline for Android malware triage, but the headline 77% accuracy is unsupported by the paper's own confusion matrices, which do not add up.","tokens_in":10296,"tokens_out":1546,"would_cite":false,"duration_ms":14545,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A general-purpose LLM, prompted but not fine-tuned, can summarize Android apps and flag malware with 77% accuracy, then trace its verdict down to the specific malicious function.","keywords":["Android malware","large language models","code summarization","prompt engineering","malware categorization","static analysis","reverse engineering","hierarchical summarization"],"falsifier":"Take a set of APKs whose malicious code lives in classes that Dex2Jar or CFR demonstrably fail to recover (confirmed by comparing the decompiled class list against the actual dex bytecode), run MalParse's Malware-Scoped pipeline on them, and check whether the classification accuracy drops measurably below 77%; if it stays near 77%, the decompilation-preservation premise is wrong, and if it drops, the premise is confirmed as the load-bearing limitation.","tokens_in":9176,"feed_emoji":"🛡️","tokens_out":4816,"duration_ms":43529,"temperature":0.7,"pith_summary":"The paper claims that a general large language model, with no fine-tuning on malware, can serve as the core of an Android malware analysis pipeline by summarizing an app's code at function, class, and package levels and then labeling the app malicious or benign. With prompts that teach the model what malware behavior looks like, the system correctly classified 154 of 200 apps (77%), while neutral prompts managed only 49.5%. The layered summaries also allow backtracking, so an analyst can move from a package-level verdict to the exact code snippet responsible for the malicious action. If this holds, LLMs could offer fast first-pass triage and explanation for reverse engineers facing unknown or zero-day Android samples.","feed_headline":"Plain LLM flags Android malware with 77% accuracy","feed_subtitle":"MalParse's layered summaries trace each malware verdict down to the exact code snippet, no fine-tuning required.","key_machinery":"The load-bearing mechanism is the hierarchical-tiered summarization chain, a bottom-up pipeline that compresses the large codebase of an APK into compact, security-relevant summaries at function, class, and package granularity, allowing a model with a finite context window to process an entire application. It is driven by three prompt scopes (Vanilla, API-Scoped, and Malware-Scoped) that inject progressively more security knowledge into the model; the Malware-Scoped prompts, which list suspicious behaviors such as dynamic class loading, rooting, privilege escalation, and data exfiltration, are what push accuracy from 49.5% to 77%. The chain also provides explainability, because each summary level is derived from the level below, so tracing a package-level claim back to the code recovers the root cause.","core_discovery":"MalParse, built on GPT-4o-mini, decompiles an Android APK into Java source and runs a bottom-up hierarchical summarization chain: each function is summarized, function summaries are merged into class summaries, and class summaries are merged into a package summary. By injecting malware-specific context through what the authors call Malware-Scoped prompting, the pipeline reaches a balanced classification rate of 77%, correctly identifying 76% of benign apps and 78% of malware samples, all without any training on malware or benign examples and without revealing APK names to the model. Backtracking from the package summary through class and function summaries lets the system pinpoint the precise code snippet responsible, demonstrated on a rooting malware sample where the function using a DexClassLoader is identified. The paper presents this as evidence that prompt engineering alone can give a general LLM meaningful semantic understanding of malicious Android code.","pith_inferences":["The 77% figure is dataset-specific and should be read as an upper bound; a deployment would need validation on a much larger and more diverse APK corpus before relying on that rate.","Although the paper hides APK names to prevent the model from relying on memory, the decompiled Java itself may still contain package or class identifiers that leak app identity to the LLM's pretraining, making the 'no prior knowledge' claim less clean than stated.","A natural extension is to run the same pipeline on obfuscated and packed malware, where static decompilation is known to degrade; the paper's own limitation section predicts accuracy will drop, turning this weakness into a testable prediction.","The backtracking capability could be formalized as an evidence requirement: if every package-level verdict had to cite at least one function-level snippet, then an analyst's trust in a verdict could be quantified by checking whether that snippet is actually malicious."],"forward_implications":["Without fine-tuning, a general LLM can achieve 77% balanced accuracy on a 100-malware/100-benign Android set, suggesting that zero-day triage may not require retraining for every new malware family.","Prompt context matters more than model specialization: moving from generic to malware-scoped prompts raised accuracy from 49.5% to 77%, making prompt design an effective control knob in LLM-based security analysis.","Analysts can use the hierarchical summaries as a navigation map, reading the package-level verdict and then drilling into classes and functions to see the specific code that triggered it, which shortens manual reverse engineering.","Because only the prompt carries the malware knowledge, expanding the Malware-Scoped prompt's list of suspicious behaviors could directly reduce the 22% false-negative rate, as the paper itself hypothesizes.","The same summarization chain could be applied to any LLM, not just GPT-4o-mini, providing a baseline to evaluate cheaper or open-source models on the same malware analysis task."],"supporting_citations":[{"why":"Dex2Jar converts APK dex bytecode to Java bytecode, the necessary first transformation in the decompilation pipeline whose output feeds all summarization.","marker":"[12]"},{"why":"CFR decompiles the Java bytecode into human-readable Java source, which is the input analyzed by the hierarchical summarization chain.","marker":"[13]"},{"why":"Provides the iterative text summarization strategy that motivates MalParse's bottom-up approach for large contexts.","marker":"[14]"},{"why":"Supports the claim that iterative, chain-of-thought summarization produces expert-aligned output, underpinning the design of the package-level summary.","marker":"[15]"},{"why":"LangChain is used to implement the custom LLM chains and prompt templates that drive the three prompt scopes.","marker":"[16]"},{"why":"Establishes a prior baseline for LLM-based malware detection, in the npm ecosystem, that MalParse extends to Android with a hierarchical approach.","marker":"[10]"}],"fun_headline_variants":["LLM pinpoints Android malware code without fine-tuning","Hierarchical LLM summaries trace Android malware to source","77% accuracy: LLM categorizes Android malware, no training","MalParse LLM finds exact malicious code in Android APKs","From APK to code: LLM maps Android malware precisely"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole chain assumes that turning an APK into Java source with Dex2Jar and CFR preserves enough of the app's behavior that the LLM's summaries reflect what the malware actually does.","fun_headline_variants_meta":{"raw":{"variants":["LLM pinpoints Android malware code without fine-tuning","Hierarchical LLM summaries trace Android malware to source","77% accuracy: LLM categorizes Android malware, no training","MalParse LLM finds exact malicious code in Android APKs","From APK to code: LLM maps Android malware precisely"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1504,"prompt_tokens":958,"completion_tokens":546,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":463}},"tokens_in":574,"tokens_out":546,"duration_ms":5263,"temperature":1.0,"reasoning_tokens":463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:23:34.067112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of APKs whose malicious code lives in classes that Dex2Jar or CFR demonstrably fail to recover (confirmed by comparing the decompiled class list against the actual dex bytecode), run MalParse's Malware-Scoped pipeline on them, and check whether the classification accuracy drops measurably below 77%; if it stays near 77%, the decompilation-preservation premise is wrong, and if it drops, the premise is confirmed as the load-bearing limitation.","supporting_citations":[{"cited_title":"dex2jar,","cited_arxiv_id":null,"evidence_quote":"Dex2Jar converts APK dex bytecode to Java bytecode, the necessary first transformation in the decompilation pipeline whose output feeds all summarization."},{"cited_title":"Cfr decompiler,","cited_arxiv_id":null,"evidence_quote":"CFR decompiles the Java bytecode into human-readable Java source, which is the input analyzed by the hierarchical summarization chain."},{"cited_title":"Summit: Iterative text summarization via chatGPT,","cited_arxiv_id":null,"evidence_quote":"Provides the iterative text summarization strategy that motivates MalParse's bottom-up approach for large contexts."},{"cited_title":"Element-aware summarization with large language models: Expert-aligned evaluation and chain- of-thought method,","cited_arxiv_id":null,"evidence_quote":"Supports the claim that iterative, chain-of-thought summarization produces expert-aligned output, underpinning the design of the package-level summary."},{"cited_title":"Langchain,","cited_arxiv_id":null,"evidence_quote":"LangChain is used to implement the custom LLM chains and prompt templates that drive the three prompt scopes."}],"review_version":1}