{"id":"06750f73-3ebf-4a21-be1f-59dfb8712ebb","arxiv_id":"2505.12106","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MalVis-B encodes Android DEX bytecode into RGB images using entropy and N-gram features, and CNN classifiers trained on these images reach 95.19% accuracy on a binary malware detection benchmark.","lead":"This paper introduces MalVis, a new way to turn Android app code into color images that highlight suspicious patterns, plus a dataset of over 1.3 million such images. The authors report that deep learning models trained on these images detect malicious apps with about 95% accuracy in a binary test, though the gains over older image methods are modest and real-world validation is still missing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Image-level split may leak app information: 1.3M images come from 184k apps, yet the 80/10/10 split is not described as app-grouped.","rationale":"The central claim is empirical: MalVis-B's encoding yields 95.19% accuracy, 90.81% F1, and 98.06% ROC-AUC on binary detection, and MalVis is claimed to be the largest public Android malware visualization dataset. The most load-bearing condition is that the evaluation measures generalization to new apps rather than familiarity with app-specific patterns. The paper reports 1,300,822 images from 184,474 apps, and Section 3.4 describes the split without any app-level grouping. Since Section 3.2 maps each DEX file to one image, multiple images can be derived from a single APK. A random image-level split can therefore place near-duplicate DEX images in both training and test partitions, which CNNs exploit to inflate absolute metrics and potentially alter model rankings. This is not a disagreement with external consensus; it is an internal threat to the validity of the reported numbers. The concrete check is an app-hash-grouped re-split and retraining of the headline row. The reader's weakest assumption identifies exactly this issue, so I agree with that diagnosis. The CONDITIONAL verdict remains appropriate: the paper should be revised to use app-level splits (ideally plus a temporal or family-disjoint held-out set), report error bars, and provide the promised versioned artifacts. I do not move the verdict because the fix is feasible and the visualization idea remains plausible; the concern strengthens the condition rather than changing the overall decision.","tokens_in":16491,"tokens_out":5422,"duration_ms":52268,"concrete_test":"Use the released dataset metadata to map each image to its source APK/SHA-256, and count images per app. Re-run the binary Inception-V3 experiment with a group-based split: random 80/10/10 split at the app level so all images of a given APK stay in one partition, keeping preprocessing, hyperparameters, and epochs identical. If the test accuracy or F1 drops by more than about one point from 95.19%/90.81%, or if MalVis-B no longer beats the entropy-only baseline, the paper's image-level split leaks app-level information. Also report the fraction of test images whose source app appears in training under the original split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 specifies an 80/10/10 partition but does not state whether the split unit is an image or an app. Section 3.1 gives 49,150 malware and 135,324 benign applications, while Figure 3 and Table 1 report 1,300,822 images; that is roughly seven images per app. Section 3.2 says each extracted DEX file is reshaped into one 256x256 image, so multiple images from the same APK (e.g., multi-dex classes.dex files) are plausible. If the random split is applied to images rather than apps, images from the same APK can occur in both training and test sets. DEX files from one APK share code, resources, and obfuscation structure, so a CNN can memorize app-specific features and inflate every metric in Table 2, including the headline Inception-V3 result of 95.19% accuracy and 90.81% F1. The ranking of MalVis-B against the entropy-only baseline could also change if leakage is uneven. The abstract's zero-day claim is additionally unsupported because no temporal or family-disjoint split is evaluated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MalVis, a visualization framework and dataset for Android malware classification. MalVis encodes DEX bytecode into RGB images by combining Shannon entropy (red and blue channels) with either Classbyte or bi-gram structural encoding (green channel). The authors construct a dataset of just over 1.3 million images from AndroZoo apps, covering nine malware families and a benign class, and evaluate four/five CNN architectures plus eight ensemble strategies on binary and multiclass tasks. The headline binary result is 95.19% accuracy and 90.81% F1-score with Inception-V3 using the MalVis-B encoding, claimed to outperform prior image-based encodings (MalNet, Classbyte, and the authors' own entropy-only method). The paper also claims zero-day detection capability and positions MalVis as the largest public Android malware visualization dataset.","tokens_in":16702,"tokens_out":4786,"duration_ms":43714,"significance":"If the central results hold, the paper would make a useful contribution: a large public dataset plus a reproducible image-generation pipeline is valuable for the Android malware visualization community. The controlled experiments in Section 3.3, which show the sensitivity of entropy and bi-gram channels to encryption and unstructured bytecode insertion, are a nice interpretability exercise. The comparison across multiple CNN architectures and ensemble methods is also more thorough than many prior works. However, the significance is conditional on resolving a potentially severe data-splitting flaw: with roughly seven images per app, an image-level split could leak app-specific information into the test set and inflate all reported metrics. Because the main claim is about relative ranking of encodings, not just absolute performance, this issue is load-bearing.","major_comments":[{"comment":"The split unit is not specified. The dataset contains 1,300,822 images generated from 184,474 apps (Section 3.1), and Section 3.2 states that each DEX file is reshaped into a 256x256 image, implying multiple images per app. If the 80/10/10 partition in Section 3.4 is applied at the image level rather than the app level, images from the same APK or the same malware family can appear in both training and test sets, allowing the CNN to memorize app-specific or family-specific bytecode structure. This would inflate every metric in Tables 2-5, including the headline Inception-V3 result, and could change the ranking between MalVis-B and the entropy-only baseline. The authors must state whether the split is by app or by image, describe how multiple images per app are produced (e.g., chunking large DEX files), and report results with an app-grouped split. Without this, the central claim of superiority over prior encodings is not established.","section":"§3.4, §3.2, Table 1"},{"comment":"The abstract and introduction claim improved detection of 'previously unseen (zero-day) malware,' but no experiment evaluates generalization to unseen apps, unseen families, or later time periods. All reported results use a random 80/10/10 split, which does not simulate zero-day conditions. To support the zero-day claim, the authors should add a family-disjoint or temporal split evaluation, or explicitly remove the claim from the abstract and conclusions.","section":"Abstract, §5, §6"},{"comment":"All results are single-run point estimates with no error bars, confidence intervals, or significance tests. Many of the claimed improvements over the entropy-only baseline are small (e.g., Inception-V3 accuracy 95.19% vs 94.94%, F1 90.81% vs 90.43% in Table 2), so it is unclear whether the differences are meaningful or within run-to-run variance. The authors should report mean and standard deviation over multiple random seeds, and ideally a paired significance test, before claiming consistent superiority.","section":"Tables 2-5"}],"minor_comments":[{"comment":"The text reports that ResNet50 achieved 'overall accuracy of 94.03%,' but Table 3 lists accuracy as 84.03% for this row; 94.03% appears to be the ROC-AUC value. Please correct the text or the table.","section":"§5.2, Table 3"},{"comment":"The description of Min Confidence Voting says predictions are used only when confidence exceeds 60%, but the behavior when no model meets the threshold is not defined. If such samples are excluded from the accuracy computation, the 88.65% accuracy is not directly comparable to the other rows. Clarify the abstention rule.","section":"Table 5, §5.4"},{"comment":"The dataset comparison in Table 1 mixes Android-specific datasets with general (e.g., PE-based) malware image datasets. Clarify for each row whether the images are derived from Android applications, since the claim of being the 'largest Android malware visualization dataset' depends on this distinction.","section":"§2.2, Table 1"},{"comment":"The header contains a typo: 'Precsion' should be 'Precision'.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The app-level split concern is the decisive issue. If the authors can show that the split is app-grouped, or rerun experiments with an app-grouped split and the numbers remain similar, the paper could be a solid contribution. As it stands, the uncertainty about leakage makes the headline comparison unreliable. The zero-day claim should either be supported with a proper evaluation or removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the MalVis dataset is a real community resource in the making, and the green-channel bigram encoding is a sensible new idea, but the paper does not rule out leakage between training and test images, so the headline 95% numbers should not be taken at face value.\n\nWhat is actually new: over 1.3M Android DEX images across ten classes, which would be the largest public Android image dataset; the encoding that puts Shannon entropy in red/blue and a bigram transition value in green is a genuine combination rather than a trivial rehash. The authors also do the right thing by comparing against MalNet, Classbyte, and their own earlier entropy method, and by showing controlled examples of how AES-encrypting part of a DEX changes the red/blue channels and inserting random bytes changes the green channel. MalVis-A failing to beat the entropy baseline is reported honestly. None of that is trivial.\n\nThe soft spots are mostly about the experimental protocol. Section 3.4 describes an 80/10/10 split without saying whether the unit is an image or an app. Since 1.3M images come from about 184k APKs (multiple DEX files per app), an image-level split can put near-duplicate code from the same app in both train and test. That would inflate every number in Table 2, and it could also change the ranking of MalVis-B versus the entropy-only baseline. This is not a minor issue; it is the main thing a referee should ask for. Second, all results appear to be single runs with no error bars or seeds, so differences of one or two points between methods are not interpretable. Third, the zero-day claim in the abstract and conclusion is not backed by any temporal or family-disjoint evaluation. Fourth, in the balanced multiclass setting the jump from about 66% for individual models to 88.65% for Min Confidence Voting looks too good; the 60% threshold and the behavior when no model meets it are not described, and that parameter looks selected on test data.\n\nNone of this means the encoding is useless. The interpretability experiments are evidence the channels respond to the intended bytecode properties. But the paper needs app-grouped splits, repeated trials, a described (and preferably validated) confidence-voting protocol, and a non-visual baseline before the numbers can be believed.\n\nWho this is for: researchers working on Android malware image representations and anyone who wants a large public DEX image dataset. It deserves peer review, and the likely right outcome is major revision rather than rejection.","headline":"A genuinely new entropy-plus-bigram RGB encoding and a large public Android image dataset, but the image-level split is not ruled out, so the headline accuracy numbers should not be trusted until app-grouped splits are shown.","tokens_in":17261,"tokens_out":2543,"would_cite":false,"duration_ms":24742,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MalVis-B encodes Android bytecode entropy and bi-gram transitions into RGB images and outperforms prior malware visualizations.","keywords":["Android malware detection","malware visualization","entropy encoding","N-gram analysis","convolutional neural networks","RGB bytecode images","MalVis dataset","binary classification"],"falsifier":"Compute the overlap between source APKs in the training and test partitions of MalVis, then rerun the binary Inception-V3 experiment with a group-aware split that keeps each app's images in one partition only; if accuracy falls much below 95.19%, the central generalization claim is inflated.","tokens_in":16275,"feed_emoji":"🛡️","tokens_out":9863,"duration_ms":89924,"temperature":0.7,"pith_summary":"This paper tries to establish that a visualization method called MalVis-B lets convolutional networks detect Android malware better than earlier image-based methods. MalVis-B colors each pixel of a 256×256 image three ways: Shannon entropy over 32-byte windows drives the red and blue channels, while a bi-gram value computed from consecutive bytes drives the green channel. The authors report 95.19% accuracy, 90.81% F1-score, and 98.06% ROC-AUC with Inception-V3 on binary malware-versus-benign classification, and they say MalVis-B beats the MalNet, Classbyte, and entropy-only baselines on most backbones. They also release MalVis, a public dataset of over 1.3 million such images spanning nine malware classes and one benign class. If the claim holds, security analysts get an interpretable, obfuscation-sensitive image representation and a large common benchmark for testing it.","feed_headline":"Bytecode entropy + bi-gram images detect Android malware at 95.19%","feed_subtitle":"Entropy flags encrypted code, bi-grams flag structural anomalies, and a 1.3M-image public dataset backs the result.","key_machinery":"The load-bearing object is the MalVis-B encoding, a three-channel RGB image built from the classes.dex bytes of an app. The red and blue channels come from Shannon entropy over a 32-byte sliding window, with a curved scaling on the red channel and a squared-entropy scaling on the blue channel; the green channel carries the normalized bi-gram value $g=(b_1\\cdot 256+b_2)/65535$, which treats each pair of consecutive bytes as a transition. The one-dimensional byte array is reshaped to a fixed 256×256 grid using nearest-neighbor interpolation. This machinery is what lets a generic CNN see encryption as bright high-entropy regions and repacking or other structural anomalies as bright bi-gram textures, and it is the component the paper varies when it tests MalVis-A's Classbyte green channel in place of the bi-gram.","core_discovery":"The central claim is that encoding the same classes.dex bytecode into three semantically loaded color channels—entropy in the red and blue channels, bi-gram transitions in the green channel—makes malware structure visible to CNNs in a way that byte-location color mappings do not. On binary malware-versus-benign classification, MalVis-B with Inception-V3 reaches 95.19% accuracy, 90.81% F1-score, 92.58% precision, 89.10% recall, 87.58% MCC, and 98.06% ROC-AUC, beating each baseline on most backbones, with DenseNet201 noted as the exception. On the imbalanced ten-class task, ResNet50 leads with 84.03% accuracy, and after undersampling to balance classes, a min-confidence voting ensemble restores accuracy to 88.65% with 86.32% F1. The paper also claims that MalVis is the largest public Android malware visualization dataset, containing 1,300,822 images generated from AndroZoo apps and labeled through Euphony and VirusTotal.","pith_inferences":["A direct test of the independence assumption would be to group images by source APK, split so no APK appears in both training and test, and rerun the Inception-V3 experiment; if accuracy drops substantially, the 95.19% figure reflects app-specific pattern memorization, not generalization to new apps.","The same entropy-plus-N-gram channel decomposition is likely transferable to Windows PE or Linux ELF bytecode, where packed sections also show high entropy and unusual transition patterns, so the method need not be Android-specific in principle.","A human-analyst study that asks people to locate encrypted or repacked regions using only the red, blue, and green channels would directly measure the interpretability claim the paper asserts but does not quantify.","Varying the sliding-window size (the paper fixes 32 bytes for entropy and two bytes for the bi-gram) might further separate malware families that share obfuscation techniques, since larger windows would capture longer-range structural dependencies at higher computational cost."],"forward_implications":["Encrypted, packed, or otherwise obfuscated code should appear as bright, localized high-entropy regions in the red and blue channels, giving CNNs a learnable visual signature rather than a byte-exact signature.","Because the green channel encodes transitions between consecutive instructions, repacked or structurally altered bytecode should show visible texture changes, which is why the bi-gram term adds signal beyond entropy alone.","The 1.3M-image MalVis dataset gives other research groups a common public benchmark for image-based Android malware detection, removing a data-access bottleneck.","On the balanced ten-class task, a min-confidence voting ensemble raises accuracy to 88.65% and F1 to 86.32%, showing that ensemble inference can offset the accuracy lost to undersampling.","MalVis-A's failure indicates that arbitrarily adding a semantic channel does not help; the green channel has to encode transition statistics rather than Classbyte colors."],"supporting_citations":[{"why":"Supplies the large APK corpus from which the MalVis dataset draws its malware and benign samples.","marker":"[44]"},{"why":"Prior entropy-based bytecode-to-image encoder whose red and blue channel formulas MalVis extends.","marker":"[37]"},{"why":"Source of the Classbyte green-channel encoding used in the MalVis-A variant and as a baseline for comparison.","marker":"[34]"},{"why":"The principal large-scale image-encoding baseline whose byte-location mapping MalVis-B is compared against.","marker":"[46]"},{"why":"Harmonized antivirus-labeling method used to assign the nine malware families in the multiclass dataset.","marker":"[38]"},{"why":"VirusTotal scanning used to verify the malware labels and filter ambiguous multi-label samples.","marker":"[39]"},{"why":"Tool used to extract classes.dex files from APKs before the bytecode-to-image conversion.","marker":"[59]"}],"fun_headline_variants":["Entropy + bi-gram bytecode images detect Android malware at 95.19%","MalVis: 1.3M visual samples for Android malware classification","Color-coded bytecode: entropy and N-grams expose malware patterns","MalVis: entropy and N-gram visualization for CNN malware detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 1.3 million images can be treated as independent samples in the 80/10/10 split; the paper does not establish that images from the same app or malware family never appear in both training and test partitions, and if they do, the reported accuracy overstates detection of genuinely new apps.","fun_headline_variants_meta":{"raw":{"variants":["Entropy + bi-gram bytecode images detect Android malware at 95.19%","MalVis: 1.3M visual samples for Android malware classification","Color-coded bytecode: entropy and N-grams expose malware patterns","MalVis: entropy and N-gram visualization for CNN malware detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00071,"raw_usage":{"total_tokens":3265,"prompt_tokens":1083,"completion_tokens":2182,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":2100}},"tokens_in":699,"tokens_out":2182,"duration_ms":15667,"temperature":1.0,"reasoning_tokens":2100,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:41:01.258669+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the overlap between source APKs in the training and test partitions of MalVis, then rerun the binary Inception-V3 experiment with a group-aware split that keeps each app's images in one partition only; if accuracy falls much below 95.19%, the central generalization claim is inflated.","supporting_citations":[{"cited_title":"Androguard tool by google","cited_arxiv_id":null,"evidence_quote":"Tool used to extract classes.dex files from APKs before the bytecode-to-image conversion."},{"cited_title":"Androzoo: Collecting millions of android apps for the research community","cited_arxiv_id":null,"evidence_quote":"Supplies the large APK corpus from which the MalVis dataset draws its malware and benign samples."},{"cited_title":"Improving android malware detection with entropy bytecode-to-image encoding framework","cited_arxiv_id":null,"evidence_quote":"Prior entropy-based bytecode-to-image encoder whose red and blue channel formulas MalVis extends."},{"cited_title":"Hit4mal: Hy- brid image transformation for malware classification.Transactions on Emerging Telecommunications Technologies, 31(11):e3789, 2020","cited_arxiv_id":null,"evidence_quote":"Source of the Classbyte green-channel encoding used in the MalVis-A variant and as a baseline for comparison."},{"cited_title":"Malnet: A large-scale image database of malicious software","cited_arxiv_id":null,"evidence_quote":"The principal large-scale image-encoding baseline whose byte-location mapping MalVis-B is compared against."},{"cited_title":"Euphony: harmonious unification of cacophonous anti-virus vendor labels for android malware","cited_arxiv_id":null,"evidence_quote":"Harmonized antivirus-labeling method used to assign the nine malware families in the multiclass dataset."},{"cited_title":"https://www.virustotal.com","cited_arxiv_id":null,"evidence_quote":"VirusTotal scanning used to verify the malware labels and filter ambiguous multi-label samples."}],"review_version":1}