{"id":"aceb8aa4-9f36-4c82-b8f2-9a017e7935bf","arxiv_id":"2504.21028","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":9,"one_line_summary":"Cosine-similarity-based hard negative selection during contrastive fine-tuning makes LLM embeddings separate malware families more cleanly, improving few-shot multimodal malware classification accuracy by 11 to 21 percentage points on two datasets.","lead":"This paper reports that fine-tuning large language model embeddings with contrastive learning and carefully chosen hard negative examples improves malware family classification from text descriptions. In few-shot tests on two public malware datasets, the best model reached 63% accuracy, about 11 to 21 points above baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains may reflect CFT training on the same malware samples/families used in MAML evaluation; no data split is documented.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the absence of a documented split between CFT training data and MAML evaluation data. The ground-truth family descriptions used as positives in Section III.B are exactly the family concepts being classified in Section IV.B; unless disjointness is shown, the reported gains could be exposure rather than generalization. This concern is independent of the algorithm's plausibility and affects every accuracy comparison in the paper. I considered whether the more interesting failure is Algorithm 1's apparent inversion (the condition s <= T would exclude high-similarity negatives, while the text and Figure 3 indicate s >= T is intended), but that is a typo-level inconsistency and does not change the empirical claim the way data leakage would. Missing error bars and hyperparameter sensitivity are important but secondary. A strict sample-disjoint and family-disjoint rerun would settle the question; until then CONDITIONAL remains the correct verdict. My recommendation does not move the reader's verdict.","tokens_in":9907,"tokens_out":4288,"duration_ms":46447,"concrete_test":"Audit the exact partition used in the experiments: verify whether any sample IDs used to generate the 200 CFT anchors per family, or any ground-truth family descriptions used as positives, appear in the MAML support or query sets. Then rerun the full pipeline (Tables III through V) under a strict sample-disjoint split; if feasible, also run a family-disjoint split where entire families are held out from CFT. If accuracy drops to roughly baseline levels under disjoint splits, leakage explains the reported gains; if the 11 to 21 point improvements persist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: similarity-based hard-negative CFT makes LLM embeddings discriminative enough to improve few-shot malware family classification. This requires that CFT training data and MAML evaluation data are disjoint in the relevant sense (samples, and ideally families). The paper never states such a split. Section III.B says 200 anchors are generated per family and paired with an expert ground-truth description for that family; Section III.C selects hard and mid-tier negatives from other families' descriptions; Section IV.B evaluates support and query sets per family. All of this happens within the same two datasets, and no hold-out family or sample partition is described. Because the ground-truth positive for every family is directly used in CFT, the fine-tuned LLM has seen the family-level semantic anchor that also defines the MAML classes. If the 200 anchors or their source samples overlap with the 10+20 support/query samples, the 11 to 21 point gains over baselines could be memorization or exposure rather than generalization. The method might still work, but Tables III through V would not establish it. This is the load-bearing assumption because every reported accuracy comparison depends on it, and it is unstated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a contrastive fine-tuning (CFT) method that selects hard negative samples by cosine similarity in the embedding space of LLM-generated malware descriptions, and combines the resulting embeddings with binary behavioral features in a MAML-based few-shot multimodal classifier. Experiments on CIC-AndMal-2020 and BODMAS report large accuracy gains over pre-trained embeddings and over random negative sampling, with the best result being 63.15% on CIC-AndMal-2020, and a RAGAS-based evaluation of description quality is also included.","tokens_in":9990,"tokens_out":2475,"duration_ms":27136,"significance":"If the reported results are valid, the paper makes a useful contribution: it demonstrates a concrete way to adapt LLM embeddings for fine-grained malware family discrimination and shows consistent gains across four LLMs and two datasets, which is a nontrivial qualitative pattern. The multimodal integration of fine-tuned text embeddings with behavioral features under MAML is also an interesting direction. However, the central claim depends on the contrastive fine-tuning data being disjoint from the few-shot evaluation data in a way that the manuscript never states, and the quantitative comparisons lack any uncertainty estimation, so the significance of the claimed 11--21 percentage point improvements is currently not established.","major_comments":[{"comment":"The manuscript never describes a data split between the contrastive fine-tuning (CFT) training data and the MAML evaluation data. Section III.B says 200 anchors are generated per family from the same two datasets, and Section III.C selects hard negatives from other families' descriptions using the family-specific ground-truth description. Section IV.B then evaluates support and query sets per family on those same datasets. If the anchors or their source samples overlap with the 10+20 support/query samples, or if the expert ground-truth descriptions of the evaluation families were used as positives during CFT, the reported gains in Tables III--V could reflect exposure to the test families rather than generalization. This is load-bearing for every accuracy comparison in the paper, so the authors must specify the exact partition (sample-level and family-level) and confirm that no information from the evaluation families or samples enters CFT.","section":"Sections III.B--III.C and IV.B"},{"comment":"Each accuracy in Tables III, IV, and V is reported as a single number with no error bars, no number of random trials, and no significance tests. Because the MAML pipeline and the negative sampling procedure involve randomness (batch composition, random mid-tier negatives, initialization), the 11--21 percentage point improvements and the ablation differences of 10--23% could be within run-to-run variation. The authors should report means and standard deviations over multiple runs and, ideally, a paired significance test for the random-vs-similarity ablation.","section":"Tables III--V"},{"comment":"The abstract and introduction claim that the method surpasses 'prior negative sampling strategies,' but the experiments only compare similarity-based negatives with random negatives. No comparison is made to the class-label-based negatives of Supervised Contrastive Learning [7], the semantic clustering approach [10], or the distance-aware approach of Xu et al. [11], despite these being discussed in the related work. Either add these comparisons or remove the claim about surpassing prior strategies.","section":"Abstract and Section IV"},{"comment":"The BODMAS feature selection is performed using LightGBM feature importance on the full dataset before any train/test split is described. Selecting the top 64 features (and limiting to 15 families based on sufficient samples) using the entire dataset can leak information from the test portion into the model pipeline. The authors should clarify whether feature selection was done inside a cross-validation or on a separate training fold, and report how the family subset was chosen.","section":"Section III.A"}],"minor_comments":[{"comment":"The paragraph beginning 'Meta-learning, particularly Model-Agnostic Meta-Learning (MAML) [6]...' is duplicated verbatim, first in the discussion of meta-adaptation and then again in the knowledge-distillation paragraph.","section":"Section II.D"},{"comment":"The sentence referencing Figure 2 is incomplete: 'Figure 2 shows the . Although some of the families...' contains a dangling phrase and a missing noun. This needs to be rewritten for clarity.","section":"Figure 2 caption and text in Section III.B"},{"comment":"Algorithm 1 adds candidate negatives only when s ≤ T, where T is described as the 'Maximum Similarity Threshold,' but the text describes hard negatives as having high cosine similarity in a range 0.85--0.95. It is unclear whether T is an upper bound, a lower bound, or a cutoff that then requires the descending sort to find the top-20 hardest negatives. Clarify the role of T.","section":"Algorithm 1"},{"comment":"Reference [16], about multimodal detection of hateful memes, does not appear relevant to malware description generation or low-level feature analysis in the sentence where it is cited. A more germane reference on LLM-based malware analysis would be appropriate.","section":"Reference [16]"}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible and the consistent trend across models and datasets is encouraging, but the missing data split description is a serious protocol gap that must be resolved before the claims can be evaluated. The paper also would benefit from a code or artifact release, since no implementation details are provided and the single-run accuracy numbers are hard to verify. I would not recommend rejection if the authors can provide a clear and credible split description and uncertainty estimates, but the current manuscript is not yet publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the method is a reasonable extension of known contrastive tricks to LLM-generated malware descriptions, and the evaluation is broad enough to be interesting. But the paper never states whether the contrastive fine-tuning data and the MAML evaluation data are disjoint, and that gap makes the reported 11–21 point gains hard to trust as evidence of generalization.\n\nWhat's actually new: the two-tier negative selection—taking the top-20 highest cosine-similarity negatives plus a few mid-tier ones—applied to LLM embeddings of malware descriptions before feeding them into a MAML few-shot classifier. That specific combination is not in the cited work, and the consistent direction of results across four LLMs and two datasets suggests the effect is real, not a single-model fluke. The ablation showing similarity-based negatives beating random negatives by 10–20 points is also a useful data point. Credit where due: the method is described well enough to re-implement, and the RAGAS evaluation of description quality goes beyond raw accuracy.\n\nNow the soft spots, in proportion. The load-bearing one is the split question. The paper says 200 anchors per family are generated and each anchor is paired with that family's expert ground-truth description. If those anchors come from the same samples, or even the same families, used later in the MAML support/query sets, then the fine-tuned LLM has effectively seen the family labels during contrastive training. The MAML classifier then only needs to pick up on an embedding space that was explicitly shaped to separate those families. That would make the gains a measure of exposure, not generalization. The paper never clarifies this, and it's easy to fix—just state whether families were held out, and if not, frame the claim as improving same-family few-shot classification rather than generalization to new families.\n\nMinor but real: no error bars or significance tests, single-run accuracies, hand-picked thresholds (T, negative counts) with no sensitivity analysis, and no code release. The claim of being the \"first multimodal malware classification\" is also an overclaim, contradicted by their own related-work section. There's a duplicated paragraph in Section II.D and an incomplete sentence in Section III.B.\n\nThe method might well work; the evidence just doesn't yet support the strong version of the claim. This deserves a serious referee, not a desk reject—but the referee should insist on the data split, variance estimates, and ideally code.\n\nFor peer review: yes, send it out. For my own citation list: not in the next year, unless the split question gets resolved.","headline":"The hard-negative selection idea is sensible, but the missing train/eval split makes the headline gains unverifiable.","tokens_in":10665,"tokens_out":2762,"would_cite":false,"duration_ms":27843,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that selecting hard negatives by cosine similarity during contrastive fine-tuning makes LLM embeddings discriminative enough to materially improve few-shot malware family classification, reporting 63.15% accuracy with 20…","keywords":["malware classification","contrastive fine-tuning","hard negative sampling","multimodal learning","few-shot classification","LLM embeddings","cybersecurity","knowledge distillation"],"falsifier":"Run the pipeline again under a strict family-level hold-out, training contrastive fine-tuning only on families excluded from the few-shot evaluation, and compare accuracy with the reported setup. If the 11–21 point gain mostly disappears, the embeddings benefited from exposure to test-family descriptions rather than from the hard-negative selection rule.","tokens_in":9541,"feed_emoji":"🛡️","tokens_out":8410,"duration_ms":76323,"temperature":0.7,"pith_summary":"Malware families keep spawning new variants, and analysts often share text descriptions rather than raw binaries. The paper claims that standard LLM embeddings are too overlapping to tell closely related malware families apart, and that a contrastive fine-tuning step fixes this when the negative examples are chosen by cosine similarity rather than at random. On two public malware datasets, the fine-tuned embeddings, paired with binary features in a few-shot meta-learning classifier, lift accuracy by 11–21 percentage points over an attribute-only baseline, reaching 63.15% accuracy with 20 samples per family. If true, defenders could classify new variants from natural-language threat reports without large labeled sets.","feed_headline":"Cosine-picked decoys lift malware family accuracy by 11-21 points","feed_subtitle":"Choosing the closest decoys during fine-tuning makes LLM descriptions separate malware families in few-shot tests.","key_machinery":"The load-bearing mechanism is a similarity-ranked negative-selection procedure wrapped around the InfoNCE contrastive loss. Hard negatives are descriptions from other families that look almost identical to the positive in the pre-trained embedding space; the procedure selects the top 20 by cosine similarity and adds 12 randomly drawn mid-similarity negatives. Each anchor-positive pair is trained with five hard and three diverse negatives per contrastive sample. The refined embeddings, obtained by mean pooling final hidden states, are then evaluated in a Model-Agnostic Meta-Learning (MAML) multimodal classifier that fuses them with binary attributes through knowledge distillation. This negative-selection rule is what forces the model to repel near-identical descriptions while retaining enough diversity to generalize.","core_discovery":"The central discovery is that negative-sample selection, not the contrastive loss itself, determines whether LLM text embeddings become discriminative for malware families. For each family's expert-written reference description, the authors rank candidate descriptions from other families by cosine similarity and keep the top 20 (scores around 0.85–0.95) as hard negatives, plus 12 mid-tier negatives for diversity. InfoNCE training on anchor-positive-hard-negative triples then separates families whose language is nearly indistinguishable in the pre-trained space. The refined embeddings are fused with binary behavioral features in a Model-Agnostic Meta-Learning classifier with knowledge distillation, and the paper reports consistent gains over random-negative contrastive fine-tuning and over pre-trained embeddings alone. The stated interpretation is that the model learns finer semantic distinctions and generates more attribute-specific descriptions that generalize to unseen samples.","pith_inferences":["Editorial inference: the same cosine-similarity hard-negative rule should transfer to other high-overlap text classification tasks, such as phishing reports, fraud narratives, or medical symptom notes, wherever a similarity-ranked candidate pool can be built.","Editorial inference: a strict temporal evaluation, fine-tuning on older variants and evaluating on newer variants of the same families, would test whether the generalization claim reflects family-level attributes or dataset-specific phrasing.","Editorial inference: publishing per-family confusion matrices would show whether the gains come from separating the hardest overlapping pairs, such as the Android adware families, or from easier families, and would isolate the mechanism.","Editorial inference: the method relies on one expert-written ground-truth description per family; testing multiple or crowd-sourced positives would show whether the hard-negative signal survives positive-sample variation."],"forward_implications":["If the claim holds, LLM-generated malware descriptions become usable as a classification signal, not just analyst reading material, because fine-tuned embeddings separate families that raw embeddings blur.","Few-shot accuracy of 63.15% with 20 samples per family on CIC-AndMal-2020 means new families could be handled with tiny labeled sets, which matters when samples are scarce and reports are abundant.","Similarity-based negative selection generalizes across model sizes and datasets, with larger models gaining more, so the method scales with better base LLMs.","Because the same embeddings improve RAGAS correctness and similarity scores, the fine-tuned models produce descriptions that are both more machine-actionable and more interpretable for analysts.","The multimodal fusion with binary features means the text modality adds information beyond dynamic or static attributes rather than substituting for them."],"supporting_citations":[{"why":"Supplies the InfoNCE contrastive loss that the fine-tuning step optimizes.","marker":"[8]"},{"why":"Provides the MAML meta-learning framework in which the embeddings are evaluated few-shot.","marker":"[6]"},{"why":"Supplies the CIC-AndMal-2020 dataset with dynamic behavioral features used for the main malware-family evaluation.","marker":"[4]"},{"why":"Supplies the BODMAS dataset with static binary features used for the second evaluation.","marker":"[5]"},{"why":"Defines the distance-aware contrastive-learning baseline that motivates and is compared against the similarity-based negative selection.","marker":"[11]"},{"why":"Provides the EMBER feature-reduction approach used to select the top 64 BODMAS features.","marker":"[24]"},{"why":"Supplies the knowledge-distillation method used to fuse textual embeddings with binary features in the multimodal classifier.","marker":"[19]"}],"fun_headline_variants":["Cosine-selected decoys boost malware family accuracy by up to 21 points","Closest decoys as negatives sharpen malware family embeddings","Similarity-ranked negatives beat random for few-shot malware classification","Hard negatives by cosine similarity improve malware ID in few-shot tests","Cosine-similarity decoys for fine-tuning lift malware classification accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume the contrastive fine-tuning step never saw descriptions from the malware families used in the few-shot evaluation; the paper does not state that such a split was made.","fun_headline_variants_meta":{"raw":{"variants":["Cosine-selected decoys boost malware family accuracy by up to 21 points","Closest decoys as negatives sharpen malware family embeddings","Similarity-ranked negatives beat random for few-shot malware classification","Hard negatives by cosine similarity improve malware ID in few-shot tests","Cosine-similarity decoys for fine-tuning lift malware classification accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000745,"raw_usage":{"total_tokens":3337,"prompt_tokens":977,"completion_tokens":2360,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":2273}},"tokens_in":593,"tokens_out":2360,"duration_ms":20612,"temperature":1.0,"reasoning_tokens":2273,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:26:43.984474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline again under a strict family-level hold-out, training contrastive fine-tuning only on families excluded from the few-shot evaluation, and compare accuracy with the reported setup. If the 11–21 point gain mostly disappears, the embeddings benefited from exposure to test-family descriptions rather than from the hard-negative selection rule.","supporting_citations":[{"cited_title":"unb.ca/cic/datasets/andmal2020.html, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the CIC-AndMal-2020 dataset with dynamic behavioral features used for the main malware-family evaluation."},{"cited_title":"Bodmas: An open dataset for learning based temporal analysis of pe malware","cited_arxiv_id":null,"evidence_quote":"Supplies the BODMAS dataset with static binary features used for the second evaluation."},{"cited_title":"Enhancing information maximization with distance-aware contrastive learning for source-free cross-domain few- shot learning","cited_arxiv_id":null,"evidence_quote":"Defines the distance-aware contrastive-learning baseline that motivates and is compared against the similarity-based negative selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the EMBER feature-reduction approach used to select the top 64 BODMAS features."}],"review_version":1}