{"id":"dd4f8f60-0d37-4306-9dd2-f6ddcca1c6d4","arxiv_id":"2505.00034","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Explanation-augmented LoRA fine-tuning lets small LLMs detect phishing emails with accuracy and F1 around 0.94 to 0.96 on the SpamAssassin test set, while transferring to unseen datasets.","lead":"This paper tests whether small language models (1.5B to 3.8B parameters) can detect phishing emails as well as much larger models after fine-tuning on labels plus explanations. If the results hold, cheap and interpretable phishing filters could run on consumer-grade GPUs instead of large server clusters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transfer claim rests on 'unseen' datasets, but the paper never verifies that the 1,000-sample training subsets are disjoint from SpamAssassin/CEAS_08/Ling/Enron test sets; overlap would directly inflate reported gains.","rationale":"The paper's in-domain gains are plausible and the ablation points in the right direction, but the transferability claim hinges on Ling/Enron being genuinely unseen. The absence of any deduplication or contamination analysis leaves a concrete, checkable hole: public email corpora are frequently repackaged and duplicated, and the cited Kaggle dataset aggregates multiple corpora. If overlap exists, the central 'unseen' claim weakens substantially; if overlap is shown absent, the concern is resolved. The reader's weakest assumption already flagged leakage/contamination, and my independent review converges on the same load-bearing point. The Section 4.2 metric inconsistency is secondary but reinforces the need for raw data, splits, and code to be released. I therefore keep the conditional verdict rather than moving to accept or reject.","tokens_in":10139,"tokens_out":6448,"duration_ms":69866,"concrete_test":"Obtain or reconstruct the exact 1,000-sample fine-tuning subsets and all test sets (SpamAssassin, CEAS_08, Ling, Enron). Normalize headers and body text (lowercase, strip whitespace/line breaks) and compute exact-match hashes plus MinHash near-duplicate similarity at a threshold such as 0.9. If any test email matches a training email, rerun all experiments after removing every overlapping message from the test sets and report whether the accuracy/F1 gains persist. Also run a membership probe: fine-tune on the 1,000 samples and compare model confidence on test-set emails versus held-out control emails from the same distribution; a large gap would suggest memorization rather than generalizable detection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that explanation-augmented LoRA fine-tuning improves small LLMs and transfers to unseen Ling/Enron—requires that the evaluation sets were not seen during fine-tuning. The paper cites a single Kaggle dataset (ref [12]) that aggregates public email corpora, and such corpora are known to contain duplicate or near-duplicate messages across sources. The authors do not report any deduplication, overlap analysis, or contamination check before fine-tuning on the 1,000-sample subset and testing on 1,069/1,000 samples. If any test message, or a near-duplicate, appears in the fine-tuning subset, the reported accuracy/F1 and especially the transfer numbers are inflated by memorization rather than by explanation-augmented generalization. Because GPT-4o-mini generated explanations from the same labeled emails, this risk is not mitigated. A secondary concern is that Section 4.2 misreports LLaMA's vanilla F1 as 0.543 when Table 1 lists 0.354, which further underscores that raw predictions and splits should be made available for verification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for improving phishing email detection with small LLMs (LLaMA-3.2-3B-Instruct, Phi-4-mini-Instruct, and Qwen-2.5-1.5B-Instruct), consisting of structured prompt engineering, explanation-augmented LoRA fine-tuning on 1,000-sample subsets, and two ensemble methods (confidence ensemble based on length-normalized logprobs, and majority vote). The authors report substantial gains on SpamAssassin and CEAS_08 over vanilla prompting, competitiveness with SVM/XGBoost baselines, and stronger transferability to unseen Enron and Ling datasets than traditional ML baselines. They also present an ablation showing that removing explanations degrades performance, and they list limitations on dataset coverage, transfer methods, cost quantification, and ensemble strategies.","tokens_in":10381,"tokens_out":4031,"duration_ms":40228,"significance":"If the reported results hold, the main contribution is a practical and interpretable recipe: explanation-augmented LoRA fine-tuning lets models with 1.5-3.8B parameters approach or exceed the accuracy of much larger zero-shot LLMs while retaining text explanations, and the transfer results suggest generalization beyond the training corpus. The ablation isolating the effect of explanation augmentation and the explicit limitation section are strengths. However, the paper ships no code, data splits, or hyperparameters, reports single runs without error bars, and does not verify that test sets are truly unseen; these omissions currently prevent verification of the central empirical claims. The paper's value is empirical rather than theoretical, so those missing pieces are material.","major_comments":[{"comment":"The transferability claim depends on Ling and Enron being truly unseen, yet the paper never demonstrates that the 1,000-sample fine-tuning subsets drawn from SpamAssassin and CEAS_08 are disjoint from the corresponding test sets or from the Ling and Enron evaluation sets. All datasets are cited from a single Kaggle aggregation (ref [12]), and public email corpora are known to contain duplicate or near-duplicate messages. Please add an overlap analysis using exact and normalized subject/body matching, report the number of overlapping messages, and rerun any affected results; otherwise the transfer gains could be inflated by memorization.","section":"Section 4.5 / Reference [12]"},{"comment":"There is an internal inconsistency in the reported improvements: Section 4.2 states that on SpamAssassin, LLaMA-3.2-3B-Instruct's F1 rose from 0.543 to 0.928, but Table 1 lists the SpamAssassin F1 as 0.354 and the value 0.543 is the CEAS_08 recall. This discrepancy must be corrected and explained, because it directly affects the claimed magnitude of the fine-tuning gain.","section":"Section 4.2 / Table 1"},{"comment":"Tables 1-4 report a single run per configuration with no error bars, and the text does not specify the LoRA rank r, scaling alpha, learning rate, batch size, number of epochs, sequence-length truncation, train/test split construction, or the number of random seeds. Training on only 1,000 samples makes split and seed variance potentially material to the reported deltas; releasing code, data splits, and hyperparameters is necessary to support the quantitative claims.","section":"Sections 4.2-4.4"},{"comment":"The claim that fine-tuned small LLMs 'outperform larger LLMs' compares LoRA-fine-tuned small models against zero-shot prompted GPT-3.5-Turbo, GPT-4o-mini, and LLaMA-3.1-70B-Instruct. This is not an apples-to-apples comparison; the large models are not fine-tuned on the same data. The claim should either be limited to zero-shot large-model baselines or the large models should be fine-tuned under comparable conditions.","section":"Section 4.2"},{"comment":"Explanation augmentation is the core intervention, but the paper provides no validation of the GPT-4o-mini-generated explanations: no human agreement study, no format-adherence check, and no analysis of whether explanation errors propagate into the fine-tuned model's decisions. The assertion that explanations 'mitigate hallucination' is unsupported. Please add at least a manual audit of a random sample of generated explanations and report the error rate.","section":"Section 3.3"}],"minor_comments":[{"comment":"The prompt text contains stray spaces ('Y ou', 'Y our') and Table 3 and Figure 1 contain 'V ote' instead of 'Vote'; these typos should be corrected.","section":"Section 3.2 / Table 3"},{"comment":"The phrase 'regular-sized LLMs' is imprecise; the paper earlier uses 'standard-sized LLMs,' and the terminology should be consistent.","section":"Section 4.5"},{"comment":"The paper does not specify the exact parsing rule for extracting '###Phishing###' or '###Safe###' from model output, nor what happens when a model outputs neither marker; please state the extraction rule and any fallback or rejection procedure.","section":"Section 3.2 / Section 4"},{"comment":"Table 3 reports only accuracy and F1 for the ensemble methods, while Table 2 includes precision and recall; adding precision and recall for the ensembles would not affect any conclusions but would improve comparability.","section":"Table 3"},{"comment":"Please describe the composition, label balance, and preprocessing of each dataset (SpamAssassin, CEAS_08, Ling, Enron), because class priors strongly affect F1 and accuracy interpretation.","section":"Reference [12] / Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"This is a straightforward empirical engineering paper with a potentially useful practical result, but the missing reproducibility artifacts, the unresolved overlap risk, and the inconsistent baseline number make the central claims currently unverifiable. If the authors provide code, data splits, deduplication/overlap analysis, multiple seeds, and corrected comparisons, the paper could become publishable at the claimed level of significance. No citation or novelty concerns beyond the reliance on a single Kaggle aggregation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper does one concrete thing: it takes three small LLMs, prompts them to reason, fine-tunes with LoRA on 1,000 emails augmented with GPT-4o-mini-generated explanations, and reports accuracy/F1 on SpamAssassin and CEAS_08 plus transfer to Ling and Enron. The headline numbers are plausible, and the transfer results would be the most interesting part if they hold: fine-tuned 3B models beat SVM and XGBoost on datasets they were not trained on. The paper is clearly written, and the authors are upfront that ensembling gives only modest gains on CEAS_08.\n\nWhat is new is narrow: the specific combination on these datasets. There is no new mechanism. That is fine for an applied paper, but it means the value rides entirely on the reliability of the measurements.\n\nThat reliability has real problems. No error bars, no code, no data, no hyperparameters; a single run per condition. Section 4.2 says LLaMA's F1 rose from 0.543 to 0.928, but Table 1 lists 0.354 for SpamAssassin; 0.543 is the CEAS_08 recall. That mismatch matters when precision is the point.\n\nThe transferability claim needs a contamination check. The paper cites one Kaggle dataset for SpamAssassin, CEAS_08, Ling and Enron. These public corpora are known to contain duplicate or near-duplicate messages. The authors never report deduplicating between the 1,000-sample fine-tuning subset and the test sets. If any overlap exists, the transfer numbers are memorization, not generalization. This is the soft spot that could sink the central claim.\n\nThere are two more minor issues. The comparison to GPT-4o-mini, GPT-3.5-Turbo and LLaMA-3.1-70B is zero-shot vs fine-tuned; the authors acknowledge it, but the discussion overreaches when it says small beats big. And the ablation is odd: no-EA fine-tuning makes LLaMA worse than vanilla prompting (F1 0.219 vs 0.354), which suggests the no-EA setup may be broken, not just missing explanations.\n\nWho is it for? People building practical phishing filters with local models, and researchers who want benchmark numbers for small-LLM detectors. It deserves a serious referee, but only with the artifacts and a contamination analysis. I'd send it out rather than desk-reject, and require code/data and the metric fix before acceptance.","headline":"Plausible applied result showing explanation-augmented LoRA fine-tuning helps small LLMs on phishing benchmarks, but thin reporting and no contamination check leave the transfer claim unverified.","tokens_in":10897,"tokens_out":4419,"would_cite":false,"duration_ms":41645,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that explanation-augmented LoRA fine-tuning lets small LLMs of 1.5–3.8 billion parameters match or exceed much larger models and traditional classifiers on phishing email detection.","keywords":["phishing email detection","small language models","explanation-augmented fine-tuning","LoRA","model ensemble","transferability","prompt engineering"],"falsifier":"Re-run the SpamAssassin and CEAS_08 experiments with the same 1,000 emails but with explanations randomly swapped across emails or replaced by correct labels with misleading reasoning; if accuracy stays near 0.96, the explanation texts are not causally responsible. Separately, measure n-gram overlap between training and test splits; high overlap would make the reported gains and the transfer claims difficult to interpret.","tokens_in":9931,"feed_emoji":"🎣","tokens_out":7287,"duration_ms":67095,"temperature":0.7,"pith_summary":"The paper tries to establish that small LLMs with roughly 1.5–3.8 billion parameters can become strong phishing detectors when fine-tuned with LoRA on a small set of emails whose labels are augmented with natural-language explanations. On SpamAssassin, Phi-4-mini-Instruct reaches 0.968 accuracy and 0.944 F1, and LLaMA-3.2-3B-Instruct reaches 0.963 and 0.928, beating the larger GPT-4o-mini and LLaMA-3.1-70B-Instruct in the authors' comparisons. The authors argue that the decisive ingredient is adding a human-readable explanation to each fine-tuning target so the task resembles the open-ended generation small LLMs were pretrained for; without the explanation, performance collapses. They also report that the fine-tuned models transfer to unseen Ling and Enron datasets better than SVM and XGBoost baselines. The upshot is a recipe for explainable phishing detection that runs on a single consumer-grade GPU.","feed_headline":"Fine-tuned 3B LLMs outperform 70B models on phishing","feed_subtitle":"Explanation-augmented LoRA fine-tuning lifts 1.5-3.8B models to ~0.97 accuracy on SpamAssassin, with transfer to unseen datasets.","key_machinery":"The load-bearing object is the explanation-augmented training instance: each email subject and body is paired with a label and a GPT-4o-mini-generated paragraph reasoning about whether it is a phishing attempt, and the small LLM is LoRA fine-tuned to reproduce both the reasoning and the label. This converts a closed-form classification target into an open-ended generative target, which the authors argue is better aligned with how the models were pretrained. The other components are a prompt that forces the final answer into a parseable '###Phishing###/###Safe###' wrapper, and a confidence ensemble that scores competing answers by the length-normalized geometric mean of token logprobabilities.","core_discovery":"On the paper's own account, the central discovery is that the form of the fine-tuning target matters as much as the amount of data. When small instruction-tuned LLMs are fine-tuned with LoRA on just 1,000 emails annotated with both a label and a natural-language explanation of that label, their phishing detection accuracy jumps: on SpamAssassin, Phi-4-mini-Instruct goes from 0.647 accuracy and 0.537 F1 under vanilla prompting to 0.968 and 0.944, and LLaMA-3.2-3B-Instruct goes from 0.587 and 0.354 to 0.963 and 0.928. On CEAS_08, Phi-4-mini reaches 0.917 accuracy and 0.927 F1. The ablation shows label-only fine-tuning is far weaker, with LLaMA-3.2-3B dropping to 0.571 accuracy and 0.219 F1 on SpamAssassin, while ensemble voting adds only a small further gain. Transfer experiments on unseen Ling and Enron data find that the fine-tuned small models generalize across domains better than SVM and XGBoost baselines.","pith_inferences":["If the mechanism is what the paper thinks, the fluency of generated explanations may matter more than their factual accuracy, because only the label is scored; a testable extension would fine-tune on deliberately wrong-but-plausible explanations and check whether accuracy survives.","The absence of contamination checks means the SpamAssassin gains could partly reflect dataset-specific artifacts; a clean test would train on a time-separated slice of emails and evaluate on later mail from the same organization.","The approach should extend naturally to other security text-classification tasks, such as malicious URL descriptions or smishing messages, where small explainable models are desirable, though that extension is not tested here."],"forward_implications":["On the paper's evidence, a 3-billion-parameter model fine-tuned this way can replace a 70-billion-parameter model for phishing triage at a fraction of the compute, while still giving a human-readable explanation for each decision.","Because the method needs only about 1,000 labeled emails and LoRA adapters, fine-tuning on an organization's own mailbox distribution is feasible on a single RTX 3090 GPU.","The ablation implies that label-only fine-tuning is not enough for small LLMs, so deployment should keep explanation augmentation in the training loop.","The transfer results suggest one fine-tuned adapter may serve multiple mail domains, since the small LLMs generalized to unseen Ling and Enron data better than embedding-based SVM and XGBoost baselines."],"supporting_citations":[{"why":"Supplies the LoRA low-rank update that makes fine-tuning feasible on a single GPU.","marker":"[27]"},{"why":"Generates the natural-language explanations that transform the training set into the central explanation-augmented format.","marker":"[16]"},{"why":"Supplies the sentence embeddings that feed the traditional machine-learning baselines the method is compared against.","marker":"[29]"},{"why":"Provides the 3-billion-parameter LLaMA model that is fine-tuned and evaluated.","marker":"[14]"},{"why":"Provides the Phi-4-mini model, which achieves the best reported results.","marker":"[15]"},{"why":"Provides the 1.5-billion-parameter Qwen model included as the smallest test subject.","marker":"[28]"},{"why":"Supplies the unseen Ling and Enron datasets used for the transferability evaluation.","marker":"[12]"},{"why":"Represents the prior large-model phishing detection result the paper positions itself against.","marker":"[25]"}],"fun_headline_variants":["Explanation fine-tuning turns 3B LLMs into phishing detectors matching 70B","Tiny fine-tuned LLMs rival 70B models on phishing detection","1k explanations unlock 3B LLMs for phishing: 0.97 accuracy","Small LLMs learn phishing fast with explanation-augmented LoRA","Transferable phishing detection from 3B LLMs via explanation fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result collapses if the large gains come not from the explanations themselves but from test-set contamination, dataset-specific artifacts in the 1,000-sample training slice, or unreliable generated explanations, none of which the paper tests for.","fun_headline_variants_meta":{"raw":{"variants":["Explanation fine-tuning turns 3B LLMs into phishing detectors matching 70B","Tiny fine-tuned LLMs rival 70B models on phishing detection","1k explanations unlock 3B LLMs for phishing: 0.97 accuracy","Small LLMs learn phishing fast with explanation-augmented LoRA","Transferable phishing detection from 3B LLMs via explanation fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000661,"raw_usage":{"total_tokens":3043,"prompt_tokens":988,"completion_tokens":2055,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":1954}},"tokens_in":604,"tokens_out":2055,"duration_ms":14709,"temperature":1.0,"reasoning_tokens":1954,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:19:31.407328+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the SpamAssassin and CEAS_08 experiments with the same 1,000 emails but with explanations randomly swapped across emails or replaced by correct labels with misleading reasoning; if accuracy stays near 0.96, the explanation texts are not causally responsible. Separately, measure n-gram overlap between training and test splits; high overlap would make the reported gains and the transfer claims difficult to interpret.","supporting_citations":[{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"Supplies the LoRA low-rank update that makes fine-tuning feasible on a single GPU."},{"cited_title":"GPT-4o-mini, https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence, 2024","cited_arxiv_id":null,"evidence_quote":"Generates the natural-language explanations that transform the training set into the central explanation-augmented format."},{"cited_title":"Llama 3.2, https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the 3-billion-parameter LLaMA model that is fine-tuned and evaluated."},{"cited_title":"Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras","cited_arxiv_id":null,"evidence_quote":"Provides the Phi-4-mini model, which achieves the best reported results."},{"cited_title":"Introducing Qwen, https://qwenlm.github.io/blog/qwen, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the 1.5-billion-parameter Qwen model included as the smallest test subject."},{"cited_title":"Phishing email dataset, https://www.kaggle.com/datasets/naserabdullahalam/phishing- email-dataset, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the unseen Ling and Enron datasets used for the transferability evaluation."}],"review_version":1}