{"id":"bcbddeff-b64e-49b2-955a-251d7c04b718","arxiv_id":"2501.04985","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuned Mixtral outperforms zero-shot and few-shot LLM prompting for SMS spam detection, reaching 98.6% accuracy on a balanced test set, though the evaluation protocol has methodological weaknesses.","lead":"This paper tests several large language models for SMS spam detection using zero-shot, few-shot, chain-of-thought, and fine-tuning. The authors report that a fine-tuned Mixtral model reaches 98.6% accuracy with balanced error rates, and that fine-tuning improves resilience to adversarial edits and concept drift.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 98.61% fine-tuned accuracy may be inflated by train/test duplication in the Super Dataset; the paper never de-duplicates or groups messages, so the central generalization claim is not yet established.","rationale":"The reader identified the random split as the weakest assumption, but attributed the bias mainly to few-shot example selection on the test set. I see a more direct and checkable threat to the central fine-tuning claim: dataset duplication. The paper's own description of the Super Dataset lists multiple aggregated sources with no mention of de-duplication, and SMS spam is highly templated, so a message-level random split is likely to leak campaign content into training. This would inflate all test numbers, including Table 5. The proposed near-duplicate removal test would settle whether the 98.61% accuracy reflects generalization or memorization. If the accuracy holds after de-duplication, the central claim is materially stronger; if it drops, the paper should be revised or the claim weakened. The adversarial robustness protocol (Table 10) also has internal inconsistencies when compared with Tables 1-5, but that is secondary to the accuracy claim. I keep the reader's CONDITIONAL verdict because the issue is testable and the paper can be corrected, but the concern is load-bearing rather than cosmetic.","tokens_in":26121,"tokens_out":6209,"duration_ms":57830,"concrete_test":"Download or reconstruct the Super Dataset, reproduce the Section 2.1 random split, and first compute exact-duplicate overlap between train and test. Then remove from the test set all messages that have a near-duplicate in training (e.g., normalized Levenshtein similarity >= 0.9 after stripping digits, URLs, and phone numbers), or better, hold out entire spam campaigns/sources. Re-run the Mixtral fine-tuning on the reduced training set and evaluate on the clean test set. If accuracy drops by more than about 2 percentage points or FNR rises above 3%, the published 98.61% is inflated by train/test leakage; if accuracy holds, the central claim survives this objection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 4, Table 5) is that fine-tuned Mixtral reaches 98.61% accuracy with FPR 0.96% and FNR 1.85%, satisfying the paper's self-defined 'Good' criteria. That estimate rests entirely on one random split of the Super Dataset into train and test (Section 2.1). The paper reports no de-duplication, near-duplicate removal, or grouping of messages by spam campaign or source. The Super Dataset aggregates UCI, NUS, Spamhunter, Twitter-sourced, and observatory-sourced messages, and real-world SMS spam is heavily templated: the same campaign is sent with slight text variants and repeated phone numbers or URLs. A message-level random split can therefore place near-identical content in both train and test. If so, the model may memorize campaign templates rather than learn a generalizable detector, and the 98.61% number would overstate performance on genuinely new messages. This concern is distinct from the reader's few-shot test-set-tuning point: it directly affects the fine-tuning result itself and is checkable from the published dataset. The missing QLoRA hyperparameters and error bars make the result harder to interpret, but the de-duplication check is the decisive one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates the effectiveness of several LLMs—GPT-4, DeepSeek, LLAMA-2, Mixtral, and Mistral—for SMS spam detection under zero-shot, few-shot, chain-of-thought, and fine-tuning settings. Using a combined 67,018-message dataset, the authors report that fine-tuned Mixtral (8×7B) achieves 98.61% accuracy with FPR 0.96% and FNR 1.85%, satisfying their self-defined 'Good' criteria. They also examine adversarial robustness against perceptible and imperceptible perturbations, and profile the best model under concept drift, concluding that fine-tuned LLMs are more robust than traditional ML baselines.","tokens_in":26395,"tokens_out":2827,"duration_ms":26194,"significance":"If the central claims hold, the paper would provide a practical demonstration that locally deployable open-source LLMs can serve as strong SMS spam detectors, with balanced error rates and improved resilience to adversarial manipulation and concept drift compared to conventional models. The paper's strengths include a broad model comparison, inclusion of both perceptible and imperceptible attack families, a dedicated concept-drift experiment, and its grounding in a recent dataset. However, the evaluation has several methodology gaps that directly affect the reliability of the reported numbers, particularly the use of the test set for few-shot example selection, the absence of de-duplication or campaign-level grouping in the train/test split, and missing fine-tuning hyperparameters and uncertainty estimates. These issues currently limit the credibility of the central accuracy claims.","major_comments":[{"comment":"The few-shot examples were selected by iterating on the evaluation set itself: \"The experiment was repeated multiple times for each LLM with different SMS messages containing similar keywords until the highest accuracy was no longer increased.\" Consequently, the reported few-shot accuracies (Tables 3, 4, 8) are optimized on the test set and are optimistic estimates of true few-shot performance. The authors should select examples using a separate development set or via cross-validation, and then report performance on the held-out test set without further iteration.","section":"Section 3.3.2"},{"comment":"The central fine-tuning result rests on a single random split of the Super Dataset into 5,000 spam and 5,000 ham test messages. The paper reports no de-duplication, near-duplicate removal, or grouping by spam campaign, source, or phone number/URL. Because the Super Dataset aggregates multiple sources and real-world SMS spam is heavily templated, a message-level random split can place near-identical content in both training and test sets, potentially inflating the fine-tuned accuracy of 98.61% (Table 5). The authors should quantify the extent of train/test leakage (e.g., exact or near-duplicate matching) and report results on a leakage-free split.","section":"Section 2.1 / Table 5"},{"comment":"The fine-tuning procedure is not reproducible as reported. The paper states that QLoRA was used but does not disclose the LoRA rank, learning rate, number of epochs, batch size, quantization configuration, or any regularization/seed settings. Without these details and without confidence intervals or multiple runs, the reader cannot assess the stability of the 98.61% accuracy figure or the claim that it generalizes. The authors should provide the hyperparameters and either run multiple seeds or report bootstrap confidence intervals.","section":"Section 4 / Table 5"},{"comment":"The adversarial robustness evaluation in Table 10 reports accuracy only on the 300-message holdout spam set; it does not measure false positive rates on legitimate messages after perturbation. Thus the claim that LLMs are 'robust' to adversarial manipulations is only about the preservation of spam detection, not about the overall classifier behavior (e.g., whether attacks also cause ham messages to be misclassified). The confidence in this robustness claim would be much higher if the authors reported full confusion-matrix metrics on a balanced ham/spam evaluation set under each attack.","section":"Section 5 / Table 10"}],"minor_comments":[{"comment":"The model name is inconsistent: 'Mistral (7B)' is sometimes written as 'Mixtral (7B)' (e.g., Table 3 rows); please correct to avoid confusion between the Mixtral mixture-of-experts model and the Mistral 7B model.","section":"Throughout"},{"comment":"Typo: 'finetunning' should be 'fine-tuning'.","section":"Section 2.2.3"},{"comment":"Typo: 'poortransferability' should be 'poor transferability'.","section":"Section 1"},{"comment":"The phrase 'refer to Section 5' for the train/test split should refer to Section 2.1, where the split is actually described.","section":"Section 4"},{"comment":"The text states that GPT-4 achieved a 'precision of 97.18%' in the medium-length few-shot setting; this should say 'accuracy' (as in Table 8).","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's few-shot example selection process, as described in Section 3.3.2, is a serious methodological issue that reviewers will likely flag. Additionally, the use of the authors' own Super Dataset without external validation may raise scope questions; although not fatal, independent replication on another SMS corpus would strengthen the claims. The paper has interesting breadth, but the central fine-tuning claim needs stronger evidence against data leakage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read. The paper is a broad, honest evaluation of six LLMs on SMS spam detection across zero-shot, few-shot, CoT, fine-tuning, adversarial robustness, and concept drift. That scope is genuinely useful; the field has plenty of BERT/ML baselines but few head-to-head LLM comparisons on a modern (2012-2023) dataset. The finding that zero-shot is unreliable, GPT-4 wins on few-shot/CoT, and fine-tuned Mixtral hits 98.61% accuracy with balanced FPR/FNR is the kind of concrete result practitioners would want. The robustness and concept drift sections are also well structured, and comparing against traditional ML baselines under the same drift protocol is the right call.\n\nThe soft spots are real but not uniform. The most serious is the stress-test concern: the Super Dataset is an aggregation of multiple public datasets plus Twitter and observatory sources, and real SMS spam is heavily templated. The paper does not de-duplicate, near-duplicate, or group by campaign before the random split. If near-identical messages appear in both train and test, the fine-tuning estimate is optimistic. This is checkable from the published dataset, and it is the first thing I would ask for.\n\nSecond, the few-shot ablation in Section 3.3.2 explicitly selects examples by iterating on a 400-message evaluation set until accuracy stops increasing. That turns the reported few-shot numbers into an optimistic in-sample estimate. It doesn't sink the fine-tuning claim, but it means the GPT-4 few-shot numbers should be read with caution.\n\nThird, minor but annoying: no QLoRA hyperparameters, no training details, no error bars, and no code release. That slows verification. Also, the adversarial robustness section's baseline is a bit underspecified—it's not clear whether the 'Original' row is the 300-message holdout or the full test set.\n\nNet: the central direction is plausible, and the paper deserves a serious referee. I would recommend conditional accept: require de-duplication/grouping analysis, a proper validation split for few-shot selection, and disclosure of fine-tuning hyperparameters. If the 98.61% survives the de-duplication check, this becomes a citable benchmark result.","headline":"Useful LLM-vs-SMS-spam benchmark, but the headline fine-tuning accuracy may be inflated by train/test duplication and the few-shot numbers are overfit; needs a data-cleaning pass before the 98.61% claim is credible.","tokens_in":26930,"tokens_out":2802,"would_cite":false,"duration_ms":27247,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning an open-source LLM, Mixtral (8×7B), yields an SMS spam detector with 98.61% accuracy and false positive/negative rates below 2%, and the same model resists adversarial edits and concept drift far better than classical baselines.","keywords":["SMS spam detection","large language models","fine-tuning","few-shot learning","chain-of-thought prompting","adversarial robustness","concept drift","Mixtral"],"falsifier":"Evaluate the same fine-tuned Mixtral checkpoint on a fresh test set of SMS messages collected after the training period, for example live 2024 carrier traffic, and check whether accuracy stays near 98.6% with false positive and false negative rates below 2%; a substantial drop would show the reported numbers are split-specific rather than a stable property of the fine-tuned model.","tokens_in":25922,"feed_emoji":"📱","tokens_out":5527,"duration_ms":48575,"temperature":0.7,"pith_summary":"The paper asks whether large language models can serve as reliable SMS spam detectors, comparing zero-shot, few-shot, chain-of-thought, and fine-tuned settings across commercial and open-source models. Its central claim is that fine-tuning makes open-source LLMs practical for this task: Mixtral (8×7B) reaches 98.61% accuracy on a balanced 5,000-spam/5,000-ham test set, with a false positive rate of 0.96% and a false negative rate of 1.85%, meeting the authors' 'Good' criteria. The study also finds that zero-shot detection is unreliable, few-shot results depend heavily on carefully selected examples and vary across models, and fine-tuned models are more resistant to adversarial text manipulations and to concept drift than classical machine-learning baselines. If the claims hold, a locally deployable open-source model could filter SMS spam with error rates balanced enough for real-world use, without sending message content to commercial APIs.","feed_headline":"Fine-tuned Mixtral catches SMS spam at 98.61% accuracy","feed_subtitle":"Open-source model hits balanced false-positive and false-negative rates under 2%, beating zero-shot and prompting approaches.","key_machinery":"The load-bearing mechanism is a comparative evaluation scaffold built on the Super Dataset (67,018 labeled SMS messages collected 2012–2023), split into a training set and a test set of 5,000 spam and 5,000 ham messages. Around this scaffold the paper layers four learning strategies: zero-shot prompting, few-shot prompting with exemplars tuned per model through an ablation study, chain-of-thought prompting, and QLoRA fine-tuning of selected open-source models. Performance is judged against three thresholds the authors define: Satisfactory (FPR and FNR below 5%), Good (below 3%), and Perfect (below 1%). Adversarial robustness is measured with six perceptible attacks implemented by extending TextAttack (spacing, character insertion, deletion, swap, substitution, and word substitution) plus three imperceptible attacks (invisible characters, homoglyphs, and bidirectional reorderings); concept drift is measured by training on the 2012 UCI dataset and testing on newer spam against five classical baselines.","core_discovery":"On the paper's own terms, the discovery is that parameter-efficient fine-tuning transforms open-source LLMs from mediocre zero-shot classifiers into deployable SMS spam detectors. Fine-tuning LLAMA-2 (70B), LLAMA-2 (13B), and Mixtral (8×7B) with QLoRA improves all three substantially over their zero- and few-shot results; Mixtral achieves 98.61% accuracy with a true positive rate of 98.15% and a true negative rate of 99.04%, corresponding to a false positive rate of 0.96% and a false negative rate of 1.85%, the only open-source model that keeps both errors below 3%. The paper further claims that fine-tuned Mixtral maintains 90.4% accuracy when trained on an older UCI dataset and tested on 2020–2023 spam, far above the 53–64% range of the SVM, random forest, LightGBM, LSTM, and CNN baselines under the same drift, and rises to 96.2% when five recent spam examples are added in a few-shot prompt. In adversarial settings, the fine-tuned Mixtral reaches 100% accuracy on several perceptible attacks and stays above 97% on imperceptible Unicode and homoglyph attacks.","pith_inferences":["If the fine-tuning numbers are stable, network operators could refresh a small set of few-shot exemplars periodically instead of full retraining; the concept-drift experiment hints at this direction but does not test a repeated update schedule.","The few-shot results were obtained by selecting examples after iterating on the same test set until accuracy stopped improving, so those numbers likely overstate what a practitioner would achieve without test-set access; a clean validation-split design would yield a more conservative estimate.","The robustness tests transform only the spam messages themselves, so they do not cover an adversary who crafts messages specifically against the fine-tuned detector; a dedicated adaptive attack could target Mixtral's blind spots.","A testable extension is whether smaller open-source models (7B–13B) can reach Mixtral-level balance with more data or longer fine-tuning, which would lower the hardware cost of deployment."],"forward_implications":["A fine-tuned open-source LLM can be deployed locally as an SMS spam filter, avoiding the privacy and latency costs of sending messages to a commercial API.","Zero-shot LLM spam detection is not reliable enough for deployment, and few-shot performance without careful per-model example selection is inconsistent.","Fine-tuning improves adversarial robustness, with Mixtral reaching perfect accuracy on several perceptible attack types after adaptation.","LLM-based spam detectors drift much less than classical models: Mixtral trained on old data and tested on new spam keeps 90.4% accuracy, versus 53–64% for conventional baselines.","Combining fine-tuning with few-shot examples drawn from recent spam can further close the concept-drift gap, lifting Mixtral from 90.4% to 96.2% accuracy."],"supporting_citations":[{"why":"Provides the Super Dataset used for training and testing and documents the vulnerability of classical ML spam detectors to adversarial and concept-drift degradation.","marker":"[35]"},{"why":"QLoRA is the parameter-efficient fine-tuning method that made adapting 70B-scale open-source models feasible on local hardware.","marker":"[10]"},{"why":"Supplies the older UCI SMS spam dataset used as the training source in the concept-drift experiment.","marker":"[5]"},{"why":"Defines the imperceptible non-printable-Unicode and homoglyph attacks used in the adversarial robustness evaluation.","marker":"[6]"},{"why":"The TextAttack framework the authors extended to generate the six perceptible adversarial perturbations.","marker":"[27]"},{"why":"Contributes recent Spamhunter messages to the Super Dataset and defines the fraud and promo spam categories used for few-shot exemplar selection.","marker":"[37]"},{"why":"Introduces chain-of-thought prompting, the reasoning strategy evaluated alongside zero-shot and few-shot learning.","marker":"[39]"},{"why":"LoRA is the low-rank adaptation basis that QLoRA builds on for memory-efficient fine-tuning.","marker":"[17]"}],"fun_headline_variants":["Fine-tuned Mixtral hits 98.6% SMS spam detection","Fine-tuning LLMs beats SMS spam drift at 90.4%","Open-source Mixtral fine-tuned: 98.6% spam accuracy","SMS spam: fine-tuned Mixtral robust to attacks, 98.6% acc","Fine-tuned Mixtral: 98.6% spam, under 2% false rates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy and balanced error rates assume that the single random split of the Super Dataset into 5,000 spam and 5,000 ham test messages gives an unbiased estimate of how the fine-tuned models will behave on new SMS messages; the few-shot numbers additionally assume that selecting examples by iterating on test-set performance does not inflate them.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned Mixtral hits 98.6% SMS spam detection","Fine-tuning LLMs beats SMS spam drift at 90.4%","Open-source Mixtral fine-tuned: 98.6% spam accuracy","SMS spam: fine-tuned Mixtral robust to attacks, 98.6% acc","Fine-tuned Mixtral: 98.6% spam, under 2% false rates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1907,"prompt_tokens":1078,"completion_tokens":829,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":694,"completion_tokens_details":{"reasoning_tokens":718}},"tokens_in":694,"tokens_out":829,"duration_ms":7411,"temperature":1.0,"reasoning_tokens":718,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:21:24.600066+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the same fine-tuned Mixtral checkpoint on a fresh test set of SMS messages collected after the training period, for example live 2024 carrier traffic, and check whether accuracy stays near 98.6% with false positive and false negative rates below 2%; a substantial drop would show the reported numbers are split-specific rather than a stable property of the fine-tuned model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Super Dataset used for training and testing and documents the vulnerability of classical ML spam detectors to adversarial and concept-drift degradation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The TextAttack framework the authors extended to generate the six perceptible adversarial perturbations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes recent Spamhunter messages to the Super Dataset and defines the fraud and promo spam categories used for few-shot exemplar selection."}],"review_version":1}