Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Machine Learning Driven Smishing Detection Framework for Mobile Security

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding NoSlang text normalization to a Naive Bayes classifier raises smishing detection accuracy from 88.2% to 96.2% on the authors' curated SMS dataset.

desk verdict The 8-point normalization gain rests on 40 undocumented test-message labels; the idea is thin and the evaluation is not reproducible. read the letter →

arxiv 2412.09641 v1 pith:NOQG2WS7 submitted 2024-12-09 cs.CR cs.LG

Diksha Goel , Hussain Ahmad , Ankit Kumar Jain , Nikhil Kumar Goel This is my paper · ORCID
classification cs.CRcs.LG
keywords smishingdetectionSMSspamfilteringtextnormalizationNoSlangdictionaryNaiveBayesclassifiermobilesecurityshortmessageservice
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a content-based smishing detector should normalize informal SMS language before classification, not feed raw text to the learner. Its core claim is that replacing slang and abbreviations with standard forms through the NoSlang dictionary lets a Naive Bayes classifier lift accuracy from 88.2% to 96.2%, with the true positive rate rising from 94.28% to 97.14% and the true negative rate from 87.74% to 96.12%. If this is right, a cheap preprocessing step delivers a substantial gain without changing the classifier, which matters because SMS is a high-open-rate channel and smishing is a growing attack vector.

What carries the argument

The load-bearing mechanism is a two-phase pipeline: a preprocessing phase that applies the NoSlang dictionary to convert slang, abbreviations, and nonstandard spellings into standard forms, then removes stop words and stems, followed by a Naive Bayes classification phase that estimates per-word probabilities of appearing in ham versus smishing messages and applies Bayes' theorem to score each message. The NoSlang dictionary is the component that distinguishes this framework from earlier content-based filters; its job is to collapse linguistic variability so that terms like 'ur' and 'your' count as the same feature. The authors quantify the dictionary's contribution by comparing the same classifier with and without the normalization step.

What would settle it

Have two or more independent annotators, working only from a public definition of smishing, label the same SMS Spam Collection v.1 spam messages and the 71 Pinterest messages; if their labels disagree with the authors' extraction, or if a classifier trained on the consensus labels fails to reproduce the 96.2% accuracy, the claimed normalization gain is not robust.

Watch

Extended reading notes

Core claim

On a dataset built by manually extracting smishing messages from the SMS Spam Collection v.1 spam category and adding 71 smishing messages from Pinterest (4,807 ham and 362 smishing in total), the proposed framework first tokenizes, lowercases, normalizes nonstandard terms with the NoSlang dictionary, removes stop words, and stems. A Naive Bayes classifier trained on the normalized text then classifies each message by comparing the posterior probability that it is smishing against the posterior probability that it is ham. The authors report that this pipeline reaches 96.2% accuracy, 97.14% TPR, 96.12% TNR, a false positive rate of 3.87%, and a false negative rate of 2.85%, compared with 88.2% accuracy, 94.28% TPR, and 87.74% TNR without normalization. The claim is that the normalization step, not the classifier choice, drives the improvement.

Load-bearing premise

The central claim rests on the validity of the authors' manual labels that picked 362 smishing messages out of the SMS Spam Collection v.1 spam category, since every accuracy and rate is computed against those labels and no annotation protocol or inter-rater check is reported.

Editorial extensions

If this is right

  • Normalization becomes a standalone, cheap upgrade: any content-based SMS filter could add the NoSlang step without replacing its classifier.
  • The improvement to TNR from 87.74% to 96.12% implies that normalization cuts false alarms on legitimate messages, which matters for user trust in a mobile filter.
  • The drop in FPR from 12.25% to 3.87% means flagged smishing messages are far more likely to be genuinely malicious, reducing the review burden.
  • Because the dataset is publicly derived, the 96.2% figure gives other researchers a concrete baseline to beat on the same 5,169-message set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same normalization step may transfer to other short-text classifiers such as SVM or random forests, and to informal-language domains like chat-app phishing, but the paper only tests Naive Bayes.
  • A natural next experiment would hold the classifier fixed and vary only the dictionary, comparing NoSlang against a domain-specific banking or financial lexicon to isolate where the eight-point gain comes from.
  • The manual labeling step is the most audit-prone part: independent re-annotation of the spam category could shift the reported numbers, and a published annotation protocol would make the improvement reproducible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a content-based smishing detection framework that applies a NoSlang dictionary for text normalization, followed by a Naive Bayes classifier, and reports that normalization improves accuracy from 88.2% to 96.2%, with TPR increasing from 94.28% to 97.14% and TNR from 87.74% to 96.12%. The evaluation uses a modified version of the SMS Spam Collection v.1 dataset, for which the authors manually extracted smishing messages from the spam category and added 71 messages from a Pinterest source. The paper also presents comparisons with two unnamed baseline systems and concludes that the proposed framework outperforms existing methods.

Significance. If the reported improvement were robust, the contribution would be a simple and potentially useful preprocessing step for content-based smishing detection, and the use of a public base dataset is a strength. The paper also gives clear algorithmic descriptions of the preprocessing and classification pipeline. However, the significance is currently limited by evaluation issues: the ground-truth labels are manually curated with no documented criteria, the baseline systems are unnamed, the training split counts are inconsistent with the stated dataset sizes, and the reported gain rests on a very small number of test messages. These issues must be resolved before the result can be taken as evidence of a general improvement.

major comments (5)
  1. [Section V-A / V-B, Algorithm 2] The reported training counts (4,342 ham and 327 smishing) are not 90% of the stated dataset sizes (4,807 ham and 362 smishing), which would be approximately 4,326.3 ham and 325.8 smishing. The paper should state the exact split procedure and raw test-set counts; as written, the discrepancy prevents independent verification of the reported metrics.
  2. [Section V-B, Tables IX and XI] The entries labeled "Baseline 1" and "Baseline 2" in Tables IX and XI are never named or cited. Without identifying these existing systems and their configurations, the comparative claim that the proposed framework "significantly outperforms existing methods" is unsupported. The authors should specify the baselines and provide citations or descriptions.
  3. [Section V-A] The ground-truth smishing labels are produced by "manually extracting smishing messages from the spam category" and adding 71 Pinterest messages, but no annotation protocol, inclusion criteria, or inter-rater reliability is reported. Because every reported metric is computed against these labels, the central 8-percentage-point accuracy improvement is no stronger than the undocumented label curation. The authors should provide the annotation guidelines and, ideally, release the labeled dataset.
  4. [Section V-B, Table X] The reported TPR improvement from 94.28% to 97.14% corresponds to a change from 33 to 34 true positives out of 35 smishing test messages under the implied 500-message test set (465 ham, 35 smishing), i.e., a single-message effect. The paper should report the raw confusion matrices and confidence intervals; otherwise, the claim of a significant TPR improvement is not supported.
  5. [Section V-B, Algorithm 2] The evaluation uses a single 90/10 split with no cross-validation, repeated trials, or statistical significance testing. Given the small smishing test set and the sensitivity of the result to a few messages, the robustness of the 96.2% accuracy claim is not established. The authors should use k-fold cross-validation or bootstrap resampling and report variance.
minor comments (5)
  1. [Section V-A] The text states that SMS Spam Collection v.1 contains 4,827 ham messages, but the final dataset is reported as 4,807 ham; the source of this 20-message reduction should be explained.
  2. [Table X] The TPR and FNR values do not exactly sum to 100% because of rounding (94.28% + 5.71% = 99.99%); the paper should state that values are rounded and perhaps give FPR/FNR to consistent precision.
  3. [Algorithm 2, line 16] The comparison operator appears as "¿" instead of ">" in the pseudocode.
  4. [References [25], [26]] The NoSlang dictionary and the Pinterest source are cited as web pages with access dates from 2017; for reproducibility, provide stable snapshots or archived versions of these resources.
  5. [Abstract and Section V-A] The abstract says the results are "validated on a publicly available dataset," but the evaluation actually uses a manually curated subset of that dataset plus 71 external messages; this should be stated explicitly wherever the dataset is described.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the reported accuracy gain is an empirical measurement on an external SMS dataset, not a quantity forced by the method's definitions or by a self-citation chain.

full rationale

The paper's central claim is that NoSlang normalization improves Naive Bayes smishing detection from 88.2% to 96.2% accuracy. This is an experimental result, not a derived theorem. Algorithm 1 is a deterministic preprocessing pipeline, Algorithm 2 is a standard Naive Bayes training and classification procedure, and Table X reports measured confusion-matrix metrics before and after normalization on a 90/10 train/test split. No parameter is fitted to the test set and then renamed as a prediction; the normalization dictionary is an external resource (NoSlang, reference [25]) and is not fitted to the dataset. The dataset itself is external (SMS Spam Collection v.1 plus a Pinterest smishing collection), although the authors manually extract smishing labels from the spam category; that manual curation is a data-validity concern, not a circularity, because the accuracy numbers are claims about classifier performance relative to those labels, not derivations of the labels from the algorithm's outputs. The paper contains several self-citations, including references [7], [8], [38], [39], [40], and [41], but none is load-bearing for the normalization effect: the key components are the external NoSlang dictionary, the standard Bayes classifier, and the measured confusion matrices. No pattern of self-definition, fitted-input-as-prediction, uniqueness imported from authors, ansatz smuggled in via citation, or renaming of a known result is present. The evaluation may be limited by undocumented label curation and lack of a close normalization baseline, but those are correctness and external-validity risks, not circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small, manually curated dataset, a slang dictionary, and a simple Naive Bayes model. The only fitted numbers are the classifier's per-word probabilities; the only hand-chosen setting is the single 90/10 split. No new entities are introduced.

free parameters (2)
  • Naive Bayes word probability table = learned from training data, not tabulated fully
    The classifier's output is determined by per-word probabilities p(w|ham) and p(w|smish) computed in Algorithm 2; these are fitted to the authors' curated training split. The paper only shows nine terms.
  • Train/test split fraction and split choice = 90/10, single split
    The 90/10 split is a hand-chosen number and no cross-validation is used. The reported accuracy depends on this split, and the stated training counts (4,342 ham and 327 smishing) do not match 90% of the stated totals (4,807 ham and 362 smishing).
assumptions (4)
  • domain assumption The NoSlang dictionary provides sufficient normalization coverage for SMS language in the dataset.
    Invoked in Section IV-A as the normalization resource; if many slang or abbreviations are missing, the claimed benefit of normalization is overstated.
  • domain assumption The SMS Spam Collection v.1 labels are correct and the manual extraction of smishing messages from the spam category is valid.
    Section V-A builds the ground truth by manual extraction; no criteria or inter-rater agreement is provided.
  • domain assumption The 'without normalization' condition is a valid control, meaning the only difference is removal of dictionary-based normalization.
    Section V-B compares the same Naive Bayes pipeline with and without normalization; the paper does not state whether all other preprocessing steps, such as stop word removal and stemming, are identical.
  • standard math Naive Bayes treats words as conditionally independent given the class.
    Equation (1) and Algorithm 2 assume word independence, a standard but often false modeling assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Machine Learning Driven Smishing Detection Framework for Mobile Security." pith.science (2026). https://pith.science/paper/NOQG2WS7

@misc{pith2026241209641,
  author       = {Pith},
  title        = {Pith review of: Machine Learning Driven Smishing Detection Framework for Mobile Security},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NOQG2WS7}},
  note         = {Machine review of arXiv:2412.09641}
}
read the original abstract

The increasing reliance on smartphones for communication, financial transactions, and personal data management has made them prime targets for cyberattacks, particularly smishing, a sophisticated variant of phishing conducted via SMS. Despite the growing threat, traditional detection methods often struggle with the informal and evolving nature of SMS language, which includes abbreviations, slang, and short forms. This paper presents an enhanced content-based smishing detection framework that leverages advanced text normalization techniques to improve detection accuracy. By converting nonstandard text into its standardized form, the proposed model enhances the efficacy of machine learning classifiers, particularly the Naive Bayesian classifier, in distinguishing smishing messages from legitimate ones. Our experimental results, validated on a publicly available dataset, demonstrate a detection accuracy of 96.2%, with a low False Positive Rate of 3.87% and False Negative Rate of 2.85%. This approach significantly outperforms existing methodologies, providing a robust solution to the increasingly sophisticated threat of smishing in the mobile environment.

Figures

Figures reproduced from arXiv: 2412.09641 by the authors.

Figure 1
Figure 1. Sequence Diagram of a Smishing Attack [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Examples of Smishing Messages 2) Use of Fake Phone Numbers: Users are asked to confirm their identity by calling a provided number, which connects them to a fraudulent call centre. Here, attackers use social engineering to extract sensitive information. 3) Impersonation of Known Entities: Attackers impersonate individuals or entities familiar to the user, such as friends or colleagues, to establish trust and gather … view at source ↗
Figure 3
Figure 3. Architecture of the Smishing Detection Framework. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example of Preprocessing and Normalization. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Future of AI: Exploring the Potential of Large Concept Models

    cs.CL 2025-01 conditional novelty 1.0 of 10

    A survey of grey literature describing Meta's Large Concept Models, their proposed advantages over token-based LLMs, and their speculative applications across many industries.

Reference graph

Works this paper leans on

41 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [23]

    A., Silva, T

    Almeida, T. A., Silva, T. P., Santos, I., & Hidalgo, J. M. G. (2016). Text normalization and semantic indexing to enhance Instant Messaging and SMS spam filtering. Knowledge-Based Systems, 108, 25-32

    2016

  2. [24]

    Kaur, H., & Mann, E. J. S. (2016). Text Normalization using Statistical Machine Approach

    work page 2016

  3. [1]

    Verkijika, S. F. (2018). Understanding smartphone security behaviours: An extension of the protection motivation theory with anticipated regret. Computers & Security

    work page 2018

  4. [2]

    B., Tewari, A., Jain, A

    Gupta, B. B., Tewari, A., Jain, A. K., & Agrawal, D. P. (2017). Fighting against phishing attacks: state of the art and future challenges. Neural Computing and Applications, 28(12), 3629-3654

    work page 2017

  5. [3]

    W., Moon, S

    Joo, J. W., Moon, S. Y ., Singh, S., & Park, J. H. (2017). S-Detector: an enhanced security model for detecting Smishing attacks for mobile computing. Telecommunication Systems, 1-10

    work page 2017

  6. [4]

    Mishra, S., & Soni, D. (2019). A content-based approach for detecting smishing in a mobile environment. In Proceedings of International Confer- ence on Sustainable Computing in Science, Technology and Management (SUSCOM), Amity University Rajasthan, Jaipur-India

    work page 2019

  7. [5]

    W., Lee, J

    Seo, J. W., Lee, J. S., Kim, H., Lee, J., Han, S., Cho, J., & Lee, C. H. (2024). On-Device Smishing Classifier Resistant to Text Evasion Attack. IEEE Access

    work page 2024

  8. [6]

    N., Gbenle, O., Abikoye, O

    Akande, O. N., Gbenle, O., Abikoye, O. C., Jimoh, R. G., Akande, H. B., Balogun, A. O., & Fatokun, A. (2023). SMSPROTECT: An automatic smishing detection mobile application. ICT Express, 9(2), 168-176

    work page 2023

Show all 41 references
  1. [7]

    Goel, D., & Jain, A. K. (2017). Mobile phishing attacks and defence mechanisms: state of the art and open research challenges. Computers & Security

    2017

  2. [8]

    Abdulsatar, M., Ahmad, H., Goel, D., & Ullah, F. (2024). Towards Deep Learning Enabled Cybersecurity Risk Assessment for Microservice Architectures. arXiv preprint arXiv:2403.15169

    2024 arXiv

  3. [9]

    K., Ahmad, H., Goel, D., Syed, M

    Jayalath, R. K., Ahmad, H., Goel, D., Syed, M. S., & Ullah, F. (2024). Microservice Vulnerability Analysis: A Literature Review with Empirical Insights. IEEE Access

    2024

  4. [10]

    Infosec Institute, Smishing, 2018, http://resources.infosecinstitute.com/ category/enterprise/phishing/phishing-variations/phishing-variations- smishing/#gref, Accessed February 2018

    2018

  5. [11]

    Number of Smartphone users worldwide from 2014 to 2020, 2018, https://www.statista.com/statistics/330695/number-of-smartphone- users-worldwide/.Accessed March 2018

    2014

  6. [12]

    State of the Phish, 2018, https://www.wombatsecurity.com/hubfs/2018% 20State%20of%20the%20Phish/Wombat- StateofPhish2018.pdf?submissionGuid=2f8a7968-4f1e-4578-ba7c- 127516efa6fb, Accessed March 2018

    2018

  7. [13]

    Deepak, G., & Pradeep, B. S. (2012). Challenging issues and limitations of mobile computing. International Journal of Computer Technology & Applications, 3(1), 177-181

    2012

  8. [14]

    (2011, March)

    Yadav, K., Kumaraguru, P., Goyal, A., Gupta, A., & Naik, V . (2011, March). Smsassassin: Crowdsourcing-driven mobile-based system for sms spam filtering. In Proceedings of the 12th Workshop on Mobile Computing Systems and Applications (pp. 1-6). ACM

    2011

  9. [15]

    El-Alfy, E. S. M., & AlHasan, A. A. (2016). Spam filtering framework for multimodal mobile communication based on dendritic cell algorithm. Future Generation Computer Systems, 64, 98-107

    2016

  10. [16]

    Accessed February 2017

    Smishing Defender, http://www.hauri.co.kr/support/hauriNews view.html? intSeq=303&page=7&keyfield=&key. Accessed February 2017

    2017

  11. [17]

    Lee, A., Kim, K., Lee, H., & Jun, M. (2016). A Study on Realtime Detecting Smishing on Cloud Computing Environments. In Advanced Multimedia and Ubiquitous Engineering (pp. 495-501). Springer, Berlin, Heidelberg

    2016

  12. [18]

    Ahmad, H., Treude, C., Wagner, M., & Szabo, C. (2024). Towards Resource-Efficient Reactive and Proactive Auto-Scaling for Microservice Architectures. Available at SSRN 4918202

    2024

  13. [19]

    K., & Gupta, B

    Jain, A. K., & Gupta, B. B. (2018). Rule-Based Framework for Detec- tion of Smishing Messages in Mobile Environment. Procedia Computer Science, 125, 617-623

    2018

  14. [20]

    Karami, A., & Zhou, L. (2014). Improving static SMS spam detection by using new content-based features

    2014

  15. [21]

    Mishra, S., & Soni, D. (2020). Smishing Detector: A security model to detect smishing through SMS content analysis and URL behaviour analysis. Future Generation Computer Systems, 108, 803-815

    2020

  16. [22]

    M., Almeida, T

    Silva, R. M., Almeida, T. A., & Yamakami, A. (2017). MDLText: An efficient and lightweight text classifier. Knowledge-Based Systems, 118, 152-164

    2017

  17. [25]

    http://www.noslang.com/dictionary/full/, Accessed December 2017

    NoSlang: Internet Slang Dictionary & Translator. http://www.noslang.com/dictionary/full/, Accessed December 2017

    2017

  18. [26]

    https://in.pinterest.com/seceduau/smishing- dataset/?lp=true

    Smishing message images. https://in.pinterest.com/seceduau/smishing- dataset/?lp=true. Accessed November 2017

    2017

  19. [27]

    S., Sanghi, D., & Conti, M

    Somani, G., Gaur, M. S., Sanghi, D., & Conti, M. (2016). DDoS attacks in cloud computing: collateral damage to non-targets. Computer Networks, 109, 157-171

    2016

  20. [28]

    Ahmad, H., Treude, C., Wagner, M., & Szabo, C. (2024). Smart HPA: A Resource-Efficient Horizontal Pod Auto-scaler for Microservice Architectures. arXiv preprint arXiv:2403.07909

    2024 arXiv

  21. [29]

    Chopra S., Ahmad, H., Goel, D., & Szabo, C. (2024). ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Models. arXiv preprint arXiv:2412.04756

    2024 arXiv

  22. [30]

    Ahmad, H., Dharmadasa, I., Ullah, F., & Babar, M. A. (2023). A review on c3i systems’ security: Vulnerabilities, attacks, and countermeasures. ACM Computing Surveys, 55(9), 1-38

    2023

  23. [31]

    Mouton, F., Leenen, L., & Venter, H. S. (2016). Social engineering attack examples, templates and scenarios. Computers & Security, 59, 186-209

    2016

  24. [32]

    K., & Gupta, B

    Jain, A. K., & Gupta, B. B. (2017). Phishing Detection: Analysis of Visual Similarity Based Approaches. Security and Communication Networks, 2017

    2017

  25. [33]

    S., Agyemang, J

    Awumee, G. S., Agyemang, J. O., Boakye, S. S., & Bempong, D. (2023, October). SmishShield: A Machine Learning-Based Smishing Detection System. In International Conference on Wireless Intelligent and Distributed Environment for Communication (pp. 205-221). Cham: Springer Nature...

    2023

  26. [34]

    E., Kitulgoda, T

    Kohilan, R., Warakagoda, H. E., Kitulgoda, T. T., Skandhakumar, N., & Kuruwitaarachchi, N. (2023, November). A Machine Learning-based Approach for Detecting Smishing Attacks at End-user Level. In 2023 IEEE International Conference on e-Business Engineering (ICEBE) (pp. 149-154). IEEE

    2023

  27. [35]

    Sender. (2024). SMS Open Rates: Everything You Need to Know. Retrieved from https://www.sender.net/blog/sms-open-rates/

    2024

  28. [36]

    Brewer, R. (2016). Ransomware attacks: detection, prevention and cure. Network Security, 2016(9), 5-9

    2016

  29. [37]

    K., Panday, A., and Goel, D

    Jain 1, A. K., Panday, A., and Goel, D. (2024). S-Defender: A Smishing Detection Approach. In *Cyber Warfare, Security and Space Computing: Second International Conference on Cyber Warfare, Security and Space Computing, SpacSec 2024, Jaipur, India, February 22–23, 2024, Procee...

    2024

  30. [38]

    Goel, D., and Jain, A. K. (2018). Smishing-classifier: A novel framework for detection of smishing attack in mobile environment. In *Smart and Innovative Trends in Next Generation Computing Technologies: Third International Conference, NGCT 2017, Dehradun, India, October 30-31...

    2018

  31. [39]

    K., Goel, D., Agarwal, S., Singh, Y ., and Bajaj, G

    Jain, A. K., Goel, D., Agarwal, S., Singh, Y ., and Bajaj, G. (2020). Pre- dicting spam messages using back propagation neural network. *Wireless Personal Communications*, **110**, 403–422

    2020

  32. [40]

    Goel, D., and Jain, A. K. (2018). Overview of smartphone security: Attack and defense techniques. In *Computer and Cyber Security*, pages 249–279. Auerbach Publications

    2018

  33. [41]

    Goel, D. (2023). Enhancing network resilience through machine learning-powered graph combinatorial optimization: Applications in cyber defense and information diffusion. *arXiv preprint arXiv:2310.10667*

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.