POSTER: A Multi-Signal Model for Detecting Evasive Smishing

Sanchari Das; Shaghayegh Hosseinpour

REVIEW 4 major objections 5 minor 19 references

POSTER: A Multi-Signal Model for Detecting Evasive Smishing

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that fusing semantic, structural, character-level, and phrase-context signals into one architecture detects evasive smishing at 97.89% accuracy, with F1 0.963 and AUC 99.73%, beating each single-stream model.

desk verdict A plausible ensemble for smishing with a real ablation, but the headline accuracy is defined by the authors' own keyword list, making the central claim circular; useful as a work-in-progress poster, not as a reliable detector. read the letter →

arxiv 2505.18233 v2 pith:I5L2QRAU submitted 2025-05-23 cs.LG cs.AI

Shaghayegh Hosseinpour , Sanchari Das This is my paper · ORCID

classification cs.LGcs.AI

keywords smishingSMSphishingmulti-signallearningcharacter-levelCNNnamedentityrecognitioncontextualembeddingsmobilesecuritydetection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that smishing—phishing sent by SMS—evades single-stream detectors because it is short, informal, regionally adapted, and rich in structural tricks. To catch it, the authors build a unified model that fuses four independent views: country-aware semantic tags, structural placeholders for URLs and phone numbers, character-level style, and contextual phrase embeddings. On a relabeled corpus of more than 84,000 messages, the fused model reaches 97.89% accuracy, a 0.963 F1 score, and 99.73% AUC, beating each stream on its own. The practical point is that no one feature type is enough; the combination, and especially the character and structural channels, carries the performance. The stated aim is a modular, region-aware detector that can partly run on-device.

What carries the argument

The load-bearing machinery is a four-stream feature fusion pipeline. Each message is processed in parallel: (1) an NER semantic stream that replaces countries, organizations, and currencies with abstract tags and adds the detected country name; (2) a regex structural stream that replaces URLs, emails, and phone numbers with placeholders; (3) a character-level CNN (CharCNN) over dense character embeddings; and (4) a DistilBERT contextual stream, with curated phrases tagged [legitimate_like] or [smishing_like], whose [CLS] embedding feeds a CNN. Each stream's vector is reduced with Truncated SVD, the four reduced vectors are concatenated, and a multi-layer perceptron with dropout and attention classifies the result. The mechanism that carries the argument is this late-fusion design: it lets each channel contribute a different kind of evidence instead of forcing one model to learn all cues.

What would settle it

Annotate a random sample of the 24,086 smishing-labeled messages with independent human judges who do not see the keyword list, then compare agreement and measure the model's accuracy on messages containing none of the keyword-smishing words; if performance collapses on keyword-free messages or human agreement is low, the labels—and therefore the 97.89% claim—are artifacts of the keyword rule.

Watch

Extended reading notes

Core claim

The paper's central claim is that a four-channel fusion architecture reliably detects evasive smishing and outperforms every single-feature model trained on the same data. In their experiments, the combined model reaches 97.89% accuracy, 0.963 F1, and 99.73% AUC; individually the best stream (character-level CharCNN) reaches 96.62% accuracy, and the contextual phrase stream lags at 88.29%. The authors attribute the gain to complementary signals: NER-based country tags catch region-specific scams, regex structural tags capture intent patterns such as embedded URLs and phone numbers, CharCNN captures obfuscation and emoji, and phrase embeddings catch repeated benign-sounding phishing language. Ablations show measurable drops when any stream is removed, especially character and structural channels.

Load-bearing premise

The load-bearing premise is that the smishing labels are valid: messages were relabeled as smishing when spam messages contained frequent smishing words from the authors' own keyword list [7], and the same kind of phrase evidence is later injected as [smishing_like] tags in the contextual stream; if that keyword rule is not a valid proxy for smishing, the reported accuracy measures detection of the rule rather than of evasive smishing.

Editorial extensions

If this is right

Deploying the fused model in a real SMS gateway should yield higher recall on evasive, region-specific smishing than any single-stream filter it replaces.
Lightweight components such as structural regex tagging and CharCNN can run on-device, while semantic processing is handled in the cloud, so the modular design is compatible with mobile deployment.
The feature streams are modular: replacing the NER tagger or the phrase list lets an operator adapt the same architecture to a new country or language without retraining the whole model.
Ablation results imply that defensive systems should not rely on contextual embeddings alone; character-level and structural cues contribute the largest robustness gains.
Because each stream remains interpretable through country tags, structural placeholders, and phrase tags, the model can give human-readable reasons for flagging a message.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the per-stream accuracies all cluster near 96%, which suggests the four datasets share strong base-rate or style signals; a harder test would be cross-dataset evaluation where training and test messages come from different sources.
Editorial inference: because the label keyword list is also used to create the [smishing_like] phrase tags, the contextual stream may be partly circular; a cleaned re-run that removes that overlap could lower the fused score and reveal how much of the gain is genuine.
Editorial inference: the country NER results, such as 'UK' appearing predominantly in smishing, point toward a testable regional-tuning claim: a country-specific model that conditions on the detected country should beat the fusion on regional benchmarks.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's main contribution is a multi-stream fusion for smishing detection: NER-based country tags, structural regex tagging, a CharCNN, and DistilBERT phrase embeddings, trained on a merged 84k-message corpus. The ablation showing each stream helps—with the character and structural streams most valuable—is a legitimate internal result. The authors are also honest that the contextual stream alone is weak (88.29% accuracy), which gives the ensemble story some credibility.

But the headline numbers are not independently supported. The core problem is that smishing labels come from the authors' own GitHub keyword list (ref [7]), and the contextual stream uses similar hand-curated phrase tags ([smishing_like]). Features and labels share the same source, so 97.89% accuracy, 0.963 F1, and 99.73% AUC likely describe detection of the labeling rule, not of evasive smishing. There is no external baseline, no comparison to a simple keyword classifier, no cross-validation, no error bars, and no released data. The internal ablation is meaningful on the authors' labels, but the external claim is circular.

What is actually new is the specific combination of those four channels and the effort to assemble a relatively large relabeled corpus. That's a legitimate moderate extension of existing spam/phishing filters, not a paradigm shift. The modular design (lightweight structural and character channels, heavier semantic channel) is a practical touch worth keeping.

The soft spots are real and load-bearing. If the keyword rule is a valid proxy, the approach might work; if not, the results describe a self-fulfilling match. The paper does not provide evidence for the rule's validity, no human validation sample, and no comparison against the keyword rule itself. That is the single biggest fix needed.

For a poster, this is acceptable as a proposal, not as a finished result. I would send it to peer review with the expectation of heavy revision: validate or replace the label rule, add an external baseline, release code/data, and report error bars. The topic matters, and the ablation has some value, but the headline claim needs to earn its way back in.

Referee Report

4 major / 5 minor

Summary. This paper proposes a multi-signal smishing detection model that fuses four feature streams: country-specific NER-based semantic tagging, structural pattern tagging, character-level CharCNN, and contextual phrase embeddings from DistilBERT. The authors merge and relabel five public SMS datasets, reporting 24,086 smishing samples, and claim the combined model achieves 97.89% accuracy, 0.963 F1, and 99.73% AUC, outperforming each individual stream. An ablation study is described in which removing any stream degrades performance. The paper also outlines a modular deployment vision with lightweight components for mobile devices.

Significance. If the central claim holds, the paper provides a useful demonstration that fusing heterogeneous feature streams improves SMS phishing detection over single-signal approaches, and the modular architecture is practically appealing for mobile deployment. The authors deserve credit for curating a large combined corpus from five public sources and for constructing a clean, interpretable multi-channel pipeline with an internally consistent ablation ordering. However, the significance is limited by the unvalidated label-generation rule, which creates a potential circularity between features and labels; the external validity of the reported accuracy numbers therefore remains unestablished.

major comments (4)

[Section 3 (label generation) and Section 3.4] The smishing labels are defined by checking spam messages for 'frequent smishing words [7],' where reference [7] is the authors' own GitHub keyword list, and the same style of phrase evidence is later reintroduced as [smishing_like] tags in the Contextual Phrase stream. This creates a circular benchmark: the label rule and a predictive feature derive from the same unvalidated source. Consequently, the headline 97.89% accuracy may simply measure detection of the keyword rule rather than of evasive smishing. The authors must validate the keyword-based labeling against an independent standard (e.g., human annotation of a random sample, or an external smishing dataset with its own ground truth) and demonstrate that the model also succeeds on labels not derived from the phrase list.
[Section 4 and Table 1] The performance differences between individual streams are small (96.34%–96.62% accuracy), and the paper reports no confidence intervals, cross-validation, or statistical significance tests. The claims that 'each stream added value' and that ablation caused 'notable performance drops' are unquantified. Please report mean and standard deviation over multiple train/test splits, provide the exact per-ablation accuracies, and include a significance test (e.g., paired bootstrap or McNemar's test) to support the fusion benefit.
[Section 3 (dataset construction)] The merging of five datasets and the relabeling procedure are described in a single sentence, with no details on the class distribution after merging, deduplication, or how the 80/20 split was performed relative to dataset boundaries. Without this information, the risk of train/test contamination from duplicate messages across datasets is not addressed. Please provide per-dataset counts, a deduplication process, and a description of how the split prevents leakage from overlapping sources.
[Title and Section 4] The title claims detection of 'evasive' smishing, but the test set consists of spam messages that contain frequent smishing keywords, and there is no evaluation against adversarial obfuscations, rephrased attacks, or real-world evasive messages. The claim of robustness to evasive attacks is therefore not supported by the current experiments. The authors should either temper the scope of the claim or include an evaluation on a separate evasive-smishing test set.

minor comments (5)

[Abstract] The phrase 'In such, we present' is grammatically awkward; it should be 'In this work, we present.'
[Section 3.1] The description 'appending the detected country name to each message' is ambiguous: it is unclear whether the country name is added as a discrete token that the classifier treats as a feature, or as part of the NER tag. Please clarify the exact preprocessing step.
[Section 3.4] The 'curated benign and suspicious phrases' used to generate [legitimate_like] and [smishing_like] tags are not listed or referenced. A sample list and the curation method (e.g., manual review or frequency analysis) are needed for reproducibility.
[Table 1] AUC is reported as a percentage (e.g., 99.73%). Please state whether this is AUC-ROC or AUC-PRC, and ensure the metric is defined identically across all rows.
[References] Reference [7] is a GitHub repository. The paper should state explicitly whether the keyword list and the relabeled dataset are publicly released, and under what license, so that the results can be independently reproduced.

Circularity Check

2 steps flagged · score 6.0 of 10

Label-feature circularity: smishing labels are assigned using the authors' own keyword list [7], and the contextual stream injects [smishing_like] phrase tags, so the headline accuracy partly measures agreement with the labeling rule.

self definitional [Section 3, first paragraph (dataset relabeling) and Section 3, item 4 (Contextual Phrase Embeddings)]
"Messages in these datasets were labeled as spam or non-spam, with spam messages further classified as smishing if they contained frequent smishing words [7]. ... We tagged curated benign and suspicious phrases with labels (e.g., [legitimate_like], [smishing_like]), embedded the messages using DistilBERT, and extracted the [CLS] token."

The positive class is defined by the presence of 'frequent smishing words' [7], and the contextual feature stream injects [smishing_like] tags for suspicious phrases into the model. The paper gives no external or human validation of [7] or of the curated phrase list, so the label and the feature encode the same kind of phrase evidence. A model can reach high accuracy by detecting the [smishing_like] tags or their embeddings, which is equivalent to detecting the label-generation rule rather than evasive smishing in the wild. The internal ablation remains valid as a comparison among streams on fixed labels, but the external 97.89% claim is not independently supported.
self citation load bearing [Section 3, first paragraph; reference [7]]
"with spam messages further classified as smishing if they contained frequent smishing words [7] ... [7] Shaghayegh Hosseinpour. 2025. Smishing Dataset. https://github.com/shaghayegh-hp/Smishing_Dataset."

The only stated criterion for relabeling the 24,086 smishing samples is the authors' own GitHub keyword list. This self-resource is load-bearing: it defines the ground truth for every reported metric. The paper supplies no independent validation, inter-annotator agreement, or error analysis of the keyword rule, and no separate source for the suspicious phrases tagged [smishing_like]. If the keyword list is not a valid smishing standard, labels and contextual features shift together, so the reported accuracy cannot confirm detection of real smishing.

full rationale

The derivation chain is: curate five public spam datasets -> relabel as smishing using a frequent-smishing-words criterion [7] -> extract four feature streams, including contextual phrase embeddings tagged [smishing_like] -> fuse and report 97.89% accuracy / 0.963 F1 / 99.73% AUC. The problematic link is that the ground-truth label is generated from the same type of phrase evidence that is later encoded as a feature. This is not a full equation identity: the model also uses NER, structural regexes, a CharCNN, and DistilBERT embeddings, and the ablation comparing streams on the fixed labels is internally meaningful. However, the paper's central claim of robust evasive-smishing detection is not externally supported because the labels themselves rest on the authors' own unvalidated keyword resource [7], with no external benchmark, human validation, or error analysis. A model that learns the [smishing_like] phrase tags is partly detecting the labeling rule, so the headline numbers are self-fulfilling to a substantial degree. This warrants a score of 6.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

All results rest on a label definition that the authors control, plus unreported hyperparameters; the original datasets and public label sources are the only external anchors. The keyword relabeling rule and phrase tag list are the main free inputs, and the model's features overlap with them. No new physical entities are introduced.

free parameters (3)

Smishing keyword list for relabeling = Not disclosed; encoded in GitHub repo [7]
Central label generator: spam messages become smishing if they contain these words; the model is trained and tested on these labels, so the list is a hand-chosen input on which all results depend.
Curated phrase list for [legitimate_like] and [smishing_like] tags = Not disclosed
The contextual phrase stream relies on a hand-curated set of benign and suspicious phrases; no list or validation is provided. It overlaps conceptually with the keyword relabeling rule.
Model hyperparameters (CharCNN, DistilBERT, MLP, SVD, TF-IDF) = Not reported
Architecture details such as convolution widths, embedding dimensions, dropout, learning rate, and SVD components are unstated, so the dependence of the reported scores on these choices cannot be assessed.

assumptions (3)

domain assumption The five public datasets' original spam/ham labels are mostly correct and directly usable as ground truth for relabeling.
Section 3 mixes five datasets with varying label quality and treats their binary spam/non-spam labels as reliable inputs before applying the smishing keyword rule.
ad hoc to paper Spam messages containing the author-defined 'frequent smishing words' are a valid definition of smishing for benchmarking.
The keyword rule in Section 3 determines all 24,086 positive labels; no validation or inter-annotator agreement is reported.
domain assumption Surface features from TF-IDF plus Random Forest, CharCNN, and DistilBERT are sufficient to represent the signals that distinguish smishing in this corpus.
The method assumes these feature extractors capture semantic, structural, stylistic, and phrase-level cues; no theoretical or empirical justification of completeness is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of POSTER: A Multi-Signal Model for Detecting Evasive Smishing." pith.science (2026). https://pith.science/paper/I5L2QRAU

@misc{pith2026250518233,
  author       = {Pith},
  title        = {Pith review of: POSTER: A Multi-Signal Model for Detecting Evasive Smishing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I5L2QRAU}},
  note         = {Machine review of arXiv:2505.18233}
}

read the original abstract

Smishing, or SMS-based phishing, poses an increasing threat to mobile users by mimicking legitimate communications through culturally adapted, concise, and deceptive messages, which can result in the loss of sensitive data or financial resources. In such, we present a multi-channel smishing detection model that combines country-specific semantic tagging, structural pattern tagging, character-level stylistic cues, and contextual phrase embeddings. We curated and relabeled over 84,000 messages across five datasets, including 24,086 smishing samples. Our unified architecture achieves 97.89% accuracy, an F1 score of 0.963, and an AUC of 99.73%, outperforming single-stream models by capturing diverse linguistic and structural cues. This work demonstrates the effectiveness of multi-signal learning in robust and region-aware phishing.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 18 canonical work pages

[7]

Shaghayegh Hosseinpour. 2025. Smishing Dataset. https://github.com/ shaghayegh-hp/Smishing_Dataset

work page 2025
[1]

Olusola Abayomi-Alli, Sanjay Misra, and Adebayo Abayomi-Alli. 2022. A deep learning method for automatic SMS spam classification: Performance of learning algorithms on indigenous dataset. Concurrency and Computation: Practice and Experience 34, 17 (2022), e6989

work page 2022
[2]

Eric B Blancaflor, Ken Jeremy R Cruz, France Junnifer C Monta, and Paolo E Flores. 2024. Unmasking the Threat: Analyzing and Mitigating SMS Smishing Attacks. In International Congress on Information and Communication Technology . Springer, 69–78

work page 2024
[3]

Joel De Goma, John Adam Bravo, Springtime Prudente, and Robert Francis Rondilla. 2024. Detection of SMS Spam Messages Using TF-IDF Vectorizer and Deep Learning Models. In Proceedings of the 2024 9th International Conference on Intelligent Information Technology. 245–249

work page 2024
[4]

Shakthidhar Gopavaram, Jayati Dev, Marthie Grobler, DongInn Kim, Sanchari Das, and L Jean Camp. 2021. Cross-national study on phishing resilience. In Proceedings of the Workshop on Usable Security and Privacy (USEC)

work page 2021
[5]

Anuj Gupta. 2024. Detection of Spam and Fraudulent calls Using Natural Lan- guage Processing Model. In 2024 Sixth International Conference on Computational Intelligence and Communication Technologies (CCICT) . IEEE, 423–427

work page 2024
[6]

José María Gómez Hidalgo, Tiago A Almeida, and Akebo Yamakami. 2012. On the validity of a new SMS spam collection. In 2012 11th International Conference on Machine Learning and Applications , Vol. 2. IEEE, 240–245

work page 2012
[8]

Suhaima Jamal, Hayden Wimmer, and Iqbal H Sarker. 2024. An improved transformer-based model for detecting phishing, spam and ham emails: A large language model approach. Security and Privacy 7, 5 (2024), e402

work page 2024

Show all 19 references

[9]

Phyo Htet Kyaw, Jairo Gutierrez, and Akbar Ghobakhlou. 2024. A Systematic Review of Deep Learning Techniques for Phishing Email Detection. Electronics 13, 19 (2024), 3823

2024
[10]

S Mishra. 2022. Implementation of smishing detector: an efficient model for smishing detection using neural network, SN Computer Science

2022
[11]

Naheem Noah, Abebe Tayachew, Stuart Ryan, and Sanchari Das. 2022. Phish- erCop: developing an NLP-based automated tool for phishing detection. In Pro- ceedings of the Human Factors and Ergonomics Society Annual Meeting , Vol. 66. SAGE Publications Sage CA: Los Angeles, CA, 2093–2097

2022
[12]

Konstantinos I Roumeliotis, Nikolaos D Tselikas, and Dimitrios K Nasiopoulos
[13]

Michiavelly Rustam, Agung Brotokuncoro, and Rusdianto Roestam. 2024. Deteksi Email Spam dengan Continuous Bag-Of-Words dan Random Forest. Ranah Research: Journal of Multidisciplinary Research and Development 6, 4 (2024), 758– 765

2024
[14]

Muhammad Salman, Muhammad Ikram, Nardine Basta, and Mohamed Ali Kaafar
[15]

Chilukuri Lekhya Sri, D Dhana Lakshmi, Kodati Ravali, Vinay Kukreja, and Shanmugasundaram Hariharan. 2024. Improved spam detection through LSTM- based approach. In 2024 Third International Conference on Intelligent Techniques in Control, Optimization and Signal Processing (INC...

2024
[16]

Anne C Tally, Jacob Abbott, Ashley Bochner, Sanchari Das, and Christena Nippert- Eng. 2023. What mid-career professionals think, know, and feel about phishing: opportunities for university it departments to better empower employees in their anti-phishing decisions. Proceedings...

2023
[17]

Daniel Timko and Muhammad Lutfor Rahman. 2024. Smishing dataset i: Phishing sms dataset from smishtank. com. InProceedings of the Fourteenth ACM Conference on Data and Application Security and Privacy . 289–294

2024
[2024]

Electronics 13, 11 (2024), 2034

Next-generation spam filtering: Comparative fine-tuning of LLMs, NLPs, and CNN models for email spam classification. Electronics 13, 11 (2024), 2034

2024
[2025]

arXiv preprint arXiv:2501.04985 (2025)

SpaLLM-Guard: Pairing SMS Spam Detection Using Open-source and Commercial LLMs. arXiv preprint arXiv:2501.04985 (2025)

2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [7]

Shaghayegh Hosseinpour. 2025. Smishing Dataset. https://github.com/ shaghayegh-hp/Smishing_Dataset

work page 2025

[2] [1]

Olusola Abayomi-Alli, Sanjay Misra, and Adebayo Abayomi-Alli. 2022. A deep learning method for automatic SMS spam classification: Performance of learning algorithms on indigenous dataset. Concurrency and Computation: Practice and Experience 34, 17 (2022), e6989

work page 2022

[3] [2]

Eric B Blancaflor, Ken Jeremy R Cruz, France Junnifer C Monta, and Paolo E Flores. 2024. Unmasking the Threat: Analyzing and Mitigating SMS Smishing Attacks. In International Congress on Information and Communication Technology . Springer, 69–78

work page 2024

[4] [3]

Joel De Goma, John Adam Bravo, Springtime Prudente, and Robert Francis Rondilla. 2024. Detection of SMS Spam Messages Using TF-IDF Vectorizer and Deep Learning Models. In Proceedings of the 2024 9th International Conference on Intelligent Information Technology. 245–249

work page 2024

[5] [4]

Shakthidhar Gopavaram, Jayati Dev, Marthie Grobler, DongInn Kim, Sanchari Das, and L Jean Camp. 2021. Cross-national study on phishing resilience. In Proceedings of the Workshop on Usable Security and Privacy (USEC)

work page 2021

[6] [5]

Anuj Gupta. 2024. Detection of Spam and Fraudulent calls Using Natural Lan- guage Processing Model. In 2024 Sixth International Conference on Computational Intelligence and Communication Technologies (CCICT) . IEEE, 423–427

work page 2024

[7] [6]

José María Gómez Hidalgo, Tiago A Almeida, and Akebo Yamakami. 2012. On the validity of a new SMS spam collection. In 2012 11th International Conference on Machine Learning and Applications , Vol. 2. IEEE, 240–245

work page 2012

[8] [8]

Suhaima Jamal, Hayden Wimmer, and Iqbal H Sarker. 2024. An improved transformer-based model for detecting phishing, spam and ham emails: A large language model approach. Security and Privacy 7, 5 (2024), e402

work page 2024