{"id":"b8ef2c1b-2ae7-4d00-9ea6-5e43d7c40636","arxiv_id":"2412.09641","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Text normalization with a slang dictionary improves a Naive Bayes smishing classifier from 88.2% to 96.2% accuracy in the authors' experiments, though the evaluation lacks a reproducible dataset and named baselines.","lead":"This paper proposes a smishing detection system that cleans up SMS slang with a dictionary before feeding messages to a Naive Bayes classifier. The authors report accuracy rising from 88.2% to 96.2% on a manually curated dataset, but details needed to reproduce the result are missing.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 8-point accuracy gain rests entirely on manually curated smishing labels; without an annotation protocol, the improvement could be an artifact of mislabeling.","rationale":"The reader's weakest assumption is exactly the load-bearing point: all reported metrics are computed against labels that the authors created by an undocumented manual process. My reconstruction of the confusion matrix shows this is not a vague worry. The 8-point accuracy improvement is precisely 40 test messages, of which 39 are ham messages that the classifier assigns to different classes before and after normalization. Those 39 messages determine the headline result, and their ground-truth labels are the ones with no annotation protocol. The paper does not provide a definition of smishing, inter-rater validation, or a reproducibility package, so there is no way to check whether the labels correspond to real smishing. If the labels are wrong, the direction of the normalization effect could even reverse. Other issues, such as the unnamed baselines and the 90%/10% split wording, are secondary; the reported numbers actually match a 500-message test set, so the counting inconsistency is less damaging than the reader's rationale suggests. The label validity concern is sufficient to maintain the REJECT verdict.","tokens_in":12008,"tokens_out":8164,"duration_ms":82309,"concrete_test":"Have two independent annotators re-label all 5,169 messages (or a stratified sample of at least 1,000 including all 362 smishing labels and all 39 FP-to-TN messages) using a pre-registered smishing definition (e.g., message induces the recipient to call a number, click a link, or reply with personal/financial credentials). Compute inter-annotator agreement; then recompute Table X on the same 500-message test split using only labels on which both annotators agree. If the normalization-based accuracy improvement is not at least 5 percentage points, or a substantial portion of the 39 messages are actually smishing, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Introduction; Tables VIII/X) is that NoSlang normalization raises Naive Bayes accuracy from 88.2% to 96.2%. The reported metrics are exactly reproduced by a 500-message test set (465 ham, 35 smishing) with the following confusion matrices: without normalization TP=33, FN=2, TN=408, FP=57; with normalization TP=34, FN=1, TN=447, FP=18. Thus the entire accuracy gain is 40 messages: 39 ham messages move from FP to TN and 1 smishing message moves from FN to TP. The 39 messages are the entire source of the accuracy improvement, and their true nature is determined solely by the authors' manual labeling in Section V-A, where smishing messages were 'manually extracting' from the SMS Spam Collection spam category and supplemented by 71 messages from a Pinterest board. No inclusion criteria, annotation instructions, or inter-rater agreement are reported. If any of those 39 messages are actually smishing (mislabeled ham), the normalization is harming detection of real smishing while improving a corrupted accuracy metric. The TPR improvement, in particular, is only 1 out of 35 smishing test messages (33 to 34), so the '97.14%' TPR claim is a single-message effect. Consequently, the central 8-percentage-point claim is no stronger than the undocumented label curation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a content-based smishing detection framework that applies a NoSlang dictionary for text normalization, followed by a Naive Bayes classifier, and reports that normalization improves accuracy from 88.2% to 96.2%, with TPR increasing from 94.28% to 97.14% and TNR from 87.74% to 96.12%. The evaluation uses a modified version of the SMS Spam Collection v.1 dataset, for which the authors manually extracted smishing messages from the spam category and added 71 messages from a Pinterest source. The paper also presents comparisons with two unnamed baseline systems and concludes that the proposed framework outperforms existing methods.","tokens_in":12292,"tokens_out":4766,"duration_ms":49103,"significance":"If the reported improvement were robust, the contribution would be a simple and potentially useful preprocessing step for content-based smishing detection, and the use of a public base dataset is a strength. The paper also gives clear algorithmic descriptions of the preprocessing and classification pipeline. However, the significance is currently limited by evaluation issues: the ground-truth labels are manually curated with no documented criteria, the baseline systems are unnamed, the training split counts are inconsistent with the stated dataset sizes, and the reported gain rests on a very small number of test messages. These issues must be resolved before the result can be taken as evidence of a general improvement.","major_comments":[{"comment":"The reported training counts (4,342 ham and 327 smishing) are not 90% of the stated dataset sizes (4,807 ham and 362 smishing), which would be approximately 4,326.3 ham and 325.8 smishing. The paper should state the exact split procedure and raw test-set counts; as written, the discrepancy prevents independent verification of the reported metrics.","section":"Section V-A / V-B, Algorithm 2"},{"comment":"The entries labeled \"Baseline 1\" and \"Baseline 2\" in Tables IX and XI are never named or cited. Without identifying these existing systems and their configurations, the comparative claim that the proposed framework \"significantly outperforms existing methods\" is unsupported. The authors should specify the baselines and provide citations or descriptions.","section":"Section V-B, Tables IX and XI"},{"comment":"The ground-truth smishing labels are produced by \"manually extracting smishing messages from the spam category\" and adding 71 Pinterest messages, but no annotation protocol, inclusion criteria, or inter-rater reliability is reported. Because every reported metric is computed against these labels, the central 8-percentage-point accuracy improvement is no stronger than the undocumented label curation. The authors should provide the annotation guidelines and, ideally, release the labeled dataset.","section":"Section V-A"},{"comment":"The reported TPR improvement from 94.28% to 97.14% corresponds to a change from 33 to 34 true positives out of 35 smishing test messages under the implied 500-message test set (465 ham, 35 smishing), i.e., a single-message effect. The paper should report the raw confusion matrices and confidence intervals; otherwise, the claim of a significant TPR improvement is not supported.","section":"Section V-B, Table X"},{"comment":"The evaluation uses a single 90/10 split with no cross-validation, repeated trials, or statistical significance testing. Given the small smishing test set and the sensitivity of the result to a few messages, the robustness of the 96.2% accuracy claim is not established. The authors should use k-fold cross-validation or bootstrap resampling and report variance.","section":"Section V-B, Algorithm 2"}],"minor_comments":[{"comment":"The text states that SMS Spam Collection v.1 contains 4,827 ham messages, but the final dataset is reported as 4,807 ham; the source of this 20-message reduction should be explained.","section":"Section V-A"},{"comment":"The TPR and FNR values do not exactly sum to 100% because of rounding (94.28% + 5.71% = 99.99%); the paper should state that values are rounded and perhaps give FPR/FNR to consistent precision.","section":"Table X"},{"comment":"The comparison operator appears as \"¿\" instead of \">\" in the pseudocode.","section":"Algorithm 2, line 16"},{"comment":"The NoSlang dictionary and the Pinterest source are cited as web pages with access dates from 2017; for reproducibility, provide stable snapshots or archived versions of these resources.","section":"References [25], [26]"},{"comment":"The abstract says the results are \"validated on a publicly available dataset,\" but the evaluation actually uses a manually curated subset of that dataset plus 71 external messages; this should be stated explicitly wherever the dataset is described.","section":"Abstract and Section V-A"}],"recommendation":"reject","confidential_remarks":"The paper's central empirical claim is not currently verifiable: the label construction is undocumented, the baselines are unnamed, the split counts are internally inconsistent, and the reported TPR gain is a single-message effect. These are not merely presentation issues; they undermine the headline result. I would be open to reconsidering a substantially revised version with a documented and released dataset, named baselines, and a proper cross-validated evaluation, but as submitted the evidence does not support the conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the central claim—normalization lifts Naive Bayes smishing detection accuracy from 88.2% to 96.2%—is an 8-point gain that on closer inspection rests on 40 test messages, and the ground truth for those messages is undocumented. The idea is sensible and the paper is clearly written, but the evaluation does not support the headline.\n\nWhat's actually new: applying the NoSlang dictionary to the smishing sub-problem rather than SMS spam filtering generally. Their own related work cites Almeida et al. [23] and Kaur et al. [24] for exactly this normalization idea, so the novelty is thin. The within-paper comparison (same classifier with and without normalization) is good practice, and the descriptive statistics on URL and symbol frequencies in smishing messages are useful.\n\nSoft spots: the labels. Section V-A says smishing messages were 'manually extracting' from the spam category of SMS Spam Collection v.1, plus 71 from Pinterest. No protocol, no inclusion criteria, no inter-rater agreement. The stress-test note reconstructs the confusion matrices: the entire accuracy gain is 39 ham messages moving from false positive to true negative and 1 smishing message moving from false negative to true positive. If any of those 39 are actually smishing, the normalization is doing harm. The TPR gain is literally 1 out of 35 test smishing messages. Also the training-count numbers don't match the dataset sizes (4,807*0.9 = 4,326, not 4,342), Baselines 1 and 2 in Tables IX and XI are unnamed, and no code, data, or cross-validation is provided. These are not cosmetic.\n\nRecommendation: desk reject in current form. The authors should release the labeled dataset, document the annotation process, and compare against the normalization baselines they already cite. If they do that, a revised version might merit a serious referee.","headline":"The 8-point normalization gain rests on 40 undocumented test-message labels; the idea is thin and the evaluation is not reproducible.","tokens_in":12817,"tokens_out":3513,"would_cite":false,"duration_ms":34850,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding NoSlang text normalization to a Naive Bayes classifier raises smishing detection accuracy from 88.2% to 96.2% on the authors' curated SMS dataset.","keywords":["smishing detection","SMS spam filtering","text normalization","NoSlang dictionary","Naive Bayes classifier","mobile security","short message service"],"falsifier":"Have two or more independent annotators, working only from a public definition of smishing, label the same SMS Spam Collection v.1 spam messages and the 71 Pinterest messages; if their labels disagree with the authors' extraction, or if a classifier trained on the consensus labels fails to reproduce the 96.2% accuracy, the claimed normalization gain is not robust.","tokens_in":11820,"feed_emoji":"📱","tokens_out":5043,"duration_ms":45081,"temperature":0.7,"pith_summary":"The paper tries to establish that a content-based smishing detector should normalize informal SMS language before classification, not feed raw text to the learner. Its core claim is that replacing slang and abbreviations with standard forms through the NoSlang dictionary lets a Naive Bayes classifier lift accuracy from 88.2% to 96.2%, with the true positive rate rising from 94.28% to 97.14% and the true negative rate from 87.74% to 96.12%. If this is right, a cheap preprocessing step delivers a substantial gain without changing the classifier, which matters because SMS is a high-open-rate channel and smishing is a growing attack vector.","feed_headline":"Text normalization lifts smishing detection to 96.2%","feed_subtitle":"A Naive Bayes filter with NoSlang dictionary preprocessing beats earlier SMS filters by about eight points.","key_machinery":"The load-bearing mechanism is a two-phase pipeline: a preprocessing phase that applies the NoSlang dictionary to convert slang, abbreviations, and nonstandard spellings into standard forms, then removes stop words and stems, followed by a Naive Bayes classification phase that estimates per-word probabilities of appearing in ham versus smishing messages and applies Bayes' theorem to score each message. The NoSlang dictionary is the component that distinguishes this framework from earlier content-based filters; its job is to collapse linguistic variability so that terms like 'ur' and 'your' count as the same feature. The authors quantify the dictionary's contribution by comparing the same classifier with and without the normalization step.","core_discovery":"On a dataset built by manually extracting smishing messages from the SMS Spam Collection v.1 spam category and adding 71 smishing messages from Pinterest (4,807 ham and 362 smishing in total), the proposed framework first tokenizes, lowercases, normalizes nonstandard terms with the NoSlang dictionary, removes stop words, and stems. A Naive Bayes classifier trained on the normalized text then classifies each message by comparing the posterior probability that it is smishing against the posterior probability that it is ham. The authors report that this pipeline reaches 96.2% accuracy, 97.14% TPR, 96.12% TNR, a false positive rate of 3.87%, and a false negative rate of 2.85%, compared with 88.2% accuracy, 94.28% TPR, and 87.74% TNR without normalization. The claim is that the normalization step, not the classifier choice, drives the improvement.","pith_inferences":["The same normalization step may transfer to other short-text classifiers such as SVM or random forests, and to informal-language domains like chat-app phishing, but the paper only tests Naive Bayes.","A natural next experiment would hold the classifier fixed and vary only the dictionary, comparing NoSlang against a domain-specific banking or financial lexicon to isolate where the eight-point gain comes from.","The manual labeling step is the most audit-prone part: independent re-annotation of the spam category could shift the reported numbers, and a published annotation protocol would make the improvement reproducible."],"forward_implications":["Normalization becomes a standalone, cheap upgrade: any content-based SMS filter could add the NoSlang step without replacing its classifier.","The improvement to TNR from 87.74% to 96.12% implies that normalization cuts false alarms on legitimate messages, which matters for user trust in a mobile filter.","The drop in FPR from 12.25% to 3.87% means flagged smishing messages are far more likely to be genuinely malicious, reducing the review burden.","Because the dataset is publicly derived, the 96.2% figure gives other researchers a concrete baseline to beat on the same 5,169-message set."],"supporting_citations":[{"why":"Supplies the SMS Spam Collection v.1 dataset used for the experiments and the prior text-normalization approach the framework builds on.","marker":"[23]"},{"why":"The NoSlang dictionary is the source of the slang-to-standard mappings used in the normalization phase.","marker":"[25]"},{"why":"Adds the 71 smishing messages that complete the curated 5,169-message dataset.","marker":"[26]"},{"why":"SMSAssassin is the Naive Bayes/SVM SMS spam filter baseline the paper compares against.","marker":"[14]"},{"why":"S-Detector is the Naive Bayes smishing detector whose content-based approach is extended with normalization.","marker":"[3]"}],"fun_headline_variants":["NoSlang normalization lifts smishing detection to 96.2%","Text normalization alone lifts smishing detection to 96.2%","Dictionary preprocessing lifts smishing detection to 96.2%","96.2% smishing detection via NoSlang normalization","Normalization step, not model, drives smishing detection to 96.2%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the validity of the authors' manual labels that picked 362 smishing messages out of the SMS Spam Collection v.1 spam category, since every accuracy and rate is computed against those labels and no annotation protocol or inter-rater check is reported.","fun_headline_variants_meta":{"raw":{"variants":["NoSlang normalization lifts smishing detection to 96.2%","Text normalization alone lifts smishing detection to 96.2%","Dictionary preprocessing lifts smishing detection to 96.2%","96.2% smishing detection via NoSlang normalization","Normalization step, not model, drives smishing detection to 96.2%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000922,"raw_usage":{"total_tokens":3931,"prompt_tokens":902,"completion_tokens":3029,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":2935}},"tokens_in":518,"tokens_out":3029,"duration_ms":23717,"temperature":1.0,"reasoning_tokens":2935,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:48:51.502306+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two or more independent annotators, working only from a public definition of smishing, label the same SMS Spam Collection v.1 spam messages and the 71 Pinterest messages; if their labels disagree with the authors' extraction, or if a classifier trained on the consensus labels fails to reproduce the 96.2% accuracy, the claimed normalization gain is not robust.","supporting_citations":[{"cited_title":"http://www.noslang.com/dictionary/full/, Accessed December 2017","cited_arxiv_id":null,"evidence_quote":"The NoSlang dictionary is the source of the slang-to-standard mappings used in the normalization phase."},{"cited_title":"https://in.pinterest.com/seceduau/smishing- dataset/?lp=true","cited_arxiv_id":null,"evidence_quote":"Adds the 71 smishing messages that complete the curated 5,169-message dataset."},{"cited_title":"(2011, March)","cited_arxiv_id":null,"evidence_quote":"SMSAssassin is the Naive Bayes/SVM SMS spam filter baseline the paper compares against."},{"cited_title":"W., Moon, S","cited_arxiv_id":null,"evidence_quote":"S-Detector is the Naive Bayes smishing detector whose content-based approach is extended with normalization."}],"review_version":1}