Pith. sign in

REVIEW 4 major objections 5 minor 57 references

Current large language models are only moderate scam detectors — best micro-F1 around 65% — and their real advantage over a fine-tuned BERT is robustness to unseen scam data, not raw accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:14 UTC pith:RMC4HRH7

load-bearing objection Useful new scam-detection benchmark and a broad LLM sweep, but the headline generalization claim rests on one model pair and the result tables have arithmetic problems. the 4 major comments →

arxiv 2607.17353 v1 pith:RMC4HRH7 submitted 2026-07-19 cs.CR

Measuring and Evaluating the Performance of Generative AI Models for Scam Detection

Cem Topcuoglu , Seyed Ali Akhavani , Harel Berger , Sadia Afroz , Michalis Pachilakis , Vibhor Sehgal , Leyla Bilge , Engin Kirda This is my paper
classification cs.CR
keywords scam detectiononline scamslarge language modelsprompt engineeringfew-shot learningchain-of-thoughtfine-tuningbenchmark dataset
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether off-the-shelf LLMs can detect real-world scams without task-specific fine-tuning, a belief that has spread as LLM-based filters appear in email gateways and browser extensions. It builds a 2,742-sample benchmark from Reddit scam posts, Google Images screenshots, and MTurk submissions, labels them Scam/Clean/Uncertain, and runs nine LLMs under zero-shot, chain-of-thought, common-signs, URL-intelligence, and few-shot prompts, plus temperature and top-p variations. The headline results are that larger models land near 65% micro-F1, prompting helps smaller models more than large ones, and a fine-tuned BERT that scores 59% on the test set drops to 39% on a 59,991-sample proprietary dataset while Llama 3.1 70B stays near 60%. The paper concludes that LLMs are useful but not a universal solution, and points toward hybrid systems rather than pure LLM deployment.

Core claim

The central claim, stated as the authors would state it: current LLMs can serve as moderately accurate, zero-training scam detectors, and their value shows up most clearly when the distribution shifts. ChatGPT 4o1 reaches 64% micro-F1 with a simple zero-shot prompt; Llama 3.1 70B reaches 65% in a 10-shot setting; and both keep similar performance on a 59,991-sample proprietary dataset where fine-tuned BERT collapses from 59% to 39% micro-F1. Because the proprietary samples were unseen by the models, the authors read the gap as evidence that pre-trained knowledge, not fine-tuning, drives robustness. They also show that prompt effects are model-specific — the common-signs prompt lifts Llama 3.

What carries the argument

The load-bearing object is the curated three-class benchmark (Scam / Clean / Uncertain) assembled from 2,742 public examples across 25 scam types, used as a fixed testbed while the paper varies model family, parameter count, prompt design, sampling parameters, and fine-tuning. Micro-F1 — the harmonic mean of precision and recall with every sample weighted equally — is the headline metric, and the mechanism that carries the argument is the controlled shift: the same evaluation is re-run on a 59,991-sample proprietary set, which isolates generalization from in-distribution fit. The fine-tuned BERT comparison is what makes the generalization story visible.

Load-bearing premise

The central claim depends on the 2,742 public benchmark examples — and the 59,991 proprietary validation samples — not having been memorized by the LLMs during pretraining; if they were, the paper's attribution of performance to pre-trained generalization collapses.

What would settle it

Collect scam and clean messages posted after the models' knowledge cutoffs, run the paper's best prompts on them, and compare micro-F1 to the reported ~65%: a drop to near chance on fresh data would refute the generalization claim. A quicker check is an exact and near-duplicate search of the benchmark texts against known pretraining corpora; high overlap would indicate memorization rather than transfer.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Security product teams should expect roughly a third of scam inputs to be missed by today's best general-purpose LLMs, so LLM-based filters need complementary signals rather than being deployed as the whole answer.
  • Prompt selection is model-specific: the common-signs prompt moves Llama 3.1 8B from 50% to 57% micro-F1, a larger gain than most sampling-parameter changes, so prompt tuning should be treated as part of model selection.
  • Fine-tuned BERT remains competitive in-distribution (59% micro-F1) but loses most of its value on shifted data (39%), which suggests LLMs earn their cost at distribution shift rather than in static benchmarks.
  • 10-shot prompting raised Llama 3.1 70B to 65% micro-F1, but shot count and even prompting direction help or hurt depending on the model, so per-model experimentation is required.
  • The paper's own recommendation is hybrid architectures: LLMs for semantic judgment and ambiguity, classical classifiers for deterministic decisions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Not directly tested in the paper: treating the 'Uncertain' output as a flag-for-human-review instead of an error would change the practical calculus, since the reported category metrics show LLMs hedge frequently and the paper's three-class scoring counts those hedges as mistakes.
  • An implication the paper leaves implicit: a membership-inference or canary-string test on the public benchmark could separate memorization from genuine transfer; without it, the generalization claim rests partly on the assumption that 2024 Reddit posts and Google Images screenshots did not appear in pretraining.
  • A testable extension: because the paper's scam-type analysis shows ChatGPT 4o1 is weakest on Eshop and phishing scams with embedded URLs, one could benchmark URL-reputation injection specifically for those two categories (the paper's generic URL-intelligence prompt was not promising, but category-targeted use was not tried).

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents the first systematic benchmark study of LLMs for scam detection. The authors curate a 2,742-example public dataset from Reddit, Google Images, and MTurk, plus a 59,991-example proprietary dataset from a security company, and evaluate nine LLMs (Mistral, Llama 3, Llama 3.1, Llama 3.2, ChatGPT 4/4o/4o1) under zero-shot, few-shot, chain-of-thought, common-signs, URL-intelligence, temperature/top-p, and fine-tuning settings. They also fine-tune Mistral 7B, Llama 3.1 8B, and BERT-base. The headline claims are that larger LLMs generally outperform smaller ones, that prompting can substantially help smaller models, and that LLMs generalize better than fine-tuned BERT to unseen scams.

Significance. If the results are correct, the released dataset and evaluation framework would be a useful resource for the security community, and the conclusion that no model is a universal silver bullet is a valuable counterweight to deployment hype. The breadth of prompts, model families, and the inclusion of a proprietary real-world test set give the study potential practical relevance. However, the strength of the central generalization claim is not matched by the evidence, and some reported metrics appear internally inconsistent with the stated formulas.

major comments (4)
  1. [Tables III and VI] The reported metrics do not satisfy the paper's own definitions in Section IV-E. In single-label multi-class classification, micro-precision and micro-recall are equal (both equal overall accuracy), yet Table III rows show different values, e.g., ChatGPT 4o1 Main Prompt: P=0.641, R=0.612, F1=0.649; the stated F1 formula gives 0.626, not 0.649. Similarly, class F1 values are inconsistent with the class P/R values (e.g., F1S=0.924 vs PS=0.638, RS=0.705 gives F1=0.670). The same issue appears in Table VI. Since the empirical results are the paper's core contribution, these inconsistencies must be corrected or the alternative computation explained.
  2. [Abstract, Section VI, Table VI] The headline claim that LLMs "are better at generalizing to unseen scams compared to fine-tuned models" rests on a comparison of exactly one zero-shot LLM (Llama 3.1 70B) with one fine-tuned BERT on one proprietary dataset. Table VI reports no variance, confidence intervals, repeated runs, or significance tests, despite the paper elsewhere repeating experiments five times. The proprietary dataset's collection, labeling, and class distribution are also not described. This is too narrow to support a general cross-family conclusion; the gap could be model-specific or dataset-specific. The appended 'SKEPTIC' note after the references makes the same observation. The claim should be narrowed to a preliminary observation or supported by multiple models and repeated runs.
  3. [Section V-B, Figure 1] Few-shot examples are sampled from the same 2,742-example pool that is used for evaluation, with no description of a train/evaluation split. If any K-shot example overlaps with a test instance, the few-shot results can be inflated by in-context memorization. The paper must state whether the K examples are disjoint from the evaluation set, or re-run the experiments with explicit train/test separation.
  4. [Section VI and VII] The paper attributes the LLMs' out-of-distribution performance to "pre-trained knowledge" but provides no check for pretraining contamination. The public data is scraped from Reddit r/Scams and Google Images, and the proprietary data comes from a security company; the sizes and release dates make overlap with Llama/ChatGPT pretraining corpora plausible. At minimum, the contamination risk should be acknowledged as a threat to the causal attribution, and ideally a contamination analysis (e.g., n-gram overlap or membership probing) should be reported.
minor comments (5)
  1. [Table I] The Google Images row sums to 615, consistent with the text, but the description says 615 images were gathered and OCR applied; please clarify whether the counts in Table I are after OCR/manual filtering.
  2. [References] References [15] and [16] are duplicates (both PentestGPT), and [35] and [36] are duplicates (both ChatGPT vulnerability management). Please deduplicate and fix citation numbering.
  3. [Table III] The table header as rendered is hard to parse: 'P/R/F1 micro' appears to combine three columns, and the row/column alignment is not visually clear. Please split into separate columns or use a machine-readable table.
  4. [Table VII] Typographical issues: 'Goverment' should be 'Government', 'Eshop' should be 'E-shop', and the sub-category ordering is inconsistent (alphabetical was not maintained).
  5. [Section V-D] The ensemble experiment says 'we test all different combinations of models' but gives no details on the aggregation method (e.g., majority vote, probability averaging) or the number of combinations. Please describe the setup.

Circularity Check

0 steps flagged

No significant circularity: the paper's claims are empirical measurements, not derivations from their inputs.

full rationale

This paper is an empirical benchmark study. It defines scam detection operationally (Section II.E), collects a dataset (Section III), and measures LLM/BERT performance under different prompts, hyperparameters, and fine-tuning (Sections IV-V). No equation in the paper derives a prediction from a fitted parameter; no fitted value is renamed as a predicted result. The central generalization claim (LLMs generalize better than fine-tuned BERT) rests on Table VI, a direct measurement on a separate proprietary dataset, not on a mathematical identity or on a self-citation. The self-citations (e.g., [27], [29]) appear only in related work and are not load-bearing for any result. Potential concerns—few-shot examples sampled from the same public pool as the evaluation set, the proprietary dataset being accessible only through the authors' partner, and the single-model comparison in Table VI—are threats to external validity or to the attribution of 'pre-trained knowledge,' but they are not circularity: none of the reported numbers is equivalent to its input by construction. Under the hard-rule standard requiring a quoted reduction, no circular step can be exhibited, so the appropriate finding is no significant circularity (score 0).

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The central claims rest on label accuracy, OCR fidelity, absence of pretraining contamination, and comparability of the proprietary data. None of these are independently verified in the preprint; there are no theoretical entities or fitted physical constants.

free parameters (6)
  • temperature/top-p grid = {0.1, 0.5, 0.9}
    Sampling parameters chosen by hand to span typical ranges; Table IV shows negligible impact, so they are weakly load-bearing.
  • few-shot example count K = 1, 5, 10
    Chosen arbitrarily; best K varies by model, indicating the choice is model-dependent and could affect comparisons.
  • fine-tuning hyperparameters (epochs, batch size, learning rate) = 1-4 epochs, batch 16-32, LR 1e-5-5e-5
    Selected by searching a small grid on the validation set; the best configuration is then reported as the model's performance without held-out retraining.
  • common signs list (12 indicators) = 12 hand-written scam signs
    An author-authored prompt bias; functions as an injected prior that improves some models and is not derived from data.
  • Argue-for-and-against CoT template = hand-authored reasoning prompt
    Reasoning prompt designed by the authors; outcome varies by model, so it is a fitted prompt rather than a fixed probe.
  • majority-vote runs = 5 runs
    Results are majority-voted over 5 runs; the number of runs is a methodological choice that affects variance.
axioms (6)
  • domain assumption Human labels (Scam/Clean/Uncertain) on Reddit, Google Images, and MTurk are correct and consistent.
    All downstream metrics inherit label quality; no inter-annotator agreement is reported (Section III).
  • domain assumption OCR of screenshots faithfully captures the scam-relevant text.
    Reddit/Google images often contain screenshots; OCR errors would distort both LLM and BERT inputs (Section III).
  • domain assumption Pretrained LLMs have not memorized the evaluation samples.
    Public sources (Reddit r/Scams, Google Images) may overlap training corpora; no contamination analysis is done, yet Section VI attributes performance to 'pre-trained knowledge.'
  • domain assumption The proprietary 59,991-sample labels are trustworthy and comparable to the public labels.
    Table VI uses company data as ground truth for the generalization claim; labeling standards are not described.
  • domain assumption The 2,742 pooled examples are representative of the real-world scam distribution.
    Reddit self-selection and MTurk recruitment introduce sampling bias; acknowledged in Section VII as 'smaller and unbalanced than ideal.'
  • standard math Evaluation metrics are computed as defined in Eqs. 1-5.
    But Table III numbers do not consistently satisfy F1=2PR/(P+R), so this assumption appears to be violated in the rendering.

pith-pipeline@v1.3.0-alltime-deepseek · 17680 in / 18656 out tokens · 173822 ms · 2026-08-01T18:14:08.773836+00:00 · methodology

0 comments
read the original abstract

Online scams continue to cause substantial financial and personal harm. As a result, detection systems based on Large Language Models (LLMs) have been integrated into security products ranging from email gateways and browser extensions to fraud-monitoring dashboards. As this adoption accelerates, a common belief has taken hold: that these models are broadly suitable for scam detection. In this work, we investigate whether LLMs, with their strong capabilities in understanding intent, context, and reasoning, can effectively detect scams across diverse scenarios without task-specific fine-tuning. We curate and release a unique benchmark dataset of real-world scams spanning multiple formats and topics. We evaluate nine LLMs of varying sizes and architectures, examining their performance under different prompting strategies and comparing them to a fine-tuned BERT-based classifier. Our results show that while larger LLMs generally outperform smaller ones, effective prompting substantially boosts the performance of smaller models. Moreover, LLMs are better at generalizing to unseen scams compared to fine-tuned models, suggesting that pre-trained knowledge contributes meaningfully to scam detection. We release our dataset and evaluation framework to facilitate future research in robust scam detection using language models.

Figures

Figures reproduced from arXiv: 2607.17353 by Cem Topcuoglu, Engin Kirda, Harel Berger, Leyla Bilge, Michalis Pachilakis, Sadia Afroz, Seyed Ali Akhavani, Vibhor Sehgal.

Figure 1
Figure 1. Figure 1: Many-Shot Prompt Micro Precision, Recall, and F1 score across models, and 1, 5, and 10 shot. [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cohen’s Kappa Scores, representing the agreement [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 11 linked inside Pith

  1. [1]

    Social-guard: Detecting scammers in online dating

    Suhaib Al-Rousan, Abdullah Abuhussein, Faisal Alsubaei, Ozkan Kahveci, Hazem Farra, and Sajjan Shiva. Social-guard: Detecting scammers in online dating. In2020 IEEE International Conference on Electro Information Technology (EIT), pages 416–422. IEEE, 2020

    2020

  2. [2]

    An automatic detection and analysis of the bitcoin generator scam

    Emad Badawi, Guy-Vincent Jourdan, Gregor Bochmann, and Iosif- Viorel Onut. An automatic detection and analysis of the bitcoin generator scam. In2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pages 407–416. IEEE, 2020

    2020

  3. [3]

    game hack

    Emad Badawi, Guy-Vincent Jourdan, Gregor Bochmann, Iosif-Viorel Onut, and Jason Flood. The “game hack” scam. InInternational Conference on Web Engineering, pages 280–295. Springer, 2019

    2019

  4. [4]

    hello. this is the irs calling

    Morvareed Bidgoli and Jens Grossklags. “hello. this is the irs calling.”: A case study on scams, extortion, impersonation, and phone spoofing. In 2017 APWG Symposium on Electronic Crime Research (eCrime), pages 57–69. IEEE, 2017. TABLE VII: Scam categories and the number of correct classifications by ChatGPT 4o1 using the main prompt. Scam Type # Correct ...

    2017

  5. [5]

    Beyond phish: Toward detecting fraudulent e-commerce websites at scale

    Marzieh Bitaab, Haehyun Cho, Adam Oest, Zhuoer Lyu, Wei Wang, Jorij Abraham, Ruoyu Wang, Tiffany Bao, Yan Shoshitaishvili, and Adam Doup ´e. Beyond phish: Toward detecting fraudulent e-commerce websites at scale. In2023 IEEE Symposium on Security and Privacy (SP), pages 2566–2583. IEEE Computer Society, 2023

    2023

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  7. [7]

    Scam call detection using nlp and na¨ıve bayes classifier.INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT, 2024

    Valarmathi C and S Sharanya. Scam call detection using nlp and na¨ıve bayes classifier.INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT, 2024

    2024

  8. [8]

    Learning to detect and mea- sure fake ecommerce websites in search-engine results

    Claudio Carpineto and Giovanni Romano. Learning to detect and mea- sure fake ecommerce websites in search-engine results. InProceedings of the international conference on web intelligence, pages 403–410, 2017

    2017

  9. [9]

    Scam statistics, 2025

    National Anti-Scam Centre. Scam statistics, 2025. https:// www.scamwatch.gov.au/research-and-resources/scam-statistics

    2025

  10. [10]

    Can ai keep you safe? a study of large language models for phishing detection

    Robin Chataut, Prashnna Kumar Gyawali, and Yusuf Usman. Can ai keep you safe? a study of large language models for phishing detection. In2024 IEEE 14th Annual Computing and Communication Workshop and Conference (CCWC), pages 0548–0554. IEEE, 2024

    2024

  11. [11]

    Scam detection in twitter

    Xiaoling Chen, Rajarathnam Chandramouli, and Koduvayur P Subbal- akshmi. Scam detection in twitter. InData Mining for Service. Springer, 2014

    2014

  12. [12]

    New ftc data show a big jump in reported losses to fraud to $12.5 billion in 2024, 2025

    Federal Trade Commission. New ftc data show a big jump in reported losses to fraud to $12.5 billion in 2024, 2025. https://www.ftc.gov/news-events/news/press-releases/2025/03/new- ftc-data-show-big-jump-reported-losses-fraud-125-billion-2024

    2024

  13. [13]

    Understanding security issues in the nft ecosystem

    Dipanjan Das, Priyanka Bose, Nicola Ruaro, Christopher Kruegel, and Giovanni Vigna. Understanding security issues in the nft ecosystem. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, 2022

    2022

  14. [14]

    Hey chatgpt, is this message phishing? In2024 22nd Mediterranean Communication and Computer Networking Conference (MedComNet)

    Sara de Rosa, Francesco Gringoli, and Gabriele Bellicini. Hey chatgpt, is this message phishing? In2024 22nd Mediterranean Communication and Computer Networking Conference (MedComNet). IEEE, 2024

    2024

  15. [15]

    PentestGPT: Evaluating and harnessing large language models for automated penetration testing

    Gelei Deng, Yi Liu, V ´ıctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. PentestGPT: Evaluating and harnessing large language models for automated penetration testing. In33rd USENIX Security Symposium (USENIX Security 24), pages 847–864, Philadelphia, PA, August 2024. USENIX Association

    2024

  16. [16]

    PentestGPT: Evaluating and harnessing large language models for automated penetration testing

    Gelei Deng, Yi Liu, V ´ıctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. PentestGPT: Evaluating and harnessing large language models for automated penetration testing. In33rd USENIX Security Symposium (USENIX Security ’24), pages 847–864, 2024

    2024

  17. [17]

    Bert: Pre-training of deep bidirectional transformers for language un- derstanding.arXiv preprint arXiv:1810.04805, 2018

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language un- derstanding.arXiv preprint arXiv:1810.04805, 2018

    Pith/arXiv arXiv 2018

  18. [18]

    Large language models for code analysis: Do LLMs really do their job? In33rd USENIX Security Symposium (USENIX Security ’24), pages 829–846, 2024

    Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Asmita, Ryan Tsang, Najmeh Nazari, Han Wang, and Houman Homayoun. Large language models for code analysis: Do LLMs really do their job? In33rd USENIX Security Symposium (USENIX Security ’24), pages 829–846, 2024

    2024

  19. [19]

    New study estimates as much as $75 billion in global victims’ losses to pig-butchering scam, 2024

    Zeke Faux. New study estimates as much as $75 billion in global victims’ losses to pig-butchering scam, 2024. https://time.com/6836703/ pig-butchering-scam-victim-loss-money-study-crypto/

    arXiv 2024

  20. [20]

    Feature engineering vs bert on twitter data.ArXiv, abs/2210.16168, 2022

    Ryiaadh Gani and Lisa Andreevna Chalaguine. Feature engineering vs bert on twitter data.ArXiv, abs/2210.16168, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    Garrido-Merch ´an

    Santiago Gonz ´alez-Carvajal and E.C. Garrido-Merch ´an. Comparing bert against traditional machine learning text classification.ArXiv, abs/2005.13012, 2020

    Pith/arXiv arXiv 2005

  22. [22]

    How do crypto flows finance slavery? the economics of pig butchering.The Economics of Pig Butchering (February 29, 2024), 2024

    John M Griffin and Kevin Mei. How do crypto flows finance slavery? the economics of pig butchering.The Economics of Pig Butchering (February 29, 2024), 2024

    2024

  23. [23]

    Incorporating bert into parallel sequence decoding with adapters.ArXiv, abs/2010.06138, 2020

    Junliang Guo, Zhirui Zhang, Linli Xu, Hao-Ran Wei, Boxing Chen, and Enhong Chen. Incorporating bert into parallel sequence decoding with adapters.ArXiv, abs/2010.06138, 2020

    Pith/arXiv arXiv 2010

  24. [24]

    Llm- tikg: Threat intelligence knowledge graph construction utilizing large language model.Computers & Security, 145:103999, 2024

    Yuelin Hu, Futai Zou, Jiajia Han, Xin Sun, and Yilei Wang. Llm- tikg: Threat intelligence knowledge graph construction utilizing large language model.Computers & Security, 145:103999, 2024

    2024

  25. [25]

    Brown, Andrew Miller, Edward Chi, Anthony Michaud, Anselm Levskaya, Mostafa Dehghani, Michael Collins, and Lillian Lee

    Zhengxiao Jiang, Tom B. Brown, Andrew Miller, Edward Chi, Anthony Michaud, Anselm Levskaya, Mostafa Dehghani, Michael Collins, and Lillian Lee. Mistral: Efficient supervision for large language models. arXiv preprint arXiv:2309.10836, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    (security) assertions by large language models.IEEE Transactions on Information Forensics and Security, 19:4374–4389, 2024

    Rohan Kande, Hammond Pearce, Ben Tan, Brendan Dolan-Gavitt, Saurabh Thakur, Ramesh Karri, and Jeyavijayan Rajendran. (security) assertions by large language models.IEEE Transactions on Information Forensics and Security, 19:4374–4389, 2024

    2024

  27. [27]

    Surveylance: automatically detecting online survey scams

    Amin Kharraz, William Robertson, and Engin Kirda. Surveylance: automatically detecting online survey scams. In2018 IEEE Symposium on Security and Privacy (SP), pages 70–86. IEEE, 2018

    2018

  28. [28]

    Chatphishdetector: Detecting phishing sites using large language models.IEEE Access, 2024

    Takashi Koide, Hiroki Nakano, and Daiki Chiba. Chatphishdetector: Detecting phishing sites using large language models.IEEE Access, 2024

    2024

  29. [29]

    Scamdog millionaire: Detecting e-commerce scams in the wild

    Platon Kotzias, Kevin Roundy, Michalis Pachilakis, Iskander Sanchez- Rola, and Leyla Bilge. Scamdog millionaire: Detecting e-commerce scams in the wild. InIn Proceedings of the 39th Annual Computer Security Applications Conference, 2023

    2023

  30. [30]

    From ml to llm: Evaluating the robustness of phishing web- page detection models against adversarial attacks.arXiv preprint arXiv:2407.20361, 2024

    Aditya Kulkarni, Vivek Balachandran, Dinil Mon Divakaran, and Tamal Das. From ml to llm: Evaluating the robustness of phishing web- page detection models against adversarial attacks.arXiv preprint arXiv:2407.20361, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Spam-t5: Benchmarking large language models for few-shot email spam detection.arXiv preprint arXiv:2304.01238, 2023

    Maxime Labonne and Sean Moran. Spam-t5: Benchmarking large language models for few-shot email spam detection.arXiv preprint arXiv:2304.01238, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    White, and Sujay Ku- mar Jauhar

    Dong-Ho Lee, Jay Pujara, Mohit Sewak, Ryen W. White, and Sujay Ku- mar Jauhar. Making large language models better data creators. In Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  33. [33]

    “hot” chatgpt: The promise of chatgpt in detecting and discriminating hateful, offensive, and toxic comments on social media.ACM Transactions on the Web, 18(2):1–36, 2024

    Lingyao Li, Lizhou Fan, Shubham Atreja, and Libby Hemphill. “hot” chatgpt: The promise of chatgpt in detecting and discriminating hateful, offensive, and toxic comments on social media.ACM Transactions on the Web, 18(2):1–36, 2024

    2024

  34. [34]

    KnowPhish: Large language mod- els meet multimodal knowledge graphs for enhancing Reference-Based phishing detection

    Yuexin Li, Chengyu Huang, Shumin Deng, Mei Lin Lock, Tri Cao, Nay Oo, Hoon Wei Lim, and Bryan Hooi. KnowPhish: Large language mod- els meet multimodal knowledge graphs for enhancing Reference-Based phishing detection. In33rd USENIX Security Symposium (USENIX Security 24), pages 793–810, Philadelphia, PA, August 2024. USENIX Association

    2024

  35. [35]

    Exploring ChatGPT’s capabilities on vulnerability management

    Peiyu Liu, Junming Liu, Lirong Fu, Kangjie Lu, Yifan Xia, Xuhong Zhang, Wenzhi Chen, Haiqin Weng, Shouling Ji, and Wenhai Wang. Exploring ChatGPT’s capabilities on vulnerability management. In33rd USENIX Security Symposium (USENIX Security 24), pages 811–828, Philadelphia, PA, August 2024. USENIX Association

    2024

  36. [36]

    Exploring ChatGPT’s capabilities on vulnerability management

    Peiyu Liu, Junming Liu, Lirong Fu, Kangjie Lu, Yifan Xia, Xuhong Zhang, Wenzhi Chen, Haiqin Weng, Shouling Ji, and Wenhai Wang. Exploring ChatGPT’s capabilities on vulnerability management. In33rd USENIX Security Symposium (USENIX Security ’24), pages 811–828, 2024

    2024

  37. [37]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.arXiv preprint arXiv:2104.08786, 2021

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pon- tus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.arXiv preprint arXiv:2104.08786, 2021

    Pith/arXiv arXiv 2021

  38. [38]

    Large language model guided protocol fuzzing

    Ruijie Meng, Martin Mirchev, Marcel B ¨ohme, and Abhik Roychoudhury. Large language model guided protocol fuzzing. InProceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS), 2024

    2024

  39. [39]

    Dial one for scam: Analyzing and detecting technical support scams

    Najmeh Miramirkhani, Oleksii Starov, and Nick Nikiforakis. Dial one for scam: Analyzing and detecting technical support scams. In22nd Annual Network and Distributed System Security Symposium (NDSS), volume 16, 2016

    2016

  40. [40]

    On sms phishing tactics and infrastructure

    Aleksandr Nahapetyan, Sathvik Prasad, Kevin Childs, Adam Oest, Yeganeh Ladwig, Alexandros Kapravelos, and Brad Reaves. On sms phishing tactics and infrastructure. InIEEE Symposium on Security and Privacy (SP), 2024

    2024

  41. [41]

    Norton genie your free ai-powered scam detector, 2024

    Norton. Norton genie your free ai-powered scam detector, 2024. https: //us.norton.com/products/genie-scam-detector

    2024

  42. [42]

    Tranco: A research-oriented top sites ranking hardened against manipulation.arXiv preprint arXiv:1806.01156, 2018

    Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczy ´nski, and Wouter Joosen. Tranco: A research-oriented top sites ranking hardened against manipulation.arXiv preprint arXiv:1806.01156, 2018

    Pith/arXiv arXiv 2018

  43. [43]

    Resource networks of pet scam websites

    Benjamin Price and Matthew Edwards. Resource networks of pet scam websites. In2020 Symposium on Electronic Crime Research. Institute of Electrical and Electronics Engineers (IEEE), 2020

    2020

  44. [44]

    Reddit r/scams, 2024

    Reddit. Reddit r/scams, 2024. https://www.reddit.com/r/Scams/

    2024

  45. [45]

    From chatbots to phishbots?: Phishing scam generation in commercial large language models

    Sayak Saha Roy, Poojitha Thota, Krishna Vamsi Naragam, and Shirin Nilizadeh. From chatbots to phishbots?: Phishing scam generation in commercial large language models. InIEEE Symposium on Security and Privacy (SP), 2024

    2024

  46. [46]

    Investigating evasive techniques in sms spam filtering: A comparative analysis of machine learning models.IEEE Access, 2024

    Muhammad Salman, Muhammad Ikram, and Mohamed Ali Kaafar. Investigating evasive techniques in sms spam filtering: A comparative analysis of machine learning models.IEEE Access, 2024

    2024

  47. [47]

    Your anti-scam partner, keeping you safe!, 2024

    ScamAdviser. Your anti-scam partner, keeping you safe!, 2024. https: //www.scamadviser.com/home

    2024

  48. [48]

    Detection of internet scam using logistic regression

    Mehrbod Sharifi, Eugene Fink, and Jaime G Carbonell. Detection of internet scam using logistic regression. In2011 IEEE International Conference on Systems, Man, and Cybernetics, pages 2168–2172. IEEE, 2011

    2011

  49. [49]

    Outside the closed world: On using machine learning for network intrusion detection

    Robin Sommer and Vern Paxson. Outside the closed world: On using machine learning for network intrusion detection. In2010 IEEE symposium on security and privacy, pages 305–316. IEEE, 2010

    2010

  50. [50]

    Exposing search and advertisement abuse tactics and infras- tructure of technical support scammers

    Bharat Srinivasan, Athanasios Kountouras, Najmeh Miramirkhani, Monjur Alam, Nick Nikiforakis, Manos Antonakakis, and Mustaque Ahamad. Exposing search and advertisement abuse tactics and infras- tructure of technical support scammers. InProceedings of the 2018 World Wide Web Conference, pages 319–328, 2018

    2018

  51. [51]

    Automatically dismantling online dating fraud.IEEE Transactions on Information Forensics and Security, 15:1128–1137, 2019

    Guillermo Suarez-Tangil, Matthew Edwards, Claudia Peersman, Gi- anluca Stringhini, Awais Rashid, and Monica Whitty. Automatically dismantling online dating fraud.IEEE Transactions on Information Forensics and Security, 15:1128–1137, 2019

    2019

  52. [52]

    An innovative GPT-based open-source intelligence using historical cyber incident reports.Natural Language Processing (NLP) Journal, page 100074, 2024

    Faiz Sufi. An innovative GPT-based open-source intelligence using historical cyber incident reports.Natural Language Processing (NLP) Journal, page 100074, 2024

    2024

  53. [53]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  55. [55]

    Counterfighting counterfeit: detecting and taking down fraudulent webshops at a cctld

    Thymen Wabeke, Giovane Moura, Nanneke Franken, and Cristian Hesselman. Counterfighting counterfeit: detecting and taking down fraudulent webshops at a cctld. InInternational Conference on Passive and Active Network Measurement, pages 158–174. Springer, Cham, 2020

    2020

  56. [56]

    Chain of thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022

    Pith/arXiv arXiv 2022

  57. [57]

    Who are the phishers? phishing scam detection on ethereum via network embedding.IEEE Transactions on Systems, Man, and Cybernetics: Systems, 52:1156–1166, 2019

    Jiajing Wu, Qi Yuan, Dan yan Lin, Wei You, Weili Chen, Chuan Chen, and Zibin Zheng. Who are the phishers? phishing scam detection on ethereum via network embedding.IEEE Transactions on Systems, Man, and Cybernetics: Systems, 52:1156–1166, 2019

    2019