Pith. sign in

REVIEW 5 major objections 5 minor 159 references

Paladin pre-arms open-weight LLMs with hidden tags so phishing outputs flag themselves, claiming over 90% detection accuracy even after attackers fine-tune the model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 22:27 UTC pith:ZBYSPHKK

load-bearing objection Clever defensive trigger-tag idea with a genuinely useful narrow result (explicit trigger + explicit tag), but the 'over 90% across all scenarios' claim is contradicted by the paper's own table and robustness rests on a tag-unaware attacker; worth refereeing but needs substantive revision. the 5 major comments →

arxiv 2509.07287 v1 pith:ZBYSPHKK submitted 2025-09-08 cs.CR cs.AI

Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigm

Yan Pang , Wenlong Meng , Xiaojing Liao , Tianhao Wang This is my paper
classification cs.CR cs.AI
keywords phishing detectiontrigger-tag associationdefensive fine-tuningopen-weight LLMszero-width charactersDPOGRPOjailbreak robustness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes that a defender can pre-arm open-weight language models with trigger-tag associations before releasing them publicly. If a malicious vendor later fine-tunes or jailbreaks the model into a phishing assistant, the generated email will automatically carry a hidden marker—a zero-width character or a statistical pattern—that a lightweight detector can spot in under a second. The paper reports over 90% phishing detection accuracy across three model families (LLaMA 2, LLaMA 3, Qwen 2.5) and four LoRA rank settings, with implicit tags also surviving moderate malicious fine-tuning better than explicit ones. The point is to move phishing detection from expensive semantic analysis at the email gateway to cheap tag lookup at the model output.

Core claim

The paper's central claim is that trigger-tag associations can be embedded into a vanilla open-weight LLM through defensive fine-tuning, creating an instrumented LLM. When such a model is later repurposed into an ill-intentioned LLM application, prompts for phishing content automatically produce outputs containing a detectable tag. The tag is explicit (zero-width Unicode characters matched by regex) or implicit (tokens perturbed so the output's negative log-likelihood shifts detectably under the instrumented model). The insertion is formulated as an optimization problem with constraints on task-level loss, parameter deviation, and output-distribution KL divergence, instantiated by three stra

What carries the argument

The trigger-tag association: a fine-tuned coupling between trigger words in the prompt (an explicit phrase such as 'phishing email', or an implicit paraphrase) and tags in the output. Explicit tags are zero-width characters decoded by a regular expression; implicit tags are embedded by entropy-weighted logit perturbations and detected by comparing negative log-likelihood under the instrumented versus the vanilla model. Three insertion strategies—SFT, DPO, and GRPO—realize progressively stricter constraints that keep the instrumented model's normal outputs close to the vanilla model while making phishing outputs self-marking.

Load-bearing premise

The malicious vendor is assumed to have limited resources—250 malicious samples, LoRA rank 8, and 5 epochs of fine-tuning—and to neither inspect outputs for hidden characters nor sanitize them, even though the paper itself notes that removing whitespace can eliminate zero-width tag characters.

What would settle it

Fine-tune an instrumented model on a larger malicious corpus (for example, 1,000+ samples, rank 64, 20 epochs) and check whether implicit-tag detection accuracy collapses; or run outputs through a whitespace-stripper and test explicit-tag detection. The paper's own result—explicit-tag accuracy dropping over 30% under its limited 250-sample attack—shows that a stronger or sanitizing attacker is the natural stress test for the claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Model owners can publish instrumented checkpoints whose normal outputs stay nearly unchanged (low KL divergence) while phishing prompts reliably produce tagged responses.
  • Detection becomes a regex match or a single forward pass, making internet-scale screening feasible; the paper measures under one second versus roughly 300–440 seconds for an LLM-based detector.
  • The defense holds against jailbreak prompts and against fine-tuning limited to 250 samples, LoRA rank 8, and 5 epochs; implicit tags resist this attack better, while explicit tags degrade by more than 30%.
  • Higher LoRA ranks generally improve detection accuracy but worsen stealthiness (higher KL divergence), so deployment must trade robustness against visibility.
  • Explicit trigger-tag pairs are fastest but fail on paraphrased malicious requests; implicit triggers and tags are slower but handle rephrased intent substantially better.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If tag vocabularies were standardized across model releases, a single shared detector could monitor outputs from every instrumented model, turning phishing detection into an ecosystem-level registry rather than per-deployment analysis.
  • The defense assumes the attacker refines the model rather than sanitizing its output; a determined attacker can strip zero-width characters with a simple text-cleaning pass, so the explicit-tag variant is best read as a speed bump, not a permanent barrier.
  • The same trigger-tag machinery naturally extends to other abuse categories—malicious code generation, disinformation, fraud—wherever a fixed set of sensitive topics can be specified at release time.
  • Robustness has only been shown within the paper's attack budget; an attacker with more data, higher rank, or more epochs may erase even implicit tags, so deployment should monitor tag yield over time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Paladin, a defense that fine-tunes open-weight LLMs to associate 'trigger' inputs (e.g., queries containing 'phishing email') with 'tags' in generated outputs. Four configurations are considered: explicit/implicit triggers and explicit/implicit tags. Three insertion strategies are developed—Paladin-base (SFT), Paladin-core (DPO), and Paladin-pro (GRPO with KL/parameter constraints). Evaluation on LLaMA 2, LLaMA 3, and Qwen 2.5 measures tag-detection accuracy (Atag), safe-email accuracy (Asafe), and KL divergence to the vanilla model, plus jailbreak and malicious-fine-tuning robustness. The headline claim is over 90% detection accuracy across all scenarios, with negligible detection cost, even after attackers fine-tune the model.

Significance. If the claims held, Paladin would be a significant new defense: a model publisher could post-train a released open-weight model so that outputs triggered by phishing content are self-marking, and detection is a regex or a single forward pass rather than an expensive LLM-based classifier. The paper ships code, evaluates three popular open-weight models, and compares with ChatSpamDetector; the explicit-trigger/explicit-tag configuration is demonstrated at near-perfect accuracy with sub-second detection, which is a useful empirical result. However, the broad 'across all scenarios' claim is not supported by the paper's own data, and the robustness evaluation omits the most obvious countermeasure (output sanitization). The contribution is therefore best viewed as a narrow feasibility study under a specific threat model, not as a general 'over 90% in all scenarios' defense.

major comments (5)
  1. [Abstract / Table IV] The claimed 'over 90% detection accuracy across all scenarios' is contradicted by Table IV. For ImT+ExG, Atag is 0.476–0.523 across LoRA ranks; ExT+ImG is 0.710–0.765; ImT+ImG is 0.748–0.845. Only ExT+ExG reaches ~0.99. The Introduction and Conclusion repeat similar overclaims ('over 85% in most cases', 'nearly 90% accuracy across all four LoRA settings'). Please either restrict the claim to ExT+ExG or report the per-scenario numbers prominently.
  2. [§V-E2, §IV-B] Robustness is evaluated only against a resource-limited, tag-unaware attacker. The malicious fine-tuning setup (250 samples, LoRA rank 8, 5 epochs) does not include any output post-processing. Yet §IV-B concedes 'removing whitespace may eliminate zero-width characters used as tags'. A one-line sanitizer (e.g., stripping U+200B) would destroy ExT+ExG detection without changing the phishing content's meaning. Since the central deployment claim is persistence after attacker modification, the paper must test sanitization and/or an attacker who knows the tag mechanism; otherwise the conclusion applies only to an attacker who chooses not to inspect outputs.
  3. [§IV-A / §V-B, Eq. (8)] The stealthiness metric is circular for Paladin-pro. The GRPO objective in Eq. (8) explicitly contains a KL term -γ DKL(Mθ* || Mθ) that enforces closeness to the vanilla model, and §V-B then measures stealthiness with the same DKL. Low DKL therefore partly restates the training objective rather than providing independent evidence that instrumented outputs are indistinguishable from vanilla outputs. Please add an independent stealthiness evaluation (e.g., human/LLM discrimination, perplexity, or character-level inspection). Note also that explicit zero-width tags are trivially detectable by Unicode inspection, so 'stealthy' for ExT+ExG needs separate support.
  4. [§III-D / §V-E2] The threat model is underspecified about attacker awareness. §III-D says 'once malicious vendors become aware of their presence, they can easily remove them via fine-tuning', yet §V-E2 assumes an attacker with only 250 samples, LoRA rank 8, and 5 epochs, and does not allow the attacker to post-process outputs. If the threat model is an aware attacker, the experiments do not test it; if it is an unaware attacker, that assumption should be stated as a central limitation and the abstract/introduction should not imply robustness against arbitrary malicious fine-tuning.
  5. [Table IV / §V-D] The 'only 1% of the time' claim applies only to explicit tags. In Table IV, ExT+ImG and ImT+ImG require 244–368 s and 156–272 s, respectively, essentially the same order as ChatSpamDetector (310–442 s in Table III). The paper itself concedes that 'the efficiency advantage diminishes... implicit tags at scale is not a viable solution' (§V-D). Please qualify all efficiency claims by scenario.
minor comments (5)
  1. [Figure 3] The baseline name is misspelled as 'ChatSpamDector' in the figure legend; should be 'ChatSpamDetector'.
  2. [Table III] Typo: 'rejecte' should be 'rejected' in the paragraph discussing Paladin-core preference pairs.
  3. [§V-A / Appendix F] The paper reports 200 jailbreak prompts in §V-A but 250 jailbreak prompts in Appendix F. Please reconcile the numbers and state the dataset size consistently.
  4. [Notation] Dtag is used both for the tag dataset and for the detection classifier (Eq. (2) and §III-C). This overloading is confusing; please use distinct symbols, e.g., D_tr_tag and C_tag.
  5. [§IV-B] The abbreviations ExT, ExG, ImT, ImG are used in Table IV before they are defined; define them in the table caption or immediately before the table, not only in Appendix E.

Circularity Check

2 steps flagged

Partial circularity: two headline metrics are defined by the training procedure itself (DKL is the GRPO regularizer; Atag for explicit tags checks the exact inserted tag), but the core method has independent empirical content via Asafe and the ChatSpamDetector baseline.

specific steps
  1. fitted input called prediction [Section V-B (Evaluation Metrics); Appendix D, Eq. (8)]
    "We define stealthiness as the similarity between the outputs of the instrumented model and the vanilla model... we use the Kullback–Leibler divergence between the output distributions of the two models. ... θ∗ = arg max ... − γ · DKL(Mθ∗(y | x) ∥ Mθ(y | x))"

    The stealthiness metric reported in the evaluation, DKL(Mθ* || Mθ), is literally the KL-regularization term that the Paladin-pro GRPO objective in Eq. (8) is trained to minimize. Therefore the low DKL values for Paladin-pro are not an independent prediction about the instrumented model; they are the value of the loss term the optimizer was explicitly set to reduce. The result is forced by construction. The metric retains some independent content for Paladin-base, whose objective does not include this KL term.

  2. self definitional [Section IV-B (Trigger-Tag Design); Section V-C (Explicit trigger and tag evaluation)]
    "For the explicit tag, the detection mechanism Dtag(·) is a regular expression used to match predefined tags. ... we insert a zero-width space (Unicode U+200B) after 'Dear' or 'Subject'."

    The explicit-tag detector is a regex for the exact zero-width character that the training procedure inserts into the Dtag training outputs, and the training objective (Eqs. 2 and 6) directly minimizes the negative log-likelihood of these tagged outputs. Consequently, Atag for explicit tags measures whether the model reproduced the precise tag it was trained to emit; the reported >90% accuracy is largely a training-success check rather than an independent detection result. The Asafe metric on benign emails and the comparison against ChatSpamDetector provide the independent evidence.

full rationale

Paladin is an engineering/defense paper rather than a theory paper, so most of its evaluation is empirical. However, two of the headline metrics are partially circular. First, the stealthiness claim for Paladin-pro is supported by DKL values that are the same KL-to-vanilla quantity minimized by the GRPO objective; reporting low DKL is reporting the training loss, not an independent outcome. Second, the effectiveness metric for the explicit-tag configuration is defined by the presence of the same zero-width character that was inserted into the training data and that the model was optimized to emit, making high Atag partly definitional. These issues do not destroy the paper's contribution: the safe-email accuracy (Asafe), the comparison with ChatSpamDetector, and the robustness experiments after jailbreak and limited fine-tuning provide independent evidence that the trigger-tag association can be learned and persists under the stated threat model. I do not find any load-bearing self-citation or imported uniqueness theorem. The more serious weakness is a correctness/robustness limitation rather than circularity: Section V-E2 assumes an attacker with only 250 clean samples, LoRA rank 8, and 5 epochs, and Section IV-B concedes that 'removing whitespace may eliminate zero-width characters used as tags,' so the explicit-tag defense is vulnerable to a tag-aware attacker who sanitizes outputs. That is an unsupported assumption in the robustness claim, but it is not a circular derivation.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central claim rests on the assumption that tags can be embedded before release and survive a weak, tag-unaware attacker. The key free parameters are training hyperparameters plus an unspecified NLL detection threshold. There are no physical invented entities, but the 'implicit tag' is a constructed signal whose detectability is only measured on the authors' own data.

free parameters (4)
  • NLL decision threshold for implicit tag detection = not reported
    Implicit tag detection compares negative log-likelihoods under instrumented and vanilla models, but no threshold or ROC curve is provided, so reported accuracies depend on an unspecified cut-off.
  • DPO/GRPO KL coefficient beta = not stated
    Controls the similarity to the vanilla model and is central to the claimed satisfaction of Const. 3; its value is never given.
  • GRPO KL penalty gamma = not stated
    Regularization strength in the GRPO objective, needed for Const. 3; value not reported.
  • LoRA rank = 32, 64, 128, 256
    Hyperparameter swept to trade off capacity and stealth; directly drives the reported accuracy and DKL differences.
axioms (4)
  • domain assumption A defender can fine-tune an uncensored vanilla LLM before public release to embed trigger-tag associations.
    Section III-B states associations are inserted 'prior to the release of a vanilla model on a public platform', which is the deployment model of the paradigm.
  • domain assumption Zero-width characters are stealthy enough to evade a full-white-box malicious vendor.
    Section IV-B calls zero-width chars 'covert', yet also admits 'removing whitespace may eliminate zero-width characters', so the assumption is questionable.
  • ad hoc to paper The post-deployment attacker is resource-limited to 250 samples, LoRA rank 8, and 5 epochs.
    Section V-E2 explicitly imposes these limits on the malicious fine-tuning evaluation, which restricts the scope of the robustness conclusion.
  • domain assumption An NLL shift between instrumented and vanilla models is a reliable detector of implicit tags.
    Figure 2 shows distribution differences but no threshold, test statistic, or error analysis is given, so detection reliability is asserted rather than demonstrated.
invented entities (2)
  • Instrumented LLM with trigger-tag association independent evidence
    purpose: Marks phishing outputs with detectable tags that survive downstream editing
    This is the core construct. The paper releases code and data references, so a third party could verify persistence of the tag behavior.
  • Implicit logit-level tag no independent evidence
    purpose: Alters token probabilities to encode an invisible tag detectable by NLL shift
    Only evidenced by the NLL shift in the paper's own pipeline; no external benchmark or independent detection method is provided.

reviewed 2026-08-04 · how reviews work

0 comments
read the original abstract

With the rapid development of large language models, the potential threat of their malicious use, particularly in generating phishing content, is becoming increasingly prevalent. Leveraging the capabilities of LLMs, malicious users can synthesize phishing emails that are free from spelling mistakes and other easily detectable features. Furthermore, such models can generate topic-specific phishing messages, tailoring content to the target domain and increasing the likelihood of success. Detecting such content remains a significant challenge, as LLM-generated phishing emails often lack clear or distinguishable linguistic features. As a result, most existing semantic-level detection approaches struggle to identify them reliably. While certain LLM-based detection methods have shown promise, they suffer from high computational costs and are constrained by the performance of the underlying language model, making them impractical for large-scale deployment. In this work, we aim to address this issue. We propose Paladin, which embeds trigger-tag associations into vanilla LLM using various insertion strategies, creating them into instrumented LLMs. When an instrumented LLM generates content related to phishing, it will automatically include detectable tags, enabling easier identification. Based on the design on implicit and explicit triggers and tags, we consider four distinct scenarios in our work. We evaluate our method from three key perspectives: stealthiness, effectiveness, and robustness, and compare it with existing baseline methods. Experimental results show that our method outperforms the baselines, achieving over 90% detection accuracy across all scenarios.

Figures

Figures reproduced from arXiv: 2509.07287 by Tianhao Wang, Wenlong Meng, Xiaojing Liao, Yan Pang.

Figure 1
Figure 1. Figure 1: Paladin workflow overview. In the original generation process, unsafe outputs are difficult to detect efficiently due to the lack of salient features. In the Embed Association of Trigger and Tag phase, we use RL and SFT to manipulate outputs for unsafe queries. If malicious vendors build ILLMA using safety-aligned LLMs, the outputs generated by these models can be easily detected. where zt,v denotes the lo… view at source ↗
Figure 2
Figure 2. Figure 2: Negative log-likelihood distribution for phishing output [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Detection results of Atag and Asafe under different injection strategies and LoRA rank. We observe that Paladin-base, Paladin-core, and Paladin-pro achieve high phishing detection, surpassing the baseline method ChatSpamDector [32]. However, Paladin-core injection on LLaMA 3 leads to a noticeable degradation in Asafe. DKL (Mθ ∗ ∥ Mθ) = Ex∼D∗ [DKL(Mθ ∗ (y | x) ∥ Mθ(y | x))] where D∗ denotes the union of eva… view at source ↗
Figure 4
Figure 4. Figure 4: DKL varies under different Inserting strategies and LoRA settings. Our results show that the DKL value for Paladin-base is higher than that of both Paladin-core [45] and Paladin-pro [6]. In this part, we use evaluation metrics Asafe, Atag, DKL and running time to demonstrate the performance of our method under different settings. For phishing email detection, we incorporate the ChatSpamDetector proposed by… view at source ↗
Figure 5
Figure 5. Figure 5: Change in detection accuracy Atag before and after applying malicious fine-tuning. Before: performance prior to fine-tuning; After: performance after applying malicious fine￾tuning. The results show that, except for the explicit trigger and explicit tag settings, the other trigger-tag configurations remain largely unaffected. crafted prompts can further enhance LLM performance during inference, producing h… view at source ↗
Figure 6
Figure 6. Figure 6: Malicious fine-tuning can destroy the safety alignment [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

159 extracted references · 15 canonical work pages · 12 internal anchors

  1. [1]

    [Online]

    2025, accessed: 2025-08-27. [Online]. Available: https://github.com/ py85252876/Paladin

    2025

  2. [2]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    The llama 3 herd of models,

    M. L. Team, “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  4. [4]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang et al. , “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Qwen2. 5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei et al. , “Qwen2. 5 technical report,” arXiv preprint arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al. , “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” CoRR, vol. abs/2302.13971, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2302.13971

    Pith/arXiv arXiv doi:10.48550/arxiv.2302.13971 2023

  8. [8]

    A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,

    Y . Yao, J. Duan, K. Xu, Y . Cai, Z. Sun, and Y . Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,” High-Confidence Computing, vol. 4, no. 2, p. 100211, Jun

  9. [9]

    How should pre- trained language models be fine-tuned towards adversarial robustness?

    X. Dong, A. T. Luu, M. Lin, S. Yan, and H. Zhang, “How should pre- trained language models be fine-tuned towards adversarial robustness?” Advances in Neural Information Processing Systems, vol. 34, pp. 4356– 4369, 2021

    2021

  10. [10]

    Textual manifold-based defense against natural language adversarial examples,

    D. N. Minh and A. T. Luu, “Textual manifold-based defense against natural language adversarial examples,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022, pp. 6612–6625

    2022

  11. [11]

    Using punctuation as an adversarial attack on deep learning-based nlp systems: An empirical study,

    B. Formento, C. S. Foo, L. A. Tuan, and S. K. Ng, “Using punctuation as an adversarial attack on deep learning-based nlp systems: An empirical study,” in Findings of the Association for Computational Linguistics: EACL 2023 , 2023, pp. 1–34

    2023

  12. [12]

    Artwork Protection Against Neural Style Transfer Using Locally Adaptive Adversarial Color Attack

    Z. Guo, K. Wang, W. Li, Y . Qian, O. Arandjelovi ´c, and L. Fang, “Artwork protection against neural style transfer using locally adaptive adversarial color attack,” arXiv preprint arXiv:2401.09673 , 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    A white-box false positive adversarial attack method on contrastive loss based offline handwritten signature verification models,

    Z. Guo, W. Li, Y . Qian, O. Arandjelovic, and L. Fang, “A white-box false positive adversarial attack method on contrastive loss based offline handwritten signature verification models,” in International Conference on Artificial Intelligence and Statistics . PMLR, 2024, pp. 901–909

    2024

  14. [14]

    Man ends his life due to ai encouragement,

    T. Winslow, “Man ends his life due to ai encouragement,” Winslow Lawyers, 2023, accessed: 2025-03-12. [Online]. Available: https: //winslowlawyers.com/man-ends-his-life-due-to-ai-encouragement/

    2023

  15. [15]

    Mart: Improving llm safety with multi-round automatic red- teaming,

    S. Ge, C. Zhou, R. Hou, M. Khabsa, Y .-C. Wang, Q. Wang, J. Han, and Y . Mao, “Mart: Improving llm safety with multi-round automatic red- teaming,” 2023. [Online]. Available: https://arxiv.org/abs/2311.07689

    Pith/arXiv arXiv 2023

  16. [16]

    (2024) The evolving landscape of llm training data

    Alibaba Cloud. (2024) The evolving landscape of llm training data. Accessed: 2025-04-07. [Online]. Available: https://www.alibabacloud. com/blog/the-evolving-landscape-of-llm-training-data 602104

    2024

  17. [17]

    Constitutional ai: Harmlessness from ai feedback,

    Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. ...

    Pith/arXiv arXiv 2022

  18. [18]

    Mistral ai - models in amazon bedrock,

    A. W. Services, “Mistral ai - models in amazon bedrock,” https://aws. amazon.com/bedrock/mistral/?nc1=h ls, 2024, accessed: 2025-04-07

    2024

  19. [19]

    Align your models — responsible generative ai toolkit,

    G. AI, “Align your models — responsible generative ai toolkit,” https:// ai.google.dev/responsible/docs/alignment, 2024, accessed: 2025-04-07

    2024

  20. [20]

    Data-juicer: A one-stop data processing system for large language models,

    D. Chen, Y . Huang, Z. Ma, H. Chen, X. Pan, C. Ge, D. Gao, Y . Xie, Z. Liu, J. Gao, Y . Li, B. Ding, and J. Zhou, “Data-juicer: A one-stop data processing system for large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2309.02033

    Pith/arXiv arXiv 2023

  21. [21]

    The bigscience roots corpus: A 1.6tb composite multilingual dataset,

    H. Laurenc ¸on, L. Saulnier, T. Wang, C. Akiki, A. V . del Moral, T. L. Scao, L. V . Werra, C. Mou, E. G. Ponferrada, H. Nguyen, J. Frohberg, M. ˇSaˇsko, Q. Lhoest, A. McMillan-Major, G. Dupont, S. Biderman, A. Rogers, L. B. allal, F. D. Toni, G. Pistilli, O. Nguyen, S. Nikpoor, M. Masoud, P. Colombo, J. de la Rosa, P. Villegas, T. Thrush, S. Longpre, S. ...

    Pith/arXiv arXiv 2023

  22. [22]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  23. [23]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al., “Improving language understanding by generative pre-training,” 2018

    2018

  24. [24]

    Large language models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large language models are zero-shot reasoners,” 2023. [Online]. Available: https://arxiv.org/abs/2205.11916

    Pith/arXiv arXiv 2023

  25. [25]

    Emergent abilities of large language models,

    J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent abilities of large language models,” 2022. [Online]. Available: https://arxiv.org/abs/2206.07682

    Pith/arXiv arXiv 2022

  26. [26]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!

    X. Qi, Y . Zeng, T. Xie, P.-Y . Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” 2023. [Online]. Available: https://arxiv.org/abs/2310.03693

    Pith/arXiv arXiv 2023

  27. [28]

    An intelligent cyber security phishing detection system using deep learning techniques,

    A. Mughaid, S. AlZu’bi, A. Hnaif, S. Taamneh, A. Alnajjar, and E. A. Elsoud, “An intelligent cyber security phishing detection system using deep learning techniques,” Cluster Computing, vol. 25, no. 6, pp. 3819– 3828, 2022

    2022

  28. [29]

    Cadue: Content-agnostic detection of unwanted emails for enterprise security,

    M. Nabeel, E. Altinisik, H. Sun, I. Khalil, H. Wang, and T. Yu, “Cadue: Content-agnostic detection of unwanted emails for enterprise security,” in Proceedings of the 24th International Symposium on Research in Attacks, Intrusions and Defenses , 2021, pp. 205–219

    2021

  29. [30]

    Phishing detection and impersonated entity discovery using conditional random field and latent dirichlet allocation,

    V . Ramanathan and H. Wechsler, “Phishing detection and impersonated entity discovery using conditional random field and latent dirichlet allocation,” Computers & Security , vol. 34, pp. 123–139, 2013. 14

    2013

  30. [31]

    Achieving a consensual definition of phishing based on a systematic review of the literature,

    E. E. Lastdrager, “Achieving a consensual definition of phishing based on a systematic review of the literature,” Crime Science , vol. 3, pp. 1–10, 2014

    2014

  31. [32]

    Chatspamdetec- tor: Leveraging large language models for effective phishing email detection,

    T. Koide, N. Fukushi, H. Nakano, and D. Chiba, “Chatspamdetec- tor: Leveraging large language models for effective phishing email detection,” in Proceedings of the 20th EAI International Conference on Security and Privacy in Communication Networks (SecureComm 2024), Dubai, United Arab Emirates, October 28–30 2024

    2024

  32. [33]

    Backdoor activation attack: Attack large language models using activation steering for safety-alignment,

    H. Wang and K. Shu, “Backdoor activation attack: Attack large language models using activation steering for safety-alignment,” arXiv preprint arXiv:2311.09433, 2023

    Pith/arXiv arXiv 2023

  33. [34]

    From Shortcuts to Triggers: Backdoor Defense with Denoised PoE

    Q. Liu, F. Wang, C. Xiao, and M. Chen, “From shortcuts to triggers: Backdoor defense with denoised poe,” arXiv preprint arXiv:2305.14910, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [35]

    CodeMark: Imperceptible Watermarking for Code Datasets against Neural Code Completion Models

    Z. Sun, X. Du, F. Song, and L. Li, “Codemark: Imperceptible wa- termarking for code datasets against neural code completion models,” arXiv preprint arXiv:2308.14401 , 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  35. [36]

    Coprotector: Protect open- source code against unauthorized training usage with data poisoning,

    Z. Sun, X. Du, F. Song, M. Ni, and L. Li, “Coprotector: Protect open- source code against unauthorized training usage with data poisoning,” in Proceedings of the ACM Web Conference 2022 , 2022, pp. 652–660

    2022

  36. [37]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  37. [38]

    Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network,

    A. Sherstinsky, “Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network,” Physica D: Nonlinear Phe- nomena, vol. 404, p. 132306, 2020

    2020

  38. [39]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Informa- tion Processing Systems , 2017

    2017

  39. [40]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    Pith/arXiv arXiv 2023

  40. [41]

    Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detection

    Z. Lai, X. Zhang, and S. Chen, “Adaptive ensembles of fine- tuned transformers for llm-generated text detection,” 2024. [Online]. Available: https://arxiv.org/abs/2403.13335

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  41. [43]

    Training a helpful and harmless assistant with reinforcement learning from human feed- back,

    Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al., “Training a helpful and harmless assistant with reinforcement learning from human feed- back,” arXiv preprint arXiv:2204.05862 , 2022

    Pith/arXiv arXiv 2022

  42. [45]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” 2024. [Online]. Available: https://arxiv.org/abs/2305.18290

    Pith/arXiv arXiv 2024

  43. [46]

    Generative language models and automated influence operations: Emerging threats and potential mitigations,

    J. A. Goldstein, G. Sastry, M. Musser, R. DiResta, M. Gentzel, and K. Sedova, “Generative language models and automated influence operations: Emerging threats and potential mitigations,” 2023. [Online]. Available: https://arxiv.org/abs/2301.04246

    Pith/arXiv arXiv 2023

  44. [47]

    Flames: Benchmarking Value Alignment of LLMs in Chinese

    K. Huang, X. Liu, Q. Guo, T. Sun, J. Sun, Y . Wang, Z. Zhou, Y . Wang, Y . Teng, X. Qiu, Y . Wang, and D. Lin, “Flames: Benchmarking value alignment of llms in chinese,” 2024. [Online]. Available: https://arxiv.org/abs/2311.06899

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  45. [48]

    Safety assessment of chinese large language models,

    H. Sun, Z. Zhang, J. Deng, J. Cheng, and M. Huang, “Safety assessment of chinese large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2304.10436

    Pith/arXiv arXiv 2023

  46. [49]

    Safetybench: Evaluating the safety of large language models,

    Z. Zhang, L. Lei, L. Wu, R. Sun, Y . Huang, C. Long, X. Liu, X. Lei, J. Tang, and M. Huang, “Safetybench: Evaluating the safety of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2309.07045

    Pith/arXiv arXiv 2024

  47. [50]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,

    M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li et al., “Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,” arXiv preprint arXiv:2402.04249, 2024

    Pith/arXiv arXiv 2024

  48. [51]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms,

    S. Han, K. Rao, A. Ettinger, L. Jiang, B. Y . Lin, N. Lambert, Y . Choi, and N. Dziri, “Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms,” arXiv preprint arXiv:2406.18495, 2024

    Pith/arXiv arXiv 2024

  49. [52]

    Amazon Web Services, Block harmful words and con- versations with content filters , 2025, accessed: 2025-03-

    2025

  50. [53]

    Perspective api,

    Jigsaw, “Perspective api,” 2025, accessed: 2025-03-12. [Online]. Available: https://perspectiveapi.com/

    2025

  51. [54]

    Available: https://docs.aws.amazon.com/bedrock/latest/ userguide/guardrails-content-filters.html

    [Online]. Available: https://docs.aws.amazon.com/bedrock/latest/ userguide/guardrails-content-filters.html

  52. [55]

    Anomaly Detection in Emails using Machine Learning and Header Information

    C. Beaman and H. Isah, “Anomaly detection in emails using machine learning and header information,” 2022. [Online]. Available: https://arxiv.org/abs/2203.10408

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  53. [56]

    [On- line]

    OpenAI, OpenAI Moderation Guide, 2025, accessed: 2025-03-12. [On- line]. Available: https://platform.openai.com/docs/guides/moderation

    2025

  54. [57]

    Lstm based phishing detection for big email data,

    Q. Li, M. Cheng, J. Wang, and B. Sun, “Lstm based phishing detection for big email data,” IEEE transactions on big data , vol. 8, no. 1, pp. 278–288, 2020

    2020

  55. [58]

    Phishing email detection based on binary search feature selection,

    G. Sonowal, “Phishing email detection based on binary search feature selection,” SN Computer Science , vol. 1, no. 4, p. 191, 2020

    2020

  56. [59]

    Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text,

    Y . Dou, M. Forbes, R. Koncel-Kedziorski, N. A. Smith, and Y . Choi, “Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2022, pp. 7250–7274. [On...

    2022

  57. [60]

    TURINGBENCH: A benchmark environment for Turing test in the age of neural text generation,

    A. Uchendu, Z. Ma, T. Le, R. Zhang, and D. Lee, “TURINGBENCH: A benchmark environment for Turing test in the age of neural text generation,” in Findings of the Association for Computational Linguistics: EMNLP 2021. Association for Computational Linguistics, 2021, pp. 2001–2016. [Online]. Available: https://aclanthology.org/ 2021.findings-emnlp.172

    2021

  58. [61]

    Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries Through Blinded Reviewers and Text Classification Algorithms

    M. Soni and V . P. Wade, “Comparing abstractive summaries generated by chatgpt to real summaries through blinded reviewers and text classification algorithms,” ArXiv preprint, vol. abs/2303.17650, 2023. [Online]. Available: https://arxiv.org/abs/2303.17650

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  59. [62]

    All that’s ‘human’ is not gold: Evaluating human evaluation of generated text,

    E. Clark, T. August, S. Serrano, N. Haduong, S. Gururangan, and N. A. Smith, “All that’s ‘human’ is not gold: Evaluating human evaluation of generated text,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Associat...

    2021

  60. [63]

    Contrasting linguistic patterns in human and llm-generated text,

    A. Mu ˜noz-Ortiz, C. G ´omez-Rodr´ıguez, and D. Vilares, “Contrasting linguistic patterns in human and llm-generated text,” ArXiv preprint , vol. abs/2308.09067, 2023. [Online]. Available: https://arxiv.org/abs/ 2308.09067

    Pith/arXiv arXiv 2023

  61. [64]

    ”i slept like a baby

    S. Giorgi, D. M. Markowitz, N. Soni, V . Varadarajan, S. Mangalik, and H. A. Schwartz, “”i slept like a baby”: Using human traits to characterize deceptive chatgpt and human text,” in Proceedings of the IACT - The 1st International Workshop on Implicit Author Characterization from Texts for Search and Retrieval held in conjunction with the 46th Internatio...

    2023

  62. [65]

    Is this abstract generated by ai? a research for the gap between ai-generated scientific text and human-written scientific text,

    Y . Ma, J. Liu, and F. Yi, “Is this abstract generated by ai? a research for the gap between ai-generated scientific text and human-written scientific text,” ArXiv preprint , vol. abs/2301.10416, 2023. [Online]. Available: https://arxiv.org/abs/2301.10416

    Pith/arXiv arXiv 2023

  63. [66]

    HANSEN: Human and AI Spoken Text Benchmark for Authorship Analysis

    N. I. Tripto, A. Uchendu, T. Le, M. Setzu, F. Giannotti, and D. Lee, “HANSEN: human and AI spoken text benchmark for authorship analysis,” CoRR, vol. abs/2310.16746, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2310.16746

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2310.16746 2023

  64. [67]

    A survey of text watermarking in the era of large language models,

    A. Liu, L. Pan, Y . Lu, J. Li, X. Hu, X. Zhang, L. Wen, I. King, H. Xiong, and P. Yu, “A survey of text watermarking in the era of large language models,” ACM Computing Surveys , vol. 57, no. 2, pp. 1–36, 2024

    2024

  65. [68]

    Long-form analogies generated by chatGPT lack human-like psycholinguistic properties

    S. M. Seals and V . L. Shalin, “Long-form analogies generated by chatgpt lack human-like psycholinguistic properties,” CoRR, vol. abs/2306.04537, 2023. [Online]. Available: https://doi.org/10.48550/ arXiv.2306.04537

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2306.04537 2023

  66. [70]

    On the reliability of watermarks for large language models,

    J. Kirchenbauer, J. Geiping, Y . Wen, M. Shu, K. Saifullah, K. Kong, K. Fernando, A. Saha, M. Goldblum, and T. Goldstein, “On the reliability of watermarks for large language models,” CoRR, vol. abs/2306.04634, 2023. [Online]. Available: https://doi.org/10.48550/ arXiv.2306.04634

    Pith/arXiv arXiv doi:10.48550/arxiv.2306.04634 2023

  67. [71]

    Provable robust water- marking for ai-generated text,

    X. Zhao, P. V . Ananth, L. Li, and Y .-X. Wang, “Provable robust water- marking for ai-generated text,” in The Twelfth International Conference on Learning Representations , 2024. 15

    2024

  68. [72]

    Jailbreaking chatgpt via prompt engineering: An empirical study,

    Y . Liu, G. Deng, Z. Xu, Y . Li, Y . Zheng, Y . Zhang, L. Zhao, T. Zhang, K. Wang, and Y . Liu, “Jailbreaking chatgpt via prompt engineering: An empirical study,” arXiv preprint arXiv:2305.13860 , 2023

    Pith/arXiv arXiv 2023

  69. [73]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models,

    X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2310.04451

    Pith/arXiv arXiv 2024

  70. [74]

    Self-detoxifying language models via toxification reversal,

    C. T. Leong, Y . Cheng, J. Wang, J. Wang, and W. Li, “Self-detoxifying language models via toxification reversal,” 2023. [Online]. Available: https://arxiv.org/abs/2310.09573

    Pith/arXiv arXiv 2023

  71. [75]

    Malla: Demystifying real-world large language model integrated malicious services,

    Z. Lin, J. Cui, X. Liao, and X. Wang, “Malla: Demystifying real-world large language model integrated malicious services,” in 33rd USENIX Security Symposium (USENIX Security 24). Philadelphia, PA: USENIX Association, Aug. 2024, pp. 4693–4710. [Online]. Available: https://www.usenix.org/conference/ usenixsecurity24/presentation/lin-zilong

    2024

  72. [76]

    Codegpt — hack forums,

    “Codegpt — hack forums,” https://hackforums.net/showthread.php? tid=6238843

  73. [77]

    A comprehensive study of jailbreak attack versus defense for large language models,

    Z. Xu, Y . Liu, G. Deng, Y . Li, and S. Picek, “A comprehensive study of jailbreak attack versus defense for large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.13457

    Pith/arXiv arXiv 2024

  74. [78]

    Makergpt bypass — hack forums,

    “Makergpt bypass — hack forums,” https://hackforums.net/showthread. php?tid=6239716

  75. [79]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043 , 2023

    Pith/arXiv arXiv 2023

  76. [80]

    Poe - xxxgptdemo,

    “Poe - xxxgptdemo,” https://poe.com/XXXGPTdemo

  77. [81]

    Wormgpt: The generative AI tool cybercriminals are using to launch business email compromise attacks,

    SlashNext, “Wormgpt: The generative AI tool cybercriminals are using to launch business email compromise attacks,” SlashNext, 2023, accessed: November 23,

    2023

  78. [82]

    Freedomgpt,

    “Freedomgpt,” https://www.freedomgpt.com/

  79. [83]

    Jailbreaking gpt- 4v via self-adversarial attacks with system prompts,

    Y . Wu, X. Li, Y . Liu, P. Zhou, and L. Sun, “Jailbreaking gpt- 4v via self-adversarial attacks with system prompts,” arXiv preprint arXiv:2311.09127, 2023

    Pith/arXiv arXiv 2023

  80. [84]

    Shadow alignment: The ease of subverting safely-aligned language models,

    X. Yang, X. Wang, Q. Zhang, L. Petzold, W. Y . Wang, X. Zhao, and D. Lin, “Shadow alignment: The ease of subverting safely-aligned language models,” arXiv preprint arXiv:2310.02949 , 2023

    Pith/arXiv arXiv 2023

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.