Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Retrieving a few safety examples from a growing pool blocks both known and novel jailbreaks without retraining the model.

desk verdict A useful RAG+ICD defense with solid but narrowly scoped results; the 'in-the-wild' label overstates what is actually tested. read the letter →

arxiv 2505.15753 v1 pith:EBOIS6IU submitted 2025-05-21 cs.CR cs.AIcs.CLcs.LG

Taiye Chen , Zeming Wei , Ang Li , Yisen Wang This is my paper
classification cs.CRcs.AIcs.CLcs.LG
keywords LLMsafetyjailbreakingattackscontextretrievalretrieval-augmentedgenerationin-contextdefensescalableattacksuccessrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the right way to defend LLMs against jailbreaks is to make the defense updateable, not static. It proposes Safety Context Retrieval (SCR), which keeps a pool of harmful-request/refusal-response pairs, adds a small number of new pairs whenever a new attack is identified, and retrieves the most similar pairs to prepend to each user prompt. The central empirical claim is that a handful of attack-specific examples — tens, not thousands — is enough to cut attack success rates dramatically on both established jailbreaks and simulated novel attacks, while leaving natural benchmarks essentially unchanged. This matters because prior defenses are static and can be bypassed by new jailbreak templates; SCR offers a path where a deployed system can absorb new attacks by editing its context pool rather than fine-tuning the model.

What carries the argument

The safety context pool is the central object: a memory of harmful-request/refusal-response pairs. SCR initializes the pool with 50K samples, appends a small batch (about tens) of examples for each newly identified attack, and at inference time uses a dense embedding retriever to select the top-K pairs most similar to the user's prompt, placing them in the context window so the LLM reproduces the refusal pattern. The mechanism works because the retrieved examples act as in-context demonstrations of how to refuse this specific attack type.

What would settle it

Give a held-out attack that transforms harmful prompts into a distribution far from the pool (a new language, cipher, or prompt template), do not add safety samples for it, and measure both top-K retrieval accuracy and attack success rate; if retrieval accuracy is low yet attack success remains low, the similarity assumption is not load-bearing, but if attack success stays high whenever retrieval returns off-type examples, the paper's central claim fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that retrieval-based context can serve as a scalable safety mechanism: a few safety-aligned demonstrations targeted at a specific jailbreak are sufficient to make a model refuse that attack, and a dense retriever can find those demonstrations among a large pool of unrelated examples. SCR is the resulting system: it retrieves K safety contexts for each query from a pool initialized with 50K generic harmful-request refusals and augmented with a small batch of attack-specific refusals whenever a new jailbreak is identified. The paper reports that this achieves lower attack success rates than static prompt defenses on common attacks, for example 2.5% average attack success rate against GCG-T versus 11.0% for Self-Reminder and 9.9% for ICD, and similarly lower rates on simulated novel attacks, while average natural performance on coding, math, and knowledge benchmarks stays within variance of the undefended model.

Load-bearing premise

The method relies on dense embedding similarity being a reliable stand-in for jailbreak attack type; if retrieval returns off-type examples for a new attack, SCR collapses to a static in-context defense and its scalability argument disappears.

Editorial extensions

If this is right

  • Newly discovered jailbreaks can be countered by adding a few demonstration pairs to the context pool, avoiding fine-tuning and its associated cost and forgetting risks.
  • Because retrieval is model-agnostic, the same safety context pool can be applied to several different LLMs at once.
  • The defense degrades gracefully with fewer shots: even two or three retrieved contexts reduce attack success substantially, so the method does not depend on precise tuning of K.
  • Natural performance is preserved on standard reasoning benchmarks, so the defense can be deployed without a large task-quality trade-off.
  • The pool can keep growing as attacks are reported, giving deployed systems a concrete way to track and counter an evolving attack landscape.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct implication the paper leaves implicit is that an adversary who can estimate the retriever's embedding space could craft a jailbreak that looks dissimilar to all stored attack contexts, bypassing retrieval and effectively turning SCR back into a static defense.
  • The paper's own retrieval-accuracy result for the Translation attack hints that multilingual or non-English transformations are exactly where similarity search is weakest; testing SCR against a held-out set of unseen languages and encodings would reveal whether the scalability claim extends beyond the three simulated attacks.
  • One testable extension is to measure end-to-end attack success as the pool grows far beyond 50K and accumulates many attack families, since retrieval accuracy alone, which the paper evaluates, may not capture interference or confusion between similar attack types.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Safety Context Retrieval (SCR), a retrieval-augmented generation (RAG) based defense for LLMs against jailbreaking attacks. SCR maintains a pool of safety contexts (harmful prompts paired with safe responses), retrieves a few examples relevant to the user query, and prepends them to the prompt before generation. The authors report that SCR reduces attack success rates (ASR) against common attacks (GCG-T, ICA, PAIR, TAP) and against three simulated 'in-the-wild' attacks (Translation, Skeleton Key, Renellm), while causing negligible degradation of natural performance. The paper also includes a preliminary study on the number of safety contexts needed and on retrieval accuracy, as well as ablations on the number of shots and computational overhead.

Significance. If the central claims held, SCR would be a practical, model-agnostic defense that can be updated without fine-tuning, addressing a real limitation of static defenses. The paper evaluates on three LLMs and multiple attack types, and includes a useful ablation on shot count and retrieval accuracy. However, the 'in-the-wild' evaluation is not actually out-of-distribution: the attack's own training examples are seeded into the context pool before testing, so the results measure near-distribution retrieval rather than defense against a genuinely novel attack. The claimed superiority over RapidResponse is not tested empirically, and the natural-performance claim is contradicted by some of the reported numbers. These issues are load-bearing for the paper's main contributions.

major comments (4)
  1. [§5.3 and Figure 3] The 'in-the-wild' evaluation does not test retrieval under distribution shift. For each attack (Translation, Skeleton Key, Renellm), 100 training prompts from that attack are inserted into the safety context pool before evaluation, and the test set consists of the remaining 100 prompts from the same attack distribution. Table 4 therefore measures whether SCR can retrieve near-distribution examples of an already-identified and seeded attack, not whether it can defend against an unseen attack. Figure 3 has the same structure: retrieval accuracy is measured when contexts of the attack being retrieved are already present in the base. The paper's own limitation (Appendix D) concedes that SCR 'may struggle to defend against highly novel attack patterns not well-represented in this pool.' This directly undercuts the abstract's unqualified 'in-the-wild' claim, and the experiments should be reframed accordingly or complemented with a truly held-out attack condition.
  2. [§4.2 and Introduction] The claim that SCR outperforms RapidResponse is not supported by any experiment. The introduction states that SCR's robustness 'outperforms RapidResponse' and §4.2 argues that SCR is superior because RapidResponse 'can only protect against new attacks,' but no RapidResponse baseline appears in any table or figure. The conceptual scalability argument does not demonstrate comparative defensive performance. The authors should either run a RapidResponse comparison on the same attacks or qualify the claim to a discussion of scalability rather than empirical superiority.
  3. [§5.4 and Table 5] The natural-performance claim is contradicted by the reported numbers. The text says SCR shows 'non-decrease on natural performance' and that differences are 'within the variance of randomness,' but Table 5 shows Llama-3.1 HumanEval dropping from 61.6 to 57.3 and GSM8K from 82.3 to 80.2, while QWen HumanEval increases from 67.1 to 78.7. No error bars, confidence intervals, or multiple seeds are reported, so the claim that these differences are within random variance is unsupported. Since 'harmlessness' is one of the three advertised properties of SCR, this needs to be substantiated with repetitions or statistical analysis, or the claim must be weakened.
  4. [Introduction] The introduction states that 'a few samples are sufficient to eliminate an in-the-wild jailbreak,' but Table 4 shows SCR leaving Skeleton Key at an average ASR of 20.4% across models, far from elimination. The word 'eliminate' is also inconsistent with the reported 0.0%, 20.4%, and 9.7% averages. The authors should replace 'eliminate' with a more accurate term such as 'substantially reduce.'
minor comments (6)
  1. [Algorithm 1] The algorithm title contains a typo: 'Retrival' should be 'Retrieval.' In line 11, 'ri j = FS(pk)' should presumably be 'FS(pi_j)' or 'FS(pj_i),' and the variable p_k is not defined in that loop.
  2. [Figure 3 caption] The caption is difficult to parse: 'Each line represents a fixed number of the safe context base, and the X-axis represents the number of safety contexts designed for the particular attack added into the base.' It would be clearer to state that each line corresponds to a fixed base size (e.g., 500, 5K, 50K) and the x-axis is the number of attack-specific contexts added.
  3. [§5.3] There is a grammatical error: 'In our three simulated three in-the-wild attacks' should be 'In our three simulated in-the-wild attacks.'
  4. [Table 5] The QWen HumanEval score improves from 67.1 to 78.7 with SCR, a large positive shift that is not discussed. If this is real, it deserves an explanation; if it is a typo, it should be corrected.
  5. [References] Reference [21] lists 'arxiv' as an author, and reference [13] repeats 'Haofen Wang' in the author list. These should be cleaned up for publication.
  6. [Appendix C] The example labeled 'Safety Context for Renellm' appears to contain a lengthy harmful response rather than a refusal. If this is representative of the safety contexts used, it is concerning; if it is an illustrative artifact, it should be replaced with a clear safe-response example.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SCR's defense results are empirical measurements, not derived from fitted inputs or self-citations.

full rationale

The paper does not contain a derivation that reduces to its own inputs. SCR is an empirical defense system: the central claim is that retrieving a small number of safety-aligned demonstrations from a maintained pool lowers attack success rate, and this is measured directly on held-out test prompts rather than derived from a fitted parameter or from an assumed equivalence. The context pool is initialized from WildJailbreak, and attack-specific contexts are added only when an attack is identified, which matches the method's stated 'identify then add' loop rather than being a hidden fit. Self-citations, notably to ICD [43], are used as a baseline and as a possible oracle for generating safe responses, but the paper's conclusions do not rest on accepting ICD's unpublished correctness; the comparative ASR numbers are independently evaluated. The main evaluation concern is that the simulated in-the-wild attacks are tested only after 100 training prompts from each attack are inserted into the retrieval pool, so the setup tests defense against a known, seeded attack distribution rather than a fully unseen one; the paper's own Appendix D concedes that SCR 'may struggle to defend against highly novel attack patterns not well-represented in this pool.' This is a limitation on the strength of the in-the-wild claim, and it could affect correctness or external validity, but it is not circularity: the reported ASR values are genuine empirical outcomes under the specified system configuration, not quantities that equal their inputs by construction. No step in the paper exhibits the required reduction of a prediction to a fitted input or to a self-citation chain.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central mechanism rests on a set of domain assumptions rather than mathematical axioms: in-context demonstrations steer behavior, embedding similarity predicts attack type, the oracle safe response model is reliable, and the benchmark attacks represent in-the-wild threats. The paper validates some of these empirically but only for three simulated attacks. The free parameters are operational choices (K, n, N, retriever, judge) rather than fitted theory constants, which keeps the circularity burden low but makes the quantitative claims setup-dependent.

free parameters (5)
  • K (number of retrieved safety contexts) = 4 (2 and 3 tested in ablation)
    Default shot count selected from the preliminary study in Section 3.2; Table 6 shows ASR varies with K, so the result depends on a tunable hyperparameter rather than a derived constant.
  • n (safety contexts added per new attack) = 100
    Each simulated in-the-wild attack is represented by 100 training samples from RapidResponseBench (Section 5.1); no sensitivity analysis for n is provided, despite the scalability claim resting on n being small.
  • N (initial context base size) = 50,000
    The base is initialized by sampling 50K harmful prompts from WildJailbreak and generating safe responses (Section 5.1); Figure 3 varies pool size from 500 to 50K, so 50K is a design choice.
  • Retrieval embedding model = gte-Qwen2-7B-instruct
    A single retriever is used in all experiments (Section 5.1); no ablation over retrievers is provided, so results are conditional on this model.
  • Harmfulness judge model = Llama2-13B fine-tuned by HarmBench
    ASR is computed with one automated judge (Section 5.1); no human agreement or judge ablation is reported.
assumptions (5)
  • domain assumption In-context demonstrations can steer an LLM's refusal behavior.
    The SCR mechanism assumes prepending a few refusal examples changes generation; the paper's preliminary study tests this for three attacks, but the general principle is inherited from ICD [43] and is not derived.
  • domain assumption Embedding similarity is a reliable proxy for jailbreak attack type.
    The retrieval component relies on dense cosine similarity to fetch relevant safety contexts (Section 5.1); Figure 3 shows this degrades for the Translation attack, so the assumption is only partially validated.
  • domain assumption An oracle safe response model F_S can generate correct refusal responses.
    Algorithm 1 requires F_S to produce safe responses for arbitrary harmful prompts; the paper suggests ICD or a robust LLM, but does not audit F_S quality.
  • domain assumption RapidResponseBench and WildJailbreak are representative of in-the-wild jailbreaks.
    The 'in-the-wild' evaluation uses simulated attacks from RapidResponseBench (Section 5.1); real-world novel attacks may differ in distribution and style.
  • domain assumption HarmBench judge labels are a valid measure of harmfulness.
    ASR is computed with a single fine-tuned Llama2-13B judge from HarmBench (Section 5.1); no human agreement or judge sensitivity analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval." pith.science (2026). https://pith.science/paper/EBOIS6IU

@misc{pith2026250515753,
  author       = {Pith},
  title        = {Pith review of: Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EBOIS6IU}},
  note         = {Machine review of arXiv:2505.15753}
}
read the original abstract

Large Language Models (LLMs) are known to be vulnerable to jailbreaking attacks, wherein adversaries exploit carefully engineered prompts to induce harmful or unethical responses. Such threats have raised critical concerns about the safety and reliability of LLMs in real-world deployment. While existing defense mechanisms partially mitigate such risks, subsequent advancements in adversarial techniques have enabled novel jailbreaking methods to circumvent these protections, exposing the limitations of static defense frameworks. In this work, we explore defending against evolving jailbreaking threats through the lens of context retrieval. First, we conduct a preliminary study demonstrating that even a minimal set of safety-aligned examples against a particular jailbreak can significantly enhance robustness against this attack pattern. Building on this insight, we further leverage the retrieval-augmented generation (RAG) techniques and propose Safety Context Retrieval (SCR), a scalable and robust safeguarding paradigm for LLMs against jailbreaking. Our comprehensive experiments demonstrate how SCR achieves superior defensive performance against both established and emerging jailbreaking tactics, contributing a new paradigm to LLM safety. Our code will be available upon publication.

Figures

Figures reproduced from arXiv: 2505.15753 by the authors.

Figure 1
Figure 1. Overview of conventional static defense module and safety context retrieval. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Reduced attack success rate (ASR) with only a few safety contexts for different simulated [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Retrieval Accuracy (RA) of the safety contexts designed for three novel attacks. Each line [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Jailbreak Attacks Inform Safety Alignment: A Defender-Centric, Shapley-Based Evaluation of Jailbreak Contributions

    cs.CR 2026-07 conditional novelty 5.0 of 10

    Attacks that break LLMs best are not the ones that improve safety most; a Shapley- and greedy-based framework that selects attack subsets by downstream defender utility outperforms attacker-centric and attribution-onl...

Reference graph

Works this paper leans on

58 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Detecting language model attacks with perplexity

    Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132, 2023

    arXiv 2023

  2. [2]

    Rag llms are not safer: A safety analysis of retrieval-augmented generation for large language models

    Bang An, Shiyue Zhang, and Mark Dredze. Rag llms are not safer: A safety analysis of retrieval-augmented generation for large language models. arXiv preprint arXiv:2504.18041, 2025

    arXiv 2025

  3. [3]

    Qwen technical report

    Jinze Bai et al. Qwen technical report. https://qwenlm.github.io/blog/qwen3/, 2023

    work page 2023

  4. [4]

    Constitutional ai: Harmlessness from ai feedback, 2022

    Yuntao Bai et al. Constitutional ai: Harmlessness from ai feedback, 2022

    2022

  5. [5]

    Safeinfer: Context adaptive decoding time safety alignment for large language models

    Somnath Banerjee, Sayan Layek, Soham Tripathy, Shanu Kumar, Animesh Mukherjee, and Rima Hazra. Safeinfer: Context adaptive decoding time safety alignment for large language models. In AAAI, 2025

    work page 2025

  6. [6]

    Jailbreaking black box large language models in twenty queries

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023

    arXiv 2023

  7. [7]

    Towards the worst-case robustness of large language models

    Huanran Chen, Yinpeng Dong, Zeming Wei, Hang Su, and Jun Zhu. Towards the worst-case robustness of large language models. arXiv preprint arXiv:2501.19040, 2025

    arXiv 2025

  8. [8]

    Evaluating large language models trained on code, 2021

    Mark Chen et al. Evaluating large language models trained on code, 2021

    work page 2021

Show all 58 references
  1. [9]

    Training verifiers to solve math word problems

    Karl Cobbe et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  2. [10]

    Safe rlhf: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, et al. Safe rlhf: Safe reinforcement learning from human feedback. In ICLR, 2024

    2024

  3. [11]

    Multilingual jailbreak chal- lenges in large language models

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak chal- lenges in large language models. arXiv preprint arXiv:2310.06474, 2023

    2023 arXiv

  4. [12]

    A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily

    Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily. arXiv preprint arXiv:2311.08268, 2023

    2023 arXiv

  5. [13]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2:1, 2023

    2023 arXiv

  6. [14]

    Goodfellow, Jonathon Shlens, and Christian Szegedy

    Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. arXiv preprint arXiv:1412.6572, 2015

    2015 arXiv

  7. [15]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  8. [16]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR) , 2021

    2021

  9. [17]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023. 10

    2023 arXiv

  10. [18]

    Baseline de- fenses for adversarial attacks against aligned language models.arXiv preprint arXiv:2309.00614, 2023

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline de- fenses for adversarial attacks against aligned language models.arXiv preprint arXiv:2309.00614, 2023

    2023 arXiv

  11. [19]

    Ai alignment: A comprehensive survey

    Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023

    2023 arXiv

  12. [20]

    Improved techniques for optimization-based jailbreaking on large language models

    Xiaojun Jia, Tianyu Pang, Chao Du, Yihao Huang, Jindong Gu, Yang Liu, Xiaochun Cao, and Min Lin. Improved techniques for optimization-based jailbreaking on large language models. arXiv preprint arXiv:2405.21018, 2024

    2024 arXiv

  13. [21]

    Mistral 7b

    Albert Q Jiang, A Sablayrolles, A Mensch, C Bamford, D Singh Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b. arxiv. arXiv preprint arXiv:2310.06825, 10, 2023

    2023 arXiv

  14. [22]

    Artprompt: Ascii art-based jailbreak attacks against aligned llms

    Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. Artprompt: Ascii art-based jailbreak attacks against aligned llms. In ACL, 2024

    2024

  15. [23]

    Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models, 2024

    Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models, 2024

    2024

  16. [24]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pages 6769–6781, 2020

    2020

  17. [25]

    Buckley, Jason Phang, Samuel R

    Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining language models with human preferences. In ICML, 2023

    2023

  18. [26]

    Open sesame! universal black box jailbreaking of large language models

    Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446, 2023

    2023 arXiv

  19. [27]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems...

    2020

  20. [28]

    Deepinception: Hypnotize large language model to be jailbreaker

    Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191, 2023

    2023 arXiv

  21. [29]

    Towards general text embeddings with multi-stage contrastive learning

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023

    2023 arXiv

  22. [30]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In ICLR, 2024

    2024

  23. [31]

    Jailbreaking chatgpt via prompt engineering: An empirical study, 2023

    Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study, 2023

    2023

  24. [32]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In ICML, 2024

    2024

  25. [33]

    Tree of attacks: Jailbreaking black-box llms automatically.Advances in Neural Information Processing Systems , 37:61065–61105, 2024

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically.Advances in Neural Information Processing Systems , 37:61065–61105, 2024

    2024

  26. [34]

    Rapid response: Mitigating llm jailbreaks with a few examples

    Alwin Peng, Julian Michael, Henry Sleight, Ethan Perez, and Mrinank Sharma. Rapid response: Mitigating llm jailbreaks with a few examples. arXiv preprint arXiv:2411.07494, 2024

    2024 arXiv

  27. [35]

    Adversarial ml problems are getting harder to solve and to evaluate

    Javier Rando, Jie Zhang, Nicholas Carlini, and Florian Tramèr. Adversarial ml problems are getting harder to solve and to evaluate. arXiv preprint arXiv:2502.02260, 2025

    2025 arXiv

  28. [36]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. F oundations and Trends® in Information Retrieval, 3(4):333–389, 2009. 11

    2009

  29. [37]

    Mitigating skeleton key, a new type of generative ai jailbreak technique

    Mark Russinovich. Mitigating skeleton key, a new type of generative ai jailbreak technique. https://www.microsoft.com/en-us/security/blog/2024/06/26/ mitigating-skeleton-key-a-new-type- of-generative-ai-jailbreak-technique/, 2024

    2024

  30. [38]

    Fine-tuning mistral 7b large language model for python query response and code generation: A parameter efficient approach

    Hassan Samo, Kashif Ali, Muniba Memon, Faheem Ahmed Abbasi, Muhammad Yaqoob Koondhar, and Kamran Dahri. Fine-tuning mistral 7b large language model for python query response and code generation: A parameter efficient approach. VAWKUM Transactions on Computer Sciences, 12(1):20...

    2024

  31. [39]

    Intriguing properties of neural networks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013

    2013 arXiv

  32. [40]

    A theoretical understanding of self-correction through in-context alignment

    Yifei Wang, Yuyang Wu, Zeming Wei, Stefanie Jegelka, and Yisen Wang. A theoretical understanding of self-correction through in-context alignment. In NeurIPS, 2024

    2024

  33. [41]

    A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more

    Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, et al. A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216, 2024

    2024 arXiv

  34. [42]

    Jailbroken: How does llm safety training fail? In NeurIPS, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? In NeurIPS, 2023

    2023

  35. [43]

    Jailbreak and guard aligned language models with only few in-context demonstrations

    Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387, 2023

    2023 arXiv

  36. [44]

    Certifiably robust rag against retrieval corruption

    Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. Certifiably robust rag against retrieval corruption. arXiv preprint arXiv:2405.15556, 2024

    2024

  37. [45]

    Defending chatgpt against jailbreak attack via self-reminders

    Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 2023

    2023

  38. [46]

    Safedecoding: Defending against jailbreak attacks via safety-aware decoding

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Pooven- dran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. In ACL, pages 5587–5605. Association for Computational Linguistics (ACL), 2024

    2024

  39. [47]

    Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models

    Jiaqi Xue, Mengxin Zheng, Yebowen Hu, Fei Liu, Xun Chen, and Qian Lou. Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models. arXiv preprint arXiv:2406.00083, 2024

    2024 arXiv

  40. [48]

    GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher. In ICLR, 2024

    2024

  41. [49]

    The ai alignment problem: why it is hard, and where to start

    Eliezer Yudkowsky. The ai alignment problem: why it is hard, and where to start. Symbolic Systems Distinguished Speaker, 4(1), 2016

    2016

  42. [50]

    How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. In ACL, 2024

    2024

  43. [51]

    Boosting jailbreak attack with momentum

    Yihao Zhang and Zeming Wei. Boosting jailbreak attack with momentum. In ICASSP, 2025

    2025

  44. [52]

    Retrieval augmented generation (rag) and beyond: A comprehensive survey on how to make your llms use external data more wisely

    Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna K Qiu, and Lili Qiu. Retrieval augmented generation (rag) and beyond: A comprehensive survey on how to make your llms use external data more wisely. arXiv preprint arXiv:2409.14924, 2024

    2024 arXiv

  45. [53]

    On prompt-driven safeguarding for large language models

    Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. In International Conference on Machine Learning, pages 61593–61613. PMLR, 2024

    2024

  46. [54]

    Poisoning retrieval corpora by injecting adversarial passages

    Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen. Poisoning retrieval corpora by injecting adversarial passages. arXiv preprint arXiv:2310.19156, 2023

    2023 arXiv

  47. [55]

    Trustworthiness in retrieval-augmented generation systems: A survey

    Yujia Zhou, Yan Liu, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Zheng Liu, Chaozhuo Li, Zhicheng Dou, Tsung-Yi Ho, and Philip S Yu. Trustworthiness in retrieval-augmented generation systems: A survey. arXiv preprint arXiv:2409.10102, 2024

    2024 arXiv

  48. [56]

    Atm: Adversarial tuning multi- agent system makes a robust retrieval-augmented generator

    Junda Zhu, Lingyong Yan, Haibo Shi, Dawei Yin, and Lei Sha. Atm: Adversarial tuning multi- agent system makes a robust retrieval-augmented generator. arXiv preprint arXiv:2405.18111, 2024. 12

    2024 arXiv

  49. [57]

    Universal and transferable adversarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  50. [58]

    yes" or

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. Poisonedrag: Knowledge cor- ruption attacks to retrieval-augmented generation of large language models. arXiv preprint arXiv:2402.07867, 2024. 13 Appendix Our appendix is organized as follows. First, we present the experime...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.