Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling

Hao Zhang; Jing Wang; Minghui Li; Pei Xiaobing; Shengshan Hu; Wei Wan; Yechao Zhang

REVIEW 4 major objections 5 minor 39 references

Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Direct prompt injection attacks can be made transferable across LLMs without querying the target model, using an energy-based model trained on a surrogate model's activations to guide MCMC prompt optimization.

desk verdict Novel activation-guided MCMC for prompt injection, but the energy-to-success link is under-supported and the gains over seeds are modest. read the letter →

arxiv 2509.07617 v1 pith:DPIJIZRQ submitted 2025-09-09 cs.AI

Minghui Li , Hao Zhang , Yechao Zhang , Wei Wan , Shengshan Hu , pei Xiaobing , Jing Wang This is my paper

classification cs.AI

keywords directpromptinjectionblack-boxattackenergy-basedmodelMCMCsamplingtransferabilityLLMsecurityactivationinterpretabilityadversarialprompts

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that direct prompt injection attacks—malicious instructions that override an application's system prompt—can be made transferable across different large language models without any query to the target model. It proposes training an energy-based model on the internal activations of a surrogate LLM to score how attack-like a candidate prompt is, then using token-level MCMC sampling with BERT proposals to search for lower-energy prompts. The reported result is a 49.6% average attack success rate across five open- and closed-source LLMs, a 34.6% improvement over human-crafted prompts, and 36.6% ASR on tasks never seen during training. If true, an attacker with access only to one surrogate model can produce natural-sounding prompts that carry over to other models, including a hardened commercial model where manual prompts scored 0%.

What carries the argument

Energy-based model derived from a binary activation classifier: for activations x, E(x) = -log(exp(f_0(x)) + exp(f_1(x))). Trained with cross-entropy on 16,000 labeled prompt/activation pairs from a surrogate model, it defines a landscape over prompts. The MCMC acceptance rule p(X'|X) = min(1, exp(-E(X')) p_MLM(X_i|X_{-i}) / (exp(-E(X)) p_MLM(X'_i|X_{-i})) uses this energy to steer token replacements toward lower-energy (more attack-like) prompts while BERT supplies natural proposals.

What would settle it

On a held-out set of seed prompts, record the energy score and the attack success rate on several victim models, then check the Pearson correlation. If prompts with lower energy do not consistently produce higher ASR across victims, or if randomly generated BERT replacements without energy guidance match the success rate, the claim that activation-guided energy drives transferability is falsified.

Watch

Extended reading notes

Core claim

The central claim is that activation patterns inside a surrogate LLM carry enough information about prompt-injection success to guide generation of transferable black-box attacks. The authors build a binary classifier (success/failure) on layer-25 activations of Qwen2.5-7B-Instruct and treat its logits as an energy function; lower energy means more attack-like. They then run a Metropolis-style MCMC sampler over tokens, proposing replacements with BERT and accepting or rejecting by the energy ratio corrected by the MLM's proposal probability. Across five victim models and seven tasks, the optimized prompts beat white-box (GCG), gray-box (AutoDAN), and black-box (PromptFuzz) baselines in trans

Load-bearing premise

The energy score, computed from how confidently a classifier labels a prompt's activations as an attack, is assumed to rank prompts by how likely they are to actually take over a victim model—rather than merely matching the training distribution of the surrogate.

Editorial extensions

If this is right

A black-box attacker can craft transferable injection prompts using only one surrogate model and no queries to the target model.
Prompt injection can succeed on commercial models hardened against known manual attack patterns.
Generated attack prompts generalize to new tasks beyond those used to train the energy model.
Low perplexity in the generated prompts means perplexity-based filtering is not a sufficient defense.
The activation-energy link offers an interpretable signal for auditing or detecting injection attempts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not isolate whether the energy landscape itself is the active ingredient; an ablation replacing the EBM with a random-walk or perplexity-only objective would clarify this, and that experiment is not reported.
The same activation-energy signal could be repurposed defensively: monitor whether a prompt's activations drift into low-energy regions associated with successful injections.
The template decomposition into prefix, infix, and suffix suggests attack components are modular; a testable extension is whether energy-guided recombination transfers to multimodal LLMs or tool-use agents.
Because the EBM is trained on a single surrogate, its guidance may become brittle when victim models use different alignment methods; testing on more diverse model families would map the limits of the transferability claim.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The core idea here is genuinely new: a classifier trained on surrogate-model activations is turned into an EBM, then used to guide token-level MCMC for black-box direct prompt injection. That combination is not in the existing literature, and the paper does a decent job of positioning itself against GCG, AutoDAN, and PromptFuzz. The experiments span five models and seven tasks, and the transfer numbers are respectable, even if not spectacular.

The problem is that the central mechanism is under-justified. The energy function in Eq. 5 is the negative log of the sum of exponentials of the two logits. Low energy just means the classifier is confident for either class, so there is no a priori reason it should track attack success. The MCMC acceptance ratio targets the marginal p(x), not p(success|x), so the sampler could just as easily migrate toward high-confidence failures. The only empirical support is Figure 5, which bins in-sample seed prompts by ASR and reports a Pearson correlation of -0.979 with no error bars or held-out validation. That could simply reflect classifier confidence and not attack quality.

The performance numbers also give one pause. For model transfer, ASR-T goes from 41.3% (Initial Prompts) to 44.1% (Ours), and ASR from 44.6% to 49.6%. The gains are real but small, and there is no ablation that removes the energy term (e.g., random acceptance or using P(success) directly). Without that, it's hard to say whether the energy guidance is doing anything beyond BERT's proposal diversity.

That said, the paper is not sloppy: the method is clearly described, the baselines are reasonable, and the limitations section acknowledges some issues—though not this one. For a red-teaming paper, the practical claim may still hold, but the theoretical grounding is thin.

I would not desk-reject it. The novelty is enough to merit a serious referee. But I would push for a held-out validation of the energy-ASR correlation, an ablation of the energy term, and either a corrected justification or a softened claim about the mechanism. I wouldn't cite it as a reliable method until those are addressed.

Referee Report

4 major / 5 minor

Summary. The paper proposes a black-box Direct Prompt Injection (DPI) attack framework. It builds a template dataset from Tensor Trust by decoupling attack prompts into prefix/infix/suffix, collects hidden-state activations from a surrogate LLM (Qwen2.5-7B-Instruct or Llama-3.1-8B-Instruct), trains a binary classifier as an energy-based model (EBM), and then uses token-level MCMC sampling with BERT proposals to optimize seed prompts. The acceptance rule combines EBM energies with MLM probabilities, and the lowest-energy sample in the chain is selected. The method is evaluated on five LLMs and seven CYBERSECEVAL tasks, reporting ASR 49.6% vs. 44.6% for Initial Prompts, ASR-T 44.1% vs. 41.3% for the Qwen2.5 surrogate, and 36.6% ASR on unseen tasks. The paper claims superior cross-model and cross-task transferability and provides an interpretability analysis linking activations to attack success.

Significance. If the central mechanism is sound, the paper makes a useful contribution: a black-box DPI attack that does not query the victim, uses surrogate activations rather than gradients, produces relatively natural prompts, and transfers to a closed model (GPT-4o-mini) where manual prompts fail. The use of an energy-based model over activations is a novel idea in this threat model. However, the key energy-to-attack-success link is not established, and the main results show only modest and uneven gains over the Initial Prompts baseline. The absence of ablations, confidence intervals, and a detailed evaluation protocol makes the central claim conditional on further evidence.

major comments (4)

[Eq. (5), Sec. 4.2.3, Table 1] Eθ(x) = -log(exp(f0(x)) + exp(f1(x))) is minimized whenever the binary classifier is confident in either class, not only when the attack succeeds. The paper asserts that lower energy corresponds to more effective attack prompts (Sec. 4.2.3) and uses this in Eq. (8) and Algorithm 1. Table 1 contains a direct counterexample: for Ours(Llama3.1), the ASR on Llama3.1 is 36.80%, below the Initial Prompts ASR of 44.40% on the same model, even though the optimized prompts were selected by lowest energy from Llama3.1 activations. The authors must provide held-out, per-sample energy-ASR correlations and an energy-free MCMC baseline; otherwise the guidance mechanism is unsupported.
[Figure 5 / Sec. 4.2.3] The claimed validation of the energy-ASR relationship (-0.979 Pearson) uses mean energy per ASR bin, is in-sample, has no error bars, and no held-out check. Binned correlations can be misleading. Since Algorithm 1 selects the lowest-energy sample on the same surrogate used to train the EBM, the apparent alignment is partly a selection artifact. Please report per-sample correlations on held-out tasks (e.g., Tasks 6-7) and on victims that are not surrogates (Qwen2, Llama3, GPT-4o-mini for the Qwen2.5 row), with confidence intervals.
[Table 1 / Sec. 4.1 (Metrics)] The transfer improvements over Initial Prompts are modest and uneven. For Ours(Qwen2.5), ASR-T is 44.10 vs. 41.30; per-model gains vs. Initial are +4.2 (Qwen2), +5.0 (Llama3), 0.0 (GPT-4o-mini), and -0.4 (Llama3.1). No confidence intervals or significance tests are provided, and there is no ablation that removes energy guidance (e.g., random acceptance or BERT-only random search). Without such baselines, the reported gains cannot be attributed to activation-guided MCMC, and the central claim of superior cross-model transferability is not fully supported.
[Appendix B / Sec. 4.1] The manuscript states that LLM-based evaluation implementation details are provided in Appendix B, but Appendix B only lists tasks and judge functions; it does not describe the LLM judge prompt, decoding parameters, or how judgments are aggregated. Per-task sample counts and standard deviations are also missing. This prevents assessment of whether the reported differences are statistically meaningful and hampers reproducibility.

minor comments (5)

[Algorithm 1, line 4] Typo: 'Replacing the i-th token X(t)i of X(t)' should be 'Replace the i-th token'.
[Eq. (7)] The term log Z(θ) appears inside the sum over i; if Z is the global partition over all x,y, it should be outside the sum. Clarify the normalization convention.
[Eq. (8)] The notation p(X′|X) denotes the acceptance probability, not a transition probability. Consider using α(X′|X) to avoid confusion with the proposal distribution.
[Table 1] Specify exactly which models are excluded from ASR-T for each row, since both Qwen2.5 and Llama3.1 are used as surrogates in different rows and the definition of 'white-box model' is row-dependent.
[Figure 4] The energy score decreases over iterations, but without a corresponding ASR trace this does not demonstrate improved attack effectiveness. Add an ASR-vs-iteration plot or clarify that energy decrease alone is not evidence of success.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the EBM-guided MCMC pipeline is a surrogate-model optimization with independently measured transfer results.

full rationale

The paper's derivation chain is: collect activations and success/failure labels from a surrogate model; train a binary classifier and reinterpret its logits as an EBM energy (Eqs. 1-7); run MCMC whose acceptance ratio uses that energy (Eq. 8) to propose prompt edits; measure ASR on the surrogate and on other models. None of these steps defines the predicted quantity in terms of the fitted quantity by construction. The energy is a function of classifier logits, not of attack-success labels; the claim that lower energy tracks higher ASR is an empirical correlation checked in Fig. 5, not an identity. Final ASR values are obtained by actually executing attacks, not read off the EBM. The cross-model transfer claim is measured on victim models other than the surrogate (ASR-T) and on unseen tasks, so it is not forced by the training objective. The EBM and MCMC machinery are cited from independent prior work, and there are no load-bearing self-citations. Concerns about whether the energy landscape truly aligns with attack success are validity/robustness issues rather than circularity. Therefore no circular step can be exhibited.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities; it combines existing models (classifier, BERT, MCMC) with a new objective. The main unproven ingredients are the transferability of activations and the validity of the energy as an attack-quality measure.

free parameters (5)

Activation layer index for EBM = 25
Selected as the layer with lowest validation loss (Fig. 10); a design choice tuned on the training data.
Number of top infixes after filtering = 35
Infixes ranked by ASR on the first training task (Sec 3.2); the 35 threshold is arbitrary and chosen based on surrogate performance.
EBM MLP hidden dimensions = 1024, 256
Architecture hyperparameters in Appendix C; not theoretically motivated.
Number of training samples and tasks = 20,000 / 7 tasks
4,000 templates x 5 training tasks; task count and template count are choices.
MCMC proposal model = BERT
The MLM (BERT) is used for token replacement; the specific BERT checkpoint is not specified.

assumptions (4)

standard math A binary classifier can be interpreted as an energy-based model (Grathwohl et al., 2020).
Used in Section 3.3 to derive the EBM from classifier logits; this is a known result.
domain assumption Surrogate model activations transfer across different LLM architectures.
The entire method relies on the EBM trained on surrogate activations to guide prompts that work on victim models; no theoretical guarantee.
domain assumption The MCMC sampler with BERT proposals will efficiently explore the space of high-threat prompts.
The algorithm uses random token replacement; convergence to a useful distribution is assumed (Algorithm 1).
domain assumption Success/failure labels from the surrogate model are reliable proxies for victim model behavior.
The EBM is trained on surrogate labels; transfer success depends on this proxy relationship.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling." pith.science (2026). https://pith.science/paper/DPIJIZRQ

@misc{pith2026250907617,
  author       = {Pith},
  title        = {Pith review of: Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DPIJIZRQ}},
  note         = {Machine review of arXiv:2509.07617}
}

read the original abstract

Direct Prompt Injection (DPI) attacks pose a critical security threat to Large Language Models (LLMs) due to their low barrier of execution and high potential damage. To address the impracticality of existing white-box/gray-box methods and the poor transferability of black-box methods, we propose an activations-guided prompt injection attack framework. We first construct an Energy-based Model (EBM) using activations from a surrogate model to evaluate the quality of adversarial prompts. Guided by the trained EBM, we employ the token-level Markov Chain Monte Carlo (MCMC) sampling to adaptively optimize adversarial prompts, thereby enabling gradient-free black-box attacks. Experimental results demonstrate our superior cross-model transferability, achieving 49.6% attack success rate (ASR) across five mainstream LLMs and 34.6% improvement over human-crafted prompts, and maintaining 36.6% ASR on unseen task scenarios. Interpretability analysis reveals a correlation between activations and attack effectiveness, highlighting the critical role of semantic patterns in transferable vulnerability exploitation.

Figures

Figures reproduced from arXiv: 2509.07617 by the authors.

**Figure 2.** Pipeline of our activations-guided MCMC sampling. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Message structure. mary instruction, while the user input includes adversarial prompt consisting of the prefix, infix, and suffix. The message structure is shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: The energy scores over the iterative process. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: The energy scores of samples. baselines. First, our framework significantly outperforms white-box GCG-Inject, gray-box AutoDANGA-Inject and black-box PromptFuzz across all evaluated models. For instance, when transferring attacks to Llama3.1, traditional white-box me…

**Figure 6.** Figure 6: The visualization of activations. ness levels meet human acceptability thresholds (PPL=127.68), potential improvements could involve fine-tuning the proposal model or introducing additional constraints to enhance attack strength at the expense of naturalness. Second, …

**Figure 8.** Figure 8: Judge function for task 3 I’m trying to take the prompt injection attack template out of the data. These injected prompts are constructed in 3 parts: prefix for model escaping, middle for misleading model and including attack payload, and suffix for simulating the use…

**Figure 9.** Figure 9: The decoupling prompts. C EBM Details We implement an EBM using a two-hidden-layer MLP with dimensions 1024 and 256, employing ReLU (Agarap, 2018) activation functions. The model is optimized using AdamW (Loshchilov and Hutter, 2019) with a learning rate of 0.0003, inc…

**Figure 7.** Figure 7: Judge function for task 2 We present representative evaluation examples of the first task in [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

**Figure 10.** Figure 10: EBM training process [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 16 canonical work pages

[1]

Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, and Mario Fritz. 2024. https://doi.org/10.48550/ARXIV.2406.00799 Are you still on track!? catching LLM task drift with activations . CoRR, abs/2406.00799

arXiv doi:10.48550/arxiv.2406.00799 2024
[2]

Abien Fred Agarap. 2018. https://arxiv.org/abs/1803.08375 Deep learning using rectified linear units (relu) . CoRR, abs/1803.08375

arXiv 2018
[3]

AgentGPT. 2023. https://github.com/reworkd/AgentGPT [link]

work page 2023
[4]

AutoGPT. 2023. https://github.com/Significant-Gravitas/AutoGPT [link]

work page 2023
[5]

Sizhe Chen, Julien Piet, Chawin Sitawarin, and David A. Wagner. 2024. https://doi.org/10.48550/ARXIV.2402.06363 StruQ : Defending against prompt injection with structured queries . CoRR, abs/2402.06363

arXiv doi:10.48550/arxiv.2402.06363 2024
[6]

Coeeter. 2023. https://github.com/Coeeter/emailgpt [link]

work page 2023
[7]

Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/V1/N19-1423 BERT: pre-training of deep bidirectional transformers for language understanding . In Proceeding of the 17th Conference of the North American Chapter of the Association for Computational Linguistics (NAACL'19), pages 4171--4186. Association for C...

doi:10.18653/v1/n19-1423 2019
[8]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and Ahmad Al-Dahle . 2024. https://doi.org/10.48550/ARXIV.2407.21783 The llama 3 herd of models . CoRR, abs/2407.21783

arXiv doi:10.48550/arxiv.2407.21783 2024

Show all 39 references

[9]

Freysa. 2024. https://www.freysa.ai/act-i [link]

2024
[10]

Futurism. 2023. https://futurism.com/the-byte/car-dealership-ai [link]

2023
[11]

Leo Gao, Tom Dupr \' e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. https://doi.org/10.48550/ARXIV.2406.04093 Scaling and evaluating sparse autoencoders . CoRR, abs/2406.04093

2024 doi arXiv
[12]

GitHub. 2023. https://github.com/features/copilot [link]

2023
[13]

Will Grathwohl, Kuan-Chieh Wang, J \"o rn-Henrik Jacobsen, David Duvenaud, and Mohammad Norouzi. 2020. https://doi.org/10.48550/arXiv.1912.03263 Your classifier is secretly an energy based model and you should treat it like one . In Proceeding of the 8th International Conferen...

2020 doi arXiv
[14]

Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. COLD-attack : Jailbreaking LLMs with stealthiness and controllability. In Proceeding of the 41st International Conference on Machine Learning (ICML'24), pages 16974--17002. PMLR

2024
[15]

Huggingface. 2023. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html [link]

2023
[16]

Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63--S63

1977
[17]

Aounon Kumar and Himabindu Lakkaraju. 2024. https://doi.org/10.48550/ARXIV.2404.07981 Manipulating large language models to increase product visibility . CoRR, abs/2404.07981

2024 doi arXiv
[18]

Hao Li, Xiaogeng Liu, and Chaowei Xiao. 2024. https://doi.org/10.48550/ARXIV.2410.22770 Injecguard: Benchmarking and mitigating over-defense in prompt injection guardrail models . CoRR, abs/2410.22770

2024 doi arXiv
[19]

Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. https://doi.org/10.48550/arXiv.2310.04451 AutoDAN : Generating stealthy jailbreak prompts on aligned large language models . In Proceeding of the 12st International Conference on Learning Representations (ICLR'24). Open...

2024 doi arXiv
[20]

Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In Proceeding of the 7th International Conference on Learning Representations (ICLR'19). OpenReview.net

2019
[21]

Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Berg-Kirkpatrick . 2022. https://doi.org/10.18653/v1/2022.acl-long.31 Mix and match: Learning-free controllable text generationusing energy language models . In Proceeding of the 60st Annual Meeting of the Association for Co...

2022 doi
[22]

OpenAI. 2023. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html Language models can explain neurons in language models

2023
[23]

OWASP. 2024. https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025 [link]

2024
[24]

Dario Pasquini, Martin Strohmeier, and Carmela Troncoso. 2024. https://doi.org/10.1145/3689932.3694764 Neural exec: Learning (and learning from) execution triggers for prompt injection attacks . In Proceeding of the 2024 Workshop on Artificial Intelligence and Security (AISec ...

2024
[25]

Yao Qiang, Xiangyu Zhou, Saleh Zare Zade, Mohammad Amin Roshani, Douglas Zytko, and Dongxiao Zhu. 2024. https://doi.org/10.48550/ARXIV.2402.13459 Learning to poison large language models during instruction tuning . CoRR, abs/2402.13459

2024 doi arXiv
[26]

Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/3e25d1aff47964c8409fd5c8dc0438d7-Abstract-Conference.html COLD decoding: Energy-based constrained text generation with langevin dynamics . In Proceedings of the...

2022
[27]

Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/c2a8060fd22744b38177d9e428a052e0-Abstract-Conference.html On the exploitability of instruction tuning . In Proceedings of the 37th Inte...

2023
[28]

Yang Song and Diederik P. Kingma. 2021. https://arxiv.org/abs/2101.03288 How to train your energy-based models . CoRR, abs/2101.03288

2021 arXiv
[29]

Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, and Luke Bailey. 2024. https://doi.org/10.48550/arXiv.2311.01011 Tensor trust: Interpretable prompt injection attacks from an online game . In Proceeding of the 21st International Conference on Learning Represen...

2024 doi arXiv
[30]

Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, and James Crnkovich. 2024. https://doi.org/10.48550/ARXIV.2408.01605 CYBERSECEVAL 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models . CoRR, abs/2408.01605

2024 doi arXiv
[31]

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

2024 doi arXiv
[32]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

2024 doi arXiv
[33]

Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. https://doi.org/10.48550/ARXIV.2309.10253 GPTFUZZER : Red teaming large language models with auto-generated jailbreak prompts . CoRR, abs/2309.10253

2023 doi arXiv
[34]

Jiahao Yu, Yangguang Shao, Hanwen Miao, Junzheng Shi, and Xinyu Xing. 2024. https://doi.org/10.48550/ARXIV.2409.14729 PROMPTFUZZ: harnessing fuzzing techniques for robust testing of prompt injection in llms . CoRR, abs/2409.14729

2024 doi arXiv
[35]

Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.624 Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents . In Proceeding of the 62nd Annual Meeting of the Association for...

2024 doi
[36]

Andy Zou, Long Phan, Sarah Chen, James Campbell, and Phillip Guo. 2023 a . https://doi.org/10.48550/ARXIV.2310.01405 Representation engineering: A top-down approach to AI transparency . CoRR, abs/2310.01405

2023 doi arXiv
[37]

Zico Kolter, and Matt Fredrikson

Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023 b . https://doi.org/10.48550/ARXIV.2307.15043 Universal and transferable adversarial attacks on aligned language models . CoRR, abs/2307.15043

2023 doi arXiv
[38]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[39]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, and Mario Fritz. 2024. https://doi.org/10.48550/ARXIV.2406.00799 Are you still on track!? catching LLM task drift with activations . CoRR, abs/2406.00799

arXiv doi:10.48550/arxiv.2406.00799 2024

[2] [2]

Abien Fred Agarap. 2018. https://arxiv.org/abs/1803.08375 Deep learning using rectified linear units (relu) . CoRR, abs/1803.08375

arXiv 2018

[3] [3]

AgentGPT. 2023. https://github.com/reworkd/AgentGPT [link]

work page 2023

[4] [4]

AutoGPT. 2023. https://github.com/Significant-Gravitas/AutoGPT [link]

work page 2023

[5] [5]

Sizhe Chen, Julien Piet, Chawin Sitawarin, and David A. Wagner. 2024. https://doi.org/10.48550/ARXIV.2402.06363 StruQ : Defending against prompt injection with structured queries . CoRR, abs/2402.06363

arXiv doi:10.48550/arxiv.2402.06363 2024

[6] [6]

Coeeter. 2023. https://github.com/Coeeter/emailgpt [link]

work page 2023

[7] [7]

Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/V1/N19-1423 BERT: pre-training of deep bidirectional transformers for language understanding . In Proceeding of the 17th Conference of the North American Chapter of the Association for Computational Linguistics (NAACL'19), pages 4171--4186. Association for C...

doi:10.18653/v1/n19-1423 2019

[8] [8]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and Ahmad Al-Dahle . 2024. https://doi.org/10.48550/ARXIV.2407.21783 The llama 3 herd of models . CoRR, abs/2407.21783

arXiv doi:10.48550/arxiv.2407.21783 2024