Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes that the fine-tuning interface of closed-weight LLMs leaks loss-like values that can substitute for logprobs, enabling optimization-based prompt injection attacks on Gemini with up to 82% success.

desk verdict A real result: using fine-tuning loss as a graybox optimization oracle for prompt injection on closed-weight models, with solid experiments and a vendor-confirmed mitigation, though the attack's practicality hinges on a vendor implementation detail that the paper's fallback claims don't actually cover. read the letter →

arxiv 2501.09798 v2 pith:LUAGXS2B submitted 2025-01-16 cs.CR cs.CL

Andrey Labunets , Nishit V. Pandya , Ashish Hooda , Xiaohan Fu , Earlence Fernandes This is my paper · ORCID
classification cs.CRcs.CL
keywords fine-tuningpromptinjectionoptimization-basedattacklossleakageGeminidiscreteoptimizationgrayboxLLMsecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the fine-tuning interface offered by a closed-weight LLM vendor is itself a usable optimization oracle: by sending a fine-tuning request with a very small learning rate, an attacker can read out loss-like values that approximate the log-probabilities of target tokens. If this holds, then attacks that previously required model weights or logprobs from the inference endpoint can be mounted with nothing more than a developer account and the vendor's own fine-tuning API. The authors demonstrate the idea on Google's Gemini family, turning the reported training loss into a guide for a greedy token-substitution search, and report attack success rates of 65.3% on Gemini 1.5 Flash and 82.0% on Gemini 1.0 Pro on a 40-example subset of a standard prompt-injection benchmark. The significance is that the loss signal is a byproduct of a legitimate developer feature, so restricting it collides directly with the utility that fine-tuning is meant to provide.

What carries the argument

The load-bearing mechanism is the training-loss proxy: for a tiny learning rate, the fine-tuning loss on an input-output pair behaves as a linear function of the output's average log-probability plus an input-dependent constant, and the paper validates this with an $R^2$ that approaches 1 as output length grows. A second component recovers the deterministic shuffle the interface applies to the training set by submitting progressively corrupted target outputs, whose losses rise monotonically, and reusing the recovered permutation to map reported losses to candidate prompts. These two pieces feed a greedy coordinate-search algorithm that evaluates candidate token substitutions in batches through the fine-tuning endpoint.

What would settle it

Send the same dataset twice to the fine-tuning endpoint and check whether the order of reported losses is identical; if the permutation changes between requests or depends on the data content, the greedy search cannot be guided and the attack's success rate should collapse to the random-ablation level.

Watch

Extended reading notes

Core claim

The paper's central claim is that the loss value returned by a commercial fine-tuning API can approximate the log-probability of a target output, and that this signal is strong enough to drive a greedy discrete search for adversarial prompt tokens. Under this premise, the authors produce the first optimization-based prompt injections against the Gemini family, reporting 65.3% attack success on Gemini 1.5 Flash and 82.0% on Gemini 1.0 Pro on a 40-example subset of a standard indirect-injection benchmark. The attacks work by running a single fine-tuning step with a very small learning rate, so the reported training loss stays close to the base model's cross-entropy loss, and by recovering the fixed permutation the API applies to the training set so that losses can be matched back to individual candidate prompts.

Load-bearing premise

The attack depends on the vendor applying the same fixed training-set shuffle to every fine-tuning request of a given size, so that a permutation learned once stays valid for later requests.

Editorial extensions

If this is right

  • An attacker with only API access to a closed-weight model can obtain a signal previously thought to require logprobs or model weights.
  • The approach transfers across Gemini model versions, with attacks computed on one model often succeeding above 70% on others.
  • The attack is cheap: fewer than 100 fine-tuning calls per benchmark example and under $10 in inference costs in the paper's setting.
  • Hyperparameter restrictions such as a minimum learning rate or larger minimum batch size can block this specific channel, but the paper argues they also reduce benign fine-tuning utility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fixed-shuffle property holds for other fine-tuning APIs that expose loss metrics, the same loss-proxy technique could be adapted to other closed-weight vendors, subject to their hyperparameter constraints.
  • The loss-proxy signal is not limited to prompt injection; any attack objective that can be written as a target string (e.g., targeted jailbreaks or data-extraction probes) could in principle be optimized through the fine-tuning interface.
  • A vendor-side fix that randomizes the training-set shuffle per request, or adds calibrated noise to reported losses, would break the specific correspondence between candidates and losses without sacrificing hyperparameter control; this is a testable defense hypothesis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper identifies a new graybox attack channel against proprietary LLMs: the loss values reported by a remote fine-tuning API can be used as a proxy for log-probabilities, enabling optimization-based prompt injection. The authors show that with a very small learning rate, the Gemini fine-tuning API returns losses that approximate average log-probs of a target output, and that these losses arrive in a fixed, size-dependent permutation that can be recovered and reused. They combine these observations into a greedy discrete-optimization algorithm ("fun-tuning") that wraps existing prompt-injection instructions with optimized prefix/suffix tokens. On a 40-example subset of the PurpleLlama benchmark, they report attack success rates of 65.3% on Gemini 1.5 Flash and 82.0% on Gemini 1.0 Pro, with the attack costing 90 fine-tuning calls per example and under $10 total. They also report transfer across Gemini models, an ablation against random signals, and a vendor-confirmed mitigation deployed in April 2025.

Significance. If the result holds, it demonstrates a practically exploitable utility-security tradeoff in commercial fine-tuning interfaces: a feature intended for legitimate developers leaks a signal strong enough to guide discrete prompt optimization on closed-weight models. The paper is careful in several respects: it validates the loss signal with an R² curve against independent log-prob estimates, compares its approximate permutation recovery to a provably correct method, includes a random-signal ablation, and reports transfer evaluation across multiple Gemini model versions. The vendor-confirmed mitigation independently corroborates that the identified mechanism was real. The main limitation is that the demonstrated efficiency depends on a specific vendor implementation detail (a fixed, reusable data shuffle), which Google subsequently changed; the paper's general claim about the fine-tuning interface as an attack surface is broader than the concrete evidence supports. The significance is therefore a valuable, but historically and vendor-scoped, characterization of a real vulnerability.

major comments (4)
  1. [§4.1, §5.1, §7] The reported efficiency of 90 fine-tuning calls per PPL40 example (Tables 3 and 4) depends on the empirical observation, made only against the November 2024 Gemini API, that the fine-tuning endpoint applies a fixed, size-dependent permutation σ_N across different requests. The Section 7 dismissal of per-request randomization says an attacker can still extract losses atomically or via duplicated training examples, but this is not quantitatively supported. Atomic extraction for the configured attack would require about 45 iterations × 2 phases × 1000 candidates = 90,000 fine-tuning calls per example, and the duplication-based method for K=1000 candidates requires roughly 500,500 training examples, both far outside the demonstrated budget. This is load-bearing: without a reusable fixed shuffle, Algorithm 1 cannot rank candidates, and the core feasibility claim is not robust to vendor API changes, which Google's April 2025 mitigation already shows can occur. The authors should either provide a quantitative feasibility analysis of the fallback strategies or explicitly scope the central practicality claim to APIs that expose a reusable fixed shuffle.
  2. [§6.1, §6.2] The headline ASR numbers are computed on PPL40, a 40-example subset that deliberately excludes the token-smuggling category and other non-standard encodings because "Gemini doesn't follow instructions encoded in non-standard encodings," and with manually rewritten "strict" judge questions. This makes the reported 65.3% and 82.0% numbers not directly comparable to results on the full PurpleLlama benchmark, and the excluded categories are precisely the ones where the attack is expected to fail. Please report results on the full indirect set (or provide a convincing argument for why the exclusions do not bias the main claim) and publish the rewritten judge questions alongside the code so that the evaluation is reproducible and comparable to prior work.
  3. [§4.2, §4.3] The reverse-engineering hypothesis TrainingLoss(Y|X) = K(X) + l·AvgLogprobs(Y|X) includes an input-dependent offset K(X). The R² validation in Figure 3 varies the output length on a fixed set of inputs, so it does not directly establish that the training loss can rank candidates that differ in the adversarial prefix/suffix (i.e., differ in X). The needed check is Section 4.3, but it is limited to three questions with N=10 candidates each, whereas the actual attack compares 1000 candidates per iteration over 40 mask positions. Given that the central claim is that the fine-tuning loss is a usable optimization signal, please add a larger-scale evaluation of the rank correlation between training loss and true log-probs on candidates generated by the actual attack procedure, or explicitly narrow the claim about why the K(X) offset does not break candidate ranking.
  4. [§6.4, Tables 3–4] The random-signal ablation still achieves 43.8% ASR on Gemini 1.5 Flash and 61.3% on Gemini 1.0 Pro, which is a substantial fraction of the fun-tuning result (65.3% and 82.0%, respectively). The paper's claim that these differences are outside standard deviation is based on aggregate mean ± std over 20 scoring runs; this does not establish that the improvement is systematic across the 40 individual examples, especially if the effect is driven by a small subset of examples. Please add per-example paired comparisons or a statistical test (e.g., a paired test over the 40 examples) to support the claim that the true loss signal, rather than the random token substitution strategy, is responsible for the improvement.
minor comments (5)
  1. [Appendix B] The manuscript includes a "Meta-Review" from the IEEE S&P program committee. A journal submission should not contain internal review documents; please remove this appendix and fold any relevant limitations into a standard limitations/discussion section.
  2. [Table 9] The header says "Purle Llama" (typo); please change to "Purple Llama."
  3. [§6.3, §6.5] The text in Section 6.5 says ASR improves after "0th, 15th, and 45th iterations," while Section 6.3 states that restarts occur at the 15th and 30th iterations. Please check the restart schedule and make the descriptions consistent.
  4. [References] References [24] and [31] appear to be the same paper ("PAL: Proxy-Guided Black-Box Attack on Large Language Models"); please deduplicate and renumber.
  5. [Throughout] The term "fun-tuning" is sometimes capitalized and sometimes not; please choose a single convention and apply it consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the attack's guidance signal is validated against external logprobs, and its success is measured by an independent judge on an external benchmark with a random-signal ablation ruling out chance.

full rationale

The paper's derivation chain is self-contained and externally grounded. The core claim is that a small-learning-rate fine-tuning loss approximates a usable optimization signal; this is established empirically in Sections 4.1-4.3 by comparing training loss against Vertex AI average logprobs and by a rank-distribution experiment showing the loss-selected candidate is better than random, not by assuming the target attack succeeds. The adversarial search in Algorithm 2 minimizes fine-tuning loss toward a fixed target string, while attack success is measured independently with GPT-4o judge questions on the external PurpleLlama/PPL40 benchmark, and the ablation that replaces losses with random numbers keeps all other components fixed, demonstrating that the loss signal itself is responsible for the ASR gain. The permutation-recovery step of Section 5.1 is validated against a provably-correct three-request method in Appendix A, and the approximate sigma_N is an implementation detail of Gemini's API, not an equation that re-imports the paper's conclusions. Section 4.2's hypothesized K(X) offset is estimated from the same comparison, but the authors explicitly state that the attack does not need its exact form, and the proxy's utility is established independently in Section 4.3; this is a weak validation design, not a circular derivation. The only self-citation in the paper appears in related-work context and is not load-bearing. The fragility of the fixed-shuffle assumption is a robustness and scope caveat, not a circularity, so it does not affect the score.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on empirical assumptions about a closed API (loss semantics, fixed shuffle, small-LR invariance) and on hand-set attack hyperparameters. No new physical or mathematical entities are introduced. The fitted K(X) offset appears only in the reverse-engineering hypothesis and is not used to set attack constants.

free parameters (5)
  • K(X) offset in training-loss model = Not reported; varies per input.
    In Section 4.2 the authors hypothesize TrainingLoss(Y|X) = K(X) + l * AvgLogprobs(Y|X) and fit K(X) as the input-dependent intercept. It is not needed for the attack because it is constant when only suffix and prefix tokens change, but it is a fitted quantity in the reverse-engineering claim.
  • candidate set size = 1000 candidates per iteration.
    Chosen from local simulation on gemma-2-9b-it; the paper shows ASR plateaus after 1000 candidates in Section 6.6. It is a hand-set attack parameter, not fitted to the Gemini ASR results.
  • number of optimizer iterations and restarts = 45 iterations with restarts at 15 and 30.
    Selected by the authors in Section 6.3. ASR is reported as the maximum over iterations, so the iteration count affects the headline number.
  • prefix and suffix length = 20 tokens each.
    Manually chosen in Section 6.3 to fit within the attack size limits; not derived from data.
  • learning rate = Approximately 1e-45.
    Set at the bottom of the range where the reported loss is independent of learning rate in Section 4.1. The attack requires this small-learning-rate capability, even though the exact value is not fitted.
assumptions (5)
  • domain assumption The scalar returned by the Gemini fine-tuning API is a next-token cross-entropy-style training loss over the supplied input-output pair.
    The paper partially reverse engineers this in Section 4.2, but it never inspects the vendor's code. The attack treats every returned scalar as a comparable loss for candidate prompts.
  • domain assumption Learning rates in the range 1e-13 to 1e-45 leave the model parameters effectively unchanged, so the returned loss reflects the base model and not the single fine-tuning step.
    Established empirically in Section 4.1. If a vendor applies a hidden learning-rate multiplier, this assumption can fail silently.
  • domain assumption For a fixed training set size N, the API applies the same fixed permutation in every fine-tuning request.
    Section 4.1 concludes this from duplicate-counting experiments. It is load-bearing for the de-permutation in Sections 5.1 and 5.2, and Google's subsequent mitigation changed the parameters the attack relied on.
  • domain assumption Progressively garbling the target output increases the training loss monotonically for the constructed D_garbled dataset.
    Section 5.1 uses this monotonicity to recover the permutation by sorting. Appendix A.1 compares against a provably correct method and reports 7 to 8 percent Hamming error, so the assumption is approximate.
  • domain assumption The manually rewritten judge questions correctly distinguish successful from failed injections.
    Section 6.2 replaces the original PurpleLlama judge questions to reduce false positives. The reported ASR numbers depend on this subjective rewrite, which is not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface." pith.science (2026). https://pith.science/paper/LUAGXS2B

@misc{pith2026250109798,
  author       = {Pith},
  title        = {Pith review of: Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LUAGXS2B}},
  note         = {Machine review of arXiv:2501.09798}
}
read the original abstract

We surface a new threat to closed-weight Large Language Models (LLMs) that enables an attacker to compute optimization-based prompt injections. Specifically, we characterize how an attacker can leverage the loss-like information returned from the remote fine-tuning interface to guide the search for adversarial prompts. The fine-tuning interface is hosted by an LLM vendor and allows developers to fine-tune LLMs for their tasks, thus providing utility, but also exposes enough information for an attacker to compute adversarial prompts. Through an experimental analysis, we characterize the loss-like values returned by the Gemini fine-tuning API and demonstrate that they provide a useful signal for discrete optimization of adversarial prompts using a greedy search algorithm. Using the PurpleLlama prompt injection benchmark, we demonstrate attack success rates between 65% and 82% on Google's Gemini family of LLMs. These attacks exploit the classic utility-security tradeoff - the fine-tuning interface provides a useful feature for developers but also exposes the LLMs to powerful attacks.

Figures

Figures reproduced from arXiv: 2501.09798 by the authors.

Figure 1
Figure 1. Example prompt injection with our method on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. The correlation between average logprobs and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Rank distribution of top candidate from training [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Fun-tuning attack against Gemini 1.0 Pro gains [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Fun-tuning attack against Gemini 1.5 Flash results [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Locally simulated attack against gemma-2-9b-it [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: ASR of our attack methods against Gemini 1.0 Pro [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: ASR of our attack methods against Gemini 1.5 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Example prompt injection with our method [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Security Concerns for Large Language Models: A Survey

    cs.CR 2025-05 conditional novelty 5.0 of 10

    A survey that classifies LLM security threats and argues that intrinsic agentic risks, such as scheming, are underappreciated and poorly defended.

Reference graph

Works this paper leans on

65 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    Tricking llms into disobedience: Understanding, analyzing, and preventing jailbreaks,

    A. Rao, S. Vashistha, A. Naik, S. Aditya, and M. Choudhury, “Tricking llms into disobedience: Understanding, analyzing, and preventing jailbreaks,” arXiv preprint arXiv:2305.14965 , 2023

    arXiv 2023

  2. [2]

    Jailbreaking chatgpt via prompt engineering: An empirical study,

    Y . Liu, G. Deng, Z. Xu, Y . Li, Y . Zheng, Y . Zhang, L. Zhao, T. Zhang, K. Wang, and Y . Liu, “Jailbreaking chatgpt via prompt engineering: An empirical study,” arXiv preprint arXiv:2305.13860 , 2023

    arXiv 2023

  3. [3]

    Baseline defenses for adversarial attacks against aligned language models,

    N. Jain, A. Schwarzschild, Y . Wen, G. Somepalli, J. Kirchenbauer, P. yeh Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, “Baseline defenses for adversarial attacks against aligned language models,” 2023

    2023

  4. [4]

    Jailbroken: How does llm safety training fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” 2023

    work page 2023

  5. [5]

    Gptfuzzer: Red teaming large lan- guage models with auto-generated jailbreak prompts,

    J. Yu, X. Lin, and X. Xing, “Gptfuzzer: Red teaming large lan- guage models with auto-generated jailbreak prompts,” arXiv preprint arXiv:2309.10253, 2023

    arXiv 2023

  6. [6]

    Jailbreaking attack against multimodal large language model,

    Z. Niu, H. Ren, X. Gao, G. Hua, and R. Jin, “Jailbreaking attack against multimodal large language model,” arXiv preprint arXiv:2402.02309 , 2024

    arXiv 2024

  7. [7]

    Artprompt: Ascii art-based jailbreak attacks against aligned llms,

    F. Jiang, Z. Xu, L. Niu, Z. Xiang, B. Ramasubramanian, B. Li, and R. Poovendran, “Artprompt: Ascii art-based jailbreak attacks against aligned llms,” arXiv preprint arXiv:2402.11753 , 2024

    arXiv 2024

  8. [8]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” 2023

    work page 2023

Show all 65 references
  1. [9]

    Prompt injection attack against llm-integrated applications,

    Y . Liu, G. Deng, Y . Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y . Liu, H. Wang, Y . Zheng et al., “Prompt injection attack against llm-integrated applications,” arXiv preprint arXiv:2306.05499 , 2023

    2023 arXiv

  2. [10]

    New prompt injection attack on chatgpt web version. markdown images can steal your chat data

    R. Samoilenko, “New prompt injection attack on chatgpt web version. markdown images can steal your chat data.” 2023

    2023

  3. [11]

    Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents,

    Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents,” arXiv preprint arXiv:2403.02691 , 2024

    2024 arXiv

  4. [12]

    Benchmarking and defending against indirect prompt injection attacks on large language models,

    J. Yi, Y . Xie, B. Zhu, K. Hines, E. Kiciman, G. Sun, X. Xie, and F. Wu, “Benchmarking and defending against indirect prompt injection attacks on large language models,” arXiv preprint arXiv:2312.14197 , 2023

    2023 arXiv

  5. [13]

    Agent hijacking: The true impact of prompt injection at- tacks,

    SnykSec, “Agent hijacking: The true impact of prompt injection at- tacks,” https://dev.to/snyk/agent-hijacking-the-true-impact-of-prompt- injection-attacks-983, 2024, [Accessed 23-09-2024]

    2024

  6. [14]

    Fine-tuning with the Gemini API — Google AI for Devel- opers — ai.google.dev,

    Google, “Fine-tuning with the Gemini API — Google AI for Devel- opers — ai.google.dev,” https://ai.google.dev/gemini-api/docs/model- tuning, 2024, [Accessed 23-09-2024]

    2024

  7. [15]

    Fine-tuning now available for gpt-4o,

    OpenAI, “Fine-tuning now available for gpt-4o,” https://openai.com/ index/gpt-4o-fine-tuning/, 2024, [Accessed 22-09-2024]

    2024

  8. [16]

    Fine-tune anthropic’s claude 3 haiku in amazon bedrock to boost model accuracy and quality,

    Amazon Web Services (AWS), “Fine-tune anthropic’s claude 3 haiku in amazon bedrock to boost model accuracy and quality,” https://aws.amazon.com/blogs/machine-learning/fine-tune- anthropics-claude-3-haiku-in-amazon-bedrock-to-boost-model- accuracy-and-quality/, 2023, accessed: ...

    2023

  9. [17]

    Jailbreaking leading safety-aligned llms with simple adaptive attacks,

    M. Andriushchenko, F. Croce, and N. Flammarion, “Jailbreaking leading safety-aligned llms with simple adaptive attacks,” 2024. [Online]. Available: https://arxiv.org/abs/2404.02151

    2024 arXiv

  10. [18]

    ChatGPT-Dan-Jailbreak,

    A. ONeal, “ChatGPT-Dan-Jailbreak,” https://gist.github.com/coolaj86/ 6f4f7b30129b0251f61fa7baaa881516, 2023, [Accessed 23-09-2024]

    2023

  11. [19]

    Jailbreaking black box large language models in twenty queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” 2024. [Online]. Available: https://arxiv.org/abs/2310.08419

    2024 arXiv

  12. [20]

    Tree of attacks: Jailbreaking black-box llms automatically,

    A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y . Singer, and A. Karbasi, “Tree of attacks: Jailbreaking black-box llms automatically,” 2023

    2023

  13. [21]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” 2023

    2023

  14. [23]

    Query-based adversarial prompt generation,

    J. Hayase, E. Borevkovic, N. Carlini, F. Tram `er, and M. Nasr, “Query-based adversarial prompt generation,” arXiv preprint arXiv:2402.12329, 2024

    2024 arXiv

  15. [25]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models,

    X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2310.04451

    2024 arXiv

  16. [26]

    Chat create top logprobs — openai api refer- ence,

    OpenAI, “Chat create top logprobs — openai api refer- ence,” https://platform.openai.com/docs/api-reference/chat/create#chat- create-top logprobs, 2024, [Accessed 22-09-2024]

    2024

  17. [27]

    Generating content — Gemini API,

    Google, “Generating content — Gemini API,” https://ai.google.dev/ api/generate-content#generatecontentresponse, 2024, [Accessed 23-09- 2024]

    2024

  18. [28]

    Stealing part of a production language model,

    N. Carlini, D. Paleka, K. D. Dvijotham, T. Steinke, J. Hayase, A. F. Cooper, K. Lee, M. Jagielski, M. Nasr, A. Conmy, I. Yona, E. Wallace, D. Rolnick, and F. Tram `er, “Stealing part of a production language model,” 2024. [Online]. Available: https://arxiv.org/abs/2403.06634

    2024 arXiv

  19. [29]

    Covert malicious finetuning: Challenges in safeguarding llm adaptation,

    D. Halawi, A. Wei, E. Wallace, T. T. Wang, N. Haghtalab, and J. Steinhardt, “Covert malicious finetuning: Challenges in safeguarding llm adaptation,” arXiv preprint arXiv:2406.20053 , 2024

    2024 arXiv

  20. [30]

    Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models,

    M. Bhatt, S. Chennabasappa, Y . Li, C. Nikolaidis, D. Song, S. Wan, F. Ahmad, C. Aschermann, Y . Chen, D. Kapil et al., “Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models,” arXiv preprint arXiv:2404.13161 , 2024

    2024 arXiv

  21. [31]

    Pal: Proxy- guided black-box attack on large language models,

    C. Sitawarin, N. Mu, D. Wagner, and A. Araujo, “Pal: Proxy- guided black-box attack on large language models,” arXiv preprint arXiv:2402.09674, 2024

    2024 arXiv

  22. [32]

    Struq: Defending against prompt injection with structured queries,

    S. Chen, J. Piet, C. Sitawarin, and D. Wagner, “Struq: Defending against prompt injection with structured queries,” 2024. [Online]. Available: https://arxiv.org/abs/2402.06363

    2024 arXiv

  23. [33]

    Finetuned language models are zero-shot learners,

    J. Wei, M. P. Bosma, V . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” 2022. [Online]. Available: https: //openreview.net/forum?id=gEZrGCozdqR

    2022

  24. [34]

    When scaling meets llm finetuning: The effect of data, model and finetuning method,

    B. Zhang, Z. Liu, C. Cherry, and O. Firat, “When scaling meets llm finetuning: The effect of data, model and finetuning method,” 2024. [Online]. Available: https://arxiv.org/abs/2402.17193

    2024 arXiv

  25. [35]

    The instruction hierarchy: Training llms to prioritize privileged instructions,

    E. Wallace, K. Xiao, R. Leike, L. Weng, J. Heidecke, and A. Beutel, “The instruction hierarchy: Training llms to prioritize privileged instructions,” 2024. [Online]. Available: https://arxiv.org/abs/2404. 13208

    2024

  26. [36]

    Llm research insights: Instruction masking and new lora finetuning experiments,

    P. Sebastian Raschka, “Llm research insights: Instruction masking and new lora finetuning experiments,” https://www.linkedin.com/pulse/llm- research-insights-instruction-masking-new-lora-raschka-phd-7p1oc/, Jun. 2024, accessed: 2024-11-14

    2024

  27. [37]

    Instruction tuning with loss over instructions,

    Z. Shi, A. X. Yang, B. Wu, L. Aitchison, E. Yilmaz, and A. Lipani, “Instruction tuning with loss over instructions,” 2024. [Online]. Available: https://arxiv.org/abs/2405.14394

    2024 arXiv

  28. [38]

    Sparse fine-tuning for inference acceleration of large language models,

    E. Kurtic, D. Kuznedelev, E. Frantar, M. Goin, and D. Alistarh, “Sparse fine-tuning for inference acceleration of large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06927

    2023 arXiv

  29. [39]

    Neural exec: Learning (and learning from) execution triggers for prompt injection attacks,

    D. Pasquini, M. Strohmeier, and C. Troncoso, “Neural exec: Learning (and learning from) execution triggers for prompt injection attacks,”

  30. [40]

    Gemma 2: Improving open language models at a practical size,

    Gemma Team, “Gemma 2: Improving open language models at a practical size,” 2024. [Online]. Available: https://arxiv.org/abs/2408. 00118

    2024

  31. [41]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    Gemini Team, “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” 2024. [Online]. Available: https://arxiv.org/abs/2403.05530

    2024 arXiv

  32. [42]

    How we estimate the risk from prompt injection attacks on ai sys- tems,

    Agentic AI Security Team at Google DeepMind, “How we estimate the risk from prompt injection attacks on ai sys- tems,” https://security.googleblog.com/2025/01/how-we-estimate-risk- from-prompt.html, Jan. 2025, [Accessed 29-01-2025]

    2025

  33. [43]

    Fine-tune claude 3 haiku,

    Anthropic, “Fine-tune claude 3 haiku,” https://www.anthropic.com/ news/fine-tune-claude-3-haiku, 2024, [Accessed 31-03-2025]

    2024

  34. [44]

    Model tuning with gemini api,

    G. C. AI, “Model tuning with gemini api,” https://ai.google.dev/gemini- api/docs/model-tuning, 2023, accessed: 2024-11-14

    2023

  35. [45]

    Fine-tuning llms: Lora or full parameter? an in-depth analysis with llama 2,

    Anyscale, “Fine-tuning llms: Lora or full parameter? an in-depth analysis with llama 2,” https://www.anyscale.com/blog/fine-tuning- llms-lora-or-full-parameter-an-in-depth-analysis-with-llama-2, 2023, accessed: 2024-11-14

    2023

  36. [46]

    Misusing tools in large language mod- els with visual adversarial examples,

    X. Fu, Z. Wang, S. Li, R. K. Gupta, N. Mireshghallah, T. Berg- Kirkpatrick, and E. Fernandes, “Misusing tools in large language mod- els with visual adversarial examples,” arXiv preprint arXiv:2310.03185, 2023

    2023 arXiv

  37. [47]

    Ai injections: Direct and indirect prompt injections and their implications,

    J. Rehberger, “Ai injections: Direct and indirect prompt injections and their implications,” https://embracethered.com/blog/posts/2023/ai- injections-direct-and-indirect-prompt-injection-basics/, 2023

    2023

  38. [48]

    Prompt injection: What’s the worst that can happen?

    S. Willison, “Prompt injection: What’s the worst that can happen?” https://simonwillison.net/2023/Apr/14/worst-that-can-happen/, 2023

    2023

  39. [49]

    Ignore previous prompt: Attack techniques for language models,

    F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” 2022

    2022

  40. [50]

    Multi-step jailbreaking privacy attacks on chatgpt,

    H. Li, D. Guo, W. Fan, M. Xu, J. Huang, F. Meng, and Y . Song, “Multi-step jailbreaking privacy attacks on chatgpt,” 2023. [Online]. Available: https://arxiv.org/abs/2304.05197

    2023 arXiv

  41. [51]

    Many-shot jailbreaking — anthropic.com,

    C. A. et al., “Many-shot jailbreaking — anthropic.com,” https://www. anthropic.com/research/many-shot-jailbreaking, 2024, [Accessed 27- 09-2024]

    2024

  42. [52]

    OpenAI’s latest model will block the ‘ignore all previous instructions’ loophole,

    K. Robison, “OpenAI’s latest model will block the ‘ignore all previous instructions’ loophole,” https://www.theverge.com/2024/ 7/19/24201414/openai-chatgpt-gpt-4o-prompt-injection-instruction- hierarchy, 2024, [Accessed 27-09-2024]

    2024

  43. [53]

    Fast adversarial attacks on language models in one gpu minute,

    V . S. Sadasivan, S. Saha, G. Sriramanan, P. Kattakinda, A. Chegini, and S. Feizi, “Fast adversarial attacks on language models in one gpu minute,” 2024. [Online]. Available: https://arxiv.org/abs/2402.15570

    2024 arXiv

  44. [54]

    Poisoning language models during instruction tuning,

    A. Wan, E. Wallace, S. Shen, and D. Klein, “Poisoning language models during instruction tuning,” in International Conference on Machine Learning. PMLR, 2023, pp. 35 413–35 425

    2023

  45. [55]

    Learning and forgetting unsafe examples in large language models,

    J. Zhao, Z. Deng, D. Madras, J. Zou, and M. Ren, “Learning and forgetting unsafe examples in large language models,” arXiv preprint arXiv:2312.12736, 2023

    2023 arXiv

  46. [56]

    Removing rlhf protections in gpt-4 via fine-tuning,

    Q. Zhan, R. Fang, R. Bindu, A. Gupta, T. Hashimoto, and D. Kang, “Removing rlhf protections in gpt-4 via fine-tuning,” arXiv preprint arXiv:2311.05553, 2023

    2023 arXiv

  47. [57]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!

    X. Qi, Y . Zeng, T. Xie, P.-Y . Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” arXiv preprint arXiv:2310.03693 , 2023

    2023 arXiv

  48. [58]

    Stealing machine learning models via prediction {APIs},

    F. Tram`er, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing machine learning models via prediction {APIs},” in 25th USENIX security symposium (USENIX Security 16) , 2016, pp. 601–618

    2016

  49. [59]

    Leaky dnn: Stealing deep-learning model secret with gpu context-switching side- channel,

    J. Wei, Y . Zhang, Z. Zhou, Z. Li, and M. A. Al Faruque, “Leaky dnn: Stealing deep-learning model secret with gpu context-switching side- channel,” in 2020 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) . IEEE, 2020, pp. 125–137

    2020

  50. [60]

    On the sizes of openai api models,

    L. Gao, “On the sizes of openai api models,” https://blog.eleuther.ai/ gpt3-model-sizes/, 2021, accessed: [Date Accessed]

    2021

  51. [61]

    Anthropic tokenizer,

    J. Rando, “Anthropic tokenizer,” https://github.com/javirandor/ anthropic-tokenizer. Appendix A. A.1. Provably correct method to recover permuta- tions Given an input sequence of size N: X = (x1, x2, ..., xN ). The fine-tuning API permutes this sequence before compute the trai...

    2020

  52. [63]

    The paper identifies a vulnerability that significantly improves the effectiveness of prompt injection at- tacks against real-world services

  53. [64]

    The paper describes and overcomes several technical challenges in order to use the fine-tuning function- ality for this purpose

  54. [65]

    The paper provides a thorough evaluation including an ablation study to demonstrate that fine-tuning training loss can be a valuable signal for prompt injection optimization. B.4. Noteworthy Concerns

  55. [66]

    Appendix C

    The attack has been demonstrated on a single service, so it is not yet known which other services might be vulnerable to this type of technique. Appendix C. Response to the Meta-Review

  56. [67]

    The authors welcome the raised noteworthy concern and encourage more studies to understand or to exclude the attack feasibility for other services

  57. [2024]

    Available: https://arxiv.org/abs/2403.03792

    [Online]. Available: https://arxiv.org/abs/2403.03792

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.