REVIEW 4 major objections 6 minor 17 references
Prompt Optimization and Evaluation for LLM Automated Red Teaming
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Repeated attacks reveal hidden success patterns that sharpen LLM red-team prompt optimization.
desk verdict The ASR-delta idea is plausible and the control is commendable, but the headline comparison is undercut by selection bias and missing statistics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the ASR distribution: for each unique attack, run it multiple times against a randomly seeded target, record the binary judge outcomes, and use their sample mean as that attack's success score. Around this, the method builds ASR-delta pair mining, which embeds attacks, finds cosine-similar nearest-neighbor pairs, and keeps pairs whose per-attack ASR difference exceeds a threshold; these contrastive pairs are given to an optimization-by-prompting loop that proposes additions to the attack generator's system prompt. The prompt addition that yields the highest ASR distribution mean is kept, and the same pipeline with single-try binary success in place of per-attack ASR serves as the control comparison.
What would settle it
Re-run the ASR-delta and single-try OPRO pipelines on a held-out target model or judge that played no role in optimization; if the ASR-delta generator no longer has a higher ASR distribution mean, the central claim is falsified. A cheaper check is to vary only the random seed and see whether the reported ordering reproduces.
Extended reading notes
Core claim
The central claim is that attack quality is at least partly characterized by discoverability: how often a repeated attack succeeds, not whether it succeeds once. The paper extends attack success rate from a generator-level mean to a per-attack numeric feature, then treats the collection of those features as a distribution over Bernoulli parameters. It argues that this distribution's mean converges to the single-try ASR, so the extra value lies in the shape: two generators with the same mean can differ in how much of their attack mass sits at high success rates. The paper's experimental result is that optimizing a generator with ASR-delta pair mining, which selects semantic nearest neighbors whose per-attack success rates differ by at least a threshold, produces an ASR distribution shifted to higher success rates, and this improvement is larger than the improvement from optimizing with single-try binary success labels.
Load-bearing premise
The measured improvement reflects genuine attack-generator quality rather than overfitting to the particular target model, judge, and random seeds used during optimization and evaluation.
Editorial extensions
If this is right
- If per-attack ASR carries real signal, single-try ASR can misrank generators whose attacks differ in reliability, so repeated-trial evaluation is needed for fair comparison.
- Attack generators can be improved without rewriting individual attacks, just by adding instruction text to the generator's system prompt.
- The method can be applied iteratively, because the optimized generator's own attacks can be scored and mined for new contrastive pairs.
- Defenders and evaluators can watch for clusters of near-duplicate, reliably successful attacks instead of relying only on average success.
- Pairs of near-identical attacks that differ sharply in per-attack ASR offer a route to identifying the minimal linguistic differences that drive attack success.
Reading between the lines
- The authors leave implicit that the ASR-delta advantage over single-try optimization is demonstrated on one target, judge, and seed configuration; testing the same mined pairs against a second target would show whether the contrasts transfer.
- The same distributional analysis could be turned into a defense tool: finding attacks that succeed reliably across seeds would flag vulnerable clusters for patching before deployment.
- A cost-sensitive variant could adapt the number of repetitions per attack, spending fewer trials on attacks whose early outcomes are extreme and more on borderline ones.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes applying Attack Success Rate (ASR) to individual attacks rather than to a generator's whole output set: each unique attack is repeated m times against a randomly seeded target, and the per-attack empirical success probability is treated as a feature. The collection of these features is called the ASR distribution. The authors then use this distribution for attack-generator prompt optimization via OPRO, selecting semantically similar attack pairs whose ASR differs by at least a threshold (ASR-delta pair mining). They compare this against single-try OPRO (ST-OPRO), which uses binary single-try success instead of per-attack ASR, and report that ASR-delta pair mining produces a larger rightward shift in the ASR distribution and thus a better optimized attack generator.
Significance. The conceptual idea of using per-attack success probabilities estimated over repeated trials is a reasonable and potentially useful extension of the standard ASR metric, particularly for settings where repeated attempts matter. The comparison against ST-OPRO is a sensible control, and the paper includes a Limitations section that names several threats to validity. However, the central claim that ASR-delta pair mining outperforms ST-OPRO is currently supported only by qualitative density plots and a selection procedure whose bias is not addressed. If the quantitative evidence and generalization checks were added, the contribution could be of interest to the automated red-teaming community; at present the evidence is preliminary.
major comments (4)
- [§4.2, Figures 4 and 5] The central claim that ASR-delta pair mining outperforms ST-OPRO is not supported by the reported evidence. The manuscript provides only density plots; it reports no numerical ASR means, no standard errors or confidence intervals, no number of independent runs, and no statistical test for the comparison. The statement that ST-OPRO improvements are 'demonstrably smaller' is therefore not demonstrated. Please report exact values and uncertainty for at least the unoptimized, ST-OPRO, and ASR-delta conditions, ideally over repeated seeds.
- [§4.1, §4.2] The evaluation used for selection is also the evaluation used for the headline result. The authors generate 10 prompt additions and keep the one with the highest ASR distribution mean, where each mean is estimated from n=384 attacks and m=50 repetitions. Maximizing over 10 noisy estimates produces an upward bias in the reported improvement; if the candidate means for ASR-delta have different variance than those for ST-OPRO, the relative comparison can be biased as well. The text does not state whether the distributions in Figures 4 and 5 come from a fresh, held-out set of attacks or from the same attacks used in selection. Please use a held-out attack set or an unbiased selection protocol and report the variance of the candidate means.
- [§3.2, Limitations] The experiment uses GPT-4o as the generator, target, and judge, and the Limitations section explicitly concedes that results may not generalize to other models. Without at least one additional target model or judge, or an analysis of judge stability, the observed improvement could reflect overfitting to the specific GPT-4o configuration used during optimization. Please add a generalization check or clearly scope the claim as a proof-of-concept for this single configuration.
- [§4.2, Algorithm 1] The ST-OPRO baseline is described only verbally, and no information is given about how many contrastive pairs were supplied to the optimizer in each condition, what threshold Δ was used, or how the single-try criterion was applied when multiple nearest neighbors tie. If the two methods receive different amounts or qualities of in-context data, the comparison conflates the information source with the data quantity. Please report these details and ideally match the number of pairs across conditions.
minor comments (6)
- [§3.2] The sentence 'the mean of the ASR distribution converges to the single-try ASR' should be stated as an equality in expectation; as written it suggests an asymptotic claim that is not what is shown.
- [Appendix A.2] The sentence 'A full evaluation would require n × n = 147,456 total attack evaluations' is unexplained and inconsistent with the n × m = 19,200 cost of the described pipeline; please correct or justify the n × n figure.
- [§3.2] The Beta-mixture model is proposed but never fitted or used in the experiments; either include an analysis of it or remove the proposal.
- [Algorithm 1] The nearest-neighbor condition and tie handling are underspecified, and the value of Δ used in the experiments is not reported, even though the Limitations section says Δ may significantly affect results.
- [Figures 4 and 5] Add axis labels and numerical annotations so the distribution shift can be read quantitatively; the current plots cannot be interpreted without the underlying numbers.
- [References] There are minor reference issues: 'Vashney' should be 'Varshney', and Chouldechova et al. is cited without a year in Section 3.2.
Circularity Check
No significant circularity: the central claim rests on a controlled OPRO comparison, and no derivation reduces to its inputs.
full rationale
The paper's central derivation is not circular. The ASR distribution is introduced by definition in Section 3.1, where individual-attack ASR is computed as the sample mean over m repeated judged trials; this is a measurement choice, not an assumption of the conclusion. The statement that the mean of the ASR distribution converges to single-try ASR is a standard expectation identity and is not used to justify the pair-mining advantage. The main claimed result, that ASR-delta pair mining outperforms ST-OPRO, is supported by a controlled comparison in Section 4.2 in which both methods use OPRO and are evaluated and selected by the same ASR-distribution-mean criterion; the outcome is therefore not forced by construction. No load-bearing self-citations or imported uniqueness theorems appear in the derivation chain; Chouldechova et al. and Yang et al. are external prior work. The fact that the optimization objective and reported evaluation metric coincide is standard in prompt optimization and does not by itself constitute circularity. The absence of repeated seeds, confidence intervals, and generalization tests is a real statistical-evidence limitation, and the paper's own Limitations section acknowledges that generalization is unstudied, but that is a validity concern rather than a circular one. No specific step can be exhibited where an equation or fitted parameter reduces to the claimed result, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- m (repetitions per attack) =
50
- n (number of unique attacks) =
384
- Delta (ASR difference threshold) =
not specified
- Number of prompt candidates =
10
assumptions (5)
- domain assumption GPT-4o with temperature=1 is representative of stochastic LLM applications.
- domain assumption Semantic similarity in embedding space identifies near-minimal linguistic contrasts relevant to attack success.
- domain assumption The judge's binary success labels are accurate and unbiased.
- domain assumption Repeated runs against a randomly seeded target produce independent, identically distributed Bernoulli draws for each attack.
- domain assumption Using the same model for generator, target, and judge does not bias the evaluation.
Cite this review
Pith. "Pith review of Prompt Optimization and Evaluation for LLM Automated Red Teaming." pith.science (2026). https://pith.science/paper/D3JPIAJZ
@misc{pith2026250722133,
author = {Pith},
title = {Pith review of: Prompt Optimization and Evaluation for LLM Automated Red Teaming},
year = {2026},
howpublished = {\url{https://pith.science/paper/D3JPIAJZ}},
note = {Machine review of arXiv:2507.22133}
}
read the original abstract
Applications that use Large Language Models (LLMs) are becoming widespread, making the identification of system vulnerabilities increasingly important. Automated Red Teaming accelerates this effort by using an LLM to generate and execute attacks against target systems. Attack generators are evaluated using the Attack Success Rate (ASR) the sample mean calculated over the judgment of success for each attack. In this paper, we introduce a method for optimizing attack generator prompts that applies ASR to individual attacks. By repeating each attack multiple times against a randomly seeded target, we measure an attack's discoverability the expectation of the individual attack success. This approach reveals exploitable patterns that inform prompt optimization, ultimately enabling more robust evaluation and refinement of generators.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKee, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073
arXiv 2022
-
[4]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scot...
work page 2020
-
[5]
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. 2021. A survey on adversarial attacks and defences. CAAI Transactions on Intelligence Technology, 6(1):25--45
work page 2021
-
[6]
Alexandra Chouldechova, Daniel Zhang, and Jennifer Wortman Vaughan. 2023. Ai red teaming through the lens of measurement theory. In OpenReview
work page 2023
-
[7]
Zhihan Dong, Yifan Liu, Zihao Zhang, Zhenyu Wang, Xujiang Yao, Chenghua Zhou, Dongxiao Wang, and Hui Xiong. 2024. Attacks, defenses and evaluations for llm conversation safety: A survey. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics
work page 2024
-
[8]
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Long Li, Danny Hernandez, Dylan Drain, Saurav Kadavath, Jared Kaplan, Laura Weidinger, Anna Jones, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858
arXiv 2022
Show all 17 references
-
[9]
Kaver Edwin Hui. 2023. https://medium.com/@hke22/language-models-red-teaming-attack-and-defense-evaluation-ea979a9fc1c5 Language models red teaming — attack and defense evaluation . Medium
2023
-
[10]
Lizhi Lin, Honglin Mu, Zenan Zhai, Minghan Wang, Yuxia Wang, Renxi Wang, Junjie Gao, Yixuan Zhang, Wanxiang Che, Timothy Baldwin, Xudong Han, and Haonan Li. 2024. Against the achilles' heel: A survey on red teaming for generative models. arXiv preprint arXiv:2404.00629v2. V2
2024 arXiv
-
[11]
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:...
2024 arXiv
-
[12]
Perez, S
E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, et al. 2022. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419--3448
2022
-
[13]
Perez and I
F. Perez and I. Ribeiro. 2022. Attack techniques for language models. In NeurIPS ML Safety Workshop
2022
-
[14]
Kush R Vashney. 2022. Trustworthy machine learning. Independently published
2022
-
[15]
Le, Denny Zhou, and Xinyun Chen
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. https://arxiv.org/abs/2309.03409 Large language models as optimizers
2024 arXiv
-
[16]
Fangqun Yang and Yisong Wang. 2023. https://doi.org/10.3390/e25111508 Analyzing the robustness of complex networks with attack success rate . Entropy, 25(11):1508
2023 doi
-
[17]
Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, page 100211
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.