Pith. sign in

REVIEW 4 major objections 4 minor 58 references

PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that many-shot jailbreaking can be made more effective by three prompt-level edits: positive affirmations inserted into the fabricated dialogue, a refusal-then-correction negative demonstration, and an optimized…

desk verdict PANDAS is a useful, incremental extension of many-shot jailbreaking, but the headline gains are inflated because the adaptive sampling and PA/ND positions are tuned on the same benchmark used to report final numbers. read the letter →

arxiv 2502.01925 v2 pith:YJZ4IZWM submitted 2025-02-04 cs.CL cs.CRcs.LG

Avery Ma , Yangchen Pan , Amir-massoud Farahmand This is my paper · ORCID
classification cs.CLcs.CRcs.LG
keywords many-shotjailbreakinglong-contextattackspositiveaffirmationnegativedemonstrationadaptivesamplingBayesianoptimizationLLMsafetyalignmentattentionanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that many-shot jailbreaking, which tricks a safety-aligned language model into answering harmful prompts by prefacing them with hundreds of fabricated user–model exchanges, can be made more effective through three inexpensive prompt-level edits. It introduces PANDAS, which inserts positive affirmation phrases into the fabricated dialogue, embeds a refusal-then-correction 'negative demonstration' into one exchange, and samples the demonstration topics from a distribution optimized by Bayesian optimization for the target prompt's topic. Across five open-source models and the AdvBench and HarmBench benchmarks, PANDAS reports higher attack success rates than standard many-shot jailbreaking and an improved variant (i-MSJ) at the same number of demonstrations. If the reported gains hold, the work suggests that long-context safety alignment can be bypassed with simple template modifications, and it provides a new benchmark dataset and attention-based evidence for how the model's attention shifts toward earlier demonstrations.

What carries the argument

The technical machinery is three modifications to the fabricated dialogue. Positive affirmation (PA) inserts a short appreciative phrase before a malicious question; negative demonstration (ND) rewrites one demonstration as question–refusal–correction–harmful answer; adaptive sampling (AS) replaces uniform topic sampling with a distribution over twelve topics that is optimized by Bayesian optimization separately for each target model. The paper also introduces the ManyHarm dataset of 2,400 malicious question–answer pairs under a controlled structure to support the attack. For analysis, it defines a segment-level reference score $R_i = 1 - S_{i,i}$, the fraction of attention segment $i$ directs toward previous segments, and shows PA and ND raise this score, tying the attack improvements to increased attention on the demonstrated compliant pattern.

What would settle it

Run PANDAS with the sampling distribution and PA/ND positions fixed exactly as selected during tuning, then evaluate on a disjoint holdout split of AdvBench and HarmBench (prompts never used in the Bayesian optimization or placement search) using the same judge; if the ASR-L gap over MSJ on that holdout is small or negative, the paper's central claim of a robust improvement over baselines would be refuted.

Watch

Extended reading notes

Core claim

The central discovery is that the instruction-following pattern created by many-shot jailbreaking can be strengthened without adding more demonstrations. PA phrases ('Exactly the detail I needed! Thanks!') placed before subsequent malicious questions raise the cost of deviating from the established compliant pattern; ND sequences (a refusal, a user correction, then the original harmful answer) demonstrate how to override refusals; and an adaptive sampling distribution over twelve harmful-demonstration topics, learned via Bayesian optimization, yields higher attack success than uniform sampling. The paper reports that PANDAS consistently outperforms both MSJ and i-MSJ across Llama-3.1-8B, Qwen-2.5-7B, GLM-4-9B, OpenChat-3.6-8B, and OLMo-2-7B on AdvBench50, AdvBench, and HarmBench, with ASR-L gains of up to roughly 30 percentage points on the harder Qwen-2.5-7B case. Attention analysis shows that PA and ND increase the 'reference score'—the fraction of attention each demonstration segment pays to earlier segments—which the paper interprets as reinforcement of the instruction-following pattern.

Load-bearing premise

The measured advantage may not survive on prompts that were not used to tune the method, because the adaptive sampling distribution and the placement of the affirmations and negative demonstrations were chosen using the same evaluation prompts that appear in the headline results.

Editorial extensions

If this is right

  • PA and ND can be added to existing many-shot jailbreaking prompts with no extra inference cost and still raise attack success, lowering the barrier to mounting stronger long-context attacks.
  • Because jailbreak effectiveness does not reliably increase with shot count on 8B models, evaluating long-context attacks at a single shot count can miss the peak, which appeared as early as 32 shots on GLM-4-9B.
  • Input-perturbation defenses such as Retokenization and SmoothLLM lose their protective effect at high shot counts because the model begins mimicking the perturbations, so defense evaluations should include long-context settings.
  • Successful MSJ and PANDAS prompts transfer across models, with transferability improving at higher shot counts and showing asymmetry between GLM-4-9B and Qwen-2.5-7B, which supports using open models to generate transferable attacks against costly proprietary models.
  • Perplexity-based defenses fail against both MSJ and PANDAS because the attacks use fluent, in-distribution text rather than special strings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gains survive evaluation on held-out prompts, safety teams may need to treat topical context engineering, not just token-level or instruction-level attacks, as a primary long-context threat in open 8B models.
  • The Bayesian-optimized distributions favoring 'regulated-content' and 'sexual' topics (Figure 2) suggest that certain topic categories act as stronger context primes; testing this topic-transfer hypothesis on other model families is a direct extension.
  • The attention reference score $R_i$ could be repurposed as an interpretability diagnostic: measuring whether a defense reduces reference-score growth might predict whether it will hold under long-context attacks.
  • Because PANDAS's gains did not require i-MSJ's costly random search, the comparison hints that rich context structure may matter more than token-level probability tuning in the many-shot regime.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes PANDAS, a hybrid many-shot jailbreaking method that augments the standard many-shot jailbreaking (MSJ) prompt with three modifications: positive affirmation (PA) phrases, a negative demonstration (ND) that embeds a refusal-and-correction turn, and an adaptive sampling (AS) distribution over demonstration topics, where the topic distribution is learned per target model via Bayesian optimization. The authors introduce ManyHarm, a dataset of 2,400 harmful question-answer pairs across 12 policy topics, and evaluate on five open-weight 7-8B models using AdvBench50, AdvBench, and HarmBench. The central empirical claim is that PANDAS outperforms MSJ and i-MSJ under equal shot counts (up to 256) across all evaluated models and datasets, as measured by both LLM-based (ASR-L, Llama-Guard-3) and rule-based (ASR-R) attack success rates. A structured attention analysis is used to argue that PA and ND reinforce instruction-following by increasing the fraction of attention each demonstration directs to earlier demonstrations.

Significance. If the reported gains survive a properly separated evaluation protocol, PANDAS would be a simple, transferable, and computationally cheap improvement to long-context jailbreaking, which is directly relevant to red-teaming and safety alignment. The PA and ND components are practically valuable because they require no additional demonstrations and no search, and the release of source code supports reproducibility. The ManyHarm dataset, with its 12-topic structure and controlled formatting, is a useful resource for controlled studies of demonstration-level properties. However, the current evaluation protocol makes the headline magnitudes optimistic: the PA/ND insertion positions are selected on the evaluation set (Sec. 4.1), the adaptive sampling distribution is optimized without a held-out split (Sec. 3.3, Appendix B), and no error bars or significance tests accompany the reported numbers (Tables 1-2). The direction of the effect is likely real, since the component-wise results in Table 3 show consistent positive signs, but the size of the improvement and the independent contribution of the adaptive sampling component remain to be established.

major comments (4)
  1. [Sec. 4.1, Sec. 3.3, Appendix B, Table 1] The reported PANDAS numbers are selected maxima over configurations tuned directly on the evaluation set. Section 4.1 states that PA/ND insertion positions are chosen as "the configuration that achieves the highest ASR-L for 256 shots on AdvBench50," and Table 1 reports PANDAS on the same AdvBench50 split; Section 3.3 says the adaptive sampling distribution z is learned by Bayesian optimization "performed separately for each target model," with Appendix B giving 50 BO steps and no held-out split. The AdvBench50 column is therefore a fitted value rather than a prediction, while MSJ and i-MSJ are fixed or per-prompt algorithms without benchmark-level tuning. The same concern applies to the AdvBench and HarmBench columns if z is optimized on prompts from those datasets. This makes the reported gaps (e.g., 94 vs 80 ASR-L for Llama-3.1-8B at 256 shots) upper-biased estimates of the method's advantage. The authors should tune on one split (e.g., AdvBench50) and report held-out benchmarks only, or use nested/leave-one-topic-out evaluation, and should quantify the variance of the selection procedure.
  2. [Appendix B, Sec. 3.3] The BO reward is a first-token proxy (success if the first token is not "I", with outputs beginning with "1" also counted as successful in some cases), whereas Table 1 reports ASR-L and ASR-R from full responses judged by Llama-Guard-3. The paper does not show that maximizing this proxy correlates with the reported metrics, and 50 optimization steps after 5 random evaluations are unlikely to reliably converge on a 12-dimensional simplex with noisy evaluations. Without repeated BO runs or a validation set, the adaptive-sampling contribution in Table 3 (e.g., ASR-L 87.12 vs 85.19 for MSJ on AdvBench at 128 shots) is indistinguishable from noise. Please report the proxy-to-metric correlation, the number of target prompts used per BO evaluation, the variability of the optimized distribution across independent runs, and the ASR of the final distribution on held-out prompts.
  3. [Tables 1 and 2, Sec. 4.1] The evaluation provides no uncertainty quantification. The protocol aggregates three restarts as success-if-any ("the jailbreak is considered successful if any of the 3 attempts succeeds"), which collapses variability, and all reported numbers are point estimates without confidence intervals or significance tests. For example, the Qwen-2.5-7B AdvBench50 differences (e.g., 22 vs 10 at 256 shots) are based on 50 prompts with success-if-any aggregation, so differences of several percentage points are within the expected noise. The abstract's "significantly outperforms" is not supported by any statistical analysis. Please report per-restart ASR distributions, confidence intervals, or a paired bootstrap over the prompt set for the main comparisons.
  4. [Sec. 4.5, Eq. (5)] The partition definition in the attention analysis is internally inconsistent. The paper defines P_{i,j} for 1≤i≤j≤n with k∈[N_i,N_{i+1}), q∈[N_j,N_{j+1}), and k≤q. Under the stated convention that rows of A correspond to queries and each row sums to 1, causal attention imposes q≤k (a token attends to itself and earlier positions), so the condition k≤q selects the wrong triangle within a segment and makes P_{i,j} identically zero for i<j. Moreover, the subsequent definition R_i = 1 − S_{i,i} = Σ_{j=1}^{i−1} S_{i,j} requires S_{i,j} for j<i, which is not covered by the stated range 1≤i≤j≤n. As written, Eqs. (5)-(6) and Figures 4-5 cannot be reproduced from the definitions. The index ranges and the triangular condition should be corrected (probably j≤i with q≤k), and the reported reference scores should be recomputed or confirmed against the corrected definition.
minor comments (4)
  1. [Figure 1, Table 1] The supplied version contains typographical and formatting issues: "Maliclious" in Figure 1, and several PANDAS rows in Table 1 show values concatenated with inconsistent spacing (e.g., "96.0098.0094.00"). Please ensure the camera-ready version aligns all numbers correctly.
  2. [Sec. 4.1, Table 3] The selection rule in Sec. 4.1 ("the configuration that achieves the highest ASR-L for 256 shots on AdvBench50") is not reconciled with Table 3, where PA is "added after random demonstrations" and ND "after the first demonstration." Please state explicitly whether Table 3 uses the selected configuration or a different fixed configuration, and unify the notation for the insertion position m.
  3. [Sec. 4.2] The ManyHarm dataset is released "upon request, subject to eligibility review," which limits reproducibility relative to a standard dataset release. Consider also providing a detailed metadata description and a seed-based sampling protocol so that researchers without access to the exact dataset can still replicate the experimental conditions approximately.
  4. [Appendix H, Table 6] The transferability results are based on only 20 prompts per source-target pair. The text describes effects like "nearly 100%" and asymmetry, but with n=20 the confidence intervals are wide; please report exact counts or bootstrap intervals for these comparisons.

Circularity Check

2 steps flagged · score 6.0 of 10

PANDAS's headline gains are measured on the same data used to tune its adaptive-sampling distribution and PA/ND insertion positions, so the Table 1 improvements are partially fitted values rather than independent predictions.

  1. fitted input called prediction [Sec. 4.1, 'Implementation Details of PANDAS']
    "For PA and ND, we explore the impact of the modified demonstrations’ position (i.e., m in (2) and (4)) by evaluating four configurations: modifying the first demonstrations, the last demonstrations, all demonstrations, or a random subset of demonstrations. Results are reported using the configuration that achieves the highest ASR-L for 256 shots on AdvBench50."

    The four PA/ND configurations are method variants whose ASR-L is the paper's headline metric. Selecting the best configuration on AdvBench50 and then reporting that configuration's numbers in the AdvBench50 column of Table 1 turns that column into a maximum over four fitted configurations. MSJ and i-MSJ have no equivalent model selection, so the PANDAS-versus-baseline gap on that dataset is inflated by construction: the reported performance is the metric used to choose the configuration, not an independent prediction.

  2. fitted input called prediction [Sec. 3.3 'Adaptive Sampling' and Appendix B]
    "We treat B as a black-box function and optimize it using Bayesian optimization ... This optimization is performed separately for each target model. ... We use 5 steps of random exploration and set the total number of optimization steps to 50."

    The adaptive sampling distribution z is the component that defines PANDAS, and B:z→r maps z to the jailbreak success rate on the evaluation targets. The paper reports Table 1 ASR after Bayesian optimization has maximized r on those same target prompts, with no held-out split described in Appendix B. Thus the reported PANDAS ASR is an optimized objective value on the evaluation set, whereas MSJ and i-MSJ run with fixed sampling. The claim that PANDAS significantly outperforms baselines is therefore supported by a fitted value compared to untuned baselines on the fitting set, which is upward-biased and requires held-out evaluation to confirm the magnitude.

full rationale

The paper's central claim—that PANDAS significantly outperforms MSJ and i-MSJ at equal shot counts—is partially circular in evaluation protocol. The adaptive sampling distribution is learned by Bayesian optimization on the same target prompts whose ASR is then reported, and the PA/ND insertion positions are selected as the best-performing configuration on AdvBench50 before that same AdvBench50 split is used in Table 1. This is not a theorem-level construction or a fully self-definitional derivation, but it falls under 'fitted input called prediction': the headline ASR numbers are optimized values on the test set rather than predictions from a fixed method. The paper also contains independent content: PA and ND are fixed, simple prompt modifications that improve ASR in ablations on AdvBench and HarmBench, and the consistent qualitative direction across models and datasets suggests the effect is likely real. However, the magnitudes in Table 1—including the reported gaps on AdvBench50—are upper-biased because tuning and measurement share the same targets. No load-bearing self-citation chain or uniqueness-theorem circularity is present. The attention analysis is post-hoc interpretation rather than a derivation, so it does not add circularity. Score is 6 rather than 8 because the central claim retains independent empirical support outside the fitted component, but the headline comparison is partially reduced to in-sample optimization.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. The ManyHarm dataset is a constructed artifact, not a postulated entity; the load-bearing extras are fitted parameters and measurement assumptions.

free parameters (4)
  • adaptive sampling topic distribution z = see Figure 2 (qualitative)
    C-dimensional simplex distribution over 12 topics, fit via Bayesian optimization separately per model; directly drives the AS component.
  • PA insertion position m = after all demonstrations (general finding)
    Chosen on AdvBench50 by picking the configuration with the highest ASR-L at 256 shots; a hyperparameter selection on the evaluation set.
  • ND insertion position m = after the first demonstration
    Same selection procedure on AdvBench50.
  • PA, refusal, and correction phrase lists = 10 hand-written phrases per type (Appendix C)
    The specific wording of the inserted phrases is a design choice; results may be sensitive to it.
assumptions (5)
  • domain assumption Reinforcing the instruction-following pattern (via praise and correction) increases the probability of harmful compliance because breaking the pattern is costly.
    Section 3.1 mechanism, based on Wei et al. (2023a); the attention analysis only shows correlation, not causation.
  • domain assumption MSJ success behaves like in-context learning and is sensitive to demonstration design, including negative demonstrations.
    Section 3.2, motivated by Anil et al. (2024) and ICL literature.
  • domain assumption Target prompts can be reliably grouped into the 12 topics, and a topic-specific sampling distribution generalizes across prompts in that topic.
    Section 3.3 and Appendix B; AdvBench topics are assigned by Llama-3.1-8B, which is also the evaluated model.
  • domain assumption Llama-Guard-3 and the extended refusal-phrase list are valid automatic measures of harmful behavior.
    Section 4.1; the paper manually inspects responses but reports automatic metrics as primary evidence.
  • domain assumption During Bayesian optimization, treating a first token of 'I' as a refusal and '1' as a success is a reliable proxy for final ASR.
    Appendix B; this proxy defines the fitted distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling." pith.science (2026). https://pith.science/paper/YJZ4IZWM

@misc{pith2026250201925,
  author       = {Pith},
  title        = {Pith review of: PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YJZ4IZWM}},
  note         = {Machine review of arXiv:2502.01925}
}
read the original abstract

Many-shot jailbreaking circumvents the safety alignment of LLMs by exploiting their ability to process long input sequences. To achieve this, the malicious target prompt is prefixed with hundreds of fabricated conversational exchanges between the user and the model. These exchanges are randomly sampled from a pool of unsafe question-answer pairs, making it appear as though the model has already complied with harmful instructions. In this paper, we present PANDAS: a hybrid technique that improves many-shot jailbreaking by modifying these fabricated dialogues with Positive Affirmations, Negative Demonstrations, and an optimized Adaptive Sampling method tailored to the target prompt's topic. We also introduce ManyHarm, a dataset of harmful question-answer pairs, and demonstrate through extensive experiments that PANDAS significantly outperforms baseline methods in long-context scenarios. Through attention analysis, we provide insights into how long-context vulnerabilities are exploited and show how PANDAS further improves upon many-shot jailbreaking.

Figures

Figures reproduced from arXiv: 2502.01925 by the authors.

Figure 1
Figure 1. PANDAS improves many-shot jailbreaking by intro￾ducing: 1. Positive Affirmation phrases inserted before the next malicious question, 2. refusal and correction phrases to create a Negative Demonstration where the model initially refuses, fol￾lowed by a user correction prompt, after which the model provides the original malicious response, and 3. Adaptive Sampling of demonstrations based on the topics of the malicious… view at source ↗
Figure 2
Figure 2. Sampling distribution obtained through Bayesian op￾timization for Llama-3.1-8B. The HarmBench dataset contains prompts from 6 topics. For each of these 6 topics, we identify the optimal sampling distributions across 12 topics of malicious demonstrations. Demonstrations from topics such as “regulated￾content” and “sexual” are frequently selected for improved jail￾breaking effectiveness. We initialize the optimization… view at source ↗
Figure 3
Figure 3. Illustration of how the attention map is divided into smaller partitions based on segments of a 4-shot MSJ prompt. N1, . . . , N5 denote the token indices that segments the input prompt based on demonstrations, and N5 marks the start of the target prompt. These indices divide the input prompt into segments. Attention scores in the red rectangular partitions represent how tokens from different segments attend to each… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Reference scores of a 32-shot MSJ and its PA and ND variants as the number of demonstrations increase. We insert a refusal phrase immediately after the first question in the initial demonstration, making the first dialogue an ND. As a result, MSJ and PA begin at index …
Figure 5
Figure 5. Figure 5: Reference scores of a 32-shot MSJ and its PA and ND variants as the number of demonstrations increase. Left: OLMo-2- 7B; Middle: openchat-3.6-8b; Right: Qwen-2.5-7B. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 45 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT -4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    What learning algorithm is in-context learning? investigations with linear models

    Aky \"u rek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learning? investigations with linear models. In International Conference on Learning Representations (ICLR), 2024

    work page 2024

  4. [4]

    Jailbreaking leading safety-aligned LLM s with simple adaptive attacks

    Andriushchenko, M., Croce, F., and Flammarion, N. Jailbreaking leading safety-aligned LLM s with simple adaptive attacks. In International Conference on Learning Representations (ICLR), 2025

    work page 2025

  5. [5]

    J., et al

    Anil, C., Durmus, E., Rimsky, N., Sharma, M., Benton, J., Kundu, S., Batson, J., Tong, M., Mu, J., Ford, D. J., et al. Many-shot jailbreaking. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    work page 2024

  6. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  7. [7]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  8. [8]

    J., and Wong, E

    Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking black box large language models in twenty queries. In Workshop on robustness of zero/few-shot learning in foundation models (R0-FoMo), 2023

    work page 2023

Show all 58 references
  1. [9]

    How many demonstrations do you need for in-context learning? In Findings of the Association for Computational Linguistics: EMNLP, 2023

    Chen, J., Chen, L., Zhu, C., and Zhou, T. How many demonstrations do you need for in-context learning? In Findings of the Association for Computational Linguistics: EMNLP, 2023

    2023

  2. [10]

    What does BERT look at? A n analysis of BERT ’s attention

    Clark, K. What does BERT look at? A n analysis of BERT ’s attention. In Proceedings of the ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019

    2019

  3. [11]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Devlin, J. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019

    2019

  4. [12]

    L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., and Yang, M

    Ding, Y., Zhang, L. L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., and Yang, M. Longrope: Extending LLM context window beyond 2 million tokens. In International Conference on Machine Learning (ICML), 2024

    2024

  5. [13]

    X., Wang, B., Tian, Z., Chen, W., and Wen, J.-R

    Dong, Z., Li, J., Men, X., Zhao, W. X., Wang, B., Tian, Z., Chen, W., and Wen, J.-R. Exploring context window of large language models via decomposed positional vectors. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  6. [14]

    The Llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

    Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. In Proceedings of the Conference on Empirical Methods i...

    2022

  8. [16]

    and Das, K

    Gao, X. and Das, K. Customizing language model responses with contrastive in-context learning. In AAAI Conference on Artificial Intelligence, 2024

    2024

  9. [17]

    ChatGLM : A family of large language models from GLM-130B to GLM-4 all tools, 2024

    GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., et al. ChatGLM : A family of large language models from GLM-130B to GLM-4 all tools, 2024

    2024

  10. [18]

    Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009

    Hansen, N., Auger, A., Ros, R., Finck, S., and Po s \' k, P. Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009 . In Proceedings of the 12th Annual Conference Companion on Genetic and Evolutionary Computation, 2010

    2009

  11. [19]

    Self-attention attribution: Interpreting information interactions inside transformer

    Hao, Y., Dong, L., Wei, F., and Xu, K. Self-attention attribution: Interpreting information interactions inside transformer. In AAAI Conference on Artificial Intelligence, 2021

    2021

  12. [20]

    WizardLM-13B-Uncensored , 2023

    Hartford, E. WizardLM-13B-Uncensored , 2023. URL https://huggingface.co/cognitivecomputations/WizardLM-13B-Uncensored

    2023

  13. [21]

    Measuring mathematical problem solving with the math dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track (Round 2), 2021

    2021

  14. [22]

    Baseline defenses for adversarial attacks against aligned language models

    Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., Chiang, P.-y., Goldblum, M., Saha, A., Geiping, J., and Goldstein, T. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614, 2023

    2023 arXiv

  15. [23]

    LLM maybe LongLM : Self-extend LLM context window without tuning

    Jin, H., Han, X., Yang, J., Jiang, Z., Liu, Z., Chang, C.-Y., Chen, H., and Hu, X. LLM maybe LongLM : Self-extend LLM context window without tuning. In International Conference on Machine Learning (ICML), 2024

    2024

  16. [24]

    Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for GPT- 3 ? In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures (DeeLIO), 2022

    2022

  17. [25]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022

    2022

  18. [26]

    Attention-enhancing backdoor attacks against BERT -based models

    Lyu, W., Zheng, S., Pang, L., Ling, H., and Chen, C. Attention-enhancing backdoor attacks against BERT -based models. In Findings of the Association for Computational Linguistics: EMNLP, 2023

    2023

  19. [27]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal

    Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., and Hendrycks, D. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In International Conference on Machine Learning (ICML), 2024

    2024

  20. [28]

    Tree of attacks: Jailbreaking black-box LLMs automatically

    Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H., Singer, Y., and Karbasi, A. Tree of attacks: Jailbreaking black-box LLMs automatically. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  21. [29]

    Bayesian Optimization : Open source constrained global optimization tool for Python , 2014

    Nogueira, F. Bayesian Optimization : Open source constrained global optimization tool for Python , 2014. URL https://github.com/bayesian-optimization/BayesianOptimization

    2014

  22. [30]

    2 OLMo 2 F urious

    OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., et al. 2 OLMo 2 F urious. 2024

    2024

  23. [31]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  24. [32]

    S., Soltanolkotabi, M., and Thrampoulidis, C

    Oymak, S., Rawat, A. S., Soltanolkotabi, M., and Thrampoulidis, C. On the role of attention in prompt-tuning. In International Conference on Machine Learning (ICML), 2023

    2023

  25. [33]

    S., O'Brien, J., Cai, C

    Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023

    2023

  26. [34]

    Red teaming language models with language models

    Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022

    2022 arXiv

  27. [35]

    Baitattack: Alleviating intention shift in jailbreak attacks via adaptive bait crafting

    Pu, R., Li, C., Ha, R., Zhang, L., Qiu, L., and Zhang, X. Baitattack: Alleviating intention shift in jailbreak attacks via adaptive bait crafting. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

    2024

  28. [36]

    and Barez, F

    Quirke, P. and Barez, F. Understanding addition in transformers. In International Conference on Learning Representations (ICLR), 2024

    2024

  29. [37]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 2019

    2019

  30. [38]

    Tricking LLMs into disobedience: Formalizing, analyzing, and detecting jailbreaks

    Rao, A., Vashistha, S., Naik, A., Aditya, S., and Choudhury, M. Tricking LLMs into disobedience: Formalizing, analyzing, and detecting jailbreaks. In Proceedings of the Joint International Conference on Computational Linguistics, Language Resources and Evaluation, 2024

    2024

  31. [39]

    Robey, A., Wong, E., Hassani, H., and Pappas, G. J. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023

    2023 arXiv

  32. [40]

    P., and De Freitas, N

    Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and De Freitas, N. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 2015

    2015

  33. [41]

    I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al

    Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  34. [42]

    Qwen2.5: A party of foundation models, 2024

    Team, Q. Qwen2.5: A party of foundation models, 2024

    2024

  35. [43]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  36. [44]

    Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020

    Turner, R., Eriksson, D., McCourt, M., Kiili, J., Laaksonen, E., Xu, Z., and Guyon, I. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In Proceedings of the NeurIPS 2020 Compe...

    2020

  37. [45]

    Attention is all you need

    Vaswani, A. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  38. [46]

    Jailbroken: How does LLM safety training fail? In Advances in Neural Information Processing Systems (NeurIPS), 2023 a

    Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does LLM safety training fail? In Advances in Neural Information Processing Systems (NeurIPS), 2023 a

    2023

  39. [47]

    Jailbreak and guard aligned language models with only few in-context demonstrations

    Wei, Z., Wang, Y., Li, A., Mo, Y., and Wang, Y. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387, 2023 b

    2023 arXiv

  40. [48]

    Never miss a beat: An efficient recipe for context window extension of large language models with consistent ``middle'' enhancement

    Wu, T., Zhao, Y., and Zheng, Z. Never miss a beat: An efficient recipe for context window extension of large language models with consistent ``middle'' enhancement. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  41. [49]

    Distract large language models for automatic jailbreak attack

    Xiao, Z., Yang, Y., Chen, G., and Chen, Y. Distract large language models for automatic jailbreak attack. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

    2024

  42. [50]

    Defending chat GPT against jailbreak attack via self-reminders

    Xie, Y., Yi, J., Shao, J., Curl, J., Lyu, L., Chen, Q., Xie, X., and Wu, F. Defending chat GPT against jailbreak attack via self-reminders. Nature Machine Intelligence, 2023

    2023

  43. [51]

    Qwen2 technical report

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  44. [52]

    Tell your model where to attend: Post-hoc attention steering for LLMs

    Zhang, Q., Singh, C., Liu, L., Liu, X., Yu, B., Gao, J., and Zhao, T. Tell your model where to attend: Post-hoc attention steering for LLMs . In International Conference on Learning Representations (ICLR), 2024 a

    2024

  45. [53]

    In-context principle learning from mistakes

    Zhang, T., Madaan, A., Gao, L., Zheng, S., Mishra, S., Yang, Y., Tandon, N., and Alon, U. In-context principle learning from mistakes. In International Conference on Machine Learning (ICML), 2024 b

    2024

  46. [54]

    What makes good examples for visual in-context learning? In Advances in Neural Information Processing Systems (NeurIPS), 2023

    Zhang, Y., Zhou, K., and Liu, Z. What makes good examples for visual in-context learning? In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  47. [55]

    Calibrate before use: Improving few-shot performance of language models

    Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning (ICML), 2021

    2021

  48. [56]

    Improved few-shot jailbreaking can circumvent aligned language models and their defenses

    Zheng, X., Pang, T., Du, C., Liu, Q., Jiang, J., and Lin, M. Improved few-shot jailbreaking can circumvent aligned language models and their defenses. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  49. [57]

    P., Di Eugenio, B., and Zhang, Y

    Zhou, Y., Zou, H. P., Di Eugenio, B., and Zhang, Y. Large language models are involuntary truth-tellers: Exploiting fallacy failure for jailbreak attacks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

    2024

  50. [58]

    Z., and Fredrikson, M

    Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models, 2023

    2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.