REVIEW 6 cited by
Adversarial Demonstration Attacks on Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
With the emergence of more powerful large language models (LLMs), such as ChatGPT and GPT-4, in-context learning (ICL) has gained significant prominence in leveraging these models for specific tasks by utilizing data-label pairs as precondition prompts. While incorporating demonstrations can greatly enhance the performance of LLMs across various tasks, it may introduce a new security concern: attackers can manipulate only the demonstrations without changing the input to perform an attack. In this paper, we investigate the security concern of ICL from an adversarial perspective, focusing on the impact of demonstrations. We propose a novel attack method named advICL, which aims to manipulate only the demonstration without changing the input to mislead the models. Our results demonstrate that as the number of demonstrations increases, the robustness of in-context learning would decrease. Additionally, we also identify the intrinsic property of the demonstrations is that they can be used (prepended) with different inputs. As a result, it introduces a more practical threat model in which an attacker can attack the test input example even without knowing and manipulating it. To achieve it, we propose the transferable version of advICL, named Transferable-advICL. Our experiment shows that the adversarial demonstration generated by Transferable-advICL can successfully attack the unseen test input examples. We hope that our study reveals the critical security risks associated with ICL and underscores the need for extensive research on the robustness of ICL, particularly given its increasing significance in the advancement of LLMs.
Forward citations
Cited by 6 Pith papers
-
Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
Self-attention's local Lipschitz constant can be bounded using the attention probability distribution, and the softmax Jacobian spectral norm is shown to be at most 1/2, leading to a new robustness regularizer.
-
Visual Persuasion: What Influences Decisions of Vision-Language Models?
Naturalistic edits to background, lighting, and staging systematically shift the choices of frontier vision-language models, and a new optimization-plus-interpretation framework (CVPO) discovers and explains the visua...
-
SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses
A systemization of LLM jailbreak security that adds linked taxonomies, an evaluation platform, and JailbreakDB, while its main attack–defense comparison results remain deferred.
-
Innocence in the Crossfire: Roles of Skip Connections in Jailbreaking Visual Language Models
The paper reports higher harmful-output rates in three open-source VLMs from detailed image descriptions, in-context examples, and positive openings, and from a skip connection between internal layers, with memes riva...
-
Exploring Explanations Improves the Robustness of In-Context Learning
Generating and comparing label-by-label explanations in the prompt improves out-of-distribution accuracy of in-context learning on most tested NLU benchmarks, at the cost of in-distribution performance and higher infe...
-
SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Models
Across 510 HarmBench behaviors and seven attack methods, GPT-4 models show more consistent jailbreak resilience than DeepSeek models, whose vulnerability grows with scale.
Discussion (0). Sign in to comment.