REVIEW 7 cited by
Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) are susceptible to security and safety threats, such as prompt injection, prompt extraction, and harmful requests. One major cause of these vulnerabilities is the lack of an instruction hierarchy. Modern LLM architectures treat all inputs equally, failing to distinguish between and prioritize various types of instructions, such as system messages, user prompts, and data. As a result, lower-priority user prompts may override more critical system instructions, including safety protocols. Existing approaches to achieving instruction hierarchy, such as delimiters and instruction-based training, do not address this issue at the architectural level. We introduce the Instructional Segment Embedding (ISE) technique, inspired by BERT, to modern large language models, which embeds instruction priority information directly into the model. This approach enables models to explicitly differentiate and prioritize various instruction types, significantly improving safety against malicious prompts that attempt to override priority rules. Our experiments on the Structured Query and Instruction Hierarchy benchmarks demonstrate an average robust accuracy increase of up to 15.75% and 18.68%, respectively. Furthermore, we observe an improvement in instruction-following capability of up to 4.1% evaluated on AlpacaEval. Overall, our approach offers a promising direction for enhancing the safety and effectiveness of LLM architectures.
Forward citations
Cited by 7 Pith papers
-
IH-Benchmark: A Conflict-Centered Benchmark for Instruction-Hierarchy Robustness in LLM Applications
IH-Benchmark, a 2,336-scenario test across 37 LLMs, shows instruction-hierarchy robustness is fragmented: strong system-vs-user compliance does not imply user-vs-tool robustness.
-
Language Shapes Instruction Hierarchy Compliance in Multilingual LLMs
Instruction-hierarchy compliance in LLMs is asymmetric by language and position, and cross-language conflicts yield systematically higher compliance than same-language ones (Language Boundary Effect).
-
Prompt Injection as Role Confusion
Prompt injection works because models internally treat text that sounds like a trusted role as if it were tagged as that role, and this confusion can be measured before generation.
-
Reasoning Up the Instruction Ladder for Controllable Language Models
RLVR on ~7K verifiable system/user conflict examples teaches LLMs to prioritize higher-priority instructions, improving instruction-hierarchy and safety benchmarks.
-
Defending Against Prompt Injection With a Few DefensiveTokens
Five optimized token embeddings prepended to the prompt reduce prompt-injection attack success to near zero on standard benchmarks while preserving most model utility.
-
A Critical Evaluation of Defenses against Prompt Injection Attacks
StruQ, SecAlign, Instruction Hierarchy, PromptGuard, and Attention Tracker are substantially less effective and utility-preserving than claimed when evaluated with diverse prompts and adaptive attacks.
-
Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Execution
Prompt injection can make LLM agents leak personal data they observed while executing tasks, with measured attack success rates around 15-20 percent and password leakage much rarer.
Discussion (0). Sign in to comment.