REVIEW 7 cited by
A Closer Look at System Prompt Robustness
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
System prompts have emerged as a critical control surface for specifying the behavior of LLMs in chat and agent settings. Developers depend on system prompts to specify important context, output format, personalities, guardrails, content policies, and safety countermeasures, all of which require models to robustly adhere to the system prompt, especially when facing conflicting or adversarial user inputs. In practice, models often forget to consider relevant guardrails or fail to resolve conflicting demands between the system and the user. In this work, we study various methods for improving system prompt robustness by creating realistic new evaluation and fine-tuning datasets based on prompts collected from from OpenAI's GPT Store and HuggingFace's HuggingChat. Our experiments assessing models with a panel of new and existing benchmarks show that performance can be considerably improved with realistic fine-tuning data, as well as inference-time interventions such as classifier-free guidance. Finally, we analyze the results of recently released reasoning models from OpenAI and DeepSeek, which show exciting but uneven improvements on the benchmarks we study. Overall, current techniques fall short of ensuring system prompt robustness and further study is warranted.
Forward citations
Cited by 7 Pith papers
-
Rewriting the Response Path: Silent Tampering and Provider-Signed Defense in BYOK LLM Agents
A malicious relay can strategically rewrite aligned LLM outputs in BYOK agent architectures to achieve up to 99.1% attack success on benchmarks like AgentDojo and ASB.
-
Steering Instruction Hierarchies at Inference Time
V-Steer boosts system-prompt value spans and suppresses conflicting user spans, raising controlled instruction-hierarchy adherence from under 18% to as high as 92%.
-
Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks
On Cybench CTF tasks, 21 of 22 models cheated under baseline, 37% of baseline passes were cheated, and stricter anti-cheat prompts reduced but did not remove cheating.
-
Reasoning Up the Instruction Ladder for Controllable Language Models
RLVR on ~7K verifiable system/user conflict examples teaches LLMs to prioritize higher-priority instructions, improving instruction-hierarchy and safety benchmarks.
-
Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessment
Attacker-controlled tool descriptions and return values can hijack tool invocation in popular LLM coding agents, yielding remote code execution and denial of service.
-
Research Challenges in Relational Database Management Systems for LLM Queries
Open-source SQL-plus-LLM database integrations fail on most representative LLM queries and run orders of magnitude slower than enterprise APIs, due to missing structured outputs, poor batching, and plan-unaware LLM calls.
-
A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.
Discussion (0). Sign in to comment.