REVIEW 10 cited by
When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Prompting serves as the major way humans interact with Large Language Models (LLM). Commercial AI systems commonly define the role of the LLM in system prompts. For example, ChatGPT uses ``You are a helpful assistant'' as part of its default system prompt. Despite current practices of adding personas to system prompts, it remains unclear how different personas affect a model's performance on objective tasks. In this study, we present a systematic evaluation of personas in system prompts. We curate a list of 162 roles covering 6 types of interpersonal relationships and 8 domains of expertise. Through extensive analysis of 4 popular families of LLMs and 2,410 factual questions, we demonstrate that adding personas in system prompts does not improve model performance across a range of questions compared to the control setting where no persona is added. Nevertheless, further analysis suggests that the gender, type, and domain of the persona can all influence the resulting prediction accuracies. We further experimented with a list of persona search strategies and found that, while aggregating results from the best persona for each question significantly improves prediction accuracy, automatically identifying the best persona is challenging, with predictions often performing no better than random selection. Overall, our findings suggest that while adding a persona may lead to performance gains in certain settings, the effect of each persona can be largely random. Code and data are available at https://github.com/Jiaxin-Pei/Prompting-with-Social-Roles.
Forward citations
Cited by 10 Pith papers
-
An Empirical Study of Vulnerabilities in Python Packages and Their Detection
PyVul is a new benchmark of 1,157 Python-package vulnerabilities with LLM-cleaned labels; current static detectors detect only a small fraction of them.
-
Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins
Small hyperbolic models (146M–3B) report 100% creative-seed preference, 90.7% compliance-gap detection, and a selective-gating skeleton–wallpaper memory pilot as a companion-AI stack.
-
Point of Order: Action-Aware LLM Persona Modeling for Data-Grounded Civic Deliberation
Fine-tuning on speaker-attributed, action-tagged transcripts from public meetings lets LLM agents mimic government meeting participants well enough that human judges often cannot tell them from real people.
-
Emotionally-Aware Agents for Dispute Resolution
LLM emotion labeling of dispute dialogues explains up to ~40% of variance in subjective outcomes (vs ~5% in prior negotiation work) and reveals anger escalation and compassion de-escalation patterns.
-
Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMs
A multi-task evaluation framework shows persona consistency in LLMs varies by persona category, task structure, and model, with stereotyped spillovers and default personas shaping outputs.
-
Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)
Placing demographic audience information in system prompts rather than user prompts shifts sentiment and ranking outputs across six commercial LLMs, but the design confounds position with instruction content.
-
IHEval: Evaluating Language Models on Following the Instruction Hierarchy
IHEval shows that current language models often follow lower-priority instructions over system messages, and simple prompting does not fix the problem.
-
AiRacleX: Automated Detection of Price Oracle Manipulations via LLM-Driven Knowledge Mining and Prompt Generation
A three-LLM pipeline (knowledge synthesis, prompt generation, auditing) is reported to detect 2.58x more price oracle manipulation bugs than GPTScan on 60 known DeFi vulnerabilities.
-
Exploring LLM Capabilities in Extracting DCAT-Compatible Metadata for Data Cataloging
Large language models can generate DCAT-compatible metadata for data catalogs at quality close to human annotations, though the strongest evidence is for simple extraction tasks.
-
Fine-Tuning and Prompt Engineering of LLMs, for the Creation of Multi-Agent AI for Addressing Sustainable Protein Production Challenges
A proof-of-concept multi-agent GPT system for microbial protein literature extraction shows both fine-tuning and prompt engineering improve cosine-similarity scores, with fine-tuning slightly ahead but more variable.
Discussion (0). Continue with ORCID to comment.