REVIEW 4 cited by
Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Language
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We are exposed to much information trying to influence us, such as teaser messages, debates, politically framed news, and propaganda - all of which use persuasive language. With the recent interest in Large Language Models (LLMs), we study the ability of LLMs to produce persuasive text. As opposed to prior work which focuses on particular domains or types of persuasion, we conduct a general study across various domains to measure and benchmark to what degree LLMs produce persuasive language - both when explicitly instructed to rewrite text to be more or less persuasive and when only instructed to paraphrase. We construct the new dataset Persuasive-Pairs of pairs of a short text and its rewrite by an LLM to amplify or diminish persuasive language. We multi-annotate the pairs on a relative scale for persuasive language: a valuable resource in itself, and for training a regression model to score and benchmark persuasive language, including for new LLMs across domains. In our analysis, we find that different 'personas' in LLaMA3's system prompt change persuasive language substantially, even when only instructed to paraphrase.
Forward citations
Cited by 4 Pith papers
-
Tailored untruths: How personalisation challenges LLM safeguards
A 1.6-million-text study of eight LLMs in four languages finds that adding demographic personae to disinformation prompts raises jailbreak rates from 78% to 82%.
-
It Matters How You Say It: Exploring Rhetorical Patterns for AI-Assisted Information Evaluation
In a 98-participant fact-checking study, the rhetorical style of AI advice changed accuracy, confidence, and preference, with step-by-step explanations helping most and user preference diverging from performance.
-
Mind What You Ask For: Emotional and Rational Faces of Persuasion by Large Language Models
Emotional prompts make LLMs produce more cognitively complex language than rational prompts, and models systematically differ in their use of Cialdini influence principles across prompt types.
-
AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language Models
AssertBench measures how often LLMs keep the same true/false evaluation of a fact across contradictory user framings, and finds most tested models agree with the user's framing more when they do not know the fact.
Discussion (0). Continue with ORCID to comment.