Pith. sign in

REVIEW 3 cited by

PERSONA: A Reproducible Testbed for Pluralistic Alignment

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.17387 v1 pith:RG7NW77Y submitted 2024-07-24 cs.CL

Louis Castricato , Nathan Lile , Rafael Rafailov , Jan-Philipp Fränken , Chelsea Finn This is my paper · ORCID
classification cs.CL
keywords alignmentdatasetpersonadiversepluralisticuserapproachesbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The rapid advancement of language models (LMs) necessitates robust alignment with diverse user values. However, current preference optimization approaches often fail to capture the plurality of user opinions, instead reinforcing majority viewpoints and marginalizing minority perspectives. We introduce PERSONA, a reproducible test bed designed to evaluate and improve pluralistic alignment of LMs. We procedurally generate diverse user profiles from US census data, resulting in 1,586 synthetic personas with varied demographic and idiosyncratic attributes. We then generate a large-scale evaluation dataset containing 3,868 prompts and 317,200 feedback pairs obtained from our synthetic personas. Leveraging this dataset, we systematically evaluate LM capabilities in role-playing diverse users, verified through human judges, and the establishment of both a benchmark, PERSONA Bench, for pluralistic alignment approaches as well as an extensive dataset to create new and future benchmarks. The full dataset and benchmarks are available here: https://www.synthlabs.ai/research/persona.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. More Is Not More: What Matters for Diversity in LLM Opinions?

    cs.CL 2026-05 conditional novelty 7.0 of 10

    Diversity in LLM opinions comes mostly from the first persona sentence and from combining different interaction architectures, not from richer personas, temperature, or diversity instructions.

  2. Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models

    cs.LG 2025-07 conditional novelty 7.0 of 10

    A demographically diverse annotation dataset shows that safety perceptions for text-to-image outputs vary by rater identity and that conventional safety classifiers under-detect bias harms flagged by minority-group raters.

  3. Whispers of Many Shores: Cultural Alignment through Collaborative Cultural Expertise

    cs.AI 2025-05 reject novelty 4.0 of 10

    A multi-agent router that selects culturally specialized LLM personas reports a jump in self-scored cultural alignment from 0.208 to 0.820, but the metric and the claimed method are not independently validated.

Pith tools