Pith. sign in

REVIEW 3 major objections 6 minor 48 references

The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLMs match human moral judgments only under high consensus, and that a prompting method called Dynamic Moral Profiling cuts the judgment-distribution gap by 64.3% while also broadening the moral values models express.

desk verdict Solid dataset and descriptive findings; the DMP improvement claim is likely inflated by test-set leakage and needs a held-out evaluation before it can be trusted. read the letter →

arxiv 2507.17216 v1 pith:MZDA4Z46 submitted 2025-07-23 cs.CL cs.AI

Giuseppe Russo , Debora Nozza , Paul Röttger , Dirk Hovy This is my paper
classification cs.CLcs.AI
keywords moraljudgmentdistributionalalignmentvaluepluralismlargelanguagemodelsdilemmasDirichletsamplingpromptingvalues
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

People increasingly ask large language models for moral advice, but models are usually judged against majority opinions rather than the full spread of human views. This paper builds a dataset of 1,618 real-world dilemmas paired with distributions of human judgments and free-text rationales, and measures how closely LLMs reproduce the entire distribution, not just the majority label. It finds that models align with humans only when human consensus is high, and that in disputed dilemmas models default toward a single judgment while relying on a narrow set of moral values. The paper argues that a prompting method called Dynamic Moral Profiling, which samples value profiles from a Dirichlet distribution fit to human rationales, closes much of this gap: average disagreement falls by 64.3%, with the largest gains in low-consensus cases. If this holds, value-conditioned prompting is a cheap way to make LLM moral advice more pluralistic.

What carries the argument

The load-bearing object is the dynamic moral profile: a topic-specific probability distribution $G_t$ over a 60-value taxonomy, constructed by taking the global empirical value distribution $G_0$ over all human rationales and fitting $G_t \sim \mathrm{Dirichlet}(\alpha G_0)$ per topic with $\alpha = 10$. For each dilemma, $N_i$ profiles are sampled from the multinomial $G_{t_i}$, each profile is a short list of top values with normalized importance weights, and the profile is inserted into the prompt so the model judges the dilemma through that value lens. The mechanism works on two levels: it injects human value diversity into individual generations, and because the sampled profiles vary across the $N_i$ queries, it reshapes the aggregated model judgment distribution toward the human one.

What would settle it

Estimate the global and topic value profiles on a training split of the Moral Dilemma Dataset and apply DMP only to held-out dilemmas; if the average absolute difference no longer falls from 22 to 8 percentage points, the reported improvement is at least partly an artifact of conditioning on the evaluation data.

Watch

Extended reading notes

Core claim

The central discovery is that the divergence between LLM and human moral judgment is distributional rather than only a matter of matching the majority answer. Across the 1,618 dilemmas, the average absolute difference between human and LLM proportions of Acceptable judgments is 22 percentage points for the best baseline, a council of models, and it rises to roughly 30 points in the most disputed dilemmas. Using a 60-value taxonomy built from 3,783 value expressions, the paper shows that LLMs place 81.6% of their value mentions in their top 10 values, versus 35.2% for humans, indicating a narrower moral vocabulary. Dynamic Moral Profiling conditions each model response on a profile of values sampled from a topic-level Dirichlet distribution over the taxonomy; this reduces the average absolute difference from 22 to 8 percentage points, a 64.3% improvement, and raises normalized value entropy from 0.46 to 0.52, closer to the human 0.57.

Load-bearing premise

The load-bearing assumption is that the value profiles used to steer the model are estimated from the same dilemmas that are then used to measure alignment, so the reported 64.3% improvement assumes no leakage from the test dilemmas into the profiles; a held-out evaluation could shrink the gain.

Editorial extensions

If this is right

  • With the reported 64.3% improvement, a prompt-only intervention reduces LLM-human moral judgment disagreement without fine-tuning or preference data.
  • The biggest gains occur in low-consensus dilemmas, precisely the cases where people are most likely to seek moral advice, because DMP breaks models' default to a single Acceptable judgment.
  • DMP broadens the value distribution in model rationales: the top-10 value concentration drops from 81.6% to 51.3%, and value entropy approaches the human baseline.
  • The failure of Moral Foundations Theory baselines suggests that the specificity of the data-driven 60-value taxonomy, not just any value framing, is what carries the alignment improvement.
  • Since DMP samples a fresh value profile for each query, its pluralistic alignment depends on repeated sampling rather than a single user-facing response.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper estimates the value profiles on the same dilemmas used to measure alignment; a held-out split would test whether the 64.3% improvement survives when profiles are built from different dilemmas, and the paper reports no such check.
  • The taxonomy is derived from one online community, so the pluralism DMP injects is pluralism within that community's value space; applying the procedure to other cultural contexts would likely require rebuilding the taxonomy rather than reusing the 60 values.
  • Because each query gets one sampled profile, an extension would be to compress the plural distribution into a single response that reports the spread of moral perspectives instead of one verdict.
  • A further testable extension is to measure downstream effects, such as whether people exposed to DMP-generated rationales make more considered or more diverse decisions than people exposed to standard model outputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces the Moral Dilemma Dataset (MDD), 1,618 real-world moral dilemmas from r/AmITheAsshole with human binary judgments and free-text rationales, and studies distributional alignment between LLM and human moral judgments. RQ1 shows that LLM-human alignment degrades as human consensus decreases. RQ2 builds a 60-value taxonomy from human and LLM rationales using the Value Kaleidoscope and reports that LLMs concentrate their value mentions on a narrow top-10 set (81.6% vs. 35.2% for humans). RQ3 proposes Dynamic Moral Profiling (DMP), a Dirichlet-based method that conditions LLM prompts on value profiles sampled from topic-specific distributions derived from all human rationales; the paper reports a 64.3% reduction in the average absolute judgment difference relative to the model council baseline and an increase in value diversity. The central RQ3 claim is, however, compromised by a leakage problem in the evaluation: the value profiles are estimated from all human rationales, including the same dilemmas used to measure alignment, so the reported improvement conflates genuine steering with access to the human answer distribution.

Significance. If the leakage issue were resolved, this paper would be a solid empirical contribution. The MDD dataset fills a gap by providing ecologically valid, distribution-level moral judgments and rationales, and the RQ1 finding that alignment drops sharply with lower human consensus is a clean, well-supported observation. The RQ2 value-diversity result is plausible and useful, though it depends on the LLM-based Value Kaleidoscope extractor and the clustering pipeline. The DMP method is a creative attempt to steer pluralistic model outputs, and the theory-driven MFT baselines are a reasonable control. However, the headline 64.3% improvement is not interpretable as a generalizable steering effect because the conditioning profiles are built from the test dilemmas themselves. The paper would be strengthened by a held-out or leave-one-dilemma-out evaluation, a sensitivity analysis for the free parameters, and a more precise specification of the generative procedure.

major comments (3)
  1. [Section 5 (DMP evaluation) and Section 5.1] The evaluation of DMP is affected by test-set leakage. In Section 5, the base measure G0 is defined as the empirical frequency distribution over all human rationales in the dataset, and the topic-specific Gt is fit via Dirichlet(alpha*G0) using those same rationales. Profiles for dilemma di are then sampled from Gti and used to condition the model, and alignment is measured on the same dilemma di with the procedure of Section 3.1. Thus the conditioning profile for each dilemma includes value-frequency information drawn from that dilemma's own human rationales. The model council baseline receives no such information, so the reported reduction from 22pp to 8pp (64.3% improvement) confounds the steering effect with leakage of the human judgment distribution. RQ1 and RQ2 are not affected because they do not use Gt, but the central RQ3 claim is not supported by the reported experiment. The authors should evaluate DMP on held-out dilemmas or use leave-one-dilemma-out cross-validation when estimating G0 and Gt, and report the resulting alignment and diversity metrics.
  2. [Section 5 (Dirichlet concentration parameter)] The concentration parameter alpha is set to 10 with no sensitivity analysis. Because DMP's behavior is directly controlled by how strongly the topic-specific profiles are allowed to deviate from the global prior, the reported 64.3% improvement and the diversity gains could be highly sensitive to this arbitrary choice. The authors should report the main results across a range of alpha values (e.g., 1, 5, 10, 20, 50) and show that the qualitative conclusions are stable.
  3. [Section 5 (generative specification)] The generative definition of Gt is underspecified. The text states 'we fit a topic-specific distribution Gt ∼ Dirichlet(alpha*G0) for each topic t', but it does not state whether Gt is a posterior estimate conditioned on the topic's rationales, a single random draw, or a MAP estimate, nor does it explain how topics are defined and how many topics exist. This ambiguity prevents exact reproduction of the DMP procedure and should be resolved with a precise algorithm or pseudocode.
minor comments (6)
  1. [Section 3.2] The text 'reported results for temperature 1 due to its superior performance (see Appendix 4 for comparison)' should refer to the actual appendix table, namely Table 4 in Appendix A.2.1.
  2. [Section 2.1] The phrase 'Evaluation fall into five standard categories' should be 'Evaluations fall into five standard categories'.
  3. [Section 2.1] The phrase 'The buckets where almost balanced' should be 'The buckets were almost balanced'.
  4. [Section 4.2] The phrase 'thee top 10 values account for 81.6%' contains a typo; it should be 'the top 10 values'.
  5. [Section 5.1] The phrase 'low-consensus dilemmans' should be 'low-consensus dilemmas'.
  6. [Section 3.3] The phrase 'All other models exhibited notably worst alignment' should be 'notably worse alignment'.

Circularity Check

1 steps flagged · score 6.0 of 10

DMP's headline 64.3% alignment gain is an in-sample quantity: the value profiles used to condition model outputs are estimated from all human rationales, including the same dilemmas on which alignment is measured.

  1. fitted input called prediction [Section 5 (RQ3: Model Steering), G0 definition and profile sampling; Section 5.1 results]
    "We define the base measure G0 as the empirical frequency distribution over our 60-value taxonomy, aggregated across all human rationales in the dataset. ... Given the set of topics associated with the dilemmas T = {t1,...,tM}, we fit a topic-specific distribution Gt ∼ Dirichlet(αG0) for each topic t. ... For each dilemma di, associated with topic ti and Ni human judgments, we sample Ni value profiles Pi ... from the multinomial distribution Gti. ... We evaluate the change in alignment due to this approach following the same procedure described in Section 3.1."

    The conditioning profiles for dilemma d_i are sampled from G_{t_i}, and G_{t_i} is fit from G0, the empirical frequency of values in all human rationales in the dataset. Since G0 aggregates over every human rationale, it includes the rationales for d_i itself. The alignment gain is then measured by the Section 3.1 procedure, which compares the LLM judgment distribution to the human judgment distribution on those same d_i dilemmas. Thus the steering signal is derived from the target human evaluations that define the metric; the 64.3% reduction is an in-sample fit rather than a prediction on unseen dilemmas. No held-out split, cross-validation, or leave-one-topic-out evaluation is reported, and the Limitations section does not disclose this leakage.

full rationale

The descriptive parts of the paper are not circular: RQ1 uses independent model elicitation against human judgments without fitting to the target, and RQ2's taxonomy is built from value expressions and then used to measure prevalence, which is a descriptive analysis. No load-bearing self-citation chain or imported uniqueness theorem is present; citations such as Value Kaleidoscope are external tools. The central RQ3 claim, however, is evaluated in-sample: G0 and G_t are fit to all human rationales, including the rationales of the evaluation dilemmas, and DMP's alignment improvement is computed on those same dilemmas. This does not make the 64.3% number a mathematical identity — the model could in principle ignore the injected profiles — but it does mean the headline number cannot be interpreted as a generalizable predictive result, and the comparison to the model council is not apples-to-apples because DMP receives human-derived information from the test set. The lack of any sensitivity analysis on alpha=10 and the absence of any leakage discussion in Limitations reinforce the concern. Score 6 reflects partial, not total, circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or ontological entities are introduced; 'pluralistic moral gap' is a descriptive label, and DMP is a prompting procedure. The central claims rest on the dataset and taxonomy assumptions listed above, plus the additional assumption that in-sample profile fitting is valid for evaluation, which is not met.

free parameters (4)
  • alpha = 10
    Dirichlet concentration parameter in DMP, set to 10 by hand to allow moderate deviation from the global prior; no sensitivity analysis is provided (Section 5).
  • temperature = 1
    Model sampling temperature chosen after comparing T=0 and T=1; reported as superior in Appendix A.2.1. This affects the model distribution and alignment results.
  • profile_size = top-3
    Each sampled value profile is represented by the top-3 most salient values, normalized to sum to 1; this truncation is arbitrary and not varied (Section 5).
  • retention_threshold = 7.2%
    Only posts never flagged as AITA by GPT-4o-mini in three checks are retained; this choice determines dataset composition and may introduce selection bias (Section 2.2).
assumptions (5)
  • domain assumption Value Kaleidoscope accurately extracts moral values from both human and LLM rationales.
    The entire value analysis in Section 4 relies on this LLM-based classifier, adapted to output only values. If it is biased toward certain text styles, the human-model comparison is confounded.
  • domain assumption The 60-value taxonomy is a valid common vocabulary for comparing human and machine moral reasoning.
    Derived via clustering and human refinement (Section 4.1); assumes the clusters are semantically coherent and comparable across human and model language.
  • ad hoc to paper The Dirichlet-multinomial model with alpha=10 captures how topics modulate human value invocation.
    The generative model in Section 5 is assumed to produce realistic human value profiles; alpha is fixed without validation or sensitivity analysis.
  • domain assumption AITA commenters' judgments represent human moral judgment for the purpose of benchmarking LLMs.
    The dataset is built from a single Reddit forum, acknowledged in Limitations as biased toward online, expressive users who share personal experiences.
  • domain assumption GPT-4o-mini rewriting preserves the moral content of the original dilemma.
    The reformulation pipeline (Section 2.2) assumes the abstracted retelling retains the dilemma's moral features after filtering out AITA-specific language.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models." pith.science (2026). https://pith.science/paper/MZDA4Z46

@misc{pith2026250717216,
  author       = {Pith},
  title        = {Pith review of: The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MZDA4Z46}},
  note         = {Machine review of arXiv:2507.17216}
}
read the original abstract

People increasingly rely on Large Language Models (LLMs) for moral advice, which may influence humans' decisions. Yet, little is known about how closely LLMs align with human moral judgments. To address this, we introduce the Moral Dilemma Dataset, a benchmark of 1,618 real-world moral dilemmas paired with a distribution of human moral judgments consisting of a binary evaluation and a free-text rationale. We treat this problem as a pluralistic distributional alignment task, comparing the distributions of LLM and human judgments across dilemmas. We find that models reproduce human judgments only under high consensus; alignment deteriorates sharply when human disagreement increases. In parallel, using a 60-value taxonomy built from 3,783 value expressions extracted from rationales, we show that LLMs rely on a narrower set of moral values than humans. These findings reveal a pluralistic moral gap: a mismatch in both the distribution and diversity of values expressed. To close this gap, we introduce Dynamic Moral Profiling (DMP), a Dirichlet-based sampling method that conditions model outputs on human-derived value profiles. DMP improves alignment by 64.3% and enhances value diversity, offering a step toward more pluralistic and human-aligned moral guidance from LLMs.

Figures

Figures reproduced from arXiv: 2507.17216 by the authors.

Figure 1
Figure 1. Overview Given a Moral Dilemma, we col￾lect Human Moral Evaluations with (i) a moral judg￾ment (i.e., whether the action is acceptable) and (ii) a supporting rationale. We generate an equal number of LLM Moral Evaluations with the same structure. We compare the distribution of human and LLM judgments and moral values (extracted from the rationales) to as￾sess their distributional alignment. limited understanding of … view at source ↗
Figure 2
Figure 2. Model-human alignment across moral dilemmas. (a) Average absolute difference (the lower the better) between models and human judgments across all dilemmas (y-axis) for all baselines tested (x-axis). (b) Average absolute difference stratified by consensus level (x-axis). Models perform well in high-consensus dilemmas but show increasing misalignment as consensus decreases. Persona-based Prompting Given that each mode… view at source ↗
Figure 3
Figure 3. Normalized entropy of value distributions across consensus levels. (a)-(b): We compute the Shannon entropy of values expressed in Human and LLM rationales for each dilemma (y-axis) and group them by levels of human consensus (x-axis). Human responses consistently exhibit higher entropy, reflecting greater diversity in the moral values invoked. The gap between humans and LLMs widens in ambiguous cases, suggesting tha… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Steering LLMs with Dynamic Moral Profiles. (a) Absolute difference in the distribution of moral evaluations between humans and LLMs, stratified by human consensus levels. Dynamic Moral Profiling (DMP) substantially improves alignment, reducing the average divergence by…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 19 canonical work pages

  1. [1]

    Marwa Abdulhai, Gregory Serapio-Garcia, Cl \'e ment Crepy, Daria Valter, John Canny, and Natasha Jaques. 2023. Moral foundations of large language models. arXiv preprint arXiv:2310.15337

    arXiv 2023

  2. [2]

    Divij Bajaj, Yuanyuan Lei, Jonathan Tong, and Ruihong Huang. 2024. Evaluating gender bias of llms in making morality judgements. arXiv preprint arXiv:2410.09992

    arXiv 2024

  3. [3]

    David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993--1022

    2003

  4. [4]

    Tim R Davidson, Viacheslav Surkov, Veniamin Veselovsky, Giuseppe Russo, Robert West, and Caglar Gulcehre. 2024. Self-recognition in language models. arXiv preprint arXiv:2407.06946

    arXiv 2024

  5. [5]

    Danica Dillion, Debanjan Mondal, Niket Tandon, and Kurt Gray. 2025. Ai language model rivals expert ethicist in perceived moral expertise. Scientific Reports, 15(1):4084

    work page 2025

  6. [6]

    Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-D \"u nner. 2024. Questioning the survey responses of large language models. Advances in Neural Information Processing Systems, 37:45850--45878

    2024

  7. [7]

    Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, and 1 others. 2023. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388

    arXiv 2023

  8. [8]

    Ion Stagkos Efstathiadis, Guilherme Paulino-Passos, and Francesca Toni. 2022. Explainable patterns for distinction and prediction of moral judgement on reddit. arXiv preprint arXiv:2201.11155

    work page Pith review arXiv 2022

Show all 48 references
  1. [9]

    Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. 2024. Modular pluralism: Pluralistic alignment via multi-llm collaboration. arXiv preprint arXiv:2406.15951

    2024 arXiv

  2. [10]

    Dean Fulgoni, Jordan Carpenter, Lyle Ungar, and Daniel Preo t iuc-Pietro. 2016. An empirical exploration of moral foundations theory in partisan news sources. In Proceedings of the tenth international conference on Language Resources and Evaluation (LREC'16), pages 3730--3736

    2016

  3. [11]

    Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P Wojcik, and Peter H Ditto. 2013. Moral foundations theory: The pragmatic validity of moral pluralism. In Advances in experimental social psychology, volume 47, pages 55--130. Elsevier

    2013

  4. [12]

    Jesse Graham, Jonathan Haidt, Matt Motyl, Peter Meindl, Carol Iskiwitch, and Marlon Mooijman. 2018. Moral foundations theory: On the advantages of moral pluralism over moral monism

    2018

  5. [13]

    Kunal Handa, Alex Tamkin, Miles McCain, Saffron Huang, Esin Durmus, Sarah Heck, Jared Mueller, Jerry Hong, Stuart Ritchie, Tim Belonax, and 1 others. 2025. Which economic tasks are performed with ai? evidence from millions of claude conversations. arXiv preprint arXiv:2503.04761

    2025 arXiv

  6. [14]

    Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2021. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint arXiv:2111.13654

    2021 arXiv

  7. [15]

    Frederic R Hopp, Jacob T Fisher, Devin Cornell, Richard Huskey, and Ren \'e Weber. 2021. The extended moral foundations dictionary (emfd): Development and applications of a crowd-sourced approach to extracting moral intuitions from text. Behavior research methods, 53:232--246

    2021

  8. [16]

    Saffron Huang, Esin Durmus, Miles McCain, Kunal Handa, Alex Tamkin, Jerry Hong, Michael Stern, Arushi Somani, Xiuruo Zhang, and Deep Ganguli. 2025. Values in the wild: Discovering and analyzing values in real-world language model interactions. arXiv preprint arXiv:2504.15236

    2025 arXiv

  9. [17]

    Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. 2024. Collective constitutional ai: Aligning a language model with public input. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pag...

    2024

  10. [18]

    Yuen J Huo and Ludwin E Molina. 2006. Is pluralism a viable model of diversity? the benefits and limits of subgroup respect. Group Processes & Intergroup Relations, 9(3):359--376

    2006

  11. [19]

    Jianchao Ji, Yutong Chen, Mingyu Jin, Wujiang Xu, Wenyue Hua, and Yongfeng Zhang. 2024. Moralbench: Moral evaluation of llms. arXiv preprint arXiv:2406.04428

    2024 arXiv

  12. [20]

    Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, and 1 others. 2021. Can machines learn morality? the delphi experiment. arXiv preprint arXiv:2110.07574

    2021 arXiv

  13. [21]

    Junseok Kim, Nakyeong Yang, and Kyomin Jung. 2024. Persona is a double-edged sword: Enhancing the zero-shot reasoning by ensembling the role-playing and neutral prompts. arXiv e-prints, pages arXiv--2408

    2024

  14. [22]

    Giuseppe Russo Latona, Manoel Horta Ribeiro, Tim R Davidson, Veniamin Veselovsky, and Robert West. 2024. The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates. arXiv preprint arXiv:2405.02150

    2024 arXiv

  15. [23]

    Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. 2024. Causal estimation of memorisation profiles. arXiv preprint arXiv:2406.04327

    2024 arXiv

  16. [24]

    Xuelin Liu, Pengyuan Liu, and Dong Yu. 2025. What’s the most important value? invp: Investigating the value priorities of llms through decision-making in social scenarios. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4725--4752

    2025

  17. [25]

    Benchmarking distributional alignment of large language models, 2024

    Nicole Meister, Carlos Guestrin, and Tatsunori Hashimoto. Benchmarking distributional alignment of large language models, 2024. URL https://arxiv. org/abs/2411.05403

    2024 arXiv

  18. [26]

    Jared Moore, Tanvi Deshpande, and Diyi Yang. 2024. Are large language models consistent over value-laden questions? arXiv preprint arXiv:2407.02996

    2024 arXiv

  19. [27]

    Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, and 1 others. 2023. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Comp...

    2023

  20. [28]

    Vjosa Preniqi, Iacopo Ghinassi, Julia Ive, Charalampos Saitis, and Kyriaki Kalimeri. 2024. Moralbert: A fine-tuned language model for capturing moral values in social discussions. In Proceedings of the 2024 International Conference on Information Technology for Social Good, pa...

    2024

  21. [29]

    o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Sch \

    Paul R \"o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Sch \"u tze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. arXiv preprint arXiv:2402.16786

    2024 arXiv

  22. [30]

    Giuseppe Russo, Christoph Gote, Laurence Brandenberger, Sophia Schlosser, and Frank Schweitzer. 2022. Disentangling active and passive cosponsorship in the us congress. arXiv preprint arXiv:2205.09674

    2022 arXiv

  23. [31]

    Giuseppe Russo, Maciej Styczen, Manoel Horta Ribeiro, and Robert West. 2025. Does content moderation lead users away from fringe movements? evidence from a recovery community. In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, pages 1719--1734

    2025

  24. [32]

    Giuseppe Russo, Luca Verginer, Manoel Horta Ribeiro, and Giona Casiraghi. 2023. Spillover of antisocial behavior from fringe platforms: The unintended consequences of community banning. In Proceedings of the international AAAI conference on web and social media, volume 17, pag...

    2023

  25. [33]

    Pratik S Sachdeva and Tom van Nuenen. 2025. Normative evaluation of large language models with everyday moral dilemmas. arXiv preprint arXiv:2501.18081

    2025 arXiv

  26. [34]

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971--30004. PMLR

    2023

  27. [35]

    Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. 2023. Evaluating the moral beliefs encoded in llms. Advances in Neural Information Processing Systems, 36:51778--51809

    2023

  28. [36]

    Taylor Sorensen, Liwei Jiang, Jena D Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, and 1 others. 2024 a . Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties. In Proceedings of the AAA...

    2024

  29. [37]

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, and 1 others. 2024 b . A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070

    2024 arXiv

  30. [38]

    Localizing paragraph memorization in language models, 2024

    Niklas Stoehr, Mitchell Gordon, Chiyuan Zhang, and Owen Lewis. Localizing paragraph memorization in language models, 2024. URL https://arxiv. org/abs/2403.19851, page 28

    2024 arXiv

  31. [39]

    Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, Ankur Rathi, Saffron Huang, Alfred Mountfield, Jerry Hong, Stuart Ritchie, and 1 others. 2024. Clio: Privacy-preserving insights into real-world ai use. arXiv preprint arXiv:2412.13678

    2024 arXiv

  32. [40]

    Philip E Tetlock. 1986. A value pluralism model of ideological reasoning. Journal of personality and social psychology, 50(4):819

    1986

  33. [41]

    Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. Do llms exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguistics, 12:1011--1026

    2024

  34. [42]

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796

    2024 arXiv

  35. [43]

    Anvesh Rao Vijjini, Rakesh R Menon, Jiayi Fu, Shashank Srivastava, and Snigdha Chaturvedi. 2024. Socialgaze: Improving the integration of human social norms in large language models. arXiv preprint arXiv:2410.08698

    2024 arXiv

  36. [44]

    Isaac Waller and Ashton Anderson. 2021. Quantifying social organization and political polarization in online platforms. Nature, 600(7888):264--268

    2021

  37. [45]

    Justin Zhao, Flor Miriam Plaza-del Arco, Benjamin Genchel, and Amanda Cercas Curry. 2024 a . Language model council: Democratically benchmarking foundation models on highly subjective tasks. arXiv preprint arXiv:2406.08598

    2024 arXiv

  38. [46]

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024 b . Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470

    2024 arXiv

  39. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.