Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

PrefPalette: Personalized Preference Modeling with Latent Attributes

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PrefPalette decomposes preference into 19 latent attributes, beating GPT-4o by 46.6% in prediction accuracy and exposing community values.

desk verdict Solid empirical paper with a genuinely new counterfactual synthesis pipeline, but the load-bearing attribute-isolation assumption is asserted, not measured, so the interpretability claims need hard evidence. read the letter →

arxiv 2507.13541 v1 pith:A5LES7BG submitted 2025-07-17 cs.AI

Shuyue Stella Li , Melanie Sclar , Hunter Lang , Ansong Ni , Jacqueline He , Puxin Xu , Andrew Cohen , Chan Young Park
show 2 more authors
Yulia Tsvetkov Asli Celikyilmaz
This is my paper
classification cs.AI
keywords preferencemodelingpersonalizationmulti-attributedecisionmakingcounterfactualdatasynthesisknowledgedistillationattentionweightscommunitynormsinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PrefPalette sets out to show that human preference judgments are attribute-mediated rather than black-box: people judge content by weighing interpretable dimensions such as formality, humor, sarcasm, and cultural values, and different social communities weigh these dimensions differently. The paper builds a model that first learns to recognize 19 such attributes from counterfactually synthesized training pairs, then learns context-dependent importance weights over those attributes through an attention mechanism. It reports that this approach predicts community-approved comments in 45 Reddit communities at 84.9% average accuracy, beating GPT-4o as a judge by 46.6% and the strongest trained baseline by 1.4%. The payoff would be double: more accurate preference prediction, and a transparent per-community profile of what values drive approval, such as verbosity in r/AskHistorians and sarcasm in r/MaliciousCompliance.

What carries the argument

The mechanism that carries the argument is counterfactual attribute synthesis followed by contrastive attribute distillation, feeding into attention-based attribute integration. For each response, a strong generator rewrites it at five intensity levels along one attribute while attempting to hold semantic meaning and the other 18 attributes fixed; the resulting pairs train specialized attribute predictors through a pairwise ranking objective. At preference time, the 19 predictor hidden states are combined with the content hidden state through an attention layer, producing per-example importance weights, and the weighted sum is added to the content representation before the final score. A gradual feature reduction stage randomly drops the attribute branch with increasing probability during training so that the final model can be deployed without the attribute predictors, forcing the preference model to internalize attribute-informed patterns.

What would settle it

Run all 19 trained attribute predictors on both members of a held-out set of generated counterfactual pairs and measure how often the non-target attribute predictors change by more than a small threshold. If a formality pair also moves the humor or politeness scores, the attribute predictors are confounded, and the attention weights cannot be read as isolating the intended cognitive dimensions. A second, cheaper check is to retrain PrefPalette with the attribute-branch inputs randomly shuffled; if accuracy does not drop, the reported gains are not attribute-mediated.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that inserting latent attribute representations between content and preference changes both accuracy and interpretability. PrefPalette defines 19 attributes—nine sociolinguistic norms and ten cultural values—and trains one small predictor per attribute using contrastive pairs generated by a much larger teacher model, so that each pair is intended to differ only along the target attribute. The preference model takes the hidden states of these predictors, computes an attention distribution over them conditioned on the instruction-response pair, and adds the weighted attribute state to the content representation before scoring. On the 45 subreddit test domains the paper measures an average of 84.9% preference accuracy, a 1.4% gain over the strongest trained baseline and a 46.6% gain over prompting GPT-4o directly. The same learned weights align with known community norms and with human-rated correlations, which the paper takes as evidence that the attribute structure is not just a performance trick.

Load-bearing premise

The load-bearing premise is that the large teacher model can vary one attribute at a time in a response without changing the other 18 attributes or the core meaning; the paper asserts this isolation but does not measure it.

Editorial extensions

If this is right

  • Attribute-mediated preference modeling reaches 84.9% average accuracy across 45 Reddit communities, 1.4% above the strongest trained baseline and 46.6% above GPT-4o prompted as a judge.
  • Per-community attention weights produce concrete normative profiles, such as verbosity and stimulation in r/AskHistorians, sarcasm and directness in r/MaliciousCompliance, and empathy and supportiveness in support-oriented communities.
  • On temporally shifted test data from January 2023, the attribute-mediated model degrades less (69.3% accuracy) than the trained baseline that lacks attributes (56.2%), indicating that attribute structure helps out-of-distribution prediction.
  • Gradual feature reduction removes attribute predictors at inference, so the gains come without extra per-query model cost, and the weights themselves can be read as a transparent explanation for each prediction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the pipeline is attribute-agnostic, the same counterfactual-distillation-and-attention recipe could be applied to other latent dimensions, such as technical accuracy or factual correctness, on other platforms or languages.
  • If the generated pairs do not truly isolate a single attribute, the attention weights would reflect correlated dimensions rather than the intended cognitive ones; retraining with shuffled attribute labels is a quick check of whether the 1.4% gain is actually attribute-mediated.
  • The paper models community-level norms, so a natural extension is individual-level personalization, where personal history and community membership both shape the attribute weights; that extension would inherit the same isolation assumption.
  • The learned attribute weights could serve as input features for content moderation or for tracking how community norms drift over time, since they provide a compact, interpretable description of what a community rewards.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces PrefPalette, a two-stage framework for preference modeling: (1) counterfactual attribute synthesis with Llama 3 405B generates pairs of responses that are claimed to differ only along each of 19 attribute dimensions (9 sociolinguistic norms and 10 Schwartz values), and small Llama 3 1B models are contrastively trained on these pairs to serve as attribute predictors; (2) an attention-based preference model learns community-specific weights over the attribute hidden states, with gradual feature reduction to allow inference without attribute predictors. The model is trained and evaluated on Reddit upvote data across 45 subreddits. The main results are a 1.4 percentage-point average accuracy gain over ValueScope (84.9% vs 83.7%), a 46.6% gain over zero-shot GPT-4o, and qualitative community-attribute profiles (e.g., r/AskHistorians values verbosity, r/MaliciousCompliance values sarcasm). A small human validation in Appendix D reports correlations consistent with the model's high- and low-importance attribute predictions.

Significance. The framework is well-motivated and the controlled comparison to ValueScope is a credible experimental setup: the 1.4% gain is measured with standard deviations across seeds on real upvote data, and the authors provide a public code link. If the attribute isolation assumption holds, the attention-weight analysis would be a useful interpretability tool for community-specific norms. The main risk is that the interpretability claims rest on an unverified premise about the synthetic counterfactuals, and the paper also contains an overclaimed headline comparison and an unsupported temporal-robustness claim. With revisions to address the attribute-validation gap and to reposition the claims, this could become a solid contribution.

major comments (4)
  1. [Section 2.1 / Table 4] The paper's central interpretability claim depends on Eq. (1)'s condition that A_k(y_{a,l}) ≈ A_k(y) for all k ≠ a, and on the statement that 'the pairwise differences only exist along the target attribute dimension.' This isolation is asserted but never measured. The attribute predictor accuracies in Table 5 (98.4–100%) are computed on held-out pairs from the same generation process, so they only measure distillation fidelity to the teacher's synthetic labels, not whether the labels correspond to the intended attribute dimensions. Strikingly, the illustrative example in Table 4 shows the 'supportiveness' counterfactuals also varying in politeness, empathy, and emotional intensity, which is exactly the confounding the method claims to avoid. Since the attention weights in Table 2 and the community-norm interpretations rest on this isolation, the interpretability contribution is not yet established. Please provide an independent check: human or separate-model ratings of non-target attributes on the generated pairs, and/or an out-of-distribution evaluation of the attribute predictors on naturally occurring Reddit comments with human annotations. Without this, the attribute representations and attention weights may reflect a confounded mixture of attributes rather than distinct cognitive dimensions.
  2. [Table 1 / Abstract] The headline 'outperforms GPT-4o by 46.6%' is misleading as a scientific claim. The GPT-4o baseline is used zero-shot without any training on the Reddit preference distribution, while PrefPalette is trained on millions of preference pairs from the same domains. The fair, controlled comparison is against ValueScope, where the gain is 1.4 percentage points (84.9% vs 83.7%). The abstract and introduction should lead with the controlled comparison and clearly position GPT-4o as an untrained reference point rather than a state-of-the-art competitor.
  3. [Section 4.4 / Table 1 last row] Section 4.4 claims that 'attribute-mediated preference modeling exhibits stronger distributional robustness than baseline approaches.' The numbers in the last row of Table 1 do not support this: PrefPalette degrades by 15.6% (84.9 to 69.3), nearly identical to ValueScope's 15.7% and PrefPalette-Score's 16.0%. PrefPalette has higher absolute accuracy on the temporally shifted test set, but it does not degrade more gracefully than the baselines. Please revise the robustness claim to reflect the actual finding of matched relative degradation.
  4. [Section 3 / Appendix B] The paper states that 'we randomly sample 45 domains for evaluation,' but Appendix B describes only a within-domain split of 3% of posts for validation and test. It is not stated whether the preference model is trained on all 680 subreddits (including the 45 evaluated ones) or only on the remaining 635. If the model is trained on the same 45 subreddits with held-out posts, the main results are in-domain per-community preference modeling rather than cross-domain generalization. Please state the exact train/test split and discuss the implications for the 'personalization' claim.
minor comments (6)
  1. [Table 2] In the bottom section, 'Powser' should be corrected to 'Power'.
  2. [Table 1 caption / Appendix A] The caption says 'after training for three epochs,' while Appendix A says training runs for 100,000 steps; please reconcile these numbers.
  3. [Section 2.2 / Appendix A] The dropout notation is inconsistent: the main text uses β ∼ Bernoulli(1−λ) while Eq. (5)-(6) in the appendix use λ as both a dropout probability and a scaling factor for the attribute representation. Please use distinct symbols for the dropout probability and the learned scaling parameter γ.
  4. [Section 3 / Appendix B] The main text says 'we sample up to 100,000 preference pairs per domain,' but Appendix B says 'extract 10,000 comment pairs' per subreddit, yielding approximately 6.8M pairs total. Please correct the number or clarify the discrepancy.
  5. [Section 2.2 / Section 4.2] The paper states that attribute predictors are 'eliminated at inference time,' yet the interpretability results in Section 4.2 report per-example attention weights. Clarify whether attribute predictors are still run during evaluation solely to compute the attention weights α_i, even though they are not needed for the preference prediction itself.
  6. [Section 3 / Baselines] The description of the GPT-4o-as-a-judge baseline does not specify whether the prompt includes the subreddit context or the upvote-based preference definition; please add the prompt details for reproducibility.

Circularity Check

2 steps flagged · score 5.0 of 10

Attribute 'ground truth' is defined by the generator itself, and predictor validation measures self-consistency, making the interpretability layer partially circular; the real-upvote preference benchmark remains independent.

  1. self definitional [Section 2.1, Counterfactual Attribute Synthesis]
    "By explicitly controlling for confounding variables and enabling a systematic coverage of the full attribute space—including rare attributes underrepresented in common interactions—our approach establishes the ground truth through the generation process itself. For any counterfactual response pair (ya,l1 , ya,l2 )where l1 < l2, note that ya,l2 exhibits a higher level of attributea than ya,l1, establishing a scalable learning signal along the target attribute dimension. Our novel generative approach ensures that the pairwise differences only exist along the target attribute dimension."

    The attribute labels A_a(y) are not anchored to any external measurement; the paper states that the ground truth is established by the generation process itself. The contrastive predictor is trained with Lattr = -log σ(r_a(y_{a,l2}) - r_a(y_{a,l1})) on these self-labeled pairs, so the predictor learns the generator's counterfactual levels rather than independently verified attribute intensities. The assertion that pairwise differences exist only along the target attribute is assumed, not measured. Since the attention weights in Section 2.2 are computed from these predictors, the interpretable community profiles in Table 2 and Figure 4 inherit the generator's assumption rather than an independently verified attribute decomposition.

  2. fitted input called prediction [Appendix C, Universal Attribute Predictor Training Details; Table 5]
    "We evaluate our universal attribute predictors on held-out test sets of counterfactual comment and post pairs to verify their ability to distinguish between different attribute intensity levels. The high accuracy scores across all 19 attributes (ranging from 98.4% to 100%) in Table 5 demonstrate that our contrastive attribute distillation approach successfully transfers attribute understanding from the strong teacher model to the smaller specialized predictors."

    The held-out test pairs are produced by the same counterfactual synthesis that created the training pairs, and the 'attribute intensity levels' are assigned by construction rather than by human annotation or by measurement of non-target attributes. The near-perfect accuracy therefore measures how well the distilled predictor replicates the teacher's own labels—distillation fidelity to the generator's definition—not whether the predictors isolate the intended latent attributes. The paper then treats this accuracy as evidence that the predictors 'reliably extract latent attribute information,' which is the load-bearing premise for the attention-weight interpretations in Section 4.2.

full rationale

The paper's headline preference-accuracy result is not circular: PrefPalette is trained on real Reddit upvote pairs, and the 84.9% average versus ValueScope's 83.7% (Table 1) is an external benchmark, so the core predictive claim stands or falls independently of the attribute-layer circularity. The circularity lies in the attribute representation and interpretability chain. Section 2.1 states that 'our approach establishes the ground truth through the generation process itself,' so the 19 attribute predictors are trained and then evaluated on labels produced by the same Llama 3 405B counterfactual generator. Table 5's 98.4-100% accuracies therefore measure self-consistency (the student reproduces the teacher's labels), not whether the labels correspond to the intended cognitive dimensions or whether non-target attributes were held fixed. Because the attention weights in Section 2.2 and the community profiles in Table 2 and Figure 4 are computed from these predictor hidden states, the interpretability results inherit the generator's assumption. The paper itself concedes in Section 7: 'The latent attribute dimensions learned by PrefPalette are not directly validated via human annotations.' The Appendix D human study covers only two high- and two low-importance attributes in r/AskHistorians, with modest correlations (e.g., verbosity r=0.12, power r=-0.06), and it does not test whether non-target attributes were held fixed in the training pairs. Thus the attribute layer is partially self-definitional and the interpretability claims are partly circular, but the independent upvote-based benchmark and the small human study prevent the whole paper from collapsing into pure circularity.

Assumptions & free parameters 4 free parameters · 6 assumptions · 1 invented entities

The central claim rests on the hand-selected attribute taxonomy, the synthetic generation procedure that defines attribute ground truth, and the assumption that attention weights capture real community values. The preference accuracy itself is evaluated on real upvotes, but the interpretability layer is heavily dependent on the synthetic teacher and is only weakly validated externally.

free parameters (4)
  • Attribute taxonomy (19 dimensions) = 9 sociolinguistic norms + 10 Schwartz values
    The choice of which attributes to model is a hand-selected set that defines the entire attribute space; the paper does not justify why these 19 are sufficient or why other attributes are excluded.
  • Counterfactual Likert levels = 5 levels (extremely low to extremely high)
    The granularity of attribute intensity used to synthesize training pairs is chosen without sensitivity analysis.
  • Dropout cap and schedule = 0.8 cap, linear schedule over T steps
    Hyperparameters for gradual feature reduction; no ablation is reported to show sensitivity to these values.
  • Gamma initial value = 0.5
    Initial weight for combining attribute and content features; it is learned, but the initial value is hand-picked.
assumptions (6)
  • domain assumption Counterfactual synthesis varies exactly one attribute while preserving all others and semantic content
    Load-bearing assumption that attribute predictors learn isolated dimensions; asserted in Section 2.1 but never measured.
  • domain assumption Upvotes and downvotes reflect community preference
    The evaluation label is net votes; the paper acknowledges in the Ethics Statement that upvotes can reflect popularity rather than normative desirability.
  • ad hoc to paper The 19 attributes are sufficient and appropriately characterize multi-attribute decision making
    The set is grounded in sociolinguistics and Schwartz values, but sufficiency is not justified; the paper admits it is not exhaustive.
  • domain assumption Llama 3 405B has correct and consistent attribute intensity understanding
    The teacher model's counterfactual generations are taken as ground truth for attribute intensity; no validation against human ratings for most attributes.
  • domain assumption Attention weights correspond to attribute importance
    The interpretability claim rests on this mapping, validated only weakly on a few attributes in one domain (Appendix D).
  • domain assumption Gradual feature reduction lets the model internalize attributes for inference without them
    The dropout training is intended to embed attribute patterns into weights, but no analysis shows this succeeds beyond the main accuracy gain.
invented entities (1)
  • 19 latent attribute dimensions (e.g., supportiveness, sarcasm, benevolence)
    purpose: Intermediate representation between content and preference predictions; the mechanism by which PrefPalette decomposes and reweights preferences.
    The attributes are operationalized through synthetic data from a teacher model and are not independently measured; only a few were spot-checked via human annotation in one community (r/AskHistorians), with weak correlations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PrefPalette: Personalized Preference Modeling with Latent Attributes." pith.science (2026). https://pith.science/paper/A5LES7BG

@misc{pith2026250713541,
  author       = {Pith},
  title        = {Pith review of: PrefPalette: Personalized Preference Modeling with Latent Attributes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A5LES7BG}},
  note         = {Machine review of arXiv:2507.13541}
}
read the original abstract

Personalizing AI systems requires understanding not just what users prefer, but the reasons that underlie those preferences - yet current preference models typically treat human judgment as a black box. We introduce PrefPalette, a framework that decomposes preferences into attribute dimensions and tailors its preference prediction to distinct social community values in a human-interpretable manner. PrefPalette operationalizes a cognitive science principle known as multi-attribute decision making in two ways: (1) a scalable counterfactual attribute synthesis step that involves generating synthetic training data to isolate for individual attribute effects (e.g., formality, humor, cultural values), and (2) attention-based preference modeling that learns how different social communities dynamically weight these attributes. This approach moves beyond aggregate preference modeling to capture the diverse evaluation frameworks that drive human judgment. When evaluated on 45 social communities from the online platform Reddit, PrefPalette outperforms GPT-4o by 46.6% in average prediction accuracy. Beyond raw predictive improvements, PrefPalette also shed light on intuitive, community-specific profiles: scholarly communities prioritize verbosity and stimulation, conflict-oriented communities value sarcasm and directness, and support-based communities emphasize empathy. By modeling the attribute-mediated structure of human judgment, PrefPalette delivers both superior preference modeling and transparent, interpretable insights, and serves as a first step toward more trustworthy, value-aware personalized applications.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Interactive Task Alignment as a POMDP

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Under ambiguous user requests, current LLMs recover the intended task only 22–32% of the time, well below human accuracy of 48%, and post-training only partially closes the gap.

Reference graph

Works this paper leans on

19 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [13]

    Personalizing reinforcement learning from human feedback with variational preference learning

    Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.https://openreview.net/forum?id=gRG6SzbW9p. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefan...

    arXiv 2024

  2. [17]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    arXiv

  3. [18]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 10582–10592, 2024a. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Nar...

    arXiv 2024

  4. [19]

    with associated metadata including upvote counts and timestamps, and set apart 3% of posts each for the validation and test sets. To prepare source data for each attribute in the attribute representation learning module, we randomly sample 100 comments from each subreddit and generate counterfactual comments along 5 Likert scale levels, resulting in680× 1...

    work page 2020

  5. [1952]

    Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob...

    arXiv 2023

  6. [1986]

    Dialogue response ranking training with large-scale human feedback data

    11 Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. Dialogue response ranking training with large-scale human feedback data. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 386–395,

    work page 2020

  7. [1995]

    Balancing the picture: Debiasing vision-language datasets with synthetic contrast sets.arXiv preprint arXiv:2305.15407,

    Brandon Smith, Miguel Farinha, Siobhan Mackenzie Hall, Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain. Balancing the picture: Debiasing vision-language datasets with synthetic contrast sets.arXiv preprint arXiv:2305.15407,

    arXiv

  8. [1998]

    Language (technology) is power: A critical survey of" bias" in nlp.arXiv preprint arXiv:2005.14050,

    Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of" bias" in nlp.arXiv preprint arXiv:2005.14050,

    arXiv 2005

Show all 19 references
  1. [1999]

    Predictive biases in natural language processing models: A conceptual framework and overview.arXiv preprint arXiv:1912.11078,

    Deven Shah, H Andrew Schwartz, and Dirk Hovy. Predictive biases in natural language processing models: A conceptual framework and overview.arXiv preprint arXiv:1912.11078,

    1912 arXiv

  2. [2003]

    Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023.https://arxiv.org/abs/2312.06674

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023.https://arxiv.org/abs/2312.06674....

    2023 arXiv

  3. [2005]

    nobody sees it, nobody gets mad

    Shuyue Stella Li, Jimin Mun, Faeze Brahman, Jonathan S. Ilgen, Yulia Tsvetkov, and Maarten Sap. Aligning llms to ask good questions a case study in clinical reasoning, 2025a.https://arxiv.org/abs/2502.14860. Xinyu Li, Ruiyang Zhou, Zachary Chase Lipton, and Liu Leqi. Personali...

    arXiv

  4. [2011]

    Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessandro Allievi

    W. Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessandro Allievi. Models of human preference for learning reward functions, 2023.https://arxiv.org/abs/2206.02231. Sachin Kumar, Chan Young Park, Yulia Tsvetkov, Noah A. Smith, and Hannane...

    2023 arXiv

  5. [2017]

    When people are floods: Analyzing dehumanizing metaphors in immigration discourse with large language models.arXiv preprint arXiv:2502.13246,

    Julia Mendelsohn and Ceren Budak. When people are floods: Analyzing dehumanizing metaphors in immigration discourse with large language models.arXiv preprint arXiv:2502.13246,

  6. [2018]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Yuntao Bai, Andy Jones, Kinden Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    arXiv

  7. [2019]

    Fspo: Few-shot preference optimization of synthetic preference data in llms elicits effective personalization to real users, 2025.https://arxiv.org/abs/2502.19312

    Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, and Chelsea Finn. Fspo: Few-shot preference optimization of synthetic preference data in llms elicits effective personalization to real users, 2025.https://arxiv.org/abs/2502...

    2025 arXiv

  8. [2020]

    Bradley Knox, Serena Booth, Scott Niekum, and Peter Stone

    Stephane Hatgis-Kessell, W. Bradley Knox, Serena Booth, Scott Niekum, and Peter Stone. Influencing humans to conform to preference models for rlhf, 2025.https://arxiv.org/abs/2501.06416. Juan Manuel Hernández-Campoy.Sociolinguistic styles. John Wiley & Sons,

    2025 arXiv

  9. [2021]

    Jon Doyle

    doi: 10.1073/pnas.2023301118.https://www.pnas.org/doi/abs/10.1073/pnas.2023301118. Jon Doyle. Prospects for preferences.Computational Intelligence, 20(2):111–136,

    doi

  10. [2022]

    Valuescope: Unveiling implicit norms and values via return potential model of social interactions, 2024a

    Chan Young Park, Shuyue Stella Li, Hayoung Jung, Svitlana Volkova, Tanushree Mitra, David Jurgens, and Yulia Tsvetkov. Valuescope: Unveiling implicit norms and values via return potential model of social interactions, 2024a. https://arxiv.org/abs/2407.02472. Chan Young Park, S...

    arXiv

  11. [2025]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Pai...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.