Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Value Drifts: Tracing Value Alignment During LLM Post-Training

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A model's expressed values are largely set during supervised fine-tuning, and subsequent preference optimization rarely re-aligns them unless the preference data carries an explicit value contrast. This claim is established by tracing stanc

desk verdict A well-documented empirical study showing SFT sets value priors and standard preference data barely moves them; the headline claim is a bit overstated, but the core result is legitimate and should be refereed. read the letter →

arxiv 2510.26707 v2 pith:BDKYUUXF submitted 2025-10-30 cs.CL cs.CYcs.LG

Mehar Bhatia , Shravan Nayak , Gaurav Kamath , Marius Mosbach , Karolina Stańczak , Vered Shwartz , Siva Reddy This is my paper
classification cs.CLcs.CYcs.LG
keywords valuealignmentpost-trainingdynamicssupervisedfine-tuningpreferenceoptimizationdriftLLMstancesUltraFeedbackHH-RLHF
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that, in the standard two-stage post-training pipeline, the supervised fine-tuning (SFT) phase is the dominant force shaping an LLM's values, and that preference optimization (PPO, DPO, SimPO) only weakly corrects those values when trained on popular preference datasets. It argues that these datasets fail to re-align values because the chosen and rejected responses in each preference pair exhibit nearly identical stance distributions, a property the paper calls a small value-gap. Using a synthetic preference dataset with a deliberately large value-gap, the paper shows that preference optimization algorithms can re-align values, but in different ways: PPO preserves the SFT values, DPO amplifies stances already favored by the SFT prior, and SimPO produces smaller, slower drifts. If correct, these findings give practitioners a concrete rule: to change a model's values, the strongest lever is the SFT corpus, not the subsequent preference optimization stage.

What carries the argument

The central machinery is the operationalization of values as stance distributions: a model's value on a topic is defined as the expected probability of support, neutral, or oppose stances across generated responses, measured by GPT-4o classification of responses to a curated prompt set (V-PRISM). Two metrics, drift magnitude and drift time, quantify how much and how quickly a model's stance distribution changes between checkpoints. To explain why preference optimization often has little effect, the paper introduces the value-gap: the Euclidean distance between the stance distributions of the chosen and rejected responses in a preference pair. The synthetic preference dataset, built by genera

What would settle it

Re-run the value-drift measurements using a stance classifier that separates style from content (for example, asking annotators to judge the underlying position even when the response is hedged, or using a continuous stance scale rather than the three-way neutral-inclusive scheme). If, on standard preference datasets like UltraFeedback and HH-RLHF, the chosen and rejected responses then show large value gaps, and if PPO or DPO on those datasets produces substantial stance shifts, the paper's central claim that SFT fixes values and preference optimization rarely moves them would be refuted.

Watch

Extended reading notes

Core claim

The paper's central discovery is that value formation in LLM post-training is largely completed during supervised fine-tuning. By sampling checkpoints throughout training and measuring stance distributions over a curated set of value-laden prompts (the V-PRISM set), the authors observe that SFT rapidly and strongly shifts a model's stances toward the value distribution of the instruction-tuning data, and that subsequent preference optimization with standard datasets (UltraFeedback and HH-RLHF) barely moves those stances. The paper attributes this to a small value-gap in standard preference data: chosen and rejected response pairs show nearly identical stance distributions, providing a weak t

Load-bearing premise

The central measurement assumption is that a GPT-4o three-way stance label (support/neutral/oppose), averaged over five generations, captures a model's latent values independently of response style; if the judge's 'neutral' category mostly absorbs hedged or refusal-style text, then the small value-gap and the drift-to-neutral results could be partly artifacts of the measurement rather than genuine value properties.

Editorial extensions

If this is right

  • Data curation: to steer a model's values, the SFT corpus is the primary control; preference data without explicit stance contrast will not correct the SFT-imposed value profile.
  • Algorithm selection: if SFT has already produced the desired values, PPO is a safe choice because it preserves those values; DPO is preferable when the goal is to amplify a value the model already leans toward.
  • Preference data design: a small value-gap in preference pairs is a measurable diagnosis for why a preference optimization run may fail to change model values.
  • Evaluation practice: checkpoint-level value drift during SFT can be used for early attribution of a model's eventual stances, enabling detection of value misalignment before the final model is deployed.
  • Hyperparameter effects: the KL coefficient in PPO and the β in DPO control the strength of value drift, with smaller coefficients allowing more drift, giving practitioners a tuning lever for value alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: the small value-gap in standard preference datasets may partly be a measurement artifact, since the GPT-4o judge's neutral category absorbs hedged, refusal-like responses; re-measuring with a style-controlled judge could reveal larger value contrasts than the paper reports.
  • The paper's finding implies that SFT is not merely teaching instruction-following but is the de facto value-alignment stage; this reframes the role of the SFT dataset in any safety or pluralism effort, suggesting that SFT data should be audited for value biases directly.
  • Looking beyond the paper, the same measurement framework could be applied to pretraining checkpoints to test whether SFT is imprinting new values or simply exposing latent stances already acquired during pretraining; if the latter, value alignment begins even earlier than this study shows.
  • The algorithm-specific divergence on synthetic data suggests a practical recipe for controlled value editing: combine a large-gap preference dataset with DPO to amplify a chosen stance, or with a small KL penalty in PPO to override the SFT prior, and validate using the paper's drift metrics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces a methodology (V-PRISM, drift magnitude, drift time) for tracing how LLM stances on value-laden topics change during post-training. Using Llama-3 and Qwen-3 models, SFT on WildChat/Alpaca, and PPO/DPO/SIMPO on UltraFeedback/HH-RLHF, the authors report three findings: (1) SFT rapidly establishes the model's value profile; (2) standard preference optimization induces little additional value drift, which they attribute to a small value-gap between chosen and rejected responses; and (3) a synthetic preference dataset with a deliberately large value-gap reveals algorithm-specific behavior (PPO preserves SFT values, DPO amplifies SFT-consistent stances, SIMPO gives smaller/slower drifts). The paper includes extensive appendices with per-topic tables, hyperparameter ablations, and downstream-task checks.

Significance. If the measurement is valid, the central claim—that SFT is the dominant locus of value formation and preference optimization is a weak corrector under typical data—is an important, actionable result for alignment research and data curation. The paper's strengths are its controlled synthetic-preference manipulation, transparent methodology with detailed appendices (Tables 9–16), explicit hyperparameter ablations (§J), and the stated plan to release code, checkpoints, and data. The main risk is that the stance classifier may conflate response style with latent value, which would undermine the empirical basis for the central claim.

major comments (3)
  1. [§2.1, Eq. (1); §D; §K.3] The central measurement v_θ(T) is based on GPT-4o three-way stance labels with 'neutral' defined as 'neither supports nor opposes, or avoids expressing a stance.' Appendix D (Tables 5–7) shows that WildChat-SFT models produce long, hedged, refusal-like responses that the judge likely maps to neutral. Thus the large SFT drift toward neutral (e.g., M_neutral = 0.38–0.76 in Tables 9–12) and the apparent stability during PO may be style imprinting, not latent value change. The same confound affects the value-gap analysis in §K.3: if most responses collapse to neutral, the near-zero Euclidean distances may be a floor effect. The Ethics Statement acknowledges semantic flattening but not this style confound. Please provide robustness evidence—e.g., a forced-choice or style-controlled annotation, or conditioning the value-gap analysis on response length/hedging—to show that the low value-gap and
  2. [§5.2, Table 1] The claim of 'minimal to no value drift' overstates the paper's own numbers. Table 1 reports SIMPO climate-change support drift magnitude 0.27 and PPO 0.20; these are comparable to many SFT drift magnitudes in Tables 9–12. The drift-time value for SIMPO climate-change oppose is 0.84, which contradicts the statement that 'drift time is also low.' The conclusion should be re-calibrated to 'small but non-negligible drifts, with magnitudes varying by algorithm and topic' rather than 'minimal to no drift,' unless a quantitative threshold for 'minimal' is specified.
  3. [§6.1, Table 4] Table 4 lists the same value for the 'Value (Support)' and 'Value (Oppose)' columns for Climate Change Opinions ('support climate change action' in both). This is either a typo or a serious error in the synthetic data generation: the oppose-aligned condition for climate change would not actually oppose the value. Since Tables 13–16 report full results across all topics, this affects the validity of the climate-change rows in the synthetic experiment and needs to be corrected and the affected results re-examined.
minor comments (4)
  1. [§3, drift time] The definition of drift time uses the 95% confidence interval of the extremum, but it is unclear how the extremum is determined when the trajectory is noisy or non-monotonic. Please clarify the estimation procedure and any smoothing.
  2. [§K.1] The cosine-similarity threshold of 0.5 used to retrieve dataset datapoints is not motivated. A brief sensitivity analysis (e.g., thresholds 0.4/0.6) would strengthen the value-gap conclusion.
  3. [Appendix K, Fig. 17] The caption for Fig. 17 is incomplete: it ends with 'On the other hand,' and the sentence is cut off. Please complete the caption.
  4. [Table 4] Aside from the climate-change row, consider adding a note explaining how the value mapping was validated (e.g., the manual verification of 100 pairs described in §6.1) to preempt concerns about prompt adherence.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; claims are empirical measurements with disclosed controls.

full rationale

The paper's central claims—SFT establishes a model's value profile and preference optimization rarely re-aligns it—are empirical findings about measured stance distributions (Eq. 1), not consequences of how those distributions are defined. v_theta(T) is computed from GPT-4o stance labels on model generations (§3), independently of the SFT/PO objectives, so no 'prediction' is equivalent to a fitted input. The value-gap analysis (§K.3) measures a property of preference datasets with the same judge; even if the judge's neutral bucket creates a floor effect (a validity concern, acknowledged in the Ethics Statement as stance proxy oversimplification), it is not a construction that forces the drift results. The synthetic experiment (§6.1) is a controlled manipulation: support/oppose responses are generated by an external model and labels are flipped; the observed algorithm differences (PPO preserving SFT values, DPO amplifying SFT-consistent stances, SIMPO moderate drifts) are not entailed by the dataset construction and are supported by hyperparameter ablations (§J). The one self-citation in the introduction (Stańczak et al., 2025) is a framing citation for the importance of human-value alignment and is not load-bearing; no uniqueness theorem, ansatz, or known-result renaming is used. Consequently there is no circular step by the paper's own equations or citation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or architectural entities are postulated. Free parameters are confined to measurement design choices (retrieval threshold, checkpoint cadence, judge configuration). The axioms are measurement and generalization assumptions typical of an empirical LLM evaluation study; the most fragile is the GPT-4o-stance operationalization, which the paper itself acknowledges in the Ethics Statement only at the semantic level, not the style-confound level.

free parameters (3)
  • Dataset retrieval cosine similarity threshold = 0.5
    §K.1: datapoints are retrieved from WildChat/Alpaca/UltraFeedback/HH-RLHF to approximate their stance distributions using cosine similarity >= 0.5. Hand-chosen; this threshold underlies the 'small value-gap' comparison in Fig. 17 and the dataset-distribution claims in Fig. 16.
  • Checkpoint cadence for drift-time measurement = 500 steps (WildChat), 100 steps (Alpaca); 3 epochs
    §4.1 and §3 (drift time definition): the cadence sets drift-time resolution to ~0.09 for WildChat models, so 'values established within first 9% of training' means 'changed by the first checkpoint'. The 'very early' drift-time claims are resolution-limited.
  • GPT-4o stance label softmax over 3 options = temperature 0.7 sampling, 5 generations, softmax of option logits
    §3: all drift magnitudes and value-gap distances are computed from this judge configuration. It is verified on 100 samples but not calibrated against a broader human-judgment distribution or across checkpoints; the trinary neutral-heavy collapse could produce near-zero value-gap distances artifactually (§K.3).
assumptions (4)
  • domain assumption Stance (support/neutral/oppose) on value-laden prompts is a valid operationalization of latent values
    §2.1, Eq. 1; the Ethics Statement concedes it flattens distinct values (economic vs. cultural opposition to immigration are both 'oppose'). Load-bearing for all three findings.
  • domain assumption GPT-4o stance labels approximate human judgments across all checkpoints, models, and response styles
    §3: verified on a manual sample of 100 prompt-generation pairs; not calibrated per checkpoint or per model. Since WildChat-fine-tuned outputs are longer and more hedged, the judge's neutral label may track style rather than latent value.
  • domain assumption The studied setup (Llama-3 3B/8B, Qwen-3 4B/8B; WildChat/Alpaca SFT; UltraFeedback/HH-RLHF preference data) represents the standard post-training pipeline
    §4.1, §5.1: WildChat and Alpaca are not the instruction mixtures used in production alignment recipes; the generalization of 'SFT sets values, preference optimization rarely re-aligns' to real pipelines is untested.
  • domain assumption Checkpoint sampling at 500/100-step intervals captures the value evolution trajectory
    §3 (drift time), §4.1: the first WildChat checkpoint is at step 500 of ~5000, so any drift occurring and reversing within 500 steps is unobserved; drift-time ~0.09 means 'by the first checkpoint'.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Value Drifts: Tracing Value Alignment During LLM Post-Training." pith.science (2026). https://pith.science/paper/BDKYUUXF

@misc{pith2026251026707,
  author       = {Pith},
  title        = {Pith review of: Value Drifts: Tracing Value Alignment During LLM Post-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BDKYUUXF}},
  note         = {Machine review of arXiv:2510.26707}
}
read the original abstract

As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model's post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model's values, and subsequent preference optimization rarely re-aligns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.

Figures

Figures reproduced from arXiv: 2510.26707 by the authors.

Figure 1
Figure 1. Post-training can cause value drift, shifting the stance of model generations from a neutral to support, when asked a value-probing question such as “Should we close the gates and stop immi￾gration?” In this paper, we analyze how post-training reshapes these values. largely opaque. This motivates our central research question: How does the underlying training data, algorithms, and their interaction shape the values … view at source ↗
Figure 2
Figure 2. SFT-induced values for Llama-3-3B and Qwen-3-4B models trained on WildChat and Alpaca for the topic of immigration. Each line represents the mean stance probability of support, neutral, and oppose stances, with 95% confidence intervals. In all cases, SFT leads to changes in stance distribution, often very early in training; WildChat leads to a high proportion of neutral responses, while on Alpaca leads to a higher p… view at source ↗
Figure 3
Figure 3. Values on the topic of abortion induced by training [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Value drifts induced by different preference optimization algorithms. Each line represents [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Prompt used to elicit stance distribution for each generated response. [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Preference optimization on the Llama-3-3B SFT WildChat using HHRLHF, shown for the topic abortion. Each line depicts the mean stance probability for support, neutral, and op￾pose stances, with 95% confidence intervals. Across PPO, DPO, and SIMPO, stance probabilities r…
Figure 7
Figure 7. Figure 7: System prompt used for generating synthetic preference data. The prompt instructs the [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: DPO-induced value drifts for Llama3 3B and Qwen3 4B models for Setup 1 and Setup 2, [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: DPO-induced value drifts for Llama3 3B and Qwen3 4B models for Setup 1 and Setup 2, [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: SIMPO-induced value drifts for Llama3 3B and Qwen3 4B models for Setup 1 and [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: SIMPO-induced value drifts for Llama3 3B and Qwen3 4B models for Setup 1 and Setup [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Effect of the PPO hyperparameter kl on model stance distributions across topics. Each plot shows how varying kl influences the proportion of support stances predicted by Llama3-3B SFT WildChat on three topics. base 200 400 600 800 0.0 0.2 0.4 0.6 0.8 1.0 SFT training …
Figure 13
Figure 13. Figure 13: Effect of the DPO hyperparameter β on model stance distributions across topics. Each plot shows how varying β influences the proportion of support stances predicted by Llama3-3B SFT WildChat on three topics. J.1 PPO: EFFECT OF KL PENALTY COEFFICIENT For PPO, we vary t…
Figure 14
Figure 14. Figure 14: Effect of the SIMPO hyperparameter γ on model stance distributions across topics. Each plot shows how varying γ influences the proportion of support stances predicted by Llama3-3B SFT WildChat on three topics. K.1 METHODOLOGY To approximate the data distribution, we f…
Figure 15
Figure 15. Figure 15: Prompt used to elicit stance distribution for each retrieved datapoint. [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]
Figure 16
Figure 16. Figure 16: Comparison of stance distributions for the WildChat (left) and Alpaca (right) SFT [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: Histogram of Euclidean distances between preference pairs in (a) UltraFeedback, (b) HH [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Epistemic Goggles: A Pretrained Module that Induces an Epistemic Frame via Gradient Editing

    cs.AI 2026-07 unverdicted novelty 7.0 of 10

    Goggles is a gradient-editing module trained once per base model and frame that, when applied frozen during finetuning, causes LLMs to treat unannotated documents with a specified epistemic stance (e.g., as fiction) a...

  2. Agents of Chaos

    cs.AI 2026-02 unverdicted novelty 6.0 of 10

    An exploratory red-teaming study documents eleven cases of security, privacy, and governance failures in autonomous language-model agents with tool access and persistent memory.

Reference graph

Works this paper leans on

93 extracted references · 3 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  3. [3]

    Foundational challenges in assuring alignment and safety of large language models

    Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024. URL https://arxiv.org/abs/2404.09932

    arXiv 2024

  4. [4]

    Explicitly unbiased large language models still form biased associations

    Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L Griffiths. Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences, 122 0 (8): 0 e2416228122, 2025. URL https://www.pnas.org/doi/10.1073/pnas.2416228122

    doi:10.1073/pnas.2416228122 2025

  5. [5]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. URL https://arxiv.org/abs/2204.05862

    arXiv 2022

  6. [6]

    Managing extreme AI risks amid rapid progress

    Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, et al. Managing extreme AI risks amid rapid progress. Science, 384 0 (6698): 0 842--845, 2024. URL https://www.science.org/doi/10.1126/science.adn0117

    doi:10.1126/science.adn0117 2024

  7. [7]

    Piqa: Reasoning about physical commonsense in natural language, 2019

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL https://arxiv.org/abs/1911.11641

    arXiv 2019

  8. [8]

    Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? Advances in Neural Information Processing Systems, 35: 0 3663--3678, 2022

    Rishi Bommasani, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? Advances in Neural Information Processing Systems, 35: 0 3663--3678, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/17a234c91f746d9625a75cf8a8731ee2-Abstract-Confer...

    2022

Show all 93 references
  1. [9]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952. URL https://doi.org/10.2307/2334029

    1952 doi

  2. [10]

    Density-based clustering based on hierarchical density estimates

    Ricardo JGB Campello, Davoud Moulavi, and J \"o rg Sander. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pp.\ 160--172. Springer, 2013. URL https://link.springer.com/chapter/10.1007/978-3-64...

    2013 doi

  3. [11]

    How people use chatgpt

    Aaron Chatterji, Thomas Cunningham, David J Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. How people use chatgpt. Working Paper 34255, National Bureau of Economic Research, September 2025. URL http://www.nber.org/papers/w34255

    2025

  4. [12]

    Chatbot arena: An open platform for evaluating LLMs by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating LLMs by human preference. In Forty-first International Conference...

    2024

  5. [13]

    Reward model interpretability via optimal and pessimal tokens

    Brian Christian, Hannah Rose Kirk, Jessica AF Thompson, Christopher Summerfield, and Tsvetomira Dumbalska. Reward model interpretability via optimal and pessimal tokens. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 1048--1059, 2...

    2025

  6. [14]

    Ultrafeedback: Boosting language models with high-quality feedback, 2023

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023. URL https://openreview.net/forum?id=pNkOx3IVWI

    2023

  7. [15]

    Towards measuring the representation of subjective global opinions in language models

    Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towar...

    2024

  8. [16]

    Towards analyzing and understanding the limitations of DPO : A theoretical perspective

    Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of DPO : A theoretical perspective. arXiv preprint arXiv:2404.04626, 2024. URL https://arxiv.org/abs/2404.04626

    2024 arXiv

  9. [17]

    Artificial intelligence, values, and alignment

    Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30 0 (3): 0 411--437, 2020

    2020

  10. [18]

    The delta learning hypothesis: Preference tuning on weak data can yield strong gains

    Scott Geng, Hamish Ivison, Chun-Liang Li, Maarten Sap, Jerry Li, Ranjay Krishna, and Pang Wei Koh. The delta learning hypothesis: Preference tuning on weak data can yield strong gains. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025...

    2025

  11. [19]

    Donoho, and Sanmi Koyejo

    Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes, Dhruv Bhandarkar Pai, Andrey Gromov, Dan Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo. Is model collapse inevitable? B reaking the curse...

    2024

  12. [20]

    Mixed preference optimization: Reinforcement learning with data selection and better reference model

    Qi Gou and Cam-Tu Nguyen. Mixed preference optimization: Reinforcement learning with data selection and better reference model. arXiv preprint arXiv:2403.19443, 2024. URL https://arxiv.org/abs/2403.19443

    2024 arXiv

  13. [21]

    Alignment faking in large language models

    Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, et al. Alignment faking in large language models. arXiv preprint arXiv:2412.14093, 2024. URL https://arxiv.org/abs/2412.14093

    2024 arXiv

  14. [22]

    Assessing the alignment of large language models with human values for mental health integration: Cross-sectional study using schwartz’s theory of basic values

    Dorit Hadar-Shoval, Kfir Asraf, Yonathan Mizrachi, Yuval Haber, and Zohar Elyoseph. Assessing the alignment of large language models with human values for mental health integration: Cross-sectional study using schwartz’s theory of basic values. JMIR Mental Health, 11: 0 e55988...

    2024

  15. [23]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  16. [24]

    Collective constitutional AI : Aligning a language model with public input

    Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. Collective constitutional AI : Aligning a language model with public input. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 139...

    2024

  17. [25]

    Values in the wild: Discovering and analyzing values in real-world language model interactions

    Saffron Huang, Esin Durmus, Miles McCain, Kunal Handa, Alex Tamkin, Jerry Hong, Michael Stern, Arushi Somani, Xiuruo Zhang, and Deep Ganguli. Values in the wild: Discovering and analyzing values in real-world language model interactions. arXiv preprint arXiv:2504.15236, 2025. ...

    2025 arXiv

  18. [26]

    The n+ implementation details of RLHF with PPO : A case study on TL ; DR summarization

    Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ implementation details of RLHF with PPO : A case study on TL ; DR summarization. In First Conference on Language Modeling, 2024 b . URL https://openreview.net/forum?id=kHO2ZTa8e3

    2024

  19. [27]

    Smith, Yejin Choi, and Hannaneh Hajishirzi

    Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking DPO and PPO : Disentangling best practices for learning from preference feedback. In The Thirty-eighth Annual Conference on Neu...

    2024

  20. [28]

    Evaluating and inducing personality in pre-trained language models

    Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evaluating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems, 36: 0 10622--10643, 2023. URL https://dl.acm.org/doi/10.5555/3666122.3666588

    2023

  21. [29]

    Can machines learn morality? The D elphi experiment

    Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, et al. Can machines learn morality? The D elphi experiment. arXiv preprint arXiv:2110.07574, 2021. URL https://arxiv.org/a...

    2021 arXiv

  22. [30]

    The PRISM alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models

    Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew M Bean, Katerina Margatina, Rafael Mosquera-Gomez, Juan Ciro, Max Bartolo, Adina Williams, He He, et al. The PRISM alignment dataset: What participatory, representative and individualised human feedback reveals about...

    2024

  23. [31]

    Understanding the effects of RLHF on LLM generalisation and diversity

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of RLHF on LLM generalisation and diversity. arXiv preprint arXiv:2310.06452, 2023. URL https://arxiv.org/abs/2310.06452

    2023 arXiv

  24. [32]

    What are human values, and how do we align ai to them? arXiv preprint arXiv:2404.10636, 2024

    Oliver Klingefjord, Ryan Lowe, and Joe Edelman. What are human values, and how do we align ai to them? arXiv preprint arXiv:2404.10636, 2024

    2024 arXiv

  25. [33]

    You are what you eat-- AI alignment requires understanding how data shapes structure and generalisation

    Simon Pepin Lehalleur, Jesse Hoogland, Matthew Farrugia-Roberts, Susan Wei, Alexander Gietelink Oldenziel, George Wang, Liam Carroll, and Daniel Murfet. You are what you eat-- AI alignment requires understanding how data shapes structure and generalisation. arXiv preprint arXi...

    2025 arXiv

  26. [34]

    Beyond probabilities: Unveiling the misalignment in evaluating large language models

    Chenyang Lyu, Minghao Wu, and Alham Aji. Beyond probabilities: Unveiling the misalignment in evaluating large language models. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pp.\ 109--131, 2024. URL https://aclanthology.org/2024.kno...

    2024

  27. [35]

    Treleaven, and Miguel Rodrigues Rodrigues

    Reem Masoud, Ziquan Liu, Martin Ferianc, Philip C. Treleaven, and Miguel Rodrigues Rodrigues. Cultural alignment in large language models: An explanatory analysis based on H ofstede ' s cultural dimensions. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barb...

    2025

  28. [36]

    How people use claude for support, advice, and companionship, 2025

    Miles McCain, Ryn Linthicum, Chloe Lubinski, Alex Tamkin, Saffron Huang, Michael Stern, Kunal Handa, Esin Durmus, Tyler Neylon, Stuart Ritchie, Kamya Jagadish, Paruul Maheshwary, Sarah Heck, Alexandra Sanderford, and Deep Ganguli. How people use claude for support, advice, and...

    2025

  29. [37]

    UMAP : Uniform manifold approximation and projection

    Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP : Uniform manifold approximation and projection. Journal of Open Source Software, 3 0 (29): 0 861, 2018. doi:10.21105/joss.00861. URL https://doi.org/10.21105/joss.00861

    2018 doi

  30. [38]

    Sim PO : Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Sim PO : Simple preference optimization with a reference-free reward. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=3Tzcot1LKb

    2024

  31. [39]

    S em E val-2016 task 6: Detecting stance in tweets

    Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. S em E val-2016 task 6: Detecting stance in tweets. In Steven Bethard, Marine Carpuat, Daniel Cer, David Jurgens, Preslav Nakov, and Torsten Zesch (eds.), Proceedings of the 10th International...

    2016 doi

  32. [40]

    Jared Moore, Tanvi Deshpande, and Diyi Yang. Are large language models consistent over value-laden questions? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 15185--15221, Miami, Florida, ...

    2024

  33. [41]

    Reinforcement learning finetunes small subnetworks in large language models

    Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur, and Hao Peng. Reinforcement learning finetunes small subnetworks in large language models. arXiv preprint arXiv:2505.11711, 2025. URL https://arxiv.org/abs/2505.11711

    2025

  34. [42]

    Value imprint: A technique for auditing the human values embedded in RLHF datasets

    Ike Obi, Rohan Pant, Srishti Shekhar Agrawal, Maham Ghazanfar, and Aaron Basiletti. Value imprint: A technique for auditing the human values embedded in RLHF datasets. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. ...

    2024

  35. [43]

    Attributing mode collapse in the fine-tuning of large language models

    Laura O'Mahony, Leo Grinsztajn, Hailey Schoelkopf, and Stella Biderman. Attributing mode collapse in the fine-tuning of large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL https://openreview.net/forum?id=3pDMYjpOxk

    2024

  36. [44]

    Help OpenAI fix over-refusals! https://community.openai.com/t/help-openai-fix-over-refusals/409799, October 2023

    OpenAI . Help OpenAI fix over-refusals! https://community.openai.com/t/help-openai-fix-over-refusals/409799, October 2023. Accessed: 2025-09-23

    2023

  37. [45]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  38. [46]

    Does writing with language models reduce content diversity? arXiv preprint arXiv:2309.05196, 2023

    Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? arXiv preprint arXiv:2309.05196, 2023. URL https://arxiv.org/abs/2309.05196

    2023 arXiv

  39. [47]

    Smaug: Fixing failure modes of preference optimisation with DPO -positive

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with DPO -positive. arXiv preprint arXiv:2402.13228, 2024. URL https://arxiv.org/abs/2402.13228

    2024 arXiv

  40. [48]

    Do LLMs possess a personality? M aking the MBTI test an amazing evaluation for large language models

    Keyu Pan and Yawen Zeng. Do LLMs possess a personality? M aking the MBTI test an amazing evaluation for large language models. arXiv preprint arXiv:2307.16180, 2023. URL https://arxiv.org/abs/2307.16180

    2023 arXiv

  41. [49]

    What matters in data for DPO ? arXiv preprint arXiv:2508.18312, 2025

    Yu Pan, Zhongze Cai, Guanting Chen, Huaiyang Zhong, and Chonghuan Wang. What matters in data for DPO ? arXiv preprint arXiv:2508.18312, 2025. URL https://arxiv.org/abs/2508.18312

    2025

  42. [50]

    Enhancing alignment using curriculum learning & ranked preferences

    Pulkit Pattnaik, Rishabh Maheshwary, Kelechi Ogueji, Vikas Yadav, and Sathwik Tejaswi Madhusudhan. Enhancing alignment using curriculum learning & ranked preferences. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Lin...

    2024 doi

  43. [51]

    AI psychometrics: Assessing the psychological profiles of large language models through psychometric inventories

    Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. AI psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science, 19 0 (5): 0 808--826, 2024. URL h...

    2024 doi

  44. [52]

    Discovering language model behaviors with model-written evaluations

    Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Lin...

    2023

  45. [53]

    The lock-in hypothesis: Stagnation by algorithm

    Tianyi Qiu, Zhonghao He, Tejasveer Chugh, and Max Kleiman-Weiner. The lock-in hypothesis: Stagnation by algorithm. In ICLR 2025 Workshop on Bidirectional Human-AI Alignment, 2025. URL https://openreview.net/forum?id=4CRMWP1tYc

    2025

  46. [54]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 0 53728--53741, 2023. URL https://dl.ac...

    2023

  47. [55]

    Balancing the budget: Understanding trade-offs between supervised and preference-based finetuning

    Mohit Raghavendra, Junmo Kang, and Alan Ritter. Balancing the budget: Understanding trade-offs between supervised and preference-based finetuning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of t...

    2025 doi

  48. [56]

    Close Encounters of the AI Kind: A Survey of Public Sentiment About Artificial Intelligence

    Lee Rainie. Close Encounters of the AI Kind: A Survey of Public Sentiment About Artificial Intelligence . Report, Elon University - Imagining the Digital Future Center and Pew Research Center, March 2025. URL https://imaginingthedigitalfuture.org/reports-and-publications/close...

    2025

  49. [57]

    Scalpel vs

    Neel Rajani, Aryo Pradipta Gema, Seraphina Goldfarb-Tarrant, and Ivan Titov. Scalpel vs. hammer: Grpo amplifies existing capabilities, sft replaces them, 2025. URL https://arxiv.org/abs/2507.10616

    2025 arXiv

  50. [58]

    Sentence- BERT : Sentence embeddings using S iamese BERT -networks

    Nils Reimers and Iryna Gurevych. Sentence- BERT : Sentence embeddings using S iamese BERT -networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internationa...

    2019

  51. [59]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA : A graduate-level google-proof Q&A benchmark, 2023. URL https://arxiv.org/abs/2311.12022

    2023 arXiv

  52. [60]

    Sutherland

    Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tPNHOoZFl9

    2025

  53. [61]

    The nature of human values

    Milton Rokeach. The nature of human values. NSF Award, 72 0 (7205473): 0 5473, 1972. URL https://philpapers.org/rec/ROKTNO

    1972

  54. [62]

    Political compass or spinning arrow? T owards more meaningful evaluations for values and opinions in large language models

    Paul R \"o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. Political compass or spinning arrow? T owards more meaningful evaluations for values and opinions in large language models. In Lun-Wei Ku, Andre Martins, and Viv...

    2024

  55. [63]

    Unintended impacts of LLM alignment on global representation

    Michael J Ryan, William Held, and Diyi Yang. Unintended impacts of LLM alignment on global representation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2024

  56. [64]

    Personal values across cultures

    Lilach Sagiv and Shalom H Schwartz. Personal values across cultures. Annual review of psychology, 73 0 (1): 0 517--546, 2022. URL https://www.annualreviews.org/content/journals/10.1146/annurev-psych-020821-125100

    2022 doi

  57. [65]

    A note on the pure theory of consumer's behaviour

    Paul A Samuelson. A note on the pure theory of consumer's behaviour. In The Foundations of Price Theory Vol 4, pp.\ 101--116. Routledge, 2024. URL http://www.jstor.org/stable/2548836

    2024

  58. [66]

    Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML'23

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023. URL https://dl.acm.org/doi/10.5555/36184...

    2023

  59. [67]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  60. [68]

    Extending the cross-cultural validity of the theory of basic human values with a different method of measurement

    Shalom H Schwartz, Gila Melech, Arielle Lehmann, Steven Burgess, Mari Harris, and Vicki Owens. Extending the cross-cultural validity of the theory of basic human values with a different method of measurement. Journal of cross-cultural psychology, 32 0 (5): 0 519--542, 2001. UR...

    2001 doi

  61. [69]

    Personality traits in large language models

    Gregory Serapio-Garc \' a, Mustafa Safdari, Cl \'e ment Crepy, Luning Sun, Stephen Fitz, Marwa Abdulhai, Aleksandra Faust, and Maja Matari \'c . Personality traits in large language models. 2023. URL https://arxiv.org/abs/2307.00184

    2023 arXiv

  62. [70]

    Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Pere...

    2024

  63. [71]

    AI models collapse when trained on recursively generated data

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. AI models collapse when trained on recursively generated data. Nature, 631 0 (8022): 0 755--759, 2024. URL https://www.nature.com/articles/s41586-024-07566-y

    2024

  64. [72]

    Recognizing stances in ideological on-line debates

    Swapna Somasundaran and Janyce Wiebe. Recognizing stances in ideological on-line debates. In Diana Inkpen and Carlo Strapparava (eds.), Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text , pp.\ 116--124, Los Ang...

    2010

  65. [73]

    Position: A roadmap to pluralistic alignment

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. Position: A roadmap to pluralistic alignment. In Proceedings of the 41st International Conference on Machin...

    2024

  66. [74]

    Value profiles for encoding human variation

    Taylor Sorensen, Pushkar Mishra, Roma Patel, Michael Henry Tessler, Michiel Bakker, Georgina Evans, Iason Gabriel, Noah Goodman, and Verena Rieser. Value profiles for encoding human variation. arXiv preprint arXiv:2503.15484, 2025. URL https://arxiv.org/abs/2503.15484

    2025

  67. [75]

    Lillicrap, Ana Marasović, Sylvie Delacroix, Gillian K

    Karolina Stańczak, Nicholas Meade, Mehar Bhatia, Hattie Zhou, Konstantin Böttinger, Jeremy Barnes, Jason Stanley, Jessica Montgomery, Richard Zemel, Nicolas Papernot, Nicolas Chapados, Denis Therien, Timothy P. Lillicrap, Ana Marasović, Sylvie Delacroix, Gillian K. Hadfield, a...

    2025 arXiv

  68. [76]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford A lpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  69. [77]

    A deep dive into the trade-offs of parameter-efficient preference alignment techniques

    Megh Thakkar, Quentin Fournier, Matthew Riemer, Pin-Yu Chen, Amal Zouaq, Payel Das, and Sarath Chandar. A deep dive into the trade-offs of parameter-efficient preference alignment techniques. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annu...

    2024 doi

  70. [78]

    Zephyr: Direct distillation of LM alignment

    Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Cl \'e mentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M Rush, and Thomas Wolf. Zephyr: Direct distillation of L...

    2024

  71. [79]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Ann...

    2023

  72. [80]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=gEZrGCozdqR

    2022

  73. [81]

    Simple synthetic data reduces sycophancy in large language models, 2025

    Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V Le. Simple synthetic data reduces sycophancy in large language models, 2025. URL https://openreview.net/forum?id=WDheQxWAo4

    2025

  74. [82]

    Generative monoculture in large language models

    Fan Wu, Emily Black, and Varun Chandrasekaran. Generative monoculture in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=yZ7sn9pyqb

    2025

  75. [83]

    Fairness feedback loops: T raining on synthetic data amplifies bias

    Sierra Wyllie, Ilia Shumailov, and Nicolas Papernot. Fairness feedback loops: T raining on synthetic data amplifies bias. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 2113--2147, 2024. URL https://dl.acm.org/doi/10.1145/3630106.3659029

    2024

  76. [84]

    Finding the sweet spot: Preference data construction for scaling preference optimization

    Yao Xiao, Hai Ye, Linyao Chen, Hwee Tou Ng, Lidong Bing, Xiaoli Li, and Roy Ka-Wei Lee. Finding the sweet spot: Preference data construction for scaling preference optimization. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings ...

    2025

  77. [85]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  78. [86]

    HellaSwag : Can a machine really finish your sentence?, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag : Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830

    2019 arXiv

  79. [87]

    Cultivating pluralism in algorithmic monoculture: The community alignment dataset

    Lily Hong Zhang, Smitha Milli, Karen Jusko, Jonathan Smith, Brandon Amos, Manon Revel, Jack Kussman, Lisa Titus, Bhaktipriya Radharapu, Jane Yu, et al. Cultivating pluralism in algorithmic monoculture: The community alignment dataset. arXiv preprint arXiv:2507.09650, 2025. URL...

    2025

  80. [88]

    Echo chamber: RL post-training amplifies behaviors learned in pretraining

    Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RL post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912, 2025. URL https://openreview.net/forum?id=dp4KWuSDzj

    2025 arXiv

  81. [89]

    WildChat : 1m chat GPT interaction logs in the wild

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. WildChat : 1m chat GPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Bl8u7ZRlbM

    2024

  82. [90]

    Secrets of RLHF in large language models part I: PPO

    Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of RLHF in large language models part I: PPO . arXiv preprint arXiv:2307.04964, 2023. URL https://arxiv.org/abs/2307.04964

    2023 arXiv

  83. [91]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  84. [92]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  85. [93]

    small value-gap

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.