Pith. sign in

REVIEW 4 major objections 4 minor 15 references

MOSLIM:Align with diverse preferences in prompts through reward classification

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MOSLIM claims to steer LLM preferences with one reward model and prompt tags, no preference-specific SFT.

desk verdict MOSLIM's central reward formula (Eq. 11) never uses the model's output, so the method as written cannot work; a plausible high-level idea with a load-bearing internal contradiction. read the letter →

arxiv 2505.20336 v1 pith:ZL6GTPK6 submitted 2025-05-24 cs.CL cs.AI

Yu Zhang , Wanli Jiang , Zhengyu Yang This is my paper
classification cs.CLcs.AI
keywords multi-objectivealignmentdynamicpreferencecontrolrewardclassificationmulti-headmodelprompt-basedRLHFoff-the-shelfintensity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper seeks to establish that multi-objective alignment of large language models can be done with a single reward model and a single policy, using prompt tags to control which preferences and intensities are expressed at generation time. It introduces MOSLIM, which trains a multi-head reward model to classify question-answer pairs by preference dimension and intensity rather than to score them, and then maps those classifications into a scalar reward for policy optimization. The claimed payoff is that preference-specific supervised fine-tuning becomes unnecessary, thousands of off-the-shelf models can be plugged into the alignment stage directly, and the resulting policy outperforms MORLHF, Rewarded Soups, and RiC on the tested benchmarks while using fewer GPU-hours. If correct, this would make controllable preference alignment a lightweight post-training step instead of a per-preference retraining burden.

What carries the argument

The load-bearing mechanism is the multi-head classification reward model, which replaces a scalar reward score with per-head intensity classification: each head is trained with cross-entropy to predict the intensity class of a (question, answer) pair for one preference dimension (Eqs. 6–10). The companion mechanism is the prompt-controlled reward mapping $(1/k)\sum_i (p_i^{target} - p_i^{avg})/p_i^{std} \cdot \mathrm{mask}_i$ (Eq. 11), which takes the preference dimensions and intensities a user has specified in the prompt, standardizes them using moving averages and standard deviations recorded during reward-model training, and combines them into a scalar reward for policy optimization. The third piece is prompt alignment, which removes preference labels from the input the reward model sees during policy training so that its input distribution matches its own training distribution.

What would settle it

Take a fixed prompt and a fixed preference prefix such as `<helpfulness 5>` and compute Eq. 11 for two different candidate answers; if the reward score is identical for both, the reinforcement learning signal cannot depend on response quality, which would contradict the paper's claim that the reward model classifies question-answer pairs.

Watch

Extended reading notes

Core claim

MOSLIM's central discovery is that a multi-head classification reward model—one head per preference dimension such as helpfulness, honesty, and harmlessness, each predicting intensity classes—can serve as the reward signal for multi-objective RLHF-style policy optimization. The paper argues that by stripping preference labels from the reward model's input during policy training and standardizing each head's intensity predictions with per-intensity means and standard deviations (Eq. 11), a single policy learns to follow preference tags such as `<helpfulness 5>` or `<harmless max>` at inference. On MT-Bench, HaluEval 2.0, and Hackaprompt, the authors report that MOSLIM-trained policies beat MORLHF, Rewarded Soups, and RiC across four data difficulty levels and three preference dimensions, and that the method composes with PPO, RLOO, and Online-DPO.

Load-bearing premise

The entire reward signal rests on Eq. 11 being a function of the generated answer, but as printed that equation uses only the preference intensity named in the prompt and stored per-intensity statistics, not the answer content.

Editorial extensions

If this is right

  • A single RLHF-trained policy can be steered at inference by prompt tags alone, with no per-preference retraining or multiple policies.
  • Off-the-shelf SFT models can be aligned to diverse preferences through the RLHF stage alone, skipping preference-specific supervised fine-tuning.
  • Multi-objective alignment becomes cheaper: MOSLIM reports fewer GPU-hours than MORLHF, Rewarded Soups, and RiC under comparable model sizes.
  • Reward-model scaling carries over to classification-style reward models, with larger classifiers giving more controllable policies.
  • MOSLIM is compatible with PPO, RLOO, and Online-DPO, and the paper reports Online-DPO as the strongest of the three under its setup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Eq. 11 is taken literally, the policy gradient does not depend on answer quality, so the reported controllability would have to be an effect of the prompt-conditioned policy itself; a testable repair is to replace the prompt-specified target intensity in Eq. 11 with the classifier's predicted probability for each head.
  • The classification-based reward view suggests a natural bridge to distributional preference modeling: instead of collapsing preferences to a scalar at the reward-model stage, a policy could consume the full per-head intensity distributions and learn to target a user's profile directly.
  • Because MOSLIM couples preference control to prompt tags, it should be composable with system-prompt steering methods, enabling a continuous preference dial across arbitrary preference vocabularies without additional training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MOSLIM, a multi-objective alignment method that uses a single multi-head classification reward model and a single policy model, with preference dimensions and intensities specified by prompt tags such as <helpfulness 5>. The authors claim that this is the first approach to achieve dynamic preference alignment with a single reward model and policy, without preference-specific SFT, and report experimental results on MT-Bench, HaluEval 2.0, and Hackaprompt, showing improvements over MORLHF, RSoups, and RiC, with lower GPU time. The core mechanism is a reward mapping function (Eq. 11) that converts classification outputs into a scalar reward, followed by PPO, RLOO, or Online-DPO policy optimization.

Significance. If the proposed mechanism were sound, the contribution would be practically valuable: a single reward model and policy able to steer generation across multiple preference dimensions and intensities would substantially reduce the cost of multi-objective alignment and enable off-the-shelf SFT models to be aligned without preference-specific training. The paper also provides a useful empirical study of reward-model scaling and policy-optimization ablations. However, the central reward mapping, as written in Eq. 11, has no dependence on the generated answer, which breaks the RL loop that the method's entire contribution rests on. The formalization therefore does not support the claimed mechanism, and the empirical results, while extensive, cannot be interpreted as evidence for a method that is not coherently defined.

major comments (4)
  1. [Section 2.3, Eq. (11)] Equation (11) defines the reward as r_score = (1/k) * sum_i (p_target_i - p_avg_i) / p_std_i * mask_i, and the text explicitly states that p_target_i is the preference intensity for the i-th dimension in the prompt. None of the quantities p_target_i, p_avg_i, p_std_i, or mask_i depend on the generated answer y. Substituting Eq. (11) into the PPO objective in Eq. (12) produces a reward that is constant with respect to the policy parameters; the only response-dependent term is the KL penalty, so the policy would collapse toward the SFT model rather than learn any preference-sensitive behavior. The abstract and Section 2.2 promise that classification outputs are converted into reward scores, but Eq. (11) never uses the reward model's class predictions. If the implementation instead uses the predicted class probability of the target intensity for the generated answer, the paper must define p_target_i accordingly and justify the z-score mapping; as written, the central mechanism is self-contradictory and the claimed RL alignment cannot occur.
  2. [Section 2.2, Eq. (10)] The combined cross-entropy loss in Eq. (10) is not the cross-entropy of the concatenated softmax outputs S1||S2||...||Sn. Concatenating softmax vectors and their labels would yield a sum of per-head cross-entropy terms, each with its own per-head softmax denominator. The formula in Eq. (10) pools the exponentials across all heads into a single denominator, which corresponds to a different loss (a single softmax over all heads' logits). This either misstates the implementation or leaves the actual multi-head training objective ambiguous. The derivation from Eq. (9) to Eq. (10) should be corrected or clarified.
  3. [Section 3.2] The experimental comparison omits CDPO (Guo et al., 2024b), which is described in Related Works as the closest prompt-based method with intensity-level control and is also based on a single model. Since the paper claims to outperform current multi-objective approaches, the absence of CDPO from Tables 2-4 weakens the empirical claim, especially because CDPO is the method that directly addresses the same prompt-driven intensity-control setting.
  4. [Section 3.2, Table 2] The statement that MOSLIM outperforms MORLHF by 57% in helpfulness on DataType 4 is misleading: the underlying scores are 3.14 vs 2.00, and MT-Bench scores are not ratio-scale measures, so a relative percentage improvement is not a meaningful quantity. This should be reported as an absolute difference or a proper effect size.
minor comments (4)
  1. [Section 2.2, Eq. (10)] The notation S1||S2||...||Sn and L1||L2||...||Ln is nonstandard for concatenation, and the label index L_j in the second line is not consistent with the per-head label notation L_i,j used in Eq. (9).
  2. [Section 3.2, Data Construction] The dataset is referred to as 'full-hh-rlhf' in Section 3.2 but as 'hh-full-rlhf' in Section 3.1; the naming should be made consistent.
  3. [Section 5, Conclusion] There is a typo in 'eliminats the need' which should read 'eliminates the need'.
  4. [Appendix F, Table 8] The table lists a separate 'Value Model: 7B' for MOSLIM, so the paper's claim of using 'a single reward model and policy model' is slightly imprecise; the method actually uses three models (policy, reward, value) during training.

Circularity Check

1 steps flagged · score 7.0 of 10

Eq. 11's 'reward' is the prompt's target intensity normalized, not the reward model's classification output; the central reward-classification mechanism reduces to the input target by construction.

  1. self definitional [Section 2.3, Eq. 11 (Reward Mapping), with Eq. 12 (PPO objective); cf. Abstract]
    "During inference phase, each preference dimension value is transformed into a sample value from a Gaussian distribution with zero mean and unit variance ... The specific reward mapping formula is defined as follows: rscore = 1/k sum_{i=0}^k (p_target_i − p_avg_i)/p_std_i mask_i ... where i denotes the preference dimension, k represents the total number of preference dimensions, and target represents preference intensity for i-th dimension in the prompt."

    Eq. 11 defines the scalar reward rscore directly from p_target_i, which the text states is 'the preference intensity for the i-th dimension in the prompt', and from p_avg_i and p_std_i, the training-time moving statistics. The reward model's classification output Si from Eq. 7 never appears in the formula. Thus the mapping promised in the Abstract—'a mapping function that converts classification results from reward model into reward scores'—is, as written, a normalized copy of the prompt's requested intensity plus fitted statistics, with no dependence on the generated answer y. Substituting Eq. 11 into the PPO objective Eq.

full rationale

Aside from Eq. 11's definitional substitution, I find no other circularity in the manuscript. The paper does not rely on load-bearing self-citations: the cited prior works (MORLHF, RSoups, RiC, CDPO, PPO, DPO, etc.) are external baselines and algorithmic components, and there is no invoked uniqueness theorem from the authors. The reward-model accuracy comparisons and policy benchmark comparisons are empirical evaluations against external datasets and baselines, so they are not circular on their own. However, the central mechanism of the paper is the reward mapping in Eq. 11. Because the text defines p_target_i as the prompt's preference intensity and never introduces the reward model's classification output into Eq. 11, the claimed mapping from 'classification results' to 'reward score' is, by the paper's own equations, a normalized version of the input target plus training statistics. This makes the headline claim that the policy is optimized by a reward derived from QA-pair classification definitionally false as written, and the controllability result is not derived from the classification head. This warrants a high circularity score because the central derivation reduces to its own input target, even though the empirical comparisons themselves are not self-referential.

Assumptions & free parameters 4 free parameters · 5 assumptions · 3 invented entities

The central mechanism rests on several choices that are fitted or asserted: the discretization of intensities into classes, the z-score statistics that scale rewards, and the multi-head classification architecture. None of these are derived from first principles or externally validated, and the paper's own Eq 11 decouples the reward from the classifier output.

free parameters (4)
  • z-score statistics p_avg_i and p_std_i = not reported numerically
    Moving average and standard deviation of each preference intensity are recorded from training data and used in Eq 11 to normalize the reward. These are fitted to the reward training data and directly determine the reward scale.
  • intensity level boundaries (DataType 1-4) = e.g., helpfulness 1-5, harmless 1-2 in DataType 4
    The mapping from UltraFeedback and UltraSafety scores to discrete intensity classes is chosen by hand; different DataTypes give different classification granularity and strongly affect reward model accuracy (Table 1).
  • number of preference heads = 3 (helpfulness, honesty, harmlessness)
    The set of preference dimensions and the number of heads is an author choice; the paper claims compatibility with any number but only tests 3.
  • KL coefficient beta and learning rates = beta 5e-2, learning rates 5e-7 to 1e-5
    Training hyperparameters chosen by hand; they affect policy optimization results but are not the main scientific parameters.
assumptions (5)
  • domain assumption A multi-head classification loss (cross-entropy per head) is a valid way to train a reward model that captures preference intensities.
    Section 2.2 assumes that converting scalar scores into discrete intensity classes and optimizing classification accuracy yields a reward signal useful for alignment; no theoretical or empirical validation is given beyond Table 1.
  • ad hoc to paper The combined loss in Eq 10 is a valid aggregate of per-head losses.
    Eq 10 computes cross-entropy over concatenated logits and labels, which is not equivalent to the sum of per-head cross-entropies in Eq 9; the paper claims 'computational convenience' without justification.
  • domain assumption Z-score normalization makes preference intensities from different dimensions additive and comparable.
    Section 2.3 assumes that subtracting the training mean and dividing by the training standard deviation for each head produces additive, comparable rewards; no empirical check supports this additivity.
  • domain assumption The policy can be trained from an SFT model that was not preference-conditioned, using only RLHF, to follow preference tags.
    The central claim that off-the-shelf models can be used without preference-specific SFT rests on this assumption; the experiments initialize from an SFT model trained on hh-full-rlhf, not a truly off-the-shelf model.
  • standard math PPO, RLOO, and Online-DPO optimize the scalar reward from Eq 11 in the intended way.
    The paper uses standard RL algorithms, but Eq 11's independence from y would make the reward constant for a given prompt under PPO, so this standard assumption is incompatible with the stated mechanism.
invented entities (3)
  • Multi-head classification reward model
    purpose: To classify question-answer pairs into preference dimensions and intensity levels instead of scoring them (Eq 6-10)
    No falsifiable prediction outside the paper; its accuracy is evaluated internally (Table 1), but the model's outputs do not appear in the reward formula as written.
  • Reward mapping function (Eq 11)
    purpose: To convert classification results into a scalar reward for reinforcement learning
    The mapping is defined in the paper, but as written it uses the prompt target and training statistics, not the reward model's predictions; it has no external validation.
  • Preference tags such as <helpfulness 5>
    purpose: Prompt-level control tokens for specifying preference dimension and intensity during training and inference
    These are new prompt tokens introduced by the paper; their semantics are only tested in the paper's own benchmarks, and they resemble tags from RiC and CDPO.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MOSLIM:Align with diverse preferences in prompts through reward classification." pith.science (2026). https://pith.science/paper/ZL6GTPK6

@misc{pith2026250520336,
  author       = {Pith},
  title        = {Pith review of: MOSLIM:Align with diverse preferences in prompts through reward classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZL6GTPK6}},
  note         = {Machine review of arXiv:2505.20336}
}
read the original abstract

The multi-objective alignment of Large Language Models (LLMs) is essential for ensuring foundational models conform to diverse human preferences. Current research in this field typically involves either multiple policies or multiple reward models customized for various preferences, or the need to train a preference-specific supervised fine-tuning (SFT) model. In this work, we introduce a novel multi-objective alignment method, MOSLIM, which utilizes a single reward model and policy model to address diverse objectives. MOSLIM provides a flexible way to control these objectives through prompting and does not require preference training during SFT phase, allowing thousands of off-the-shelf models to be directly utilized within this training framework. MOSLIM leverages a multi-head reward model that classifies question-answer pairs instead of scoring them and then optimize policy model with a scalar reward derived from a mapping function that converts classification results from reward model into reward scores. We demonstrate the efficacy of our proposed method across several multi-objective benchmarks and conduct ablation studies on various reward model sizes and policy optimization methods. The MOSLIM method outperforms current multi-objective approaches in most results while requiring significantly fewer GPU computing resources compared with existing policy optimization methods.

Figures

Figures reproduced from arXiv: 2505.20336 by the authors.

Figure 1
Figure 1. An overview of MOSLIM(OURS), MORLHF, Rewarded Soups(RSoups) during training [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Reward Model Architecture of MOSLIM In this equation, x represents input prompt, while y denotes the corresponding answer. As research on preference alignment evolved, this training paradigm for reward models is found to obscure significant preference information by adhering solely to a majority voting principle, thereby overlooking the nuances between majority and minority groups (Jang et al. 2023). Li et al. (2024… view at source ↗
Figure 3
Figure 3. Construction process of reward model training datasets. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Ablation study on different DataType . The figure illustrates the classification performance across four DataType . 3.2 POLICY OPTIMIZATION In this section we conduct our policy optimization experiments with the reward models trained in 3.1. All policy models are initi…
Figure 5
Figure 5. Figure 5: Controllability experiment results of preference intensity. From left to right, the subfigures [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Controllability experiment results across preference dimensions. The scores in each di [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Examples of input for policy and reward model [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Construction process of reinforcement learning datasets. Each sample consists of a pref [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Performance comparison between our reward model and GPT-4. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 3 canonical work pages

  1. [1]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gall´e, Marzieh Fadaee, Julia Kreutzer, Ahmet ¨Ust¨un, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740,

    arXiv

  2. [3]

    Abhimanyu Dubey et al

    URL https://arxiv.org/abs/2310.01377. Abhimanyu Dubey et al. The llama 3 herd of models. ArXiv, abs/2407.21783,

    arXiv

  3. [4]

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al

    URL https: //api.semanticscholar.org/CorpusID:271571434. Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024a. Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Hu...

    arXiv

  4. [5]

    Aligning to thousands of preferences via system message generalization

    Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to thousands of preferences via system message generalization. ArXiv, abs/2405.17977,

    arXiv

  5. [6]

    Aligning Crowd Feedback via Distributional Preference Reward Modeling

    URL https: //api.semanticscholar.org/CorpusID:270067579. Dexun Li, Cong Zhang, Kuicai Dong, Derrick-Goh-Xin Deik, Ruiming Tang, and Yong Liu. Align- ing crowd feedback via distributional preference reward modeling.ArXiv, abs/2402.09764, 2024a. URL https://api.semanticscholar.org/CorpusID:267681742. Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Wayne Xin...

    work page Pith review arXiv

  6. [7]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L

    URL https://api.semanticscholar.org/CorpusID:270067818. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lo...

    arXiv

  7. [9]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D

    URL https://api.semanticscholar.org/CorpusID:269484177. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. ArXiv, abs/2305.18290,

    arXiv

  8. [11]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    URL https://arxiv.org/abs/2311.16119. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proxi- mal policy optimization algorithms. ArXiv, abs/1707.06347,

    arXiv

Show all 15 references
  1. [14]

    13 Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao

    URL https://arxiv.org/ abs/2306.05685. 13 Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Be- yond one-preference-fits-all alignment: Multi-objective direct preference optimization. In An- nual Meeting of the Association for Computational L...

    arXiv

  2. [15]

    In the Table 5 in Ap- pendix B, <preference n > represents a preference intensity of n (1 ≤ n < nmax), where a larger n indicates a higher intensity

    To standardize these scores, we define different preference intensity ranges and partition the data accordingly. In the Table 5 in Ap- pendix B, <preference n > represents a preference intensity of n (1 ≤ n < nmax), where a larger n indicates a higher intensity. Finally, based...

    2023

  3. [2017]

    semanticscholar.org/CorpusID:28695052

    URL https://api. semanticscholar.org/CorpusID:28695052. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feed- back. ArXiv, abs/2009.01325,

    2009 arXiv

  4. [2020]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang

    URL https://api.semanticscholar.org/ CorpusID:221665105. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts.ArXiv, abs/2406.12845, 2024a. URL https://api.semanticscholar.org/Cor...

    arXiv

  5. [2022]

    Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, and Asuman E

    URL https://api.semanticscholar.org/ CorpusID:246426909. Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, and Asuman E. Ozdaglar. Rlhf from heterogeneous feedback via personalization and preference aggregation. ArXiv, abs/2405.00254,

    arXiv

  6. [2023]

    Alexandre Ram ´e, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord

    URL https://api.semanticscholar.org/ CorpusID:258959321. Alexandre Ram ´e, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diver...

    arXiv

  7. [2024]

    Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dassarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.