Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Data-adaptive Safety Rules for Training Reward Models

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that labeling preference data with the few rules on which the two responses differ most is provably the most informative choice, and an 8B reward model built this way tops the safety leaderboard.

desk verdict The adaptive rule-selection idea and ablation are genuinely useful, but the paper's theoretical guarantee (Theorem 3.4) is invalid as proven, so it needs major revisions before its claims can be trusted. read the letter →

arxiv 2501.15453 v2 pith:JPCERWN6 submitted 2025-01-26 cs.CL

Xiaomin Li , Mingye Gao , Zhiwei Zhang , Jingxuan Fan , Weiyu Li This is my paper · ORCID
classification cs.CL
keywords rewardmodelssafetyalignmentrule-basedannotationmax-discrepancyselectionmutualinformationRLHFpreferencelearningdata-adaptivelabeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that, when reward models are trained from fine-grained rule-based safety annotations, the best rules to apply are the ones on which the two candidate responses differ most. The authors prove that max-discrepancy rule selection maximizes the mutual information between the resulting rule-based preference label and the hidden ground-truth preference, assuming each rule's vote is conditionally independent given that truth. They train a small Rule Adapter to pick the five most discrepant and relevant rules for each trio, use an LLM judge to rate responses on those rules, and train an 8B reward model, RAMO, on just 1K labeled trios. On the RewardBench safety benchmark this model scored 95.1, the top safety score as of January 25, 2025, and using it as the reward model in PPO also improved policy safety. The claim matters because it offers an automatic, interpretable way to label preference data for safety alignment while making far fewer annotation calls than scoring every rule.

What carries the argument

The Rule Adapter is a multi-label classifier built on a small instruction-tuned language model; it maps a prompt and two responses to the five rules with the largest rating discrepancy, regularized by cosine relevance between each rule and the prompt. The mathematical engine is the identity $I(T; H) = D_{\mathrm{JS}}(P^+ \parallel P^-)$, combined with the fact that the Jensen-Shannon divergence between $\mathrm{Bern}(\sigma(d))$ and $\mathrm{Bern}(\sigma(-d))$ increases strictly with $|d|$. Under conditional independence of rule votes, this makes selecting the largest-$|d_i|$ rules equivalent to maximizing $I(T_s; H)$, and the reward model RAMO is then trained with the Bradley-Terry loss on the resulting binary preferences.

What would settle it

Take a preference dataset whose true labels are known, compute the joint mutual information $I(T_s; H)$ for all $r$-subsets of rules, and check whether the max-discrepancy subset achieves the maximum; a counterexample would settle the claim. A simpler check is to measure whether rule votes remain independent conditional on the hidden truth, since substantial residual correlation would invalidate the additivity step.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a selection-to-information equivalence: given a budget of $r$ rules, the $r$ rules with the largest absolute score differences between the two responses are exactly the rules that maximize the mutual information $I(T_s; H)$ between the aggregated rule-based preference label and the hidden ground-truth label $H$. The proof routes through Jensen-Shannon divergence: each rule's conditional vote distribution is modeled as $P_i^+ \sim \mathrm{Bern}(\sigma(d_i))$ and $P_i^- \sim \mathrm{Bern}(\sigma(-d_i))$, whose divergence is $\log 2 - H(\sigma(d_i))$, strictly increasing in $|d_i|$, so under conditional independence the additive mutual information is maximized by picking the largest $|\psi_i(v_A) - \psi_i(v_B)|$. In practice the selection adds a relevance term $\gamma \cdot \mathrm{sim}(x, u_i)$ and is distilled into a 3B Rule Adapter that outputs five rules per trio; the resulting 8B reward model attains 95.1 on RewardBench-Safety.

Load-bearing premise

Everything rests on assuming that, once you know which answer is truly better, each rule's vote is independent of every other rule's vote; if real rules vote in correlated ways, the proof's key additivity step does not go through, and the theorem also does not cover the relevance-weighted selection actually used in the experiments.

Editorial extensions

If this is right

  • If the equivalence holds, any reward-model training pipeline can cut its annotation budget to a handful of adaptively chosen rules without losing information about the true preference.
  • Rule-based labels produced this way can replace or refine human labels: in the paper's experiments, reward models trained on a human-preference dataset relabeled by the Rule Adapter outperformed models trained on the original human labels.
  • Because the Rule Adapter is small and trained once, the labeling step becomes cheaper and more interpretable than repeatedly querying a large judge model.
  • The PPO results imply that safety-critical RLHF fine-tuning can use a data-adaptively labeled reward model to improve policy safety from a small prompt budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the same selection rule should transfer to non-safety multi-criteria settings such as helpfulness, factuality, or reasoning, but the conditional-independence assumption may need checking because those criteria often overlap more than safety rules do.
  • A testable extension the paper does not develop is a relevance-regularized version of the theorem: the proof covers the unregularized objective, while the deployed system optimizes the regularized objective, so the guarantee is not directly about the exact method used in experiments.
  • Because the label chain begins with an LLM judge's ratings, the method inherits any judge bias; one could turn the Rule Adapter into an audit tool that surfaces which rules drive each preference judgment, an application the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a data-adaptive rule-selection method for reward-model training. Given a large pool of safety rules, it selects, for each trio of prompt and two responses, the r rules with the largest rating discrepancy between the two responses, augmented by a prompt-relevance regularization term. A 3B Rule Adapter is trained to imitate this selection on 63K pairwise comparisons, and the resulting 5 selected rules are used to label 1K preference trios, from which an 8B reward model (RAMO) is trained. The paper reports a RewardBench safety score of 95.1, the highest on the leaderboard as of January 25, 2025, along with ablations against random, fixed, all-rule, GPT-selected, and direct-GPT-labeling baselines, a generalization experiment on HH-RLHF, and a PPO alignment experiment. The central theoretical claim is Theorem 3.4, stating that max-discrepancy selection maximizes the mutual information between rule-based preference labels and a hidden ground-truth label.

Significance. If the theoretical result and the empirical results hold, the paper makes a useful contribution: it offers an interpretable, low-cost alternative to full rule enumeration for fine-grained reward-model annotation, and it releases the rule pool, synthetic dataset, Rule Adapter, and RAMO. The empirical ablation is informative: RAMO beats dynamic-random, fixed, all-rule, and GPT-selected baselines, and the HH-RLHF relabeling experiment shows consistent gains over human labels in the paper's settings. The main reservation is that the advertised theoretical guarantee is central to the paper's framing, and the proof of Theorem 3.4 contains a load-bearing error. The empirical results may stand independently, but the paper's claim to have 'demonstrated theoretically' that max-discrepancy selection maximizes mutual information is not currently supported.

major comments (3)
  1. [Appendix A, proof of Theorem 3.4] The proof asserts that under conditional independence of the rule votes T_i given H, the mutual information decomposes as I(T_s; H) = sum_{i in I_s} I(T_i; H). This identity is false. Conditional independence given H makes conditional entropies additive, H(T_s | H) = sum_i H(T_i | H), but it does not make marginal entropies additive, so I(T_s; H) = H(T_s) - H(T_s | H) is not generally equal to sum_i [H(T_i) - H(T_i | H)]. Under the paper's own model, taking H uniform and T_i independent given H with P(T_i = +1 | H = +1) = P(T_i = -1 | H = -1) = sigma(d_i), setting d_1 = d_2 = 2 gives I(T_1; H) = I(T_2; H) ≈ 0.473 bits, while I(T_1, T_2; H) ≈ 0.687 bits rather than the claimed 0.946 bits. Therefore the proof's reduction of the joint mutual information to a sum of per-rule Jensen-Shannon terms is invalid, and Theorem 3.4 is not established as stated.
  2. [Appendix A, assumptions of Theorem 3.4] The proof introduces two modeling assumptions that do not appear in the theorem statement and are not validated against any preference data: (i) the rule labels T_i are conditionally independent given the hidden label H, and (ii) each rule's vote satisfies P(T_i = +1 | H = +1) = P(T_i = -1 | H = -1) = sigma(psi_i(v_A) - psi_i(v_B)). These are substantive behavioral assumptions about how rule-based raters relate to the hidden ground truth. The theorem as stated claims a general equivalence between max-discrepancy selection and maximum mutual information; if the theorem is to be retained, these conditions must be stated as hypotheses, and the paper should provide evidence that the sigmoid conditional-independence model is a reasonable description of the actual rule-rating process.
  3. [Section 3.3 (Eq. 12) and Section 3.4 (Theorem 3.4, Eq. 15)] The actual selection method used in the experiments, including the Rule Adapter training and the reward-model data labeling, is the regularized objective of Eq. (12), which contains the term gamma * sim(x, u_i) with gamma = 2 chosen in Appendix G.1.2. Theorem 3.4, however, addresses only the unregularized objective of Eq. (11). Even if the additivity problem in Appendix A were repaired, the theorem would not justify the exact method evaluated. The paper should either extend the theoretical analysis to cover the relevance regularization or explicitly restrict the theoretical claim to the unregularized surrogate.
minor comments (6)
  1. [Appendix A, Definition A.1] The displayed definition of conditional entropy H(U | V) is missing the minus sign: it should be H(U | V) = -E_{(u,v) ~ P(U,V)} log P_{U|V}(u | v). The subsequent entropy calculations use the correct sign, but the displayed definition is mathematically wrong.
  2. [Appendix A, proof of Theorem 3.4] The proof contains internal citation inconsistencies: it refers to 'Lemma A.4' and 'Lemma A.5', but the relevant lemmas are named Lemma A.5 and Lemma A.6. In addition, the additivity equation writes I(T_s; H) = sum_{i in I_s} I(T_i; Y), using Y where the theorem uses H.
  3. [Theorem 3.4, Eq. (15)] Equation (15) writes psi(v_B) without the rule index; it should be psi_i(v_B), matching the discrepancy term |psi_i(v_A) - psi_i(v_B)| used in Eq. (11).
  4. [Appendix D] Definition 3.1 defines psi_i as taking values in [0, 1], but the actual implementation described in Appendix D uses the score P(Yes) - P(No) with range [-1, 1]. The authors note that shifting and scaling does not affect comparisons, but the notation should be made consistent for clarity.
  5. [Appendix G.1.4, Table 9] Table 9's caption reads 'xxxx', which appears to be an unfinished placeholder. The hyperparameter table should be given a proper caption and, ideally, a description of the number of repeated runs.
  6. [Section 5, Table 1] The RAMO row reports a single set of scores with no indication of whether these are from a single run or averaged over seeds. Since Table 2 reports three-trial averages and Table 3 reports two-seed averages with standard deviations, the paper should state the number of runs for the headline Table 1 result.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 3.4's optimality claim reduces to the paper's own generative assumption that each rule votes with probability σ(d_i); the empirical leaderboard result is not circular.

  1. self definitional [Section 3.4, Theorem 3.4 (Eq. 15) and Appendix A, Proof of Theorem 3.4]
    "Based on Bradley-Terry equation 5, we model the conditional distributions of Ti given H as follows, P+i ∼ Bern(σ(di)) and P−i ∼ Bern(σ(−di)) ... Under conditional independence, the mutual information of I(Ts; H) is a sum of individual mutual information: I(Ts; H) = Σ_{i∈Is} I(Ti; Y). ... By Lemma A.5, we know each function DJS(P+i ∥ P−i) is a strictly increasing function of |di|. Therefore the optimal solution happens when we choose s that maximizes those |di|."

    The theorem's conclusion is built into the modeling assumption. The proof defines each rule's vote distribution as Bern(σ(d_i)), so Lemma A.6 gives I(T_i;H) = log(2) − H(σ(d_i)), which is strictly increasing in |d_i|. Thus the per-rule mutual information is a monotone function of the discrepancy by construction, and 'max discrepancy maximizes mutual information' restates the assumed generative model rather than deriving a property of rule-based annotations and hidden ground truth. The asserted additivity of I(T_s;H) under conditional independence is also not what conditional independence implies (it makes H(T_s|H) additive, not I(T_s;H)), so the formal bridge is invalid. In addition, the theorem as stated covers Eq. (11), not the actual relevance-regularized objective in Eq. (12).

full rationale

The central theoretical claim is circular: the advertised equivalence between max-discrepancy selection and maximum mutual information is manufactured by defining each rule-vote variable through σ(d_i), so the proof's conclusion is equivalent to its input assumption. This is a genuine self-definitional reduction of the paper's theoretical guarantee. The empirical core is not circular: RAMO is trained on 1K trios labeled via the Rule Adapter and evaluated on the external RewardBench-Safety benchmark (Table 1), with ablations against random, fixed, all-rule, and GPT-based labeling (Table 2). Those results are self-contained evidence independent of the theorem, and the relevant DPP citation to Li et al. (2024) overlaps with the authors but is used only as a standard tool for selecting diverse rules and is not load-bearing for the circularity. Separately, the proof's additivity identity for mutual information under conditional independence is false, but that is a correctness defect, not an additional circularity. Overall score 6: a central 'prediction' reduces by construction, while the main empirical result remains externally testable.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim rests on several unverified modeling choices: r=5, gamma=2, R=100, prompt filtering thresholds, conditional independence and the sigmoid vote model for the theorem, Bradley-Terry assumptions, and Llama3-70B as ground-truth judge. The only invented conceptual entity is the hidden preference label H.

free parameters (5)
  • Rule budget r = 5
    Chosen by ablation (Appendix G.1.1); used for all rule selection and adapter training. The method depends on this number.
  • Relevance regularization gamma = 2
    Chosen by ablation (Appendix G.1.2, Table 7); Eq. (12) uses gamma to balance discrepancy and prompt-rule similarity, and the theoretical claim only covers gamma=0.
  • Rule pool size R = 100
    GPT-4 generated 400 raw rules and DPP deduplication selected 100; the pool size is an arbitrary design choice that defines the selection problem.
  • Safety relevance prompt threshold = 0.8
    Section C.1: only ShareGPT prompts with Llama3-8B safety-relevance score at least 0.8 were kept; this filters the training distribution.
  • Embedding dedup threshold = 0.8
    Section C.1: prompts with embedding similarity above 0.8 were removed; affects diversity of the synthetic dataset.
assumptions (5)
  • domain assumption Rule labels T_i are conditionally independent given hidden ground-truth label H
    Invoked in Appendix A proof of Theorem 3.4 after noting the rule pool is made 'as orthogonal as possible'; no empirical test is provided, and orthogonality of embeddings does not imply conditional independence of binary rule votes.
  • ad hoc to paper Each rule's vote T_i follows Bern(sigma(d_i)) with d_i equal to the difference in rule ratings
    Introduced in Appendix A solely to make mutual information maximization reduce to max |d_i|. This is not derived from data and makes the theorem's conclusion true by construction.
  • domain assumption Bradley-Terry model for reward model preference probabilities
    Section 3.2, Eq. (5); standard RLHF assumption, accepted in the field.
  • domain assumption H is uniformly distributed and represents an ideal human preference
    Section 3.4 and Appendix A; the hidden ground-truth label is unobservable, and no data verifies its existence or the stated conditional distributions.
  • domain assumption Llama3-70B logits provide valid rule-compliance ratings for synthetic and real data
    Sections 4.2, 4.3 and Appendix D; all labels and selected rule ground truths depend on this judge, with no human validation.
invented entities (1)
  • Hidden ground-truth preference label H
    purpose: Theoretical target in Theorem 3.4; mutual information with H is claimed to be maximized by max-discrepancy selection.
    H is defined as the ideal unobservable human preference; the paper provides no measurement, proxy, or falsifiable handle for H, so the mutual information claim is not testable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data-adaptive Safety Rules for Training Reward Models." pith.science (2026). https://pith.science/paper/JPCERWN6

@misc{pith2026250115453,
  author       = {Pith},
  title        = {Pith review of: Data-adaptive Safety Rules for Training Reward Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JPCERWN6}},
  note         = {Machine review of arXiv:2501.15453}
}
read the original abstract

Reinforcement Learning from Human Feedback (RLHF) is commonly employed to tailor models to human preferences, especially to improve the safety of outputs from large language models (LLMs). Traditionally, this method depends on selecting preferred responses from pairs. However, due to the variability in human opinions and the challenges in directly comparing two responses, there is an increasing trend towards fine-grained annotation approaches that evaluate responses using multiple targeted metrics or rules. The challenge lies in efficiently choosing and applying these rules to handle the diverse range of preference data. In this paper, we propose a dynamic method that adaptively selects the most important rules for each response pair. We introduce a mathematical framework that utilizes the maximum discrepancy across paired responses and demonstrate theoretically that this approach maximizes the mutual information between the rule-based annotations and the underlying true preferences. We then train an 8B reward model using this adaptively labeled preference dataset and assess its efficacy using RewardBench. As of January 25, 2025, our model achieved the highest safety performance on the leaderboard, surpassing various larger models.

Figures

Figures reproduced from arXiv: 2501.15453 by the authors.

Figure 1
Figure 1. Pipeline of our framework. First, we train a Rule Adapter that learns to identify [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bradley-Terry and Multi-Objective Reward Modeling Are Complementary

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Jointly training a Bradley-Terry preference head and a multi-attribute regression head on a shared embedding improves reward-model robustness to reward hacking and boosts multi-objective scoring performance.

Reference graph

Works this paper leans on

31 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    arXiv preprint arXiv:2311.10702 ,

    arXiv

  3. [5]

    {severe level} harm question:

    Index Rule Description 0 Accept the response that are not perceived as rude or demeaning. 1 Accept the response that actively deters self-destructive behavior. 2 Accept the response that best encourages critical thinking. 3 Accept the response that most thoroughly considers consequences. 4 Accept the response that best supports privacy-preserving practice...

    work page 2023

  4. [6]

    SteerLM: At- tribute conditioned SFT as an (user-steerable) alternative to RLHF

    Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. SteerLM: At- tribute conditioned SFT as an (user-steerable) alternative to RLHF. arXiv preprint arXiv:2310.05344 ,

    arXiv

  5. [7]

    Quantile regression for distributional reward models in RLHF

    Nicolai Dorka. Quantile regression for distributional reward models in RLHF. arXiv preprint arXiv:2409.10164,

    arXiv

  6. [9]

    Liao, Kamil˙ e Lukoˇ si¯ ut˙ e, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al

    Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamil˙ e Lukoˇ si¯ ut˙ e, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self- correction in large language models. arXiv preprint arXiv:2302.07459 ,

    arXiv

  7. [10]

    Improving alignment of dialogue agents via targeted human judgements

    Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375 ,

    arXiv

  8. [11]

    Collective Constitutional AI: Aligning a language model with public input

    Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. Collective Constitutional AI: Aligning a language model with public input. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 1395–1417,

    work page 2024

Show all 31 references
  1. [13]

    PKU-SafeRLHF: Towards multi-level safety alignment for llms with human preference

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. PKU-SafeRLHF: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406.15513 ,

    arXiv

  2. [14]

    Mistral 7B

    12 Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825 ,

    arXiv

  3. [15]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 ,

    arXiv

  4. [17]

    RewardBench: Evaluating reward models for language modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. RewardBench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787 ,

    arXiv

  5. [18]

    Rule-based data selection for large language models

    Xiaomin Li, Mingye Gao, Zhiwei Zhang, Chang Yue, and Hong Hu. Rule-based data selection for large language models. arXiv preprint arXiv:2410.04715 ,

  6. [19]

    Skywork-reward: Bag of tricks for reward modeling in LLMs

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in LLMs. arXiv preprint arXiv:2410.18451,

    arXiv

  7. [20]

    Uncertainty-aware reward model: Teaching reward models to know what is unknown

    Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. Uncertainty-aware reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.00847 ,

    arXiv

  8. [22]

    Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization

    Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kiant´ e Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language p...

    arXiv

  9. [23]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 ,

    arXiv

  10. [24]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 ,

    arXiv

  11. [25]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845 , 2024a. Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel ...

    arXiv

  12. [26]

    Helpsteer2: Open-source dataset for training top- performing reward models

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top- performing reward models. arXiv preprint arXiv:2406.08673 , 2024b. Guillaume Wenzek...

    1911 arXiv

  13. [27]

    Safetybench: Evaluating the safety of large language models with multiple choice questions

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309.07045 ,

    arXiv

  14. [28]

    − X t P+(t) logP+(t) # + 1 2

    14 A Proof of Theorem 3.4 Before presenting the main proof of the theorem, we first introduce the Jensen-Shannon divergence DJS(·∥·) below, which is known to be a symmetrized and smoothed version of the Kullback-Leibler divergenceDKL(·∥·) Kullback & Leibler (1951). Utilizing t...

    1951

  15. [29]

    16 Proof of Theorem 3.4

    is strictly increasing and H(p+) is strictly increasing for all p+ ∈ ( 1 2 , 1), the monotonicity is shown and this completes the proof. 16 Proof of Theorem 3.4. Given the preference dataset, we define the random variable H ∈ {0, 1} as the hidden ground truth label that decide...

    2024

  16. [31]

    Results are averaged over 2 trained models with different random seeds for optimal hyperparameter selection

    γ DoNot Answer Refusals Dangerous Refusals Offensive Xstest Should Refuse Xstest Should Respond Safety 0.1 80.1 95.0 98.0 97.4 95.2 93.4 0.5 83.1 95.0 99.0 97.4 96.0 94.2 1 81.6 94.0 99.0 97.4 96.4 93.9 2 88.6 96.5 99.0 97.4 94.4 94.9 10 77.2 95.0 99.0 96.8 94.8 92.6 Table 7: ...

    2024

  17. [1951]

    Specific versus general principles for Constitutional AI

    Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, et al. Specific versus general principles for Constitutional AI. arXiv preprint arXiv:2310.13798 ,

    arXiv

  18. [1952]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 ,

    2005 arXiv

  19. [1975]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models

    AI Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. Meta AI Blog. Retrieved December, 20:2024,

    2024

  20. [2022]

    The Llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  21. [2023]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022a. Yu...

    arXiv

  22. [2024]

    Introduces Claude 3.5 Sonnet with improved performance in intelligence, vision capabilities, and new Artifacts feature

    URL https://www.anthropic.com/news/ claude-3-5-sonnet . Introduces Claude 3.5 Sonnet with improved performance in intelligence, vision capabilities, and new Artifacts feature. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei...

    arXiv

  23. [2025]

    Llama 3.2: Advancing ai on edge and mobile devices

    Meta AI. Llama 3.2: Advancing ai on edge and mobile devices. https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/ , 2024a. Accessed: January 25,

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.