Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Offline RLHF and DPO reduce to one noisy logistic-regression problem, and the order of privacy and corruption determines how hard that problem is.

desk verdict A genuinely useful reduction paper with a clean CTL/LTC separation; the suboptimality bounds need one more assumption stated explicitly before the claims as written hold. read the letter →

arxiv 2505.15694 v1 pith:GFVGZIKG submitted 2025-05-21 cs.LG cs.AI

Xingyu Zhou , Yulian Wu , Francesco Orabona This is my paper
classification cs.LGcs.AI
keywords offlinealignmentRLHFDPOlocaldifferentialprivacyadversariallabelcorruptionlogisticregressionBradley-Terrymodelrelativeconditionnumber
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that offline alignment of language models—whether done through reward-then-optimize RLHF or through direct DPO—is at bottom one statistical problem: estimating a logistic-regression parameter from preference labels that have been both privatized and adversarially corrupted. Under linear reward or log-linear policy assumptions, the preference labels are exactly logistic-regression labels, so suboptimality bounds for both paradigms follow from a single estimation bound. The paper's central discovery is that the order of operations matters: if labels are privatized first and corrupted second (LTC), the error carries an extra privacy-dependent factor $c(\varepsilon)$ compared with corruption-then-privacy (CTL), making LTC the harder setting. The same reduction yields the first $O(1/\sqrt{n})$ suboptimality rate for private or corrupted DPO, and $O(\sqrt{\alpha})$ robustness for RLHF under label corruption.

What carries the argument

The load-bearing mechanism is the reduction of preference learning to logistic regression. Under the Bradley-Terry model with a linear reward, RLHF labels satisfy the logistic model with $\theta_{\mathrm{true}}=\theta^\star$ and $x_i=\phi(s_i,a_i^1)-\phi(s_i,a_i^0)$; under the log-linear policy class, DPO labels satisfy it with $\theta_{\mathrm{true}}=\beta(\theta^\star-\theta_{\mathrm{sft}})$. The estimation side is carried by a new loss $\tilde{\ell}_i(\theta)=\ln(1-\sigma(\theta^\top x_i))+(z_i+\sigma(\varepsilon)-1)c(\varepsilon)\theta^\top x_i$ with $c(\varepsilon)=(e^\varepsilon+1)/(e^\varepsilon-1)$, which debiases randomized-response labels and is agnostic to which noise order generated them. The suboptimality translation is controlled by the relative condition number $\kappa(\pi^\dagger,\pi_{\mathrm{ref}})$ for RLHF and its all-policy analogue $\kappa_{\Pi}$ for DPO, which measure how well the comparator or the whole policy class is covered by the SFT sampling distribution.

What would settle it

Simulate logistic-regression labels with $d=20$, features $x_i$ drawn from a behavior policy different from $\pi_{\mathrm{sft}}$, apply randomized response and then adversarial corruption, and run Algorithm 1; if $\|\hat{\theta}-\theta_{\mathrm{true}}\|_2$ keeps growing or fails to match the predicted $1/\sqrt{n}$ concentration as $n$ grows, the covariance identification premise is false. Alternatively, compare Algorithm 1 under LTC and CTL with $\varepsilon$ fixed: if the empirical suboptimality gap does not grow with $c(\varepsilon)$, the separation claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that privacy and corruption in preference labels do not need separate theories: both RLHF and DPO, under Assumptions 4.1 and 4.5, reduce to learning $\theta_{\mathrm{true}}$ in the logistic model $P(y=1\mid x)=\sigma(\langle\theta_{\mathrm{true}},x\rangle)$, where $x_i=\phi(s_i,a_i^1)-\phi(s_i,a_i^0)$. Algorithm 1, a randomized-response privatizer followed by a shifted-and-scaled logistic loss, estimates this parameter with the same algorithm whether corruption precedes privacy (CTL) or follows it (LTC). The resulting bounds show LTC pays an extra $c(\varepsilon)$ factor in the corruption and bias terms, establishing a separation between the two noise orders for both RLHF and DPO. As corollaries, the paper obtains explicit suboptimality guarantees in terms of relative condition numbers, the first $O(1/\sqrt{n})$ rate for robust or private DPO, and robustness rates matching the best known in standard offline reinforcement learning.

Load-bearing premise

The whole argument collapses if the preference data are not generated the way the proof assumes: the two actions in each sample must be drawn from the SFT or reference policy $\pi_{\mathrm{sft}}$ so the empirical covariance equals the population matrix $\Sigma^{\mathrm{diff}}_{\pi_{\mathrm{sft}},\pi_{\mathrm{sft}}}$, and both $\pi^\star$ and $\pi_{\mathrm{sft}}$ must be exactly representable in the log-linear class.

Editorial extensions

If this is right

  • If the reduction is correct, RLHF under strong label corruption achieves a degradation of order $O(\sqrt{\alpha})$ (and $O(\alpha/\xi)$ under uniform coverage), matching the best known corruption-robust offline reinforcement learning bounds.
  • If the reduction is correct, private or corrupted DPO attains an $O(1/\sqrt{n})$ suboptimality rate, closing the gap to the noise-free rate and improving the previously known $O(1/n^{1/4})$.
  • If the reduction is correct, LTC is strictly harder than CTL by a factor $c(\varepsilon)$ in the corruption term, so privacy protection should be applied after data cleaning whenever feasible.
  • If the reduction is correct, a single algorithm that does not know whether privacy or corruption came first achieves the stated bounds in all scenarios, because its loss is an unbiased proxy for the standard log-loss under randomized response.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The covariance identification step implicitly assumes the two actions in each preference pair are drawn from $\pi_{\mathrm{sft}}$; if deployment data are collected under a different behavior policy, the empirical covariance may not concentrate around $\Sigma^{\mathrm{diff}}_{\pi_{\mathrm{sft}},\pi_{\mathrm{sft}}}$, and the relative-condition-number bounds would not follow.
  • The logistic-regression reduction hinges on exact log-linear realizability of both $\pi^\star$ and $\pi_{\mathrm{sft}}$; without that realizability, the misspecification error falls outside the analysis, although the same shifted-and-scaled loss is still a sensible heuristic, as the paper's neural-net experiments indicate.
  • The separation between LTC and CTL is not an artifact of the linear model: the same ordering appears in the paper's GPT2-large experiments, suggesting it may be a general phenomenon worth testing in larger preference datasets.
  • A sharper editor-level conjecture, which the paper itself raises as an open direction, is that the $c(\varepsilon)$ separation under the $\sqrt{\alpha}$ regime may be improvable to $\sqrt{c(\varepsilon)}$.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper develops a unified reduction of offline RLHF and DPO, under a linear reward model (Assumption 4.1) and a log-linear policy class (Assumption 4.5), to parameter estimation in logistic regression with privately and adversarially corrupted labels. It proposes a single analyzer based on a shifted-and-scaled log-loss paired with randomized response, proves weighted-norm estimation error bounds under CTL and LTC (Theorem 5.4) and an L2-norm bound under uniform coverage (Theorem 5.7), and converts these into suboptimality bounds for RLHF (Theorems 6.1, 6.2) and DPO (Theorem 6.3). The paper's headline results include a separation between LTC and CTL in which LTC carries an extra c(epsilon) factor, improved root-alpha corruption dependence for robust RLHF, and the first O(1/sqrt(n)) rates for private or robust DPO under the log-linear policy class. The appendix contains detailed proofs and a claimed diagnosis of a sign-direction error in Mandal et al. (2024).

Significance. If the results hold as stated, the reduction framework provides an elegant unifying perspective on private and robust offline alignment, and the CTL/LTC separation is conceptually valuable. The paper ships detailed appendix proofs, a clean decomposition of the gradient into privacy, corruption, and standard logistic-regression terms, and a concrete, apparently sound identification of a sign-direction gap in prior work on robust RLHF. The improved corruption dependence for RLHF and the new O(1/sqrt(n)) DPO rates are meaningful advances over the existing state of the art. However, the headline suboptimality guarantees currently rest on an unstated data-generation assumption; this is a load-bearing gap rather than a presentation issue, and it must be addressed before the central claims can be accepted as proven.

major comments (2)
  1. [Section 3, Corollary 4.4, Proposition 4.6, Theorems 6.1 and 6.3] The suboptimality bounds in Corollary 4.4, Proposition 4.6, and hence Theorems 6.1 and 6.3 require that the empirical covariance bSigma concentrates around the population matrix Sigma^diff_{pi_sft,pi_sft} used in the relative condition numbers kappa(pi_dagger,pi_ref) and kappa_Pi. This is used in step (a) of Corollary 4.4 via Lemma H.1 and asserted in the proof of Proposition 4.6 with the sentence 'Sigma^diff_{pi_sft,pi_sft} is the corresponding population matrix of bSigma'. However, Section 3 only specifies that states s_i are i.i.d. from rho; no assumption is made about how the two actions a_i^0, a_i^1 are drawn, and neither Assumption 4.1 nor Assumption 4.5 states that they are sampled from pi_sft. If the actions are generated by a different behavior policy pi_b, then bSigma concentrates around Sigma^diff_{pi_b,pi_b}, the covariance replacement in Lemma H.1 is invalid, and the displayed bounds with kappa(pi_dagger,pi_ref) or kappa_Pi do not follow; the suboptimality may depend on a different, potentially much larger concentrability coefficient. This issue is load-bearing for the paper's main suboptimality claims. Please add an explicit data-generation assumption (e.g., a_i^0, a_i^1 ~ pi_sft(·|s_i)) to Section 3 or to Assumptions 4.1/4.5, and verify that all applications of Lemma H.1 are then justified.
  2. [Section 4.2, Proposition 4.6, Theorem 6.3] The reduction for DPO relies on the exact log-linear realizability of both pi_star and pi_sft (Assumption 4.5). The equality P(y_i=1|s_i,a_i^0,a_i^1) = sigma(beta(theta_star - theta_sft)^T x_i) holds only when both policies lie exactly in the class Pi with the known feature map phi; under misspecification, the preference labels do not follow the logistic regression model (3), and the estimation error bounds of Theorem 5.4 no longer connect to the DPO suboptimality. While this is an explicit assumption rather than an internal inconsistency, the paper should state the limitation prominently and clarify that Theorem 6.3 and the claimed O(1/sqrt(n)) rate are contingent on exact realizability. The current text sometimes reads as if the reduction applies to general policy classes, e.g., in the discussion of the rDPO implementation in Appendix D.
minor comments (5)
  1. [Theorem 5.4 and Definition 5.1] There is a typo in the theorem statement and in Definition 5.1: 'alpha, in [0, 1/2)' should be 'alpha in [0, 1/2)'.
  2. [Section 6.1 and Appendix C] The notation eO and eTheta is used without definition; please define soft-O notation when first used.
  3. [Appendix D] The experiments introduce corruption as randomly flipped labels with probability alpha, whereas the theoretical results in Theorems 5.4, 6.1, and 6.3 are for the strong adversarial corruption model of Definition 3.3. The paper should state explicitly that the experiments serve only as a sanity check for the qualitative separation and do not directly test the adversarial-corruption bounds.
  4. [Section 6.1, comparison with Mandal et al.] The comparison with Mandal et al. (2024) claims their corruption dependence is O(alpha^{1/4}) when reduced from MDP to bandit, while the present label-only setting achieves O(sqrt(alpha)). Since Mandal et al. consider corruption of both features and labels in a linear MDP, the claim that the present setting is 'easier' and the reduction of their bound to the bandit setting deserve a more careful statement; as written, the comparison could be read as comparing incomparable settings.
  5. [Section 4.2, Remark 4.8] The remark contrasts single-policy and all-policy concentrability, but the distinction is only meaningful after the missing action-sampling assumption is added; the wording should be updated to reference the assumed data-generating distribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the estimation and suboptimality bounds follow from self-contained first-order optimality and concentration arguments, with self-citations used only for comparison and conjectured tightness.

full rationale

The derivation chain is self-contained. Theorem 5.4's estimation bounds are obtained by minimizing the shifted/scaled loss (14), expanding around the true parameter, lower-bounding the Hessian by gamma, and bounding the gradient via the sub-Gaussian and corruption claims E.1-E.4; no parameter is fitted to a quantity that is later renamed a prediction. Propositions 4.2 and 4.6 reduce RLHF and DPO suboptimality to this estimation error by direct algebra plus covariance concentration, and the relative-condition-number transfer is a mathematical consequence of Lemma H.1 rather than an input. The only fragile point is an unstated data-generating assumption: the proof of Proposition 4.6 asserts that 'Sigma^diff_{pi_sft,pi_sft} is the corresponding population matrix of bSigma', while Section 3 and Assumptions 4.1 and 4.5 do not specify that the two actions in each preference sample are drawn from pi_sft; without that hypothesis, Lemma H.1 does not connect bSigma to the kappa denominators in Corollary 4.4 or Theorem 6.3. That is a missing-assumption/correctness gap, not a circular reduction, because it does not make any conclusion equal to an input by construction. Self-citations (Chowdhury et al. 2023 and Zhou & Zhang 2024) appear in related work, comparisons, and tightness remarks; deleting them would leave the main proofs unchanged, so they are not load-bearing.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claim rests on linear or log-linear realizability and on an unstated sampling assumption; no parameters are fitted to data. The theory uses standard concentration tools from cited literature. No new entities are postulated.

assumptions (6)
  • domain assumption Bradley-Terry preference model (Equation 1) generates the true labels.
    Used in Section 3 and all reductions; if human preferences are not Bradley-Terry, the logistic-regression form and all subsequent bounds fail.
  • ad hoc to paper Linear reward model with bounded features (Assumption 4.1) for RLHF.
    Powers Proposition 4.2; no guarantees outside this class, and the paper acknowledges linearity is for theoretical analysis.
  • ad hoc to paper Log-linear policy realizability for both optimal and SFT policies (Assumption 4.5) for DPO.
    Needed so DPO labels become logistic regression; misspecification breaks Proposition 4.6 and Theorem 6.3.
  • domain assumption Preference actions are drawn from πsft so bΣ tracks Σdiff_{πsft,πsft}.
    Used in Corollary 4.4 proof step (a) and in the final step of Proposition 4.6; not stated in Assumption 4.1 or 4.5.
  • domain assumption Strong label corruption model with label-only changes (Definition 3.3) and randomized response as the local randomizer (Remark 3.2).
    The adversary corrupts only labels, not features, and privacy uses randomized response; changing either mechanism changes the bounds.
  • standard math Matrix concentration inequalities from cited literature (Lemmas H.1, H.2, H.3).
    Used throughout the proofs; accepted as external background results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO." pith.science (2026). https://pith.science/paper/GFVGZIKG

@misc{pith2026250515694,
  author       = {Pith},
  title        = {Pith review of: A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GFVGZIKG}},
  note         = {Machine review of arXiv:2505.15694}
}
read the original abstract

In this paper, we theoretically investigate the effects of noisy labels in offline alignment, with a focus on the interplay between privacy and robustness against adversarial corruption. Specifically, under linear modeling assumptions, we present a unified analysis covering both reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) under different privacy-corruption scenarios, such as Local differential privacy-then-Corruption (LTC), where human preference labels are privatized before being corrupted by an adversary, and Corruption-then-Local differential privacy (CTL), where labels are corrupted before privacy protection. Our analysis leverages a reduction framework that reduces the offline alignment problem under linear modeling assumptions to parameter estimation in logistic regression. This framework allows us to establish an interesting separation result between LTC and CTL, demonstrating that LTC presents a greater challenge than CTL in offline alignment, even under linear models. As important by-products, our findings also advance the state-of-the-art theoretical results in offline alignment under privacy-only or corruption-only scenarios.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.

Reference graph

Works this paper leans on

27 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Claim E.4

    and λ >0, with probability at least 1 − δ, 1 n nX i=1 ηixi 2 ≤ C · σ · r 1 + ln(1/δ) n , for some universal constant C. Claim E.4. Let b = (b1, . . . , bn) be a vector that at least 1 − αn elements are zero, and the rest are bounded by some constant ζ >0, i.e., |bi| ≤ζ. Then, we have 1 n nX i=1 bixi 2 ≤ ζα . We are left to establish (25) for CTL, LTC, and...

    work page 2023

  2. [2]

    Thus, by Lemma H.2, we have with probability at least 1 − δ, 1 n nX i=1 ηixi 2 ≤ C · σ · r 1 + ln(1/δ) n , for some universal constant C >0

    ≤ 1 n2 , ∥M ∥ ≤1 n . Thus, by Lemma H.2, we have with probability at least 1 − δ, 1 n nX i=1 ηixi 2 ≤ C · σ · r 1 + ln(1/δ) n , for some universal constant C >0. Proof of Claim E.4. This simply holds by algebra: 1 n nX i=1 bixi 2 ≤ 1 n nX i=1 ∥xi∥ |bi| ≤ζα, which holds by the boundedness assumption of ∥xi∥ ≤1. G. Discussion on the Gap in Prior Work As we ...

    work page 2024

  3. [5]

    Finally, we use the win rate from these comparisons as our primary performance metric, following the methodology outlined in the DPO paper (Rafailov et al., 2023)

    To assess performance, we employ the llama3:70b model as a judge, comparing responses from πDPO and πrDPO against those from πSFT. Finally, we use the win rate from these comparisons as our primary performance metric, following the methodology outlined in the DPO paper (Rafailov et al., 2023). We compute the average and standard deviation across five seed...

    work page 2023

  4. [7]

    Manipulation attacks in local differential privacy

    Cheu, A., Smith, A., and Ullman, J. Manipulation attacks in local differential privacy. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 883–900. IEEE,

    work page 2021

  5. [8]

    Differentially Private Reward Estimation with Preference Feedback

    Chowdhury, S. R., Zhou, X., and Natarajan, N. Differen- tially private reward estimation with preference feedback. arXiv preprint arXiv:2310.19733,

    work page Pith review arXiv

  6. [9]

    R., Kini, A., and Natarajan, N

    Chowdhury, S. R., Kini, A., and Natarajan, N. Provably robust DPO: Aligning language models with noisy feed- back. arXiv preprint arXiv:2403.00409,

    arXiv

  7. [10]

    C., Jordan, M

    9 A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO Duchi, J. C., Jordan, M. I., and Wainwright, M. J. Local privacy and statistical minimax rates. In 2013 IEEE 54th annual symposium on foundations of computer science, pp. 429–438. IEEE,

    work page 2013

  8. [12]

    M., and Zhang, T

    Hsu, D., Kakade, S. M., and Zhang, T. A tail inequality for quadratic forms of subgaussian random vectors. arXiv preprint arXiv:1110.2842,

    arXiv

Show all 27 references
  1. [14]

    Corruption robust offline reinforce- ment learning with human feedback

    Mandal, D., Nika, A., Kamalaruban, P., Singla, A., and Radanovi´c, G. Corruption robust offline reinforce- ment learning with human feedback. arXiv preprint arXiv:2402.06734,

    arXiv

  2. [15]

    Dueling RL: reinforce- ment learning with trajectory preferences

    Pacchiano, A., Saha, A., and Lee, J. Dueling RL: reinforce- ment learning with trajectory preferences. arXiv preprint arXiv:2111.04850,

    arXiv

  3. [16]

    The importance of online data: Understanding prefer- ence fine-tuning via coverage

    Song, Y ., Swamy, G., Singh, A., Bagnell, D., and Sun, W. The importance of online data: Understanding prefer- ence fine-tuning via coverage. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists,

    2024

  4. [19]

    D., and Sun, W

    Zhan, W., Uehara, M., Kallus, N., Lee, J. D., and Sun, W. Provable offline preference-based reinforcement learning. arXiv preprint arXiv:2305.14816,

    arXiv

  5. [20]

    M., Stiennon, N., Wu, J., Brown, T

    Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593,

    1909 arXiv

  6. [21]

    Additional Related Work We discuss here more relevant work that do not fit in the main text

    11 A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO A. Additional Related Work We discuss here more relevant work that do not fit in the main text. In addition to the work discussed below, we refer readers to Huang et al. (2024) for theo...

    2024

  7. [22]

    to the setting of offline RLHF, where only relative rankings, rather than true rewards, are observed. As will be discussed in Appendix C, the lower bounds established for robust offline RL, along with their proof techniques, can be applied or adapted to derive lower bounds for...

    2022

  8. [23]

    rejected

    under both privacy and corruption with a log-linear policy class (cf. Theorem 6.3). In this section, we would like to empirically demonstrate its performance with a general function class, i.e., neural nets. 1Note that the hard instance in Zhang et al. (2022) only requires cor...

    2022

  9. [27]

    Chosen” and “Rejected

    Notice that due to the negative sign, the inequality direction should be reversed. In order to have the right direction, we need to introduce γ, which in turn introduces 1/γ in the final bound. H. Auxiliary Results Lemma H.1 (Concentration of Covariances, Lemma 39 in (Zanette ...

    2021

  10. [1952]

    Robust reinforcement learning from corrupted human feedback

    Bukharin, A., Hong, I., Jiang, H., Zhang, Q., Zhang, Z., and Zhao, T. Robust reinforcement learning from corrupted human feedback. arXiv preprint arXiv:2406.15568,

    arXiv

  11. [1965]

    J., Krishnamurthy, A., Rosset, C., Awadal- lah, A., and Rakhlin, A

    Xie, T., Foster, D. J., Krishnamurthy, A., Rosset, C., Awadal- lah, A., and Rakhlin, A. Exploratory preference optimiza- tion: Harnessing implicit Q*-approximation for sample- efficient RLHF. arXiv preprint arXiv:2405.21046,

    arXiv

  12. [2011]

    D., Sun, W., Krishna- murthy, A., and Foster, D

    Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krishna- murthy, A., and Foster, D. J. Correcting the mythos of KL-regularization: Direct alignment without overparame- terization via Chi-squared preference optimization. arXiv preprint arXiv:2407.13399,

    arXiv

  13. [2014]

    R., Yun, H., Teo, C

    Feng, Q., Kasa, S. R., Yun, H., Teo, C. H., and Bodapati, S. B. Exposing privacy gaps: Membership inference attack on preference data for LLM alignment. arXiv preprint arXiv:2407.06443,

    arXiv

  14. [2019]

    Classification under misspecification: Halfspaces, generalized linear models, and connections to evolvability

    Chen, S., Koehler, F., Moitra, A., and Yau, M. Classification under misspecification: Halfspaces, generalized linear models, and connections to evolvability. arXiv preprint arXiv:2006.04787,

    2006 arXiv

  15. [2020]

    K., Ramnath, K., Chaudhuri, S., Mehrotra, S., Mao, X.-B., Asur, S., et al

    10 A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO Wang, Z., Bi, B., Pentyala, S. K., Ramnath, K., Chaudhuri, S., Mehrotra, S., Mao, X.-B., Asur, S., et al. A comprehen- sive survey of LLM alignment techniques: RLHF, RLAIF, PPO, DPO and...

    arXiv

  16. [2021]

    Trimmed maxi- mum likelihood estimation for robust learning in gener- alized linear models

    Awasthi, P., Das, A., Kong, W., and Sen, R. Trimmed maxi- mum likelihood estimation for robust learning in gener- alized linear models. arXiv preprint arXiv:2206.04777,

    arXiv

  17. [2022]

    Training a helpful and harmless assistant with reinforce- ment learning from human feedback

    Bai, Y ., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernan- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt...

    arXiv

  18. [2023]

    Value-incentivized preference optimization: A unified approach to online and offline RLHF

    Cen, S., Mei, J., Goshvadi, K., Dai, H., Yang, T., Yang, S., Schuurmans, D., Chi, Y ., and Dai, B. Value-incentivized preference optimization: A unified approach to online and offline RLHF. arXiv preprint arXiv:2405.19320,

    arXiv

  19. [2024]

    Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T., Marks, S., Segerie, C.-R., Carroll, M., Peng, A., Christoffersen, P., Damani, M., Slocum, S., Anwar, U., Siththaranjan, A., Nadeau, M., Michau...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.