Pith. sign in

REVIEW 3 major objections 5 minor 59 references

Risk-aware Direct Preference Optimization under Nested Risk Measure

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper derives a closed-form risk-aware preference loss in which the Bradley-Terry probability becomes $\sigma(u-\delta)$, with $u$ the DPO implicit reward difference and $\delta$ a sequential risk-ratio correction.

desk verdict Ra-DPO adds a sensible token-level risk penalty to TDPO, but the main theorem's proof swaps a sampled token for an expectation inside a nonlinear risk measure, so the loss is not derived exactly as claimed. read the letter →

arxiv 2505.20359 v2 pith:RAB2QRFB submitted 2025-05-26 cs.LG cs.AI

Lijun Zhang , Lin Li , Yajie Qi , Huizhong Song , Yaodong Yang , Jun Wang , Wei Wei This is my paper
classification cs.LGcs.AI
keywords risk-awaredirectpreferenceoptimizationnestedriskmeasuresconditionalvalue-at-riskentropicmeasuretoken-levelLLMalignmentsequentialratiomodeldriftcontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Direct preference optimization (DPO) aligns language models with human preferences through an implicit reward, but its KL penalty only bounds average divergence from the reference model. This paper tries to make the alignment loss itself risk-aware by replacing the sentence-level penalty with a token-level sequential risk ratio computed through nested risk measures such as CVaR and entropic risk. The central claim is that, under a risk-aware Bellman recursion, the human preference probability factorizes as $\sigma(u - \delta)$, where $u$ is the usual DPO implicit-reward difference and $\delta$ is a difference of sequential risk ratios. The resulting Ra-DPO loss, $-\mathbb{E}[\log\sigma(u-\delta)]$, is a drop-in replacement for the DPO loss that suppresses risky deviations token by token. Experiments on three open-source text-generation benchmarks report higher reward accuracy and lower sequential KL divergence than token-level DPO baselines, so the payoff of the claim is tighter risk control without a separate reward model or RL loop.

What carries the argument

The load-bearing object is the nested risk measure $\Phi_\mu$ embedded in a state-augmented preference-based MDP. Each state is the prompt plus generated prefix $[x,y_{<t}]$, and the Bellman recursion $\tilde Q_\pi([x,y_{<t}], y_t) = \Phi_\mu(\tilde V_\pi([x,y_{<t+1}]))$, $\tilde V_\pi([x,y_{<t}]) = \mathbb{E}_\pi[\tilde Q_\pi([x,y_{<t}], y_t)]$ carries risk forward token by token. The argument hinges on a reward decomposition that shifts the cumulative past reward into the state value and on the definition of a risk-aware advantage $\tilde A_\pi([x,y_{<t}], z) = \tilde Q_\pi([x,y_{<t}], z) - \Phi_\mu(\tilde V_\pi([x,y_{<t}]))$; this advantage turns the Bradley-Terry comparison into a sum over tokens, and the closed-form policy from the KL-constrained objective converts that sum into the DPO reward difference $u$ minus the sequential risk-ratio difference $\delta$.

What would settle it

Take any trained checkpoint, fix a prefix $[x,y_{<t}]$, and compute both $\Phi_\mu(\tilde V_\pi([x,y_{<t}]))$ and $\Phi_\mu(\tilde Q_\pi([x,y_{<t}], z))$ for a sampled token $z$ under CVaR and ERM; if these differ materially, the derivation in Equation (37) fails. A second, behavioral check: train Ra-DPO with the closed-form $\delta$ and compare against the same loss with a Monte Carlo estimate of the true risk-aware advantage; identical reward accuracy and sequential KL would support the identity, while divergence would show the loss is a heuristic rather than the derived objective.

Watch

Extended reading notes

Core claim

The paper's central discovery is a closed-form identity linking the Bradley-Terry preference model to the optimal policy of a risk-averse token-level objective. Concretely, Theorem 3.6 states that $P^*_{BT}(y_1 \succ y_2 \mid x) = \sigma\bigl(u^*(x,y_1,y_2) - \delta^*(x,y_1,y_2)\bigr)$, where $u$ is the difference of $\beta$-weighted log-probability ratios between policy and reference model, and $\delta$ is the difference of sequential risk ratios $D_{SeqRR}(x,y;\pi_{ref}\|\pi_\theta) = \sum_t \Phi_\mu\bigl(\log \frac{\pi_{ref}(z\mid [x,y_{<t}])}{\pi_\theta(z\mid [x,y_{<t}])}\bigr)$ over tokens. Because the risk-measure value $\Phi_\mu$ is taken on each token's log-ratio distribution, the preference probability automatically penalizes responses whose token-level deviations from the reference are risky in the tail, not just on average. Maximizing the likelihood of $\sigma(u-\delta)$ therefore both fits human preferences and suppresses model drift, and the paper proves a policy-improvement lemma for the underlying risk-aware advantage objective.

Load-bearing premise

The closed-form loss depends on treating a risk measure applied to an average as the same as the risk measure applied to one sampled token, and on a Bellman recursion whose printed form omits the immediate reward; if either identification fails, the loss is not exactly the claimed risk-aware objective.

Editorial extensions

If this is right

  • Ra-DPO1, with loss $-\mathbb{E}[\log\sigma(u-\delta)]$, is a direct replacement for the DPO objective: it needs only policy and reference token probabilities plus a chosen risk measure, with no reward model or RL training loop.
  • Choosing a risk-averse measure such as CVaR or ERM makes the training signal penalize the tail of token-level log-ratio deviations, which the paper's experiments show reduces sequential KL divergence relative to TDPO while keeping or improving reward accuracy.
  • The second variant Ra-DPO2 stops the gradient of the preferred-response risk ratio and treats it as a baseline, which the paper argues stabilizes training while still suppressing drift.
  • On the evaluated benchmarks, Ra-DPO1 and Ra-DPO2 attain the best winrate and length-controlled winrate among compared methods, indicating the risk correction does not trade away generation quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Theorem 3.6 is taken at face value, every DPO-style objective can be made risk-aware by adding a difference of sequential risk ratios, so the framework suggests a generic recipe rather than a single loss; one could test this by applying the same $\delta$ correction to variants such as IPO or SimPO.
  • The proof's critical step treats $\Phi_\mu(\tilde V_\pi)$ as interchangeable with $\Phi_\mu(\tilde Q_\pi([x,y_{<t}], z))$ for a sampled token; for strictly concave risk measures this equality does not hold exactly, so a plausible reading is that Ra-DPO works because the risk-ratio term is itself a regularizer, with the exact closed-form derivation needing an added assumption. An empirical test wou
  • A natural extension would use other law-invariant coherent risk measures (for example mean-semideviation) in place of CVaR and ERM, or make $\mu$ adaptive per token, to see whether the reported drift suppression strengthens as tail sensitivity increases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript introduces Risk-aware Direct Preference Optimization (Ra-DPO), a token-level DPO variant that incorporates nested risk measures (CVaR and ERM) into the Bellman recursion of a Preference-based MDP. The paper defines a risk-aware advantage function, claims a closed-form mapping from the risk-aware state-action value to the optimal policy, and then claims that the Bradley-Terry preference probability can be written as sigma(u(x,y1,y2) - delta(x,y1,y2)), where u is the usual DPO implicit-reward difference and delta is a difference of sequential risk ratios DSeqRR. This yields the loss -E[log sigma(u - delta)] (Eq. 15) and a stop-gradient variant (Eq. 17). Experiments on IMDb, Anthropic HH, and AlpacaEval with GPT-2 and Pythia models compare Ra-DPO with DPO, PPO, KTO, and TDPO, reporting improved or comparable reward accuracy and lower sequential KL divergence.

Significance. If Theorem 3.6 were valid, the paper would offer a principled way to inject tail-risk control into DPO-style training while preserving a simple loss; the proposed DSeqRR regularizer is intuitive, and the experimental protocol (three datasets, two base-model families, multiple seeds in Appendix C) is a reasonable first evaluation. The code is open-sourced and the experimental setup is transparent. However, the central derivation has a load-bearing gap: the proof of Theorem 3.6 replaces a nonlinear risk measure of a value expectation by the same risk measure evaluated at a sampled token's Q-value, which is not valid for CVaR or ERM. Because this substitution is the step that turns the risk-aware advantage into the DSeqRR expression, the advertised connection between the risk-aware objective (8) and the loss (15) is not established. The paper's contribution as currently written is therefore an empirically motivated regularized DPO variant with an unsupported risk-aware derivation.

major comments (3)
  1. [Appendix B.6, Eq. (37)] The proof equates Phi_mu(tilde V_pi([x,y<t])) with Phi_mu(tilde Q_pi([x,y<t], z)) for a sampled token z. By Eq. (6), tilde V is the expectation of tilde Q over the next-token distribution, so this identity would require Phi_mu to be linear (or the Q distribution to be degenerate). For CVaR and ERM, Phi_mu(E_pi[Q]) is generally not equal to Phi_mu(Q(z)); for example, for CVaR with alpha < 1 and a non-degenerate Q distribution, the former is a conditional tail mean while the latter is a random variable. Without this identity, the chain from Eq. (37) to Eq. (38), and hence to Theorem 3.6 and the loss (15), does not go through. The authors should either prove the theorem under the additional assumption that Phi_mu is linear (which excludes the claimed CVaR/ERM instantiations) or present Ra-DPO as a heuristic regularizer rather than as the closed-form solution of the risk-aware objective.
  2. [Eq. (6), Lemma 3.1] Equation (6) defines tilde Q_pi([x,y<t], yt) = Phi_mu(tilde V_pi([x,y<t+1])), omitting the immediate reward R([x,y<t], yt) that appears in the original Bellman equation (5). The subsequent decomposition in Lemma 3.1 and the derivation of the risk-aware advantage in Appendix B.2 nonetheless manipulate expressions of the form R + gamma Phi_mu(tilde V) - Phi_mu(tilde V). As printed, it is unclear whether tilde Q is supposed to include the immediate reward or not; if it does, Eq. (6) is incorrect, and if it does not, Lemma 3.1's relation tilde V_pi([x,y<t]) = V_pi([x,y<t]) + R_{1:t-1} needs a different proof. This ambiguity propagates into Definition 3.2 and Theorem 3.6.
  3. [Section 4, Figures 2-4] The evaluation metric DSeqKL(x,y; ref) is the expectation (Phi = E) special case of the sequential risk ratio DSeqRR(x,y; pi_ref | pi_theta) that appears directly in the loss (15). Therefore the reported reduction in DSeqKL relative to TDPO is substantially by construction: the objective explicitly penalizes the preferred response's DSeqRR and the dispreferred response's DSeqRR with opposite signs. To support the claim that Ra-DPO provides risk control beyond simply optimizing its own regularizer, the paper should include an independent evaluation, such as tail statistics of reward or divergence over generations, or performance on a held-out risk-sensitive metric.
minor comments (5)
  1. [Section 3.3, Eq. (16)] The reported gradient is not the derivative of -log sigma(u - delta). Differentiating gives -(1 - sigma(u - delta)) grad(u - delta) = -sigma(delta - u)(grad u - grad delta), not -(u - delta)(grad u - grad delta). The weighting-factor interpretation in the text should be corrected.
  2. [Theorem 3.6 and Section 3.3] The theorem statement and Eqs. (13)-(14) use the symbol pi_theta where the proof of Theorem 3.6 derives the identity for the optimal policy pi*_theta. This shift should be clarified, since Eq. (10) is obtained for the optimal policy of the constrained problem, and the loss (15) is then written for the current parametrized policy.
  3. [Algorithm 1] The Ra-DPO1 update is written as theta <- theta + eta grad_theta E[log sigma(...)], but the loss defined in Eq. (15) is the negative log likelihood; the update should be a gradient descent step on L, i.e., theta <- theta - eta grad_theta L, unless the sign convention is explained.
  4. [Section 2.3 and Appendix D.1] The main text lists concavity and translation invariance as the required properties of the risk-sensitive function, while Appendix D.1 also mentions monotonicity. Add monotonicity to the definition or explain why it is not needed.
  5. [Lemma 3.5] The phrase 'sigma(z) = 1/(1 + exp(-z)) is the logistic sigmoid function for any random variable z' is imprecise; sigma is applied pointwise to the argument here, and the left-hand side P_BT is a deterministic conditional probability given the responses, not a random variable.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation chain: Ra-DPO's loss follows algebraically from its stated risk-aware objective; the drift suppression seen in experiments is a consistency check of the explicit penalty, not a fitted prediction, and the only self-citation (TDPO) is not load-bearing.

full rationale

No circular steps were found. The derivation from the risk-aware objective in Eq. (8) to the Ra-DPO loss in Eq. (15) is an algebraic consequence of the stated definitions: Lemma 3.4 solves the KL-constrained maximization, Lemma 3.5 substitutes the telescoped reward decomposition into the Bradley-Terry model, and Theorem 3.6 rewrites the resulting advantage sum in terms of the policy ratio and the defined sequential risk ratio (Eq. 14). None of these steps fits a parameter to data and then renames it a prediction, and no quantity is defined in terms of the quantity it is supposed to derive. The report that Ra-DPO lowers DSeqKL is a consistency check of the explicit DSeqRR penalty in Eq. (15), since DSeqRR reduces to DSeqKL when the risk measure is the identity; this makes the empirical drift reduction expected, but it is not a circular derivation of the method's theoretical claim. The apparent proof gap in Eq. (37), where Phi_mu(tilde V) is replaced by Phi_mu(tilde Q(z)) despite Phi_mu being nonlinear, is a correctness concern, not a circularity concern, and the stated limitations about concavity and translation invariance do not address it. The only self-citation is TDPO [10], whose author list includes Jun Wang; it supplies the token-level advantage insight, but the paper re-derives the decomposition in Appendices B.2 through B.6, so the citation is not load-bearing. Overall score 2 reflects a minor self-citation with otherwise independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim depends on the risk-measure properties, the tree-like MDP decomposition, and an unstated equality between Phi(V) and Phi(Q(z)); the latter is a gap rather than a stated axiom. The DSeqRR term is a new mathematical object whose evaluation metric is its own risk-neutral case. The coefficients Beta, Alpha, and Mu are hand-set or swept hyperparameters that affect the reported tradeoffs.

free parameters (4)
  • Beta = 0.1
    Temperature/KL coefficient in the DPO-style loss, fixed across all experiments; controls the strength of the deviation penalty.
  • Alpha = {0.3, 0.5, 0.7, 0.9}
    Coefficient for the risk-ratio term in Ra-DPO2; swept across four values, with results reported for each. Acts as a free scaling of the penalty.
  • Mu (CVaR) = {0.95, 0.97, 0.98, 0.99}
    Risk-aversion level for CVaR; swept. Higher values are more risk-averse. The paper reports results for each value and claims near-universal improvement, so the choice functions as a tuned parameter.
  • Mu (ERM) = {5, 7, 9}
    Risk-aversion parameter for entropic risk measure; swept across three values.
assumptions (5)
  • domain assumption The risk measure Phi is concave, monotone, and translation invariant.
    Section 2.3 and Appendix D.1 state this class; translation invariance is required for Lemma 3.1 and the loss derivation.
  • domain assumption The Pb-MDP can be reformulated as a tree-like MDP via state augmentation, and the cumulative reward decomposes as r = sum_t gamma^{t-1} R([x,y<t], y_t).
    Appendix B.1, Lemma 3.1; used to relate V and V_tilde. The tree-like assumption guarantees a unique path and Markovian transitions.
  • domain assumption The policy pi_theta is initialized to pi_ref and the reference model is fixed during training.
    Standard DPO assumption, used throughout Section 3.
  • standard math Preferences follow the Bradley-Terry model.
    Equation (1); the entire derivation starts from this pairwise preference model.
  • ad hoc to paper Phi(E_pi[Q]) is treated as equal to Phi(Q(z)) for a sampled token z.
    Implicit in Appendix B.6, Eq. (37). For concave nonlinear risk measures this is generally false unless the policy is deterministic or Phi is linear, so the theorem as proven relies on an unstated and unjustified equality.
invented entities (1)
  • Sequential risk ratio DSeqRR(x, y; pi_ref | pi_theta)
    purpose: Token-level risk-adjusted divergence penalty used in the Ra-DPO loss to suppress model drift.
    Defined in Theorem 3.6 as sum_t Phi_mu[log(pi_ref/pi_theta)] under pi_ref. It is a new objective term; its risk-neutral special case (Phi = expectation) equals the sequential KL used by TDPO, and the evaluation metric DSeqKL is that same special case, so the empirical drift reduction is partly forced by construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Risk-aware Direct Preference Optimization under Nested Risk Measure." pith.science (2026). https://pith.science/paper/RAB2QRFB

@misc{pith2026250520359,
  author       = {Pith},
  title        = {Pith review of: Risk-aware Direct Preference Optimization under Nested Risk Measure},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RAB2QRFB}},
  note         = {Machine review of arXiv:2505.20359}
}
read the original abstract

When fine-tuning pre-trained Large Language Models (LLMs) to align with human values and intentions, maximizing the estimated reward can lead to superior performance, but it also introduces potential risks due to deviations from the reference model's intended behavior. Most existing methods typically introduce KL divergence to constrain deviations between the trained model and the reference model; however, this may not be sufficient in certain applications that require tight risk control. In this paper, we introduce Risk-aware Direct Preference Optimization (Ra-DPO), a novel approach that incorporates risk-awareness by employing a class of nested risk measures. This approach formulates a constrained risk-aware advantage function maximization problem and then converts the Bradley-Terry model into a token-level representation. The objective function maximizes the likelihood of the policy while suppressing the deviation between a trained model and the reference model using a sequential risk ratio, thereby enhancing the model's risk-awareness. Experimental results across three open-source datasets: IMDb Dataset, Anthropic HH Dataset, and AlpacaEval, demonstrate the proposed method's superior performance in balancing alignment performance and model drift. Our code is opensourced at https://github.com/zlj123-max/Ra-DPO.

Figures

Figures reproduced from arXiv: 2505.20359 by the authors.

Figure 1
Figure 1. Comparison of loss functions for DPO, TDPO2 and Ra-DPO2 methods. The sg denotes the stop-gradient operator. The operator sg represents the stop￾gradient operator, which blocks the propagation of gradients. The pa￾rameter β can control the deviation between DSeqRR (x, y2; πref | πθ) and (βDSeqRR (x, y1; πref | πθ)). Ra-DPO2 modifies the loss function of Ra-DPO1 by disabling the gradient propagation of DSeqRR(x, yw; π… view at source ↗
Figure 2
Figure 2. The experiment on the IMDb dataset with GPT-2 Large serving as the base model. (a) and [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The experiment on the Anthropic HH dataset with Pythia-1.4B serving as the base model. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: The experiment on the Anthropic HH dataset with Pythia-1.4B serving as the base model. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The experiment on the Anthropic HH dataset with Pythia-1.4B serving as the base model. [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: The experiment on the Anthropic HH dataset with Pythia-1.4B serving as the base model. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: The experiment on the Anthropic HH dataset with Pythia-1.4B serving as the base model. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: The experiment on the Anthropic HH dataset with Pythia-1.4B serving as the base model. [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: The experiment on the Anthropic HH dataset with Pythia-2.8B serving as the base model. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: The experiment on the Anthropic HH dataset with Pythia-2.8B serving as the base model. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: The experiment on the Anthropic HH dataset with Pythia-2.8B serving as the base model. [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: The experiment on the Anthropic HH dataset with Pythia-2.8B serving as the base model. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: The experiment on the Anthropic HH dataset with Pythia-1.4B serving as the base model. [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: AlpacaEval comparison between DPO, PPO, TDPO1, TDPO2, and Ra-DPO2 methods. The win, tie, and lose rates are evaluated based on oasst-pythia-12b. • Figures 5-8 illustrate the experiment on the Anthropic HH dataset with Pythia-1.4B serving as the base model. We implemen…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 41 canonical work pages

  1. [1]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurIPS, 2017

    work page 2017

  2. [2]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022

    work page 2022

  3. [3]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  4. [4]

    Preference ranking optimization for human alignment

    Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In AAAI, 2024

    work page 2024

  5. [5]

    Direct preference optimization: your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. In NeurIPS, 2023

    work page 2023

  6. [6]

    Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints

    Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. In ICLR, 2024

    work page 2024

  7. [7]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In AISTATS, 2024

    work page 2024

  8. [8]

    Robust preference optimization through reward model distillation

    Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316, 2024

    arXiv 2024

Show all 59 references
  1. [9]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  2. [10]

    Token- level direct preference optimization

    Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. In ICML, 2024

    2024

  3. [11]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015

    2015

  4. [12]

    Deep reinforcement learning: A brief survey

    Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38, 2017

    2017

  5. [13]

    An introduction to deep reinforcement learning

    Vincent François-Lavet, Peter Henderson, Riashat Islam, Marc G Bellemare, Joelle Pineau, et al. An introduction to deep reinforcement learning. Foundations and Trends® in Machine Learning, 11(3-4):219–354, 2018

    2018

  6. [14]

    Robust risk-aware reinforcement learning

    Sebastian Jaimungal, Silvana M Pesenti, Ye Sheng Wang, and Hariom Tatsat. Robust risk-aware reinforcement learning. SIAM Journal on Financial Mathematics, 13(1):213–226, 2022

    2022

  7. [15]

    Risk-averse policy optimization via risk-neutral policy optimization

    Lorenzo Bisi, Davide Santambrogio, Federico Sandrelli, Andrea Tirinzoni, Brian D Ziebart, and Marcello Restelli. Risk-averse policy optimization via risk-neutral policy optimization. Artificial Intelligence, 311:103765, 2022

    2022

  8. [16]

    Risk-aware controller for autonomous vehicles using model-based collision prediction and reinforcement learning

    Eduardo Candela, Olivier Doustaly, Leandro Parada, Felix Feng, Yiannis Demiris, and Panagio- tis Angeloudis. Risk-aware controller for autonomous vehicles using model-based collision prediction and reinforcement learning. Artificial Intelligence, 320:103923, 2023

    2023

  9. [17]

    Risk-averse fine-tuning of large language models

    Sapana Chaudhary, Ujwal Dinesha, Dileep Kalathil, and Srinivas Shakkottai. Risk-averse fine-tuning of large language models. In NeurIPS, 2024

    2024

  10. [18]

    Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In ICML, 2024. 10

    2024

  11. [19]

    Advances in prospect theory: Cumulative representation of uncertainty

    Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5:297–323, 1992

    1992

  12. [20]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952

    1952

  13. [21]

    More risk-sensitive markov decision processes

    Nicole Bäuerle and Ulrich Rieder. More risk-sensitive markov decision processes. Mathematics of Operations Research, 39(1):105–120, 2014

    2014

  14. [22]

    Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control.Artificial Intelligence, 311:103743, 2022

    Yuheng Wang and Margaret P Chapman. Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control.Artificial Intelligence, 311:103743, 2022

    2022

  15. [23]

    Thinking coherently

    Philippe Artzner. Thinking coherently. Risk, 10:68–71, 1997

    1997

  16. [24]

    Optimization of conditional value-at-risk

    R Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of risk, 2:21–42, 2000

    2000

  17. [25]

    Risk-sensitive and robust decision- making: a cvar optimization approach

    Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a cvar optimization approach. In NeurIPS, 2015

    2015

  18. [26]

    Convex measures of risk and trading constraints

    Hans Föllmer and Alexander Schied. Convex measures of risk and trading constraints. Finance and stochastics, 6:429–447, 2002

    2002

  19. [27]

    Entropic risk optimization in discounted mdps

    Jia Lin Hau, Marek Petrik, and Mohammad Ghavamzadeh. Entropic risk optimization in discounted mdps. In AISTATS, 2023

    2023

  20. [28]

    Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret

    Yingjie Fei, Zhuoran Yang, Yudong Chen, Zhaoran Wang, and Qiaomin Xie. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret. In NeurIPS, 2020

    2020

  21. [29]

    Provably efficient iterated cvar reinforcement learning with function approximation and human feedback

    Yu Chen, Yihan Du, Pihe Hu, Siwei Wang, Desheng Wu, and Longbo Huang. Provably efficient iterated cvar reinforcement learning with function approximation and human feedback. In ICLR, 2024

    2024

  22. [30]

    Ra-pbrl: Provably efficient risk-aware preference-based reinforcement learning

    Yujie Zhao, Jose Efraim Aguilar Escamilla, Weyl Lu, and Huazheng Wang. Ra-pbrl: Provably efficient risk-aware preference-based reinforcement learning. In NeurIPS, 2024

    2024

  23. [31]

    Entropic risk optimization in discounted mdps

    Jia Lin Hau, Marek Petrik, and Mohammad Ghavamzadeh. Entropic risk optimization in discounted mdps. In AISTATS, pages 47–76, 2023

    2023

  24. [32]

    Learning word vectors for sentiment analysis

    Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, 2011

    2011

  25. [33]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    2022 arXiv

  26. [34]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  27. [35]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  28. [36]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  29. [37]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Skowron Raff, Sutawika Aviya, Wal Lintang, and Oskar van der. Pythia: A suite for analyzing large language mod...

    2023

  30. [38]

    Red-teaming large language models using chain of utterances for safety-alignment

    Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662, 2023. 11

    2023 arXiv

  31. [39]

    Safe rlhf: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In ICLR, 2024

    2024

  32. [40]

    How reliable is human feedback for aligning large language models? arXiv preprint arXiv:2410.01957, 2024

    Min-Hsuan Yeh, Leitian Tao, Jeffrey Wang, Xuefeng Du, and Yixuan Li. How reliable is human feedback for aligning large language models? arXiv preprint arXiv:2410.01957, 2024

    2024 arXiv

  33. [41]

    Fine-grained human feedback gives better rewards for language model training

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In NeurIPS, 2023

    2023

  34. [42]

    Token-level proximal policy optimization for query generation

    Yichen Ouyang, Lu Wang, Fangkai Yang, Pu Zhao, Chenghua Huang, Jianfeng Liu, Bochen Pang, Yaming Yang, Yuefeng Zhan, Hao Sun, et al. Token-level proximal policy optimization for query generation. arXiv preprint arXiv:2411.00722, 2024

    2024 arXiv

  35. [43]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  36. [44]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  37. [45]

    Rusu, Joel Veness, Marc G

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Ku- maran, Daan...

    2015

  38. [46]

    Reinforcement learning in robotic applications: a comprehensive survey

    Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications: a comprehensive survey. Artificial Intelligence Review, 55(2):945–990, 2022

    2022

  39. [47]

    A review of safe reinforcement learning: Methods, theory and applications

    Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):11216–11235, 2024

    2024

  40. [48]

    Risk-sensitive reinforcement learning with function approximation: A debiasing approach

    Yingjie Fei, Zhuoran Yang, and Zhaoran Wang. Risk-sensitive reinforcement learning with function approximation: A debiasing approach. In ICML, 2021

    2021

  41. [49]

    Regret bounds for risk- sensitive reinforcement learning

    Osbert Bastani, Yecheng Jason Ma, Estelle Shen, and Wanqiao Xu. Regret bounds for risk- sensitive reinforcement learning. In NeurIPS, 2022

    2022

  42. [50]

    Near-minimax-optimal risk-sensitive reinforce- ment learning with cvar

    Kaiwen Wang, Nathan Kallus, and Wen Sun. Near-minimax-optimal risk-sensitive reinforce- ment learning with cvar. In ICML, 2023

    2023

  43. [51]

    Provably efficient risk-sensitive reinforcement learning: Iterated cvar and worst path

    Yihan Du, Siwei Wang, and Longbo Huang. Provably efficient risk-sensitive reinforcement learning: Iterated cvar and worst path. In ICLR, 2022

    2022

  44. [52]

    Group robust preference optimization in reward-free rlhf

    Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. Group robust preference optimization in reward-free rlhf. In NeurIPS, 2024

    2024

  45. [53]

    Preference learning of latent decision utilities with a human-like model of preferential choice

    Sebastiaan De Peuter, Shibei Zhu, Yujia Guo, Andrew Howes, and Samuel Kaski. Preference learning of latent decision utilities with a human-like model of preferential choice. In NeurIPS, 2024

    2024

  46. [54]

    Robust reinforcement learning

    Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335– 359, 2005

    2005

  47. [55]

    Hamilton–jacobi reachability: Some recent theoretical advances and applications in unmanned airspace management

    Mo Chen and Claire J Tomlin. Hamilton–jacobi reachability: Some recent theoretical advances and applications in unmanned airspace management. Annual Review of Control, Robotics, and Autonomous Systems, 1(1):333–358, 2018

    2018

  48. [56]

    Coherent measures of risk

    Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. Coherent measures of risk. Mathematical finance, 9(3):203–228, 1999. 12

    1999

  49. [57]

    Equivalence notions and model minimization in markov decision processes

    Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in markov decision processes. Artificial intelligence, 147(1-2):163–223, 2003

    2003

  50. [58]

    Learning markov network structure with decision trees

    Daniel Lowd and Jesse Davis. Learning markov network structure with decision trees. InICDM, 2010

    2010

  51. [59]

    ∞X t=1 γt−1 R x, y<t , yt + γ Φµ ˜Vπ x, y<t+1 − ˜Vπ([x]) # =Eτ |π′

    Akifumi Wachi, Thien Q Tran, Rei Sato, Takumi Tanabe, and Youhei Akimoto. Stepwise alignment for constrained language model policy optimization. In NeurIPS, 2024. 13 A Supplementary Materials for Section 2 A.1 Risk Measure: A Brief Overview For quantifying and managing risks, ...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.