Pith. sign in

REVIEW 4 major objections 8 minor 33 references

Risk-Averse Finetuning of Large Language Models

T0 review · 4 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing expected reward with a CVaR-style worst-return objective in RLHF makes LLMs safer on toxic prompts and better on average, the paper claims.

desk verdict First clean CVaR-style RA-RLHF; relative gains on the proxy are solid, but the real-world safety claim is undermined by using the same classifier for reward and evaluation. read the letter →

arxiv 2501.06911 v1 pith:EVCXLGWQ submitted 2025-01-12 cs.AI cs.CL

Sapana Chaudhary , Ujwal Dinesha , Dileep Kalathil , Srinivas Shakkottai This is my paper · ORCID
classification cs.AIcs.CL
keywords risk-aversereinforcementlearningconditionalvalueatriskRLHFlargelanguagemodelstoxicitymitigationsentimentmodificationsoft-riskschedulingsafetextgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that LLM fine-tuning can be made risk-averse by optimizing a tail-risk objective instead of average reward, and that this reduces toxic or negative generations without sacrificing overall quality. It proposes RA-RLHF, a variant of RLHF that, after a warm-up phase, updates the policy only on the completions with the worst rewards in each batch, approximating Conditional Value at Risk (CVaR). On sentiment-completion and toxicity-mitigation tasks with two base LLMs (117M and 6B parameters), the resulting policies shift both negative and positive prompts toward positive or non-toxic completions, beat standard RLHF in the riskiest prompt quantiles, and also score higher on average over all test prompts. If this holds, safety-conscious fine-tuning needs no second reward model and no constrained optimization, just a change in which trajectories are used for the policy update.

What carries the argument

The engine is Conditional Value at Risk (CVaR), the average return in the worst $\alpha$ fraction of the return distribution, applied through trajectory selection on a KL-regularized reward. Concretely, RA-RLHF keeps the $B_0 = \lceil \alpha B \rceil$ lowest-return completions from each batch and runs PPO only on those, using the standard RLHF per-token reward $\bar r(s_t,a_t) = r(s_t,a_t) - \beta \log(\pi_\theta(a_t|s_t)/\pi_{\mathrm{ref}}(a_t|s_t))$, where $\pi_{\mathrm{ref}}$ is the supervised fine-tuned policy. A soft-risk schedule handles the exploration-versus-risk trade-off: for the first $i_0$ iterations the full batch is used so the model learns to produce successful completions, then the selection threshold decays at a constant rate from the full batch to $\lceil \alpha B \rceil$ by iteration $\lfloor \rho M \rfloor$. This schedule, together with the KL regularization to a positive-data fine-tuned base, is what lets the policy concentrate on hazardous prompts without losing the ability to generate coherent text.

What would settle it

Have human raters, or a toxicity or sentiment classifier never used during training, score completions from RA-RLHF and RLHF on the same tail prompts; if the RA-RLHF completions are not rated less toxic or more positive than the RLHF completions, the paper's central claim fails.

Watch

Extended reading notes

Core claim

Traditional RLHF maximizes expected reward, so rare but very harmful outputs can be masked by many safe outputs. RA-RLHF replaces that objective with a CVaR-style one: each training iteration samples a batch of completions, scores them with a fixed reward model, and keeps only the lowest-scoring $B_0$ trajectories for the PPO update, with $B_0$ initially equal to the full batch and then shrinking linearly to $\lceil \alpha B \rceil$ as training progresses. The paper finds that on IMDB sentiment completion, Jigsaw toxicity mitigation, and RealToxicityPrompts, this worst-return selection produces policies whose reward distributions shift further toward positive/non-toxic text than SFT or RLHF, that dominate RLHF on the hardest prompt quantiles, and that achieve higher average reward over all prompts with only a small perplexity increase. The authors state this as: RA-RLHF policies do not merely show risk-aversion on negative or toxic prompts, they perform better than a traditional RLHF-tuned policy over all prompts.

Load-bearing premise

The comparison rests on the scoring models used to select training trajectories and to grade outputs being accurate measures of sentiment and toxicity; if those models are gameable or miscalibrated on the hardest prompts, the claimed reduction in negative content may not be real.

Editorial extensions

If this is right

  • On the three evaluated tasks, RA-RLHF shifts the reward distribution of generated completions upward more than SFT or RLHF, with the largest gains on the prompts whose starting rewards are worst.
  • The average reward on the worst-case prompt tail is higher for RA-RLHF than for RLHF on both the 117M-parameter and the 6B-parameter base models.
  • The improvement does not come from degenerate repetition: RA-RLHF keeps response lengths near those of RLHF, maintains diversity scores, and pays only a small perplexity penalty.
  • Because RA-RLHF needs a single reward model, it offers a simpler route to tail-risk safety than approaches that require separate helpfulness and harmlessness models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test this, but because the same reward model selects training trajectories and scores the final outputs, part of the reported gap could be reward hacking; a human or held-out classifier evaluation would show whether the actual toxicity of generations drops.
  • A natural extension the authors leave implicit is to apply worst-return selection to alignment losses other than PPO-based RLHF, such as direct preference optimization, giving those methods tail-risk behavior as well.
  • The soft-risk schedule is effectively a curriculum over the return distribution; one could make the risk level or the warm-up length depend on the riskiness of each prompt class rather than using a global quantile.
  • If the toxicity reward model is miscalibrated on particular harm categories, RA-RLHF's selection will inherit that blind spot, so auditing the scorer's calibration per category is a direct test of the method's safety claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes Risk-Averse RLHF (RA-RLHF), a modification of the standard RLHF pipeline in which each PPO update uses only the B0 lowest-return trajectories, with a soft-risk schedule that starts with the full batch and gradually reduces the selection size to a CVaR-determined fraction. The authors test RA-RLHF on three conditional-generation tasks (IMDB sentiment, Jigsaw toxicity, RealToxicityPrompts) using GPT-2 and GPT-J, reporting that RA-RLHF achieves higher average and tail rewards than RLHF while maintaining comparable perplexity and diversity. The central claim is that replacing the expected-reward objective with a CVaR-style worst-return selection improves both tail behavior and average performance over a standard RLHF-tuned policy.

Significance. The paper makes a useful, practical contribution: RA-RLHF is a simple, drop-in modification to existing RLHF implementations, the code is released, and the algorithm-vs-algorithm comparison on the stated reward proxies is consistent across three GPT-2 datasets and a single-seed GPT-J run. The soft-risk scheduling and the explicit focus on worst-case prompts are sensible adaptations of prior risk-averse RL work (Greenberg et al.). The main limitation is that the safety interpretation of the results is not supported by the measurement design: the same classifier reward models are used for training selection, for the evaluation metrics, and for defining the tail prompts, so the reported gains may reflect optimization of classifier-specific artifacts rather than a genuine reduction in toxic or negative content. If the authors add independent safety evaluation (e.g., Perspective API, human ratings, or a held-out reward model) on the tail prompts and clarify the statistical reporting, the work would be a solid empirical contribution to safe LLM fine-tuning.

major comments (4)
  1. [§5, §E.1] The reward models lvwerra/distilbert-imdb and unitary/toxic-bert are used both as the training reward (Eq. (8) and Algorithm 1) and as the evaluation metric (the 'Standard task performance scores' in §5). Consequently, the reported improvements may reflect overfitting to these classifiers rather than reduced negative or toxic content. The abstract's safety claim requires independent evaluation on tail prompts (e.g., Perspective API, human ratings, or a held-out reward model) and a demonstration that the generated outputs differ in semantically meaningful ways; the perplexity and diversity metrics do not address this measurement overlap.
  2. [§5.1, Tables 3-4, §F.4] The statistical support is inconsistent. Table 3 reports standard deviations for RealToxicityPrompts-Gen, but Appendix F.4 (Table 10) states 'Results are for one seed,' and the GPT-J results in Table 4 are single-seed. The SFT baseline has no standard deviation due to 'faulty seeding'. The claim of consistent improvement over RLHF across datasets should be backed either by three-seed results for all reported configurations or by explicit labeling of preliminary single-seed results.
  3. [§4, Eq. (8), Algorithm 1] The paper states that selecting the B0 lowest-return trajectories is a 'simplified approach similar to (7),' but no derivation is provided for why this selection, combined with the KL-regularized objective and a critic updated only on selected trajectories, optimizes the CVaR objective. As written, the algorithm is a heuristic that may deviate substantially from CVaR policy gradients; the authors should either state the conditions under which the approximation holds or present RA-RLHF as an empirical heuristic rather than as a CVaR optimizer.
  4. [§1, Figs. 1-2, Table 4] The claim that RA-RLHF 'actually perform[s] better than a traditional RLHF-tuned policy over all prompts' is supported only by quantile plots without error bars; the quantitative tables (Tables 2-4) report tail rewards, not all-prompt averages. The authors should add an all-prompt average reward table with standard deviations or temper the claim to tail performance.
minor comments (8)
  1. [§4, risk schedule point C] The variable m in the formula for B0 is not defined; it should be the iteration index i, as in points A and B.
  2. [Algorithm 1, line 10] The word 'retrun' should be 'return'.
  3. [Table 2 and Table 3] The table formatting is difficult to parse: the RLHF row of Table 2 contains an extra value '0.36', and standard deviations appear as separate columns rather than subscripts; Table 3 has a similar layout. Please align the columns with the stated headers.
  4. [§5, first paragraph of Evaluation Metrics] The sentence 'The outputs in are evaluated using a standardized scoring models' is missing a noun and should read 'The outputs are evaluated using standardized scoring models'.
  5. [Table 4 and Appendix F.3, Table 9] Table 4's caption says 'Testing on reward (r), and Perplexity,' but the table only shows Tail (r) and Perplexity; either add the all-prompt Reward (r) column or adjust the caption to match the content.
  6. [Appendix F.4, Table 10] Table 10 says 'Results are for one seed,' which conflicts with the standard deviations reported for RealToxicityPrompts-Gen in Table 3; please reconcile the two presentations.
  7. [§5.3, Table 5] The warm-start iterations are denoted n in Table 5 but i0 in Section 4; please unify the notation.
  8. [Table 3 caption] The caption contains the typo 'Nagative' instead of 'Negative'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RA-RLHF's risk-averse objective and its evaluation are distinct algorithmic steps, and the shared reward model is a measurement-validity caveat, not a construction-level reduction.

full rationale

The derivation chain in Section 4 is self-contained: RA-RLHF optimizes the KL-regularized return in Eq. (8) over the B0 lowest-return trajectories, using soft-risk scheduling adapted from the external prior work of Greenberg et al. (2022). The evaluation in Section 5 compares RA-RLHF against RLHF, SFT, DExperts, and Quark using sentiment/toxicity scores, perplexity, and diversity metrics on held-out test prompts. The fact that the same reward models (lvwerra/distilbert-imdb and unitary/toxic-bert) are used to compute returns during training and to score outputs at evaluation is a measurement-overlap concern for the paper's real-world safety claim, but it is not equation-level circularity: RLHF is also trained and evaluated under the same reward models, so the algorithm-vs-algorithm comparison is not forced by construction. No fitted parameter is later relabeled as a prediction, no test-tail statistic is fit and then reported as a discovery, and no load-bearing theorem or ansatz is imported from the authors' own prior work (the paper contains no self-citations to the author list). The soft-risk schedule, CVaR-selected batches, PPO update, perplexity checks, diversity metrics, response-length monitoring, and the GPT-J/LoRA extension all provide independent empirical content. The limitations appendix (A) and broader-impact appendix (B) candidly state that the method relies on pre-existing reward models and that generalizability across domains remains open; those are honest external-validity caveats, not evidence that the central claim reduces to its inputs. Therefore the paper receives a circularity score of 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on hand-set risk-schedule hyperparameters (alpha, n, rho), a simplified equivalence between worst-batch selection and CVaR, and the validity of classifier reward models as measures of toxicity and sentiment. No new entities are introduced; the method reuses existing reward models, the MDP framing, and PPO.

free parameters (4)
  • risk level alpha = 0.4 (IMDB-Gen), 0.2 (Jigsaw-Gen); not reported for RealToxicityPrompts-Gen
    Controls the fraction of worst trajectories kept in each update. Hand-set per dataset; sensitivity is explored in Table 5 but no automatic selection rule is given.
  • warm start iterations n (i0) = 30
    Number of initial iterations that use the full batch before risk filtering begins. Chosen by hand.
  • risk schedule endpoint rho = 0.95
    Fraction of training at which the batch size reaches alpha times B. Chosen by hand.
  • KL controller parameters = KLtarget=6.0, initial beta=0.2, Kbeta=0.0128
    Balance reward optimization against deviation from the reference policy; inherited from the RLHF setup and tuned by hand.
assumptions (5)
  • domain assumption Token-level MDP with deterministic transitions is a valid model of language generation.
    Section 3 frames generation as an MDP following Ramamurthy et al. (2022); this is required to define returns and apply policy optimization.
  • domain assumption The classifier reward models faithfully measure sentiment and toxicity.
    Section 5 and Appendix E.1 use distilbert-imdb and toxic-bert as both training rewards and evaluation metrics; if these are gameable, the safety conclusions do not follow.
  • ad hoc to paper Selecting the B0 lowest-return trajectories approximates optimizing the CVaR objective.
    Section 4, after Eq. 8, calls this a simplified approach, but no equivalence proof or bias bound is given for the PPO plus KL-regularized setting.
  • domain assumption KL-regularized PPO with a reference policy maintains generation quality.
    Section 3, Eqs. 3-5; the KL penalty is the mechanism that prevents the tail-focused updates from collapsing into repetitive or incoherent text.
  • domain assumption SFT on positive or non-toxic data provides a suitable warm start for RA-RLHF.
    Section 4A(a) initializes from an SFT model; the algorithm's behavior depends on this base policy being able to generate reasonable completions before risk filtering begins.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Risk-Averse Finetuning of Large Language Models." pith.science (2026). https://pith.science/paper/EVCXLGWQ

@misc{pith2026250106911,
  author       = {Pith},
  title        = {Pith review of: Risk-Averse Finetuning of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EVCXLGWQ}},
  note         = {Machine review of arXiv:2501.06911}
}
read the original abstract

We consider the challenge of mitigating the generation of negative or toxic content by the Large Language Models (LLMs) in response to certain prompts. We propose integrating risk-averse principles into LLM fine-tuning to minimize the occurrence of harmful outputs, particularly rare but significant events. By optimizing the risk measure of Conditional Value at Risk (CVaR), our methodology trains LLMs to exhibit superior performance in avoiding toxic outputs while maintaining effectiveness in generative tasks. Empirical evaluations on sentiment modification and toxicity mitigation tasks demonstrate the efficacy of risk-averse reinforcement learning with human feedback (RLHF) in promoting a safer and more constructive online discourse environment.

Figures

Figures reproduced from arXiv: 2501.06911 by the authors.

Figure 1
Figure 1. Environment reward distribution shift, and quantile plot for IMDB-Gen. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Environment reward distribution shift, and quantile plot for Jigsaw-Gen. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Tail sentiment score plotted for one seed. Quantitative performance on test data: Performance metrics on the test datasets across tasks are presented in Tables 2,3. The numbers are reported over the worst case prompts from randomly sampled dataset of 5k test prompts. The RA-RLHF model outperforms all the other baselines on average reward for these least favorable prompts sampled from the prompt (reward distribution)… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Average environment rewards, and per batch returns during training for IMDB-Gen and [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Number of generated tokens [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Scores for train prompts of size 200 characters ( [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Scores for test prompts of size 200 characters ( [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Scores for test prompts of size 200 characters ( [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Scores for train prompts of size 60 characters ( [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Scores for test prompts of size 60 characters ( [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Clustering on Jigsaw test dataset [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: IMDB risk schedule analysis F.2 Other training statistics In [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Various training statistics for IMDB-Gen. [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Various training statistics for Jigsaw-Gen [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Environment reward distribution shift, and quantile plot for IMDB-Gen. [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Environment reward distribution shift, and quantile plot for RealToxicityPrompts-Gen. [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 10 canonical work pages

  1. [1]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. 10 Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, And...

    arXiv 1910

  2. [6]

    Toxicity in chatgpt: Analyzing persona-assigned language models

    Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335,

    arXiv

  3. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

    arXiv

  4. [8]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858,

    arXiv

  5. [10]

    WHEN I AM UNBLOCKED I SWEAR I WILL GO F**K YOUR M C

    6 4 2 0 2 4 6 8 Enviroment Score 0 2000 4000 6000 8000 10000 12000 14000No. of input prompts Non-toxic T oxic (a) Scores for all train prompts 7.0 6.5 6.0 5.5 5.0 Enviroment Score 0 20 40 60 80 100 120 140No. of input prompts Non-toxic T oxic (b) Scores for tail train prompts Figure 9: Scores for train prompts of size 60 characters (∼ 20 tokens) for Jigsa...

    work page 2000

  6. [15]

    Dexperts: Decoding-time controlled text generation with experts and anti-experts

    Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts. arXiv preprint arXiv:2105.03023,

    arXiv

  7. [17]

    Epopt: Learning robust neural network policies using model ensembles

    12 Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283,

    arXiv

  8. [20]

    The woman worked as a babysitter: On biases in language generation

    Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326,

    arXiv 1909

Show all 33 references
  1. [22]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    arXiv

  2. [23]

    Universal adversarial triggers for attacking and analyzing nlp

    Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125,

    1908 arXiv

  3. [24]

    Ethical and social risks of harm from language models

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359,

    arXiv

  4. [25]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593,

    1909 arXiv

  5. [26]

    In our work, we primarily focussed on generative tasks, and not the Question-Answer (Q&A) format

    13 A Limitations and Future Work The effectiveness of the risk-averse fine-tuning strategy may vary across different domains and languages, necessitating further investigation and adaptation. In our work, we primarily focussed on generative tasks, and not the Question-Answer (...

    2019

  6. [27]

    Furthermore, even aligned versions of LLMs are not immune to exploitation

    which highlight the risks associated with uncurated training data. Furthermore, even aligned versions of LLMs are not immune to exploitation. The aligned models can still be prompted or ‘red-teamed’ to produce harmful content under certain conditions Gehman et al. [2020], Weid...

    2020

  7. [28]

    represent pivotal methods for enhancing downstream task performance in large language models. Each technique offers a unique approach to optimizing model proficiency: RAG integrates external knowledge sources during generation knowledge-intensive tasks like question answering,...

    2019

  8. [29]

    Furthermore, even aligned versions of LLMs are not immune to exploitation

    which highlight the risks associated with uncurated training data. Furthermore, even aligned versions of LLMs are not immune to exploitation. The aligned models can still be prompted or ‘red-teamed’ to produce harmful content under certain conditions Gehman et al. [2020], Weid...

    2020

  9. [30]

    Risk Averseness in RL

    balance helpfulness and harmlessness in AI responses by decoupling these aspects during training. Risk Averseness in RL. In the RL community, risk averseness to ensure safe policy execution has been studied using various risk criteria. Examples of these criteria include mean-v...

    2001

  10. [32]

    The dataset utilized in this task is introduced by Gehman et al

    RealToxicityPrompts-Gen task has a training size of 57.9k prompts compared to IMDB’s 25k and Jigsaw’s 36.9k. The dataset utilized in this task is introduced by Gehman et al. Gehman et al. [2020]. This dataset originates from the OPEN-WEBTEXT CORPUS, a comprehensive corpus of E...

    2020

  11. [34]

    We choose to work with regularized reward for two reasons: I

    work only with the plain reward. We choose to work with regularized reward for two reasons: I. We want to measure risk in generations accounting for both the performance on the actual environment reward and the quality of language generation measured by KL- Divergence with res...

    2022

  12. [768]

    [PAD]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 100: AddedToken(

    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise _affine=True) (dropout): Dropout(p=0.1, inplace=False) ) (encoder): BertEncoder( (layer): ModuleList( (0-11): 12 x BertLayer( (attention): BertAttention( (self): BertSelfAttention( (query): Linear(in _features=768, out _fea...

    2022

  13. [1952]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    1901

  14. [2001]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  15. [2011]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332,

    arXiv

  16. [2013]

    Rlaif: Scaling reinforcement learning from human feedback with ai feedback

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267,

    arXiv

  17. [2015]

    Worst cases policy gradients.arXiv preprint arXiv:1911.03618,

    Yichuan Charlie Tang, Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients.arXiv preprint arXiv:1911.03618,

    1911 arXiv

  18. [2016]

    Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization

    Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language po...

    arXiv

  19. [2017]

    Gedi: Generative discriminator guided sequence generation

    Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367,

    2009 arXiv

  20. [2018]

    Safe rlhf: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773,

    arXiv

  21. [2019]

    Universal language model fine-tuning for text classification

    Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146,

    arXiv

  22. [2020]

    Sparks of artificial general intelligence: Early experiments with gpt-4

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712,

    arXiv

  23. [2021]

    Audrey Huang, Liu Leqi, Zachary C Lipton, and Kamyar Azizzadenesheli

    URL https://arxiv.org/abs/2106.09685. Audrey Huang, Liu Leqi, Zachary C Lipton, and Kamyar Azizzadenesheli. On the convergence and optimality of policy gradient for markov coherent risk. arXiv preprint arXiv:2103.02827,

    arXiv

  24. [2022]

    Real- toxicityprompts: Evaluating neural toxic degeneration in language models

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462,

    2009 arXiv

  25. [2023]

    Systematic rectification of language models via dead-end analysis

    Meng Cao, Mehdi Fatemi, Jackie Chi Kit Cheung, and Samira Shabanian. Systematic rectification of language models via dead-end analysis. arXiv preprint arXiv:2302.14003,

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.