{"id":"819691d2-1230-45ff-a220-4221e9b7534f","arxiv_id":"2501.06911","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning a language model on its worst-scoring responses, using a CVaR-style schedule, reduces negative and toxic generations more than standard RLHF on IMDB, Jigsaw, and RealToxicityPrompts.","lead":"Chatbot fine-tuning usually maximizes average quality; this paper instead keeps only a model's worst-scoring responses during training, a risk-averse strategy aimed at rare toxic outputs. On sentiment and toxicity benchmarks the method beats standard fine-tuning on average and on the hardest prompts, with a small coherence cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported safety gains rest on the same classifier that supplies the training reward; without an independent toxicity/sentiment evaluation on tail prompts, RA-RLHF's better scores may be reward hacking, not less toxic output.","rationale":"The reader's conditional verdict already identifies the reward models as the weakest assumption, and I agree. The concern is specific and testable: Eq. (8) and Algorithm 1 select and train on low-return trajectories according to R(τ), and Section 5 uses the same R for all headline numbers. This is a measurement-validity gap, not a disagreement with community consensus; it is a standard failure mode of reward optimization. Single-seed GPT-J, missing hyperparameters, and the omitted CVaR-bias analysis are real but secondary: even perfectly repeated experiments would not establish actual toxicity reduction if the yardstick itself is gameable. The paper does several things well—consistent pattern across three datasets, released codebase, training-stability and length checks—so the concern is not that the method is uninteresting, only that the central claim as worded overreaches the evidence. An independent tail-prompt evaluation would settle it cheaply, and it would either strengthen the paper (if RA-RLHF wins on an external metric) or require a more modest claim. I therefore leave the verdict at conditionally accepted rather than moving it.","tokens_in":29640,"tokens_out":5324,"duration_ms":56551,"concrete_test":"Using the released checkpoints for RLHF and RA-RLHF, score generations on the held-out test-tail prompts with an independent instrument: Perspective API (the original source for RealToxicityPrompts) or a separately trained toxicity classifier for Jigsaw-Gen/RealToxicityPrompts-Gen, and human raters or a non-TRL sentiment model for IMDB-Gen. Compare per-quantile mean scores, not just aggregate means, and report the correlation between the internal reward model and the external measure on the tail. If RA-RLHF no longer dominates RLHF under the external measure, the central claim should be narrowed to \"improves the training surrogate\" rather than \"performs better over all prompts\" in terms of actual content.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that RA-RLHF \"actually perform[s] better\" and reduces negative/toxic content. In the implementation, R(τ) in Algorithm 1 and Eq. (8) is computed by a fixed reward model (lvwerra/distilbert-imdb for IMDB-Gen, unitary/toxic-bert for Jigsaw-Gen and RealToxicityPrompts-Gen; §5, E.1). Those same models then define every reported evaluation metric: sentiment and toxicity scores, the tail-prompt threshold, and the quantile curves in Figs. 1–2. This shared measurement pipeline does not invalidate the algorithm-vs-algorithm comparison as surrogate optimization, but it cannot support the paper's real-world safety claim unless the classifiers are faithful on tail prompts. RLHF/RA-RLHF policies can in principle achieve high classifier scores by exploiting surface artifacts—positive trigger words, length changes, template repetition—while leaving actual sentiment or toxicity unchanged. The paper's perplexity, response-length, and diversity checks reduce some of this risk but do not test semantic content. The limitation appendix (A) discusses domain breadth, not reward-model validity. Since the same metric also selects which prompts are \"worst case\", any miscalibration on negative/toxic prompts is directly baked into both training and evaluation, making the concern load-bearing, not incidental.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Risk-Averse RLHF (RA-RLHF), a modification of the standard RLHF pipeline in which each PPO update uses only the B0 lowest-return trajectories, with a soft-risk schedule that starts with the full batch and gradually reduces the selection size to a CVaR-determined fraction. The authors test RA-RLHF on three conditional-generation tasks (IMDB sentiment, Jigsaw toxicity, RealToxicityPrompts) using GPT-2 and GPT-J, reporting that RA-RLHF achieves higher average and tail rewards than RLHF while maintaining comparable perplexity and diversity. The central claim is that replacing the expected-reward objective with a CVaR-style worst-return selection improves both tail behavior and average performance over a standard RLHF-tuned policy.","tokens_in":29905,"tokens_out":7646,"duration_ms":69386,"significance":"The paper makes a useful, practical contribution: RA-RLHF is a simple, drop-in modification to existing RLHF implementations, the code is released, and the algorithm-vs-algorithm comparison on the stated reward proxies is consistent across three GPT-2 datasets and a single-seed GPT-J run. The soft-risk scheduling and the explicit focus on worst-case prompts are sensible adaptations of prior risk-averse RL work (Greenberg et al.). The main limitation is that the safety interpretation of the results is not supported by the measurement design: the same classifier reward models are used for training selection, for the evaluation metrics, and for defining the tail prompts, so the reported gains may reflect optimization of classifier-specific artifacts rather than a genuine reduction in toxic or negative content. If the authors add independent safety evaluation (e.g., Perspective API, human ratings, or a held-out reward model) on the tail prompts and clarify the statistical reporting, the work would be a solid empirical contribution to safe LLM fine-tuning.","major_comments":[{"comment":"The reward models lvwerra/distilbert-imdb and unitary/toxic-bert are used both as the training reward (Eq. (8) and Algorithm 1) and as the evaluation metric (the 'Standard task performance scores' in §5). Consequently, the reported improvements may reflect overfitting to these classifiers rather than reduced negative or toxic content. The abstract's safety claim requires independent evaluation on tail prompts (e.g., Perspective API, human ratings, or a held-out reward model) and a demonstration that the generated outputs differ in semantically meaningful ways; the perplexity and diversity metrics do not address this measurement overlap.","section":"§5, §E.1"},{"comment":"The statistical support is inconsistent. Table 3 reports standard deviations for RealToxicityPrompts-Gen, but Appendix F.4 (Table 10) states 'Results are for one seed,' and the GPT-J results in Table 4 are single-seed. The SFT baseline has no standard deviation due to 'faulty seeding'. The claim of consistent improvement over RLHF across datasets should be backed either by three-seed results for all reported configurations or by explicit labeling of preliminary single-seed results.","section":"§5.1, Tables 3-4, §F.4"},{"comment":"The paper states that selecting the B0 lowest-return trajectories is a 'simplified approach similar to (7),' but no derivation is provided for why this selection, combined with the KL-regularized objective and a critic updated only on selected trajectories, optimizes the CVaR objective. As written, the algorithm is a heuristic that may deviate substantially from CVaR policy gradients; the authors should either state the conditions under which the approximation holds or present RA-RLHF as an empirical heuristic rather than as a CVaR optimizer.","section":"§4, Eq. (8), Algorithm 1"},{"comment":"The claim that RA-RLHF 'actually perform[s] better than a traditional RLHF-tuned policy over all prompts' is supported only by quantile plots without error bars; the quantitative tables (Tables 2-4) report tail rewards, not all-prompt averages. The authors should add an all-prompt average reward table with standard deviations or temper the claim to tail performance.","section":"§1, Figs. 1-2, Table 4"}],"minor_comments":[{"comment":"The variable m in the formula for B0 is not defined; it should be the iteration index i, as in points A and B.","section":"§4, risk schedule point C"},{"comment":"The word 'retrun' should be 'return'.","section":"Algorithm 1, line 10"},{"comment":"The table formatting is difficult to parse: the RLHF row of Table 2 contains an extra value '0.36', and standard deviations appear as separate columns rather than subscripts; Table 3 has a similar layout. Please align the columns with the stated headers.","section":"Table 2 and Table 3"},{"comment":"The sentence 'The outputs in are evaluated using a standardized scoring models' is missing a noun and should read 'The outputs are evaluated using standardized scoring models'.","section":"§5, first paragraph of Evaluation Metrics"},{"comment":"Table 4's caption says 'Testing on reward (r), and Perplexity,' but the table only shows Tail (r) and Perplexity; either add the all-prompt Reward (r) column or adjust the caption to match the content.","section":"Table 4 and Appendix F.3, Table 9"},{"comment":"Table 10 says 'Results are for one seed,' which conflicts with the standard deviations reported for RealToxicityPrompts-Gen in Table 3; please reconcile the two presentations.","section":"Appendix F.4, Table 10"},{"comment":"The warm-start iterations are denoted n in Table 5 but i0 in Section 4; please unify the notation.","section":"§5.3, Table 5"},{"comment":"The caption contains the typo 'Nagative' instead of 'Negative'.","section":"Table 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper's NeurIPS-style checklist asserts full statistical significance, but the text reports single-seed results for GPT-J, an inconsistency for RealToxicityPrompts, and no standard deviation for SFT; the editor may want to verify the underlying logs. The core algorithmic idea is useful and the empirical comparison on the given reward proxies is plausible, but the safety framing needs an independent evaluation before the claims can be taken at face value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is the first clean application of CVaR-style soft-risk scheduling to RLHF, and the core relative claim — RA-RLHF improves on expected-reward RLHF on tail and average across the three tasks — is supported by the data. The paper's stronger real-world safety language, though, rests on a measurement overlap: the same classifier supplies the training reward and the evaluation, so the gains are gains on that proxy, not independently verified reductions in toxicity.\n\nWhat's new is the combination. The ingredients — CVaR policy gradients, soft-risk scheduling from CeSoR, PPO, the standard RLHF loop — are prior work, but putting them together for LLM fine-tuning and showing consistent gains on IMDB-Gen, Jigsaw-Gen, and RealToxicityPrompts-Gen is a legitimate first application. The paper is also fairly careful. RLHF and RA-RLHF get three seeds, perplexity and diversity checks are included, the response-length plots guard against degenerate repetition, there's a hyperparameter sweep, and code is linked. The authors disclose the SFT seeding bug and the GPT-J resource limits rather than hiding them.\n\nThe stress-test concern holds up and is load-bearing. Section 5 and E.1 list lvwerra/distilbert-imdb and unitary/toxic-bert as the reward models; those same models compute every reported evaluation metric, including the quantile curves and the tail thresholds. If the classifiers are miscalibrated or gameable on tail prompts, that flaw enters both training and evaluation. Perplexity and diversity reduce the risk but don't test semantic content. The limitation appendix discusses domain breadth, not reward-model validity. This does not invalidate the relative claim — RA-RLHF beats RLHF on the surrogate — but it means the abstract's 'minimizing harmful outputs' is not independently established. The fix is either an external evaluation (e.g., Perspective API or human ratings) or a softer claim.\n\nMinor soft spots: GPT-J and RealToxicity results are single-seed, RealToxicity hyperparameters are missing from the table, and the worst-batch approximation to the CVaR gradient is stated without derivation or bias analysis. The tables are dense and hard to parse, which hurts reproducibility. None of this undermines the central comparison.\n\nThis paper is for people working on risk-sensitive RLHF or classifier-based safety training. It deserves a serious referee. The experiment setup is re-implementable, the novelty is modest but real, and the caveats are addressable. I'd send it out, asking for an independent toxicity evaluation or a softened claim plus the missing details. Desk rejection would be wrong.","headline":"First clean CVaR-style RA-RLHF; relative gains on the proxy are solid, but the real-world safety claim is undermined by using the same classifier for reward and evaluation.","tokens_in":30440,"tokens_out":2823,"would_cite":true,"duration_ms":27573,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing expected reward with a CVaR-style worst-return objective in RLHF makes LLMs safer on toxic prompts and better on average, the paper claims.","keywords":["risk-averse reinforcement learning","conditional value at risk","RLHF","large language models","toxicity mitigation","sentiment modification","soft-risk scheduling","safe text generation"],"falsifier":"Have human raters, or a toxicity or sentiment classifier never used during training, score completions from RA-RLHF and RLHF on the same tail prompts; if the RA-RLHF completions are not rated less toxic or more positive than the RLHF completions, the paper's central claim fails.","tokens_in":29436,"feed_emoji":"🛡️","tokens_out":9481,"duration_ms":81796,"temperature":0.7,"pith_summary":"The paper sets out to show that LLM fine-tuning can be made risk-averse by optimizing a tail-risk objective instead of average reward, and that this reduces toxic or negative generations without sacrificing overall quality. It proposes RA-RLHF, a variant of RLHF that, after a warm-up phase, updates the policy only on the completions with the worst rewards in each batch, approximating Conditional Value at Risk (CVaR). On sentiment-completion and toxicity-mitigation tasks with two base LLMs (117M and 6B parameters), the resulting policies shift both negative and positive prompts toward positive or non-toxic completions, beat standard RLHF in the riskiest prompt quantiles, and also score higher on average over all test prompts. If this holds, safety-conscious fine-tuning needs no second reward model and no constrained optimization, just a change in which trajectories are used for the policy update.","feed_headline":"Training on worst outputs cuts LLM toxicity, beats RLHF","feed_subtitle":"A CVaR-style objective lifts sentiment and toxicity scores on the hardest prompts while keeping generation quality.","key_machinery":"The engine is Conditional Value at Risk (CVaR), the average return in the worst $\\alpha$ fraction of the return distribution, applied through trajectory selection on a KL-regularized reward. Concretely, RA-RLHF keeps the $B_0 = \\lceil \\alpha B \\rceil$ lowest-return completions from each batch and runs PPO only on those, using the standard RLHF per-token reward $\\bar r(s_t,a_t) = r(s_t,a_t) - \\beta \\log(\\pi_\\theta(a_t|s_t)/\\pi_{\\mathrm{ref}}(a_t|s_t))$, where $\\pi_{\\mathrm{ref}}$ is the supervised fine-tuned policy. A soft-risk schedule handles the exploration-versus-risk trade-off: for the first $i_0$ iterations the full batch is used so the model learns to produce successful completions, then the selection threshold decays at a constant rate from the full batch to $\\lceil \\alpha B \\rceil$ by iteration $\\lfloor \\rho M \\rfloor$. This schedule, together with the KL regularization to a positive-data fine-tuned base, is what lets the policy concentrate on hazardous prompts without losing the ability to generate coherent text.","core_discovery":"Traditional RLHF maximizes expected reward, so rare but very harmful outputs can be masked by many safe outputs. RA-RLHF replaces that objective with a CVaR-style one: each training iteration samples a batch of completions, scores them with a fixed reward model, and keeps only the lowest-scoring $B_0$ trajectories for the PPO update, with $B_0$ initially equal to the full batch and then shrinking linearly to $\\lceil \\alpha B \\rceil$ as training progresses. The paper finds that on IMDB sentiment completion, Jigsaw toxicity mitigation, and RealToxicityPrompts, this worst-return selection produces policies whose reward distributions shift further toward positive/non-toxic text than SFT or RLHF, that dominate RLHF on the hardest prompt quantiles, and that achieve higher average reward over all prompts with only a small perplexity increase. The authors state this as: RA-RLHF policies do not merely show risk-aversion on negative or toxic prompts, they perform better than a traditional RLHF-tuned policy over all prompts.","pith_inferences":["The paper does not test this, but because the same reward model selects training trajectories and scores the final outputs, part of the reported gap could be reward hacking; a human or held-out classifier evaluation would show whether the actual toxicity of generations drops.","A natural extension the authors leave implicit is to apply worst-return selection to alignment losses other than PPO-based RLHF, such as direct preference optimization, giving those methods tail-risk behavior as well.","The soft-risk schedule is effectively a curriculum over the return distribution; one could make the risk level or the warm-up length depend on the riskiness of each prompt class rather than using a global quantile.","If the toxicity reward model is miscalibrated on particular harm categories, RA-RLHF's selection will inherit that blind spot, so auditing the scorer's calibration per category is a direct test of the method's safety claim."],"forward_implications":["On the three evaluated tasks, RA-RLHF shifts the reward distribution of generated completions upward more than SFT or RLHF, with the largest gains on the prompts whose starting rewards are worst.","The average reward on the worst-case prompt tail is higher for RA-RLHF than for RLHF on both the 117M-parameter and the 6B-parameter base models.","The improvement does not come from degenerate repetition: RA-RLHF keeps response lengths near those of RLHF, maintains diversity scores, and pays only a small perplexity penalty.","Because RA-RLHF needs a single reward model, it offers a simpler route to tail-risk safety than approaches that require separate helpfulness and harmlessness models."],"supporting_citations":[{"why":"Supplies the soft-risk scheduling and the sample-based CVaR gradient estimate that RA-RLHF adapts to the RLHF setting.","marker":"Greenberg et al. [2022]"},{"why":"Defines CVaR policy gradients, the risk objective that RA-RLHF replaces expected reward with.","marker":"Tamar et al. [2015]"},{"why":"Frames language generation as a token-level MDP and provides the IMDB-Gen task setup used for sentiment modification.","marker":"Ramamurthy et al. [2022]"},{"why":"Provides the KL-regularized, PPO-based RLHF update with adaptive KL coefficient that RA-RLHF inherits.","marker":"Ziegler et al. [2019]"},{"why":"Provides the Proximal Policy Optimization algorithm used for all policy updates.","marker":"Schulman et al. [2017]"},{"why":"Supplies the IMDB movie-review dataset used for the sentiment task.","marker":"Maas et al. [2011]"},{"why":"Supplies the Wikipedia comment toxicity dataset used for Jigsaw-Gen.","marker":"[Jigsaw, 2017]"},{"why":"Supplies the RealToxicityPrompts dataset and motivates the task of avoiding toxic continuations.","marker":"Gehman et al. [2020]"},{"why":"Quark, a selective fine-tuning baseline, is one of the state-of-the-art methods RA-RLHF is compared against on toxicity.","marker":"Lu et al. [2022]"},{"why":"DExperts is a decoding-time baseline, and its distinct-n metrics are used for evaluating textual diversity.","marker":"Liu et al. [2021]"}],"fun_headline_variants":["Worst-case training tames LLM toxicity, tops RLHF","CVaR fine-tuning beats RLHF on toxic prompts","Risk-averse RLHF: focus on worst outputs to cut toxicity","Selecting worst responses in RLHF yields safer LLMs","RA-RLHF: Shifting focus to rare harms improves LLM safety"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on the scoring models used to select training trajectories and to grade outputs being accurate measures of sentiment and toxicity; if those models are gameable or miscalibrated on the hardest prompts, the claimed reduction in negative content may not be real.","fun_headline_variants_meta":{"raw":{"variants":["Worst-case training tames LLM toxicity, tops RLHF","CVaR fine-tuning beats RLHF on toxic prompts","Risk-averse RLHF: focus on worst outputs to cut toxicity","Selecting worst responses in RLHF yields safer LLMs","RA-RLHF: Shifting focus to rare harms improves LLM safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1355,"prompt_tokens":852,"completion_tokens":503,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":414}},"tokens_in":468,"tokens_out":503,"duration_ms":5165,"temperature":1.0,"reasoning_tokens":414,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:37.634970+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human raters, or a toxicity or sentiment classifier never used during training, score completions from RA-RLHF and RLHF on the same tail prompts; if the RA-RLHF completions are not rated less toxic or more positive than the RLHF completions, the paper's central claim fails.","supporting_citations":[],"review_version":1}