REVIEW 7 cited by
Reward-Robust RLHF in LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
As Large Language Models (LLMs) continue to progress toward more advanced forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is increasingly seen as a key pathway toward achieving Artificial General Intelligence (AGI). However, the reliance on reward-model-based (RM-based) alignment methods introduces significant challenges due to the inherent instability and imperfections of Reward Models (RMs), which can lead to critical issues such as reward hacking and misalignment with human intentions. In this paper, we introduce a reward-robust RLHF framework aimed at addressing these fundamental challenges, paving the way for more reliable and resilient learning in LLMs. Our approach introduces a novel optimization objective that carefully balances performance and robustness by incorporating Bayesian Reward Model Ensembles (BRME) to model the uncertainty set of reward functions. This allows the framework to integrate both nominal performance and minimum reward signals, ensuring more stable learning even with imperfect RMs. Empirical results demonstrate that our framework consistently outperforms baselines across diverse benchmarks, showing improved accuracy and long-term stability. We also provide a theoretical analysis, demonstrating that reward-robust RLHF approaches the stability of constant reward settings, which proves to be acceptable even in a stochastic-case analysis. Together, these contributions highlight the framework potential to enhance both the performance and stability of LLM alignment.
Forward citations
Cited by 7 Pith papers
-
Multimodal Reward Hacking in Reinforcement Learning
Imperfect multimodal RL rewards systematically create new failures (NRFR > RHR); scaling and answer-aware rewards help but do not eliminate hacking, and unreliable visual verifiers actively increase it.
-
Robust General Utility for Reinforcement Learning
The paper introduces robust general-utility RL, a minimax formulation over utility uncertainty sets, and proves convergence rates for projected gradient descent-ascent and prox-extragradient algorithms.
-
SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text
A 24-dataset benchmark for inducing schema graphs from raw text, plus an auditable LLM-based pipeline that reports the highest scores on the benchmark's four schema-similarity metrics.
-
Bradley-Terry and Multi-Objective Reward Modeling Are Complementary
Jointly training a Bradley-Terry preference head and a multi-attribute regression head on a shared embedding improves reward-model robustness to reward hacking and boosts multi-objective scoring performance.
-
Ethics and Persuasion in Reinforcement Learning from Human Feedback: A Procedural Rhetorical Approach
RLHF-enhanced chatbots exert subtle procedural persuasion on users by reinforcing language norms, reshaping information seeking, and conditioning relationship expectations, creating overlooked ethical risks.
-
Knowledge Boundary of Large Language Models: A Survey
A survey that formalizes the knowledge boundary of LLMs into a four-type taxonomy and reviews detection and mitigation methods.
-
Baichuan4-Finance Technical Report
A finance-tuned LLM reportedly beats strong baselines on Chinese financial exams, but the training data includes exam questions and no contamination check is reported.
Discussion (0). Continue with ORCID to comment.