Pith. sign in

REVIEW 6 cited by

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.16635 v3 pith:33H6ZU7C submitted 2024-01-30 cs.LG cs.AIcs.CL

Shun Zhang , Zhenfang Chen , Sunli Chen , Yikang Shen , Zhiqing Sun , Chuang Gan This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords ensemblerewardhumanrlhfmodelmodelsefficientfeedback
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Reinforcement Learning from Human Feedback (RLHF) is a widely adopted approach for aligning large language models with human values. However, RLHF relies on a reward model that is trained with a limited amount of human preference data, which could lead to inaccurate predictions. As a result, RLHF may produce outputs that are misaligned with human values. To mitigate this issue, we contribute a reward ensemble method that allows the reward model to make more accurate predictions. As using an ensemble of large language model-based reward models can be computationally and resource-expensive, we explore efficient ensemble methods including linear-layer ensemble and LoRA-based ensemble. Empirically, we run Best-of-$n$ and Proximal Policy Optimization with our ensembled reward models, and verify that our ensemble methods help improve the alignment performance of RLHF outputs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCOPE: Synthetic Conditional Objectives for Policy Evolution in Black-Box Combinatorial Optimization

    cs.AI 2026-07 conditional novelty 6.0 of 10

    SCOPE evolves LLM-generated auxiliary objective functions and selects a validated portfolio of them to guide fixed combinatorial search engines under strict black-box query budgets.

  2. Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Applying importance weighting to reward model training to correct for policy distribution shift in RLHF improves final policy quality without new labels.

  3. Bradley-Terry and Multi-Objective Reward Modeling Are Complementary

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Jointly training a Bradley-Terry preference head and a multi-attribute regression head on a shared embedding improves reward-model robustness to reward hacking and boosts multi-objective scoring performance.

  4. Ensembles of Low-Rank Expert Adapters

    cs.CL 2025-01 conditional novelty 5.0 of 10

    ELREA clusters instruction-tuning data by gradient direction, trains one LoRA expert per cluster, and routes new instructions to experts via gradient similarity, giving modest benchmark gains over full-data LoRA.

  5. Towards Reliable, Uncertainty-Aware Alignment

    cs.LG 2025-07 reject novelty 4.0 of 10

    Variance-aware RLHF adds a variance-weighted KL penalty to PPO and reduces reward variance and the risk of underperforming the reference policy in the paper's experiments.

  6. Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy

    cs.AI 2025-05 reject novelty 4.0 of 10

    AI copilot preference optimization is organized into a pre-, mid-, and post-interaction taxonomy, with a unified definition of AI copilots.

Pith tools