REVIEW 6 cited by
Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Reward models (RMs) are essential for aligning large language models (LLM) with human expectations. However, existing RMs struggle to capture the stochastic and uncertain nature of human preferences and fail to assess the reliability of reward predictions. To address these challenges, we introduce the Uncertainty-aware Reward Model (URM) and its ensemble variant, URME. URM employs a probabilistic value head to capture aleatoric uncertainty by modeling the distribution of disentangled human preference attributes. URME further quantifies epistemic uncertainty by examining discrepancies among individual URMs within the ensemble, enabling identification of unreliable evaluations. Our empirical evaluations demonstrate that URM achieves strong performance on RewardBench, outperforming competitive large-scale models. Additionally, extensive experiments, including best-of-n sampling (BoN), iterative direct preference optimization (iterative DPO), and proximal policy optimization (PPO), demonstrate that URM and URME significantly enhance LLMs' generation quality. Notably, reward predictions with lower uncertainty are far more reliable, demonstrate significantly higher quality, and result in substantially improved alignment.
Forward citations
Cited by 6 Pith papers
-
CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation
Fusing hidden states of multiple frozen reward models under shared-private constraints improves pairwise preference accuracy over scalar-score fusion and single-expert adapters on LM-Arena and PPE.
-
Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
RLSF uses a model's own answer-span confidence as an intrinsic reward to create preference data, then applies DPO or PPO to improve calibration and reasoning without external labels.
-
Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries
Dialog-act and maxim-aware prompting improves LLM judge accuracy on multi-turn preference data by up to 8 points, with further gains from jury-style voting.
-
Decision Flow Policy Optimization
Decision Flow frames the gradual action generation of flow-based policies as a flow MDP and updates the flow policy with flow-level value functions, reporting state-of-the-art results on several D4RL tasks.
-
Test-Time Scaling with Repeated Sampling Improves Multilingual Text Generation
Repeated sampling with a verifier improves open-ended multilingual generation, and reward-based verifiers are needed for math and code tasks.
-
Towards Reliable, Uncertainty-Aware Alignment
Variance-aware RLHF adds a variance-weighted KL penalty to PPO and reduces reward variance and the risk of underperforming the reference policy in the paper's experiments.
Discussion (0). Continue with ORCID to comment.