REVIEW 6 cited by
Quantile Regression for Distributional Reward Models in RLHF
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Reinforcement learning from human feedback (RLHF) has become a key method for aligning large language models (LLMs) with human preferences through the use of reward models. However, traditional reward models typically generate point estimates, which oversimplify the diversity and complexity of human values and preferences. In this paper, we introduce Quantile Reward Models (QRMs), a novel approach to reward modeling that learns a distribution over rewards instead of a single scalar value. Our method uses quantile regression to estimate a full, potentially multimodal distribution over preferences, providing a more powerful and nuanced representation of preferences. This distributional approach can better capture the diversity of human values, addresses label noise, and accommodates conflicting preferences by modeling them as distinct modes in the distribution. Our experimental results show that QRM outperforms comparable traditional point-estimate models on RewardBench. Furthermore, we demonstrate that the additional information provided by the distributional estimates can be utilized in downstream applications, such as risk-aware reinforcement learning, resulting in LLM policies that generate fewer extremely negative responses. Our code and model are released at https://github.com/Nicolinho/QRM.
Forward citations
Cited by 6 Pith papers
-
Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.
-
Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
RLSF uses a model's own answer-span confidence as an intrinsic reward to create preference data, then applies DPO or PPO to improve calibration and reasoning without external labels.
-
PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalization
PersonaFeedback provides a human-labeled benchmark showing current LLMs, including strong reasoners, score only about 65-70 percent on hard personalization choices, and explicit persona information helps more than retrieval.
-
Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries
Dialog-act and maxim-aware prompting improves LLM judge accuracy on multi-turn preference data by up to 8 points, with further gains from jury-style voting.
-
Multi-Domain Explainability of Preferences
A concept-discovery plus hierarchical regression pipeline explains human, LLM-judge, and reward-model preferences at local and global levels across eight domains.
-
Safety Game: Inference-Time Alignment of Black-Box LLMs via Constrained Optimization
A linear-programming 'safety game' selects among LLM candidate answers to maximize helpfulness under a self-reported risk cap, improving safety-benchmark accuracy over reranking baselines in multiple-choice settings.
Discussion (0). Sign in to comment.