Pith. sign in

REVIEW 5 cited by

Bayesian Reward Models for LLM Alignment

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.13210 v2 pith:5FWM6ENK submitted 2024-02-20 cs.LG

Adam X. Yang , Maxime Robeyns , Thomas Coste , Zhengyan Shi , Jun Wang , Haitham Bou-Ammar , Laurence Aitchison This is my paper
classification cs.LG
keywords rewardresponsesmodelbayesiandatahighrewardsfurther
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

To ensure that large language model (LLM) responses are helpful and non-toxic, a reward model trained on human preference data is usually used. LLM responses with high rewards are then selected through best-of-$n$ (BoN) sampling or the LLM is further optimized to produce responses with high rewards through reinforcement learning from human feedback (RLHF). However, these processes are susceptible to reward overoptimization or `hacking', where responses receive high rewards due to imperfections in the reward model rather than true preference, particularly as prompts or responses deviate from the training data. To address these challenges, we propose to train a Bayesian reward model, which signals higher uncertainty further from the training data distribution. We trained Bayesian reward models using Laplace approximation on LoRA weights, and found that the resulting uncertainty estimates can effectively mitigate reward overoptimization in BoN sampling.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Aligned Query Expansion: Efficient Query Expansion for Information Retrieval through LLM Alignment

    cs.IR 2025-07 conditional novelty 6.0 of 10

    AQE uses retrieval rank as a preference signal to fine-tune T0 with RSFT and DPO, beating generate-then-filter baselines on four QA datasets.

  2. Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting

    cs.LG 2025-08 reject novelty 5.0 of 10

    S-GRPO reweights GRPO advantages by an assumed noise level p to down-weight unbalanced groups, claiming improved and more noise-robust math reasoning training.

  3. ConceptSMILE: Auditing the Trustworthiness of Concept-Based Explainable AI

    cs.AI 2026-07 conditional novelty 4.0 of 10

    A perturbation-and-surrogate audit shows MedSAM and VLM retinal concept explanations have pathway- and concept-specific reliability, not automatic trustworthiness.

  4. The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives

    cs.LG 2025-10 conditional novelty 4.0 of 10

    Bayesian IRL with sequential posterior updates can recover a usable toxicity-reduction reward from LLM demonstrations and reproduce ground-truth RLHF detoxification.

  5. Optimising Language Models for Downstream Tasks: A Post-Training Perspective

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A dissertation that repackages the author's previously published papers on continued pre-training, prompt tuning, and instruction modelling into a single narrative.

Pith tools