REVIEW 6 cited by
$\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Direct Preference Optimization (DPO) has emerged as a compelling approach for training Large Language Models (LLMs) to adhere to human preferences. However, the performance of DPO is sensitive to the fine-tuning of its trade-off parameter $\beta$, as well as to the quality of the preference data. We analyze the impact of $\beta$ and data quality on DPO, uncovering that optimal $\beta$ values vary with the informativeness of pairwise data. Addressing the limitations of static $\beta$ values, we introduce a novel framework that dynamically calibrates $\beta$ at the batch level, informed by data quality considerations. Additionally, our method incorporates $\beta$-guided data filtering to safeguard against the influence of outliers. Through empirical evaluation, we demonstrate that our dynamic $\beta$ adjustment technique significantly improves DPO's performance across a range of models and datasets, offering a more robust and adaptable training paradigm for aligning LLMs with human feedback. The code is available at \url{https://github.com/junkangwu/beta-DPO}.
Forward citations
Cited by 6 Pith papers
-
RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization
RLearner-LLM's Hybrid-DPO fuses DeBERTa NLI and LLM verifier scores to deliver up to 6x higher NLI entailment than standard SFT while preserving answer coverage across academic domains.
-
Adaptive Margin RLHF via Preference over Preferences
Adaptive margins for DPO inferred from preference-over-preference comparisons improve alignment quality, with random sampling of comparisons working best overall.
-
MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
A human-annotated multimodal preference dataset plus critique-based reward modeling and reward-margin-weighted DPO improves MLLM performance across many benchmarks.
-
LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information
LongDPO uses MCTS-collected, critique-refined stepwise preference pairs with step-level DPO to improve long-form generation quality and length adherence.
-
Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking
TailorPO aligns diffusion models by generating paired noisy samples from the same intermediate state, ranking them by the reward of their predicted clean images, and optimizing the preferred one at every denoising step.
-
R.I.P.: Better Models by Survival of the Fittest Prompts
RIP filters preference-optimization training data by keeping prompts whose rejected responses are high quality and whose chosen/rejected reward gap is small, yielding consistent benchmark gains over unfiltered data.
Discussion (0). Continue with ORCID to comment.