REVIEW 48 cited by
Direct Language Model Alignment from Online AI Feedback
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Direct alignment from preferences (DAP) methods, such as DPO, have recently emerged as efficient alternatives to reinforcement learning from human feedback (RLHF), that do not require a separate reward model. However, the preference datasets used in DAP methods are usually collected ahead of training and never updated, thus the feedback is purely offline. Moreover, responses in these datasets are often sampled from a language model distinct from the one being aligned, and since the model evolves over training, the alignment phase is inevitably off-policy. In this study, we posit that online feedback is key and improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as annotator: on each training iteration, we sample two responses from the current model and prompt the LLM annotator to choose which one is preferred, thus providing online feedback. Despite its simplicity, we demonstrate via human evaluation in several tasks that OAIF outperforms both offline DAP and RLHF methods. We further show that the feedback leveraged in OAIF is easily controllable, via instruction prompts to the LLM annotator.
Forward citations
Cited by 48 Pith papers
-
Safety Alignment of LMs via Non-cooperative Games
Jointly training an Attacker and Defender LLM in a non-zero-sum game with pairwise preference judges produces a defender with much lower jailbreak success while preserving general utility.
-
Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Function
ZSPO provably converges to a stationary policy using only the sign of preference feedback, without knowing the link function between preferences and rewards.
-
POPri: Private Federated Learning using Preference-Optimized Synthetic Data
POPri uses client similarity scores as RL rewards to DPO-tune an LLM for DP synthetic data generation, outperforming prior private evolution baselines on next-token prediction and classification.
-
Preference learning made easy: Everything should be understood through win rate
Under two axioms (preference-consistency and prevalence-consistency), the only distribution-grounded evaluation for preference learning is h-win rate, and most popular alignment methods can be classified by whether th...
-
SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
SP3O is a reward-model-free, critic-free, gradient-based RL algorithm that optimizes policies from segment-level preferences in stochastic MDPs via off-policy importance sampling and PPO-style clipping.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Noisy Pairwise-Comparison Random Search for Smooth Nonconvex Optimization
Noisy-comparison random search reaches ε-stationarity in O(k/(p²ε²)) comparisons for smooth nonconvex objectives with k-dimensional active subspace.
-
Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
LLM attention maps reveal a preplan-and-anchor pattern, and reweighting RL credit toward the flagged tokens improves math/QA reasoning.
-
Bridging Offline and Online Reinforcement Learning for LLMs
Semi-online DPO, which syncs the generation model every few update steps, performs nearly as well as fully online DPO and GRPO, while strongly beating offline DPO.
-
TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
TGDPO modifies DPO by weighting each token's log-ratio with an external token-level reward, and reports win-rate gains over DPO and SimPO on three instruction-following benchmarks.
-
Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling
Reweighting offline DPO pairs by the likelihood ratio pi_theta/pi_ref reduces reward over-optimization and keeps the model closer to the reference policy.
-
Customizing Speech Recognition Model with Large Language Model Feedback
LLM log-probability scores combined with acoustic scores serve as RL rewards to adapt ASR models to new domains without labeled data.
-
Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment
Post-training alignment on synthetic graph data yields statistically significant gains on some real-world implicit-graph tasks, but gains are inconsistent and multi-step reasoning remains fragile.
-
Online Knowledge Distillation with Reward Guidance
A preference-based knowledge distillation framework uses a confidence-set reward model in a min-max imitation game, with offline, online, and white-box variants, and outperforms prior KD baselines on LLM benchmarks.
-
MPO: Multilingual Safety Alignment via Reward Gap Optimization
MPO reduces jailbreak success in multilingual LLMs by regressing target-language reward gaps onto the English reward gap, outperforming DPO and related methods while preserving utility.
-
Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
A generative reward model trained with long chain-of-thought and rule-based RL outperforms standard and vertically scaled reward baselines on RM-Bench and RewardBench.
-
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
Two co-trained reward models that filter each other's training batches via peer review and synchronized curriculum learning improve reward-model accuracy and downstream policy win-rates under injected preference noise.
-
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning
Mixing on-policy and off-policy preference data in equal proportions (SIMPLEMIX) improves DPO alignment over either source alone, with on-policy data best for reasoning tasks and off-policy data best for open-ended tasks.
-
Calibrating Translation Decoding with Quality Estimation on LLMs
Optimizing the Pearson correlation between hypothesis likelihood and an external quality score during fine-tuning improves LLM translation quality and turns log-likelihood into a competitive reference-free quality estimator.
-
Process Reward Models for LLM Agents: Practical Framework and Directions
A lightweight actor-critic loop trains 3B LLM agents to surpass GPT-4o on ALFWorld by learning step-level reward models from rollouts or demonstrations.
-
Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator
Sampling dispreferred completions proportionally to the current reward model, as in contrastive divergence, improves preference-optimization performance and is framed as NLL estimation.
-
PILAF: Optimal Human Preference Sampling for Reward Modeling
A response-pair sampling scheme that interpolates current and reference model logits is proposed and claimed to align DPO gradients with the oracle reward gradient, with empirical gains in iterative and online DPO.
-
Online Preference Alignment for Language Models via Count-based Exploration
COPO adds a pseudo-count exploration bonus to DPO-style preference optimization and reports improved online RLHF results on Zephyr and Llama-3.
-
Understanding the Logic of Direct Preference Alignment through Logic
Direct preference alignment losses can be expressed as logical programs over model predictions, yielding an organized landscape of billions of definable losses and a route to new variants.
-
Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution
A multimodal LLM can improve itself using only unlabeled images by self-generating questions, self-enhancing answers, and adding a description-alignment loss to DPO.
-
Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
A hybrid online-plus-offline preference optimization algorithm, HPO, provably needs fewer samples than pure online or offline RLHF in linear MDP settings.
-
T-REG: Preference Optimization with Token-Level Reward Regularization
T-REG adds contrastive-prompt self-generated token-level rewards as a weighted regularizer to DPO and SimPO, improving Alpaca Eval 2 length-controlled win rate by up to 3.8% and Arena-Hard win rate by up to 4.4% over ...
-
Learning from Relevant Subgoals in Successful Dialogs using Iterative Training for Task-oriented Dialog Systems
SUIT uses counterfactual turn replacements to select high-quality subgoal training samples and iteratively retrains task-oriented dialog models, reaching a COMBINED score of 105.02 on MultiWOZ 2.2.
-
MR. Judge: Multimodal Reasoner as a Judge
MR. Judge turns MLLM judgment into a multiple-choice reasoning task and uses synthetic negative responses plus distilled reasoning traces to train a 7B judge that beats GPT-4o on VL-RewardBench.
-
Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations
Preference Tree Optimization uses look-ahead simulations scored by an AI oracle to generate DPO preference data, and the resulting Motivational Interviewing agent scores higher on that same oracle than the base model.
-
Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understanding
CoT data curated by two-round LLM prompting and VLM verification, then SFT+GRPO with fine-grained rewards, improves MapDR rule–lane association F1 from 0.642 to 0.723.
-
From Answers to Rationales: Self-Aligning Multimodal Reasoning with Answer-Oriented Chain-of-Thought
Answer-oriented chain-of-thought prompts that generate both positive and negative reasoning data, combined with iterative DPO, improve multimodal LLM reasoning on several benchmarks.
-
MOSLIM:Align with diverse preferences in prompts through reward classification
A prompt-controlled multi-objective alignment method using a multi-head classification reward model and a z-score reward mapping, claimed to work with off-the-shelf models.
-
Self-Training Large Language Models with Confident Reasoning
CORE-PO self-trains LLMs to prefer reasoning paths with high self-estimated confidence, improving answer and reasoning accuracy on several benchmarks.
-
DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
DGRO decouples the KL regularization coefficient in reward optimization into two hyperparameters and shows strong reasoning results, though its reward-variance ablation is confounded.
-
A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
A large collaborative survey organizes LLM and LLM-agent safety issues into a full-stack lifecycle framework from data preparation to deployment.
-
Compass-V2 Technical Report
Compass-v2 is an efficient MoE language model for Southeast Asian languages and e-commerce that claims top-tier performance among models of its size using a unified fast/deep reasoning design.
-
Direct Advantage Regression: Aligning LLMs with Online AI Reward
DAR aligns LLMs through advantage-weighted supervised fine-tuning on online AI scalar rewards with dual KL regularization, and reports win-rate gains over online RLHF and online preference methods.
-
LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
LARPO, an iterative preference optimization method that adapts information retrieval techniques such as listwise ranking losses, hard negatives, and candidate lists, is claimed to substantially improve LLM alignment o...
-
Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
RPO unifies DPO, IPO, SimPO, and RLOO as special cases of one reward-matching loss, and a synthetic-judge ablation finds online RPO-bwd outperforms RLOO.
-
Robust Single-Stage Fully Sparse 3D Object Detection via Detachable Latent Diffusion
The submitted manuscript's abstract and full text are mismatched; the claimed 3D detection method is not present in the body.
-
Data Diversification Methods In Alignment Enhance Math Performance In LLMs
DTS, which generates diverse solution strategies before writing solutions, improves GSM8K by 7.1 points and MATH by 4.2 points over an untuned base model at 1.03x baseline compute.
-
InfoPO: On Mutual Information Maximization for Large Language Model Alignment
InfoPO's mutual information derivation fails: its rejected-response regularizer is constant in expectation, leaving plain supervised fine-tuning on chosen responses.
-
Aligning LLMs with Domain Invariant Reward Models
Applying Wasserstein-distance domain adaptation, a known technique, to reward models lets preference signals learned on labeled source data transfer to unlabeled target domains, with consistent but modest gains across...
-
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Cal-DPO modifies DPO by adding squared losses that anchor the policy's implicit rewards of chosen and rejected responses to plus or minus 1/(2*beta), reporting gains on reasoning, summarization, and dialogue benchmarks.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
-
Reinforcement Learning from Human Feedback
The book introduces the origins, mathematical setup, and optimization stages of RLHF including reward modeling, reinforcement learning, rejection sampling, and direct alignment algorithms.
-
Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.
Discussion (0). Continue with ORCID to comment.