REVIEW 4 cited by
RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We propose Reinforcement Learning from Contrastive Distillation (RLCD), a method for aligning language models to follow principles expressed in natural language (e.g., to be more harmless) without using human feedback. RLCD creates preference pairs from two contrasting model outputs, one using a positive prompt designed to encourage following the given principles, and one using a negative prompt designed to encourage violating them. Using two different prompts causes model outputs to be more differentiated on average, resulting in cleaner preference labels in the absence of human annotations. We then use the preference pairs to train a preference model, which is in turn used to improve a base unaligned language model via reinforcement learning. Empirically, RLCD outperforms RLAIF (Bai et al., 2022b) and context distillation (Huang et al., 2022) baselines across three diverse alignment tasks--harmlessness, helpfulness, and story outline generation--and when using both 7B and 30B model scales for simulating preference data.
Forward citations
Cited by 4 Pith papers
-
Enhancing Safe and Controllable Protein Generation via Knowledge Preference Optimization
A knowledge-graph-guided preference optimization framework that fine-tunes protein language models to generate fewer sequences similar to known harmful proteins.
-
Aligning VLM Assistants with Personalized Situated Cognition
The authors present PCogAlignBench, an 18k-sample benchmark of visual scenes with role-based users, and PCogAlign, a framework using a cognition-aware reward model to produce responses aligned with each user's roles.
-
Probability-Consistent Preference Optimization for Enhanced LLM Reasoning
PCPO selects preference pairs by combining correct-answer status with token-level probability consistency, then trains with a weighted DPO+NLL loss, yielding small and partly inconsistent gains over outcome-only metho...
-
Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
A self-critique, revision, and verification loop makes small vision-language models produce more detailed and more executable robot plans, beating their own baselines and, on the paper's judge-based evaluation, plans ...
Discussion (0). Continue with ORCID to comment.