REVIEW 9 cited by
Rule Based Rewards for Language Model Safety
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Reinforcement learning based fine-tuning of large language models (LLMs) on human preferences has been shown to enhance both their capabilities and safety behavior. However, in cases related to safety, without precise instructions to human annotators, the data collected may cause the model to become overly cautious, or to respond in an undesirable style, such as being judgmental. Additionally, as model capabilities and usage patterns evolve, there may be a costly need to add or relabel data to modify safety behavior. We propose a novel preference modeling approach that utilizes AI feedback and only requires a small amount of human data. Our method, Rule Based Rewards (RBR), uses a collection of rules for desired or undesired behaviors (e.g. refusals should not be judgmental) along with a LLM grader. In contrast to prior methods using AI feedback, our method uses fine-grained, composable, LLM-graded few-shot prompts as reward directly in RL training, resulting in greater control, accuracy and ease of updating. We show that RBRs are an effective training method, achieving an F1 score of 97.1, compared to a human-feedback baseline of 91.7, resulting in much higher safety-behavior accuracy through better balancing usefulness and safety.
Forward citations
Cited by 9 Pith papers
-
MentalThink: Shaping Thoughts in Mental SVG World
MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.
-
Mach-Mind-4-Flash Technical Report
Post-training alone—parallel domain RL experts, Multi-Teacher On-Policy Distillation, and Hybrid Median-length Policy Optimization—lifts a 3B-activated MoE to roughly 100B-class agent and reasoning scores.
-
Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.
-
Statutory Construction and Interpretation for Artificial Intelligence
Prompt-based legal canons and iterative rule refinement reduce disagreement among LLM judges about whether a response complies with natural-language rules.
-
One Token to Fool LLM-as-a-Judge
LLM reward models falsely accept empty 'master key' responses such as ':' or 'Thought process:' across many models, and a fine-tuning defense reduces these false positives to near zero.
-
Saffron-1: Safety Inference Scaling
A multifurcation reward model that scores all next-token candidates in one call makes inference-time safety scaling with tree search far more compute-efficient than best-of-N sampling.
-
Contrastive Distillation of Emotion Knowledge from LLMs for Zero-Shot Emotion Recognition
A BERT-sized model, trained with contrastive learning on GPT-4-generated emotion descriptors, achieves zero-shot emotion recognition across new label spaces and task types.
-
Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
A generative reward model trained with long chain-of-thought and rule-based RL outperforms standard and vertically scaled reward baselines on RM-Bench and RewardBench.
-
Activation Reward Models for Few-Shot Model Alignment
Mean attention-head activations from a few labeled examples, injected into selected heads, turn a frozen vision-language model into a few-shot reward model that beats prompting and scoring baselines and a new reward-h...
Discussion (0). Continue with ORCID to comment.