REVIEW 6 cited by
Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Aligning Large Language Models (LLMs) is crucial for enhancing their safety and utility. However, existing methods, primarily based on preference datasets, face challenges such as noisy labels, high annotation costs, and privacy concerns. In this work, we introduce Alignment from Demonstrations (AfD), a novel approach leveraging high-quality demonstration data to overcome these challenges. We formalize AfD within a sequential decision-making framework, highlighting its unique challenge of missing reward signals. Drawing insights from forward and inverse reinforcement learning, we introduce divergence minimization objectives for AfD. Analytically, we elucidate the mass-covering and mode-seeking behaviors of various approaches, explaining when and why certain methods are superior. Practically, we propose a computationally efficient algorithm that extrapolates over a tailored reward model for AfD. We validate our key insights through experiments on the Harmless and Helpful tasks, demonstrating their strong empirical performance while maintaining simplicity.
Forward citations
Cited by 6 Pith papers
-
rePIRL: Learn PRM with Inverse RL for LLM Reasoning
rePIRL learns token-level process rewards for LLM reasoning via a guided-cost-learning-style IRL objective, and shows these rewards improve reasoning policies on math/coding benchmarks.
-
Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
RLSF uses a model's own answer-span confidence as an intrinsic reward to create preference data, then applies DPO or PPO to improve calibration and reasoning without external labels.
-
Where You Go is Who You Are: Behavioral Theory-Guided LLMs for Inverse Reinforcement Learning
SILIC uses LLM-guided inverse reinforcement learning and Theory of Planned Behavior chain reasoning to infer age, gender, income, and employment from travel trajectories, reportedly beating SVM, XGBoost, CatBoost, and...
-
OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models
The paper advocates protecting and leveraging OpenReview's peer review corpus as a community asset for LLM-based review assistance, benchmarks, and alignment.
-
Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions
LLM robustness research is organized into adversarial robustness, out-of-distribution robustness, and evaluation, with an accompanying GitHub collection of papers.
-
Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.
Discussion (0). Sign in to comment.