REVIEW 7 cited by
Chain of Hindsight Aligns Language Models with Feedback
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Learning from human preferences is important for language models to match human needs and to align with human and social values. Prior works have achieved remarkable successes by learning from human feedback to understand and follow instructions. Nonetheless, these methods are either founded on hand-picked model generations that are favored by human annotators, rendering them inefficient in terms of data utilization and challenging to apply in general, or they depend on reinforcement learning, which often suffers from imperfect reward functions and relies on extremely challenging optimizations. In this work, we propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. Our idea is inspired by how humans learn from extensive feedback presented in the form of languages. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model, allowing us to take advantage of the language comprehension capabilities of language models. We condition the model on a sequence of model generations paired with feedback. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors. Applying our method to large language models, we observed that Chain of Hindsight significantly surpasses previous methods in aligning language models with human preferences. We report significant improvements on summarization and dialogue benchmarks, with our approach markedly preferred in human evaluations.
Forward citations
Cited by 7 Pith papers
-
Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex
Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.
-
LeAct: Learning to Reason from Expert Actions
An AI can learn to reason by sampling explanations for an expert's actions and keeping only the ones that help it predict those actions.
-
From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment
SP-PRM trains process reward models from outcome reward models via score and preference consistency, improving reward-guided search on dialogue, summarization, and reasoning tasks.
-
Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time
SITAlign is an inference-time constrained decoder that maximizes a primary reward while enforcing thresholds on secondary rewards, and it reports better primary-reward win-tie rates than weighted-objective decoding.
-
Large Language Model-Based Agents for Automated Research Reproducibility: An Exploratory Study in Alzheimer's Disease
LLM agents given abstracts, methods, and data dictionary entries approximately reproduced about 53% of key abstract findings across five Alzheimer's disease studies, with frequent mismatches in statistical methods.
-
Invariant-based Robust Weights Watermark for Large Language Models
An invariant-based weights watermark embeds per-user keys into the null space of transformer invariants and uses noise to repel collusion.
-
Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives
A survey proposes a macro-meso-micro value framework for agentic AI alignment and maps applications, methods, and benchmarks onto it.
Discussion (0). Continue with ORCID to comment.