REVIEW 10 cited by
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The alignment tuning process of large language models (LLMs) typically involves instruction learning through supervised fine-tuning (SFT) and preference tuning via reinforcement learning from human feedback (RLHF). A recent study, LIMA (Zhou et al. 2023), shows that using merely 1K examples for SFT can achieve significant alignment performance as well, suggesting that the effect of alignment tuning might be "superficial." This raises questions about how exactly the alignment tuning transforms a base LLM. We analyze the effect of alignment tuning by examining the token distribution shift between base LLMs and their aligned counterpart. Our findings reveal that base LLMs and their alignment-tuned versions perform nearly identically in decoding on the majority of token positions. Most distribution shifts occur with stylistic tokens. These direct evidence strongly supports the Superficial Alignment Hypothesis suggested by LIMA. Based on these findings, we rethink the alignment of LLMs by posing the research question: how effectively can we align base LLMs without SFT or RLHF? To address this, we introduce a simple, tuning-free alignment method, URIAL. URIAL achieves effective alignment purely through in-context learning (ICL) with base LLMs, requiring as few as three constant stylistic examples and a system prompt. We conduct a fine-grained and interpretable evaluation on a diverse set of examples, named JUST-EVAL-INSTRUCT. Results demonstrate that base LLMs with URIAL can match or even surpass the performance of LLMs aligned with SFT or SFT+RLHF. We show that the gap between tuning-free and tuning-based alignment methods can be significantly reduced through strategic prompting and ICL. Our findings on the superficial nature of alignment tuning and results with URIAL suggest that deeper analysis and theoretical understanding of alignment is crucial to future LLM research.
Forward citations
Cited by 10 Pith papers
-
Training Skills Like Parameters via Self-Supervised Semantic Diffusion
Screenwriting skills can be learned from human scripts alone by compressing and reconstructing them, storing the lessons as text cards that improve generation without changing model weights.
-
Learning to Diagnose and Correct Errors: Towards Moral Sensitivity Acquisition in Large Language Models
A pragmatic-inference training method, learning2DiagCorr, lets LLMs diagnose and correct moral errors and generalizes across toxic language, social bias, and jailbreak tasks.
-
GeLaCo: An Evolutionary Approach to Layer Compression
Evolutionary search over layer-merging configurations, scored by module-wise activation similarity, yields competitive LLM compression and the first size-quality Pareto fronts.
-
LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning
Low-rank extrapolation of an aligned model's weight update (LoX) reduces how much later fine-tuning erodes safety refusal behavior.
-
What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding
FoReaL-Decoding lets a strong reasoning model generate the first few tokens of each sentence and a weaker model complete the sentence, cutting theoretical FLOPs by 30-55% while retaining 86-100% of accuracy on four ma...
-
Normalized Rewards for Preference Optimization
A regularization term that conserves the combined length-normalized probability of chosen and rejected responses reduces likelihood displacement in DPO/SimPO, improves AlpacaEval and benchmark outcomes, and acts prima...
-
We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems
MCP-powered LLM agents are vulnerable to prompt injection from third-party services, and simple detection or filtering defenses do not reliably stop these attacks.
-
SoK: Machine Unlearning for Large Language Models
A new taxonomy for LLM unlearning distinguishes removal-intended from suppression-intended methods, and argues that gradient ascent methods functionally behave like suppression.
-
Beyond the Surface: Measuring Self-Preference in LLM Judgments
The DBG metric measures LLM self-preference bias as the gap between a judge model's own win rate and the win rate assigned by an ensemble of gold judges.
-
Optimising Language Models for Downstream Tasks: A Post-Training Perspective
A dissertation that repackages the author's previously published papers on continued pre-training, prompt tuning, and instruction modelling into a single narrative.
Discussion (0). Sign in to comment.