REVIEW 6 cited by
Unfamiliar Finetuning Examples Control How Language Models Hallucinate
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models are known to hallucinate when faced with unfamiliar queries, but the underlying mechanism that govern how models hallucinate are not yet fully understood. In this work, we find that unfamiliar examples in the models' finetuning data -- those that introduce concepts beyond the base model's scope of knowledge -- are crucial in shaping these errors. In particular, we find that an LLM's hallucinated predictions tend to mirror the responses associated with its unfamiliar finetuning examples. This suggests that by modifying how unfamiliar finetuning examples are supervised, we can influence a model's responses to unfamiliar queries (e.g., say ``I don't know''). We empirically validate this observation in a series of controlled experiments involving SFT, RL, and reward model finetuning on TriviaQA and MMLU. Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations. We find that, while hallucinations from the reward model can significantly undermine the effectiveness of RL factuality finetuning, strategically controlling how reward models hallucinate can minimize these negative effects. Leveraging our previous observations on controlling hallucinations, we propose an approach for learning more reliable reward models, and show that they improve the efficacy of RL factuality finetuning in long-form biography and book/movie plot generation tasks.
Forward citations
Cited by 6 Pith papers
-
Learning Facts at Scale with Active Reading
Training LLMs on self-generated, diverse 'active reading' materials improves factual recall by 160-312% and scales to a 1T-token Wikipedia expert model.
-
Exploring the Challenges and Opportunities of AI-assisted Codebase Generation
Developers prompting codebase-level AI assistants are often dissatisfied with generated code, citing missing functionality, poor code quality, and communication gaps, despite varied prompting strategies.
-
Stochastic Chameleons: Irrelevant Context Hallucinations Reveal Class-Based (Mis)Generalization in LLMs
LLMs systematically combine abstract category cues from a query with features from irrelevant context, causing structured answer flips, a behavior the authors call class-based (mis)generalization.
-
GRAIT: Gradient-Driven Refusal-Aware Instruction Tuning for Effective Hallucination Mitigation
GRAIT selects and reweights refusal-training examples using gradient influence, reporting lower hallucination rates and better helpfulness scores than prior refusal-aware tuning baselines.
-
Connections between reinforcement learning with feedback,test-time scaling, and diffusion guidance: An anthology
RLHF, RLIF, and soft best-of-N sampling reduce to the same exponential-tilting objective under parameter matching, and test-time scaling can asymptotically implement classifier-free diffusion guidance.
-
A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models
A survey claims proactive defenses against LLM misinformation outperform post-hoc detection by up to 63%, but no meta-analysis details are provided to support the claim.
Discussion (0). Continue with ORCID to comment.