REVIEW 19 cited by
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training. It is often conjectured that this can teach the model the behavior of hallucinating factually incorrect responses, as the model is trained to generate facts that are not grounded in its pre-existing knowledge. In this work, we study the impact of such exposure to new knowledge on the capability of the fine-tuned model to utilize its pre-existing knowledge. To this end, we design a controlled setup, focused on closed-book QA, where we vary the proportion of the fine-tuning examples that introduce new knowledge. We demonstrate that large language models struggle to acquire new factual knowledge through fine-tuning, as fine-tuning examples that introduce new knowledge are learned significantly slower than those consistent with the model's knowledge. However, we also find that as the examples with new knowledge are eventually learned, they linearly increase the model's tendency to hallucinate. Taken together, our results highlight the risk in introducing new factual knowledge through fine-tuning, and support the view that large language models mostly acquire factual knowledge through pre-training, whereas fine-tuning teaches them to use it more efficiently.
Forward citations
Cited by 19 Pith papers
-
Quantifying Cross-Modality Memorization in Vision-Language Models
Fine-tuning VLMs on image-only or text-only personas yields a significant, asymmetric cross-modal memorization gap that persists with model scale, unlearning, and multi-hop reasoning.
-
Reliability Scales Inversely: Hallucinations Snowball Faster in Bigger Language Models
Hallucination in LLMs is driven by an oracle-invisible “decoding risk” term that grows with scale and causally compounds errors within a response.
-
Bridging Vision and Language: Optimal Transport-Driven Radiology Report Generation via LLMs
Optimal Transport alignment between image patches and disease labels, paired with LLM fine-tuning, improves clinical efficacy of generated radiology reports.
-
AggTruth: Contextual Hallucination Detection using Aggregated Attention Scores in LLMs
AggTruth uses four ways of aggregating attention scores over the retrieved passage to train a logistic regression detector, achieving stable cross-task AUROC across four LLMs.
-
When to Trust Context: Self-Reflective Debates for Context Reliability
SR-DCR uses an asymmetric debate plus self-confidence to gate whether a model follows context or its prior, improving ClashEval accuracy on several models.
-
Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?
Fine-tuning on data aligned with an LLM's prior knowledge induces overconfidence, and CogCalib mitigates this by gating a calibration loss to known data.
-
The Hallucination Tax of Reinforcement Finetuning
Standard RFT sharply reduces LLM refusal on unanswerable questions, and adding 10% synthetic unanswerable math during RFT restores refusal with small accuracy losses.
-
Refine Knowledge of Large Language Models via Adaptive Contrastive Learning
An adaptive contrastive learning strategy that uses a model's own sampled response accuracy to create per-region positive and negative training pairs improves LLM truthful rate by up to 6.9% over IDK-SFT.
-
OntoTune: Ontology-Driven Self-training for Aligning Large Language Models
A self-training method that uses an existing medical ontology to select and learn from the model's own inconsistent answers improves medical QA and taxonomy tasks while preserving general ability.
-
Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA
For closed-book QA via LoRA adapters, once capacity is sufficient, training-data quality—especially answer brevity—outperforms rank, learning rate, and architecture changes, lifting 15-doc accuracy from 57.7% to 85.7%.
-
REFLEX: Self-Refining Explainable Fact-Checking via Verdict-Anchored Style Control
REFLEX improves explainable fact-checking by using verdict-anchored style control and self-disagreement signals to disentangle fact from style in LLM outputs, achieving SOTA results with minimal self-refined samples.
-
Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs
Reasoning LLMs seem better at retrieving hierarchical facts not because they know more but because they navigate better; the key supporting RL experiment is missing from the paper.
-
Reliable Reasoning Path: Distilling Effective Guidance for LLM Reasoning with Knowledge Graphs
RRP generates semantic and structural reasoning paths, reranks them with a rethinking module, and reports SOTA Hits@1 of 90.0 on WebQSP and 64.5 on CWQ.
-
ClimateChat: Designing Data and Methods for Instruction Tuning LLMs to Answer Climate Change Queries
An automated pipeline creates climate instruction data, and fine-tuning a geoscience LLM on it improves climate question-answering accuracy over general instruction tuning.
-
SLearnLLM: A Self-Learning Framework for Efficient Domain-Specific Adaptation of Large Language Models
Fine-tuning only on questions a model answers incorrectly reaches nearly the same performance as full-data fine-tuning while substantially reducing training time.
-
Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report
Foundation-Sec-8B-Instruct, an instruction-tuned 8B cybersecurity LLM, is released and claimed to beat Llama 3.1-8B-Instruct on CTIBench-RCM and CTIBench-MCQA while remaining competitive on general instruction-following.
-
Reconstructing Biological Pathways by Applying Selective Incremental Learning to (Very) Small Language Models
A small BERT model trained on only 117 of 517 curated regulatory relationships selected as confident errors reaches 93% balanced accuracy, outperforming a policy that also includes uncertain correct examples.
-
Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models
A survey of LLM hallucination research that formalizes hallucination types and argues, via incompleteness and undecidability arguments, that hallucinations cannot be fully eliminated.
-
Building Task Bots with Self-learning for Enhanced Adaptability, Extensibility, and Factuality
A thesis that combines self-learning from dialog logs, schema-guided prompting, and self-aligned factuality to build task bots with minimal human intervention.
Discussion (0). Continue with ORCID to comment.