Pith. sign in

REVIEW 9 cited by

Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.05934 v3 pith:EWBGX2JG submitted 2023-12-10 cs.AI cs.CLcs.LG

Oded Ovadia , Menachem Brief , Moshik Mishaeli , Oren Elisha This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords fine-tuninginformationknowledgellmstrainingunsupervisedacrossapproaches
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) encapsulate a vast amount of factual information within their pre-trained weights, as evidenced by their ability to answer diverse questions across different domains. However, this knowledge is inherently limited, relying heavily on the characteristics of the training data. Consequently, using external datasets to incorporate new information or refine the capabilities of LLMs on previously seen information poses a significant challenge. In this study, we compare two common approaches: unsupervised fine-tuning and retrieval-augmented generation (RAG). We evaluate both approaches on a variety of knowledge-intensive tasks across different topics. Our findings reveal that while unsupervised fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge. Moreover, we find that LLMs struggle to learn new factual information through unsupervised fine-tuning, and that exposing them to numerous variations of the same fact during training could alleviate this problem.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 27 citations worldwide. Full citation record

  1. CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learning

    cs.SE 2025-07 conditional novelty 6.0 of 10

    CodeReasoner combines a concise execution-focused dataset, instruction tuning, and GRPO RL to make 7B/14B models match or beat GPT-4o on code reasoning benchmarks.

  2. Maximally-Informative Retrieval for State Space Model Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RICO ranks documents by how much they reduce an SSM's question perplexity, using gradient-document inner products, and matches BM25 while often beating E5 on answer quality without finetuning.

  3. FedRAG: A Framework for Fine-Tuning Retrieval-Augmented Generation Systems

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A new library, FedRAG, provides centralized and federated fine-tuning for RAG systems, with a lightweight experiment showing a RALT-based accuracy gain on MMLU global facts.

  4. ScienceMeter: Tracking Scientific Knowledge Updates in Language Models

    cs.CL 2025-05 reject novelty 6.0 of 10

    ScienceMeter evaluates language model knowledge updates across three axes, preservation of old scientific claims, acquisition of new claims, and projection to future findings, and finds all current methods fall short.

  5. Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA

    cs.CL 2026-07 conditional novelty 5.0 of 10

    For closed-book QA via LoRA adapters, once capacity is sufficient, training-data quality—especially answer brevity—outperforms rank, learning rate, and architecture changes, lifting 15-doc accuracy from 57.7% to 85.7%.

  6. Reasoning with RAGged events: RAG-Enhanced Event Knowledge Base Construction and reasoning with proof-assistants

    cs.CL 2025-06 reject novelty 5.0 of 10

    On a single Thucydides corpus, strong LLMs extract more events without RAG, weak LLMs benefit from simple RAG but collapse under complex RAG, and a Coq translation is proposed to justify RAG-discovered event types.

  7. Reading Between the Timelines: RAG for Answering Diachronic Questions

    cs.CL 2025-07 conditional novelty 4.0 of 10

    TA-RAG uses LLM-extracted time intervals, time-filtered retrieval with averaged temporal query embeddings, and chronologically ordered context to beat standard RAG by 13-27 points on the new ADQAB benchmark of 525 mul...

  8. Automatic Dataset Generation for Knowledge Intensive Question Answering Tasks

    cs.CL 2025-05 reject novelty 3.0 of 10

    Fine-tuning Llama-3-8b and Mistral-7b-v0.3 on LLM-generated QA pairs from IBM Technotes can improve no-context QA scores over training on human-annotated TechQA data, but the evaluation may be inflated by test-documen...

  9. Building Task Bots with Self-learning for Enhanced Adaptability, Extensibility, and Factuality

    cs.CL 2025-08 conditional novelty 2.0 of 10

    A thesis that combines self-learning from dialog logs, schema-guided prompting, and self-aligned factuality to build task bots with minimal human intervention.

Pith tools