Pith. sign in

REVIEW 6 cited by

Scaling Laws for Forgetting When Fine-Tuning Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.05605 v1 pith:TT3DJBMZ submitted 2024-01-11 cs.CL cs.LG

Damjan Kalajdzievski This is my paper · ORCID
classification cs.CLcs.LG
keywords forgettingfine-tuningnumberwhenfine-tunedlanguagelargelaws
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We study and quantify the problem of forgetting when fine-tuning pre-trained large language models (LLMs) on a downstream task. We find that parameter-efficient fine-tuning (PEFT) strategies, such as Low-Rank Adapters (LoRA), still suffer from catastrophic forgetting. In particular, we identify a strong inverse linear relationship between the fine-tuning performance and the amount of forgetting when fine-tuning LLMs with LoRA. We further obtain precise scaling laws that show forgetting increases as a shifted power law in the number of parameters fine-tuned and the number of update steps. We also examine the impact of forgetting on knowledge, reasoning, and the safety guardrails trained into Llama 2 7B chat. Our study suggests that forgetting cannot be avoided through early stopping or by varying the number of parameters fine-tuned. We believe this opens up an important safety-critical direction for future research to evaluate and develop fine-tuning schemes which mitigate forgetting

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning

    cs.LG 2026-07 accept novelty 7.0 of 10

    A parameter-free per-layer critical LoRA strength from the rectangular spiked-deformation transform on the measured spectrum of W predicts intruder onset and forgetting, and a derived spike budget cuts forgetting 62% ...

  2. MemSFT: Mitigating Alignment Tax with an External Parametric Memory

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MemSFT attaches a retriever-imitating 8B memory plus a word-level router to frozen Qwen3 backbones, boosting domain scores by ~36 points while holding general-benchmark averages essentially flat, where full SFT loses ...

  3. PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    PLATE constructs frozen weight-derived bases B and Q and trains only a small core A, reducing catastrophic forgetting during data-free continual fine-tuning.

  4. Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Finetuning forgetting follows a multiplicative scaling law in model size, finetuning tokens, and injected pretraining fraction, with 1% injection nearly eliminating forgetting.

  5. One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context

    cs.LG 2026-06 conditional novelty 5.0 of 10

    Soft-prompt teachers, each a small set of learnable tokens on a frozen backbone, can replace full fine-tuning as on-policy distillation supervisors and compose across tasks into a single student.

  6. Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.

Pith tools