Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection

T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A scaling law predicts how much finetuning makes language models forget, and 1% pretraining data in the mix prevents most of the damage.

desk verdict Solid scaling-law study with a genuinely new forgetting law and a real extrapolation check, but the '1% injection prevents forgetting' headline only holds with an effectively infinite pretraining reservoir, which the paper's own Section 4.4 demonstrates. read the letter →

arxiv 2502.06042 v2 pith:LJPVDZY5 submitted 2025-02-09 cs.LG cs.CL

Louis Bethune , David Grangier , Dan Busbridge , Eleonora Gualdoni , Marco Cuturi , Pierre Ablin This is my paper · ORCID
classification cs.LGcs.CL
keywords scalinglawscatastrophicforgettingfinetuningpretrainingdatainjectionlanguagemodelsnext-tokenpredictionoverfittingextrapolation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to turn two failure modes of finetuning — overfitting to a small target dataset and forgetting the pretraining distribution — into quantities that can be predicted before a run. The authors propose a scaling law for the pretraining loss after finetuning, measured at the best finetuning-validation checkpoint, as a function of model size, number of finetuning tokens, and the fraction of pretraining data mixed into finetuning. Across twelve target domains the law predicts the actual pretraining loss with a mean relative error of 0.40%, and it extrapolates from small models and small datasets to larger ones within about 1–2% error. A second, simpler law predicts the finetuning validation loss, which is essentially independent of the injection fraction. The headline practical result is that injecting just 1% pretraining data suppresses most forgetting with no measurable cost to finetuning loss.

What carries the argument

The carrying object is the modified multiplicative scaling law for the pretraining loss (Equation 7 of the paper), $L_{\mathrm{pt}} = L^0_{\mathrm{pt}} + A\, D_{\mathrm{ft}}^{\beta} / ((1+Bp)N)^{\alpha}$, with the term $(1+Bp)N$ as the mechanism that prices in pretraining data injection: a fraction $p$ of parameters is "owned" by the pretraining task and is $B$ times more efficient at preserving pretraining loss than parameters devoted to finetuning. The law is evaluated at the checkpoint that minimizes finetuning validation loss, i.e. the bottom of the overfitting U-curve, and $L^0_{\mathrm{pt}}$ is measured on the rewarmed model — the pretrained model at the constant learning rate used for finetuning — because the rewarming itself increases the pretraining loss. Coefficients are fitted per domain by Huber loss in log space, following the standard scaling-law fitting procedure.

What would settle it

Pretrain a 334M-parameter model on both 3.3B and 33B tokens, finetune both on the same target domain with $p=1\%$, and check whether the same $\alpha$ in Equation 7 predicts both pretraining losses; a systematic mismatch would falsify the claim that $\alpha$ captures scale alone.

Watch

Extended reading notes

Core claim

The central discovery is that forgetting during finetuning is not erratic but obeys the multiplicative law $L_{\mathrm{pt}} = L^0_{\mathrm{pt}} + A\, D_{\mathrm{ft}}^{\beta} / ((1+Bp)N)^{\alpha}$, where $L^0_{\mathrm{pt}}$ is the pretraining loss of the pretrained model before finetuning, $D_{\mathrm{ft}}$ is the number of finetuning tokens, $N$ is model size, $p$ is the fraction of pretraining data injected, and $A$, $B$, $\alpha$, $\beta$ are domain-dependent positive constants. The factor $(1+Bp)N$ encodes the idea that a fraction $p$ of the model's parameters is allocated to the pretraining task, and these parameters are $B$ times more efficient at retaining pretraining performance than those allocated to finetuning; $B\gg 1$ because pretrained features are already aligned with the pretraining loss. The paper reports a bootstrapped mean relative error of 0.40% across twelve domains of The Pile, and shows the law still holds when pretraining checkpoints are taken on the $D=10N$ isocurve and when instruction finetuning is used. For the finetuning loss itself, the paper confirms the multiplicative law $L_{\mathrm{ft}} = A/(N^{\alpha} D_{\mathrm{ft}}^{\beta}) + E$ with 0.89% error and essentially no dependence on $p$, and shows that injecting 1% pretraining data does not hurt finetuning performance while strongly reducing forgetting.

Load-bearing premise

The model-size exponent in the forgetting law is estimated from models that were always pretrained on 100 tokens per parameter, so the law does not separate model scale from pretraining data volume.

Editorial extensions

If this is right

  • Forgetting before a finetuning run can be predicted from three inputs — model size, target token count, and injection fraction — with no need to train the model first, once the domain coefficients are fitted on small runs.
  • The injection fraction $p$ can be chosen from the law: $p=1\%$ already shields pretraining loss, and larger $p$ improves finetuning validation loss only mildly before a U-curve makes it worse.
  • Small-scale experiments (models up to 334M parameters, datasets up to 3,000K tokens) predict the behavior of models up to 1.3B parameters and 30,000K tokens within about 2% on finetuning loss and 1% on forgetting.
  • Forgetting is primarily a capacity phenomenon: small models lose up to 95% of pretraining progress while large models lose about 20%, so bigger models are more robust to sequential finetuning at higher compute cost.
  • The finetuning loss obeys a multiplicative law with no measurable dependence on $p$, meaning pretraining data injection is a nearly free regularizer for the target domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the fitted coefficient $B$ can serve as a cheap, quantitative measure of a target domain's distance from the pretraining distribution: domains far from the pretraining mix (Dm mathematics, $B \approx 10^4$) need injection most, while similar domains (Wikipedia) need it least — a relationship the paper observes but does not formalize.
  • Because the pretraining-token budget is tied to model size ($D=100N$), the law's $\alpha$ conflates scale with pretraining data volume; I expect the exponents would need re-fitting for models trained at compute-optimal or other token budgets.
  • The paper measures forgetting through pretraining loss; a natural testable extension is to check whether the same law predicts accuracy drops on held-out benchmarks, grounding the loss-based proxy in task-level forgetting.
  • The observed rule that roughly 0.3 unique pretraining tokens per finetuning token suffices to avoid forgetting suggests a diversity threshold, which could be probed directly by ablating the size of the injected pretraining pool.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies catastrophic forgetting during full-parameter finetuning of GPT-style language models and proposes two per-domain scaling laws: one for the finetuning validation loss (Eq. 6, following Zhang et al.) and one for the pretraining loss after finetuning as a measure of forgetting (Eq. 7). The forgetting law predicts L_pt from the pretrained baseline L0_pt, model size N, finetuning tokens D_ft, and injection fraction p, with a multiplicative factor (1+Bp)N. The authors fit these laws on 125 configurations per domain (5 model sizes × 5 finetuning dataset sizes × 5 injection fractions) across 12 domains of The Pile, report mean relative errors of 0.89% (finetuning) and 0.40% (forgetting), and validate extrapolation to larger models and larger finetuning datasets. The headline practical claim is that injecting p=1% pretraining data into the finetuning mixture mitigates forgetting with negligible cost to finetuning loss.

Significance. The work is a careful, large-scale empirical study in a practically important setting. Its strengths include the out-of-sample extrapolation checks in Table 4, the use of multiple domains, the comparison to alternative law forms, and the explicit measurement of the rewarming offset L0_pt. If the forgetting law survives the caveats below, it would give practitioners a cheap way to predict forgetting from small-scale runs. However, the law's scope is narrower than the abstract suggests: it is fitted only on the D=100N pretraining isocurve and only for streaming (infinite-reservoir) pretraining data injection.

major comments (2)
  1. [Section 4.4, Eq. (7)] The forgetting law in Eq. (7) does not include the number of unique pretraining tokens available for injection. The authors' own experiment in §4.4 (Figure 6) shows that at fixed p=1%, the post-finetuning pretraining loss depends strongly on the reservoir size: with only 32KT-320KT unique pretraining tokens, repeated injection of the same tokens overfits the pretraining set and forgetting is not prevented. Since Eq. (7) contains no reservoir term, it is only valid in the infinite-reservoir streaming regime, while the abstract states that 'injecting as little as 1% of pretraining data... prevents the model from forgetting the pretraining set' without this caveat. This is a load-bearing limitation of the central claim; the law cannot predict the finite-reservoir failure mode shown in the paper itself. The authors should either extend the law (e.g., with a repetition or reservoir-size factor) or explicitly limit the scope of the claim and the law to streaming injection, and revise the abstract accordingly.
  2. [Section 4.5, Eq. (7) and Table 2] All pretrained checkpoints lie on the isocurve D=100N (Section 3.1, Table 1), so the fitted exponent α in Eq. (7) conflates model size N with the number of pretraining tokens D_pretrain. The statement in the Analysis that 'forgetting is primarily attributed to network capacity' is therefore not identified: larger models also received more pretraining data, and the D=10N check in Figure 11 does not separate the two covariates because it only changes the isocurve ratio, not N and D_pretrain independently. The authors should rephrase the capacity interpretation, or add experiments that vary N and D_pretrain separately, or at least prominently flag the confound as a limitation.
minor comments (5)
  1. [Section 3.1 / Figure 15] The injection fractions are listed as {0%, 0.1%, 0.5%, 1%, 5%}, but Figure 15 shows 0.2% in the legend; please reconcile this discrepancy.
  2. [Section 3.1 / Table 1] The text states pretraining uses 100 tokens per parameter, but the D/N ratios in Table 1 range from about 79 to 124; please clarify whether the isocurve is approximate.
  3. [Section 4.5 / Figure 14] The sentence 'they lose up to 95% (!) of the pretraining progress' appears to conflict with Figure 14's caption ('up to 80%'); please clarify which number is correct or specify the domain and condition.
  4. [Figure 6 caption] The sentence 'Remarkably, only 0.3 unique pretraining tokens per unique finetuning token are sufficient to avoid forgetting' is difficult to reconcile with the main text's finding that limited unique tokens increase pretraining loss; please rephrase to state the threshold condition and its dependence on D_ft.
  5. [Abstract and Conclusion] The phrase 'prevents the model from forgetting' is too absolute given the finite-reservoir results in §4.4; I suggest using 'mitigates' or adding a qualified statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the forgetting law is an empirical fit validated by held-out extrapolation, and the residual concerns are scope limitations rather than reductions to inputs.

full rationale

This paper does not claim a from-first-principles derivation; it proposes Eq. (7) as an empirical scaling law and fits A, B, alpha, and beta to 125 measured points per domain (Sec. 4.5, Appendix C). The only measured offset, L0_pt, is the pretraining loss of the rewarmed checkpoint before finetuning, which is a legitimate input rather than a fitted version of the target, and the paper explicitly tests the law without L0_pt and reports a worse MRE (1.05% vs. 0.40%), showing the offset does not force the fit. The reported 0.40% MRE is not a held-out test, but Table 4 provides genuine extrapolation: fitting only up to Medium/3,000K tokens predicts Large/XL at 9,000K/30,000K with 0.83% forgetting MRE, and Appendix A confirms the law on a separate D = 10N pretraining isocurve. These checks show the 'predictions' are not statistical identities with the fitted data. No uniqueness theorem or ansatz is imported via self-citation; the prior-work citations (e.g., Zhang et al. 2024) supply the baseline multiplicative finetuning law, while the new forgetting-law terms are fitted rather than forced by those citations. The skeptical concern that Eq. (7) omits the size of the unique pretraining reservoir is a scope and correctness limitation, not circularity: the paper itself states in Sec. 4.4 that its main runs stream pretraining data 'without repetition,' and Figure 6 shows the 1% rule degrades when the reservoir is small, qualifying the abstract's unconditional 1% claim. Similarly, the D = 100N isocurve limitation in Sec. 4.5 constrains extrapolation to other pretraining token budgets but does not make the fit circular.

Assumptions & free parameters 10 free parameters · 5 assumptions · 0 invented entities

The main quantities the paper contributes are fitted constants, not derived ones. The law is an empirical summary of 1500 runs. Because the pretraining budget is tied to N via D=100N, the fitted exponents are not identifiable as pure model-size effects. No new entities are introduced.

free parameters (10)
  • Finetuning law coefficient A (per domain) = e.g., Arxiv 95.18
    Multiplicative prefactor in Eq (6), fitted on 125 points per domain.
  • Finetuning law exponent alpha (per domain) = Arxiv 0.17
    Controls model-size scaling in Eq (6); fitted per domain.
  • Finetuning law exponent beta (per domain) = Arxiv 0.10
    Controls finetuning-token scaling in Eq (6); fitted per domain.
  • Finetuning irreducible loss E (per domain) = e.g., Enron 0.00
    Offset in Eq (6); fitted per domain.
  • Forgetting law coefficient A (per domain) = e.g., Arxiv 526
    Multiplicative prefactor in Eq (7); fitted per domain.
  • Forgetting law coefficient B (per domain) = e.g., Arxiv 392
    Efficiency gain of injected pretraining parameters in Eq (7); fitted per domain.
  • Forgetting law exponent alpha (per domain) = Arxiv 0.74
    Controls effective model-size scaling in Eq (7); fitted per domain.
  • Forgetting law exponent beta (per domain) = Arxiv 0.34
    Controls finetuning-token scaling in Eq (7); fitted per domain.
  • Finetuning learning rate multiplier = 1/30 of peak pretraining LR
    Constant LR chosen from ablation in Figure 5; all finetuning runs use it and it changes the rewarming offset L0_pt.
  • Pretraining token isocurve ratio = D_pretrain = 100N
    Chosen to align with common practice; restricts the (N, D) grid and conflates N with pretraining tokens in the fitted exponents.
assumptions (5)
  • domain assumption Next-token prediction loss on the pretraining set is a valid proxy for general knowledge and forgetting.
    Section 2.1 defines forgetting as increase in pretraining loss; if this proxy fails, the measured 'forgetting' does not reflect capability loss.
  • ad hoc to paper The multiplicative functional form of the forgetting law holds across domains and scales.
    Eq (7) is assumed rather than derived; the paper compares it to additive alternatives and reports lower MRE, but no mechanistic justification.
  • domain assumption Model checkpoints at the bottom of the finetuning U-curve are the right points to compare.
    Section 4 states all losses are reported at the lowest finetuning validation loss; this early-stopping rule determines every fitted value and is not itself predicted.
  • domain assumption Results on GPT-2 style models pretrained on RedPajamaV2 generalize to other architectures and pretraining corpora.
    Section 3.1 uses a single architecture family and one pretraining corpus; the paper claims broad applicability.
  • ad hoc to paper Variation of p in {0, 0.1, 0.5, 1, 5}% is fine enough to establish the 1% rule.
    The grid does not include values between 1% and 5%, so the 'as little as 1%' statement is only an upper bound on the threshold, not a precise estimate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection." pith.science (2026). https://pith.science/paper/LJPVDZY5

@misc{pith2026250206042,
  author       = {Pith},
  title        = {Pith review of: Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LJPVDZY5}},
  note         = {Machine review of arXiv:2502.06042}
}
read the original abstract

A widespread strategy to obtain a language model that performs well on a target domain is to finetune a pretrained model to perform unsupervised next-token prediction on data from that target domain. Finetuning presents two challenges: (i) if the amount of target data is limited, as in most practical applications, the model will quickly overfit, and (ii) the model will drift away from the original model, forgetting the pretraining data and the generic knowledge that comes with it. We aim to derive scaling laws that quantify these two phenomena for various target domains, amounts of available target data, and model scales. We measure the efficiency of injecting pretraining data into the finetuning data mixture to avoid forgetting and mitigate overfitting. A key practical takeaway from our study is that injecting as little as 1% of pretraining data in the finetuning data mixture prevents the model from forgetting the pretraining set.

Figures

Figures reproduced from arXiv: 2502.06042 by the authors.

Figure 1
Figure 1. As little as p = 1% of pretraining data injection shields the model from forgetting on the pretrain dataset. The finetuning validation follows a conventional U-curve. In this paper, we always consider the models obtained at the bottom of the U-curve, that is, models with the best validation loss on the finetuning set, indicated here by a black dot. Github dataset with small model. The minimum validation loss is bare… view at source ↗
Figure 2
Figure 2. Generalization-memorization tradeoff. Arxiv domain. Each point corresponds to the bottom of the U-curve for a model trained on datasets of sizes 300K, 900K, 3,000K, 9,000K and 30,000K tokens with mixture parameter p = 1%. Forgetting is more severe when the model is small and when the finetuning dataset is big. As shown in Equation 6, this can be attributed to the lack of capacity of the model. More parameters are as… view at source ↗
Figure 3
Figure 3. Losses as a function of the fraction of injected pretraining data p on Enron emails with 900K finetuning tokens. Data mixing improves generalization when finetuning data is scarce. The diversity of the pretraining dataset biases learning toward features that exhibit higher generalization. The optimal value of p depends on the domain, the dataset size, and the model size. The finetuning loss as a function of p also f… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Finetuning learning rate as fraction of peak pretrain￾ing learning rate. Ablation on Arxiv with XL Model (1.3B). The model’s learning rate (LR) is reduced by a factor of 100 during pretraining using cosine scheduling. For finetuning, we employ a constant LR, defined as…
Figure 6
Figure 6. Figure 6: Influence of the number of available pretraining to￾kens on the pretraining loss after fine-tuning with p = 1% of pretraining data injected to the mix. We use the Arxiv domain and a model of size “tiny”. With little finetuning data, finetuning is short and hence we can…
Figure 7
Figure 7. Figure 7: Scaling laws for finetuning loss. Wikipedia domain. We extend the multiplicative laws of Zhang et al. (2024) to take into account the fraction of injected pretraining data. At first order, this scaling law is independent of the mixture p. Left Agreement between the obs…
Figure 8
Figure 8. Figure 8: Scaling laws for forgetting. Github domain. We propose to model the increase in pretraining loss as a multiplicative scaling law (Equation 1), that takes into account model size, number of finetuning tokens available, and fraction of pretraining data injected in the da…
Figure 9
Figure 9. Figure 9: Comparison of Adam, AdamW, and Anchored AdamW on the objective function f(x, y) = (x − 2)2 + (y + 3)2 with the starting point θ0 = (4, 4). 100 steps, learning rate 0.1. Tiny (41M) Small (109M) Medium (334M) Large (665M) 1.4 1.6 1.8 2.0 2.2 2.4 Github validation loss Ti…
Figure 11
Figure 11. Figure 11: Scaling laws for finetuning and forgetting in models pre-trained at the D = 10N isocurve. Freelaw domain. The law remains robust to the choice of checkpoint, with a bootstrapped MRE of 0.57% for forgetting and 1.14% for finetuning. Setup Predict on Finetuning MRE Forg…
Figure 12
Figure 12. Figure 12: Scaling laws for finetuning and forgetting with Instruction Finetuning (IFT) on OpenHermes dataset. The dataset is much smaller than the domain of The Pile considered here, so different values of unique tokens are considered. Finally, this dataset is very diverse and …
Figure 13
Figure 13. Figure 13: Downstream tasks performance for pretrained checkpoint before finetuning, after finetuning, and with data injection. All checkpoints have been fine-tuned on dm-mathematics. Forgetting penalizes the model on ARC Easy, but finetuning improves (marginally) the model on M…
Figure 14
Figure 14. Figure 14: Cost of forgetting. Points are reported at the bottom of the U-curve for the Arxiv domain. Same setup as [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: As little as p = 1% of pretraining data shields the model from forgetting on the pretrain dataset. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Losses at the bottom of the U-curve for 6 domains of The Pile with 5 models and 5 dataset sizes, for all values of data mixture γ. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Ablation. Weight decay during pretraining improves checkpoint quality. Note that default implementations of Pytorch and Optax do not decouple the weight decay from the learning rate (per this Github issue: https://github.com/google-deepmind/ optax/issues/292), unlike …
Figure 18
Figure 18. Figure 18: Losses as a function of model size, dataset size, and data-mixing. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Example of finetuning scaling laws for several domains. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: Example of forgetting scaling laws for several domains. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Predict-then-Correct Loop Based on Few-Shot Continuous Contextual Bandit for Demand Forecasting

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A contextual-bandit correction layer with few-shot masked updates improves ML demand forecasts by 3.7–14.9% and cuts inventory costs in two retail datasets.

Reference graph

Works this paper leans on

48 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Scaling laws for generative mixed-modal language models

    Aghajanyan, A., Yu, L., Conneau, A., Hsu, W.-N., Hambardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., and Zettlemoyer, L. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning, pp.\ 265--279. PMLR, 2023

    work page 2023

  2. [2]

    Physics in Next-token Prediction

    An, H., Song, Y., and Li, X. Physics in next-token prediction, 2024. URL https://arxiv.org/abs/2411.00660

    work page Pith review arXiv 2024

  3. [3]

    An empirical study of scaling laws for transfer

    Barnett, M. An empirical study of scaling laws for transfer. arXiv preprint arXiv:2408.16947, 2024

    arXiv 2024

  4. [4]

    Chinchilla scaling: A replication attempt

    Besiroglu, T., Erdil, E., Barnett, M., and You, J. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102, 2024

    arXiv 2024

  5. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  6. [6]

    Deepseek llm: Scaling open-source language models with longtermism

    DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. URL https://github.com/deepseek-ai/DeepSeek-LLM

    arXiv 2024

  7. [7]

    The pile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    arXiv 2020

  8. [8]

    B., Werra, L

    H \"a gele, A., Bakouch, E., Kosson, A., allal, L. B., Werra, L. V., and Jaggi, M. Scaling laws and compute-optimal training beyond fixed training durations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=Y13gSfTjGr

    work page 2024

Show all 48 references
  1. [9]

    The elements of statistical learning: data mining, inference, and prediction, 2017

    Hastie, T., Tibshirani, R., and Friedman, J. The elements of statistical learning: data mining, inference, and prediction, 2017

    2017

  2. [10]

    Towards a unified view of parameter-efficient transfer learning, 2022

    He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., and Neubig, G. Towards a unified view of parameter-efficient transfer learning, 2022. URL https://arxiv.org/abs/2110.04366

    2022 arXiv

  3. [11]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021

    2021

  4. [12]

    Scaling laws for transfer, 2021

    Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer, 2021. URL https://arxiv.org/abs/2102.01293

    2021 arXiv

  5. [13]

    Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017

    2017 arXiv

  6. [14]

    Disentangling and mitigating the impact of task similarity for continual learning, 2024

    Hiratani, N. Disentangling and mitigating the impact of task similarity for continual learning, 2024. URL https://arxiv.org/abs/2405.20236

    2024 arXiv

  7. [15]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vin...

    2022 doi arXiv

  8. [16]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Ra...

    2022

  9. [17]

    Parameter-efficient transfer learning for nlp, 2019

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp, 2019. URL https://arxiv.org/abs/1902.00751

    2019 arXiv

  10. [18]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  11. [19]

    L., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., dahai li, Liu, Z., and Sun, M

    Hu, S., Tu, Y., Han, X., Cui, G., He, C., Zhao, W., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhang, X., Thai, Z. L., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., dahai li, Liu, Z., and Sun, M. Mini CPM : Unveiling the potential of sma...

    2024

  12. [20]

    L., Anthony, Q., Lesort, T., Belilovsky, E., and Rish, I

    Ibrahim, A., Thérien, B., Gupta, K., Richter, M. L., Anthony, Q., Lesort, T., Belilovsky, E., and Rish, I. Simple and scalable strategies to continually pre-train large language models, 2024. URL https://arxiv.org/abs/2403.08763

    2024 arXiv

  13. [21]

    Scaling laws for downstream task performance of large language models

    Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vassilvitskii, S., and Koyejo, S. Scaling laws for downstream task performance of large language models. arXiv preprint arXiv:2402.04177, 2024

    2024

  14. [22]

    Scaling laws for forgetting when fine-tuning large language models, 2024

    Kalajdzievski, D. Scaling laws for forgetting when fine-tuning large language models, 2024. URL https://arxiv.org/abs/2401.05605

    2024 arXiv

  15. [23]

    A., Sun, Y., Jahagirdar, H., Zhang, Y., Du, R., Sahu, A

    Kang, F., Just, H. A., Sun, Y., Jahagirdar, H., Zhang, Y., Du, R., Sahu, A. K., and Jia, R. Get more for less: Principled data selection for warming up fine-tuning in llms. arXiv preprint arXiv:2405.02774, 2024

    2024 arXiv

  16. [24]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  17. [25]

    Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

    Kudo, T. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018

    2018 arXiv

  18. [26]

    Improved fine-tuning by better leveraging pre-training data

    Liu, Z., Xu, Y., Xu, Y., Qian, Q., Li, H., Ji, X., Chan, A., and Jin, R. Improved fine-tuning by better leveraging pre-training data. Advances in Neural Information Processing Systems, 35: 0 32568--32581, 2022

    2022

  19. [27]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017

    2017

  20. [28]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  21. [29]

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning

    Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023

    2023 arXiv

  22. [30]

    Llms on the line: Data determines loss-to-loss scaling laws

    Mayilvahanan, P., Wiedemer, T., Mallick, S., Bethge, M., and Brendel, W. Llms on the line: Data determines loss-to-loss scaling laws. arXiv preprint arXiv:2502.12120, 2025

    2025 arXiv

  23. [31]

    Metaicl: Learning to learn in context

    Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.\ 2791--2809, 2022

    2022

  24. [32]

    Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36: 0 50358--50376, 2023

    2023

  25. [33]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  26. [34]

    Resolving discrepancies in compute-optimal scaling of language models, 2024

    Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Carmon, Y. Resolving discrepancies in compute-optimal scaling of language models, 2024. URL https://arxiv.org/abs/2406.19146

    2024 arXiv

  27. [35]

    Rabe, M. N. and Staats, C. Self-attention does not need o (n2) memory. arXiv preprint arXiv:2112.05682, 2021

    2021 arXiv

  28. [36]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. 2019

    2019

  29. [37]

    Multitask prompted training enables zero-shot task generalization

    Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022

    2022

  30. [38]

    Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2024

    Sardana, N., Portes, J., Doubov, S., and Frankle, J. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2024. URL https://arxiv.org/abs/2401.00448

    2024 arXiv

  31. [39]

    Sequence to sequence learning with neural networks

    Sutskever, I. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215, 2014

    2014 arXiv

  32. [40]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL https://huggingface.co/datasets/teknium/openhermes

    2023

  33. [41]

    Scaling law with learning rate annealing

    Tissue, H., Wang, V., and Wang, L. Scaling law with learning rate annealing. arXiv preprint arXiv:2408.11029, 2024

    2024 arXiv

  34. [42]

    When precision meets position: Bfloat16 breaks down rope in long-context training

    Wang, H., Liu, Q., Du, C., Zhu, T., Du, C., Kawaguchi, K., and Pang, T. When precision meets position: Bfloat16 breaks down rope in long-context training. arXiv preprint arXiv:2411.13476, 2024

    2024 arXiv

  35. [43]

    Weber, M., Fu, D. Y., Anthony, Q., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkun, B., Chalamala, R., Chen, K., Ryabinin, M., Dao, T., Liang, P., Ré, C., Rish, I., and Zhang, C. Redpajama: an open dataset for training large language ...

    2024

  36. [44]

    W., Lester, B., Du, N., Dai, A

    Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022

    2022

  37. [45]

    What makes a high-quality training dataset for large language models: A practitioners' perspective

    Yu, X., Zhang, Z., Niu, F., Hu, X., Xia, X., and Grundy, J. What makes a high-quality training dataset for large language models: A practitioners' perspective. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pp.\ 656--668, 2024

    2024

  38. [46]

    When scaling meets LLM finetuning: The effect of data, model and finetuning method

    Zhang, B., Liu, Z., Cherry, C., and Firat, O. When scaling meets LLM finetuning: The effect of data, model and finetuning method. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=5HCnKDeTws

    2024

  39. [47]

    S., Gabrielsson, R

    Zhu, J., Greenewald, K., Nadjahi, K., de Ocáriz Borde, H. S., Gabrielsson, R. B., Choshen, L., Ghassemi, M., Yurochkin, M., and Solomon, J. Asymmetry in low-rank adapters of foundation models, 2024. URL https://arxiv.org/abs/2402.16842

    2024 arXiv

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.