Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Catastrophic forgetting can be reduced to 0.25% by pruning task-vector parameters whose change is large relative to pre-trained weights.

desk verdict Useful empirical trick, but the paper's central mechanistic claim is undone by an algebraic identity the authors missed: Eq. (1) reduces to magnitude pruning reweighted by pretrained magnitudes, so the 'relative change' story is confounded and needs a control experiment. read the letter →

arxiv 2509.08255 v1 pith:YPZ6ZMJH submitted 2025-09-10 cs.LG

Wei Huang , Anda Cheng , Yinggui Wang This is my paper
classification cs.LG
keywords catastrophicforgettinglargelanguagemodelstaskvectorweightpruningfine-tuningrelativechangemagnitudeforgetting-aware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that catastrophic forgetting during full fine-tuning of large language models is governed by the relative size of the task-vector change compared with the pre-trained weight, not just the absolute change. On that basis it proposes a pruning metric, FAPM, that scores each parameter by its absolute change and penalizes parameters whose change is large relative to their pre-trained value. Pruning the fine-tuned model's task vector with this score, the authors report, reduces forgetting on general benchmarks to 0.25% while keeping 99.67% of downstream accuracy, without retraining or extra data. The significance is a post-hoc, no-training way to restore generality after fine-tuning.

What carries the argument

The central object is the Forgetting-Aware Pruning Metric (FAPM), an unstructured pruning score applied to the task vector ΔW = W_ft − W_pre. It subtracts a scaled relative-change term from the absolute-change term so that a parameter with both a large absolute update and a large update relative to its pre-trained value is penalized and pruned. The scaling by the average pre-trained magnitude |W_pre|_avg brings the two criteria onto a comparable numerical range.

What would settle it

On a benchmark where the coordinates with largest |ΔW/W_pre| are few, compare FAPM against pruning the same number of random coordinates. The paper's mechanism predicts FAPM's mask specifically recovers general-task accuracy while a random mask does not; if random masks match its general-task recovery at equal downstream accuracy, the relative-change driver is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that the ratio ΔW/W_pre — the relative change magnitude of each parameter from pre-trained to fine-tuned weights — is the quantity that governs catastrophic forgetting. Since the fine-tuned weights factor as W_pre(1 + ΔW/W_pre), and W_pre itself cannot cause forgetting, the paper argues the relative-change term is what disrupts pre-trained knowledge. FAPM uses a hybrid score S = |ΔW| − |W_pre|_avg * |ΔW| / |W_pre| to prune the task vector: it keeps parameters with large absolute change (needed for the downstream task) while removing those whose relative deviation is largest (blamed for forgetting). The paper reports this yields a better balance than magnitude pruning alo

Load-bearing premise

The claim collapses if catastrophic forgetting is not primarily caused by parameters with large relative change |ΔW/W_pre|, so pruning those parameters would not reliably restore generality at a given sparsity.

Editorial extensions

If this is right

  • FAPM can be applied after any fine-tuning run as a post-processing step, requiring only the pre-trained and fine-tuned checkpoints and no training data.
  • At 90% sparsity it preserves near-full downstream accuracy while restoring general capability to nearly pre-trained levels, so models can be specialized without sacrificing generality.
  • The same relative-change score extends to LoRA fine-tuning by treating the merged low-rank update as the task vector, reducing forgetting there too.
  • FAPM also mitigates forgetting in sequential fine-tuning, where a model trained on one task is then trained on a second, suggesting it can be applied repeatedly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the relative-change criterion could serve as a cheap diagnostic for which parameters a fine-tuning run has 'hijacked', potentially guiding data selection or hyperparameter choices rather than only pruning.
  • I infer that FAPM's gains should shrink on tasks where large absolute changes and large relative changes occur in the same coordinates, because there is little to prune that separates the two objectives.
  • I infer that the metric could be combined with training-based methods for mitigating catastrophic forgetting, a direction the paper explicitly leaves open, to push forgetting below the already-low reported level.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FAPM, a post-hoc pruning method that sparsifies the task vector ΔW = W_ft − W_pre after full fine-tuning in order to reduce catastrophic forgetting (CF) while preserving downstream task accuracy. The pruning score in Eq. (1) subtracts a pretrained-magnitude-scaled relative-change term from the absolute task-vector magnitude. Experiments on Llama3-8B and Qwen2-7B across eight downstream datasets and four general-capability benchmarks compare FAPM with L1 regularization, WiSE-FT, V-SoftMask, CoFiTune, and LoRA, and report that FAPM achieves a favorable trade-off. The paper also applies an adapted version to LoRA fine-tuning and to sequential fine-tuning. The central empirical claim is that FAPM limits CF to 0.25% while maintaining 99.67% accuracy on downstream tasks, and that the relative-change magnitude |ΔW|/|W_pre| is a key driver of CF.

Significance. If the empirical results are sound, FAPM is a simple, practical, post-hoc method for mitigating CF: it requires no changes to training, no auxiliary data, no architectural modification, and only uses the pre-trained and fine-tuned weights. The paper includes code, experiments on two model families and eight datasets, and comparisons with several baselines. These are genuine strengths. However, the central theoretical claim is currently not supported: Eq. (1) algebraically reduces to magnitude pruning reweighted by a monotone function of |W_pre|, so the proposed 'relative-change' mechanism is confounded with pretrained-magnitude gating. In addition, the headline numbers in the abstract do not match the numbers reported in Section 5.1. The paper may still describe a useful empirical method, but the explanatory contribution needs substantial rework and additional controlled experiments.

major comments (4)
  1. [Abstract and §5.1] The abstract claims 'FAPM limits CF to just 0.25% while maintaining 99.67% accuracy on downstream tasks.' In §5.1, the 0.25% is the decrease in downstream-task performance relative to Full SFT (0.816 → 0.814), while CF—the loss on general tasks—is reported as 0.33% (0.6132 → 0.6112). The figure 99.67% is the retained fraction of pre-trained general-task performance, not downstream accuracy. The headline therefore misstates both the CF rate and the downstream accuracy and must be corrected.
  2. [Eq. (1), §3] Let c_i = |W_i_pre|_avg, which is a per-layer scalar. Then Eq. (1) becomes S_i = |ΔW_i|(1 − c_i / |W_i_pre|). The 'relative change' term is therefore a monotone function of |W_i_pre| alone; it does not create a genuine joint absolute+relative criterion. FAPM is magnitude pruning reweighted by pretrained weight magnitude. Consequently, the asserted mechanism—that large relative change identifies CF-prone parameters—is confounded with the pretrained-magnitude distribution. The paper needs a control experiment, e.g., comparing FAPM with magnitude pruning at matched sparsity and with a criterion |ΔW|·h(|W_pre|) for a different monotone h, or stratifying by |W_pre| to test whether relative change adds predictive power beyond |W_pre|.
  3. [§5.3, Table 3] The experiment comparing retention by |ΔW|/|W_pre| with retention by |ΔW| does not isolate relative change. Parameters with large |ΔW|/|W_pre| are disproportionately those with small |W_pre|; without controlling for |W_pre|, the observed difference from magnitude pruning may be driven by the denominator alone. The statement that the results 'reveal a strong relationship between |ΔW|/|W_pre| and CF' is not supported by this comparison. A proper control would bin parameters by |W_pre| and vary |ΔW| within bins, or equivalently compare against a criterion that uses |W_pre| independently of |ΔW|.
  4. [§2.2] The 'Theoretical analysis' is an algebraic identity plus a conjecture. The factorization W_ft = W_pre(1 + ΔW/W_pre) is trivially true, and the claim that 'CF may be more closely related to the last term ΔW/W_pre' is not derived; it is an assumption. The heading overstates the content. If the authors wish to make a theoretical claim, a formal statement with proof, or at least a clearly labeled hypothesis with dedicated tests, is required. Otherwise the narrative should be reframed as an empirical heuristic.
minor comments (5)
  1. [Abstract / §5.1] The abstract's '99.67% accuracy on downstream tasks' should be reworded; as written it is contradicted by the body's downstream-task numbers. Please report the correct quantity (retained general-task performance) and align the CF percentage with the body.
  2. [Tables 1 and 6] The table headers are hard to parse: 'Avg. Results' appears twice, and the Pre-trained rows appear to contain numbers that are not downstream-task accuracies. Please clarify the column layout and define what the last numeric column in each block represents.
  3. [General] Only point estimates are reported; no standard deviations, confidence intervals, or repeated-seed results are given. Since many differences are on the order of 0.001–0.005, the significance of the comparisons is unclear.
  4. [§3, Eq. (1)] The notation i is used both for a layer and for an element within a layer. Please clarify that |W_i_pre|_avg is a per-layer scalar and that the score is computed elementwise.
  5. [Appendix D] The global 90% sparsity ratio is applied across all datasets and models. The ablation shows trends but does not provide a principled selection rule or sensitivity analysis. Since the headline numbers depend on this choice, report how 90% was selected and whether the conclusions are robust to it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FAPM is a hand-designed pruning score evaluated on held-out benchmarks; no fitted parameter or self-citation chain carries the central claim.

full rationale

The derivation chain is self-contained. Equation (1) defines S_i = |ΔW_i| − |W^i_pre|_avg · |ΔW_i|/|W^i_pre| using only the fine-tuned and pre-trained weights. The evaluation targets (MMLU, C-Eval, GSM8K, HumanEval, and downstream accuracy) are external held-out benchmarks; they are not used to construct or fit the pruning score. The only tunable quantity is the global sparsity ratio, which is a standard hyperparameter and is not renamed as a prediction. The algebraic identity S_i = |ΔW_i|(1 − c_i/|W^i_pre|) shows that the relative-change term acts as a per-layer rescaling by pretrained magnitude; this is a potential confound for the paper's mechanistic claim that relative change is the driver of forgetting, but it is a correctness/identification concern, not circularity. The Limitations section (no integration with training-based CF methods) is a scope limitation and does not reveal a circular step. No load-bearing self-citations were found. The claim that FAPM reduces forgetting is an empirical, falsifiable result, not a tautology.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. It relies on the task-vector definition, a heuristic assumption about relative change magnitude, and the empirically motivated choice to prune ΔW and restore W_pre. The single significant free parameter is the sparsity ratio, chosen by test-set inspection.

free parameters (2)
  • sparsity ratio (full fine-tuning) = 90%
    Set uniformly across models and datasets after inspecting ablation curves in Appendix D; the paper states this 'may represent an optimal balance'.
  • LoRA pruning rate = 3-10% (per dataset)
    Table 2 reports results at several pruning rates (0%, 3%, 10%, 90%); the text notes the rate must be small, implying per-dataset selection.
assumptions (4)
  • standard math Task vector ΔW = W_ft - W_pre captures the fine-tuning change (Ilharco et al., 2022).
    Definition borrowed from prior work, used as the object to prune.
  • domain assumption CF is related to the relative change magnitude |ΔW/W_pre| rather than only |ΔW|.
    Section 2.2 proposes this based on the algebraic identity and intuitions; no proof is given.
  • domain assumption Pruning task vector entries and restoring W_pre recovers general knowledge while retaining task accuracy.
    Section 2.1 and Figure 2; assumes redundancy in ΔW and that restored W_pre values are beneficial.
  • domain assumption Magnitude pruning of ΔW preserves downstream task accuracy up to high sparsity.
    Empirically observed in Figure 2 and used as the base of FAPM.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning." pith.science (2026). https://pith.science/paper/YPZ6ZMJH

@misc{pith2026250908255,
  author       = {Pith},
  title        = {Pith review of: Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YPZ6ZMJH}},
  note         = {Machine review of arXiv:2509.08255}
}
read the original abstract

Recent advancements in large language models (LLMs) have shown impressive capabilities in various downstream tasks but typically face Catastrophic Forgetting (CF) during fine-tuning. In this paper, we propose the Forgetting-Aware Pruning Metric (FAPM), a novel pruning-based approach to balance CF and downstream task performance. Our investigation reveals that the degree to which task vectors (i.e., the subtraction of pre-trained weights from the weights fine-tuned on downstream tasks) overlap with pre-trained model parameters is a critical factor for CF. Based on this finding, FAPM employs the ratio of the task vector to pre-trained model parameters as a metric to quantify CF, integrating this measure into the pruning criteria. Importantly, FAPM does not necessitate modifications to the training process or model architecture, nor does it require any auxiliary data. We conducted extensive experiments across eight datasets, covering natural language inference, General Q&A, Medical Q&A, Math Q&A, reading comprehension, and cloze tests. The results demonstrate that FAPM limits CF to just 0.25\% while maintaining 99.67\% accuracy on downstream tasks. We provide the code to reproduce our results.

Figures

Figures reproduced from arXiv: 2509.08255 by the authors.

Figure 1
Figure 1. Illustration of the issues in CF, the desired [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The relationship between the magnitude pruning sparsity ratio, general capability, and downstream task [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the weight matrices in different layers of Llama3-8B fine-tuned on RTE dataset. From left [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Illustration of pruning with FAPM. If |∆Wi | is large (it will be retained by magnitude pruning) and |∆Wi | |Wi pre| is also large, FAPM will penalize and possibly prune this parameter, e.g., the value 1.1 in the middle of ∆W. By doing so, most large-magnitude paramete…
Figure 5
Figure 5. Figure 5: Performance of FAPM on downstream task accuracy and mitigation of catastrophic forgetting with different sparsity ratios on Llama3-8B. 50 55 60 65 70 75 80 85 90 95 Sparsity Ratio (%) 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 Performance Downstream Task Performance MRPC …
Figure 6
Figure 6. Figure 6: Performance of FAPM on downstream task accuracy and mitigation of catastrophic forgetting with different sparsity ratios on Qwen2-7B. gradually increases. These observations suggest that a 90% sparsity ratio may represent an opti- [PITH_FULL_IMAGE:figures/full_fig_p01…
Figure 7
Figure 7. Figure 7: Visualization of the weight matrices in different layers of Qwen2-7B fine-tuned on RTE dataset. From left [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: The relationship between the magnitude prun [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 14 canonical work pages

  1. [1]

    S \'e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712

    arXiv 2023

  2. [2]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  3. [3]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  4. [4]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  5. [5]

    Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28

    work page 2015

  6. [6]

    Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608

    arXiv 2024

  7. [7]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  8. [8]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    arXiv 2021

Show all 41 references
  1. [9]

    Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. 2024. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. arXiv preprint arXiv:2403.01244

    2024 arXiv

  2. [10]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint a...

    2023 arXiv

  3. [11]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations

    2022

  4. [12]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    2021

  5. [13]

    Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual pre-training of language models. In The Eleventh International Conference on Learning Representations

    2023

  6. [14]

    Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. arXiv:1910.11473v2

    2020 arXiv

  7. [15]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of...

    2017

  8. [16]

    Yajing Kong, Liu Liu, Huanhuan Chen, Janusz Kacprzyk, and Dacheng Tao. 2023. Overcoming catastrophic forgetting in continual learning by exploring eigenvalues of hessian matrix. IEEE Transactions on Neural Networks and Learning Systems

    2023

  9. [17]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  10. [18]

    Yann LeCun, John Denker, and Sara Solla. 1989. https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf Optimal brain damage . In Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann

    1989

  11. [19]

    Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. 2023. Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprint arXiv:2309.06256

    2023 arXiv

  12. [20]

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747

    2023 arXiv

  13. [21]

    Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. 2024. Lottery ticket adaptation: Mitigating destructive interference in llms. arXiv preprint arXiv:2406.16797

    2024 arXiv

  14. [22]

    Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. 2023. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, pages 27011--27033. PMLR

    2023

  15. [23]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  16. [24]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  17. [25]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  18. [26]

    Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6107--6122

    2022

  19. [27]

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695

    2023 arXiv

  20. [28]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE : A multi-task benchmark and analysis platform for natural language understanding. In the Proceedings of ICLR

    2019

  21. [29]

    Zhicheng Wang, Yufang Liu, Tao Ji, Xiaoling Wang, Yuanbin Wu, Congcong Jiang, Ye Chao, Zhencong Han, Ling Wang, Xu Shao, et al. 2023. Rehearsal-free continual language learning via efficient parameter isolation. In Proceedings of the 61st Annual Meeting of the Association for ...

    2023

  22. [30]

    Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. 2021. https://api.semanticscholar.org/CorpusID:237420687 Robust fine-tuning of zero-shot models . 2022 IEEE/CVF Conference on Computer Vision an...

    2021

  23. [31]

    Mitigating catastrophic forgetting in online continual learning by modeling previous task interrelations via pareto optimization

    Yichen Wu, Hong Wang, Peilin Zhao, Yefeng Zheng, Ying Wei, and Long-Kai Huang. Mitigating catastrophic forgetting in online continual learning by modeling previous task interrelations via pareto optimization. In Forty-first International Conference on Machine Learning

  24. [32]

    Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2024. Reft: Representation finetuning for language models. arXiv preprint arXiv:2404.03592

    2024 arXiv

  25. [33]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36

    2024

  26. [34]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024 a . Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  27. [35]

    Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jia, and Hongying Zan. 2024 b . Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. In Proceedings of the AAAI Confere...

    2024

  28. [36]

    Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. https://doi.org/10.18653/v1/D15-1237 W iki QA : A challenge dataset for open-domain question answering . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013--2018, Lisbon, Portu...

    2015 doi

  29. [37]

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284

    2023 arXiv

  30. [38]

    Hengyuan Zhang, Yanru Wu, Dawei Li, Zacc Yang, Rui Zhao, Yong Jiang, and Fei Tan. 2024. Balancing speciality and versatility: a coarse to fine framework for supervised fine-tuning large language model. arXiv preprint arXiv:2404.10306

    2024 arXiv

  31. [39]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372

    2024 arXiv

  32. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.