Pith. sign in

REVIEW 3 major objections 4 minor 36 references

Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Domain finetuning can keep general knowledge if the training target is the unfinetuned model's soft labels, corrected token-by-token toward the ground truth; the paper reports 3-15x less general-domain perplexity damage than standard…

desk verdict A genuinely new corrective self-distillation objective with broad, consistent experiments, but the OpenWebText reference-tuning protocol likely inflates the headline degeneralization numbers. read the letter →

arxiv 2506.15702 v1 pith:XYJJMVSV submitted 2025-05-30 cs.LG cs.AI

Peter Belcak , Greg Heinrich , Jan Kautz , Pavlo Molchanov This is my paper
classification cs.LGcs.AI
keywords languagemodelfinetuningdomainadaptationcatastrophicforgettingself-distillationlow-dataregimesoftlabelsreplay-freedegeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Finetuning a language model on a new, small domain normally costs it general knowledge, and the smaller the dataset, the worse the loss. Minifinetuning (MFT) is a training objective designed to break that trade: a frozen copy of the unfinetuned model acts as teacher, and the student trains not on one-hot tokens but on the teacher's next-token distributions, corrected only enough to favor the ground-truth token by a margin $\tau$. Across three specialized domains and several model families, the paper reports that standard finetuning's general-domain perplexity detriment is three to fifteen times larger than MFT's, while MFT keeps most of the specialization, yielding degeneralization-to-specialization ratios that are 2-10 times more favorable. The value of the claim is that MFT needs no pre-training data for replay, holds up at budgets as low as 500 samples, and composes with parameter-efficient finetuning.

What carries the argument

The load-bearing mechanism is the distribution correction function $\mathrm{DC}_{\tau}$ applied token-by-token. For a teacher distribution $p_T$ and ground-truth label $l$, if $\arg\max p_T \neq l$ the training target is $p_C=(1-\alpha)p_T+\alpha\mathbf{1}_l$ with $\alpha$ chosen so $p_C^l$ exceeds $p_C^{\arg\max p_T}$ by exactly $\tau$; if the teacher already predicts $l$, the target is $p_C=(1-\beta)p_T+\beta\mathbf{1}_l$ with $p_C^l=\min(1,p_T^l+\tau)$. The student minimizes cross-entropy against these corrected per-token distributions, and the frozen teacher's unmodified soft labels carry the general-domain information. Setting $\tau=1$ collapses the formula to standard finetuning, so the method is a dial rather than a separate algorithm.

What would settle it

Run FT and MFT on the same low-data domain and stop both at the same specialized-domain validation perplexity; if their general-domain perplexities are then equal, MFT's advantage reduces to learning the new domain more slowly, and the claimed replay-like preservation of general knowledge is not needed to explain the results.

Watch

Extended reading notes

Core claim

The central claim is that the unfinetuned teacher's soft labels on new-domain inputs are a compressed replay of the general domain, so correcting them toward ground truth and training the student against them delivers specialization without the usual forgetting. Quantitatively, the paper finds three- to fifteen-fold differences in relative perplexity detriment on the general domain between FT and MFT across evaluated models and corpora, with MFT sacrificing 25-35% of specialization for that retention. A single target $\tau$ interpolates between pure self-distillation ($\tau=0$) and standard finetuning ($\tau=1$), making the specialization-retention trade a controllable parameter. The ablation shows the full doubly-corrective formula is necessary: fixing only incorrect teacher predictions hurts the ratio compared with plain finetuning, and only correcting both correct and incorrect predictions yields the reported behavior.

Load-bearing premise

The load-bearing premise is that the unfinetuned teacher's next-token distributions on the new domain's inputs carry enough general-domain information that correcting them toward the ground truth preserves general performance while still teaching the new domain; the paper does not prove this independently of the MFT results themselves.

Editorial extensions

If this is right

  • Standard finetuning's general-domain perplexity detriment is three to fifteen times larger than MFT's across the evaluated models and domains, and MFT's degeneralization-to-specialization ratios are 2-4x better in the main table and up to 10x better in the wider evaluation.
  • At data budgets down to 500 samples, MFT keeps improving on the specialized domain, whereas standard finetuning begins overfitting within a few hundred steps and can end worse than its starting point.
  • The correction target $\tau$ gives practitioners a continuous control between specialization and retention: $\tau=1$ is standard finetuning, and lower $\tau$ trades some specialization for much lower degeneralization.
  • MFT reproduces the shape of replay's degeneralization mitigation without any pre-training data, because the teacher's soft labels act as compressed replay samples.
  • MFT composes with replay and with parameter-efficient finetuning methods, so the memory savings of LoRA, DoRA, or IA3 can offset the cost of the extra teacher copy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test instruction tuning, but the same compressed-replay mechanism is a natural fit there: if MFT's anchor works, corrected teacher logits should reduce the post-training distribution shift the paper names as a motivation.
  • A testable extension the paper leaves implicit is annealing $\tau$ during training instead of fixing it at 0.25; starting high to anchor general knowledge and lowering it later could recover some of the specialization MFT gives up.
  • The mechanism predicts a measurable boundary condition: on domains where the unfinetuned teacher is nearly random, the soft-label anchor carries little general-domain information, so MFT should converge toward plain distillation and lose its degeneralization advantage.
  • Because MFT requires a second frozen copy and an extra forward pass, its relative overhead should decrease as model size grows; the paper's memory analysis supports this but the paper does not benchmark training throughput at scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes minifinetuning (MFT), a self-distillation objective for low-resource domain adaptation of generative language models. A frozen teacher (initialized as a reference checkpoint) produces per-token soft labels that are corrected toward the ground truth by a margin tau, and the student is trained with cross-entropy against these corrected distributions. Experiments on OpenELM, GPT-Neo, Phi, Gemma, Minitron, and LLaMA variants across PubMed, Pile of Law, and OpenWebMath report that MFT yields lower degeneralization and better DG/S ratios than standard finetuning, is competitive with replay-like behavior, and composes with PEFT methods.

Significance. If the reported effects are robust, MFT is a simple, training-objective-only intervention that mitigates catastrophic forgetting in low-data domain adaptation without replay data, and the ablation and sensitivity analyses are thorough. The correction formula in Section 2 is derived cleanly and correctly reduces to standard finetuning at tau = 1. The paper also benefits from a wide model/domain coverage and from an explicit incremental ablation of the correction components. However, the OpenWebText reference-checkpoint protocol confounds the central empirical claim, and the absence of error bars or multiple seeds makes the quantitative advantage uncertain.

major comments (3)
  1. [Section 3.1 (Process) and Section 7] The reference checkpoint is produced by tuning on OpenWebText (OWT), the same corpus used to measure general-domain perplexity, and this checkpoint serves both as the MFT teacher and as the common initialization for FT and MFT. Because MFT trains the student against soft labels that encode the OWT-tuned distribution, the student is by construction pushed toward the exact distribution on which DG is measured, while plain FT has no such anchor. The reported 3-15x lower DG (Section 3.2) and the abstract's 2-10x ratio improvement are therefore not cleanly attributable to the MFT objective alone. The Limitations section mentions the OWT starting point as an obstacle, but it does not quantify how much of the advantage comes from the protocol. Please add experiments starting from the original pretrained checkpoint without OWT reference tuning, and/or evaluate DG on a held-out general corpus not used to prepare the teacher.
  2. [Section 3.1 (Metrics and checkpoint selection)] Each reported S, DG, and ratio value comes from a single training run, and the reported checkpoint is selected as the one with the lowest specialized-domain validation perplexity. This oracle-style selection can inflate specialization and make the DG/S ratio look more favorable, and without multiple seeds or error bars it is not possible to assess whether the consistent-looking pattern in Tables 2 and 4 is significant. Please provide at least three seeds with means and confidence intervals, or otherwise justify that the single-run checkpoints are representative.
  3. [Section 3.2 and Abstract] The quantitative summaries are not fully supported by the displayed tables. In Table 2, the FT-to-MFT DG/S ratio improvement ranges from about 1.75x (OpenELM 450M, PubMed: 0.07 vs 0.04) to about 4.7x (OpenELM 1.1B, OpenWebMath: 0.14 vs 0.03), not 2-10x; 10x-level figures arise only when comparing ratios against near-zero DG entries. The '25-35% higher specialization' claim is also not met by several rows, for example OpenELM 1.1B PubMed is 9.3 vs 8.2, about 13% higher. Please state the observed ranges precisely or identify the specific table cells that justify the abstract's headline numbers.
minor comments (4)
  1. [Section 4] In the paragraph 'Corrective finetuning hurts models the most', the sentence 'The DG/S ratios for this method are the smallest among all the methods' is inconsistent with the preceding claim of worse ratio performance; the intended word is 'largest' or 'least favorable'.
  2. [Section 3.1 and Appendix D] The main evaluation fixes tau = 0.25 without stating a selection criterion; because tau directly controls the specialization/degeneralization trade-off, please state whether this value was chosen a priori or by validation, and consider reporting the sensitivity in the main text rather than only in Appendix D.
  3. [Table 5 caption and Section 6] The '(author?)' placeholders in the EWC citation and in the related-work discussion must be completed before publication.
  4. [Section 5] The statement that 'FT even results in worse-than-initial performance on the specialized domain after just a few hundred steps' appears to refer specifically to the 1M-token budget in Figure 5; the caption or text should state this explicitly, as the surrounding text reads like a general statement across all budgets.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported degeneralization advantage is partly by construction: the MFT teacher is a reference checkpoint tuned on OpenWebText, the same corpus used to measure DG.

  1. fitted input called prediction [Sections 3.1 (Data/Process) and 2 (Distribution correction), with results in Table 2]
    "For each model, we first prepare a reference checkpoint lightly tuned on OpenWebText on at most 4M unique tokens and choose the checkpoint with the least validation perplexity. ... To keep track of the model understanding of the general domain, we use OpenWebText. ... p_C = (1-alpha)p_T + alpha 1_l, where p_T is the teacher distribution."

    The MFT teacher is the OpenWebText-tuned reference checkpoint, and DG is the relative OpenWebText validation perplexity increase. MFT's training target is p_C = (1-alpha)p_T + alpha 1_l, a convex combination that explicitly preserves a fraction of the teacher's distribution. Thus the student is trained to stay close to a model that was selected for low OpenWebText perplexity, on the same corpus used to compute DG. Standard FT, which starts from the same checkpoint but is trained only against one-hot labels, has no such anchor. The reported 3-15x lower DG and 2-10x better ratios are therefore partly a consequence of choosing the teacher on the evaluation metric, not an independent discovery.

full rationale

The formal derivation of the MFT correction formula is not circular: alpha and beta are solved from a stated margin requirement, and the method is then evaluated against external baselines on held-out validation splits. The tau ablation in Appendix D is a genuine hyperparameter study, not a fitted prediction. There are no load-bearing self-citations: the cited DoRA and OpenELM works are baselines or model sources, not used to justify the MFT mechanism. However, one protocol-level circularity affects the central empirical claim. The reference checkpoint used as the MFT teacher is explicitly tuned on OpenWebText and selected by OpenWebText validation perplexity, and OpenWebText is exactly the corpus on which degeneralization is measured. Because the MFT loss trains the student toward a convex combination that retains the teacher's soft distribution, the student is anchored to a model optimized for the DG metric, while standard finetuning is not. This makes a substantial portion of the reported degeneralization reduction and the 2-10x ratio improvement a consequence of the evaluation protocol rather than an independent property of the method. The paper's own Section 7 acknowledges that adapting all models to one general-domain dataset is 'an obstacle' but does not quantify the effect on the headline numbers. The method still contains independent content: specialization gains, PEFT compositions, and internal ablations do not reduce to the same construction. Overall, partial circularity raises the score to 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical claim depends on one hand-chosen hyperparameter (tau), on the assumption that teacher soft labels can substitute for replay data, and on a measurement protocol that includes an under-specified OpenWebText reference-tuning step. No new physical or architectural entities are introduced.

free parameters (3)
  • Tau (target correction margin) = 0.25
    Hand-chosen default in Section 3.1; ablated in Appendix D. The central quantitative claims are reported at this value.
  • Training hyperparameters (learning rate, optimizer, schedule) = not specified
    Section 3.1 gives batch size, sequence length, and steps, but omits optimizer, learning rate, and schedule needed for exact replication.
  • OpenWebText reference-tuning schedule = not specified
    Section 3.1 Process says the reference checkpoint is 'lightly tuned' with 'at most 4M unique tokens' and early stopping on validation perplexity, but does not report steps or learning rate.
assumptions (4)
  • domain assumption Validation perplexity on OpenWebText is a sufficient measure of general-domain performance for the models tested.
    Section 3.1 states: 'To keep track of the model understanding of the general domain, we use OpenWebText.' The entire degeneralization metric rests on this corpus being representative of general text.
  • domain assumption Lightly tuning each model on OpenWebText before comparisons yields a valid common starting point and does not distort the specialization/degeneralization measurements.
    Section 3.1 Process creates a reference checkpoint tuned on OpenWebText; Section 7 Limitations admits this step 'makes the models forget parts of the knowledge gained in their post-training even before they begin to adapt to the specialized domain.'
  • ad hoc to paper The teacher's soft labels on the specialized domain act as a compressed substitute for replay data from the general domain.
    Section 3.3 states the soft labels 'act as highly compressed replay samples.' This is the core mechanism assumed to produce MFT's benefit, not a result proven elsewhere.
  • standard math The distribution correction formula preserves relative probabilities among all non-target tokens while adjusting target mass.
    Section 2 constructs p_C as a uniform scaling of p_T plus mass at the label, which preserves ratios among other token pairs by construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation." pith.science (2026). https://pith.science/paper/XYJJMVSV

@misc{pith2026250615702,
  author       = {Pith},
  title        = {Pith review of: Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XYJJMVSV}},
  note         = {Machine review of arXiv:2506.15702}
}
read the original abstract

Finetuning language models for a new domain inevitably leads to the deterioration of their general performance. This becomes more pronounced the more limited the finetuning data resource. We introduce minifinetuning (MFT), a method for language model domain adaptation that considerably reduces the effects of overfitting-induced degeneralization in low-data settings and which does so in the absence of any pre-training data for replay. MFT demonstrates 2-10x more favourable specialization-to-degeneralization ratios than standard finetuning across a wide range of models and domains and exhibits an intrinsic robustness to overfitting when data in the new domain is scarce and down to as little as 500 samples. Employing corrective self-distillation that is individualized on the sample level, MFT outperforms parameter-efficient finetuning methods, demonstrates replay-like degeneralization mitigation properties, and is composable with either for a combined effect.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 14 canonical work pages

  1. [1]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems , volume 35, pages 1950–1965. Curran As...

    work page 1950

  2. [2]

    Fine-tuning can distort pretrained features and underperform out-of-distribution.arXiv preprint arXiv:2202.10054 , 2022

    Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution.arXiv preprint arXiv:2202.10054 , 2022

    arXiv 2022

  3. [3]

    Distill and replay for continual language learning

    Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Distill and replay for continual language learning. In Proceedings of the 28th international conference on computational linguistics , pages 3569–3579, 2020

    work page 2020

  4. [4]

    Scalable language model with generalized continual learning

    Bohao Peng, Zhuotao Tian, Shu Liu, Mingchang Yang, and Jiaya Jia. Scalable language model with generalized continual learning. arXiv preprint arXiv:2404.07470 , 2024

    arXiv 2024

  5. [5]

    Continual learning of large language models: A comprehensive survey.arXiv preprint arXiv:2404.16789 , 2024

    Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, and Hao Wang. Continual learning of large language models: A comprehensive survey.arXiv preprint arXiv:2404.16789 , 2024

    arXiv 2024

  6. [6]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  7. [7]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InInternational conference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  8. [8]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685 , 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685 , 2021

    arXiv 2021

Show all 36 references
  1. [9]

    Dora: Weight-decomposed low-rank adaptation.arXiv preprint arXiv:2402.09353 , 2024

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation.arXiv preprint arXiv:2402.09353 , 2024

    2024 arXiv

  2. [10]

    Mitigating the alignment tax of rlhf, 2024

    Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. Mitigating the alignment tax of rlhf, 2024

    2024

  3. [11]

    Reuse, don’t retrain: A recipe for continued pretraining of language models.arXiv preprint arXiv:2407.07263 , 2024

    Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don’t retrain: A recipe for continued pretraining of language models.arXiv preprint arXiv:2407.07263 , 2024

    2024 arXiv

  4. [12]

    Evaluating language model finetuning techniques for low-resource languages

    Jan Christian Blaise Cruz and Charibeth Cheng. Evaluating language model finetuning techniques for low-resource languages. arXiv preprint arXiv:1907.00409 , 2019

    1907 arXiv

  5. [13]

    Fine-tuning and utilization methods of domain-specific llms.arXiv preprint arXiv:2401.02981 , 2024

    Cheonsu Jeong. Fine-tuning and utilization methods of domain-specific llms.arXiv preprint arXiv:2401.02981 , 2024

    2024 arXiv

  6. [14]

    Harnessing pre-trained neural networks with rules for formality style transfer

    Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, and Wenhan Chao. Harnessing pre-trained neural networks with rules for formality style transfer. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu...

    2019

  7. [15]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020

    2020

  8. [16]

    Openelm: An efficient language model family with open-source training and inference framework.arXiv preprint arXiv:2404.14619 , 2024

    Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. Openelm: An efficient language model family with open-source training and inference framework.arXiv preprint arX...

    2024 arXiv

  9. [17]

    GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021

    Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. If you use this software, please cite it using these metadata

    2021

  10. [18]

    Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023. 11 Minifinetuning: Low-Data Genera...

    2023 arXiv

  11. [19]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219 , 2024

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219 , 2024

    2024 arXiv

  12. [20]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 , 2024

    2024 arXiv

  13. [21]

    Compact language models via pruning and knowledge distillation.arXiv preprint arXiv:2407.14679 , 2024

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation.arXiv preprint arXiv:2407.14679 , 2024

    2024 arXiv

  14. [22]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023

    2023 arXiv

  15. [23]

    Pmc open access subset, 2024

    Bethesda. Pmc open access subset, 2024

    2024

  16. [24]

    Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset.Advances in Neural Information Processing Systems , 35:29217–29234, 2022

    Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset.Advances in Neural Information Processing Systems , 35:29217–29234, 2022

    2022

  17. [25]

    Openwebmath: An open dataset of high-quality mathematical web text.arXiv preprint arXiv:2310.06786 , 2023

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text.arXiv preprint arXiv:2310.06786 , 2023

    2023 arXiv

  18. [26]

    Openwebtext corpus.http://Skylion007.github.io/OpenWebTextCorpus, 2019

    Aaron Gokaslan and Vanya Cohen. Openwebtext corpus.http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  19. [27]

    Efficient hierarchical domain adaptation for pretrained language models

    Alexandra Chronopoulou, Matthew E Peters, and Jesse Dodge. Efficient hierarchical domain adaptation for pretrained language models. arXiv preprint arXiv:2112.08786 , 2021

    2021 arXiv

  20. [28]

    Unsupervised domain adaptation of a pretrained cross-lingual language model.arXiv preprint arXiv:2011.11499 , 2020

    Juntao Li, Ruidan He, Hai Ye, Hwee Tou Ng, Lidong Bing, and Rui Yan. Unsupervised domain adaptation of a pretrained cross-lingual language model.arXiv preprint arXiv:2011.11499 , 2020

    2011 arXiv

  21. [29]

    Effective unsupervised domain adaptation with adversarially trained language models

    Thuy-Trang Vu, Dinh Phung, and Gholamreza Haffari. Effective unsupervised domain adaptation with adversarially trained language models. arXiv preprint arXiv:2010.01739 , 2020

    2010 arXiv

  22. [30]

    Taming pre-trained language models with n-gram representations for low-resource domain adaptation

    Shizhe Diao, Ruijia Xu, Hongjin Su, Yilei Jiang, Yan Song, and Tong Zhang. Taming pre-trained language models with n-gram representations for low-resource domain adaptation. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th ...

    2021

  23. [31]

    Yangsibo Huang, Daogao Liu, Zexuan Zhong, Weijia Shi, and Yin Tat Lee.𝑘 nn-adapter: Efficient domain adaptation for black-box language models.arXiv preprint arXiv:2302.10879 , 2023

    2023 arXiv

  24. [32]

    Maybe only 0.5% data is needed: A preliminary exploration of low training data instruction tuning.arXiv preprint arXiv:2305.09246, 2023

    Hao Chen, Yiming Zhang, Qi Zhang, Hantao Yang, Xiaomeng Hu, Xuetao Ma, Yifan Yanggong, and Junbo Zhao. Maybe only 0.5% data is needed: A preliminary exploration of low training data instruction tuning.arXiv preprint arXiv:2305.09246, 2023

    2023 arXiv

  25. [33]

    Unlocking parameter-efficient fine-tuning for low-resource language translation.arXiv preprint arXiv:2404.04212 , 2024

    Tong Su, Xin Peng, Sarubi Thillainathan, David Guzmán, Surangika Ranathunga, and En-Shiun Annie Lee. Unlocking parameter-efficient fine-tuning for low-resource language translation.arXiv preprint arXiv:2404.04212 , 2024

    2024 arXiv

  26. [34]

    When scaling meets llm finetuning: The effect of data, model and finetuning method.arXiv preprint arXiv:2402.17193 , 2024

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The effect of data, model and finetuning method.arXiv preprint arXiv:2402.17193 , 2024

    2024 arXiv

  27. [35]

    Self-distillation bridges distribution gap in language model fine-tuning.arXiv preprint arXiv:2402.13669 , 2024

    Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. Self-distillation bridges distribution gap in language model fine-tuning.arXiv preprint arXiv:2402.13669 , 2024

    2024 arXiv

  28. [36]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of scien...

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.