Pith. sign in

REVIEW 5 cited by

Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2109.05687 v1 pith:RN62SPZK submitted 2021-09-13 cs.CL cs.AI

Runxin Xu , Fuli Luo , Zhiyuan Zhang , Chuanqi Tan , Baobao Chang , Songfang Huang , Fei Huang This is my paper · ORCID
classification cs.CLcs.AI
keywords fine-tuninglargepretrainedchild-tuningmodelschilddownstreameffective
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent pretrained language models extend from millions to billions of parameters. Thus the need to fine-tune an extremely large pretrained model with a limited training corpus arises in various downstream tasks. In this paper, we propose a straightforward yet effective fine-tuning technique, Child-Tuning, which updates a subset of parameters (called child network) of large pretrained models via strategically masking out the gradients of the non-child network during the backward process. Experiments on various downstream tasks in GLUE benchmark show that Child-Tuning consistently outperforms the vanilla fine-tuning by 1.5~8.6 average score among four different pretrained models, and surpasses the prior fine-tuning techniques by 0.6~1.3 points. Furthermore, empirical results on domain transfer and task transfer show that Child-Tuning can obtain better generalization performance by large margins.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Vision Transformer Finetuning Benefits from Non-Smooth Components

    cs.LG 2026-02 conditional novelty 6.0 of 10

    For vision transformers, components with higher input-output sensitivity (attention and feedforward layers) yield better and more stable fine-tuning accuracy than smoother LayerNorm components.

  2. CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentation

    cs.CV 2025-11 conditional novelty 6.0 of 10

    A Fisher-information-guided dynamic selection over a toolbox of LoRA, adapter, and frequency-adapter modules improves cross-domain remote sensing segmentation over static PEFT methods.

  3. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  4. GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation

    cs.LG 2025-08 conditional novelty 4.0 of 10

    GEM selects fine-tuning parameters by gradient-to-weight ratio and distributes the budget by layer entropy, reaching 0.1% parameter updates with small accuracy gains on several NLP tasks.

  5. Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models

    cs.CL 2025-09 reject novelty 2.0 of 10

    A gated-adapter method with a sparsity regularizer is reported to match or slightly beat full fine-tuning on MNLI and BoolQ using 1.4% of parameters, but lacks the experimental detail needed to verify the claim.

Pith tools