REVIEW 6 cited by
Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model, preserving learned features while also adapting to the new task. This paper shows that in such settings, selectively fine-tuning a subset of layers (which we term surgical fine-tuning) matches or outperforms commonly used fine-tuning approaches. Moreover, the type of distribution shift influences which subset is more effective to tune: for example, for image corruptions, fine-tuning only the first few layers works best. We validate our findings systematically across seven real-world data tasks spanning three types of distribution shifts. Theoretically, we prove that for two-layer neural networks in an idealized setting, first-layer tuning can outperform fine-tuning all layers. Intuitively, fine-tuning more parameters on a small target dataset can cause information learned during pre-training to be forgotten, and the relevant information depends on the type of shift.
Forward citations
Cited by 6 Pith papers
-
Toward Mechanistic Interpretability of an AI Foundation Model Fine-Tuned for Atmospheric Chemistry
Aurora, a foundation model fine-tuned for air quality, captures rough NOx–ozone coupling but lacks the chemical consistency and emission-plume fidelity of process-based models.
-
SAFE-Merge: Data-Free Continual Model Merging with General Knowledge Preservation
SAFE-Merge masks risk-prone parameter updates and recovers lost task information with a constrained low-rank correction, achieving the best H-score in data-free continual model merging benchmarks.
-
FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts
FLoE uses Fisher information to pick the transformer layers that matter and a Bayesian optimizer to set LoRA rank, cutting trainable parameters while keeping or improving accuracy.
-
Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performance
A CLIP fine-tuning method that penalizes output differences from the pretrained model and prediction differences under augmentation improves both ID accuracy and OOD robustness in reported experiments.
-
Neurosymbolic Artificial Intelligence for Robust Network Intrusion Detection: From Scratch to Transfer Learning
Reusing a frozen pretrained autoencoder, retrained clustering, and fine-tuned XGBoost on ACI-IoT-2023 outperforms FcNN and 1D-CNN with about half the training data, and metamodel-based UQ outperforms score-based UQ on...
-
ReStNet: A Reusable & Stitchable Network for Dynamic Adaptation on IoT Devices
ReStNet builds a hybrid model from two pre-trained networks by stitching at the most CKA-similar layer and fine-tuning only the stitching layer, claiming flexible runtime trade-offs for IoT devices.
Discussion (0). Continue with ORCID to comment.