Pith. sign in

REVIEW 28 cited by

Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2202.10054 v1 pith:XREVD5E4 submitted 2022-02-21 cs.LG cs.CV

Ananya Kumar , Aditi Raghunathan , Robbie Jones , Tengyu Ma , Percy Liang This is my paper
classification cs.LGcs.CV
keywords fine-tuninglinearprobingaccuracypretrainedbetterfeaturesfull
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

When transferring a pretrained model to a downstream task, two popular methods are full fine-tuning (updating all the model parameters) and linear probing (updating only the last linear layer -- the "head"). It is well known that fine-tuning leads to better accuracy in-distribution (ID). However, in this paper, we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large. On 10 distribution shift datasets (Breeds-Living17, Breeds-Entity30, DomainNet, CIFAR $\to$ STL, CIFAR10.1, FMoW, ImageNetV2, ImageNet-R, ImageNet-A, ImageNet-Sketch), fine-tuning obtains on average 2% higher accuracy ID but 7% lower accuracy OOD than linear probing. We show theoretically that this tradeoff between ID and OOD accuracy arises even in a simple setting: fine-tuning overparameterized two-layer linear networks. We prove that the OOD error of fine-tuning is high when we initialize with a fixed or random head -- this is because while fine-tuning learns the head, the lower layers of the neural network change simultaneously and distort the pretrained features. Our analysis suggests that the easy two-step strategy of linear probing then full fine-tuning (LP-FT), sometimes used as a fine-tuning heuristic, combines the benefits of both fine-tuning and linear probing. Empirically, LP-FT outperforms both fine-tuning and linear probing on the above datasets (1% better ID, 10% better OOD than full fine-tuning).

Discussion (0). Sign in to comment.

Forward citations

Cited by 28 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Toward Mechanistic Interpretability of an AI Foundation Model Fine-Tuned for Atmospheric Chemistry

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Aurora, a foundation model fine-tuned for air quality, captures rough NOx–ozone coupling but lacks the chemical consistency and emission-plume fidelity of process-based models.

  2. Econometrics with Pre-Trained Embeddings for Unstructured Data

    econ.EM 2026-07 accept novelty 7.0 of 10

    Pre-trained embeddings are valid in double machine learning when the target nuisance function lies in the span of the source-task representation; under that condition the downstream estimator can converge faster than ...

  3. Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Emergent misalignment in Qwen2.5 is mediated by a causal persona direction that low-rank LoRA recruits from covert code while full SFT does not and moves against it.

  4. Fine-Tuning Regimes Define Distinct Continual Learning Problems

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    The relative rankings of continual learning methods are not preserved across different fine-tuning regimes defined by trainable parameter depth.

  5. A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    In diagonal linear networks, fine-tuning generalization is governed by a tunable per-dimension penalty whose sparsity and pretraining dependence define four regimes and a trade-off between feature reuse and new-featur...

  6. Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification

    cs.SD 2025-09 conditional novelty 6.0 of 10

    A binarized prototypical probing method that pools per-class evidence from patch tokens substantially outperforms [cls]-token and attentive probes on multi-label audio classification benchmarks.

  7. How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Zapping the last layer during pretraining speeds a model's recovery after transfer, and Adam produces different learning and forgetting patterns than SGD in continual learning.

  8. Probabilistic Prototype Calibration of Vision-Language Models for Generalized Few-shot Semantic Segmentation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    FewCLIP calibrates frozen CLIP text prototypes with probabilistic visual prototypes, reporting large novel-class mIoU gains on PASCAL-5i and COCO-20i.

  9. Exploring Visual Prompting: Robustness Inheritance and Beyond

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Visual prompts built on robust source models inherit adversarial robustness but lose standard accuracy; a max-pooling over logit blocks (PBL) improves accuracy while keeping most robustness.

  10. Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Weight averaging of pretrained and fine-tuned models is shown, in a linear model and on three LLM families, to reduce overadaptation and improve both downstream and retained knowledge.

  11. Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MFT, a corrective self-distillation objective, reports 2 to 10 times better specialization-to-degeneralization ratios than standard finetuning across many models and three specialized domains.

  12. BIRD: Behavior Induction via Representation-structure Distillation

    cs.LG 2025-05 conditional novelty 6.0 of 10

    BIRD transfers aligned behavior across models with different architectures, tasks, and data by minimizing linear CKA between teacher and student representations, and three teacher representation properties explain mos...

  13. TNG-CLIP:Training-Time Negation Data Generation for Negation Awareness of CLIP

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A training-time pipeline that synthesizes diverse negation captions from batch neighbors improves CLIP's negation accuracy on matching and generation benchmarks, and a new NEG-TTOI benchmark measures negation handling...

  14. Next Token Perception Score: Analytical Assessment of your LLM Perception Skills

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A subspace-overlap score computed from frozen LLM representations and labels is proposed, with a bounding theorem in a linear setting and empirical correlations with linear probe accuracy and LoRA gains.

  15. Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing

    cs.LG 2025-05 conditional novelty 6.0 of 10

    For binary classification with noisy labels, the paper derives the Bayes-optimal function for combining a model's current predictions with the given labels during retraining, and shows a fitted version improves linear...

  16. A Lightweight Foundation Model for Collider Physics with Multi-Domain Adaptation

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A lightweight autoencoder pre-trained on LHC track data transfers to collider and out-of-domain scientific tasks, matching a transformer within ~2% at about 46x lower per-epoch training cost.

  17. Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    LayerNorm parameter shifts after fine-tuning encode domain-transition information; rescaling them via an FSR-dependent scalar lambda plus a cyclic step improves ViT classification under data scarcity and domain shift.

  18. Predicting mutational effects on protein binding from folding energy

    q-bio.BM 2025-07 conditional novelty 5.0 of 10

    StaB-ddG predicts binding-affinity changes from folding-energy differences, matching FoldX accuracy on a homology-split SKEMPIv2.0 benchmark while running about 1,000 times faster.

  19. FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    FOCoOp uses global, local, and OOD prompts with bi-level distributionally robust optimization and semi-unbalanced optimal transport to improve OOD robustness in federated prompt learning.

  20. OV-COAST: Cost Aggregation with Optimal Transport for Open-Vocabulary Semantic Segmentation

    cs.CV 2025-06 reject novelty 5.0 of 10

    Applying Sinkhorn optimal transport to the CAT-Seg cost volume yields a small mIoU improvement on the MESS benchmark, but the training mechanism is under-specified.

  21. FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A benchmark of ten VQA datasets shows SPD wins on in-distribution and near-OOD accuracy, FTP wins on far-OOD accuracy, and question shifts dominate joint embedding shifts after fine-tuning.

  22. ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation

    cs.SE 2025-05 conditional novelty 5.0 of 10

    ReqBrain, a LoRA-fine-tuned Zephyr-7b-beta model, produces software requirements that human evaluators could not reliably tell apart from human-authored ones, with automatic metrics favoring it over untuned ChatGPT-4o.

  23. Optimal Self-Distillation for Rectified Flow via Linear Probing

    stat.ML 2026-07 accept novelty 4.0 of 10

    For linear rectified flow with ridge regression on fixed interpolants, optimally mixed self-distillation strictly improves velocity risk whenever the teacher is off the ridge stationary point, with a closed-form mixin...

  24. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

  25. SAM$^{*}$: Task-Adaptive SAM with Physics-Guided Rewards

    cs.CV 2025-09 reject novelty 4.0 of 10

    Reward-guided hyperparameter search turns SAM into a task-specific microscope-particle segmenter, with validation limited to single images and no ground truth.

  26. Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning

    cs.RO 2025-08 reject novelty 4.0 of 10

    A review that categorizes large-model-empowered embodied AI into hierarchical and end-to-end decision-making, imitation and reinforcement learning, and world models.

  27. AetherVision-Bench: An Open-Vocabulary RGB-Infrared Benchmark for Multi-Angle Segmentation across Aerial and Ground Perspectives

    cs.CV 2025-06 conditional novelty 4.0 of 10

    AetherVision-Bench curates existing RGB and IR segmentation datasets across three viewpoint classes and shows open-vocabulary models degrade sharply under sensor shift and viewpoint change.

  28. Generalizing vision-language models to novel domains: A comprehensive survey

    cs.CV 2025-06 conditional novelty 3.0 of 10

    A survey of VLM generalization literature organized by transferred module, with benchmark tables and a review of multimodal LLMs.

Pith tools