REVIEW 6 cited by
Robust fine-tuning of zero-shot models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large pre-trained models such as CLIP or ALIGN offer consistent accuracy across a range of data distributions when performing zero-shot inference (i.e., without fine-tuning on a specific dataset). Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts. We address this tension by introducing a simple and effective method for improving robustness while fine-tuning: ensembling the weights of the zero-shot and fine-tuned models (WiSE-FT). Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution. On ImageNet and five derived distribution shifts, WiSE-FT improves accuracy under distribution shift by 4 to 6 percentage points (pp) over prior work while increasing ImageNet accuracy by 1.6 pp. WiSE-FT achieves similarly large robustness gains (2 to 23 pp) on a diverse set of six further distribution shifts, and accuracy gains of 0.8 to 3.3 pp compared to standard fine-tuning on seven commonly used transfer learning datasets. These improvements come at no additional computational cost during fine-tuning or inference.
Forward citations
Cited by 6 Pith papers
-
LoRA-Loop: Closing the Synthetic Replay Cycle for Continual VLM Learning
Adapting a text-to-image generator with task-specific LoRA adapters and filtering samples by the model's own confidence improves synthetic replay in continual vision-language learning.
-
Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Models
The accuracy of CLIP and CLIP-based visual question answering models is strongly correlated with how often the concept pair in an image appears together in pretraining captions.
-
AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning
Fitting one interpolation coefficient per parameter tensor on a small exemplar memory improves continual audio–image–text retrieval over individual continual-learning checkpoints.
-
DeltaMerge-LowRes: Composing Language and Task Deltas for Low-Resource Adaptation
A TIES-style merge across language-delta and task-delta axes (cross-axis TIES) beats additive and task-only composition on low-resource summarisation and QA.
-
FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering
A benchmark of ten VQA datasets shows SPD wins on in-distribution and near-OOD accuracy, FTP wins on far-OOD accuracy, and question shifts dominate joint embedding shifts after fine-tuning.
-
Revisiting Bayesian Model Averaging in the Era of Foundation Models
The paper proposes Bayesian model averaging and an entropy-minimizing weight optimizer for ensembling foundation models, reporting accuracy gains over output averaging on image and text classification tasks.
Discussion (0). Continue with ORCID to comment.