Pith. sign in

REVIEW 5 major objections 5 minor 46 references

Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performance

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning a CLIP model with a function-space alignment penalty plus a prediction-consistency penalty raises both in-distribution accuracy and out-of-distribution robustness across transformer and convolutional backbones.

desk verdict A plausible function-space regularizer for CLIP fine-tuning whose central ablation contradicts itself; the empirical claims are unverifiable without code and a corrected table. read the letter →

arxiv 2509.05328 v1 pith:LBTFFNLS submitted 2025-08-31 cs.LG cs.CV

Xiang Yuan , Jun Shu , Deyu meng , Zongben Xu This is my paper · ORCID
classification cs.LGcs.CV
keywords robustfine-tuningout-of-distributionrobustnessfunction-spaceregularizationCLIPconsistencydistributionshiftvision-languagemodelsimageclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper targets a failure mode of fine-tuning: adapting a pre-trained CLIP model to a downstream task raises in-distribution (ID) accuracy but erodes robustness to distribution shift (OOD). It argues that the standard remedy — keeping the fine-tuned model close to the pre-trained one in weight, feature, or logit space — works only on some architectures, because those spaces are imperfect proxies for how the model's output changes across inputs. The proposed alternative, FRR-FT, constrains the prediction function itself: a functional alignment term penalizes the squared output difference between fine-tuned and pre-trained models on RandAugment-transformed images, and a consistency term stabilizes predictions on perturbed inputs. With both terms added to cross-entropy, the paper reports simultaneous gains in ID and OOD accuracy across every CLIP backbone tested, including ResNet-50, where no compared regularization baseline recovers the pre-trained OOD level. If correct, this is a regularization-only recipe that makes robustness and downstream performance joint goals rather than a trade-off.

What carries the argument

The load-bearing object is the function-space distance the paper optimizes: ||f_theta − f_0||² = E_x[|f_theta(x) − f_0(x)|²], the mean squared difference between the fine-tuned and pre-trained prediction functions, estimated as a batch mean over RandAugment-transformed training images that stand in for unavailable OOD data. This yields FAR (functional alignment regularization), and a second term FCR (functional consistency regularization) penalizes KL divergence between predictions on original and perturbed inputs. The combined objective L = L_CE + lambda_1·FAR + lambda_2·FCR carries the argument: FAR preserves the pre-trained function's OOD behavior, FCR exploits downstream-data structure f

What would settle it

Fine-tune the same CLIP backbones with the FAR term computed on a shift that RandAugment cannot plausibly generate — for example, a change of sensor modality, or a label-correlated background/environment change rather than color and geometry transforms. Then evaluate on a held-out benchmark embodying that shift: if OOD accuracy falls back to vanilla-fine-tuning levels while the function distance to the pre-trained model on augmented inputs stays small, the simulated-OOD premise of Eq. (2) is the wrong link and the claimed mechanism fails. A cheaper check: remove RandAugment (set augmentations

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that OOD robustness during fine-tuning behaves like a property of the learned function, not of the parameters, features, or logits that encode it. The authors support this with a perturbation analysis: updating a pre-trained CLIP model along random directions in function space degrades OOD accuracy far less than same-magnitude updates in parameter, feature, or logit space (Finding 1). From this they derive FRR-FT, adding two terms to the fine-tuning loss: FAR, the empirical L2 distance between the fine-tuned and pre-trained prediction functions on RandAugment-transformed inputs, and FCR, the KL divergence between the model's predictions on original

Load-bearing premise

The load-bearing premise is that RandAugment-style perturbations of the in-distribution training images faithfully stand in for the real out-of-distribution data, so that pulling the fine-tuned function toward the pre-trained function on those augmented inputs transfers to actual shifts; Section 3.2.1 (Eq. 2) asserts this approximation without validating that the augmented distribution covers the shift modes of ImageNet-R, ImageNet-A, Sketch, or the WILDS geographic shifts.

Editorial extensions

If this is right

  • Robust fine-tuning would no longer require a trade-off: the same two-term regularizer improves ID accuracy and OOD robustness together on both transformer and convolutional CLIP backbones.
  • The function-space diagnosis would explain why parameter-, feature-, and logit-alignment methods are inconsistent across architectures: they do not control how the model's output varies with input, so their OOD benefit is incidental.
  • Downstream data themselves contribute to OOD robustness, not only pre-trained knowledge: the consistency term alone lifts OOD accuracy above vanilla fine-tuning, and in combination with alignment it exceeds the pre-trained model's OOD level on the ImageNet shift suite.
  • FRR-FT composes with post hoc ensembling: interpolating the FRR-FT checkpoint with pre-trained weights yields further ID and OOD gains, per the paper's ensemble experiment.
  • In cross-class zero-shot transfer, FRR-FT matches or slightly beats the zero-shot CLIP baseline on eleven held-out datasets, where most fine-tuning baselines degrade below it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ceiling of the method is set by how well RandAugment approximates the true shift; a testable extension is to swap in shift-specific augmentations (renditions, sketches, adversarial crops, style randomization) and check whether OOD gains on the matching benchmarks scale with coverage.
  • The same recipe — a function-distance term to a frozen reference model plus a self-consistency term on perturbations — could apply to other pre-trained model families (language models, speech encoders) that show the same fine-tuning robustness drop; the paper does not test this.
  • Finding 1 suggests a practical, label-free diagnostic: during fine-tuning, monitor the function-space distance to the pre-trained checkpoint as an early-warning proxy for OOD degradation, since the paper's perturbation study shows weight/feature/logit distances do not track it reliably.
  • The reported FAR–FCR synergy is characterized on three benchmarks; how the two terms trade off as a function of dataset size, shift severity, and hyperparameters (lambda_1, lambda_2) is left open, so the robustness of the synergy claim across regimes is unverified.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes FRR-FT, a regularization-based robust fine-tuning method for CLIP models. The objective (Eq. 4) adds two function-space terms to cross-entropy: FAR, which penalizes the squared L2 distance between fine-tuned and pretrained model outputs on RandAugment-augmented inputs (Eq. 2), and FCR, which penalizes the KL divergence between predictions on original and augmented inputs (Eq. 3). The authors report that FRR-FT improves both ID accuracy and OOD robustness relative to parameter-, feature-, and logit-space baselines across CLIP ViT-L/14, ViT-B/32, ViT-B/16, and ResNet-50 on ImageNet variants, WILDS-iWildCam, WILDS-FMoW, and cross-class zero-shot transfer. They also claim architecture-agnostic behavior (Finding 3) and show compatibility with weight-interpolation ensembling.

Significance. If the reported results hold, the contribution is practically significant: the objective is simple, requires no OOD data, and appears to generalize across transformer and convolutional CLIP backbones. The paper is strong in evaluation breadth, comparing many recent baselines on multiple distribution-shift benchmarks. However, the current manuscript does not permit verification. The two regularizers' ablations are contradictory between the main text and the appendix, a headline ID number differs between Table 2 and Table 5 for the same setting, and no code, hyperparameters, or repeated-seed statistics are provided. The contribution is potentially important but is not yet supported as written.

major comments (5)
  1. [§3.2.2, Finding 2, Fig. 3 vs Appendix C.3, Table 10] The ablation evidence is internally contradictory. Figure 3 reports for ViT-B/32 on ImageNet: FAR OOD 51.7 vs FCR OOD 48.5, and the main text states that 'only FCR objective cannot achieve a better OOD performance than FAR objective.' Table 10 reports the opposite ordering on the same setting: ViT-B/32 OOD FAR 48.50 vs FCR 51.67; similarly ViT-B/16 OOD FAR 58.77 vs FCR 62.66 and ResNet50 OOD FAR 44.24 vs FCR 46.06. The ID columns are also reversed (Fig. 3 FAR 78.0/FCR 78.6 vs Table 10 FAR 78.55/FCR 78.06). This is not a rounding artifact; it reverses which regularizer contributes the OOD gain. Since no code or seeds are provided, a reader cannot determine which ablation is correct. The claimed synergy of FAR and FCR in Eq. (4) is load-bearing for the central claim, so this discrepancy must be resolved and the experiments re-reported with per-seed results.
  2. [§4.1, Table 2 vs Appendix C.1.1, Table 5] The headline ViT-B/32 ID accuracy for 'Ours' is 78.98 in Table 2 but 78.75 in Table 5 for the same ImageNet fine-tuning setting, while the OOD average is 52.17 in both. This is a direct numerical inconsistency in a key reported result. The authors should identify which value is correct and ensure all tables are generated from the same runs.
  3. [§3.2.1, Eq. (2)] The approximation R_FAR ≈ (1/N) Σ |fθ(x̃_i) − f0(x̃_i)|² assumes that RandAugment-augmented ID data faithfully simulate the OOD distribution over which the functional distance should be minimized. This premise is asserted but not validated. Since FAR is the novel component, the paper should provide evidence that the augmented inputs occupy modes relevant to ImageNet-R/A/Sketch and the WILDS shifts. Concretely, the authors could vary augmentation strength and type, report per-benchmark FAR behavior, or compare against alternative simulated-OOD strategies. Without such evidence, the mechanism for OOD improvement is unverified.
  4. [Reproducibility: Eq. (4), §4.1] No λ1 or λ2 values, no RandAugment strength or number of operations, no code, no seeds, and no error bars are reported anywhere. The paper also does not state how λ1 and λ2 were selected (e.g., on ID validation, OOD validation, or manually). Many headline gains are 1–3 percentage points, so without variance or a hyperparameter sensitivity analysis the improvements cannot be assessed for statistical significance or robustness. Please release code and configuration files and report results over multiple seeds.
  5. [§3.2.3, Finding 1; Appendix B.1, Figs. 4–7] The empirical support for Finding 1 is not controlled. Perturbation magnitudes are set differently across spaces (0.0004 in parameter space vs 0.1–1.0 in feature, logit, and function spaces), and a 'function-space perturbation' is, by construction, a change in the outputs that FAR directly penalizes. The observation that function-space perturbations are less harmful is therefore partly definitional. This weakens the motivating claim. The authors should re-run with matched complexity measures, or restrict the claim to a practical statement rather than a general superiority of function space.
minor comments (5)
  1. [Throughout] Several unresolved cross-references remain: 'Section ??' appears in §3.1, §3.2.1, and after Finding 2. These need to point to actual sections.
  2. [Table 1 and References] Table 1 cites [26] for L2-SP, but Ref. [26] is the LDIFS paper ('Fine-tuning can cripple your foundation model...'). The L2-SP method is Ref. [43]. Please correct the citation.
  3. [Table 3] The FLYP row for ViT-B/32 ID metrics reports F1 = 59.76 with Recall = 30.47; this F1 is implausible given the recall value. Please verify all F1 entries in Table 3 (and Table 6).
  4. [§2, Eqs. (2)–(3)] Please clarify what fθ and f0 denote in Eqs. (2)–(3): the full classifier (image encoder plus text-embedding-based head) or only the image encoder. Table 1 and §2 use different notation.
  5. [Various] Typos and wording issues include 'Entensive validation', 'consisteny', 'appdix', 'robuts fine-tuning', and duplicate 'directions' in the Figure 2 caption. Also 'Flyp' should be 'FLYP'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central OOD results are evaluated on held-out natural distribution shifts, not on the augmented inputs used in the regularizer.

full rationale

The claimed derivation chain is not circular. The proposed objective (Eq. 4) combines cross-entropy with FAR (Eq. 2), computed on RandAugment-simulated OOD samples, and FCR (Eq. 3), a standard consistency penalty. The headline OOD numbers in Table 2 and the appendix are measured on held-out natural shifts such as ImageNet-V2, ImageNet-R, ImageNet-A, ImageNet-Sketch, WILDS-iWildCam, and WILDS-FMoW. These evaluation distributions are not the same as the augmented training samples, so the regularizer does not by construction equal the reported metric. No parameter is fitted to the OOD test sets and then renamed as a prediction. There is no load-bearing self-citation chain: the paper does not rely on the authors' own prior work to justify FAR or FCR. Finding 1 is an empirical perturbation study on actual OOD benchmarks, not a definitional equivalence, so it does not make the later evaluation circular. Two non-circular concerns remain: (a) the assumption that RandAugment-augmented ID data faithfully simulates OOD shifts is asserted rather than validated, and (b) the ablation evidence is internally contradictory—Finding 2/Figure 3 report FAR > FCR for OOD on ViT-B/32 (51.7 vs 48.5), while Table 10 reports the opposite (48.50 vs 51.67). These are correctness and reproducibility issues, not circularity, because the final evaluation is external to the training objective.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method depends on treating OOD robustness as output stability, treating augmented ID data as OOD data, and trusting standard consistency regularization. No new entities are introduced. The free parameters lambda_1, lambda_2, and RandAugment settings are unreported, which limits verification.

free parameters (3)
  • lambda_1 (weight of functional alignment regularization) = not reported
    Weights the FAR term in Eq. (4). The paper gives no value, search range, or selection procedure, so the central trade-off hyperparameter is unspecified.
  • lambda_2 (weight of functional consistency regularization) = not reported
    Weights the FCR term in Eq. (4). Like lambda_1, no value or tuning protocol is reported.
  • RandAugment strength and number of operations = not reported
    RandAugment is used to generate simulated OOD data in Eqs. (2) and (3), but the augmentation magnitude and operation count are not specified, leaving the simulated OOD distribution undefined.
assumptions (4)
  • domain assumption OOD robustness is well captured by the L2 distance between model functions under an empirical input measure
    Section 3.1 defines the function norm and Section 3.2.1 uses it as the objective. This identification of robustness with output stability is assumed, not derived.
  • domain assumption RandAugment-augmented ID data serves as a valid proxy for OOD data
    Section 3.2.1 Eq. (2) replaces the expectation over OOD data with augmented ID samples. No evidence is given that the augmentation covers the actual shift modes in the evaluation benchmarks.
  • domain assumption Consistency regularization between original and perturbed predictions improves OOD robustness
    Section 3.2.2 Eq. (3) applies a standard consistency loss and assumes it transfers to OOD generalization in the fine-tuning setting.
  • domain assumption The pretrained CLIP model is a robust anchor worth staying close to
    Throughout Section 3.2, the pretrained function is treated as the source of OOD robustness. The paper does not test whether another anchor or a learned interpolation would work better.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performance." pith.science (2026). https://pith.science/paper/LBTFFNLS

@misc{pith2026250905328,
  author       = {Pith},
  title        = {Pith review of: Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LBTFFNLS}},
  note         = {Machine review of arXiv:2509.05328}
}
read the original abstract

Robust fine-tuning aims to achieve competitive in-distribution (ID) performance while maintaining the out-of-distribution (OOD) robustness of a pre-trained model when transferring it to a downstream task. To remedy this, most robust fine-tuning methods aim to preserve the pretrained weights, features, or logits. However, we find that these methods cannot always improve OOD robustness for different model architectures. This is due to the OOD robustness requiring the model function to produce stable prediction for input information of downstream tasks, while existing methods might serve as a poor proxy for the optimization in the function space. Based on this finding, we propose a novel regularization that constrains the distance of fine-tuning and pre-trained model in the function space with the simulated OOD samples, aiming to preserve the OOD robustness of the pre-trained model. Besides, to further enhance the OOD robustness capability of the fine-tuning model, we introduce an additional consistency regularization to promote stable predictions of perturbed samples. Extensive experiments demonstrate our approach could consistently improve both downstream task ID fine-tuning performance and OOD robustness across a variety of CLIP backbones, outperforming existing regularization-based robust fine-tuning methods.

Figures

Figures reproduced from arXiv: 2509.05328 by the authors.

Figure 1
Figure 1. Performance comparison of existing regularization-based robust fine-tuning methods on [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Robustness analysis of optimization in parameter/feature/logit spaces and proposed function [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Relative performance gains over the pretrained model under full-parameter fine-tuning (FT) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Variation in data loss across different perturbation magnitudes in parameter, feature, [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Variation in accuracy across different perturbation magnitudes in parameter, feature, [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Impact of fixed perturbations magnitude, [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Impact of fixed perturbations magnitude m = 0.0004 in parameter, feature, logit, and function spaces, on the accuracy across five benchmarks (ImageNet, ImageNetV2, ImageNet-A, ImageNet-R, and ImageNet-Sketch). Function-space perturbations induce the best accuracy acros…
Figure 8
Figure 8. Figure 8: Relative performance gains over the pretrained model under full-parameter fine-tuning (FT) [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Ensemble results on the ViT-B/32, ViT-B/16 and ResNet-50. backbone obtained by [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 37 canonical work pages

  1. [1]

    Invariance principle meets information bottleneck for out-of-distribution generalization

    Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean-Christophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. Advances in Neural Information Processing Systems , 34:3438–3450, 2021

    work page 2021

  2. [2]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in neural information processing systems, volume 32, 2019

    work page 2019

  3. [3]

    Measuring and regularizing networks in function space

    Ari S Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. In International Conference on Learning Representations, 2018

    work page 2018

  4. [4]

    Benjamin, David Rolnick, and Konrad P

    Ari S. Benjamin, David Rolnick, and Konrad P. Kording. Measuring and regularizing networks in function space. In International Conference on Learning Representations (ICLR), 2019

    work page 2019

  5. [5]

    A kernel perspective for regularizing deep neural networks

    Alberto Bietti and Julien Mairal. A kernel perspective for regularizing deep neural networks. In Advances in Neural Information Processing Systems, volume 32, pages 12869–12879, 2019

    work page 2019

  6. [6]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  7. [7]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners

    work page

  8. [8]

    Burt, Sebastian W

    David R. Burt, Sebastian W. Ober, Adrià Garriga-Alonso, and Mark van der Wilk. Understanding variational inference in function-space. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), volume 108, pages 265–275. PMLR, 2020

    work page 2020

Show all 46 references
  1. [9]

    Multi-dimensional graph linear canonical transform and its application

    Jian-Yi Chen and Bing-Zhao Li. Multi-dimensional graph linear canonical transform and its application. Digital Signal Processing, 163:105222, 2025

    2025

  2. [10]

    Zonghao Chen, Xupeng Shi, Tim G. J. Rudner, Qixuan Feng, Weizhong Zhang, and Tong Zhang. A neural tangent kernel perspective on function-space regularization in neural networks. In Proceedings of the NeurIPS 2022 Workshop on Optimization for Machine Learning (OPT), 2022

    2022

  3. [11]

    Randaugment: Practical automated data augmentation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020

    2020

  4. [12]

    Improved regularization of convolutional neural networks with cutout

    Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. In arXiv preprint arXiv:1708.04552, 2017

    2017 arXiv

  5. [13]

    Sharpness-aware mini- mization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2021

    2010 arXiv

  6. [14]

    Domain-adversarial training of neural networks

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1–35, 2016

    2016

  7. [15]

    Goodfellow, Jonathon Shlens, and Christian Szegedy

    Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. arXiv preprint arXiv:1412.6572, 2015

    2015 arXiv

  8. [16]

    Finetune like you pretrain: Improved finetuning of zero-shot vision models

    Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19338–19347, 2023

    2023

  9. [17]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262–15271, 2021. 24

    2021

  10. [18]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwi ´n´ska, et al

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwi ´n´ska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of S...

    2017

  11. [19]

    Fine- tuning can distort pretrained features and underperform out-of-distribution

    Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022

    2022

  12. [20]

    Invariant risk minimization is a total variation model

    Zhaorong Lai, Weiwen Wang, Cheng Li, and Yangyu Zhang. Invariant risk minimization is a total variation model. In International Conference on Machine Learning (ICML), 2024

    2024

  13. [21]

    Temporal ensembling for semi-supervised learning

    Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Interna- tional Conference on Learning Representations (ICLR), 2017

    2017

  14. [22]

    Surgical fine-tuning improves adaptation to distribution shifts

    Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210.11466, 2022

    2022 arXiv

  15. [23]

    Towards out-of-distribution generalization: A survey

    Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624, 2021

    2021 arXiv

  16. [24]

    Context-aware robust fine-tuning

    Xiaofeng Mao, Yufeng Chen, Xiaojun Jia, Rong Zhang, Hui Xue, and Zhao Li. Context-aware robust fine-tuning. International Journal of Computer Vision, 132(5):1685–1700, 2024

    2024

  17. [25]

    Virtual adversarial training: a regularization method for supervised and semi-supervised learning

    Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979–1993, 2018

    1979

  18. [26]

    Fine-tuning can cripple your foundation model; preserving features may be the solution

    Jishnu Mukhoti, Yarin Gal, Philip Torr, and Puneet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. Transactions on Machine Learning Research, 2024

    2024

  19. [27]

    Lipsum-ft: Robust fine-tuning of zero-shot models using random text guidance

    Giung Nam, Byeongho Heo, and Juho Lee. Lipsum-ft: Robust fine-tuning of zero-shot models using random text guidance. In International Conference on Learning Representations, 2024

    2024

  20. [28]

    Dawin: Training- free dynamic weight interpolation for robust adaptation, 2024

    Changdae Oh, Yixuan Li, Kyungwoo Song, Sangdoo Yun, and Dongyoon Han. Dawin: Training- free dynamic weight interpolation for robust adaptation, 2024. ICLR 2025 cameraready

    2024

  21. [29]

    Towards calibrated robust fine-tuning of vision-language models

    Changdae Oh, Hyesu Lim, Mijoo Kim, Dongyoon Han, Sangdoo Yun, Jaegul Choo, Alexander Hauptmann, Zhi-Qi Cheng, and Kyungwoo Song. Towards calibrated robust fine-tuning of vision-language models. In Advances in Neural Information Processing Systems, volume 37, pages 12677–12707, 2024

    2024

  22. [30]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  23. [31]

    Tim G. J. Rudner, Sanyam Kapoor, Shikai Qiu, and Andrew Gordon Wilson. Function-space regularization in neural networks: A probabilistic perspective. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning R...

    2023

  24. [32]

    Test-time training with self-supervision for generalization under distribution shifts

    Anusha Saeed, Ricardo German, Hyung Jin Chun, and Soheil Feizi. Test-time training with self-supervision for generalization under distribution shifts. arXiv preprint arXiv:1909.13231, 2020

    1909 arXiv

  25. [33]

    Distributionally robust neural networks for group shifts

    Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts. In International Conference on Learning Representations (ICLR), 2020

    2020

  26. [34]

    A survey on image data augmentation for deep learning

    Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1–48, 2019. 25

    2019

  27. [35]

    Functional variational bayesian neural networks

    Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational bayesian neural networks. In International Conference on Learning Representations (ICLR), 2019

    2019

  28. [36]

    Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

    Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems (NeurIPS), pages 1195–1204, 2017

    2017

  29. [37]

    Trainable projected gradient method for robust fine-tuning

    Junjiao Tian, Zecheng He, Xiaoliang Dai, Chih-Yao Ma, Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7836–7845, 2023

    2023

  30. [38]

    Fast trainable projection for robust fine-tuning

    Junjiao Tian, Yen-Cheng Liu, James S Smith, and Zsolt Kira. Fast trainable projection for robust fine-tuning. In Advances in Neural Information Processing Systems, volume 36, pages 11374–11393, 2023

    2023

  31. [39]

    Titsias, Jonathan Schwarz, Alexander de G

    Michalis K. Titsias, Jonathan Schwarz, Alexander de G. Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with gaussian processes. In International Conference on Learning Representations (ICLR), 2020

    2020

  32. [40]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without i...

    2022

  33. [41]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision...

    2022

  34. [42]

    Unsupervised data augmentation for consistency training

    Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems (NeurIPS), 33:6256–6268, 2020

    2020

  35. [43]

    Explicit inductive bias for transfer learning with convolutional networks

    LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International conference on machine learning, pages 2825–

  36. [44]

    Sample efficiency of data augmentation consistency regularization

    Shuo Yang, Yijun Dong, Rachel Ward, Inderjit S Dhillon, Sujay Sanghavi, and Qi Lei. Sample efficiency of data augmentation consistency regularization. In International Conference on Artificial Intelligence and Statistics, pages 3825–3853. PMLR, 2023

    2023

  37. [45]

    Learning to generate novel domains for domain generalization

    Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In ECCV, pages 561–578. Springer, 2020

    2020

  38. [46]

    Domain generalization with mixstyle

    Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021. 26

    2021

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.