Pith. sign in

REVIEW 3 major objections 5 minor 82 references

Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Structure-aware regularizer keeps fine-tuned models from forgetting pretrained concepts.

desk verdict Broad, well-executed regularization study with a real if not fully deconfounded headline claim; worth sending to review with requests for trade-off curves and quantitative OTDD evidence. read the letter →

arxiv 2505.24088 v1 pith:56BENPYJ submitted 2025-05-30 cs.LG cs.CV

Chen Huang , Skyler Seto , Hadi Pouransari , Mehrdad Farajtabar , Raviteja Vemulapalli , Fartash Faghri , Oncel Tuzel , Barry-John Theobald
show 1 more author
Josh Susskind
This is my paper
classification cs.LGcs.CV
keywords conceptforgettingfeaturedistributionalignmentproxylearningfine-tuningvisionfoundationmodelsnearestneighborgraphoptimaltransportdatasetdistanceregularization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Proxy-FDA is a regularization method for fine-tuning vision foundation models that aims to stop concept forgetting, the loss of a model's ability to recognize concepts outside the fine-tuning task. The paper argues that point-wise matching of features or weights is too strong a constraint because it ignores the neighborhood structure in feature space that encodes rich knowledge such as shared attributes. To fix this, Proxy-FDA computes a nearest-neighbor graph from the frozen pretrained model and regularizes the fine-tuned features to preserve that graph, pulling neighbors together and pushing non-neighbors apart. It then adds a dynamic proxy generator that synthesizes additional features to augment this alignment, which is especially helpful when fine-tuning data are scarce. Experiments on ten classification datasets show that Proxy-FDA significantly reduces forgetting—with average transfer metric $\Delta$LP rising from -4.37 under naive fine-tuning to 1.54—while keeping downstream accuracy comparable, and the paper demonstrates a strong correlation between forgetting and a structure-aware distance metric, OTDD.

What carries the argument

The central object is the nearest-neighbor graph in the pretrained feature space, computed within each hard-mined batch of the fine-tuning dataset. For each feature point, the graph records its k neighbors and their cosine similarities to that point. FDA transfers these indices and similarities to the fine-tuned features, using a Sigmoid loss to pull neighbors together and push non-neighbors apart, weighted by the pretrained similarities. Proxy-FDA extends this by learning a small generator—one attention layer and two convolutional layers—that produces synthetic features, called proxies, from the positive and negative neighborhoods of each sample. These proxies augment the real features in the alignment loss, increasing data diversity and refining the neighborhood boundary, which is especially useful in few-shot settings. The machinery is structure-aware because it regularizes the local geometry of the feature space rather than individual feature points.

What would settle it

On a downstream classification task constructed so that the optimal feature representation requires grouping classes differently from the pretrained neighborhoods (for example, merging classes that are far apart in the pretrained feature space), Proxy-FDA should measurably reduce downstream accuracy compared to unregularized fine-tuning, indicating that the regularization suppresses necessary adaptation.

Watch

Extended reading notes

Core claim

The central claim is that structure-aware feature regularization, implemented by aligning local nearest-neighbor graphs between the pretrained and fine-tuned feature spaces, reduces concept forgetting during fine-tuning far more effectively than point-wise feature or weight matching, without hurting downstream performance. The paper introduces FDA, which transfers neighbor indices and cosine similarities from the frozen model to the fine-tuned model using a Sigmoid loss, and Proxy-FDA, which learns to generate dynamic synthetic features, or proxies, from the positive and negative neighborhoods of each sample to improve alignment in data-poor regimes. Empirically, with CLIP ViT-B/32 end-to-end fine-tuned on ten datasets, Proxy-FDA achieves a mean $\Delta$LP of 1.54 (higher is better, positive indicating forward transfer) versus 0.29 for LDIFS and -4.37 for naive fine-tuning, while keeping downstream accuracy essentially unchanged. The paper further demonstrates that concept forgetting correlates more strongly with OTDD, a distributional distance that respects local structure, than with L2 feature distance, supporting the claim that structural alignment, not point-wise closeness, is the key to preserving knowledge.

Load-bearing premise

The load-bearing assumption is that the k-nearest-neighbor graph computed from the frozen pretrained model over hard-mined batches is the structure that should be preserved, and that restricting its change does not conflict with the downstream task's need to reorganize features.

Editorial extensions

If this is right

  • If Proxy-FDA is correct, structure-aware feature regularization becomes a more effective tool than point-wise feature or weight matching for preserving pretrained knowledge during fine-tuning.
  • Concept forgetting during fine-tuning can be predicted by a structure-aware distributional distance (OTDD), suggesting that alignment quality should be measured by neighborhood structure rather than point-wise L2 distance.
  • Proxy-FDA can be applied as a plug-in regularizer to various fine-tuning methods, including prompt tuning, continual fine-tuning, and vision-language tasks, extending its benefits beyond standard classification.
  • The dynamic proxy generator provides a data-efficient way to improve alignment without external data, which is particularly valuable in few-shot settings where fine-tuning data are scarce.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The correlation between forgetting and OTDD suggests that directly minimizing OTDD could be an alternative or complementary regularizer to Proxy-FDA, though the paper does not attempt this.
  • If preserving pretrained neighborhoods is beneficial for unseen concepts, the same principle might extend to other forms of distribution shift, such as domain generalization or OOD robustness, beyond concept forgetting.
  • The assumption that pretrained neighborhoods should be preserved is untested in settings where the downstream task requires reorganizing features; a task that conflicts with the pretrained graph could expose a limit of the method.
  • The proxy generator's ability to synthesize features that encode unseen concepts might be harnessed for data augmentation in other self-supervised or semi-supervised settings, beyond fine-tuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Proxy-FDA, a regularization method for fine-tuning vision foundation models that aims to reduce concept forgetting by preserving the local neighborhood structure of the pre-trained feature space. For each sample, the fine-tuned model is regularized so that neighbors identified by the frozen pre-trained model remain neighbors, and non-neighbors are pushed apart, using a sigmoid-based contrastive loss (FDA). A lightweight proxy generator synthesizes additional positive and negative feature points, with similarity estimates, to augment the regularization, particularly in data-limited regimes. Experiments are reported for end-to-end fine-tuning of CLIP and other backbones on 10 datasets, few-shot prompt tuning with several baselines, continual fine-tuning, image captioning/VQA, and knowledge distillation. The central claim is that structure-wise alignment reduces forgetting more effectively than point-wise feature/weight matching, and that a distributional distance (OTDD) correlates with forgetting better than L2 feature distance.

Significance. If the results hold, Proxy-FDA is a broadly applicable and computationally light regularizer for robust fine-tuning, with consistent gains across architectures (CLIP, FLAVA, DINOv2, MAE), tasks, and settings. The paper is strong in scope: it covers end-to-end, few-shot, continual, captioning/VQA, and distillation settings, and includes ablations, hyperparameter sensitivity, diversity metrics, and compute-time overhead. The proxy generator is compact (23.6k parameters) and the method adds only 17--21% training time. However, the central comparison against point-wise regularization is confounded by regularization strength, and the OTDD correlation claim is asserted without a quantitative statistic. These issues need to be addressed before the structural contribution is fully established.

major comments (3)
  1. [Section 4.1, Table 1, Eq. (5), Appendix D] The central claim that structure-wise Proxy-FDA reduces forgetting better than point-wise matching rests on a single operating point per method: Proxy-FDA achieves mean ΔLP=1.54 with mean ALP=91.82, while LDIFS achieves ΔLP=0.29 with ALP=91.86. Because λ in Eq. (5) is tuned separately per dataset on a held-out set (Appendix D), a larger effective λ for Proxy-FDA could produce exactly this pattern: lower forgetting at slightly lower task accuracy. To attribute the improvement to the neighborhood-graph structure, report the ΔLP–ALP trade-off by sweeping λ for LDIFS, L2SP, and Proxy-FDA on the 10 datasets, or provide a matched-ALP comparison where methods are selected to have comparable downstream accuracy.
  2. [Section 4.1, Fig. 3] The abstract and Section 4.1 claim a 'strong correlation' between concept forgetting and OTDD, but the support is visual inspection of training trajectories for EuroSAT fine-tuning only. No correlation coefficient, confidence interval, or cross-dataset summary is given. Because this correlation is a stated contribution and motivates the structure-wise design, please compute a quantitative statistic (e.g., Spearman or Pearson correlation between final ΔLP and OTDD across all 10 datasets and all methods) and report its uncertainty.
  3. [Tables 1, 2, and 3] Most main results are reported without standard deviations or significance tests, and the gap between Proxy-FDA and FDA in Table 1 is small (mean ΔLP 1.54 vs 1.39; mean ALP 91.82 vs 91.86). Without repeated-seed variability or a significance test, the claim that the proxy component 'consistently' improves over FDA is not yet supported. Report mean ± std over at least three seeds for the main end-to-end, few-shot, and continual experiments, and note significance where relevant.
minor comments (5)
  1. [Section 3.2 and Appendix B] Clarify the optimization of the proxy generator: whether Eq. (3–4) is the only objective for the proxy generator or whether gradients from Eq. (5) also flow into it, and how it is scheduled relative to the main fine-tuning updates.
  2. [Appendix G] Fix the typos 'As metioned' and 'epseically'; also ensure that 'Table 5 in Appendix' is cross-referenced to the correct appendix location.
  3. [Figure 3] The caption says the metrics are computed during fine-tuning 'on EuroSAT', but the columns are labeled with multiple datasets; please clarify that each column corresponds to a target dataset evaluated during EuroSAT fine-tuning.
  4. [Section 4.1, Table 5] The main text says 'Table 5 in Appendix' but Table 5 appears in Appendix F; please check the cross-references.
  5. [Appendix D] Consider reporting the per-dataset λ values or a reproducibility statement, since the method has several hyperparameters (K, s, α, λ, τ, b) and the tuned values are not listed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the regularization losses, evaluation metrics, and baselines are independently defined and empirically compared.

full rationale

The paper contains no derivation that reduces a predicted quantity to a fitted input by construction. The FDA loss (Eq. 2) and Proxy-FDA loss (Eq. 5) are explicit regularization objectives that penalize changes in a kNN graph transferred from the frozen pre-trained model; the reported forgetting metric ΔLP is measured with held-out linear-probe accuracy on external datasets, not by the regularization loss itself. Hyperparameters such as λ, K, and s are tuned on validation data per dataset, which is standard empirical ML practice and, while it can confound comparisons of regularization strength (e.g., against LDIFS in Table 1), does not make the ΔLP values algebraic reductions of the tuning procedure. The OTDD correlation in Fig. 3 is used as a post hoc diagnostic and is not the optimized objective; the claim that OTDD is more correlated with forgetting than L2 distance is an empirical observation, not a definitional identity. The only apparent self-citation, OGEN (Zang et al., 2024), appears solely as a related-work baseline and comparison method, and no load-bearing assumption or uniqueness theorem is imported from the authors' prior work. All central comparisons are evaluated against external baselines (LDIFS, L2SP, PromptSRC, CLIPood, etc.), so no step reduces to the paper's inputs by construction.

Assumptions & free parameters 5 free parameters · 7 assumptions · 1 invented entities

The central claim is empirical; the method depends on several hyperparameters and domain assumptions about feature neighborhood structure. No new physical entities are introduced; the learned proxies are a technical component with internal but not external validation.

free parameters (5)
  • K (neighborhood size) = per dataset from {n, 2n, 3n, 4n}
    Chosen per dataset on validation AH (Appendix D), directly affects which structures are preserved.
  • s (proxy count scalar) = 0.4 default
    Controls number of generated proxies; sensitivity shown in Fig. 7(d).
  • α (variance loss weight) = 5
    Fixed; no meaningful tuning found (Appendix D).
  • λ (regularization weight) = tuned per dataset
    Balances task loss and Proxy-FDA loss; tuned on validation (Sec 3, Appendix D).
  • τ, b (sigmoid temperature and bias) = learned during training
    Initialized as in Zhai et al. 2023; optimized as part of the loss.
assumptions (7)
  • domain assumption Pre-trained vision foundation models encode rich concepts in the local structure of feature neighborhoods.
    Central motivation in Sec 1 and Fig. 5, assumed throughout.
  • domain assumption Preserving the kNN graph from the pre-trained feature space is a sufficient proxy for preserving pre-trained knowledge.
    Sec 3.1 defines FDA as transfer of neighbor indices and similarities; no formal equivalence is shown.
  • domain assumption Hard class mining produces batches whose local neighborhoods are meaningful for alignment.
    Appendix A; ablation shows importance, but it is a heuristic.
  • domain assumption Learned proxies lie on the true feature manifold and are diverse.
    Sec 3.2 and Appendix B; enforced only through the variance loss and manifold-matching loss, no external validation.
  • domain assumption OTDD with K-means pseudolabels accurately measures structure-aware distribution shift relevant to forgetting.
    Appendix C; used as explanation, not optimized.
  • standard math Sigmoid loss (Zhai et al. 2023) is noise-resistant and appropriate for variable positives and negatives.
    Used in Eq. 2; standard result from cited paper.
  • standard math Softmax-normalized pooling yields convex combinations of features.
    Appendix B, Eq. 6-7; simple math.
invented entities (1)
  • Learned proxies (synthetic feature points)
    purpose: Augment the FDA loss with diverse positive and negative features to increase data diversity, especially in few-shot settings.
    Generated by an attention+convolution network trained online with a manifold-matching and variance loss (Sec 3.2, Appendix B). The paper provides internal ablations showing they help beyond larger batches, but no external falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting." pith.science (2026). https://pith.science/paper/56BENPYJ

@misc{pith2026250524088,
  author       = {Pith},
  title        = {Pith review of: Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/56BENPYJ}},
  note         = {Machine review of arXiv:2505.24088}
}
read the original abstract

Vision foundation models pre-trained on massive data encode rich representations of real-world concepts, which can be adapted to downstream tasks by fine-tuning. However, fine-tuning foundation models on one task often leads to the issue of concept forgetting on other tasks. Recent methods of robust fine-tuning aim to mitigate forgetting of prior knowledge without affecting the fine-tuning performance. Knowledge is often preserved by matching the original and fine-tuned model weights or feature pairs. However, such point-wise matching can be too strong, without explicit awareness of the feature neighborhood structures that encode rich knowledge as well. We propose a novel regularization method Proxy-FDA that explicitly preserves the structural knowledge in feature space. Proxy-FDA performs Feature Distribution Alignment (using nearest neighbor graphs) between the pre-trained and fine-tuned feature spaces, and the alignment is further improved by informative proxies that are generated dynamically to increase data diversity. Experiments show that Proxy-FDA significantly reduces concept forgetting during fine-tuning, and we find a strong correlation between forgetting and a distributional distance metric (in comparison to L2 distance). We further demonstrate Proxy-FDA's benefits in various fine-tuning settings (end-to-end, few-shot and continual tuning) and across different tasks like image classification, captioning and VQA.

Figures

Figures reproduced from arXiv: 2505.24088 by the authors.

Figure 1
Figure 1. (a) Motivation: alleviating concept forgetting during model fine-tuning by a novel feature regularization method– Proxy-FDA (Proxy-based Feature Distribution Alignment). (b) Proxy-FDA aligns the pre-trained and fine-tuned feature distributions by their local neighborhood structures, which is further aided by proxies (i.e., synthetic features). We show Proxy-FDA indeed preserves the rich knowledge in local feature ne… view at source ↗
Figure 2
Figure 2. Batch construction and nearest neighbor graph transfer for our [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Three metrics computed over the course of model fine-tuning (CLIP ViT-B/32) on [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: (a-b) The average ∆New with varying number of shots per class for prompt tuning in the base-to-new setting. FDA achieves higher gains over the baselines in low-data regime, and our proxy learning further improves data efficiency. (c) PromptSRC+Proxy-FDA scales better w…
Figure 5
Figure 5. Figure 5: t-SNE visualization of the local feature neighborhood (circled) on ImageNet for the pre-trained CLIP ViT-B/16 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 7
Figure 7. Figure 7: Sensitivity analysis for hyper-parameters: (a) batch size B, (b) neighborhood size K that is fixed across datasets, (c) optimal K per dataset, and (d) scalar s that decides the percent number of generated proxies compared to that of real samples. Analysis is performed …
Figure 8
Figure 8. Figure 8: includes ablation studies on the key components of Proxy-FDA, in the few-shot prompt tuning setting. Batch sampling strategy. We start with comparing the default hard class mining method with random class sampling. Their considerable performance difference shows that h…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

82 extracted references · 58 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    NoCaps : novel object captioning at scale

    Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., and Anderson, P. NoCaps : novel object captioning at scale. In ICCV, 2019

    2019

  3. [3]

    and Fusi, N

    Alvarez-Melis, D. and Fusi, N. Geometric dataset distances via optimal transport. In NeurIPS, 2020

    work page 2020

  4. [4]

    Bossard, L., Guillaumin, M., and Gool, L. V. Food-101--mining discriminative components with random forests. In ECCV, 2014

    work page 2014

  5. [5]

    Darkrank: Accelerating deep metric learning via cross sample similarities transfer

    Chen, Y., Wang, N., and Zhang, Z. Darkrank: Accelerating deep metric learning via cross sample similarities transfer. In AAAI, 2018

    work page 2018

  6. [6]

    Remote sensing image scene classification: Benchmark and state of the art

    Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105: 0 1865--1883, 2017

    work page 2017

  7. [7]

    Describing textures in the wild

    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In CVPR, 2014

    2014

  8. [8]

    ImageNet : A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet : A large-scale hierarchical image database. In CVPR, 2009

    work page 2009

Show all 82 references
  1. [9]

    Vos: Learning what you don’t know by virtual outlier synthesis

    Du, X., Wang, Z., Cai, M., and Li, Y. Vos: Learning what you don’t know by virtual outlier synthesis. In ICLR, 2022

    2022

  2. [10]

    Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories

    Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR workshop, 2004

    2004

  3. [11]

    CLIP-Adapter : Better vision-language models with feature adapters

    Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. CLIP-Adapter : Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544, 2021

    2021 arXiv

  4. [12]

    Finetune like you pretrain: Improved finetuning of zero-shot vision models

    Goyal, S., Kumar, A., Garg, S., Kolter, Z., and Raghunathan, A. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In CVPR, 2023

    2023

  5. [13]

    Making the V in VQA matter: Elevating the role of image understanding in V isual Q uestion A nswering

    Goyal, Y., Khot, T., Summers - Stay, D., Batra, D., and Parikh, D. Making the V in VQA matter: Elevating the role of image understanding in V isual Q uestion A nswering. In CVPR, 2017

    2017

  6. [14]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016

    2016

  7. [15]

    Masked autoencoders are scalable vision learners

    He, K., Chen, X., Xie, S., Li, Y., Dollar, P., and Girshick, R. Masked autoencoders are scalable vision learners. In CVPR, 2022

    2022

  8. [16]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 2019

    2019

  9. [17]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021 a

    2021

  10. [18]

    Natural adversarial examples

    Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In CVPR, 2021 b

    2021

  11. [19]

    CLIPS core: A reference-free evaluation metric for image captioning

    Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. CLIPS core: A reference-free evaluation metric for image captioning. In EMNLP, 2021

    2021

  12. [20]

    Lifelong learning via progressive distillation and retrospection

    Hou, S., Pan, X., Change Loy, C., Wang, Z., and Lin, D. Lifelong learning via progressive distillation and retrospection. In ECCV, 2018

    2018

  13. [21]

    G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H

    Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. MobileNets : Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017

    2017 arXiv

  14. [22]

    Knowledge distillation from a stronger teacher

    Huang, T., You, S., Wang, F., Qian, C., and Xu, C. Knowledge distillation from a stronger teacher. In NeurIPS, 2022

    2022

  15. [23]

    Visual prompt tuning

    Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In ECCV, 2022

    2022

  16. [24]

    Less-forgetting learning in deep neural networks

    Jung, H., Ju, J., Jung, M., and Kim, J. Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122, 2016

    2016 arXiv

  17. [25]

    U., Rasheed, H., Maaz, M., Khan, S., and Khan, F

    Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. MaPLe : Multi-modal prompt learning. In CVPR, 2023 a

    2023

  18. [26]

    U., Wasim, S

    Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M.-H., and Khan, F. S. Self-regulating prompts: Foundational model adaptation without forgetting. In ICCV, 2023 b

    2023

  19. [27]

    U., Ferjad, M., Muzzamal, N., Gool, L

    khattak, M. U., Ferjad, M., Muzzamal, N., Gool, L. V., and Tombari, F. Learning to prompt with text only supervision for vision-language models. arXiv preprint arXiv:2401.02418, 2024

    2024 arXiv

  20. [28]

    Proxy anchor loss for deep metric learning

    Kim, S., Kim, D., Cho, M., and Kwak, S. Proxy anchor loss for deep metric learning. In CVPR, 2020

    2020

  21. [29]

    3d object representations for fine-grained categorization

    Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In ICCV workshops, 2013

    2013

  22. [30]

    Learning multiple layers of features from tiny images

    Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  23. [31]

    M., Ma, T., and Liang, P

    Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. In ICLR, 2022

    2022

  24. [32]

    G., Courville, A., and Ballas, N

    Lavoie, S., Kirichenko, P., Ibrahim, M., Assran, M., Wildon, A. G., Courville, A., and Ballas, N. Modeling caption diversity in contrastive vision-language pretraining. arXiv preprint arXiv:2405.00740, 2024

    2024 arXiv

  25. [33]

    Mnist handwritten digit database

    LeCun, Y., Cortes, C., and Burges, C. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010

    2010

  26. [34]

    Explicit inductive bias for transfer learning with convolutional networks

    Li, X., Grandvalet, Y., and Davoine, F. Explicit inductive bias for transfer learning with convolutional networks. In ICML, 2018

    2018

  27. [35]

    and Hoiem, D

    Li, Z. and Hoiem, D. Learning without forgetting. TPAMI, 2017

    2017

  28. [36]

    J., Hays, J., Perona, P., Ramanan, D., Doll \' a r, P., and Zitnick, C

    Lin, T., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Doll \' a r, P., and Zitnick, C. L. Microsoft COCO: common objects in context. In ECCV, 2014

    2014

  29. [37]

    Fine-grained visual classification of aircraft

    Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013

    2013 arXiv

  30. [38]

    Linearly mapping from image to text space

    Merullo, J., Castricato, L., Eickhoff, C., and Pavlick, E. Linearly mapping from image to text space. In ICLR, 2023

    2023

  31. [39]

    Information theoretic representation distillation

    Miles, R., Lopez-Rodriguez, A., and Mikolajczyk, K. Information theoretic representation distillation. In BMVC, 2022

    2022

  32. [40]

    No fuss distance metric learning using proxies

    Movshovitz-Attias, Y., Toshev, A., Leung, T., Ioffe, S., and Singh, S. No fuss distance metric learning using proxies. In ICCV, 2017

    2017

  33. [41]

    Mukhoti, J., Gal, Y., Torr, P., and Dokania, P. K. Fine-tuning can cripple your foundation model; preserving features may be the solution. TMLR, 2024. ISSN 2835-8856

    2024

  34. [42]

    Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop, 2011

    2011

  35. [43]

    and Zisserman, A

    Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. In ICVGIP, 2008

    2008

  36. [44]

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Maira...

    2024

  37. [45]

    Relational knowledge distillation

    Park, W., Kim, D., Lu, Y., and Cho, M. Relational knowledge distillation. In CVPR, 2019

    2019

  38. [46]

    M., Vedaldi, A., Zisserman, A., and Jawahar, C

    Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In CVPR, 2012

    2012

  39. [47]

    and Tefas, A

    Passalis, N. and Tefas, A. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018

    2018

  40. [48]

    Correlation congruence for knowledge distillation

    Peng, B., Jin, X., li, D., Zhou, S., Wu, Y., Liu, J., Zhang, Z., and Liu, Y. Correlation congruence for knowledge distillation. In ICCV, 2019

    2019

  41. [49]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  42. [50]

    Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In CVPR, 2017

    2017

  43. [51]

    Do imagenet classifiers generalize to imagenet? In ICML, 2019

    Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In ICML, 2019

    2019

  44. [52]

    Non-isotropy regularization for proxy-based deep metric learning

    Roth, K., Vinyals, O., and Akata, Z. Non-isotropy regularization for proxy-based deep metric learning. In CVPR, 2022

    2022

  45. [53]

    CLIPood : Generalizing clip to out-of-distributions

    Shu, Y., Guo, X., Wu, J., Wang, X., Wang, J., and Long, M. CLIPood : Generalizing clip to out-of-distributions. In ICML, 2023

    2023

  46. [54]

    FLAVA: A foundational language and vision alignment model

    Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. FLAVA: A foundational language and vision alignment model. In CVPR, 2022

    2022

  47. [55]

    S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z

    Smith, J. S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In CVPR, 2023

    2023

  48. [56]

    FD -align: Feature discrimination alignment for fine-tuning pre-trained models in few-shot learning

    Song, K., Ma, H., Zou, B., Zhang, H., and Huang, W. FD -align: Feature discrimination alignment for fine-tuning pre-trained models in few-shot learning. In NeurIPS, 2023

    2023

  49. [57]

    R., and Shah, M

    Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012

    2012 arXiv

  50. [58]

    Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32: 0 323--332, 2012

    2012

  51. [59]

    Non-parametric outlier synthesis

    Tao, L., Du, X., Zhu, J., and Li, Y. Non-parametric outlier synthesis. In ICLR, 2023

    2023

  52. [60]

    Clip model is an efficient continual learner

    Thengane, V., Khan, S., Hayat, M., and Khan, F. Clip model is an efficient continual learner. arXiv preprint arXiv:2210.03114, 2022

    2022 arXiv

  53. [61]

    ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models

    Tian, X., Zou, S., Yang, Z., and Zhang, J. ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models . In CVPR, 2024

    2024

  54. [62]

    and Mori, G

    Tung, F. and Mori, G. Similarity-preserving knowledge distillation. In ICCV, 2019

    2019

  55. [63]

    L., and Parikh, D

    Vedantam, R., Zitnick, C. L., and Parikh, D. CIDEr : Consensus-based image description evaluation. In CVPR, 2015

    2015

  56. [64]

    Manifold mixup: Better representations by interpolating hidden states

    Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D., and Bengio, Y. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019

    2019

  57. [65]

    Optimal Transport: Old and New

    Villani, C. Optimal Transport: Old and New. Springer Berlin Heidelberg, 2008. ISBN 9783540710509

    2008

  58. [66]

    Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019

    2019

  59. [67]

    and Yoon, K.-J

    Wang, L. and Yoon, K.-J. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. TPAMI, 44: 0 3048--3068, 2021

    2021

  60. [68]

    and Deng, W

    Wang, M. and Deng, W. Deep visual domain adaptation: A survey. Neurocomput., 312 0 (C): 0 135–153, 2018

    2018

  61. [69]

    Improving knowledge distillation via regularizing feature norm and direction

    Wang, Y., Cheng, L., Duan, M., Wang, Y., Feng, Z., and Kong, S. Improving knowledge distillation via regularizing feature norm and direction. arXiv preprint arXiv:2305.17007, 2023

    2023 arXiv

  62. [70]

    Dualprompt: Complementary prompting for rehearsal-free continual learning

    Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, 2022 a

    2022

  63. [71]

    Learning to prompt for continual learning

    Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In CVPR, 2022 b

    2022

  64. [72]

    Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A

    Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In IC...

    2022

  65. [73]

    W., Li, M., Kornblith, S., Roelofs, R., Gontijo-Lopes, R., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L

    Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Gontijo-Lopes, R., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L. Robust fine-tuning of zero-shot models. In CVPR, 2022 b

    2022

  66. [74]

    A., Oliva, A., and Torralba, A

    Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010

    2010

  67. [75]

    M., and Huang, C

    Zang, Y., Goh, H., Susskind, J. M., and Huang, C. Overcoming the pitfalls of vision-language model finetuning for OOD generalization. In ICLR, 2024

    2024

  68. [76]

    Sigmoid loss for language image pre-training

    Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training. In ICCV, 2023

    2023

  69. [77]

    SLCA : Slow learner with classifier alignment for continual learning on a pre-trained model

    Zhang, G., Wang, L., Kang, G., Chen, L., and Wei, Y. SLCA : Slow learner with classifier alignment for continual learning on a pre-trained model. In ICCV, 2023

    2023

  70. [78]

    and Yang, E.-H

    Zheng, K. and Yang, E.-H. Knowledge distillation based on transformed teacher matching. In ICLR, 2024

    2024

  71. [79]

    Preventing zero-shot transfer degradation in continual learning of vision-language models

    Zheng, Z., Ma, M., Wang, K., Qin, Z., Yue, X., and You, Y. Preventing zero-shot transfer degradation in continual learning of vision-language models. In ICCV, 2023

    2023

  72. [80]

    C., and Liu, Z

    Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. IJCV, 2022 a

    2022

  73. [81]

    C., and Liu, Z

    Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. In CVPR, 2022 b

    2022

  74. [82]

    Contrastive neighborhood alignment

    Zhu, P., Cai, Z., Xiong, Y., Tu, Z., Goncalves, L., Mahadevan, V., and Soatto, S. Contrastive neighborhood alignment. arXiv preprint arXiv:2201.01922, 2022

    2022 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.