Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

GLAD: Generalizable Tuning for Vision-Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GLAD claims that a gradient-regularized LoRA tuning of CLIP beats prompt-based methods in few-shot generalization.

desk verdict A useful, simple recipe for CLIP few-shot tuning, but the central ablation is confounded by an undefined KL regularizer; needs a revision before the gains can be trusted. read the letter →

arxiv 2507.13089 v1 pith:OBAVFK3J submitted 2025-07-17 cs.CV

Yuqi Peng , Pengfei Wang , Jianzhuang Liu , Shifeng Chen This is my paper
classification cs.CV
keywords LoRACLIPfew-shotlearningvision-languagemodelsprompttuningsharpness-awareminimizationgradientregularizationbase-to-novelgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that Low-Rank Adaptation (LoRA) of a frozen CLIP model can replace prompt tuning for few-shot classification, and that a specific gradient-based regularizer plus a light image-conditioned text head makes LoRA generalize better than established prompt-based methods. The load-bearing result is a set of benchmark numbers: GLAD reports the highest average base and novel class accuracy across 11 datasets (85.05% and 76.74%), the best harmonic mean (80.68%), and consistent gains in domain-shift and cross-dataset transfer. The authors argue this happens because the regularizer steers optimization into flatter, more stable regions of the loss landscape, preventing the overfitting that plain few-shot LoRA suffers, while AlignNet adds the cross-modal flexibility prompt methods obtain through task-specific architectures. If the claimed numbers hold, the practical consequence is that complex prompt designs and their sensitive hyperparameters are unnecessary for strong few-shot CLIP transfer.

What carries the argument

The central mechanism is the gradient-regularized update of Section 3.4, which fuses two gradient signals. First it computes the ordinary gradient g at the current parameters, then a second gradient g' at parameters shifted by the sharpness-aware perturbation epsilon = rho * grad/||grad|| (the SAM device that probes the loss neighborhood). When g and g' point in conflicting directions the method projects g' onto the subspace orthogonal to g, then forms the update gf = (1 - alpha)g + alpha g'. The claim is that this fused direction inherits SAM's flat-minima bias while preserving the original descent direction, so the model reaches a more generalizable region instead of merely a lower-loss point; the construction keeps the whole method hyperparameter-light and architecture-agnostic.

What would settle it

Train GLAD and the plain LoRA baseline on the same few-shot splits, then measure the sharpness of the two final solutions by computing the largest eigenvalue of the Hessian or the loss increase under a small parameter perturbation. If GLAD's solutions are not flatter than LoRA's, or if flatness does not track the reported novel-class and cross-dataset accuracy differences, then the flatness story in Section 3.4 is contradicted even if the benchmark table stands.

Watch

Extended reading notes

Core claim

GLAD's central claim is that a vanilla LoRA fine-tune of both CLIP encoders already matches the accuracy of recent prompt-tuning methods, and that two additions then exceed them: gradient-regularized optimization using a sharpness-aware reference gradient, and AlignNet, a small MLP that biases the static text embedding with the current image feature before computing logits. The paper reports that this combination achieves the best average harmonic mean between base and novel class accuracy (80.68%) among all compared methods on 11 datasets, improves on the CLIP-LoRA baseline by +1.67 HM points, and raises cross-dataset average accuracy by +4.63 points over the same baseline. The ablation study credits the gradient regularization with recovering the novel-class gain SAM provides while avoiding SAM's drop on base classes, and credits AlignNet with improving both.

Load-bearing premise

The whole method assumes that taking a gradient at the sharply perturbed weights, stripping the part that fights the original gradient, and averaging the two directions really leads to a flatter, more generalizable solution, and that this flatness is the reason for the accuracy gains rather than some other side effect.

Editorial extensions

If this is right

  • If GLAD's numbers are right, prompt tuning's task-specific designs (prompt insertion schemes, separate text branches) are not necessary for strong few-shot CLIP transfer; a frozen backbone plus 1% LoRA parameters suffices.
  • The gradient regularizer can be dropped into any LoRA-based fine-tuning, not just CLIP, so the flatness-seeking trick generalizes as an optimizer-level fix for few-shot overfitting.
  • Since LoRA weights merge into the backbone at inference, GLAD adds no extra latency or memory at test time, unlike methods that keep learnable prompts or auxiliary branches.
  • The ablation's separation of SAM's novel-class gain from its base-class cost suggests that the interpolation coefficient alpha controls a trade-off between fitting seen classes and generalizing to unseen ones.
  • Cross-dataset gains of +4.63 over the LoRA baseline indicate the method is not just memorizing the training distribution but transferring to 10 unseen datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never measures flatness directly; a reader should test whether GLAD's solutions are actually flatter (e.g., smaller Hessian spectral norm) than the LoRA baseline's before accepting the mechanism over the numbers.
  • Because alpha is fixed at 0.5 in all experiments, the authors leave unexplored a possible Pareto frontier between base and novel accuracy; sweeping alpha could reveal whether the harmonic mean gain is robust or tuned.
  • The same regularization logic could apply to LLM fine-tuning with LoRA under data scarcity, a setting the paper does not test but where the flatness argument transfers.
  • The reported DTD novel-class jump (+4.70 over baseline) and EuroSAT novel-class jump (+7.14) are surprisingly large relative to other datasets, so checking per-class confusion and seed variance on those two datasets would tell whether the gain is concentrated in a few categories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes GLAD, a fine-tuning framework for CLIP that combines LoRA-based internal adaptation, a lightweight AlignNet that adjusts text features with image context, and a gradient-regularization strategy inspired by Sharpness-Aware Minimization (SAM). The method is evaluated on base-to-novel class generalization across 11 datasets, domain generalization on four ImageNet variants, and cross-dataset generalization over 10 target datasets, claiming state-of-the-art average accuracy over prompt-based baselines and a CLIP-LoRA baseline. An ablation study (Table 4) attributes performance gains to the proposed Gradient Regularization and AlignNet components.

Significance. If the results are reproducible and the component attribution is valid, GLAD is a simple, parameter-efficient alternative to prompt tuning that shows strong empirical performance across several generalization settings. The paper provides extensive experiments and a clear ablative breakdown, and the promise of public code is a positive sign. However, the current manuscript leaves a key training objective (the KL divergence loss) underspecified, which could fully confound the attribution of the reported gains. With the training objective properly specified and the proposed components isolated from this regularization term, the paper would be a valuable contribution to few-shot VLM adaptation.

major comments (4)
  1. [Section 3 and Figure 2] The total training objective is never fully defined. Figure 2 states that training uses the cross-entropy loss L_CE and a KL divergence loss L_KL between the predictions of the original and fine-tuned models, but Section 3, including Algorithm 1, only refers to a generic loss L(θ) and never gives the form of L_KL, its weighting coefficient, or the full objective. Moreover, the ablation in Table 4 does not state whether L_KL is included in all rows or only in specific configurations. Since a KL-to-zero-shot-distribution term is known to improve base-to-novel generalization (e.g., as used in PromptSRC), the reported improvements of +1.37 HM for GradReg (row c vs. row a) and +0.80 HM for AlignNet (row e vs. row d) could in principle be driven entirely by this undefined KL term. The authors must specify the full loss, the KL weight, and an ablation that includes a LoRA+KL baseline to isolate the contributions of the proposed components.
  2. [Section 3.4, Eq. (7)] The central claim that the gradient regularization in Eq. (7) steers optimization toward flatter, more generalizable regions is not empirically verified. No flatness measure (e.g., largest Hessian eigenvalue, SAM loss at the solution, or loss-landscape visualization) is reported for the LoRA baseline versus GLAD. The connection to flat minima is borrowed from SAM, but the modified gradient g_f is applied outside the SAM min-max objective, so the link is heuristic. If the proposed mechanism is not demonstrated, the observed improvements could stem from other effects of the interpolation, such as implicit learning-rate modulation or noise smoothing. Please add a quantitative flatness diagnostic or a derivation showing that Eq. (7) is equivalent to, or approximates, a sharpness-aware objective.
  3. [Section 3.2 and Implementation Details] Several implementation details that are load-bearing for reproducibility are missing. The LoRA scaling factor γ in Eq. (4) is never assigned a value, even though it directly controls the magnitude of the adaptation and affects generalization. The text says LoRA is applied to "key linear projections such as the query, key, and value matrices in self-attention," but it does not specify whether the list is exhaustive or whether other projection layers are included in both encoders. Finally, the KL loss weight and the exact form of L_KL (e.g., symmetric KL, temperature, whether it uses hard or soft targets) are absent. For a framework whose stated goal is simplicity and ease of use, these missing hyperparameters prevent replication and make it hard to assess whether the chosen settings are standard or carefully tuned.
  4. [Section 4.3, Table 4] The ablation table is ambiguous about the presence of the KL regularization term in each row. If row (a) (LoRA baseline) does not include L_KL but rows (b)-(e) do, then the comparison is not controlled and the gains attributed to "GradReg" and "AlignNet" are conflated with the KL term. The caption and text should explicitly state, for every row, whether L_KL is active and with what weight. Adding a row "LoRA + L_KL" would directly resolve whether the proposed components provide any benefit beyond the already-existing zero-shot-preserving KL regularization.
minor comments (4)
  1. [Figure 1] The label "PromprSRC" in Figure 1 is a typo; it should read "PromptSRC."
  2. [Tables 1-3] The paper states that results are averaged over three random seeds, but neither the individual seeds nor standard deviations are reported. Given that some comparisons are close (e.g., GLAD 60.80 vs. PromptSRC 60.65 in Table 3), error bars or variance information would help assess whether the differences are meaningful.
  3. [Section 3.4] The sentence "without altering the main optimization direction" (around Eq. (7)) is imprecise: if g' is orthogonal to g after projection, the fused gradient (1-α)g + αg' is not parallel to g, so the update direction does change for any α>0. The text should be reworded to say that the component along g is preserved and a supplementary direction is added, or provide a formal justification for why the orthogonal component does not alter the descent direction.
  4. [Abstract and Introduction] The claim that GLAD is "the first framework dedicated to improving the generalization ability of LoRA in few-shot learning scenarios" may be too strong given that CLIP-LoRA [57] already explores LoRA for few-shot classification, albeit with less emphasis on generalization. Consider softening the novelty claim or citing additional prior work on LoRA generalization.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: GLAD's claimed gains are benchmarked on external held-out splits and the gradient regularization is computed from the same training data, not from target metrics. The only self-citation [51] is non-load-bearing, and the missing L_KL specification is a reproducibility/confound issue, not a circular reduction.

full rationale

The paper's central claim is that LoRA plus gradient regularization (Eqs. 2, 3, 6, 7) and AlignNet (Eq. 5) improve base-to-novel, domain, and cross-dataset generalization. The derivation chain is empirical rather than self-referential: the regularization is computed from gradients on the training data, the hyperparameters alpha=0.5 and rho=0.1 are fixed rather than fitted to the evaluation targets, and all headline numbers are measured on external benchmarks with held-out classes and datasets. No fitted parameter is renamed as a prediction, and no target generalization metric is used to construct the training objective. The flat-minima premise is heuristic and unverified by any flatness measurement, but an unsupported empirical assumption is a correctness risk, not circularity. The only self-citation is [51], a prior CVPR paper by co-author Pengfei Wang; it is cited alongside external references [9,35,69] to support the empirical association between flat regions and generalization, and it is not used to justify Eqs. (6)-(7) or to rule out alternative designs, so it is not load-bearing. One omitted specification must be flagged: Figure 2's caption states, 'During training, we apply the cross-entropy loss LCE for classification and the KL divergence loss LKL between the predicted distributions of the original and fine-tuned models to preserve zero-shot CLIP behavior,' but L_KL is never defined, weighted, or ablated. If the Table 4 baseline row (a) omitted L_KL while rows (b)-(e) included it, the attributed gains of GradReg and AlignNet would be confounded. However, the paper does not say this, and the central comparison against external baselines remains independent of the ablations; this is a missing-support and reproducibility concern rather than a circularity, so it does not increase the circularity score beyond a low level.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The GLAD framework relies on two hand-set hyperparameters (alpha, rho), a chosen LoRA rank, an AlignNet architecture, and several unspecified weights (gamma, KL weight). The core optimization heuristic is an ad hoc assumption that projected SAM gradients lead to flatter minima. AlignNet is an invented architectural component with only internal empirical support.

free parameters (6)
  • alpha (gradient mixing coefficient) = 0.5
    Chosen by hand; controls interpolation between original and SAM-reference gradient in Eq. (7).
  • rho (SAM perturbation radius) = 0.1
    Chosen by hand; sets the norm of the adversarial parameter perturbation in Eq. (2).
  • LoRA rank r = 8
    Chosen for all LoRA modules; determines capacity of the low-rank adaptation.
  • AlignNet hidden dimensions = 256 and 128
    Chosen architecture for the MLP that adjusts text embeddings.
  • LoRA scaling factor gamma = unspecified
    Appears in Eq. (4) but no value or schedule is given.
  • KL divergence loss weight = unspecified
    L_KL is mentioned in Figure 2 caption but no weight or formula is provided.
assumptions (5)
  • domain assumption CLIP's pretrained embeddings are useful for downstream classification and remain useful after low-rank updates to attention projections.
    Assumed throughout Section 3.2; the paper does not analyze interference between LoRA updates and pretrained representations.
  • domain assumption Flat minima in the loss landscape correlate with better generalization.
    Invoked in Sections 3.1 and 3.4 with references [9,35,51,69]; no sharpness measurement is performed here.
  • ad hoc to paper The SAM gradient at theta+epsilon is a reliable reference for a stable descent direction, even after removing its component that conflicts with the original gradient.
    This is the core heuristic behind Eq. (6)-(7); no proof or empirical flatness check is provided.
  • ad hoc to paper AlignNet can adjust text features per image without degrading base-class fitting or novel-class transfer.
    Justified only by ablation row (d) in Table 4; no theoretical analysis.
  • standard math Standard optimization and pre-training assumptions: loss is differentiable, SGD with cosine schedule works, etc.
    Background for Algorithm 1.
invented entities (1)
  • AlignNet
    purpose: Post-encoding MLP that generates a bias vector for text embeddings conditioned on the current image feature, enabling instance-level text adjustment.
    Introduced in Section 3.3; supported only by internal ablations, no external or theoretical validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GLAD: Generalizable Tuning for Vision-Language Models." pith.science (2026). https://pith.science/paper/OBAVFK3J

@misc{pith2026250713089,
  author       = {Pith},
  title        = {Pith review of: GLAD: Generalizable Tuning for Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OBAVFK3J}},
  note         = {Machine review of arXiv:2507.13089}
}
read the original abstract

Pre-trained vision-language models, such as CLIP, show impressive zero-shot recognition ability and can be easily transferred to specific downstream tasks via prompt tuning, even with limited training data. However, existing prompt tuning methods face two main challenges: (1) In few-shot scenarios, data scarcity often leads to overfitting, making the model sensitive to changes in the input domain. (2) To mitigate overfitting, these methods typically rely on complex task-specific model architectures and sensitive hyperparameter tuning, severely restricting their general applicability. To address these issues, we propose a simpler and more general framework called GLAD (Generalizable LoRA tuning with RegulArized GraDient). We show that merely applying LoRA achieves performance in downstream tasks comparable to current state-of-the-art prompt-based methods. While LoRA is effective and easy to use, it remains susceptible to overfitting in few-shot learning scenarios. To mitigate this risk, we introduce a gradient-based regularization technique. This technique effectively steers the optimization trajectory, encouraging the model to find a more stable parameter region that is robust to variations in data distribution. Through extensive experiments conducted on 15 benchmark datasets, we demonstrate that GLAD outperforms previous tuning approaches in terms of base-to-novel class generalization, image domain generalization, and cross-dataset generalization. The code will be publicly available.

Figures

Figures reproduced from arXiv: 2507.13089 by the authors.

Figure 1
Figure 1. An overview of the proposed regularization strategy and its effect on base-to-novel generalization. (a) Illustration of our [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the GLAD framework. (a) LoRA modules are inserted into each Transformer layer of CLIP’s encoders to enable [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models

    cs.LG 2026-04 conditional novelty 7.0 of 10

    GUI-Perturbed shows that GUI grounding models suffer systematic accuracy collapse under relational instructions and visual changes such as 70% zoom, with even augmented fine-tuning worsening results.

  2. TARA: Token-Aware LoRA for Composable Personalization in Diffusion Models

    cs.CV 2025-08 conditional novelty 6.0 of 10

    TARA adds token-focused masking and a token alignment loss to LoRA adapters, allowing several independently trained personalized adapters to be composed with less identity loss and feature leakage.

Reference graph

Works this paper leans on

69 extracted references · 47 canonical work pages · cited by 2 Pith papers

  1. [1]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In ECCV, pages 446–461. Springer, 2014. 6

    2014

  2. [2]

    Domain prompt learning with quaternion networks

    Qinglong Cao, Zhengqin Xu, Yuntian Chen, Chao Ma, and Xiaokang Yang. Domain prompt learning with quaternion networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26637– 26646, 2024. 3

    work page 2024

  3. [3]

    Tokenmixup: Efficient attention-guided token-level data augmentation for transformers

    Hyeong Kyu Choi, Joonmyung Choi, and Hyunwoo J Kim. Tokenmixup: Efficient attention-guided token-level data augmentation for transformers. In NeurIPS, 2023. 3

    work page 2023

  4. [4]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, pages 3606–3613, 2014. 6

    2014

  5. [5]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. Ieee, 2009. 6, 7, 8

    work page 2009

  6. [6]

    Learning to prompt for open-vocabulary ob- ject detection with vision-language model

    Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary ob- ject detection with vision-language model. In CVPR, 2022. 1

    work page 2022

  7. [7]

    Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories. In CVPR Workshop, pages 178–178. IEEE, 2004. 6

    work page 2004

  8. [8]

    Prompt- det: Towards open-vocabulary detection using uncurated im- ages

    Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Prompt- det: Towards open-vocabulary detection using uncurated im- ages. In ECCV, 2022. 1

    work page 2022

Show all 69 references
  1. [9]

    Sharpness-aware minimization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412,

    2010 arXiv

  2. [10]

    Clip-adapter: Better vision-language models with feature adapters

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544, 2021. 3

    2021 arXiv

  3. [11]

    Explaining and harnessing adversarial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. 3

    2014 arXiv

  4. [12]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022. 1

    2022

  5. [13]

    Eurosat: A novel dataset and deep learn- ing benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learn- ing benchmark for land use and land cover classification. J- STARS, 12(7):2217–2226, 2019. 6

    2019

  6. [14]

    The many faces of robust- ness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust- ness: A critical analysis of out-of-distribution generalization. In ICCV, pages 8340–8349, 2021. 7

    2021

  7. [15]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. In CVPR, pages 15262–15271, 2021. 7

    2021

  8. [16]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 1, 3

    2022

  9. [17]

    Learning a better ini- tialization for soft prompts via meta-learning

    Yukun Huang, Kun Qian, and Zhou Yu. Learning a better ini- tialization for soft prompts via meta-learning. arXiv preprint arXiv:2205.12471, 2022. 5, 6

    2022 arXiv

  10. [18]

    Patching open- vocabulary models by interpolating weights

    Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open- vocabulary models by interpolating weights. arXiv preprint arXiv:2208.05592, 2022. 3

    2022 arXiv

  11. [19]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. In ECCV, 2022. 3

    2022

  12. [20]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, pages 19113– 19122, 2023. 1, 3, 6, 7

    2023

  13. [21]

    Self-regulating prompts: Foundational model adaptation without forgetting

    Muhammad Uzair Khattak, Syed Talal Wasim, Muzam- mal Naseer, Salman Khan, Ming-Hsuan Yang, and Fa- had Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF international conference on computer vision, pages 151...

    2023

  14. [22]

    Co-mixup: Saliency guided joint mixup with super- modular diversity

    Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with super- modular diversity. In ICLR, 2021. 3

    2021

  15. [23]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV, pages 554–561, 2013. 6

    2013

  16. [24]

    Read-only prompt op- timization for vision-language few-shot learning

    Dongjun Lee, Seokwon Song, Jihee Suh, Joonmyeong Choi, Sanghyeok Lee, and Hyunwoo J Kim. Read-only prompt op- timization for vision-language few-shot learning. In ICCV,

  17. [25]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP,

  18. [26]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML,

  19. [27]

    Promptkd: Unsupervised prompt distillation for vision-language models

    Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, and Jian Yang. Promptkd: Unsupervised prompt distillation for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26617–26626, 2024. 3

    2024

  20. [28]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 3

    2023

  21. [29]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in nat- ural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hi- roaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in nat- ural language processing. ACM computing surveys , 55(9): 1–35, 2023. 2

    2023

  22. [30]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 3 9

    2019

  23. [31]

    Prompt distribution learning

    Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In CVPR, pages 5206–5215, 2022. 1, 6, 7

    2022

  24. [32]

    Image segmentation using text and image prompts

    Timo L ¨uddecke and Alexander Ecker. Image segmentation using text and image prompts. In CVPR, pages 7086–7096,

  25. [33]

    Fine-grained visual classi- fication of aircraft

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft. arXiv preprint arXiv:1306.5151 , 2013. 6

    2013 arXiv

  26. [34]

    Clipcap: Clip prefix for image captioning

    Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv:2111.09734, 2021. 1

    2021 arXiv

  27. [35]

    Lookbehind-sam: k steps back, 1 step for- ward

    Gonc ¸alo Mordido, Pranshu Malviya, Aristide Baratin, and Sarath Chandar. Lookbehind-sam: k steps back, 1 step for- ward. arXiv preprint arXiv:2307.16704, 2023. 3, 5

    2023 arXiv

  28. [36]

    When does label smoothing help? Advances in neural in- formation processing systems, 32, 2019

    Rafael M ¨uller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural in- formation processing systems, 32, 2019. 3

    2019

  29. [37]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, pages 722–729. IEEE, 2008. 6

    2008

  30. [38]

    Metropolis-hastings data augmentation for graph neu- ral networks

    Hyeonjin Park, Seunghun Lee, Sihyeon Kim, Jinyoung Park, Jisu Jeong, Kyung-Min Kim, Jung-Woo Ha, and Hyunwoo J Kim. Metropolis-hastings data augmentation for graph neu- ral networks. In NeurIPS, 2022. 3

    2022

  31. [39]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, pages 3498–3505. IEEE, 2012. 6

    2012

  32. [40]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 1, 6, 7

    2021

  33. [41]

    Do imagenet classifiers generalize to im- agenet? In ICML, pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? In ICML, pages 5389–5400. PMLR, 2019. 7

    2019

  34. [42]

    Multimodal instruction tun- ing with conditional mixture of lora

    Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wen- peng Yin, and Lifu Huang. Multimodal instruction tun- ing with conditional mixture of lora. arXiv preprint arXiv:2402.15896, 2024. 3

    2024 arXiv

  35. [43]

    Flava: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guil- laume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022. 1

    2022

  36. [44]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 6

    2012 arXiv

  37. [45]

    Dropout: a simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15 (1):1929–1958, 2014. 3

    1929

  38. [46]

    Rethinking the inception ar- chitecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception ar- chitecture for computer vision. In CVPR, pages 2818–2826,

  39. [47]

    Saliencymix: A saliency guided data augmentation strategy for better regularization

    AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bae, et al. Saliencymix: A saliency guided data augmentation strategy for better regularization. InICLR,

  40. [48]

    Manifold mixup: learning better representations by in- terpolating hidden states

    Vikas Verma, Alex Lamb, Christopher Beckham, Amir Na- jafi, Aaron Courville, Ioannis Mitliagkas, and Yoshua Ben- gio. Manifold mixup: learning better representations by in- terpolating hidden states. In ICML, 2019. 3

    2019

  41. [49]

    Tuning multi-mode token- level prompt alignment across modalities

    Dongsheng Wang, Miaoge Li, Xinyang Liu, MingSheng Xu, Bo Chen, and Hanwang Zhang. Tuning multi-mode token- level prompt alignment across modalities. Advances in Neu- ral Information Processing Systems, 36:52792–52810, 2023. 3

    2023

  42. [50]

    Learning robust global representations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019. 7

    2019

  43. [51]

    Sharpness-aware gradient matching for domain generaliza- tion

    Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generaliza- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 3769–3778,

  44. [52]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiX- uan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Sys- tems, 37:121475–121499, 2024. 3

    2024

  45. [53]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gon- tijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR, pages 7959–7971, 2022. 3

    2022

  46. [54]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, pages 3485–3492. IEEE, 2010. 6

    2010

  47. [55]

    Tcp: Textual- based class-aware prompt tuning for visual-language model

    Hantao Yao, Rui Zhang, and Changsheng Xu. Tcp: Textual- based class-aware prompt tuning for visual-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23438–23448, 2024. 1, 3

    2024

  48. [56]

    Cutmix: Regu- larization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features. In CVPR, pages 6023–6032, 2019. 3

    2019

  49. [57]

    Low-rank few-shot adaptation of vision-language models

    Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1593–1603, 2024. 3, 6, 7

    2024

  50. [58]

    Lit: Zero-shot transfer with locked-image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, pages 18123–18133, 2022. 1

    2022

  51. [59]

    Three mechanisms of weight decay regularization

    Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. In ICLR, 2019. 3

    2019

  52. [60]

    mixup: Beyond empirical risk minimiza- tion

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. arXiv preprint arXiv:1710.09412, 2017. 3 10

    2017 arXiv

  53. [61]

    Dept: Decoupled prompt tuning

    Ji Zhang, Shihan Wu, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Dept: Decoupled prompt tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12924–12933, 2024. 1, 3

    2024

  54. [62]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023. 3

    2023

  55. [63]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. In ICLR, 2024. 1

    2024

  56. [64]

    Svit: Scaling up visual instruction tuning

    Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04087, 2023. 3

    2023 arXiv

  57. [65]

    Regionclip: Region-based language-image pretraining

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In CVPR, 2022. 1

    2022

  58. [66]

    Conditional prompt learning for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Zi- wei Liu. Conditional prompt learning for vision-language models. In CVPR, pages 16816–16825, 2022. 1, 3, 5, 6, 7

    2022

  59. [67]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9):2337–2348, 2022. 1, 2, 6, 7

    2022

  60. [68]

    Prompt-aligned gradient for prompt tuning

    Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. InICCV,

  61. [69]

    Surrogate gap minimization improves sharpness-aware training

    Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.08065,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.