Pith. sign in

REVIEW 3 major objections 6 minor 89 references

Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning zero-shot models with dual risk minimization improves out-of-distribution robustness beyond FLYP and other baselines.

desk verdict Solid empirical paper on robust fine-tuning; the worst-case-risk story is under-built, but the method works and the paper is honest about its limits. read the letter →

arxiv 2411.19757 v1 pith:MZU3LHF7 submitted 2024-11-29 cs.LG cs.CV

Kaican Li , Weiyan Xie , Yongxiang Huang , Didan Deng , Lanqing Hong , Zhenguo Li , Ricardo Silva , Nevin L. Zhang This is my paper · ORCID
classification cs.LGcs.CV
keywords dualriskminimizationrobustfine-tuningzero-shotmodelsCLIPout-of-distributionrobustnessconceptdescriptionsworst-caselargelanguage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes dual risk minimization (DRM) as the training objective for fine-tuning zero-shot vision-language models, replacing the single empirical-risk objective with a sum of empirical risk and a worst-case-risk proxy. The proxy is a cross-entropy loss against soft labels computed by a pre-trained CLIP model when it is prompted with GPT-4 concept descriptions of each class's core visual features. DRM thereby preserves class-defining features while still fitting the downstream task, and the paper reports consistent out-of-distribution gains on ImageNet, WILDS-iWildCam, and WILDS-FMoW across three model sizes. A sympathetic reader would care because this is a concrete way to defend fine-tuning against distribution shifts without needing grouped domain data.

What carries the argument

The object that carries the argument is the DRM objective (7), $$\min_{\$\theta$} R_s(\$\theta$;$T^{{df}}$)+\$\lambda$ R^c_s(\$\theta$;$T^{{cd}}$),$$ where $R_s$ is the empirical risk with default prompts and $R^c_s$ is the core-feature risk. The core-feature risk is computed with respect to the soft labels $\tilde{p}_c(y|x)$ defined in Eq. (9), which come from per-class min-max normalization in Eq. (8) of the pre-trained CLIP affinities between image embeddings and text embeddings of GPT-4 concept descriptions; the normalization suppresses artifact terms that would otherwise make the soft labels high-entropy. Inference uses a mixture $\hat{p}^{dual}_\theta=\beta\hat{p}^{df}_\theta+(1-\beta)\hat{p}^{cd}_\theta$. Theorem 1's strong-duality result justifies relaxing the constrained IDRM into this weighted sum, and the empirical study in Appendix D makes the case that concept-description affinities respond to foreground core features while staying stable under background changes.

What would settle it

On a domain where LLM concept descriptions are known to be inaccurate, such as breast histology or ocular disease, compare DRM with FLYP under identical hyperparameter selection; if the OOD advantage disappears or reverses while in-distribution accuracy stays comparable, the concept-description proxy is the load-bearing part of the claim.

Watch

Extended reading notes

Core claim

The central claim is that minimizing $R_s(\theta;T^{df})+\lambda R^c_s(\theta;T^{cd})$—the standard ERM term supervised by one-hot labels using default prompts, plus a worst-case-risk term supervised by soft labels $\tilde{p}_c(y|x)$ built from CLIP affinities to LLM-generated concept descriptions—preserves core features and improves robustness. The paper proves strong duality between the idealized constrained problem and the weighted dual, and then replaces the intractable worst-case risk over all domains with the core-feature risk on the training domain. On CLIP ViT-L/14@336, DRM raises OOD ImageNet accuracy from 75.9 to 77.1, WILDS-iWildCam macro F1 from 47.1 to 51.8, and WILDS-FMoW worst-region accuracy from 50.7 to 53.1 relative to FLYP plus WiSE-FT, while maintaining comparable in-distribution accuracy.

Load-bearing premise

The load-bearing premise is that CLIP's similarity scores for GPT-4 concept descriptions reliably track the class-defining visual attributes and ignore background and context; the soft labels built from those scores are only a trustworthy stand-in for worst-case risk if this holds.

Editorial extensions

If this is right

  • DRM with WiSE-FT raises OOD ImageNet accuracy on CLIP ViT-L/14@336 from 75.9 to 77.1, WILDS-iWildCam macro F1 from 47.1 to 51.8, and WILDS-FMoW worst-region accuracy from 50.7 to 53.1.
  • On CLIP ViT-B/16, DRM's relative OOD gains over FLYP are about 5.0% on ImageNet, 12.4% on iWildCam, and 11.1% on FMoW; with WiSE-FT the gains are 1.9%, 11.6%, and 9.8%.
  • The ablation shows that using concept descriptions for both the ERM and the WRM term is worse than separating the prompts, so the dual-prompt design is load-bearing for the result.
  • DRM is not tied to one generation model: concept descriptions from GPT-3.5, GPT-4, and three Llama-3 sizes all improve OOD performance over FLYP on iWildCam.
  • Compared with FLYP, DRM adds roughly 20% to training and inference cost, from the extra text-encoder pass.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's benchmarks, the method suggests a scaling rule for robust fine-tuning: any model with a text-interface zero-shot head could receive the same dual-prompt treatment, so the recipe may transfer from CLIP to other vision-language and multimodal models.
  • The fixed per-class min-max normalization is a natural place to look for further gains; a learned temperature or a calibrated version of $\tilde{p}_c(y|x)$ could reduce sensitivity to $\lambda$ on classes with weak or noisy concept affinities.
  • Given the paper's admitted failure of GPT-4 descriptions in medical imaging, an immediate testable extension is to generate concept descriptions with image-grounded multimodal LLMs or expert curation and check whether DRM's OOD advantage reappears in those domains.
  • Since DRM improves worst-case robustness without grouped domain labels, a head-to-head against domain-invariant methods on multi-domain benchmarks would separate the contribution of the core-feature proxy from the contribution of the dual objective itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes dual risk minimization (DRM) for fine-tuning zero-shot vision-language models such as CLIP. The objective adds to the standard empirical risk, computed with default prompts, a regularization term computed with LLM-generated concept descriptions; the soft targets for this second term are obtained from the pre-trained CLIP model and normalized per class. The authors argue that this regularizer estimates the worst-case risk over domains and thereby preserves core visual features. On ImageNet, WILDS-iWildCam, and WILDS-FMoW, DRM reports consistent OOD improvements over FLYP and other robust fine-tuning baselines, including with WiSE-FT, across three CLIP model sizes. The paper includes ablations on prompt choice, inference combination, normalization, LLM source, and the hyperparameter lambda.

Significance. If the findings hold, DRM is a practically valuable and simple addition to robust fine-tuning: it requires only an LLM-generated concept description per class and a precomputed soft-label set, and the reported gains on ImageNet variants, iWildCam, and FMoW are consistent across model sizes and seeds. The empirical evaluation is a genuine strength: five-seed runs with confidence intervals, ablations in Tables 3 and 10, sensitivity analyses for lambda and for different LLMs, and cost measurements in Appendix F.3. The central conceptual claim, however, that the concept-description soft-label risk is a faithful proxy for worst-case risk over domains, is asserted more than established; the formal derivation from IDRM to DRM has a gap. The empirical OOD improvements remain credible, but the paper's mechanistic explanation is stronger than the evidence supports.

major comments (3)
  1. [Section 3, Theorem 1 and Appendix A] The strong-duality argument is carried out in prediction space, not in parameter space. Lemma 1 shows convexity of the cross-entropy in the prediction function q(y|x), and the proof then applies Slater's condition to a convex program over q. However, the actual optimizations in (IDRM) and (DRM) are over theta, the parameters of a deep network, for which the objective is not convex. The conclusion that IDRM 'reduces to' min_theta [R_s(theta) + lambda* max_d R_d(theta)] is therefore not established for the parameterized problem. Since this reduction is the formal bridge from IDRM to DRM, please either prove duality under the specific parameterization, state explicit conditions under which it holds, or reframe Theorem 1 as a motivation in function space rather than a theorem about the training objective actually used.
  2. [Section 4.2, Eqs. (8)-(9)] The identification of CLIP affinities to LLM concept descriptions with the oracle core-feature model p_c(y|x) is the load-bearing step of the method, but it is directly validated only on the 15-class Hard ImageNet study (Table 6 and Appendix D.1), and Appendix G concedes that GPT-4 produces inaccurate concept descriptions in medical imaging domains. The min-max normalization in Eq. (8) is a per-class rank transform over training images rather than a calibrated estimate of core-feature presence, and Eq. (9) distributes the residual probability mass to other classes in proportion to unnormalized affinities whenever gamma(x, y_x) < 1, so p_tilde_c can assign substantial mass on the basis of non-core similarity. The empirical OOD improvements over FLYP remain credible, but the specific claim that R_cs(theta; T_cd) is a worst-case-risk proxy is under-supported. Please provide additional evidence for the core-feature interpretation on the evaluated datasets, or soften the WRM claim and present R_cs as a regularizer motivated by core-feature preservation whose empirical value is established by the ablations.
  3. [Section 5.3, Table 3 and Table 10] The ablations convincingly show that the concept-description soft labels outperform one-hot labels and default-prompt soft labels as the regularization target, which is strong evidence against a purely generic-distillation explanation. However, the paper does not isolate the effect of the min-max normalization from the effect of the soft-label target: Row 10 of Table 3 shows that the unnormalized direct estimate fails badly, but there is no analysis of what the normalization does to the ranking of classes or to the entropy of p_tilde_c. A simple diagnostic, such as reporting the distribution of gamma(x, y_x) and the average entropy of p_tilde_c across datasets, would substantially strengthen the claim that Eqs. (8)-(9) estimate core-feature presence rather than merely rescale per-class affinities.
minor comments (6)
  1. [Section 1] The sentence 'The main challenge of applying DRM to real-world tasks is to assess of worst-case risk' contains a grammatical error; it should read 'is to assess the worst-case risk' or 'is the assessment of worst-case risk.'
  2. [Section 4.1] 'It is also know that non-visual and spurious descriptions contribute significantly to CLIP's representation' should read 'It is also known that...'.
  3. [Appendix D title] 'Quantitative study on the reliability of concept descriptions verse default prompts' should use 'versus' instead of 'verse'.
  4. [Table 3] The notation in the 'Affinity norm.' and 'w/ model' columns uses '!' and '%' symbols that are not defined in the caption; please define them or replace them with explicit labels.
  5. [Figure 1] The rendering of Figure 1 in the submitted PDF is difficult to read: the inline text is fragmented and the diagram is not self-explanatory. Please ensure that the figure is legible and that its caption explains the relationship between ERM, WRM, and DRM without relying on the surrounding prose.
  6. [References] There are several typographical artifacts in the reference list, such as 'V ondrick' and 'V on Kügelgen'; these should be cleaned up before the camera-ready version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DRM's OOD gains are measured on external benchmarks, and the core-feature proxy is an explicitly labeled, empirically checked assumption rather than a construction that reduces to its own outputs.

full rationale

The claimed derivation chain is: IDRM is relaxed by replacing max_d R_d with R_c_s(theta), where R_c_s is a cross-entropy against the oracle core-feature model p_c(y|x); p_c is estimated by p_tilde_c(y|x) from Eq. (9), computed from pre-trained CLIP affinities to GPT-4 concept descriptions; the final objective (7) is ERM with default prompts plus this regularizer. None of these steps is circular by construction: the soft labels are precomputed once from the frozen theta_0 and are not fit to any of the reported OOD numbers; the OOD accuracies in Tables 1 and 2 (ImageNet 75.9 to 77.1, iWildCam 47.1 to 51.8, FMoW 50.7 to 53.1) come from external benchmark test sets, so the central empirical claim is not forced by the inputs. The identification of CLIP-concept-description affinity with core-feature presence is an assumption, but the paper supports it with an independent operational check in Table 6 and Figure 2 using Hard ImageNet foreground/background removal: concept-description affinities drop by 55.6% when foreground is removed versus 1.9% for default prompts, and by only 10.3% versus 31.1% when background is removed. That check is external to the fine-tuning targets and provides real evidence for the proxy. The paper also transparently labels the regularizer as 'a form of self-distillation targeted at core features,' which is a description of the mechanism rather than an attempt to pass the construction off as an independently derived oracle. The self-references (Zhang et al. 2023a; Gao et al. 2024a) appear only in a list of similar data-generation models 'widely adopted in the literature' and are not load-bearing for the main claim. Appendix G honestly concedes that GPT-4 produced inaccurate concept descriptions in medical imaging domains and that CLIP pre-training data are undisclosed; these limitations weaken domain-scope claims but do not make the ImageNet, iWildCam, or FMoW results circular. The strong-duality argument in Appendix A is somewhat loose because convexity is shown with respect to prediction probabilities rather than the non-convex parameterization theta, but that is a rigor concern, not circularity. Overall, the empirical claim is self-contained against external benchmarks, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The DRM objective relies on a causal core/non-core decomposition, an informal equivalence between worst-case risk and core-feature reliance, and the reliability of CLIP affinities to LLM concept descriptions. None of these is formally proven; the only direct empirical check is the Hard ImageNet foreground/background study on 15 classes. The method's empirical gains are measured on external OOD benchmarks, which is the main independent evidence.

free parameters (3)
  • lambda (WRM weight) = chosen from {1, 2, 3, 4, 5} by ID validation; exact per-dataset values not reported
    Controls the balance between ERM and the worst-case proxy in Eq. (7). At lambda=0 DRM reduces to FLYP. Table 5 shows OOD performance is fairly stable across lambda, but all reported main results use the validation-selected value.
  • beta (inference mixture weight) = beta = 1 / (1 + lambda) by default
    Mixes the default-prompt and concept-description classifiers at inference in Eq. (10). It is tied to lambda, so not separately tuned, but it is still a design choice affecting reported results.
  • rho (WiSE-FT interpolation weight) = chosen from 0.1 to 0.9 by ID validation
    Used when DRM is combined with WiSE-FT. The reported DRM+WiSE-FT numbers depend on this validation-selected interpolation weight, as do the WiSE-FT baselines.
assumptions (5)
  • domain assumption Causal data model: X <- hX(Xc, Xn, eps), Y <- hY(Xc), with causal mechanisms and eps distribution invariant across domains.
    Introduced in Section 3 'Data model'. Needed to interpret Xc as core features and to justify the claim that a predictor using only core features controls worst-case risk.
  • domain assumption Non-core features are generally unreliable across domains: 'leveraging non-core features would almost always lead to worse performance in certain domains'.
    Stated in Section 3 after Theorem 1. This bridges the intractable max_d Rd(θ) to reliance on core features and is the conceptual basis for replacing worst-case risk with Rcs(θ;Tcd).
  • domain assumption CLIP affinities to LLM-generated concept descriptions faithfully measure core visual features.
    Used in Section 4.2 and Appendix D.1. Supported only by a 15-class Hard ImageNet foreground/background study; the paper itself notes failures in medical imaging.
  • ad hoc to paper Min-max normalized soft labels in Eq. (8)-(9) are a valid estimate of the oracle pc(y|x) and promote balanced learning.
    Eq. (8)-(9) define p~c(y|x). The normalization is a technical fix for 'artifact terms' and is not derived from first principles; it is validated only empirically through the ablation in Table 10.
  • standard math Strong duality holds for IDRM via convexity w.r.t. the prediction distribution and Slater's condition.
    Theorem 1 and Appendix A prove dual equivalence in the space of prediction distributions. Actual neural network fine-tuning optimizes non-convex parameters, so the theorem is motivation rather than a proof for the implemented method.
invented entities (2)
  • Core/non-core feature decomposition (Xc, Xn)
    purpose: Latent variables defining which visual features are robust; used to define pc(y|x) and to justify preserving core features.
    The decomposition is a modeling assumption. It is only indirectly supported by concept-description affinity experiments; the latent variables themselves are never directly measured.
  • Oracle feature extractor fc and oracle core-feature model pc(y|x)
    purpose: Theoretical benchmark model that predicts solely from core features; defines the ideal WRM proxy Rcs.
    Hypothetical constructs used in Section 3. They are never observed; the paper approximates them with CLIP affinities to LLM concept descriptions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models." pith.science (2026). https://pith.science/paper/MZU3LHF7

@misc{pith2026241119757,
  author       = {Pith},
  title        = {Pith review of: Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MZU3LHF7}},
  note         = {Machine review of arXiv:2411.19757}
}
read the original abstract

Fine-tuning foundation models often compromises their robustness to distribution shifts. To remedy this, most robust fine-tuning methods aim to preserve the pre-trained features. However, not all pre-trained features are robust and those methods are largely indifferent to which ones to preserve. We propose dual risk minimization (DRM), which combines empirical risk minimization with worst-case risk minimization, to better preserve the core features of downstream tasks. In particular, we utilize core-feature descriptions generated by LLMs to induce core-based zero-shot predictions which then serve as proxies to estimate the worst-case risk. DRM balances two crucial aspects of model robustness: expected performance and worst-case performance, establishing a new state of the art on various real-world benchmarks. DRM significantly improves the out-of-distribution performance of CLIP ViT-L/14@336 on ImageNet (75.9 to 77.1), WILDS-iWildCam (47.1 to 51.8), and WILDS-FMoW (50.7 to 53.1); opening up new avenues for robust fine-tuning. Our code is available at https://github.com/vaynexie/DRM .

Figures

Figures reproduced from arXiv: 2411.19757 by the authors.

Figure 1
Figure 1. Dual risk minimization (DRM) combines empirical risk minimization (ERM) and worst-case risk minimization (WRM) to complement their weaknesses. In this binary classifica￾tion task predicting if there are skis in a given image, (i) ERM underperforms when the core features of skis are clear but the non-core features such as background/context are spurious (i.e. negatively correlated with ski), and (ii) WRM underperform… view at source ↗
Figure 2
Figure 2. Concept descriptions better capture core features than default prompts. The affinities between images and default prompts (df) are not stable w.r.t. changes in image background (BG) containing non-core features and are insensitive to changes in image foreground (FG) containing core features, as indicated by the relative changes (gray numbers in parentheses) w.r.t. the affinities of the original images. In contrast, … view at source ↗
Figure 3
Figure 3. Concept description prompts (cd) yield affinities which are more robust to the change of context information than the affinities yielded by the default text prompts (df). D.2 Examples of repeated generations of concept descriptions To evaluate the stochasticity of LLM in generating concept descriptions, we repeatedly ask GPT-4 to generate concept descriptions for each iWildCam class. Below are some examples: White-l… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

89 extracted references · 61 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Varshney

    Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R. Varshney. Empirical or invariant risk minimization? a sample complexity perspective. In ICLR, 2021

    2021

  3. [3]

    Adapting to latent subgroup shifts via concepts and proxies

    Ibrahim Alabdulmohsin, Nicole Chiou, Alexander D’Amour, Arthur Gretton, Sanmi Koyejo, Matt J Kusner, Stephen R Pfohl, Olawale Salaudeen, Jessica Schrouff, and Katherine Tsai. Adapting to latent subgroup shifts via concepts and proxies. In AISTATS, pages 9637--9661. PMLR, 2023

    2023

  4. [4]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, volume 35, pages 23716--23736, 2022

    2022

  5. [5]

    The evolution of out-of-distribution robustness throughout fine-tuning

    Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of-distribution robustness throughout fine-tuning. arXiv preprint arXiv:2106.15831, 2021

    arXiv 2021

  6. [6]

    Invariant risk minimization

    Martin Arjovsky, L \'e on Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv:1907.02893, 2019

    arXiv 1907

  7. [7]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. NeurIPS, 32, 2019

    2019

  8. [8]

    Robust optimization, volume 28

    Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization, volume 28. Princeton university press, 2009

    2009

Show all 89 references
  1. [9]

    Robust solutions of optimization problems affected by uncertain probabilities

    Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59 0 (2): 0 341--357, 2013

    2013

  2. [10]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2 0 (3): 0 8, 2023

    2023

  3. [11]

    Generalizing from several related classification tasks to a new unlabeled sample

    Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In NeurIPS, 2011

    2011

  4. [12]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    2021 arXiv

  5. [13]

    Swad: Domain generalization by seeking flat minima

    Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In NeurIPS, volume 34, pages 22405--22418, 2021

    2021

  6. [14]

    Domain generalization by mutual-information regularization with pre-trained models

    Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual-information regularization with pre-trained models. In ECCV, pages 440--457. Springer, 2022

    2022

  7. [15]

    Disentangled prompt representation for domain generalization

    De Cheng, Zhipeng Xu, Xinyang Jiang, Nannan Wang, Dongsheng Li, and Xinbo Gao. Disentangled prompt representation for domain generalization. In CVPR, pages 23595--23604, 2024

    2024

  8. [16]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009

    2009

  9. [17]

    Statistics of robust optimization: A generalized empirical likelihood approach

    John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research, 46 0 (3): 0 946--969, 2021

    2021

  10. [18]

    u gelgen, Hamed Hassani, George J Pappas, and Bernhard Sch \

    Cian Eastwood, Alexander Robey, Shashank Singh, Julius Von K \"u gelgen, Hamed Hassani, George J Pappas, and Bernhard Sch \"o lkopf. Probable domain generalization via quantile risk minimization. In NeurIPS, volume 35, pages 17340--17358, 2022

    2022

  11. [19]

    If clip could talk: Understanding vision-language model representations through their preferred concept descriptions

    Reza Esfandiarpoor, Cristina Menghini, and Stephen H Bach. If clip could talk: Understanding vision-language model representations through their preferred concept descriptions. arXiv preprint arXiv:2403.16442, 2024

    2024 arXiv

  12. [20]

    Data determines distributional robustness in contrastive language image pre-training (clip)

    Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In ICML, pages 6216--6234. PMLR, 2022

    2022

  13. [21]

    Born again neural networks

    Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In ICML, pages 1607--1616. PMLR, 2018

    2018

  14. [22]

    Han Gao, Kaican Li, Weiyan Xie, Lin Zhi, Yongxiang Huang, Luning Wang, Caleb Chen Cao, and Nevin L. Zhang. Consistency regularization for domain generalization with logit attribution matching. In UAI, 2024 a

    2024

  15. [23]

    Out-of-domain robustness via targeted augmentations

    Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Out-of-domain robustness via targeted augmentations. In ICML, 2023

    2023

  16. [24]

    Clip-adapter: Better vision-language models with feature adapters

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132 0 (2): 0 581--595, 2024 b

    2024

  17. [25]

    Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning

    Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In CVPR, pages 1086--1095, 2017

    2017

  18. [26]

    Wichmann

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020

    2020

  19. [27]

    Finetune like you pretrain: Improved finetuning of zero-shot vision models

    Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In CVPR, pages 19338--19347, 2023

    2023

  20. [28]

    In search of lost domain generalization

    Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In ICLR, 2021

    2021

  21. [29]

    Spottune: transfer learning through adaptive fine-tuning

    Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In CVPR, pages 4805--4814, 2019

    2019

  22. [30]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, pages 8340--8349, 2021 a

    2021

  23. [31]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, pages 15262--15271, 2021 b

    2021

  24. [32]

    Refine myself by teaching myself: Feature refinement via self-knowledge distillation

    Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, and Il-Chul Moon. Refine myself by teaching myself: Feature refinement via self-knowledge distillation. In CVPR, pages 10664--10673, 2021

    2021

  25. [33]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904--4916. PMLR, 2021

    2021

  26. [34]

    Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization

    Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. arXiv preprint arXiv:1911.03437, 2019

    1911 arXiv

  27. [35]

    Last layer re-training is sufficient for robustness to spurious correlations

    Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In ICLR, 2023

    2023

  28. [36]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of scien...

    2017

  29. [37]

    Wilds: A benchmark of in-the-wild distribution shifts

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In ICML, pages 5637--5664. PMLR, 2021

    2021

  30. [38]

    Fine-tuning can distort pretrained features and underperform out-of-distribution

    Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In ICLR, 2022

    2022

  31. [39]

    Explicit inductive bias for transfer learning with convolutional networks

    Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In ICML, pages 2825--2834. PMLR, 2018

    2018

  32. [40]

    Learning causal semantic representation for out-of-distribution prediction

    Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie-Yan Liu. Learning causal semantic representation for out-of-distribution prediction. In NeurIPS, 2021

    2021

  33. [41]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, volume 36, 2023

    2023

  34. [42]

    Causality inspired representation learning for domain generalization

    Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, and Di Liu. Causality inspired representation learning for domain generalization. In CVPR, pages 8046--8056, 2022

    2022

  35. [43]

    Domain generalization using causal matching

    Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In ICML, pages 7313--7324. PMLR, 2021

    2021

  36. [44]

    Enhancing clip with gpt-4: Harnessing visual descriptions as prompts

    Mayug Maniparambil, Chris Vorster, Derek Molloy, Noel Murphy, Kevin McGuinness, and Noel E O'Connor. Enhancing clip with gpt-4: Harnessing visual descriptions as prompts. In ICCV, pages 262--271, 2023

    2023

  37. [45]

    Context-aware robust fine-tuning

    Xiaofeng Mao, Yufeng Chen, Xiaojun Jia, Rong Zhang, Hui Xue, and Zhao Li. Context-aware robust fine-tuning. IJCV, 132 0 (5): 0 1685--1700, 2024

    2024

  38. [46]

    Catastrophic interference in connectionist networks: The sequential learning problem

    Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109--165. Elsevier, 1989

    1989

  39. [47]

    Visual classification via description from large language models

    Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2022

    2022

  40. [48]

    Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization

    John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International conferen...

    2021

  41. [49]

    Representation learning via invariant causal mechanisms

    Jovana Mitrovic, Brian McWilliams, Jacob C Walker, Lars Holger Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. In ICLR, 2021

    2021

  42. [50]

    Hard imagenet: Segmentations for objects with strong spurious cues

    Mazda Moayeri, Sahil Singla, and Soheil Feizi. Hard imagenet: Segmentations for objects with strong spurious cues. NeurIPS, 35: 0 10068--10077, 2022

    2022

  43. [51]

    Domain generalization via invariant feature representation

    Krikamol Muandet, David Balduzzi, and Bernhard Sch \"o lkopf. Domain generalization via invariant feature representation. In ICML, 2013

    2013

  44. [52]

    Lipsum-ft: Robust fine-tuning of zero-shot models using random text guidance

    Giung Nam, Byeongho Heo, and Juho Lee. Lipsum-ft: Robust fine-tuning of zero-shot models using random text guidance. In ICLR, 2024

    2024

  45. [53]

    Towards calibrated robust fine-tuning of vision-language models

    Changdae Oh, Mijoo Kim, Hyesu Lim, Junhyeok Park, Euiseog Jeong, Zhi-Qi Cheng, and Kyungwoo Song. Towards calibrated robust fine-tuning of vision-language models. arXiv preprint arXiv:2311.01723, 2023

    2023 arXiv

  46. [54]

    Causal inference by using invariant prediction: identification and confidence intervals

    Jonas Peters, Peter B \"u hlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78 0 (5): 0 947--1012, 2016

    2016

  47. [55]

    Combined scaling for zero-shot transfer learning

    Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. Neurocomputing, 555: 0 126658, 2023

    2023

  48. [56]

    Ins-detclip: Aligning detection model to follow human-language instruction

    Renjie Pi, Lewei Yao, Jianhua Han, Xiaodan Liang, Wei Zhang, and Hang Xu. Ins-detclip: Aligning detection model to follow human-language instruction. In ICLR, 2024

    2024

  49. [57]

    Can domain adaptation make object recognition work for everyone? In CVPR, pages 3981--3988, 2022

    Viraj Prabhu, Ramprasaath R Selvaraju, Judy Hoffman, and Nikhil Naik. Can domain adaptation make object recognition work for everyone? In CVPR, pages 3981--3988, 2022

    2022

  50. [58]

    What does a platypus look like? generating customized prompts for zero-shot image classification

    Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV, pages 15691--15701, 2023

    2023

  51. [59]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  52. [60]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748--8763. PMLR, 2021

    2021

  53. [61]

    Do imagenet classifiers generalize to imagenet? In ICML, pages 5389--5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, pages 5389--5400. PMLR, 2019

    2019

  54. [62]

    Hashimoto, and Percy Liang

    Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In ICLR, 2020

    2020

  55. [63]

    The pitfalls of simplicity bias in neural networks

    Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In NeurIPS, volume 33, pages 9573--9585, 2020

    2020

  56. [64]

    How much can clip benefit vision-and-language tasks? In ICLR, 2022

    Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-language tasks? In ICLR, 2022

    2022

  57. [65]

    Clipood: Generalizing clip to out-of-distributions

    Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Generalizing clip to out-of-distributions. In ICML, pages 31716--31731. PMLR, 2023

    2023

  58. [66]

    Intriguing properties of neural networks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013

    2013 arXiv

  59. [67]

    Measuring robustness to natural distribution shifts in image classification

    Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. NeurIPS, 33: 0 18583--18599, 2020

    2020

  60. [68]

    Separating style and content

    Joshua Tenenbaum and William Freeman. Separating style and content. In NeurIPS, volume 9, 1996

    1996

  61. [69]

    Id and ood performance are sometimes inversely correlated on real-world datasets

    Damien Teney, Yong Lin, Seong Joon Oh, and Ehsan Abbasnejad. Id and ood performance are sometimes inversely correlated on real-world datasets. In NeurIPS, volume 36, 2023

    2023

  62. [70]

    Trainable projected gradient method for robust fine-tuning

    Junjiao Tian, Zecheng He, Xiaoliang Dai, Chih-Yao Ma, Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning. In CVPR, pages 7836--7845, 2023 a

    2023

  63. [71]

    Fast trainable projection for robust fine-tuning

    Junjiao Tian, Yen-Cheng Liu, James S Smith, and Zsolt Kira. Fast trainable projection for robust fine-tuning. NeurIPS, 36, 2023 b

    2023

  64. [72]

    Robustness may be at odds with accuracy

    Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, 2019

    2019

  65. [73]

    Statistical Learning Theory

    Vladimir Vapnik. Statistical Learning Theory. Wiley, 1998

    1998

  66. [74]

    Statistical decision functions which minimize the maximum risk

    Abraham Wald. Statistical decision functions which minimize the maximum risk. Annals of Mathematics, pages 265--280, 1945

    1945

  67. [75]

    Learning robust global representations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019

    2019

  68. [76]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR, pages 7959--7971, 2022

    2022

  69. [77]

    Demystifying clip data

    Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In ICLR, 2024

    2024

  70. [78]

    Learning concise and descriptive attributes for visual recognition

    An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In ICCV, pages 3090--3100, 2023

    2023

  71. [79]

    Language in a bottle: Language model guided concept bottlenecks for interpretable image classification

    Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR, pages 19187--19197, 2023

    2023

  72. [80]

    Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization

    Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization. In CVPR, pages 7947--7958, 2022

    2022

  73. [81]

    Continual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pages 3987--3995. PMLR, 2017

    2017

  74. [82]

    Side-tuning: a baseline for network adaptation via additive side networks

    Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In ECCV, pages 698--714. Springer, 2020

    2020

  75. [83]

    Be your own teacher: Improve the performance of convolutional neural networks via self distillation

    Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, pages 3713--3722, 2019

    2019

  76. [84]

    Self-distillation: Towards efficient and compact neural networks

    Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillation: Towards efficient and compact neural networks. TPAMI, 44 0 (8): 0 4388--4403, 2021

    2021

  77. [85]

    Zhang, Kaican Li, Han Gao, Weiyan Xie, Zhi Lin, Zhenguo Li, Luning Wang, and Yongxiang Huang

    Nevin L. Zhang, Kaican Li, Han Gao, Weiyan Xie, Zhi Lin, Zhenguo Li, Luning Wang, and Yongxiang Huang. A causal framework to unify common domain generalization approaches. arXiv preprint arXiv:2307.06825, 2023 a

    2023 arXiv

  78. [86]

    Pointclip: Point cloud understanding by clip

    Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In CVPR, pages 8552--8562, 2022

    2022

  79. [87]

    Connecting unseen domains: Cross-domain invariant learning in recommendation

    Yang Zhang, Yue Shen, Dong Wang, Jinjie Gu, and Guannan Zhang. Connecting unseen domains: Cross-domain invariant learning in recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1894--1898, 2023 b

    2023

  80. [88]

    Freelb: Enhanced adversarial training for natural language understanding

    Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In ICLR, 2020

    2020

  81. [89]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.