Pith. sign in

REVIEW 5 major objections 5 minor 60 references

Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An unsupervised alignment score, VEGA, ranks vision-language models by downstream zero-shot accuracy using only unlabeled data and class names.

desk verdict A sensible new problem and useful benchmarks, but VEGA's graph alignment is mostly a sharpened confidence score and the edge term is mis-signed; worth reviewing but needs major revision. read the letter →

arxiv 2412.20682 v1 pith:LOKKHRUF submitted 2024-12-30 cs.CV cs.LG

Yuhe Ding , Bo Jiang , Aihua Zheng , Qin Xu , Jian Liang This is my paper · ORCID
classification cs.CVcs.LG
keywords unsupervisedmodelselectionvision-languagemodelszero-shotclassificationcross-modalalignmentgraphperformancepredictionrankingCLIP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a practical problem: given a set of candidate vision-language models (VLMs) such as CLIP-style models, and only an unlabeled downstream dataset with class names, which model will give the best zero-shot classification accuracy? The authors propose a score, VEGA (Visual-tExtual Graph Alignment), that ranks the candidates by measuring how aligned the visual and textual feature structures are in the model's shared representation space. They claim that VLMs with a higher VEGA score are more likely to achieve better downstream performance, and they validate this on three benchmarks: CLIP-family checkpoints, models from different pre-training algorithms, and model–prompt-template combinations. If right, a user can choose a VLM for a new task without labels, large language models, or an annotated reference set like ImageNet.

What carries the argument

The load-bearing object is the pair of class-level graphs constructed separately from text prompts and from unlabeled images. The textual graph uses class text features as nodes and cosine similarities as edges; the visual graph is built by pseudo-labeling each image with the candidate model's own highest-similarity class, fitting a Gaussian to each cluster, and using Bhattacharyya distances between clusters as edges. The VEGA score is the sum of node similarity (softmax-normalized closeness of cluster members to the corresponding text node) and edge similarity (Pearson correlation between edge matrices). This graph-alignment construction is what ties the unlabeled data to the model's zero-shot accuracy without any labels.

What would settle it

Build a candidate pool of several VLMs plus a deliberately broken model whose text encoder is replaced or perturbed so its class embeddings are wrong but internally consistent. If VEGA ranks the broken model highly (because its visual pseudo-clusters align well with its own text graph) while the model's true zero-shot accuracy is low, the central claim fails. A cleaner measurement: restrict to candidate pools where every model's accuracy is below a modest threshold, and check whether the VEGA–accuracy correlation survives or collapses.

Watch

Extended reading notes

Core claim

VEGA is defined as $s = s_n + s_e$, the sum of a node-level and an edge-level similarity between two graphs built in the shared cross-modal feature space. The textual graph places each class's text-prompt feature as a node and connects classes by cosine similarity. The visual graph assigns each unlabeled image to the class whose text feature is most similar, models each resulting cluster as a Gaussian, and connects classes by Bhattacharyya distance between these Gaussians. Node similarity $s_n$ is the weighted average softmax-normalized closeness of images to their assigned class text feature, and edge similarity $s_e$ is the Pearson correlation between the two edge matrices, rescaled to $[0,1]$. The paper's claim is that this summed graph alignment serves as a reliable estimator of the VLM's zero-shot accuracy on the unlabeled task.

Load-bearing premise

The method assumes that the candidate model's own pseudo-labels — assigning every image to its most similar class name — are accurate enough that the visual clusters and their distances reflect true class structure; for a weak or biased model, the graph becomes an artifact of that model's own mistakes.

Editorial extensions

If this is right

  • A practitioner can rank an entire model zoo for a target dataset using only the unlabeled images and class names, with no annotation effort or LLM API calls.
  • The score also ranks prompt-template and model combinations, so it can guide prompt selection in deployment without validation labels.
  • The score transfers across model families: it was evaluated on CLIP checkpoints, on models from different pre-training algorithms, and on template combinations, with consistently higher correlation than training-data-free baselines.
  • Because VEGA requires only forward passes and no backpropagation, it is cheap enough for mid-range GPUs and CPUs, as the paper notes.
  • The ablation shows node and edge similarities contribute different information, so combining them is what gives the reliable ranking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The self-referential pseudo-labeling means VEGA could be fooled by a model whose errors are self-consistent; a natural stress test is to evaluate candidate pools with deliberately mismatched text encoders.
  • The reported correlations may be driven by contrast between clearly good and clearly bad models; whether VEGA preserves fine-grained ranking among a pool of uniformly mediocre models is not settled by the paper's benchmarks.
  • The visual graph could be made less dependent on the candidate's own pseudo-labels by using an external or ensembled clustering, though that would violate the strict information limit of the proposed setting.
  • One could test the score as a stopping rule: rank checkpoints during VLM training and check whether VEGA tracks the accuracy trajectory, which would extend the method to model development rather than only deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces the problem of unsupervised vision-language model selection (UVMS), where a practitioner must rank candidate VLMs on an unlabeled downstream dataset using only the dataset and class names. The proposed method, VEGA, constructs a textual graph whose nodes are class-name text features and a visual graph whose nodes are Gaussian models of pseudo-labeled image features, with edges given by cosine similarity and Bhattacharyya distance, respectively. The VEGA score combines a node-level term (Eq. 11-12) and an edge-level term (Eq. 13-14), and the paper reports experiments on three benchmarks: 31 OpenCLIP models, 17 models from different pretraining algorithms, and 100 model-template combinations, across 10 datasets. The paper claims VEGA 'consistently provides reliable and accurate estimates of VLMs' performance on unlabeled downstream tasks' and outperforms existing training-data-free baselines.

Significance. If the central claim held, VEGA would be a practical, annotation-free, LLM-free and training-data-free tool for selecting among the rapidly growing zoo of open VLMs, and the three benchmarks would be a useful resource for the community. The paper also reports a commendably broad experimental sweep (31 CLIP models, 17 algorithm models, 100 model-template combinations, 10 datasets) and states that the implementation and predicted scores are provided in the supplementary material, which supports reproducibility. However, the claimed graph-alignment mechanism is not supported by the mathematics of the score: the node term reduces to a rescaling of the standard Confidence baseline, and the edge term has a sign inconsistency and is shown by the paper's own ablation to be weak or harmful. The empirical gains over Confidence are modest and are reported without any error bars or significance tests, so the distinctive contribution of the paper is not yet established.

major comments (5)
  1. [§IV-B, Eqs. (11)-(12)] The node term sn is algebraically equivalent to a per-dataset rescaling of the Confidence baseline in Eq. (16). Because the indicator I(yhat_i = c_k) partitions the image indices, the double sum collapses: sn = (1/K) * sum_i max_k softmax_k(phi(x_i); t), where the softmax uses temperature t. Thus sn carries no graph-structure information whatsoever, and the statement in the text that 'the range of node similarity sn is constrained to the range of 0 to 1' is incorrect; the range is [0, N/K]. Consequently, the claimed novelty of VEGA as a graph-alignment method rests entirely on the edge term, whose contribution the ablation does not support (see below).
  2. [§IV-B, Eq. (13)-(14) and Textual Graph definition] The textual graph edges are defined as cosine similarity (eT_ij = cos(...)), while the visual graph edges are Bhattacharyya distances (Eq. (10)). The Pearson correlation in Eq. (13) is computed between a similarity matrix and a distance matrix. For two aligned graphs, one expects high visual distance to correspond to low textual similarity, i.e., a negative correlation; under the current sign convention, an aligned model would receive a small se, not a large one. Either the sign in Eq. (13) is wrong, or the experiments must be using cosine distance for the textual edges. This ambiguity must be resolved before the edge contribution and the 'graph alignment' story can be evaluated.
  3. [§V-D, Table IV] The ablation does not support the design s = sn + se. The text claims 'In all cases, the full VEGA method ... achieves the highest predictive accuracy', but in benchmark (a) the full method has tau5 = 0.20 versus 0.49 for sn alone, and in benchmark (c) the full method has R5 = 0.36 versus 0.42 and tau5 = -0.07 versus 0.03 for sn alone. The edge-only term has negative or near-zero tau5 in benchmarks (a) and (c). These numbers contradict the stated conclusion and suggest that the edge term, as defined and signed, does not provide the claimed benefit.
  4. [§V-A to §V-C] No error bars, confidence intervals, or significance tests are reported for any of the ranking metrics. This is particularly important because the improvement over the Confidence baseline is small in several cases: in Table I the average tau gain is 0.62 vs 0.55 (VEGA vs Conf), and in Table III the average R5 is exactly tied at 0.36. Since sn is mathematically the Confidence score up to temperature and a per-dataset constant, the observed gains could plausibly be due to the temperature choice alone. Without a significance analysis or at least repeated subsampling, the claim that VEGA 'consistently' outperforms Confidence is not established.
  5. [§IV-B, Eqs. (8)-(10)] The visual graph is built from each candidate model's own pseudo-labels (argmax class assignments), so for low-accuracy models the Gaussian class nodes and the Bhattacharyya edges are artifacts of the model being evaluated. The paper does not validate this assumption for weak models, even though model selection is precisely the regime where weak models are in the candidate set. A concrete test would be to report the correlation between VEGA and accuracy separately for the bottom half of models by accuracy; if the correlation disappears or reverses there, the method's applicability to a realistic model zoo is limited.
minor comments (5)
  1. [§IV-B, Eq. (10)] The quantity in Eq. (10) is called the Bhattacharyya coefficient, but the formula is the Bhattacharyya distance (the expression with the 1/8 and log terms). Please use consistent terminology.
  2. [§V-D, Table IV] The table header lists 'R5, tau5, tau, Top-1 Acc.' for each of the three methods, but the text also refers to 'R2 and rho values'; the table and the text should use the same metric names, and the column layout is visually ambiguous.
  3. [§IV-A, Eq. (7)] In the displayed formula for the VLM pretraining loss, the fractions inside the logarithms are missing a visible fraction bar; the two log terms appear as products rather than ratios. This is a LaTeX rendering issue that should be fixed.
  4. [§V-A] The paper states that the candidate CLIP models are 'the same as those used in LOVM [8]' and defers details to the supplementary material; since model identity and prompt-template averaging can affect reproducibility, at least a table of the 31 model names and source datasets should appear in the main text or an appendix.
  5. [§V-B] LOVM-G and LOVM-C are excluded from the algorithm-model benchmark because the authors 'cannot guarantee the reliability of our reproducible results'; this is a reasonable caution, but it should be stated more explicitly in the main text rather than as a sentence in the experimental setup, since it removes two of the paper's principal baselines from one benchmark.

Circularity Check

1 steps flagged · score 6.0 of 10

VEGA's node similarity sn reduces by construction to a per-dataset rescaling of the Confidence baseline (Eqs. 11-12 vs. Eq. 16); the claimed graph-alignment advantage is largely a temperature-sharpened confidence score.

  1. renaming known result [Sec. IV-B, Eqs. (11)-(12) and Eq. (16); ablation Table IV]
    "sn = 1 K PK i=1 sim(nT k , nV k ) · Nk, ... sim(nT k , nV k ) = 1 Nk PN i=1 exp(cos(ϕ(xi), ξ(ck))/t) P K k′=1 exp(cos(ϕ(xi), ξ(ck′))/t) · I( ˆyi = ck) ... sConf = 1 N PN i=1 max({P (xi)[k]}K k=1)"

    Because I(ŷ_i=c_k) is nonzero only for the argmax class, summing over k turns Σ_k softmax_k·I(argmax=k) into max_k softmax_k. Thus sn = (1/K) Σ_i max_k softmax_t(x_i) = (N/K) times the average max-softmax confidence at temperature t=0.05. For a fixed downstream dataset, N and K are constant across candidate models, so the ranking produced by sn is identical to that of the Confidence baseline in Eq. (16) up to a per-dataset scale factor and temperature. The node term therefore carries no graph-structure information, and Table IV shows it is nearly the whole method: for benchmark (a), sn alone gives R5=0.62 and τ=0.60 vs. full VEGA's 0.64 and 0.62, while se alone has τ5=-0.13.

full rationale

VEGA does not fit any parameter to the ground-truth accuracy, and it uses no labels or external supervised data, so there is no target-fitting circularity. The self-citation [18] (a transferability survey by the same authors) is used only for background and is not load-bearing. However, the central node-level term of VEGA is not new: substituting the argmax indicator in Eq. (12) makes Eq. (11) collapse to the average maximum softmax confidence, which is exactly the Confidence baseline of Eq. (16) under temperature t=0.05 and a per-dataset constant. Because the ablation shows that sn dominates the full score and se alone is weak and even negatively correlated on τ5 for benchmark (a), the paper's headline result is substantially a renaming of the Confidence baseline rather than an independent graph-alignment mechanism. The edge term is a genuine additional component, but its reported contribution is small and, separately, its sign is suspect since it correlates cosine similarities with Bhattacharyya distances. Overall, the reduction of sn to the Confidence baseline is a partial circularity that inflates the apparent novelty of VEGA; the score is therefore 6, not higher, because the full method retains an independent (if weak) edge component and no external fitted constants enter the derivation.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The method introduces one hand-set hyperparameter, temperature t=0.05. It relies on several domain assumptions about CLIP-like embeddings and pseudo-label reliability. No new physical or ontological entities are introduced; VEGA is a score, not an entity.

free parameters (1)
  • temperature t in node similarity (Eq. 12) = 0.05
    Sharpening temperature for the softmax over text-class similarities; chosen by hand and held fixed across all datasets and models. Sensitivity analysis (Fig. 6) shows stable performance across t=0.005 to 0.5, reducing but not eliminating the concern.
assumptions (5)
  • domain assumption Contrastive pretraining (Eq. 7) maps images and texts with the same semantics into a shared space where they cluster together.
    Invoked in Sec. IV-A as the motivation for measuring modality graph alignment.
  • domain assumption Improved cross-modal alignment implies improved zero-shot classification accuracy.
    Sec. IV-A states that zero-shot classification performance improves as alignment improves; this monotonic link is assumed, not derived.
  • domain assumption The candidate model's own argmax pseudo-labels (Eq. 8) are accurate enough to define meaningful class clusters.
    Visual graph nodes and node similarity depend on these pseudo-labels; no validation is provided for low-accuracy models.
  • ad hoc to paper Visual features within each pseudo-class are approximately Gaussian, so the Bhattacharyya distance (Eq. 10) is a meaningful edge measure.
    Eq. (10) models each class as N(nV_k, Sigma_k); real CLIP feature distributions are not shown to be Gaussian.
  • standard math Pearson correlation between text-edge and visual-edge matrices removes scale differences and preserves ranking information.
    Eqs. (13)-(14) use Pearson correlation without justification beyond scale invariance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks." pith.science (2026). https://pith.science/paper/LOKKHRUF

@misc{pith2026241220682,
  author       = {Pith},
  title        = {Pith review of: Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LOKKHRUF}},
  note         = {Machine review of arXiv:2412.20682}
}
read the original abstract

Vision language models (VLMs) like CLIP show stellar zero-shot capability on classification benchmarks. However, selecting the VLM with the highest performance on the unlabeled downstream task is non-trivial. Existing VLM selection methods focus on the class-name-only setting, relying on a supervised large-scale dataset and large language models, which may not be accessible or feasible during deployment. This paper introduces the problem of \textbf{unsupervised vision-language model selection}, where only unsupervised downstream datasets are available, with no additional information provided. To solve this problem, we propose a method termed Visual-tExtual Graph Alignment (VEGA), to select VLMs without any annotations by measuring the alignment of the VLM between the two modalities on the downstream task. VEGA is motivated by the pretraining paradigm of VLMs, which aligns features with the same semantics from the visual and textual modalities, thereby mapping both modalities into a shared representation space. Specifically, we first construct two graphs on the vision and textual features, respectively. VEGA is then defined as the overall similarity between the visual and textual graphs at both node and edge levels. Extensive experiments across three different benchmarks, covering a variety of application scenarios and downstream datasets, demonstrate that VEGA consistently provides reliable and accurate estimates of VLMs' performance on unlabeled downstream tasks.

Figures

Figures reproduced from arXiv: 2412.20682 by the authors.

Figure 1
Figure 1. Paradigm of unsupervised vision language model selection, where [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of VEGA involves encoding class names and unlabeled images into a shared cross-modality feature space. Subsequently, we construct a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the correlation between the actual zero-shot classification accuracy and predicted scores for various VLMs in the CLIP family. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Visualization of correlations between the actual zero-shot classification accuracy and the predicted scores for VLMs from various popular pre-training [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualization of correlations between the actual downstream accuracy and the predicted scores across combinations of model and prompt template. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Sensitivity analysis on temperature t in Eq. (12). The Y-axis is the average results of the prediction for VLMs from CLIP family (Sec V-A). VI. CONCLUSION This paper introduces a novel method called Visual-tExtual Graph Alignment (VEGA) for unsupervised vision language…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 47 canonical work pages

  1. [1]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in Proc. ICML, 2021, pp. 8748–8763

    work page 2021

  2. [2]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in Proc. ICML , 2021, pp. 4904–4916

    work page 2021

  3. [3]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Proc. ICCV, 2023, pp. 11 975–11 986

    work page 2023

  4. [4]

    Sgva-clip: Semantic- guided visual adapting of vision-language models for few-shot image classification,

    F. Peng, X. Yang, L. Xiao, Y . Wang, and C. Xu, “Sgva-clip: Semantic- guided visual adapting of vision-language models for few-shot image classification,” IEEE Transactions on Multimedia , vol. 26, pp. 3469– 3480, 2023

    work page 2023

  5. [5]

    Clip-vg: Self-paced curriculum adapting of clip for visual grounding,

    L. Xiao, X. Yang, F. Peng, M. Yan, Y . Wang, and C. Xu, “Clip-vg: Self-paced curriculum adapting of clip for visual grounding,” IEEE Transactions on Multimedia , vol. 26, pp. 4334–4347, 2023

    work page 2023

  6. [6]

    Effective end-to-end vision language pre- training with semantic visual loss,

    X. Yang, F. Liu, and G. Lin, “Effective end-to-end vision language pre- training with semantic visual loss,” IEEE Transactions on Multimedia , vol. 25, pp. 8408–8417, 2023

    work page 2023

  7. [7]

    Neural logic vision language explainer,

    ——, “Neural logic vision language explainer,” IEEE Transactions on Multimedia, vol. 26, pp. 3331–3340, 2024

    work page 2024

  8. [8]

    Lovm: Language- only vision model selection,

    O. Zohar, S.-C. Huang, K.-C. Wang, and S. Yeung, “Lovm: Language- only vision model selection,” in Proc. NeurIPS Workshops, 2024

    work page 2024

Show all 60 references
  1. [9]

    Bridge the modality and capacity gaps in vision-language model selection,

    C. Yi, D.-C. Zhan, and H.-J. Ye, “Bridge the modality and capacity gaps in vision-language model selection,” arXiv preprint arXiv:2403.13797 , 2024

    2024 arXiv

  2. [10]

    Imagenet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, pp. 211–252, 2015

    2015

  3. [11]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” in arXiv preprint arXiv:2303.08774 , 2024

    2024 arXiv

  4. [12]

    Leveraging unlabeled data to predict out-of-distribution performance,

    S. Garg, S. Balakrishnan, Z. C. Lipton, B. Neyshabur, and H. Sedghi, “Leveraging unlabeled data to predict out-of-distribution performance,” in Proc. ICLR, 2022

    2022

  5. [13]

    Are labels always necessary for classifier accuracy evaluation?

    W. Deng and L. Zheng, “Are labels always necessary for classifier accuracy evaluation?” in Proc. CVPR, 2021, pp. 15 069–15 078

    2021

  6. [14]

    Predicting out-of- distribution error with the projection norm,

    Y . Yu, Z. Yang, A. Wei, Y . Ma, and J. Steinhardt, “Predicting out-of- distribution error with the projection norm,” in Proc. ICML, 2022, pp. 25 721–25 746

    2022

  7. [15]

    Data determines distributional robustness in contrastive language image pre-training (clip),

    A. Fang, G. Ilharco, M. Wortsman, Y . Wan, V . Shankar, A. Dave, and L. Schmidt, “Data determines distributional robustness in contrastive language image pre-training (clip),” in Proc. ICML , 2022, pp. 6216– 6234

    2022

  8. [16]

    Does clip’s generalization performance mainly stem from high train- test similarity?

    P. Mayilvahanan, T. Wiedemer, E. Rusak, M. Bethge, and W. Brendel, “Does clip’s generalization performance mainly stem from high train- test similarity?” in Proc. ICLR, 2024

    2024

  9. [17]

    A survey on evaluation of out-of-distribution generalization,

    H. Yu, J. Liu, X. Zhang, J. Wu, and P. Cui, “A survey on evaluation of out-of-distribution generalization,” in arXiv preprint arXiv:2403.01874 , 2024

    2024 arXiv

  10. [18]

    Which model to transfer? a survey on transferability estimation,

    Y . Ding, B. Jiang, A. Yu, A. Zheng, and J. Liang, “Which model to transfer? a survey on transferability estimation,” arXiv preprint arXiv:2402.15231, 2024

    2024 arXiv

  11. [19]

    Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank,

    Q. Garrido, R. Balestriero, L. Najman, and Y . Lecun, “Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank,” in Proc. ICML, 2023, pp. 10 929–10 974

    2023

  12. [20]

    Identifying useful learnwares for heterogeneous label spaces,

    L.-Z. Guo, Z. Zhou, Y .-F. Li, and Z.-H. Zhou, “Identifying useful learnwares for heterogeneous label spaces,” in Proc. ICML, 2023, pp. 12 122–12 131

    2023

  13. [21]

    Etran: Energy-based transferability estimation,

    M. Gholami, M. Akbari, X. Wang, B. Kamranian, and Y . Zhang, “Etran: Energy-based transferability estimation,” in Proc. ICCV, 2023, pp. 18 613–18 622

    2023

  14. [22]

    Predicting out-of-distribution error with confidence optimal transport,

    Y . Lu, Z. Wang, R. Zhai, S. Kolouri, J. Campbell, and K. Sycara, “Predicting out-of-distribution error with confidence optimal transport,” in Proc. NeurIPS, 2023

    2023

  15. [23]

    Data analysis and regression. a second course in statistics,

    F. Mosteller and J. W. Tukey, “Data analysis and regression. a second course in statistics,” Addison-Wesley series in behavioral science: quan- titative methods, 1977

    1977

  16. [24]

    Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density,

    K. Saito, D. Kim, P. Teterwak, S. Sclaroff, T. Darrell, and K. Saenko, “Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density,” in Proc. ICCV, 2021, pp. 9184–9193

    2021

  17. [25]

    Covariate shift adap- tation by importance weighted cross validation

    M. Sugiyama, M. Krauledat, and K.-R. M ¨uller, “Covariate shift adap- tation by importance weighted cross validation.” Journal of Machine Learning Research, vol. 8, no. 5, 2007

    2007

  18. [26]

    Towards accurate model selection in deep unsupervised domain adaptation,

    K. You, X. Wang, M. Long, and M. Jordan, “Towards accurate model selection in deep unsupervised domain adaptation,” inProc. ICML, 2019, pp. 7124–7133

    2019

  19. [27]

    Stochastic gradient methods for dis- tributionally robust optimization with f-divergences,

    H. Namkoong and J. C. Duchi, “Stochastic gradient methods for dis- tributionally robust optimization with f-divergences,” in Proc. NeurIPS, 2016

    2016

  20. [28]

    Invariant risk minimization,

    M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, “Invariant risk minimization,” in arXiv preprint arXiv:1907.02893 , 2019

    1907 arXiv

  21. [29]

    Stable learning via sample reweighting,

    Z. Shen, P. Cui, T. Zhang, and K. Kunag, “Stable learning via sample reweighting,” in Proc. AAAI, 2020, pp. 5692–5699

    2020

  22. [30]

    A baseline for detecting misclassified and out-of-distribution examples in neural networks,

    D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in Proc. ICLR , 2017

    2017

  23. [31]

    What does rotation prediction tell us about classifier accuracy under varying testing environments?

    W. Deng, S. Gould, and L. Zheng, “What does rotation prediction tell us about classifier accuracy under varying testing environments?” in Proc. ICML, 2021, pp. 2579–2589

    2021

  24. [32]

    Agreement-on-the- line: Predicting the performance of neural networks under distribution shift,

    C. Baek, Y . Jiang, A. Raghunathan, and J. Z. Kolter, “Agreement-on-the- line: Predicting the performance of neural networks under distribution shift,” in Proc. NeurIPS, 2022, pp. 19 274–19 289

    2022

  25. [33]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in Proc. ICML, 2022, pp. 12 888–12 900

    2022

  26. [34]

    I. mathematical contributions to the theory of evolu- tion.—vii. on the correlation of characters not quantitatively measur- able,

    K. Pearson, “I. mathematical contributions to the theory of evolu- tion.—vii. on the correlation of characters not quantitatively measur- able,” Philosophical Transactions of the Royal Society of London. , vol. 195, no. 262-273, pp. 1–47, 1900

    1900

  27. [35]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, “Learning multiple layers of features from tiny images,” Master’s thesis, University of Tront , 2009

    2009

  28. [36]

    Cats and dogs,

    O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, “Cats and dogs,” in Proc. CVPR, 2012, pp. 3498–3505

    2012

  29. [37]

    Automated flower classification over a large number of classes,

    M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in Proc. ICVGIP, 2008, pp. 722–729

    2008

  30. [38]

    Reading digits in natural images with unsupervised feature learning,

    Y . Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y . Ng et al. , “Reading digits in natural images with unsupervised feature learning,” in Proc. NeurIPS Workshops, 2011

    2011

  31. [39]

    Detection of traffic signs in real-world images: The german traffic sign detection benchmark,

    S. Houben, J. Stallkamp, J. Salmen, M. Schlipsing, and C. Igel, “Detection of traffic signs in real-world images: The german traffic sign detection benchmark,” in Proc. IJCNN, 2013, pp. 1–8

    2013

  32. [40]

    Describing textures in the wild,

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, “Describing textures in the wild,” in Proc. CVPR, 2014, pp. 3606–3613

    2014

  33. [41]

    Yfcc100m: The new data in multimedia research,

    B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li, “Yfcc100m: The new data in multimedia research,” Communications of the ACM, vol. 59, no. 2, pp. 64–73, 2016

    2016

  34. [42]

    Sun database: Large-scale scene recognition from abbey to zoo,

    J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “Sun database: Large-scale scene recognition from abbey to zoo,” in Proc. CVPR, 2010, pp. 3485–3492

    2010

  35. [43]

    Gradient-based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” in Proc. IEEE, 1998, pp. 2278–2324

    1998

  36. [44]

    Challenges in representation learning: Facial expression recognition challenge,

    Dumitru, G. Ian, C. Will, and B. Yoshua, “Challenges in representation learning: Facial expression recognition challenge,” 2013. [Online]. Available: https://kaggle.com/competitions/ challenges-in-representation-learning-facial-expression-recognition-challenge

    2013

  37. [45]

    On the importance of feature separability in predicting out-of-distribution error,

    R. Xie, H. Wei, L. Feng, Y . Cao, and B. An, “On the importance of feature separability in predicting out-of-distribution error,” in Proc. NeurIPS, 2023

    2023

  38. [46]

    Unsupervised representation learning by predicting image rotations,

    S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” in Proc. ICLR, 2018

    2018

  39. [47]

    The use of multiple measurements in taxonomic prob- lems,

    R. A. Fisher, “The use of multiple measurements in taxonomic prob- lems,” Annals of eugenics , vol. 7, no. 2, pp. 179–188, 1936

    1936

  40. [48]

    Silhouettes: a graphical aid to the interpretation and validation of cluster analysis,

    P. J. Rousseeuw, “Silhouettes: a graphical aid to the interpretation and validation of cluster analysis,” Journal of computational and applied mathematics, vol. 20, pp. 53–65, 1987

    1987

  41. [49]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. CVPR, 2016, pp. 770–778

    2016

  42. [50]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. ICLR, 2021

    2021

  43. [51]

    A convnet for the 2020s,

    Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in Proc. CVPR, 2022, pp. 11 976–11 986. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11

    2022

  44. [52]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,

    C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, “Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,” in Proc. NeurIPS Workshops, 2021

    2021

  45. [53]

    AltCLIP: Altering the language encoder in CLIP for extended language capabilities,

    Z. Chen, G. Liu, B.-W. Zhang, Q. Yang, and L. Wu, “AltCLIP: Altering the language encoder in CLIP for extended language capabilities,” in Proc. ACL, 2023, pp. 8666–8682

    2023

  46. [54]

    Groupvit: Semantic segmentation emerges from text supervision,

    J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang, “Groupvit: Semantic segmentation emerges from text supervision,” in Proc. CVPR, 2022, pp. 18 134–18 144

    2022

  47. [55]

    Learning generalized zero-shot learners for open-domain image geolocalization,

    L. Haas, S. Alberti, and M. Skreta, “Learning generalized zero-shot learners for open-domain image geolocalization,” in arXiv preprint arXiv:2302.00275, 2023

    2023 arXiv

  48. [56]

    Demystifying clip data,

    H. Xu, S. Xie, X. E. Tan, P.-Y . Huang, R. Howes, V . Sharma, S.-W. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer, “Demystifying clip data,” in arXiv preprint arXiv:2309.16671 , 2023

    2023 arXiv

  49. [57]

    Large-scale domain-specific pretraining for biomedical vision-language processing,

    S. Zhang, Y . Xu, N. Usuyama, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wong, M. Lungren, T. Naumann, and H. Poon, “Large-scale domain-specific pretraining for biomedical vision-language processing,” in arXiv preprint arXiv:2303.00915 , 2023

    2023 arXiv

  50. [58]

    Quilt-1m: One million image-text pairs for histopathology,

    W. O. Ikezogwo, M. S. Seyfioglu, F. Ghezloo, D. S. C. Geva, F. S. Mohammed, P. K. Anand, R. Krishna, and L. Shapiro, “Quilt-1m: One million image-text pairs for histopathology,” in arXiv preprint arXiv:2306.11207, 2023

    2023 arXiv

  51. [59]

    BioCLIP: A vision foundation model for the tree of life,

    S. Stevens, J. Wu, M. J. Thompson, E. G. Campolongo, C. H. Song, D. E. Carlyn, L. Dong, W. M. Dahdul, C. Stewart, T. Berger-Wolf, W.- L. Chao, and Y . Su, “BioCLIP: A vision foundation model for the tree of life,” in Proc. CVPR, 2024

    2024

  52. [60]

    Gpt-4: Generative pre-trained transformer,

    OpenAI, “Gpt-4: Generative pre-trained transformer,” https://openai. com/gpt-4, 2023, accessed: 2024-08-14

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.