Pith. sign in

REVIEW 5 major objections 6 minor 19 references

Occam's model: Selecting simpler representations for better transferability estimation

T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that a pre-trained model's transferability to a target classification task is determined by how simply its frozen embeddings separate the target classes, and introduces two embedding-only metrics—INT and Concept…

desk verdict Plausible idea, big experiment, but the main metric is scale-dependent and the headline gains are unquantified — worth refereeing, not worth relying on yet. read the letter →

arxiv 2502.06925 v1 pith:ZZILQQEX submitted 2025-02-10 cs.LG cs.AI

Prabhant Singh , Sibylle Hess , Joaquin Vanschoren This is my paper · ORCID
classification cs.LGcs.AI
keywords transferabilityestimationpre-trainedmodelselectionrepresentationcomplexityinterclassdistanceconceptvarianceKendall'staufine-tuningsource-independent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the best pre-trained model to fine-tune for a target classification task is the one whose frozen embeddings already look simplest with respect to that task's labels—classes well separated, local label structure regular. It tackles the source-independent transferability setting, where a model zoo and a labeled target set are available but the models' source data are not, and fine-tuning every candidate is too expensive. The proposed metrics, INT (average inter-class embedding distance) and Concept Variance (spread of distance-weighted neighbor-label disagreement), are computed from one forward pass over the target data. On a new seven-domain benchmark of modern vision transformers, both rank fine-tuned models by accuracy better than state-of-the-art transferability metrics, with INT improving weighted Kendall's tau by up to 32% over the strongest baselines and running faster than the leading competitors. If the simplicity hypothesis holds, these scores give practitioners a cheap screening tool for the large public model repositories where the best candidate is otherwise found only by brute-force fine-tuning.

What carries the argument

The load-bearing identity is Theorem 3.1: any multinomial logistic regression model $h_{W,b}(x)=\mathrm{softmax}(Wx+b)$ can be written with class centers $\mu_1,\ldots,\mu_C$ as $h_{W,b}(x)_y=\exp(-\frac{1}{2}\|x-\mu_y\|^2)/\sum_{c}\exp(-\frac{1}{2}\|x-\mu_c\|^2)$. This recasts cross-entropy training as a nearest-centre problem, so the loss on a point is approximately $-\frac{1}{2}\|\phi(x_n)-\mu_c\|^2$ against the nearest other class, making interclass distance a proxy for how easily a linear head can separate the embeddings. INT operationalizes this as the pairwise-normalized sum of mean Euclidean distances between classes, and Concept Variance adapts a Boolean-domain regularity measure by weighting neighbour pairs with $2^{-\alpha D_{ij}/(\sqrt{d}-D_{ij})}$ and scoring the standard deviation of per-example label disagreement.

What would settle it

On the paper's benchmark, a falsifying observation would be a target dataset and model pair where the model ranked first by INT (or Concept Variance) fine-tunes to below-median accuracy while a model ranked last fine-tunes to above-median accuracy; such a pair would directly contradict the claimed monotonic relationship between embedding simplicity and post-fine-tuning performance.

Watch

Extended reading notes

Core claim

The central claim is that transferability is an emergent property of representation geometry: a pre-trained feature extractor will fine-tune well to a target dataset exactly when its embeddings for that dataset already separate the target classes with little effort. The paper proves that any multinomial logistic regression classifier can be re-expressed as a normalized nearest-centre classifier, so cross-entropy training is equivalent to pulling each class's points toward its centre and away from other centres, making interclass distance a proxy for how easily the head can be fine-tuned. The paper therefore defines INT as the average pairwise distance between embeddings of different classes, and Concept Variance as the standard deviation of a per-example, distance-weighted measure of how often neighbours carry different labels. On the authors' benchmark of seven domains and modern transformer backbones, INT achieves the highest average weighted Kendall's tau in the main image-classification and limited-data settings, outperforming LogME, SFDA, N-LEEP, ETran, TransRate, and NCTI; the combined INT+Concept Variance score also leads in several settings, and both metrics are several times faster than the strongest baselines.

Load-bearing premise

The load-bearing premise is that an embedding that already separates target classes cleanly before fine-tuning is the one that will fine-tune to the highest accuracy; no proof links that frozen-geometry condition to post-fine-tuning performance.

Editorial extensions

If this is right

  • On the paper's seven-domain benchmark, ranking with INT gives an average weighted Kendall's tau of 0.70 in the main image-classification experiment, against 0.38 for the best baseline (TransRate).
  • Both scores are cheap enough for large-scale screening: the reported wall-clock time is 1.28 seconds for INT and 0.20 seconds for Concept Variance on the test setup, versus 6.32 seconds for SFDA and 56.35 seconds for TransRate.
  • In the limited-data setting with 40 examples per class, combining INT with Concept Variance reaches an average tau of 0.65, better than either score alone, so the two simplicity notions capture complementary signal.
  • The metrics also rank self-supervised encoders on CIFAR and Caltech-101 on par with SFDA, and remain competitive in a source-selection setup where source models were fine-tuned on specialized domains, indicating the idea generalizes beyond ImageNet-pretrained backbones.
  • If these rankings hold, model selection on public hubs can be reduced to scoring frozen embeddings once per target task, avoiding thousands of fine-tuning runs and making transferability estimation practical at repository scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the authors do not make explicit: because INT measures linear separability, it may also forecast linear-probe accuracy, giving a cheap selection criterion for self-supervised checkpoints when full fine-tuning is too expensive.
  • A testable consequence of the simplicity hypothesis: an embedding with artificially inflated class separation (for instance, via spurious features) should still score high on INT yet fine-tune poorly, so constructing that pair would check whether real separation, not just any separation, drives transferability.
  • The paper names non-classification tasks as future work; a direct adaptation would replace the label-disagreement kernel with agreement on dense targets such as segmentation masks, testing whether the simplicity principle survives beyond discrete labels.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes two source-independent transferability metrics for ranking pre-trained models by how easily their frozen embeddings separate target classes: INT, the normalized pairwise interclass Euclidean distance, and Concept Variance, a weighted neighbor-label disagreement summarized by its standard deviation. The authors evaluate these against LogME, SFDA, LDA, N-LEEP, ETran, TransRate, and NCTI on seven image classification datasets with a small Vision Transformer zoo, plus limited-data, self-supervised, source-selection, larger-network, and timing experiments. They report large weighted Kendall's tau gains for INT (e.g., average 0.70 in the main experiment versus 0.38 for TransRate) and include code and fine-tuning details.

Significance. If the reported rankings are reliable, INT would be a practically attractive source-independent transferability metric: it is fast, has no fitted constants (alpha is fixed from the literature), and the paper ships code and a more realistic small-model zoo than earlier transferability benchmarks. The strength of the claim is conditional on two issues: ruling out a scale artifact in INT, and adding statistical support for the ranking comparisons. The theoretical link in Section 3.2 is heuristic rather than a derivation, but the empirical setup is broad and the proposed metrics are simple enough to be independently reimplemented.

major comments (5)
  1. [Section 3.2, Eqs. (1)-(2), Appendix D] INT is computed on raw, unnormalized embeddings: Appendix D shows pairwise Euclidean distances with no normalization before the interclass sums. Scaling every embedding by a positive constant c multiplies every pairwise distance and hence INT by c, while the ground-truth fine-tuning accuracy is invariant to such rescaling for a linear classifier head, since the head weights can absorb the scale. Because the model zoo mixes architectures with very different embedding magnitudes, the reported tau gains in Tables 1, 2, 4, and 5 could be driven by embedding norm rather than class separability. The paper should report a normalized INT (for example, dividing by the mean embedding norm or per-dimension standard deviation) and include a trivial baseline that ranks models by mean L2 norm of the embeddings.
  2. [Tables 1-5, Section 5] Each reported tau value is computed from a single ranking of only 7 to 24 models, and the paper provides no confidence intervals, bootstrap estimates, or significance tests. The word "significantly" in the conclusion (Section 5) is therefore not supported by the evidence as presented. A bootstrap over target samples or over model subsets would show whether the observed tau gaps of 0.1-0.3 are stable, and would allow a fair comparison of the two leading metrics in the SSL experiment.
  3. [Abstract and Section 5] The percentage claims are not traceable to the tables. The abstract says "increase Kendall's Tau by up to 32%" and Section 5 says "by 38% in Experiment 4.1 and 33% in Experiment 4.2," but from Table 1 the average INT is 0.70 versus 0.38 for TransRate, a relative increase of 84% (and an absolute difference of 0.32). The conclusion's 38% and 33% values do not match either Table 1 or Table 2. The baseline used for each percentage should be identified and the calculation defined explicitly.
  4. [Section 4.3, Table 3] In the self-supervised learning experiment, INT achieves an average tau of 0.752 versus 0.749 for SFDA, so INT only matches SFDA rather than outperforming it. This is at odds with the abstract's blanket statement that the metrics outperform state-of-the-art baselines. The claim in the conclusion is appropriately limited to Experiments 4.1 and 4.2, but the abstract and Section 5 should be made consistent with the SSL result.
  5. [Section 3.2 and Table 6] Theorem 3.1 and the cross-entropy argument justify squared Euclidean distance to class centers, but INT uses non-squared Euclidean distance in Eq. (2). The stated reason (less sensitivity to outliers) is not derived, and Table 6 shows that squared Euclidean performs nearly identically to Euclidean in the ablation. The theoretical discussion therefore does not explain the chosen geometry; either a bound or derivation for the non-squared case should be provided, or the ablation should be presented explicitly as empirical model selection.
minor comments (6)
  1. [Table 1 caption] The caption says "in limited data settings," but the limited-data experiment is reported in Table 2; Table 1 corresponds to the full-data Experiment 4.1, so the caption is likely wrong.
  2. [Eq. (6)] Equation (6) contains typos: "1≤i≤jM" should be "1 ≤ i < j ≤ M" and "sgn(x)is" should be "sgn(x) is".
  3. [Appendix B.1] The text refers to the "100-Sports dataset" but then says it covers 73 different sports; this inconsistency should be corrected.
  4. [Appendix B.1] The appendix cites Cimpoi et al. (2014) for the DTD dataset, but that reference is missing from the reference list.
  5. [Section 3.3, Eq. (5)] Using the standard deviation of per-example concept variation as the final metric is a departure from the original v_total in Eq. (4), and the paper does not justify this choice or ablate it against using the mean.
  6. [Table 3] The missing TransRate entries for Caltech101 and CIFAR100 should be marked as not computed in the table itself, not only mentioned in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed metrics are defined directly from embeddings and labels, contain no parameters fitted to the fine-tuning ground truth, and the cited softmax/nearest-center theorem is re-proved in the appendix.

full rationale

The paper's central claims are empirical rather than definitional. INT (Eq. 1-2) and Concept Variance (Alg. 1, Eq. 5) are computed directly from frozen embeddings and target labels, with no fitted constants: the alpha in Concept Variance is fixed at the literature value of 2, and the normalization in Concept Variance is a deterministic min-max scaling. The transferability ground truth is obtained separately by fine-tuning each model and measuring test accuracy, so the reported Kendall tau values are comparisons against an external benchmark, not quantities forced by the metric definitions. The theoretical foundation (Theorem 3.1) is attributed to Hess et al. (2020), which is a self-citation by co-author Sibylle Hess, but the paper includes a self-contained proof in Appendix A (Theorems A.1 and A.2), and the theorem is used only to motivate a geometric intuition about cross-entropy, not to define the score or to derive the ranking claim. The hypothesis in Section 3.1 that simpler embeddings fine-tune better is a modeling assumption and could be false, but that is a correctness/validity concern rather than circularity: the paper does not define 'simpler' in terms of the fine-tuning ground truth, nor does it fit INT or Concept Variance to the accuracies it later claims to predict. The choice of Euclidean distance in Experiment 4.7 is made after inspecting the same benchmark, and the unnormalized INT is sensitive to embedding scale; these are legitimate robustness critiques, but they do not make the reported result equivalent to its inputs by construction. Overall the derivation chain is self-contained and no circular step is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central empirical claim does not fit any parameters to target data beyond the fixed alpha and the min-max normalization. However, the theoretical motivation leans on an author's previous theorem, and the simplicity hypothesis itself is assumed. The benchmark is self-constructed, which is a design choice rather than a free parameter.

free parameters (1)
  • alpha (concept variation weight exponent) = 2 (default from Pérez & Rendell 1996; ablation shows 2-5 similar)
    Controls how quickly neighbor weights decay with distance. Chosen from prior literature, not tuned per dataset; however the metric's ranking can shift with alpha (Table 10), so it is a free knob of the method.
assumptions (4)
  • domain assumption Hypothesis: simpler embeddings fine-tune better
    Section 3.1 states this as a hypothesis; it is the core justification for both metrics but is not derived or independently tested.
  • standard math Theorem A.1 (softmax regression equals nearest-centroid classification)
    Reproduced from Hess et al. 2020 (an author's own prior work); requires W to have full column rank. Used in Section 3.2 to motivate distance-based analysis.
  • ad hoc to paper Euclidean distance is the right geometry for class-center separation
    Section 3.2 argues class centers differ from centroids, so Euclidean (not squared) is less outlier-sensitive. This is a heuristic, and Table 6 shows cosine distance fails, confirming the choice matters.
  • ad hoc to paper Continuous weighted adaptation of concept variation approximates the Boolean-space definition
    The original vtotal from Pérez & Rendell assumes Boolean features and full space enumeration; Algorithm 1 replaces this with MinMax-normalized real embeddings and a distance-based weight matrix, which is an unvalidated approximation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Occam's model: Selecting simpler representations for better transferability estimation." pith.science (2026). https://pith.science/paper/ZZILQQEX

@misc{pith2026250206925,
  author       = {Pith},
  title        = {Pith review of: Occam's model: Selecting simpler representations for better transferability estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZZILQQEX}},
  note         = {Machine review of arXiv:2502.06925}
}
read the original abstract

Fine-tuning models that have been pre-trained on large datasets has become a cornerstone of modern machine learning workflows. With the widespread availability of online model repositories, such as Hugging Face, it is now easier than ever to fine-tune pre-trained models for specific tasks. This raises a critical question: which pre-trained model is most suitable for a given task? This problem is called transferability estimation. In this work, we introduce two novel and effective metrics for estimating the transferability of pre-trained models. Our approach is grounded in viewing transferability as a measure of how easily a pre-trained model's representations can be trained to separate target classes, providing a unique perspective on transferability estimation. We rigorously evaluate the proposed metrics against state-of-the-art alternatives across diverse problem settings, demonstrating their robustness and practical utility. Additionally, we present theoretical insights that explain our metrics' efficacy and adaptability to various scenarios. We experimentally show that our metrics increase Kendall's Tau by up to 32% compared to the state-of-the-art baselines.

Figures

Figures reproduced from arXiv: 2502.06925 by the authors.

Figure 1
Figure 1. Given a set of pretrained models and a target dataset, extract the embeddings and estimate a complexity score DINT to rank pretrained models based on their suitability for the target task. computationally too expensive in this scenario. Fine-tuning each model on the target data for a search strategy like Bayesian optimization is not feasible since the fine-tuning step is too expensive. This raises the question: ”How… view at source ↗
Figure 2
Figure 2. We show how INT performs with a 2D toy problem. Overlapping classes receive lower INT scores than well-separated clusters. 3.3. Concept Variation Our second metric is based on concept variation (Perez & ´ Rendell, 1996), a measure that reflects the irregularity of class label distributions. Understanding this variation helps to assess the structural consistency of a concept C, which can be evaluated by analyzing how… view at source ↗
Figure 3
Figure 3. Alpha α studies larger version of models introduced in Section 4.1, details of selected models can be found in Appendix B.2. The embed￾ding size also quadruples in most of the models. We report our findings for this experiment in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 15 canonical work pages

  1. [1]

    The Digital Images of Bacteria Species dataset (DIBaS) (https://github.com/gallardorafael/DIBaS-Dataset) is a dataset of 33 bacterial species with around 20 images for each species

    DIBaS: Digital Image of Bacterial Species (DIBaS). The Digital Images of Bacteria Species dataset (DIBaS) (https://github.com/gallardorafael/DIBaS-Dataset) is a dataset of 33 bacterial species with around 20 images for each species

    work page

  2. [2]

    Flowers: Flowers dataset from Visual Geometry Group, University of Oxford. The Flowers dataset(https://www.robots.ox.ac.uk/ vgg/data/flowers/102/index.html) consists of a variety of flowers gath- ered from different websites and some are photographed by the original creators. These flowers are commonly found in the UK. The images generally have large scal...

    work page

  3. [3]

    Images are 224x224x3 in size and in .jpg format

    Sports: The 100-Sports dataset(https://www.kaggle.com/datasets/gpiosenka/sports-classification) is a collection of sports images covering 73 different sports. Images are 224x224x3 in size and in .jpg format. Images were gathered from internet searches. The images were scanned with a duplicate image detector program and all duplicate images were removed

    work page

  4. [4]

    and Zisserman, A

    2 Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In Indian Con- ference on Computer Vision, Graphics and Image Pro- cessing, Dec 2008. 5 Okuta, R., Unno, Y ., Nishino, D., Hido, S., and Loomis, C. Cupy: A numpy-compatible library for nvidia gpu calcula- tions. In Proceedings of Workshop on Machine Learn...

    work page 2008

  5. [5]

    Textures: The original Textures dataset is a combination of 4 texture datasets: KTH-TIPS and KTH-TIPS 2 (https://www.csc.kth.se/cvap/databases/kth-tips/index.html), Kylberg Textures Dataset (http://www.cb.uu.se/ gustaf/tex- ture/) and UIUC Textures Dataset. The data in all four datasets is collected in laboratory conditions, i.e., images were captured in ...

    work page 2004

  6. [6]

    Most images are on the road, but some have different backgrounds, and each image has only one car

    Cars: The original Cars dataset (https://ai.stanford.edu/ jkrause/cars/car dataset.html) was collected in 2013, and it contains more than 16 000 images from 196 classes of cars. Most images are on the road, but some have different backgrounds, and each image has only one car. Each class can have 48 to 136 images of variable resolutions. 13 Occam’s model: ...

    work page 2013

  7. [7]

    org/CorpusID:247939839

    URL https://api.semanticscholar. org/CorpusID:247939839. 5 Ullah, I., Carrion, D., Escalera, S., Guyon, I. M., Huis- man, M., Mohr, F., van Rijn, J. N., Sun, H., Vanschoren, J., and Vu, P. A. Meta-album: Multi-domain meta- dataset for few-shot image classification. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks...

    work page arXiv 2022

  8. [8]

    Each class has 5,000 training samples and 1,000 testing samples

    CIFAR10: The dataset contains 60,000 color images in 10 classes, with each image in the size of 32×32. Each class has 5,000 training samples and 1,000 testing samples

    work page

Show all 19 references
  1. [9]

    org/CorpusID:231985863

    URL https://api.semanticscholar. org/CorpusID:231985863. 2, 5 Zielinski, B., Plichta, A., Misztal, K., Spurek, P., Brzychczy-Wloch, M., and Ochonska, D. Deep learn- ing approach to bacterial colony classification. PLOS ONE, 12(9):1–14, 09 2017. doi: 10.1371/journal. pone.01845...

    2017 doi

  2. [10]

    The number of images in each category is between 40 and 800

    CALTECH 101: The dataset contains 9,146 images from 101 object categories. The number of images in each category is between 40 and 800. B.2. Models Pretrained timm models used in Experiment 4.1, Experiment 4.2, Experiment 4.6 and Experiment 4.7. Model Name Parameters (M) FLOPs...

  3. [13]

    The original image resolution is 256x256 px

    Plants: The Plant Village datasethttps://data.mendeley.com/datasets/tywbtsjrjv/1 contains cam- era photos of 17 crop leaves. The original image resolution is 256x256 px. This collection covers 26 plant diseases and 12 healthy plants

  4. [16]

    The data authors strive to provide a challenging dataset by increasing both within-class diversity and between-class similarity, as well as integrating many image variations

    RESISC: RESISC45 dataset(https://gcheng-nwpu.github.io/) gathers 700 RGB images of size 256x256 px for each of 45 scene categories. The data authors strive to provide a challenging dataset by increasing both within-class diversity and between-class similarity, as well as integ...

  5. [17]

    nl/10.1007/978-3-031-19830-4_17

    URL https://doi-org.dianus.libr.tue. nl/10.1007/978-3-031-19830-4_17 . 2, 5, 6 Tan, Y ., Li, Y ., and Huang, S.-L. Otce: A transferability metric for cross-domain cross-task representations. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2 Ti...

    2021 doi

  6. [18]

    DTD (Cimpoi et al., 2014) The dataset consists of

    CIFAR 100: The dataset is the same as CIFAR-10 except that it has 100 classes each of which contains 500 training images and 100 testing images. DTD (Cimpoi et al., 2014) The dataset consists of

    2014

  7. [2017]

    7 Papyan, V ., Han, X., and Donoho, D

    URL http://learningsys.org/nips17/ assets/papers/paper_16.pdf. 7 Papyan, V ., Han, X., and Donoho, D. L. Prevalence of neural collapse during the terminal phase of deep learn- ing training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020. 3 P´erez, E...

    2020 arXiv

  8. [2019]

    X., and Lin, D

    5 Wu, Z., Xiong, Y ., Yu, S. X., and Lin, D. Unsupervised fea- ture learning via non-parametric instance discrimination. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. doi: 10.1109/CVPR.2018. 00393. 6 Xu, W., Xu, Y ., Chang, T. A., and Tu, Z. Co-...

    2018 doi

  9. [2020]

    Remote sensing image scene classification: Benchmark and state of the art

    6 Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. CoRR, abs/1703.00121, 2017. URL http://arxiv.org/ abs/1703.00121. 5 Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchic...

    2017 arXiv

  10. [2021]

    org/CorpusID:233346705

    URL https://api.semanticscholar. org/CorpusID:233346705. 5 Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. Computer Vision and Image Understanding, 2007. d...

    2007

  11. [2022]

    5 Nguyen, C

    URL https://openreview.net/forum? id=vh-0sUt8HlG. 5 Nguyen, C. V ., Hassner, T., Seeger, M., and Archambeau, C. Leep: a new measure to evaluate transferability of learned representations. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org,

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.