Pith. sign in

REVIEW 3 major objections 5 minor 25 references

Analysis of Transferability Estimation Metrics for Surgical Phase Recognition

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Source-independent transferability estimation can pick the best pre-trained model for surgical phase recognition without retraining, with LogME most aligned and TransRate inverted.

desk verdict Useful first benchmark of SITE metrics for surgical phase recognition, but the headline robustness claim fades under the paper's own pruning ablation. read the letter →

arxiv 2508.16730 v1 pith:BKXAMKVL submitted 2025-08-22 eess.IV cs.CVcs.LG

Prabhant Singh , Yiping Li , Yasmina Al Khalil This is my paper
classification eess.IVcs.CVcs.LG
keywords SurgicalphaserecognitionTransferabilityestimationLogMEH-ScoreTransRatePre-trainedmodelsModelselectionFine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a cheap, retraining-free score can tell which pre-trained model will fine-tune best for surgical phase recognition, a task where labeled video is expensive. It formalizes source-independent transferability estimation (SITE) for this setting, then benchmarks three metrics—LogME, H-Score, and TransRate—across two datasets (RAMIE and AutoLaparo) using 11–14 candidate models. The central finding is that LogME, especially when aggregated by the minimum per-surgery score, closely tracks actual fine-tuning accuracy; H-Score gives only weak signal, and TransRate systematically inverts the true model ranking. An ablation shows the rankings depend on including both strong and weak models; when candidates cluster in a narrow accuracy band, the metrics lose discriminative power. If the claim holds, model selection in surgical AI can skip most fine-tuning.

What carries the argument

The central machinery is LogME (maximum label marginalized likelihood), which turns a pre-trained model's features on target frames into a transferability score without retraining. The paper's contribution is to compute LogME separately for each surgery subset of the target dataset and then summarize the per-subset scores with a statistical function (mean, min, or max); the minimum aggregation is what best mirrors fine-tuning accuracy. The evaluation harness pairs this with weighted Kendall's tau against a two-stage fine-tuned baseline (frame-wise backbone plus MS-TCN temporal refinement).

What would settle it

Fine-tune each candidate model at least three times with different random seeds, compute per-seed Kendall tau between each metric and mean fine-tuned accuracy, and check whether LogME's tau remains above H-Score's and TransRate's with non-overlapping confidence intervals; if not, the claimed ranking advantage fails.

Watch

Extended reading notes

Core claim

Formally, the paper claims that for surgical phase recognition, the LogME transferability score computed from frame-wise embeddings of a pre-trained model is a reliable proxy for that model's fine-tuned accuracy, with a per-subset (per-surgery) aggregation by the minimum score giving the closest alignment (Pearson r ≈ 0.66–0.67, Kendall τ ≈ 0.83 on both datasets). H-Score, by contrast, provides only weak predictive power, and TransRate produces negative Kendall τ values, meaning it ranks better models as worse. The paper also claims that the usefulness of these estimates hinges on the diversity of the candidate pool: removing the single strongest model on AutoLaparo drops LogME's weighted Ke

Load-bearing premise

The claimed rankings assume that the single fine-tuned accuracy per model is a stable, low-noise ground truth, yet with only 11–14 models, one seed, and no confidence intervals, removing one top model cuts LogME's Kendall tau from 0.83 to 0.36.

Editorial extensions

If this is right

  • Practitioners can pre-rank surgical phase recognition models by LogME on a small labeled frame set, avoiding full fine-tuning of each candidate.
  • Aggregate per-surgery LogME scores by the minimum to get the most reliable overall model ranking.
  • H-Score should not be relied on alone; its mean and max variants are near zero or negative.
  • TransRate should be avoided in this setting because its ranking is inverted.
  • When candidate models have similar fine-tuning accuracy (within roughly 5–7%), transferability scores lose discriminative power; maintain pool diversity or add validation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because LogME ranks via embedding-label compatibility, it may generalize to other frame-wise surgical tasks such as tool presence or action recognition, but this is untested.
  • Beyond the paper: the ablation's sensitivity to removing one top model implies the reported tau values should not be read as stable; a practical tool would need confidence intervals, which the paper itself flags for future work.
  • Beyond the paper: TransRate's consistent inversion suggests the mutual-information estimate has a sign or normalization issue for temporally correlated surgical frames, a diagnostic direction the paper does not pursue.
  • Beyond the paper: minimum-per-subset aggregation working best hints that worst-case surgery performance, not average, drives fine-tuning success; a testable extension is to check whether LogME(min) predicts per-surgery generalization gaps.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper formalizes source-independent transferability estimation (SITE) for surgical phase recognition and benchmarks three metrics—LogME, H-Score, and TransRate—on two datasets (RAMIE and AutoLaparo). For each of 11–14 pre-trained models, the authors compute transferability scores from frozen embeddings, aggregate per-surgery scores via mean/min/max, and compare the resulting rankings with fine-tuned test accuracies using Pearson r and Kendall's tau. The central claim is that LogME, especially with minimum aggregation, correlates best with downstream accuracy, H-Score is weak, and TransRate inverts true rankings. Ablation studies in §3.2 show that these correlations are highly sensitive to the inclusion of extreme-performance models.

Significance. If the result were robust, it would offer a practical shortcut for selecting pre-trained models for surgical phase recognition without fine-tuning each candidate, which is attractive given annotation and compute costs. The paper is among the first to evaluate SITE in surgical video analysis, and the code release is a useful contribution. However, the current evidence is not yet strong enough to support the advertised practical guidance: the model pool is small (11–14), fine-tuning accuracies come from a single seed, no confidence intervals are reported, and the paper's own ablations show that the headline Kendall tau values collapse when extreme models are removed. The significance is therefore conditional on a more careful, uncertainty-aware evaluation.

major comments (3)
  1. [§3.2, Table 2] The headline claim that LogME 'aligns most closely with fine-tuning accuracy' is not robust to pool composition. On AutoLaparo, removing the single strongest model (SurgNet_XL) reduces tau from 0.825 to 0.36; on RAMIE, removing the top three models drops tau from 0.835 to 0.24, and further pruning inverts it to -0.07. These results, acknowledged in the text, demonstrate that the high tau values are driven by a few extreme models. Since the stated practical use case is choosing among candidate models with comparable performance (§3.2, Conclusion), the abstract's unconditional phrasing is not supported. Please report bootstrap confidence intervals or leave-one-out tau values, and reframe the claim as valid only when the candidate pool spans a wide performance range.
  2. [§2.3, Fine-Tuning details] The ground truth accuracy G_m is obtained from a single fine-tuning run per model; no seed variation is reported and no variance measure is given. With M=11 on AutoLaparo and M=14 on RAMIE, Kendall's tau is a coarse statistic, and small changes in G_m could reorder models. The small numerical differences between LogME variants (e.g., tau=0.833 vs 0.825) are not interpretable without repeated seeds or at least bootstrap confidence intervals. The paper's Future Work promises statistical significance testing, but the current conclusions and practical guidelines rest on these unsupported point estimates.
  3. [§3.1, Table 2 and Abstract] The abstract's emphasis on 'minimum per-subset score' is not supported by the reported Kendall tau values. LogME(mean) has tau=0.833/0.835 on AutoLaparo/RAMIE; LogME(min) has tau=0.825/0.835. The min variant is only better in Pearson r, and the difference is within sampling variability given the small M. Since the min aggregation appears to have been selected after inspecting the results, the claim of superiority is post hoc and should be removed or supported by a significance test. Additionally, LogME(max) still attains tau≈0.70–0.75, so the conclusion should focus on LogME as a family rather than on a particular aggregation.
minor comments (5)
  1. [§2.3, Eq. (2)] Equation (2) is the standard unweighted Kendall tau formula, but the text says the weighted version is used. Please provide the explicit weighted definition or clarify that the simple formula is used in practice.
  2. [Table 1/Table 2] The number of evaluated models differs between datasets (the footnote lists three models not evaluated on AutoLaparo). State the per-dataset M explicitly in the Table 2 caption or in §2.3, as M directly affects the granularity and stability of tau.
  3. [§2.2] The notation reuses T_m for the per-subset set and for the scalar global score. Use e.g. T_m^{(a)} for per-subset scores and T_m for the summarized value to avoid confusion.
  4. [Figure 2] The point labels in Figure 2 overlap heavily and use inconsistent capitalization ('surgenet' vs 'SurgeNet'). This makes the figure hard to read; consider using a legend or callouts.
  5. [Abstract] The phrase 'first comprehensive benchmark' is strong given the evaluation covers two datasets and three metrics; 'first benchmark' would be more accurate and less open to criticism.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: pre-existing transferability metrics are compared against independently measured fine-tuning accuracy.

full rationale

The paper is a benchmark rather than a derivation. LogME, H-Score, and TransRate are pre-existing metrics with fixed definitions; the ground-truth G_m is obtained by independently fine-tuning each model with a fixed TeCNO two-stage protocol and measuring test accuracy. Transferability scores T_m are computed from pretrained embeddings and target labels, and Eq. (1) only states the desired ranking condition, not a definitional equivalence. The per-subset mean/min/max summaries are fixed statistics and all variants are reported, so highlighting the minimum aggregation is a selection among reported results, not a fitted parameter renamed as a prediction. Self-citations ([10], [11], [13]) supply pretrained models and prior SITE context, but the benchmark outcome is not deduced from them; the evaluation protocol is independent of those citations. The paper's own ablation showing tau dropping from 0.825 to 0.36 when one model is removed is a robustness/statistical-power limitation, not a circular step. No equation or self-citation chain forces the reported rankings.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new axioms beyond standard supervised learning assumptions. The key domain assumptions are that the TeCNO fine-tuning protocol produces a valid ground truth, that static frame features are sufficient for transferability assessment in a temporal task, and that the small model pools represent realistic candidate sets. These are stated implicitly, not tested.

assumptions (3)
  • domain assumption The TeCNO two-stage fine-tuning protocol yields a valid ground truth for transferability.
    G_m is defined as test accuracy after this protocol; any noise or bias in this protocol propagates to all correlations.
  • domain assumption Static frame embeddings at 1 fps capture the transferability relevant for a temporal classification task.
    SITE metrics are computed on per-frame features, while final accuracy includes temporal refinement by MS-TCN; the paper does not test whether this step changes model rankings.
  • domain assumption The model pools (14 models on RAMIE, 11 on AutoLaparo) are representative of pre-trained models used in practice.
    The performance span is wide for LogME to show strong signal; the ablation shows fragility when the span is narrowed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analysis of Transferability Estimation Metrics for Surgical Phase Recognition." pith.science (2026). https://pith.science/paper/BKXAMKVL

@misc{pith2026250816730,
  author       = {Pith},
  title        = {Pith review of: Analysis of Transferability Estimation Metrics for Surgical Phase Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BKXAMKVL}},
  note         = {Machine review of arXiv:2508.16730}
}
read the original abstract

Fine-tuning pre-trained models has become a cornerstone of modern machine learning, allowing practitioners to achieve high performance with limited labeled data. In surgical video analysis, where expert annotations are especially time-consuming and costly, identifying the most suitable pre-trained model for a downstream task is both critical and challenging. Source-independent transferability estimation (SITE) offers a solution by predicting how well a model will fine-tune on target data using only its embeddings or outputs, without requiring full retraining. In this work, we formalize SITE for surgical phase recognition and provide the first comprehensive benchmark of three representative metrics, LogME, H-Score, and TransRate, on two diverse datasets (RAMIE and AutoLaparo). Our results show that LogME, particularly when aggregated by the minimum per-subset score, aligns most closely with fine-tuning accuracy; H-Score yields only weak predictive power; and TransRate often inverses true model rankings. Ablation studies show that when candidate models have similar performances, transferability estimates lose discriminative power, emphasizing the importance of maintaining model diversity or using additional validation. We conclude with practical guidelines for model selection and outline future directions toward domain-specific metrics, theoretical foundations, and interactive benchmarking tools.

Figures

Figures reproduced from arXiv: 2508.16730 by the authors.

Figure 1
Figure 1. Estimating transferability via frame-wise estimation. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Relationship between mean-aggregated LogME transferability score and [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 20 canonical work pages

  1. [1]

    In: 2019 IEEE International Conference on Image Processing (ICIP)

    Bao, Y., Li, Y., Huang, S.L., Zhang, L., Zheng, L., Zamir, A., Guibas, L.: An information-theoretic approach to transferability in task transfer learning. In: 2019 IEEE International Conference on Image Processing (ICIP). pp. 2309–2313 (2019)

    work page 2019

  2. [2]

    Interna- tional Journal of Computer Assisted Radiology and Surgery19(6), 1085– 1091 (2024)

    Batić, D., Holm, F., Özsoy, E., Czempiel, T., Navab, N.: Endovit: pretrain- ing vision transformers on a large collection of endoscopic images. Interna- tional Journal of Computer Assisted Radiology and Surgery19(6), 1085– 1091 (2024)

    work page 2024

  3. [3]

    Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? In: ICML. vol. 2, p. 4 (2021)

    2021

  4. [4]

    Medical Image Analysis98, 103298 (2024)

    Boers, T.G., Fockens, K.N., van der Putten, J.A., Jaspers, T.J., Kusters, C.H., Jukema, J.B., Jong, M.R., Struyvenberg, M.R., de Groof, J., Bergman, J.J., et al.: Foundation models in gastrointestinal endoscopic ai: Impact of architecture, pre-training approach and data efficiency. Medical Image Analysis98, 103298 (2024)

    work page 2024

  5. [5]

    In: MICCAI 2020

    Czempiel, T., Paschali, M., Keicher, M., Simson, W., Feussner, H., Kim, S.T., Navab, N.: Tecno: Surgical phase recognition with multi-stage tempo- ral convolutional networks. In: MICCAI 2020. pp. 343–352. Springer (2020)

    work page 2020

  6. [6]

    In: 2009 IEEE Conference on Com- puter Vision and Pattern Recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Com- puter Vision and Pattern Recognition. pp. 248–255 (2009)

    work page 2009

  7. [7]

    arXiv preprint arXiv:2010.11929 (2020)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Un- terthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    arXiv 2010

  8. [8]

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recogni- tion.In:ProceedingsoftheIEEEconferenceoncomputervisionandpattern recognition. pp. 770–778 (2016)

    work page 2016

Show all 25 references
  1. [9]

    In: International Conference on Machine Learn- ing

    Huang, L.K., Huang, J., Rong, Y., Yang, Q., Wei, Y.: Frustratingly easy transferability estimation. In: International Conference on Machine Learn- ing. pp. 9201–9225. PMLR (2022)

    2022

  2. [10]

    arXiv preprint arXiv:2501.09436 (2025)

    Jaspers, T.J., de Jong, R.L., Li, Y., Kusters, C.H., Bakker, F.H., van Jaarsveld, R.C., Kuiper, G.M., van Hillegersberg, R., Ruurda, J.P., Brinkman, W.M., et al.: Scaling up self-supervised learning for improved surgical foundation models. arXiv preprint arXiv:2501.09436 (2025)

    2025

  3. [11]

    Juodelyte, D., Ferrante, E., Lu, Y., Singh, P., Vanschoren, J., Cheplygina, V.: On dataset transferability in medical image classification (2024),https: //arxiv.org/abs/2412.20172

    2024 arXiv

  4. [12]

    In: Proceedings of the IEEE/CVF Conference CVPR

    Li, Y., Jia, X., Sang, R., Zhu, Y., Green, B., Wang, L., Gong, B.: Ranking neural checkpoints. In: Proceedings of the IEEE/CVF Conference CVPR. pp. 2663–2673 (2021) Title Suppressed Due to Excessive Length 11

    2021

  5. [13]

    In: Medical Imaging 2025: Image-Guided Procedures, Robotic Inter- ventions, and Modeling

    Li, Y., van Jaarsveld, R., de Jong, R., Bongers, J., Kuiper, G., van Hil- legersberg, R., Ruurda, J., Breeuwer, M., Khalil, Y.A.: Benchmarking and enhancing surgical phase recognition models for robot-assisted esophagec- tomy. In: Medical Imaging 2025: Image-Guided Procedures,...

    2025

  6. [14]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11976–11986 (2022)

    2022

  7. [15]

    In: International Conference on Machine Learning

    Nguyen, C., Hassner, T., Seeger, M., Archambeau, C.: Leep: A new mea- sure to evaluate transferability of learned representations. In: International Conference on Machine Learning. pp. 7294–7305. PMLR (2020)

    2020

  8. [16]

    arXiv preprint arXiv:2403.05949 (2024)

    Schmidgall, S., Kim, J.W., Jopling, J., Krieger, A.: General surgery vi- sion transformer: A video pre-trained foundation model for general surgery. arXiv preprint arXiv:2403.05949 (2024)

    2024 arXiv

  9. [17]

    In: European Conference on Computer Vision

    Shao, W., Zhao, X., Ge, Y., Zhang, Z., Yang, L., Wang, X., Shan, Y., Luo, P.: Not all models are equal: Predicting model transferability in a self- challenging fisher space. In: European Conference on Computer Vision. pp. 286–302. Springer (2022)

    2022

  10. [18]

    In: Proceed- ings of the 24th International Conference on World Wide Web

    Vigna, S.: A weighted correlation index for rankings with ties. In: Proceed- ings of the 24th International Conference on World Wide Web. WWW ’15, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE (2015)

    2015

  11. [19]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao,L.: Pyramidvisiontransformer: Aversatilebackbone fordensepredic- tion without convolutions. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 568–578 (2021)

    2021

  12. [20]

    In: International Conference on Medical Image Computing and Computer-Assisted Interven- tion

    Wang, Z., Liu, C., Zhang, S., Dou, Q.: Foundation model for endoscopy video analysis via large-scale self-supervised pre-train. In: International Conference on Medical Image Computing and Computer-Assisted Interven- tion. pp. 101–111. Springer (2023)

    2023

  13. [21]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang, Z., Luo, Y., Zheng, L., Huang, Z., Baktashmotlagh, M.: How far pre- trained models are from neural collapse on the target dataset informs their transferability. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5549–5558 (2023)

    2023

  14. [22]

    In: MICCAI 2022

    Wang, Z., Lu, B., Long, Y., Zhong, F., Cheung, T.H., Dou, Q., Liu, Y.: Au- tolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy. In: MICCAI 2022. pp. 486–496

    2022

  15. [23]

    In: Greenspan, H., Madabhushi, A., Mousavi, P., Salcudean, S., Duncan, J., Syeda-Mahmood, T., Taylor, R

    Yang, Y., Wei, M., He, J., Yang, J., Ye, J., Gu, Y.: Pick the best pre-trained model: Towards transferability estimation for medical image segmentation. In: Greenspan, H., Madabhushi, A., Mousavi, P., Salcudean, S., Duncan, J., Syeda-Mahmood, T., Taylor, R. (eds.) MICCAI 2023....

    2023

  16. [24]

    In: ICML

    You, K., Liu, Y., Wang, J., Long, M.: Logme: Practical assessment of pre- trained models for transfer learning. In: ICML. pp. 12133–12143 (2021)

    2021

  17. [25]

    IEEE Transactions on Pattern Analysis and Machine Intelligence46(2), 896–912 (2023)

    Yu, W., Si, C., Zhou, P., Luo, M., Zhou, Y., Feng, J., Yan, S., Wang, X.: Metaformer baselines for vision. IEEE Transactions on Pattern Analysis and Machine Intelligence46(2), 896–912 (2023)

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.