REVIEW 4 major objections 41 references
SciLT: Long-tailed Image Classification under Scientific Image Domains
T0 review · 4 major / 0 minor · reviewed 2026-07-13 · grok-4.5
Pith's one-line read Fine-tuning foundation models on long-tailed scientific images yields only limited gains, but fusing penultimate and final layer features with dual supervision balances head and tail accuracy.
desk verdict Useful domain-shift diagnosis for scientific long-tail PEFT, but the headline “foundation models barely help” rests on a confounded ResNet vs CLIP-ViT comparison; SciLT itself is a modest, clean dual-head recipe. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
SciLT: adaptive routing-calibrated fusion of penultimate- and final-layer features, dual heads supervised by logit adjustment (fused features) and cross-entropy (final features), with logit ensembling at inference.
What would settle it
Re-run the Table 2 / Finding 1 protocol with a convolutional backbone of similar capacity that is first pre-trained on natural images then fine-tuned the same way; if that setup closes the gap to training from scratch on NIH-Chest and erases SciLT’s BScore advantage, the claim that domain shift makes foundation PEFT ineffective would be undermined.
Extended reading notes
Core claim
Under a purely visual fine-tuning setup, foundation models give only limited—and sometimes negative—gains on scientific long-tailed data relative to training from scratch, while penultimate-layer features supply complementary signal that is particularly helpful for tail classes; jointly using penultimate and final features via adaptive fusion and dual LA/CE supervision (SciLT) improves balanced accuracy across Blood, ISIC, and NIH-Chest.
Load-bearing premise
That comparing an AdaptFormer-tuned CLIP ViT-B/16 to a ResNet-18 trained from scratch is a fair measure of whether foundation-model fine-tuning helps on scientific long-tailed data.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies long-tailed image classification on scientific domains (Blood, ISIC, NIH-Chest) under a purely visual PEFT setup with CLIP ViT-B/16 and AdaptFormer. Empirically, it reports that fine-tuning foundation models yields only limited or even negative gains relative to training from scratch, and that penultimate-layer features can match or exceed final-layer features, especially for tail classes. Motivated by a measured Wasserstein gap between penultimate and final features, the authors propose SciLT: adaptive fusion of the two layers, dual supervision (LA on the fused head, CE on the final-layer head), and logit ensembling at inference. On the three benchmarks SciLT improves a proposed BScore (harmonic mean of overall and macro accuracy) over CE and LA, with the largest gains on NIH-Chest medium/few classes.
Significance. Scientific long-tailed recognition under large domain shift is a genuine and underexplored setting; most foundation-model long-tail work is confined to natural-image benchmarks. The penultimate-vs-final analysis (Tables 3–5) and the purely visual design are useful contributions if the limited-gains narrative is cleaned up. SciLT itself is a simple, practical PEFT recipe with clear BScore gains on ISIC and NIH-Chest. The work would be a solid empirical baseline paper for adapting foundation models to scientific long-tail data, provided the architecture confound in Finding 1 is fixed and SciLT is compared against stronger same-backbone long-tail methods.
major comments (4)
- §4.2, Table 2 and Finding 1: the claim that “foundation models merely benefit” scientific long-tailed data rests on AdaptFormer-tuned CLIP ViT-B/16 versus ResNet-18 trained from scratch. Capacity, inductive bias, optimizer dynamics, and pre-training status all differ, so the gap cannot be attributed cleanly to domain shift. This comparison is load-bearing for the paper’s motivation. Please add architecture-matched controls (e.g., ViT-B/16 from scratch or ImageNet-pretrained ResNet/ViT under the same PEFT protocol) before stating Finding 1 in its current form.
- §6.1, Tables 6–8: SciLT is evaluated almost exclusively against CE and LA on the same backbone. Table 2 already lists CB, LDAM, Focal, and LADE for ResNet, but those methods are not re-run under AdaptFormer-CLIP. Without same-backbone long-tail baselines (e.g., LA, LDAM, PaCo-style, or classifier re-training on the fused features), the claim that SciLT “consistently outperforms existing methods” is overstated. At minimum, report LA/CE/LDAM/Focal with the identical AdaptFormer setup and the same dual-head budget.
- Appendix A / NIH-Chest protocol: multi-label ChestX-ray labels are collapsed to single-label by “selecting the first non-No Finding label.” This is an ad-hoc reduction that can systematically bias class frequencies and co-occurrence structure. Please justify or ablate this choice (e.g., random disease among positives, multi-label metrics, or official multi-label evaluation) and show that SciLT’s BScore gains are not an artifact of the conversion rule.
- §5.3, Theorem 5.3 and Remark: the Rademacher bound is a standard sum-space argument for ensembles; the claimed link from Wasserstein distance (Table 5) to a positive ensemble gain Δ_ens is informal and not used in the proof. Either tighten the theory to a concrete condition under which W-distance implies Δ_ens > complexity penalty, or reframe §5.3 as a brief complexity discussion rather than a formal justification of SciLT’s advantage.
Circularity Check
No significant circularity: empirical PEFT study with standard training/evaluation and non-load-bearing theory.
full rationale
This is a purely empirical methods paper. Observations (limited PEFT gains on scientific LT data vs. ResNet-from-scratch, penultimate-layer utility for tails, Wasserstein gap) are measured on held-out splits and motivate SciLT; they do not define or force the method's success. SciLT trains adaptive fusion gates and dual heads (LA on fused features, CE on final-layer) on labeled data in the ordinary supervised way, then ensembles logits at inference. BScore is an evaluation harmonic mean of OvAcc and Macro, not a fitted target or training objective. The Rademacher/generalization bounds (Lemmas 5.1–5.2, Thm. 5.3) are standard textbook applications that upper-bound complexity of the sum hypothesis class and do not reduce the empirical claims to their inputs by construction. No self-definitional loops, no fitted-parameter-as-prediction, no load-bearing self-citation uniqueness theorems, and no ansatz smuggled via overlapping-author citations. The confounded ResNet-vs-CLIP comparison is a validity concern, not circularity. The derivation chain is self-contained against external benchmarks.
Assumptions & free parameters
free parameters (5)
- Per-dataset learning rates (Blood 0.012, ISIC 0.008, NIH-Chest 0.01)
- Fusion interaction coefficient γ and gating parameters W1, W2
- Logit-adjustment temperature / class-prior strength in LA
- AdaptFormer bottleneck rank and which blocks receive adapters
- Training epochs (10/10/5) and batch size 128
assumptions (6)
- domain assumption CLIP ViT-B/16 pretraining plus AdaptFormer is a representative foundation-model fine-tuning paradigm for scientific long-tailed recognition.
- ad hoc to paper ResNet-18 trained from scratch is a fair reference for whether foundation models 'merely benefit' scientific datasets.
- domain assumption Wasserstein distance between penultimate and final features indicates complementary, non-redundant information useful for long-tail learning.
- ad hoc to paper NIH-Chest multi-label labels can be reduced to single-label by taking the first non-'No Finding' disease label.
- ad hoc to paper BScore = 2·OvAcc·Macro/(OvAcc+Macro) is an appropriate primary summary of balanced scientific long-tail performance on imbalanced test sets.
- standard math Standard Rademacher complexity bounds for sum hypothesis classes apply to SciLT’s ensemble of two heads.
invented entities (3)
-
SciLT adaptive multi-level fusion + dual-supervision framework
-
BScore (harmonic mean of overall and macro accuracy)
-
Routing-calibrated aggregation module (gates r and interaction Ψ)
Cite this review
Pith. "Pith review of SciLT: Long-tailed Image Classification under Scientific Image Domains." pith.science (2026). https://pith.science/paper/2604.03687
@misc{pith2026260403687,
author = {Pith},
title = {Pith review of: SciLT: Long-tailed Image Classification under Scientific Image Domains},
year = {2026},
howpublished = {\url{https://pith.science/paper/2604.03687}},
note = {Machine review of arXiv:2604.03687}
}
read the original abstract
Long-tailed recognition has benefited from foundation models and fine-tuning paradigms, yet existing studies and benchmarks are mainly confined to natural image domains, where pre-training and fine-tuning data share similar distributions. In contrast, scientific images exhibit distinct visual characteristics and supervision signals, raising questions about the effectiveness of fine-tuning foundation models in such settings. In this work, we investigate scientific long-tailed recognition under a purely visual and fine-tuning paradigm. Experiments on three scientific benchmarks show that fine-tuning foundation models yields limited gains, and reveal that penultimate-layer features play an important role, particularly for tail classes. Motivated by these findings, we propose SciLT, a framework that exploits multi-level representations through adaptive feature fusion and dual-supervision learning. By jointly leveraging penultimate- and final-layer features, SciLT achieves balanced performance across head and tail classes. Extensive experiments demonstrate that SciLT consistently outperforms existing methods, establishing a strong and practical baseline for scientific long-tailed recognition and providing valuable guidance for adapting foundation models to scientific data with substantial domain shifts.
Reference graph
Works this paper leans on
-
[1]
Large-scale long-tailed recognition in an open world
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2537–2546, 2019
2019
-
[2]
Class-balanced loss based on effective number of samples
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9268–9277, 2019
2019
-
[3]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020
arXiv 2010
-
[4]
Long-tail learning with foundation model: Heavy fine-tuning hurts
Jiang-Xin Shi, Tong Wei, Zhi Zhou, Jie-Jing Shao, Xin-Yan Han, and Yu-Feng Li. Long-tail learning with foundation model: Heavy fine-tuning hurts. InForty-first International Conference on Machine Learning, 2024
2024
-
[5]
Lpt: Long-tailed prompt tuning for image classification.arXiv preprint arXiv:2210.01033, 2022
Bowen Dong, Pan Zhou, Shuicheng Yan, and Wangmeng Zuo. Lpt: Long-tailed prompt tuning for image classification.arXiv preprint arXiv:2210.01033, 2022
arXiv 2022
-
[6]
Vl-ltr: Learning class- wise visual-linguistic representation for long-tailed visual recognition
Changyao Tian, Wenhai Wang, Xizhou Zhu, Jifeng Dai, and Yu Qiao. Vl-ltr: Learning class- wise visual-linguistic representation for long-tailed visual recognition. InEuropean Conference on Computer Vision, pages 73–91. Springer, 2022
2022
-
[7]
Ltgc: Long-tail recognition via leveraging llms-driven generated content
Qihao Zhao, Yalun Dai, Hao Li, Wei Hu, Fan Zhang, and Jun Liu. Ltgc: Long-tail recognition via leveraging llms-driven generated content. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19510–19520, 2024
2024
-
[8]
Zero-shot text-to-image generation
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InInternational conference on machine learning, pages 8821–8831. Pmlr, 2021
2021
Show all 41 references
-
[9]
Long-tail learning via logit adjustment.arXiv preprint arXiv:2007.07314, 2020
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment.arXiv preprint arXiv:2007.07314, 2020
2007 arXiv
-
[10]
Balanced meta-softmax for long- tailed visual recognition.Advances in Neural Information Processing Systems, 33:4175–4186, 2020
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition.Advances in Neural Information Processing Systems, 33:4175–4186, 2020
2020
-
[11]
Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings
Hao Guo and Song Wang. Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15089–15098, 2021
2021
-
[12]
Imbalanced continual learning with partitioning reservoir sampling
Chris Dongjoo Kim, Jinseo Jeong, and Gunhee Kim. Imbalanced continual learning with partitioning reservoir sampling. InEuropean Conference on Computer Vision, pages 411–428. Springer, 2020
2020
-
[13]
A simple long-tailed recognition baseline via vision-language model.arXiv preprint arXiv:2111.14745, 2021
Teli Ma, Shijie Geng, Mengmeng Wang, Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao. A simple long-tailed recognition baseline via vision-language model.arXiv preprint arXiv:2111.14745, 2021
2021 arXiv
-
[14]
Learning from neighbors: Category extrapolation for long-tail learning
Shizhen Zhao, Xin Wen, Jiahui Liu, Chuofan Ma, Chunfeng Yuan, and Xiaojuan Qi. Learning from neighbors: Category extrapolation for long-tail learning. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 30483–30492, 2025
2025
-
[15]
Matthew Ragoza, Julien Hochuli, Ernesto Idrobo, Jocelyn Sunseri, and David R. Koes. Protein– ligand scoring with convolutional neural networks.Journal of Chemical Information and Modeling, 57(4):942–957, 2017
2017
-
[16]
Kdeep: Protein– ligand absolute binding affinity prediction via 3d convolutional neural networks.Journal of Chemical Information and Modeling, 58(2):287–296, 2018
Javier Jiménez, Miha Skalic, Gerard Martinez-Rosell, and Gianni De Fabritiis. Kdeep: Protein– ligand absolute binding affinity prediction via 3d convolutional neural networks.Journal of Chemical Information and Modeling, 58(2):287–296, 2018. 10
2018
-
[17]
Transfusion: Understanding transfer learning for medical imaging.Advances in neural information processing systems, 32, 2019
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging.Advances in neural information processing systems, 32, 2019
2019
-
[18]
Big self-supervised models advance medical image classification
Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, et al. Big self-supervised models advance medical image classification. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021
2021
-
[19]
Climd: A curriculum learning framework for imbalanced multimodal diagnosis
Kai Han, Chongwen Lyu, Lele Ma, Chengxuan Qian, Siqi Ma, Zheng Pang, Jun Chen, and Zhe Liu. Climd: A curriculum learning framework for imbalanced multimodal diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 65–74. Sprin...
2025
-
[20]
Deciphering the extremes: A novel approach for pathological long-tailed recognition in scientific discovery
Zhe Zhao, HaiBin Wen, Xianfu Liu, Rui Mao, Pengkun Wang, Liheng Yu, Linjiang Chen, Bo An, Qingfu Zhang, and Yang Wang. Deciphering the extremes: A novel approach for pathological long-tailed recognition in scientific discovery. InThe Thirty-ninth Annual Conference on Neural In...
2025
-
[21]
Decoupling representation and classifier for long-tailed recognition.arXiv preprint arXiv:1910.09217, 2019
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition.arXiv preprint arXiv:1910.09217, 2019
1910 arXiv
-
[22]
Improving calibration for long-tailed recognition
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16489–16498, 2021
2021
-
[23]
Parametric contrastive learning
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 715–724, 2021
2021
-
[24]
Learning imbalanced data with vision transformers
Zhengzhuo Xu, Ruikang Liu, Shuo Yang, Zenghao Chai, and Chun Yuan. Learning imbalanced data with vision transformers. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15793–15803, 2023
2023
-
[25]
The inaturalist species classification and detection dataset
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 87...
2018
-
[26]
Wbcatt: a white blood cell dataset annotated with detailed morphological attributes.Advances in Neural Information Processing Systems, 36:50796–50824, 2023
Satoshi Tsutsui, Winnie Pang, and Bihan Wen. Wbcatt: a white blood cell dataset annotated with detailed morphological attributes.Advances in Neural Information Processing Systems, 36:50796–50824, 2023
2023
-
[27]
Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly- supervised classification and localization of common thorax diseases
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly- supervised classification and localization of common thorax diseases. InProceedings of the IEEE conference on co...
-
[28]
Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).arXiv preprint arXiv:1902.03368, 2019
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging ...
2018 arXiv
-
[29]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...
2021
-
[30]
Adaptformer: Adapting vision transformers for scalable visual recognition.Advances in Neural Information Processing Systems, 35:16664–16678, 2022
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition.Advances in Neural Information Processing Systems, 35:16664–16678, 2022. 11
2022
-
[31]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016
2016
-
[32]
Learning imbal- anced datasets with label-distribution-aware margin loss
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbal- anced datasets with label-distribution-aware margin loss. InAdvances in Neural Information Processing Systems, pages 1567–1578, 2019
2019
-
[33]
Focal loss for dense object detection
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. InProceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017
2017
-
[34]
Disentangling label distribution for long-tailed visual recognition
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6626–6636, 2021
2021
-
[35]
Dualprompt: Complementary prompting for rehearsal-free continual learning
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. InEuropean Conference on Computer Vision, pages 631–648. Springer, 2022
2022
-
[36]
Resclip: Residual attention for training- free dense vision-language inference
Yuhang Yang, Jinhong Deng, Wen Li, and Lixin Duan. Resclip: Residual attention for training- free dense vision-language inference. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 29968–29978, 2025
2025
-
[37]
Clearclip: Decomposing clip representations for dense vision-language inference
Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decomposing clip representations for dense vision-language inference. InEuropean Conference on Computer Vision, pages 143–160. Springer, 2024
2024
-
[38]
Sinkhorn distances: Lightspeed computation of optimal transport.Advances in neural information processing systems, 26, 2013
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport.Advances in neural information processing systems, 26, 2013
2013
-
[39]
Rademacher and gaussian complexities: Risk bounds and structural results.Journal of machine learning research, 3(Nov):463–482, 2002
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results.Journal of machine learning research, 3(Nov):463–482, 2002
2002
-
[40]
Analysis of representations for domain adaptation.Advances in neural information processing systems, 19, 2006
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation.Advances in neural information processing systems, 19, 2006
2006
-
[41]
No Finding
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition...
2022
Reviewed July 13, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.