Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Open-Set Semi-Supervised Learning for Long-Tailed Medical Datasets

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Open-set SSL for long-tailed medical datasets improves with feature-level regularization and classifier weight normalization, beating strong baselines.

desk verdict A sensible but modest assembly of known components for open-set SSL on long-tailed medical images; the closed-set evidence holds, but the open-set accuracy numbers are not yet reproducible because the evaluation threshold is never specified and no error bars are reported. read the letter →

arxiv 2505.14846 v1 pith:3RJX6BP5 submitted 2025-05-20 cs.CV

Daniya Najiha A. Kareem , Jean Lahoud , Mustansar Fiaz , Amandeep Kumar , Hisham Cholakkal This is my paper · ORCID
classification cs.CV
keywords open-setrecognitionsemi-supervisedlearninglong-tailedclassificationmedicalimagefeatureregularizationclassifierweightnormalizationsimplexequiangulartightframe
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the performance drop caused by long-tailed class imbalance in semi-supervised medical image classification can be corrected by two regularizers on the supervised branch: one that pulls class feature centers toward a maximally separated simplex (a simplex equiangular tight frame) and one that bounds the closed-set classifier weight norms. It claims that these regularizers, combined with an open-set branch that fuses closed-set and multi-binary classifier outputs, improve both seen-class (closed-set) and unseen-class (open-set) accuracy over the state-of-the-art open-set SSL methods on ISIC2018, ISIC2019, and TissueMNIST with only 25% labelled data. A sympathetic reader would care because real medical datasets are imbalanced and contain novel classes, and the method improves accuracy on both without needing class distribution information. The paper also reports that the regularizers help the open-set targets become more reliable, which is the mechanism behind the gains.

What carries the argument

The central mechanism is the open-set target defined in Eq. 6: for each unlabeled sample, the product of the closed-set classifier's probability and the multi-binary classifier's inlier score for known classes, and the sum over products with outlier scores for the final unknown class. This fused target is used as a soft pseudo-label to train the open-set classifier. Two regularizers carry the long-tail correction: (1) feature-center regularization using a fixed simplex ETF classifier to pull class feature centers toward maximal equiangular separation, and (2) classifier weight normalization that projects weight vectors onto a ball of radius $a$. The multi-binary classifier is trained with hard-negative sampling, following the one-vs-all design.

What would settle it

Run the same pipeline on a benchmark like CIFAR-10 with a manually created long-tail and an open-set split, comparing with and without the two regularizers. If closed-set accuracy improves but open-set accuracy is unchanged or worse, the claimed link between long-tail correction and open-set gains breaks. Additionally, reporting the AUROC or a threshold sweep for the open-set evaluator on ISIC2018 would show whether the 41.16% open-set accuracy is robust or a threshold artifact.

Watch

Extended reading notes

Core claim

The central claim is that fixing the long-tail problem at the feature and classifier level is enough to make open-set semi-supervised learning work well on medical images. Specifically, it claims that regularizing feature centers to approximate a simplex equiangular tight frame and normalizing closed-set classifier weights to a bounded ball yields higher closed-set and open-set accuracy than OpenMatch and IoMatch when only 25% of labelled data is available for a subset of seen classes. On ISIC2018 it reports closed-set accuracy 84.08 versus 82.70 for IoMatch and open-set accuracy 41.16 versus 36.75; on ISIC2019 closed-set 79.16 versus 77.22 and open-set 40.90 versus 40.10; on TissueMNIST closed-set 72.30 versus 71.72 and open-set 40.93 versus 40.86. The paper also shows that the open-set accuracy comes close to a fully supervised upper bound of 43.80 on ISIC2018, suggesting the method nearly closes the gap.

Load-bearing premise

The method assumes that the open-set targets computed from the product of closed-set and multi-binary classifier outputs are reliable training signals under class imbalance and sparse labels, and it assumes the reported open-set accuracy is not an artifact of the unspecified hard threshold used during evaluation.

Editorial extensions

If this is right

  • If the central claim is right, long-tail correction on the supervised branch can improve open-set detection without any explicitly labelled outlier data.
  • The two regularizers can be plugged into other open-set SSL frameworks to improve their closed-set and open-set accuracy.
  • The method works with as few as 25% labelled data, making it practical for medical datasets where expert annotations are scarce.
  • The improvement in open-set accuracy comes mainly from better closed-set balance, not from the open-set branch alone.
  • The results should transfer to other long-tailed medical classification tasks, such as chest X-ray or pathology image classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader might test the method's generality by benchmarking it on more recent open-set SSL methods or on non-medical long-tailed datasets, which the paper does not do.
  • The reported open-set accuracy depends on an unspecified hard threshold during evaluation; a threshold-independent metric like AUROC would tell whether the improvement is real or an artifact of threshold choice.
  • The paper deliberately avoids loss re-weighting or class-balanced sampling; combining these with the proposed regularizers could yield further gains, but that combination is not explored.
  • The feature-center regularization assumes final-layer features converge to a simplex ETF under balanced training; if that assumption fails on very high-resolution or multi-modal medical images, the gains may diminish.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes OpenLTR, an open-set semi-supervised learning framework for long-tailed medical image classification. The method combines a closed-set classification branch with feature-center regularization using a simplex equiangular tight frame (ETF) and classifier weight normalization, together with an open-set recognition branch that uses a multi-binary classifier. Open-set targets are computed by fusing closed-set and multi-binary classifier outputs, and are used to train an open-set classifier under an open-set loss and a pseudo-label consistency loss. Experiments are reported on ISIC2018, ISIC2019, and TissueMNIST with 25% labeled data, comparing against SSL baselines such as OpenMatch and IoMatch. The paper claims consistent improvements in closed-set and open-set accuracy on all three datasets.

Significance. If the reported results are reliable, the paper addresses a relevant and under-explored combination of challenges in medical image classification: long-tailed class distributions, open-set classes, and limited labeled data. The proposed components (feature-center regularization and classifier weight normalization) are grounded in existing ideas from neural collapse and long-tail recognition, and the method builds on published building blocks rather than fitting constants to data. The claimed improvements, however, are small in some cases (e.g., TissueMNIST open-set 40.93 vs. 40.86) and are not backed by error bars or a fully specified evaluation protocol. The central empirical claim is therefore not yet verifiable; the methodological ideas are plausible and the study could become a solid contribution after the missing experimental details are provided.

major comments (3)
  1. [Section 4, Tables 1 and 2] The open-set accuracy results are not well-defined because the hard threshold used at evaluation is never specified. The text says only that 'open-set accuracies for the above experiments were obtained by applying a hard threshold during the evaluation phase,' but does not give the threshold value, the score to which it is applied, or the selection rule. Open-set accuracy is threshold-dependent, and in Table 2 the reported advantage over IoMatch is 0.07 percentage points (40.93 vs. 40.86), which could easily be reversed by a different threshold choice. Please specify the exact evaluation protocol and report either threshold-independent metrics (e.g., AUROC or open-set FPR at a fixed TPR) or a sweep over thresholds with error bars over multiple seeds.
  2. [Section 2, Implementation; Section 3] The implementation states that experiments are conducted with 50%, 25%, and 10% labeled data, and the abstract claims evaluation with 'various numbers of labelled samples,' but only 25% labeled experiments are presented in Tables 1 and 2. Without the promised 10% and 50% results, the claim of consistent improvement across label fractions is unsupported. Please include the missing results or revise the text to state that only the 25% setting was evaluated.
  3. [Section 2.2, Eq. (6)] The fused open-set targets in Eq. (6) are formed by the product of closed-set softmax outputs and multi-binary classifier outputs. Under long-tail imbalance, the two classifiers can have very different score scales and confidences, and the paper provides no evidence that this multiplicative fusion produces reliable training targets for the open-set classifier. The claim in Section 2.2 that feature regularization and weight normalization improve the quality of open-set targets is plausible but not directly validated. Please add a diagnostic study, for example measuring the precision/recall of the open-set targets as training progresses, or an ablation showing that the open-set loss improves open-set accuracy only when the regularization components are active.
minor comments (5)
  1. [Section 2.1, first paragraph] The input images are denoted by Y ∈ R^{H×W×D}, but standard notation would use X for input images; also the dimensions seem to be for a 3D volume, while the datasets are 2D images. Please clarify the notation.
  2. [Section 2.2, Eq. (6)] The symbols ezi,k and ow_{i,k} are not fully defined before Eq. (6); earlier text uses zi for the closed-set classifier output and (o_i,k, ¯o_i,k) for the multi-binary output. Please define the fused variables and the relationship between ezi,k and zi explicitly.
  3. [Section 4, Tables 3 and 4] The lower-bound experiment Exp 1 in Table 3 reports 82.60 closed-set accuracy, while the baseline in Table 4 (semi-supervised setting with CE loss) reports 82.73; the descriptions look similar. Please clarify whether these are the same configuration and why the numbers differ.
  4. [Section 2.2, 'Classifier Weight Balancing' paragraph] It is unclear whether weight decay is a component of the proposed method or only a baseline technique; Table 4 lists 'Weight Decay' in the ablation, but the method description does not specify a weight-decay coefficient or how it interacts with classifier weight normalization. Please clarify the role and hyperparameter.
  5. [Introduction, first paragraph] The sentence 'semi-supervised learning (SSL) has been introduced as a promising approach [3, 4], [5], and [6, 7]' has awkward citation grouping; please rephrase for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the method is an assembly of standard SSL components and external regularizers, and the reported gains rest on independent held-out benchmarks.

full rationale

The paper's derivation chain does not reduce to its own inputs. The open-set targets in Eq. 6 are generated by the current closed-set and multi-binary classifiers and then used to train the open-set classifier in Eq. 7; this is standard self-training feedback in semi-supervised learning, not a logical circularity, because the final evaluation is performed on held-out test data and the targets are not fitted to the test labels. The feature regularization (Eq. 4) and classifier weight normalization are borrowed from external prior work on neural collapse and long-tail recognition (references [11] and [9]), not from the authors' own claims, and they are defined independently of the reported results. The authors' self-citations ([1], [2]) concern earlier medical image segmentation papers and are not load-bearing for the proposed open-set SSL framework. The only substantive weakness is that Section 4 says open-set accuracies were obtained by applying a hard threshold during evaluation without specifying the threshold or selection rule, which makes the open-set ranking hard to reproduce; however, this is a reproducibility and verification concern, not a circularity, because there is no quoted step showing a fitted parameter being renamed as a prediction or an equation being defined in terms of its own output. Accordingly, no circular step can be exhibited and the score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The method depends on empirically motivated regularizers and on the self-generated open-set targets being sufficiently accurate; these are assumptions rather than derived facts.

free parameters (4)
  • confidence threshold tau_r (open-set loss) = not reported
    Used in Eq. 7 to select unlabeled samples with max open-set target above threshold; chosen on validation, no value given.
  • confidence threshold tau_p (pseudo-label filtering) = not reported
    Used in Eq. 8 to filter pseudo-labels; chosen on validation, no value given.
  • weight norm bound a = not reported
    Radius for classifier weight normalization in Section 2.1; value not stated.
  • open-set evaluation threshold = not reported
    Hard threshold applied during evaluation to decide known vs unknown; mentioned in Section 4 but never specified.
assumptions (3)
  • domain assumption Balanced training converges to a simplex ETF of feature centers, and enforcing this structure helps long-tailed open-set learning.
    Borrowed from neural collapse literature (Refs. 11,12); assumed to transfer to imbalanced semi-supervised medical data.
  • domain assumption Closed-set and multi-binary classifier outputs can be multiplied to form calibrated open-set targets (Eq. 6).
    Assumed in Section 2.2 without calibration analysis.
  • domain assumption The one-vs-rest binary classifier with hard-negative sampling provides reliable outlier scores under class imbalance.
    Treated as a given from OVANet (Ref. 13) and applied here without additional validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Open-Set Semi-Supervised Learning for Long-Tailed Medical Datasets." pith.science (2026). https://pith.science/paper/3RJX6BP5

@misc{pith2026250514846,
  author       = {Pith},
  title        = {Pith review of: Open-Set Semi-Supervised Learning for Long-Tailed Medical Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3RJX6BP5}},
  note         = {Machine review of arXiv:2505.14846}
}
read the original abstract

Many practical medical imaging scenarios include categories that are under-represented but still crucial. The relevance of image recognition models to real-world applications lies in their ability to generalize to these rare classes as well as unseen classes. Real-world generalization requires taking into account the various complexities that can be encountered in the real-world. First, training data is highly imbalanced, which may lead to model exhibiting bias toward the more frequently represented classes. Moreover, real-world data may contain unseen classes that need to be identified, and model performance is affected by the data scarcity. While medical image recognition has been extensively addressed in the literature, current methods do not take into account all the intricacies in the real-world scenarios. To this end, we propose an open-set learning method for highly imbalanced medical datasets using a semi-supervised approach. Understanding the adverse impact of long-tail distribution at the inherent model characteristics, we implement a regularization strategy at the feature level complemented by a classifier normalization technique. We conduct extensive experiments on the publicly available datasets, ISIC2018, ISIC2019, and TissueMNIST with various numbers of labelled samples. Our analysis shows that addressing the impact of long-tail data in classification significantly improves the overall performance of the network in terms of closed-set and open-set accuracies on all datasets. Our code and trained models will be made publicly available at https://github.com/Daniyanaj/OpenLTR.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 30 canonical work pages

  1. [1]

    Open-Set Semi-Supervised Learning for Long-Tailed Medical Datasets

    INTRODUCTION Medical image recognition includes classifying images, such as pathological images, X-rays, MRI scans, and CT scans into different classes to aid in diagnosis, disease monitoring, and treatment planning [1, 2]. In the practical scenarios, the classification task faces numerous challenges. Often, labelled data for categories of interest is lim...

    work page Pith review arXiv 2025

  2. [2]

    The input images, denoted byY ∈ RH×W×D , are processed to form labelled batchI C and unlabelled batchI U

    METHODOLOGY We propose an integrated framework to classify known cate- gories in the long tail medical data from a few samples and identify the unknown classes within the dataset. The input images, denoted byY ∈ RH×W×D , are processed to form labelled batchI C and unlabelled batchI U. Our network is organized into two main branches, a closed-set classific...

    work page

  3. [3]

    All the networks were trained on 25% labelled data with ResNet18 backbone for 100 epochs with similar hyper-parameters

    RESULTS We benchmark our method against the top-performing semi- supervised models in the literature. All the networks were trained on 25% labelled data with ResNet18 backbone for 100 epochs with similar hyper-parameters. The performance of our method on the ISIC2018 and ISIC2019 datasets is shown in Table 1. Our method has a clear improvement over the st...

    work page

  4. [4]

    In our network setup, we train the network only with samples of five classes with the remaining reserved for open-set classifica- tion

    DISCUSSION AND ABLATIONS Semi-Supervised Learning:We here study the lower and upper bounds for the semi-supervised learning task. In our network setup, we train the network only with samples of five classes with the remaining reserved for open-set classifica- tion. The lower bound accuracies for this data partition were found as 82.6 (Exp 1) and 80.81 (Ex...

    work page

  5. [5]

    To alleviate the effect of class imbalance, we employ feature regularization and classifier weight normalization

    CONCLUSION We propose a novel open-set framework that addresses long- tail classification in medical images with few-shot learning. To alleviate the effect of class imbalance, we employ feature regularization and classifier weight normalization. It helps in preserving the innate properties of the classifier exhibited in a balanced data classification task...

    work page 2018

  6. [6]

    As an upper-bound, we train the network with whole training data for the first five classes, employing a multi-class classifier, which showed an accuracy of 86.65 (Exp 5)

    Experiments 1 and 2 were trained with 25% of labelled samples with multiclass classifier and multi-binary classifier respectively. As an upper-bound, we train the network with whole training data for the first five classes, employing a multi-class classifier, which showed an accuracy of 86.65 (Exp 5). The open-set accuracies for the above experiments were...

    work page

  7. [7]

    Semi-supervised clas- sification of radiology images with noteacher: A teacher that is not mean,

    Balagopal U., Cuong N., et al., “Semi-supervised clas- sification of radiology images with noteacher: A teacher that is not mean,”MIA, vol. 73, pp. 102148, 2021

    work page 2021

  8. [8]

    Improving 3-d medical image segmentation at bound- ary regions using local self-attention and global volume mixing,

    Daniya Najiha Abdul Kareem, Mustansar Fiaz, Noa Novershtern, Jacob Hanna, and Hisham Cholakkal, “Improving 3-d medical image segmentation at bound- ary regions using local self-attention and global volume mixing,”IEEE Transactions on Artificial Intelligence, vol. 5, no. 6, pp. 3233–3244, 2024

    work page 2024

Show all 31 references
  1. [9]

    Medical image seg- mentation using directional window attention,

    Daniya Najiha A. Kareem, Mustansar Fiaz, Noa Nover- shtern, and Hisham Cholakkal, “Medical image seg- mentation using directional window attention,” in2024 IEEE International Symposium on Biomedical Imaging (ISBI), 2024, pp. 1–5

    2024

  2. [10]

    Pseudo-labeling generative adversarial networks for medical image classification,

    J. Mao, X. Yin, et al., “Pseudo-labeling generative adversarial networks for medical image classification,” Computers in Biology and Medicine, vol. 147, pp. 105729, 2022

    2022

  3. [11]

    Boosting deep learning risk prediction with generative adversarial networks for elec- tronic health records,

    Z. Che, Yu Cheng, et al., “Boosting deep learning risk prediction with generative adversarial networks for elec- tronic health records,” inICDM. IEEE, 2017, pp. 787– 792

    2017

  4. [12]

    Dira: Discriminative, restorative, and adversarial learning for self-supervised medical image analysis,

    Fatemeh H., Mohammad R. H. Taher, et al., “Dira: Discriminative, restorative, and adversarial learning for self-supervised medical image analysis,” inIEEE CVPR, 2022, pp. 20824–20834

    2022

  5. [13]

    Self-supervised mean teacher for semi-supervised chest x-ray classification,

    Fengbei Liu, Yu Tian, et al., “Self-supervised mean teacher for semi-supervised chest x-ray classification,” inInternational Workshop on Machine Learning in Medical Imaging. Springer, 2021, pp. 426–436

    2021

  6. [14]

    Mixmatch: A holistic approach to semi- supervised learning,

    David B. et al., “Mixmatch: A holistic approach to semi- supervised learning,”NeurIPS, vol. 32, 2019

    2019

  7. [15]

    Neural collapse: A review on modelling principles and generalization,

    Vignesh Kothapalli, “Neural collapse: A review on modelling principles and generalization,”arXiv preprint arXiv:2206.04041, 2022

    2022 arXiv

  8. [16]

    Long-tailed recognition via weight balancing,

    Shaden A. et al., “Long-tailed recognition via weight balancing,” inIEEE CVPR, 2022, pp. 6897–6907

    2022

  9. [17]

    Investigating the impact of data normalization on classification performance,

    D. Singh and B. Singh, “Investigating the impact of data normalization on classification performance,”Applied Soft Computing, vol. 97, pp. 105524, 2020

    2020

  10. [18]

    Prevalence of neural collapse during the terminal phase of deep learn- ing training,

    Vardan P., XY Han, and David L D., “Prevalence of neural collapse during the terminal phase of deep learn- ing training,”Proceedings of the National Academy of Sciences, vol. 117, no. 40, pp. 24652–24663, 2020

    2020

  11. [19]

    Understanding imbal- anced semantic segmentation through neural collapse,

    Z. Zhong, Jiequan Cui, et al., “Understanding imbal- anced semantic segmentation through neural collapse,” inCVPR, 2023

    2023

  12. [20]

    Ovanet: One-vs-all network for universal domain adaptation,

    K. Saito and Kate S., “Ovanet: One-vs-all network for universal domain adaptation,” inIEEE ICCV, 2021, pp. 9000–9009

    2021

  13. [21]

    Noel C. F. Codella, D. Gutman, et al., “Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collabo- ration (isic),” in2018 IEEE ISBI, 2018, pp. 168–172

    2017

  14. [22]

    Flexmatch: Boosting semi- supervised learning with curriculum pseudo labeling,

    Bowen Z., Y . Wang, et al., “Flexmatch: Boosting semi- supervised learning with curriculum pseudo labeling,” NeurIPS, vol. 34, pp. 18408–18419, 2021

    2021

  15. [23]

    Simmatch: Semi-supervised learning with similarity matching,

    M. Zheng, S. You, et al., “Simmatch: Semi-supervised learning with similarity matching,” inIEEE CVPR, 2022, pp. 14471–14481

    2022

  16. [24]

    Table 2 shows that our method outperforms the existing methods

    where 2 head classes with 13800 and 9800 samples, 2 tail classes with 1466 and 1926 samples, and the rest of the classes contain samples in the range (2500, 6500). Table 2 shows that our method outperforms the existing methods

    1926

  17. [25]

    Comatch: Semi-supervised learning with contrastive graph regularization,

    J. Li, C. Xiong, et al., “Comatch: Semi-supervised learning with contrastive graph regularization,” in ICCV, 2021, pp. 9475–9484

    2021

  18. [26]

    Openmatch: open-set consistency regularization for semi-supervised learning with outliers,

    K. Saito, D. Kim, and Kate S., “Openmatch: open-set consistency regularization for semi-supervised learning with outliers,” inNeurIPS, 2021, pp. 25956–25967

    2021

  19. [27]

    Iomatch: Simplifying open-set semi-supervised learning with joint inliers and outliers utilization,

    Z. Li, Lei Qi, et al., “Iomatch: Simplifying open-set semi-supervised learning with joint inliers and outliers utilization,” inICCV, 2023, pp. 15870–15879

    2023

  20. [28]

    Deep residual learning for image recognition,

    K. He, X. Zhang, et al., “Deep residual learning for image recognition,” inIEEE CVPR, 2016, pp. 770–778

    2016

  21. [29]

    The ham10000 dataset: A large collection of multi- source dermatoscopic images of common pigmented skin lesions,

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler, “The ham10000 dataset: A large collection of multi- source dermatoscopic images of common pigmented skin lesions,”Scientific Data, vol. 5, 08 2018

    2018

  22. [30]

    Bcn20000: Dermoscopic lesions in the wild,

    H. Carlos, Marc C., et al., “Bcn20000: Dermoscopic lesions in the wild,”Scientific Data, vol. 11, 06 2024

    2024

  23. [31]

    Medmnist v2 - a large-scale lightweight benchmark for 2d and 3d biomedical image classification,

    Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni, “Medmnist v2 - a large-scale lightweight benchmark for 2d and 3d biomedical image classification,”Scientific Data, vol. 10, no. 1, Jan. 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.