NoduLoCC2026: Lung Nodule Localization and Classification Contest from Chest X-Ray Images
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-26 14:50 UTCgrok-4.3pith:XZKJBPSJrecord.jsonopen to challenge →
The pith
A challenge on chest X-ray images shows classification of lung nodules reaches moderate accuracy while localization stays limited.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Results from the NoduLoCC2026 submissions establish that current methods can classify the presence of lung nodules in chest X-rays at balanced accuracy 0.72 and AUC-ROC 0.79, yet the same approaches identify the correct number of nodules in just 53 percent of external test images and locate them with a median distance error of 12.83 mm, confirming localization as the more difficult task.
What carries the argument
The NoduLoCC2026 dataset together with its dual evaluation protocol that scores classification by balanced accuracy and AUC-ROC while scoring localization by nodule-count match rate and median Euclidean distance.
Load-bearing premise
The external test dataset is representative of real clinical variability and the chosen metrics capture clinical utility.
What would settle it
A follow-up test collection drawn from different hospitals or scanners that drops the leading method's balanced accuracy below 0.60 or its correct-count rate below 40 percent.
Figures
read the original abstract
We propose NoduLoCC2026, a challenge on lung nodule detection and localization in chest X-ray images. We have provided a dataset for both tasks and received submissions from 5 international teams. The participating teams' solutions are presented in this work along with results on an external dataset used for testing. Proposed methods show good performance on the classification task. The best method shows a balanced accuracy score of 0.72 and AUC-ROC of 0.79. We highlight the limitations of current approaches for the localization task, with the best approach having predicted the correct number of nodules on 53\% of the test images with a median distance of 12.83mm, showing that it is a more challenging task than the first one. The challenge website is available via https://gt-i2mdp.github.io/website/nodule_challenge.html.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents the NoduLoCC2026 challenge for lung nodule localization and classification from chest X-ray images. A dataset is provided for both tasks, submissions are received from 5 international teams, and the teams' solutions are evaluated on an external held-out test set. The best classification method achieves a balanced accuracy of 0.72 and AUC-ROC of 0.79; the best localization method predicts the correct nodule count on 53% of test images with a median Euclidean distance of 12.83 mm, indicating that localization remains more challenging.
Significance. If the dataset construction, annotation protocol, and test-set representativeness can be established, the work would provide a useful community benchmark that quantifies the current performance gap between classification and localization tasks on chest X-rays and could guide future method development.
major comments (3)
- [Abstract] Abstract: performance metrics (balanced accuracy 0.72, AUC-ROC 0.79, 53% correct nodule count, median distance 12.83 mm) are stated without any accompanying information on training or test dataset size, annotation protocol, inter-annotator agreement, or statistical testing, so the support for the headline claims cannot be verified.
- [Abstract] Abstract: the manuscript supplies no description or validation showing that the external test set matches real clinical distributions (patient demographics, imaging conditions, nodule appearance variability), which is load-bearing for the claim that the reported scores demonstrate method performance rather than dataset-specific artifacts.
- [Abstract] Abstract: no details are given on the five submitted methods, their architectural choices, training procedures, or comparisons against standard baselines, preventing assessment of whether the observed scores reflect meaningful progress.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback. We agree that the abstract requires expansion to provide necessary context and will revise it accordingly. We also address the need for additional discussion on test-set limitations and method details in the main text.
read point-by-point responses
-
Referee: [Abstract] Abstract: performance metrics (balanced accuracy 0.72, AUC-ROC 0.79, 53% correct nodule count, median distance 12.83 mm) are stated without any accompanying information on training or test dataset size, annotation protocol, inter-annotator agreement, or statistical testing, so the support for the headline claims cannot be verified.
Authors: We agree the abstract is overly concise. The manuscript body (Section 2) provides dataset sizes, annotation protocol by multiple radiologists, inter-annotator agreement metrics, and Section 5 includes statistical testing. We will revise the abstract to incorporate these key details and cross-references to the relevant sections. revision: yes
-
Referee: [Abstract] Abstract: the manuscript supplies no description or validation showing that the external test set matches real clinical distributions (patient demographics, imaging conditions, nodule appearance variability), which is load-bearing for the claim that the reported scores demonstrate method performance rather than dataset-specific artifacts.
Authors: We acknowledge that the manuscript does not include a formal validation or comparison of the external test set to broader clinical distributions. The test set was deliberately collected from a separate institution to create a challenging held-out evaluation. We will add a dedicated limitations paragraph in the discussion section addressing potential dataset shift and the absence of explicit demographic matching. revision: yes
-
Referee: [Abstract] Abstract: no details are given on the five submitted methods, their architectural choices, training procedures, or comparisons against standard baselines, preventing assessment of whether the observed scores reflect meaningful progress.
Authors: Section 4 of the manuscript summarizes the five submitted solutions, but we concur that greater technical depth is needed. We will expand this section with additional architectural and training details supplied by the teams and will include explicit comparisons against standard baselines (e.g., ResNet classifiers and Faster R-CNN detectors). revision: yes
Circularity Check
No circularity: empirical contest results from independent teams on held-out test set
full rationale
The paper reports classification and localization performance metrics (balanced accuracy 0.72, AUC-ROC 0.79, nodule count accuracy 53%, median distance 12.83 mm) obtained by external teams on an external test dataset. No equations, fitted parameters, derivations, or self-citations are used to define or predict these outcomes; the results are direct empirical measurements. No self-definitional, fitted-input, or uniqueness-imported patterns exist. The derivation chain is absent, rendering circularity analysis inapplicable.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Data From LIDC-IDRI, 2015
Samuel G Armato, III, Geoffrey McLennan, Luc Bidaut, Michael F McNitt-Gray, Charles R Meyer, Anthony P Reeves, Binsheng Zhao, Denise R Aberle, Claudia I Henschke, Eric A Hoffman, Ella A Kazerooni, Heber MacMahon, Edwin J R Van Beek, David Yankelevitz, Al- berto M Biancardi, Peyton H Bland, Matthew S Brown, Roger M Engelmann, Gary E Laderach, Daniel Max, R...
2015
-
[2]
Computed tomography—an increasing source of radiation exposure.New England journal of medicine, 357(22):2277–2284, 2007
David J Brenner and Eric J Hall. Computed tomography—an increasing source of radiation exposure.New England journal of medicine, 357(22):2277–2284, 2007
2007
-
[3]
Padchest: A large chest x-ray image dataset with multi- label annotated reports.Medical Image Analysis, 66:101797, December 2020
Aurelia Bustos, Antonio Pertusa, Jose-Maria Sali- nas, and Maria de la Iglesia-Vay´ a. Padchest: A large chest x-ray image dataset with multi- label annotated reports.Medical Image Analysis, 66:101797, December 2020
2020
-
[4]
De- sign and development of a new multi-projection x-ray system for chest imaging.IEEE Trans
Amarpreet S Chawla, Sarah Boyce, Lacey Wash- ington, H Page McAdams, and Ehsan Samei. De- sign and development of a new multi-projection x-ray system for chest imaging.IEEE Trans. Nucl. Sci., 56(1):36–45, 2009
2009
-
[5]
A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation
Zhihong Chen, Maya Varma, Jean-Benoit Del- brouck, Magdalini Paschali, Louis Blanke- meier, Dave Van Veen, Jeya Maria Jose Vala- narasu, Alaa Youssef, Joseph Paul Cohen, Ed- uardo Pontes Reis, Emily B. Tsai, Andrew John- ston, Cameron Olsen, Tanishq Mathew Abra- ham, Sergios Gatidis, Akshay S Chaudhari, and Curtis Langlotz. Chexagent: Towards a founda- ti...
work page Pith review arXiv 2024
-
[6]
Tsai, Christopher F
Zhihong Chen, Maya Varma, Justin Xu, Mag- dalini Paschali, Dave Van Veen, Andrew John- ston, Alaa Youssef, Louis Blankemeier, Christian Bluethgen, Stephan Altmayer, Jeya Maria Jose Valanarasu, Mohamed Siddig Eltayeb Muneer, Eduardo Pontes Reis, Joseph Paul Cohen, Cameron Olsen, Tanishq Mathew Abraham, Emily B. Tsai, Christopher F. Beaulieu, Jenia Jitsev, ...
2024
-
[7]
Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.Biometrics, 44(3):837–845, September 1988
E R DeLong, D M DeLong, and D L Clarke- Pearson. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.Biometrics, 44(3):837–845, September 1988
1988
-
[8]
An image is worth 16x16 words: Transformers for image recognition at scale, 2021
Alexey Dosovitskiy, Lucas Beyer, Alexan- der Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021
2021
-
[9]
Mamba: Linear-time sequence modeling with selective state spaces, 2024
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024
2024
-
[10]
Gbd cancer compare data visualiza- tion
Institute for Health Metrics and Evaluation (IHME). Gbd cancer compare data visualiza- tion. IHME, University of Washington, Seattle, WA, 2025. Accessed: 30 apr. 2026
2025
-
[11]
World fact sheet: Global cancer observatory
International Agency for Research on Cancer. World fact sheet: Global cancer observatory. Fact sheet, World Health Organization, 2024
2024
-
[12]
Mong, Safwan S
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. Chexpert: A large chest radiogra...
2019
-
[13]
MIMIC-CXR, a de-identified pub- licly available database of chest radiographs with free-text reports.Scientific Data, 6(1):317, De- cember 2019
Alistair E W Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-Ying Deng, Roger G Mark, and Steven Horng. MIMIC-CXR, a de-identified pub- licly available database of chest radiographs with free-text reports.Scientific Data, 6(1):317, De- cember 2019
2019
-
[14]
Key challenges for delivering clinical impact with artificial intelligence.BMC Med., 17(1):195, October 2019
Christopher J Kelly, Alan Karthikesalingam, Mustafa Suleyman, Greg Corrado, and Dominic King. Key challenges for delivering clinical impact with artificial intelligence.BMC Med., 17(1):195, October 2019
2019
-
[15]
Focal loss for dense object detection.IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):318–327, February 2020
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaim- ing He, and Piotr Dollar. Focal loss for dense object detection.IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):318–327, February 2020
2020
-
[16]
Vmamba: Visual state space model, 2024
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jian- bin Jiao, and Yunfan Liu. Vmamba: Visual state space model, 2024
2024
-
[17]
Dhayalini M and Revathi alias Ponmozhi B. Multi-phase deep learning framework with multi- scale adaptive swin transformer and embedding attention for precision lung nodule detection and classification.Scientific Reports, 16(1):1652, Dec 2025
2025
-
[18]
U-mamba: Enhancing long-range dependency for biomedical image segmentation, 2024
Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation, 2024
2024
-
[19]
Guidelines for manage- ment of incidental pulmonary nodules detected on ct images: from the fleischner society 2017
Heber MacMahon, David P Naidich, Jin Mo Goo, Kyung Soo Lee, Ann NC Leung, John R Mayo, Atul C Mehta, Yoshiharu Ohno, Charles A Pow- ell, Mathias Prokop, et al. Guidelines for manage- ment of incidental pulmonary nodules detected on ct images: from the fleischner society 2017. Radiology, 284(1):228–243, 2017
2017
-
[20]
Ad- vances in imaging for lung emphysema.Ann
Katharina Martini and Thomas Frauenfelder. Ad- vances in imaging for lung emphysema.Ann. Transl. Med., 8(21):1467, 2020
2020
-
[21]
Is it time to replace cnns with transformers for medical im- ages?, 2021
Christos Matsoukas, Johan Fredin Haslum, Mag- nus S¨ oderberg, and Kevin Smith. Is it time to replace cnns with transformers for medical im- ages?, 2021
2021
-
[22]
Development and val- idation of deep learning–based automatic detec- tion algorithm for malignant pulmonary nodules on chest radiographs.Radiology, 290(1):218–228, 2019
Ju Gang Nam, Sunggyun Park, Eui Jin Hwang, Jong Hyuk Lee, Kwang-Nam Jin, Kun Young Lim, Thienkai Huy Vu, Jae Ho Sohn, Sangheum Hwang, Jin Mo Goo, et al. Development and val- idation of deep learning–based automatic detec- tion algorithm for malignant pulmonary nodules on chest radiographs.Radiology, 290(1):218–228, 2019
2019
-
[23]
Nguyen, Khanh Lam, Linh T
Ha Q. Nguyen, Khanh Lam, Linh T. Le, Hieu H. Pham, Dat Q. Tran, Dung B. Nguyen, Dung D. Le, Chi M. Pham, Hang T. T. Tong, Diep H. Dinh, Cuong D. Do, Luu T. Doan, Cuong N. Nguyen, Binh T. Nguyen, Que V. Nguyen, Au D. Hoang, Hien N. Phan, Anh T. Nguyen, Phuong H. Ho, Dat T. Ngo, Nghia T. Nguyen, Nhan T. Nguyen, Minh Dao, and Van Vu. Vindr-cxr: An open datas...
2022
-
[24]
Eduardo H. P. Pooch, Pedro Ballester, and Ro- drigo C. Barros. Can we trust deep learning based diagnosis? the impact of domain shift in chest radiograph classification. InThoracic Image Analysis: Second International Workshop, TIA 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 8, 2020, Proceedings, page 74–83, Berlin, Heidelberg, 2020. S...
2020
-
[25]
Castro, Anton Schwaighofer, Matthew P
Fernando P´ erez-Garc´ ıa, Harshita Sharma, Sam Bond-Taylor, Kenza Bouzid, Valentina Salvatelli, Maximilian Ilse, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Matthew P. Lun- gren, Maria Teodora Wetscherek, Noel Codella, Stephanie L. Hyland, Javier Alvarez-Valle, and Ozan Oktay. Exploring scalable medical image encoders beyond text supervision.Na...
2025
-
[26]
Lungren, and Andrew Y
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpan- skaya, Matthew P. Lungren, and Andrew Y. Ng. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning, 2017
2017
-
[27]
Asymmetric loss for multi- label classification
Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi- label classification. In2021 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), page 82–91. IEEE, October 2021
2021
-
[28]
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, C´ ıan Hughes, Charles Lau, et al. Medgemma technical report.arXiv preprint arXiv:2507.05201, 2025. 14
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[29]
Laleh Seyyed-Kalantari, Haoran Zhang, Matthew B. A. McDermott, Irene Y. Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intel- ligence algorithms applied to chest radiographs in under-served patient populations.Nature Medicine, 27(12):2176–2182, Dec 2021
2021
-
[30]
Dino-cxr: A self supervised method based on vision transformer for chest x- ray classification, 2023
Mohammadreza Shakouri, Fatemeh Iranmanesh, and Mahdi Eftekhari. Dino-cxr: A self supervised method based on vision transformer for chest x- ray classification, 2023
2023
-
[31]
Development of a Digi- tal Image Database for Chest Radiographs With and Without a Lung Nodule.American Jour- nal of Roentgenology, 174(1):71–74, 2000
Shiraishi, Junji and Katsuragawa, Shigehiko and Ikezoe, Junpei and Matsumoto, Tsuneo and Kobayashi, Takeshi and Komatsu, Ken-ichi and Matsui, Mitate and Fujita, Hiroshi and Kodera, Yoshie and Doi, Kunio. Development of a Digi- tal Image Database for Chest Radiographs With and Without a Lung Nodule.American Jour- nal of Roentgenology, 174(1):71–74, 2000. P...
2000
-
[32]
Sidhu, Jainy Sachdeva, and D
R.K. Sidhu, Jainy Sachdeva, and D. Katoch. Seg- mentation of retinal blood vessels by a novel hybrid technique- principal component analysis (pca) and contrast limited adaptive histogram equalization (clahe).Microvascular Research, 148:104477, 2023
2023
-
[33]
Mingxing Tan and Quoc V. Le. Efficientnetv2: Smaller models and faster training, 2021
2021
-
[34]
Gemma 3, 2025
Gemma Team. Gemma 3, 2025
2025
-
[35]
Reduced lung-cancer mortality with low-dose computed tomographic screening.New England Journal of Medicine, 365(5):395–409, 2011
National Lung Screening Trial Research Team. Reduced lung-cancer mortality with low-dose computed tomographic screening.New England Journal of Medicine, 365(5):395–409, 2011
2011
-
[36]
Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 2097–2106, 2017
2097
-
[37]
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba...
2022
-
[38]
Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.PLoS Med., 15(11):e1002683, November 2018
John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.PLoS Med., 15(11):e1002683, November 2018
2018
-
[39]
Sigmoid loss for language image pre-training, 2023
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023
2023
-
[40]
The blind spots on chest computed tomography: what do we miss.J Thorac Dis, 16(12):8782–8795, 2024
Li Zhang, Xin Wen, Jing-Wen Ma, Jian-Wei Wang, Yao Huang, Ning Wu, and Meng Li. The blind spots on chest computed tomography: what do we miss.J Thorac Dis, 16(12):8782–8795, 2024
2024
-
[41]
Learning deep features for discriminative localization
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), page 2921–2929. IEEE, June 2016. 15
2016
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.