Pith. sign in

REVIEW 3 major objections 4 minor 37 references

A tissue and cell-level annotated H&E and PD-L1 histopathology image dataset in non-small cell lung cancer

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read The paper claims to introduce the first public NSCLC histopathology dataset with manually annotated H&E from metastatic sites and cell-level PD-L1 IHC annotations, validated by baseline models across three tasks.

desk verdict A genuinely useful NSCLC pathology dataset with real novelty in metastatic H&E and PD-L1 IHC; the canonical-annotator selection is partially circular but the released all-reader data mitigates it. read the letter →

arxiv 2507.16855 v1 pith:KQYNBZEB submitted 2025-07-21 q-bio.QM cs.CVeess.IV

classification q-bio.QMcs.CVeess.IV
keywords non-smallcelllungcancercomputationalpathologyPD-L1immunohistochemistrytissuesegmentationnucleidetectionwhole-slideimagestumorimmunemicroenvironmentannotateddataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims to fill a specific gap in public resources for computational pathology of non-small cell lung cancer: no existing public dataset provides manual annotations of H&E slides from metastatic sites or cell-level PD-L1 immunohistochemistry annotations. The authors present IGNITE, a multi-stain, multi-center, multi-scanner release of 887 fully annotated regions from 155 patients, spanning 16-class H&E tissue segmentation, nuclei detection in PD-L1 IHC, and PD-L1-positive tumor cell detection. Baseline models trained on the data reach an F1 of 0.79 for H&E segmentation (0.81 when classes are grouped for tumor-infiltrating-lymphocyte analysis), 0.87 for nuclei detection, and 0.61 for PD-L1-positive tumor cells against a multi-reader reference whose own inter-reader F1 is 0.70. A sympathetic reading is that the dataset, if reliable, gives the field a common training and benchmarking ground for the metastatic and immunotherapy-relevant cases that previously had no annotated public data.

What carries the argument

The central object is the dataset itself: 887 annotated regions of interest with pixel masks and cell point labels. The PD-L1-positive tumor cell annotations are produced by a two-step semi-automated procedure: annotators hand-draw polygon regions grouping cells by class, a nuclei detector trained on the point annotations predicts nucleus centers, and each detected nucleus receives the polygon's label. The H&E annotations use an iterative human-in-the-loop process in which a preliminary segmentation model and its uncertainty maps highlight candidate regions for correction, and serial CD68-stained sections guide macrophage labels. Test sets are manually annotated by multiple readers so that model performance can be compared with human inter-rater agreement, and a canonical reader per test set is proposed as the reference standard.

What would settle it

Re-annotate a random sample of the released PD-L1 test ROIs by an independent pathology group and compute pairwise F1 against the canonical readers; if the independent group's agreement with the canonical labels falls appreciably below the reported 0.70 reader-reader F1, the reference standard is not reproducible. A second direct check: train a model on the PD-L1 labels and compare its predicted positive-tumor-cell fraction with clinically scored TPS on an external cohort; weak correlation would indicate the cell-level annotations do not capture the biomarker signal they are meant to support.

Watch

Extended reading notes

Core claim

The central claim is that the IGNITE data toolkit is the first public NSCLC dataset with manual annotations of H&E in metastatic sites and of PD-L1 IHC at cell level. The authors support this by releasing 407 ROIs with 16-class H&E tissue masks covering 166 mm², 135 ROIs with 91,164 point-annotated nuclei, and 344 ROIs with 859,681 label-assigned cells for PD-L1-positive tumor, PD-L1-negative tumor, and non-tumor classes. They define reference standards for the detection test sets through multi-reader annotations and select canonical readers for future benchmarks. Technical validation shows that baseline deep-learning models can be trained on the data and reach performance within the range of human agreement for nuclei detection, while the PD-L1 tumor-cell task remains harder with reader-algorithm F1 of 0.61 against 0.70 reader-reader F1.

Load-bearing premise

The dataset is only as good as its annotations; the load-bearing premise is that the semi-automated PD-L1-positive tumor cell labels, which human annotators reproduce only at F1 0.70, are consistent enough to serve as a reference standard for training and benchmarking, and that the AI-assisted annotations in the training set do not quietly inject the model's own mistakes back into the ground truth.

Editorial extensions

If this is right

  • A segmentation model trained on the H&E component reaches F1 0.79 across 16 classes and 0.81 for TIL-relevant grouped classes, supporting quantification of tumor, stroma, inflammation, necrosis, and macrophages.
  • A nuclei detection model trained on PD-L1 IHC reaches F1 0.87, matching reader-reader agreement, so the dataset can serve as a supervised training source for cell detection in IHC.
  • PD-L1-positive tumor cell detection reaches F1 0.61 against readers whose mutual agreement is 0.70, so the released canonical-reader annotations give future benchmarks a definite reference to compare against.
  • Because the slides come from two centers, three PD-L1 antibody clones, and multiple scanners, models trained and evaluated here provide a stain- and scanner-diversity benchmark for NSCLC.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If used broadly, the PD-L1-positive tumor cell detections could be aggregated into a tumor proportion score analogue and tested against clinical TPS in an independent cohort, something the paper does not itself do.
  • The 0.70 inter-reader F1 on PD-L1 tumor cells implies that supervised models should be judged against human noise; near-perfect agreement with the released labels would be implausible, and benchmarks should report reader variability.
  • Because metastatic sites include liver and bone, the H&E taxonomy may transfer to other cancers with similar non-lung stroma, but that extension is untested in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript introduces the IGNITE data toolkit, a publicly released collection of 887 annotated regions of interest (ROIs) from 155 unique patients with non-small cell lung cancer (NSCLC). The toolkit comprises three complementary annotated datasets: (i) H&E tissue segmentation with 16 tissue classes covering primary and metastatic sites, (ii) nuclei detection on PD-L1 IHC slides, and (iii) PD-L1-positive tumor cell detection on PD-L1 IHC slides. The authors describe a human-in-the-loop annotation workflow with AI-guided ROI selection and AI-assisted annotation for training data, while test sets were manually annotated. They provide technical validation by training nnUNet-based segmentation models and YOLOv5-based detection models, reporting F1 scores on hold-out test sets with multi-reader agreement analysis. The paper positions the dataset as the first public NSCLC resource with manual H&E annotations in metastatic sites and cell-level PD-L1 IHC annotations.

Significance. If the dataset is released as described, it addresses a genuine gap in public computational pathology resources for NSCLC. The inclusion of metastatic samples and PD-L1 IHC with cell-level annotations is valuable for developing and benchmarking models for TIME quantification and biomarker research. The paper's strengths include the detailed annotation protocol, multi-center and multi-scanner data, multiple expert readers on test sets, and the release of model weights and evaluation code. The explicit reporting of inter-reader variability (e.g., mean reader-reader F1 of 0.70 for PD-L1+ tumor cell detection) is a useful and honest characterization of label noise. However, the benchmark reference standard for the PD-L1 detection test set is chosen with the aid of baseline model performance, which introduces a circularity that needs to be addressed before the resource can be used as a fair evaluation benchmark.

major comments (3)
  1. [Usage Notes; Testing] The proposed canonical annotator for the PD-L1 test sets is selected using the baseline algorithm's F1 score. The Usage Notes state that the canonical reader is chosen by 'the best combined ranking of two outcomes: i) highest F1 score among the readers and ii) the highest F1 score versus the respective baseline algorithms.' For the RUMC cases, P2 was selected with the highest F1 score versus the baseline model (0.7); for SCDC, P5 was selected with the second-highest score versus the baseline model (0.59). Since pdl1_annotations.json distributes these canonical readers' annotations as the proposed reference standard, any future method evaluated against this file is being scored against a reader chosen partly for agreement with a particular YOLOv5 model. The statement in the Testing section that 'model output was not inspected for generating the annotations nor selecting the ROIs' does not apply to this selection step. This is a load-bearing issue for the benchmarking claim. I recommend either selecting canonical annotators without reference to algorithm outputs (e.g., solely on reader-reader agreement), or clearly documenting that the canonical standard is algorithm-influenced and designating the all-reader annotations as the primary reference standard for fairness assessments.
  2. [Abstract; Data Records] The abstract reports 887 fully annotated ROIs, but the Data Records section reports 407 ROIs for H&E segmentation, 135 ROIs for nuclei detection, and 344 ROIs for PD-L1+ tumor cell detection, which sum to 886. This discrepancy in the headline dataset count should be reconciled in the text and in the Zenodo repository metadata before publication.
  3. [Annotations for PD-L1 IHC nuclei & positive tumor cell detection datasets] The PD-L1+ tumor cell annotations are produced by a semi-automated process: hand-drawn polygon labels are intersected with detections from a nuclei detector, and the mean inter-reader F1 score for this task is only 0.70. While the authors acknowledge variability, the paper does not quantify how much of the remaining label noise stems from the nuclei detection step versus genuine pathologist disagreement. Because this annotation pipeline is used to create a large fraction of the released training data, a systematic error analysis (e.g., comparing fully manual annotations against the semi-automated output on a small sample) would strengthen confidence that models trained on this resource learn biological signal rather than detector artifacts.
minor comments (4)
  1. [Performance metrics] The word 'deteciton' appears to be a typo for 'detection'.
  2. [Author contributions] There are inconsistent spacing and punctuation artifacts in author initials (e.g., 'M.V .D.V', 'V .D', 'R ˛ aczkowska' in reference 21); these should be cleaned up.
  3. [Technical Validation] The reported F1 scores are presented as point estimates without confidence intervals or variance measures across folds/cases. Given the multi-reader design, adding bootstrap confidence intervals or per-case spreads would better convey the uncertainty in these benchmark results.
  4. [Background & Summary] The claim of being 'the first public NSCLC dataset with manual annotations of H&E in metastatic sites and PD-L1 IHC' is stated twice; a brief comparison with the most closely related existing datasets (e.g., in terms of IHC annotations) would make the novelty claim easier to verify.

Circularity Check

1 steps flagged · score 3.0 of 10

PD-L1 test-set canonical annotator is selected using baseline-model F1 scores, so the proposed reference standard is partly circular.

  1. fitted input called prediction [Usage Notes, canonical annotator paragraph]
    "Moreover, we propose to use a single reader per set as the canonical annotator. This canonical annotator functions as a proposed reference standard for future benchmarks and as a way for users of the data to concisely report their own benchmarking results. For this purpose, we choose the three readers who have the best combined ranking of two outcomes: i) highest F1 score among the readers and ii) the highest F1 score versus the respective baseline algorithms."

    The canonical reader distributed in pdl1_annotations.json is chosen partly by the 'highest F1 score versus the respective baseline algorithms' criterion (P2 for RUMC, highest at 0.7; P5 for SCDC, second-highest at 0.59). The proposed reference standard is therefore, by construction, a reader selected for concordance with one particular baseline detector. Any future benchmark scored against pdl1_annotations.json evaluates against that model-influenced reader, so reader-algorithm agreement is not an independent estimate against an unbiased expert reader. The paper's assertion that 'model output was not inspected for generating the annotations nor selecting the ROIs' applies only to annotation generation and ROI selection, not to the canonical-reader choice.

full rationale

The central claim of the paper is the release of an annotated NSCLC dataset, and that claim is not circular: H&E and PD-L1 annotations were produced by human experts (with the AI-assisted merging of nuclei detections and polygons disclosed for the PD-L1 training set), and the technical validation used held-out test sets whose annotations were made manually without AI assistance for generation or ROI selection. The only load-bearing circular element is the proposal of a single canonical annotator as the reference standard for future benchmarks: that reader is selected using the baseline algorithm's F1 score, so the released pdl1_annotations.json is partly model-dependent. This is a localized benchmark-design issue rather than a circularity in the dataset construction itself, and the all-reader annotation files remain available for model-independent comparison, so the overall circularity is partial.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The dataset introduces no new theoretical entities or fitted parameters. The central contribution is empirical: a new annotated dataset. The main assumptions are domain assumptions about the validity of the annotation categories and staining guidance.

assumptions (3)
  • domain assumption The 16 H&E tissue classes form a complete and biologically meaningful partition of NSCLC morphology.
    The classification excludes an 'other' class and relies on the defined classes covering the relevant tissue types. Invoked in the description of the H&E annotation classes.
  • domain assumption CD68 immunohistochemistry on serial sections is a valid reference for identifying macrophages in H&E slides.
    Used in annotation process to guide macrophage annotations, as described in the Methods section.
  • domain assumption PD-L1 positivity defined as partial or complete circumferential membranous staining above background level is a consistent and reproducible criterion.
    Definition used for all annotators and centers, as described in the PD-L1 annotation section.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A tissue and cell-level annotated H&E and PD-L1 histopathology image dataset in non-small cell lung cancer." pith.science (2026). https://pith.science/paper/KQYNBZEB

@misc{pith2026250716855,
  author       = {Pith},
  title        = {Pith review of: A tissue and cell-level annotated H&E and PD-L1 histopathology image dataset in non-small cell lung cancer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KQYNBZEB}},
  note         = {Machine review of arXiv:2507.16855}
}
read the original abstract

The tumor immune microenvironment (TIME) in non-small cell lung cancer (NSCLC) histopathology contains morphological and molecular characteristics predictive of immunotherapy response. Computational quantification of TIME characteristics, such as cell detection and tissue segmentation, can support biomarker development. However, currently available digital pathology datasets of NSCLC for the development of cell detection or tissue segmentation algorithms are limited in scope, lack annotations of clinically prevalent metastatic sites, and forgo molecular information such as PD-L1 immunohistochemistry (IHC). To fill this gap, we introduce the IGNITE data toolkit, a multi-stain, multi-centric, and multi-scanner dataset of annotated NSCLC whole-slide images. We publicly release 887 fully annotated regions of interest from 155 unique patients across three complementary tasks: (i) multi-class semantic segmentation of tissue compartments in H&E-stained slides, with 16 classes spanning primary and metastatic NSCLC, (ii) nuclei detection, and (iii) PD-L1 positive tumor cell detection in PD-L1 IHC slides. To the best of our knowledge, this is the first public NSCLC dataset with manual annotations of H&E in metastatic sites and PD-L1 IHC.

Figures

Figures reproduced from arXiv: 2507.16855 by the authors.

Figure 2
Figure 2. Overview of the three datasets in the IGNITE data toolkit. In (A), we report the total amount of annotations per class, expressed in square millimeters for the H&E tissue segmentation dataset and in number of annotated cells for the nuclei/PD-L1+ tumor cell detection datasets. Some regions of interest (ROIs) in the nuclei/PD-L1+ tumor cell detection dataset were annotated by multiple readers; in such cases, we add t… view at source ↗
Figure 3
Figure 3. An overview of the technical validation of our datasets. For each dataset, we trained fully-supervised models on top of the data and then evaluated their performance on hold-out test sets. In (A), we show statistics regarding the train/validation/test split for the baseline models trained on each of the three datasets. In (B), we show the pairwise F1 scores of readers and the predictions of the algorithms for both t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 31 canonical work pages

  1. [1]

    & Hellmann, M

    Reck, M., Remon, J. & Hellmann, M. D. First-line immunotherapy for non–small-cell lung cancer. J. Clin. Oncol. 40, 586–597, https://doi.org/10.1200/JCO.21.01497 (2022)

  2. [2]

    Reck, M. et al. Five-year outcomes with pembrolizumab versus chemotherapy for metastatic non–small-cell lung cancer with pd-l1 tumor proportion score≥ 50%. J. Clin. Oncol. 39, 2339–2349, https://doi.org/10.1200/JCO.21.00174 (2021)

  3. [3]

    Sezer, A. et al. Cemiplimab monotherapy for first-line treatment of advanced non-small-cell lung cancer with pd-l1 of at least 50%: a multicentre, open-label, global, phase 3, randomised, controlled trial. The Lancet 397, 592–604, https://doi.org/10.1016/s0140-6736(21)00228-2 (2021)

  4. [4]

    Herbst, R. et al. Fp13. 03 impower110: updated os analysis of atezolizumab vs platinum-based chemotherapy as first-line treatment in pd-l1–selected nsclc. J. Thorac. Oncol. 16, S224–S225, https://doi.org/10.1016/j.jtho.2021.01.142 (2021)

  5. [5]

    Meng, Y .et al. Efficacy and safety of perioperative, neoadjuvant, or adjuvant immunotherapy alone or in combination with chemotherapy in early-stage non-small cell lung cancer: a systematic review and meta-analysis of randomized clinical trials. Ther. Adv. Med. Oncol. 16, 17588359241284929 (2024)

  6. [6]

    F., Wang, Y ., Liu, B

    Yang, F., Wang, J. F., Wang, Y ., Liu, B. & Molina, J. R. Comparative analysis of predictive biomarkers for pd-1/pd-l1 inhibitors in cancers: developments and challenges. Cancers 14, 109, https://doi.org/10.3390/cancers14010109 (2021)

  7. [7]

    Binnewies, M. et al. Understanding the tumor immune microenvironment (time) for effective therapy. Nat. medicine 24, 541–550 (2018)

  8. [8]

    & Zhou, Q

    Wang, F., Yang, M., Luo, W. & Zhou, Q. Characteristics of tumor microenvironment and novel immunotherapeutic strategies for non-small cell lung cancer. J. Natl. Cancer Cent. 2, 243–262 (2022)

Show all 37 references
  1. [9]

    Hendry, S. et al. Assessing tumor-infiltrating lymphocytes in solid tumors: A practical review for pathologists and proposal for a standardized method from the international immuno-oncology biomarkers working group: Part 2: TILs in melanoma, gastrointestinal tract carcinomas, ...

  2. [10]

    Kos, Z. et al. Pitfalls in assessing stromal tumor infiltrating lymphocytes (stils) in breast cancer. NPJ breast cancer 6, 17 (2020)

  3. [11]

    & Yanagita, M

    Sato, Y ., Silina, K., van den Broek, M., Hirahara, K. & Yanagita, M. The roles of tertiary lymphoid structures in chronic diseases. Nat. Rev. Nephrol. 19, 525–537 (2023)

  4. [12]

    Hendry, S. et al. Assessing tumor-infiltrating lymphocytes in solid tumors: A practical review for pathologists and proposal for a standardized method from the international immunooncology biomarkers working group: Part 1: Assessing the host immune response, TILs in invasive b...

  5. [13]

    Niazi, M. K. K., Parwani, A. V . & Gurcan, M. N. Digital pathology and artificial intelligence.The lancet oncology 20, e253–e261 (2019)

  6. [14]

    Backman, M. et al. Spatial immunophenotyping of the tumour microenvironment in non–small cell lung cancer. Eur. J. Cancer 185, 40–52 (2023)

  7. [15]

    Park, S. et al. Artificial intelligence–powered spatial analysis of tumor-infiltrating lymphocytes as complementary biomarker for immune checkpoint inhibition in non–small-cell lung cancer. J. Clin. Oncol. 40, 1916–1928 (2022)

  8. [16]

    Spronck, J. et al. 14p deep learning-based quantification of immune infiltrate for predicting response to pembrolizumab from pre-treatment biopsies of metastatic non-small cell lung cancer: A study on the pembro-rt phase ii trial. Immuno-Oncology Technol. 16, 100119, https://d...

  9. [17]

    Kludt, C. et al. Next-generation lung cancer pathology: Development and validation of diagnostic and prognostic algorithms. Cell Reports Medicine 5, https://doi.org/10.1016/j.xcrm.2024.101697 (2024). 11/12

  10. [18]

    van Rijthoven, M. et al. Multi-resolution deep learning characterizes tertiary lymphoid structures and their prognostic relevance in solid tumors. Commun. Medicine 4, 5 (2024)

  11. [19]

    Li, Z. et al. Deep learning methods for lung cancer segmentation in whole-slide histopathology images—the acdc@ lunghp challenge 2019. IEEE J. Biomed. Heal. Informatics 25, 429–440, https://doi.org/10.1109/JBHI.2020.3039741 (2020)

  12. [20]

    Verma, R. et al. Monusac2020: A multi-organ nuclei segmentation and classification challenge. IEEE Transactions on Med. Imaging 40, 3413–3423, https://doi.org/10.1109/TMI.2021.3085712 (2021)

  13. [21]

    Deep learning-based tumor microenvironment segmentation is predictive of tumor mutations and patient survival in non-small-cell lung cancer

    R ˛ aczkowska, A.et al. Deep learning-based tumor microenvironment segmentation is predictive of tumor mutations and patient survival in non-small-cell lung cancer. BMC cancer 22, 1001 (2022)

  14. [22]

    Han, C. et al. Wsss4luad: Grand challenge on weakly-supervised tissue semantic segmentation for lung adenocarcinoma. arXiv preprint arXiv:2204.06455 (2022)

  15. [23]

    Komura, D. et al. Restaining-based annotation for cancer histology segmentation to overcome annotation-related limitations among pathologists. Patterns 4 (2023)

  16. [24]

    Aubreville, M. et al. Mitosis domain generalization in histopathology images—the midog challenge. Med. Image Analysis 84, 102699 (2023)

  17. [25]

    Riihimäki, M. et al. Metastatic sites and survival in lung cancer. Lung cancer 86, 78–84 (2014)

  18. [26]

    Specific organ metastases and survival in metastatic non-small-cell lung cancer

    Tamura, T.et al. Specific organ metastases and survival in metastatic non-small-cell lung cancer. Mol. clinical oncology 3, 217–221 (2015)

  19. [27]

    Spronck, J. et al. nnunet meets pathology: bridging the gap for application to whole-slide images and computational biomarkers. In Medical Imaging with Deep Learning (2023)

  20. [28]

    van Eekelen, L. et al. Comparing deep learning and pathologist quantification of cell-level pd-l1 expression in non-small cell lung cancer whole-slide images. Sci. Reports 14, 7136 (2024)

  21. [29]

    Salgado, R. et al. The evaluation of tumor-infiltrating lymphocytes (TILs) in breast cancer: recommendations by an international TILs working group 2014. Ann. Oncol. 26, 259–271 (2015)

  22. [30]

    S., Khoshbin, A

    Sedighzadeh, S. S., Khoshbin, A. P., Razi, S., Keshavarz-Fathi, M. & Rezaei, N. A narrative review of tumor-associated macrophages in lung cancer: regulation of macrophage polarization and therapeutic implications. Transl. Lung Cancer Res. 10, 1889–1916 (2021)

  23. [31]

    Litjens, G. et al. 1399 h&e-stained sentinel lymph node sections of breast cancer patients: the camelyon dataset. GigaScience 7, giy065 (2018)

  24. [32]

    van Rijthoven, M. et al. Tumor-infiltrating lymphocytes in breast cancer through artificial intelligence: biomarker analysis from the results of the tiger challenge. medRxiv 2025–02 (2025)

  25. [33]

    Automate slide analysis platform (asap) (2017)

    Litjens, G. Automate slide analysis platform (asap) (2017)

  26. [34]

    F., Kohl, S

    Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J. & Maier-Hein, K. H. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. methods 18, 203–211 (2021)

  27. [35]

    Jocher, G. et al. ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation, 10.5281/zenodo.7347926 (2022)

  28. [36]

    Spronck, J. et al. Ignite data toolkit: a tissue and cell-level annotated h&e and pd-l1 histopathology image dataset in non-small cell lung cancer, 10.5281/zenodo.15674785 (2025)

  29. [37]

    Microsoft coco: Common objects in context

    Lin, T.-Y .et al. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, 740–755 (Springer, 2014). 12/12

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.