Pith. sign in

REVIEW 3 major objections 6 minor 27 references

A Novel Method to Evaluate Models on Unreliable, Noisy and Inconsistent Labels: Adaptive Resolution Label Aggregation (ARLA)

T0 review · 3 major / 6 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ARLA re-aggregates labels and predictions at coarser resolution so evaluation metrics track true model error instead of annotation noise.

desk verdict Simple, useful evaluation trick for noisy segmentation labels; the flood demos are persuasive, but the claim that it cleanly separates label noise from model error is only weakly supported by hand-tuned examples. read the letter →

arxiv 2607.11214 v1 pith:3OD5OCYP submitted 2026-07-13 cs.CV

Natasha Randall , Gernot Heisenberg This is my paper · ORCID
classification cs.CV
keywords labelnoisesegmentationevaluationweaklabelsadaptiveresolutionfloodmappingremotesensingmodelperformancemetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Segmentation labels used for testing are often inconsistent, noisy, or wrong at class borders, and that noise systematically distorts accuracy numbers even when the model itself is fine. The paper claims there has been almost no method for evaluating segmentation models under such labels, only methods for training on them. It introduces Adaptive Resolution Label Aggregation (ARLA): at inference time both the label and the prediction are partitioned into a grid of sub-patches, each sub-patch is reassigned a single class by a sensitivity threshold (or majority vote), and metrics are then computed on the coarser pair. On real flood-mapping labels that omit water under cloud and inconsistently mark flooded forest, ARLA raises precision from 0.52 to 0.80 while still revealing genuine under-prediction, outperforming buffer and soft-label baselines. The free parameters let a user match the aggregation scale to expected label error size so that label noise is minimised without erasing model mistakes that matter.

What carries the argument

Adaptive Resolution Label Aggregation (ARLA): partition both label and prediction into an s imes s grid of sub-patches; set each sub-patch to the positive class if the fraction of positive pixels meets sensitivity au (or take majority class for multi-class); compute ordinary metrics on the resulting coarser maps.

What would settle it

On a segmentation benchmark whose test labels have been deliberately corrupted at known spatial scales, measure whether ARLA recovers the same ranking and absolute metrics that the clean labels would have produced, while buffer and soft-label baselines do not.

Watch

Extended reading notes

Core claim

The central claim is that jointly coarsening the label and the model prediction with the same spatial partition and sensitivity threshold extracts a clearer signal of true model performance: label error is minimised while genuine model error is retained, so evaluation metrics become trustworthy even when the original ground-truth is unreliable.

Load-bearing premise

That a user can choose the grid size and sensitivity so they match the unknown spatial scale of the label noise without also hiding genuine model errors of interest.

Editorial extensions

If this is right

  • Practitioners can report model performance across a range of aggregation resolutions rather than a single misleading full-resolution score.
  • Flood and remote-sensing models can be ranked by whether they correctly detect inundation under cloud or forest, not by how closely they copy annotator omissions.
  • The same aggregation can be applied to multi-class maps by majority vote or class-priority thresholds without retraining.
  • Existing models need not be retrained; ARLA is applied only at evaluation time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the method generalises beyond remote sensing, it could serve as a standard secondary reporting layer for any benchmark whose test labels are known to contain border or occlusion noise.
  • A natural next experiment is an automatic rule that sets s from measured inter-annotator disagreement maps rather than manual calibration.
  • ARLA’s coarsening may also act as a diagnostic: regions whose metrics change sharply with s are precisely the regions of highest label uncertainty.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Adaptive Resolution Label Aggregation (ARLA), an inference-time procedure for evaluating segmentation models when ground-truth labels are unreliable, noisy, or inconsistent. Both the label and the model prediction are partitioned into an s×s grid of subpatches; each subpatch is binarised (or majority-voted for multi-class) according to a sensitivity threshold τ, and standard metrics are then computed on the aggregated maps. The method is motivated by remote-sensing flood labels (inconsistent forest inclusion, cloud-obscured regions, ambiguous borders) and is illustrated on synthetic exemplars, two hand-fixed “reliable” labels versus buffer/soft-label baselines (Table 1), and one real Copernicus flood event where precision rises from 0.52 to 0.80 under coarse aggregation. Parameters s and τ are presented as adjustable to label precision or noise level rather than as performance-maximising hyperparameters. Code and figures are released.

Significance. If the central claim holds—that ARLA systematically reduces the contribution of label error while preserving genuine model error—the work fills a genuine and under-served gap: most noisy-label literature addresses training, not evaluation, and the paper correctly notes that even small test-label errors can distort model ranking. The method is simple, model-agnostic, requires no re-training, and is immediately usable in remote sensing and other mapping tasks where perfect labels are often unobtainable. Public code and reproducible figures are a clear strength. Significance is currently limited by thin quantitative support (hand-tuned parameters, post-hoc reliable labels, a single real event, no controlled noise-injection study), so the practical impact depends on whether stronger validation can be supplied.

major comments (3)
  1. Abstract and §4–§5 claim that ARLA “minimises the label error” while still “retaining true model error.” This separation is not demonstrated under the conditions the paper itself identifies as realistic. In §5.2 / Table 1 the “reliable” labels are constructed by the authors after the fact, and both ARLA’s (s, τ) and the buffer width are calibrated to the known spatial size of the forest/cloud errors. In §5.3 / Fig. 7 the 1×1 / 2.5% setting that lifts precision from 0.52 to 0.80 is likewise chosen after inspecting the cloud artefacts. Without a pre-specified selection rule or an experiment that injects label noise of known scale into a trusted ground truth and checks recovery of a known model error, the reported gains are consistent with deliberate suppression of disagreements rather than a validated separation of label noise from model error.
  2. §6 explicitly states that estimating label-error size is difficult without the original annotators’ knowledge, yet every quantitative result in the paper uses parameters matched to that size. The free parameters s and τ are therefore load-bearing for the central claim, but no a-priori rule, cross-validation protocol, or sensitivity analysis that does not peek at the target errors is provided. A controlled synthetic-noise study (varying spatial scale and density of injected label errors, measuring residual disagreement against a known true model error) is needed before the abstract claim can be treated as established rather than plausible.
  3. Evaluation scope is too narrow to support a general method paper. Quantitative evidence consists of two hand-crafted binary examples (Table 1) and one flood event (§5.3); multi-class is only visual (Fig. 3, Cityscapes); there are no public segmentation benchmarks, no error bars, no multiple models, and no comparison against label-cleaning or multi-annotator baselines beyond the two border-focused methods. Extending the experiments—at least with controlled noise on a standard dataset and one additional application domain—is necessary for the contribution to be proportionate to the claim of a “novel method to evaluate models on unreliable labels.”
minor comments (6)
  1. Title and running headers contain missing spaces (“A NOVELMETHOD”, “UNRELIABLE, NOISY ANDINCONSISTENTLABELS”); clean for camera-ready.
  2. Figure order in the text is awkward: Fig. 6 (F1 grid) is discussed before Fig. 7 (the flood event it summarises). Renumber or reorder for narrative flow.
  3. The formal definition of V(Sa,b,fL) in §4 is correct but dense; a short algorithmic box or pseudocode would help practitioners reimplement without ambiguity about edge handling and non-divisible image sizes.
  4. §2 asserts that “to the best of our knowledge, no works currently exist on evaluating segmentation models on unreliable labels.” Soft metrics, boundary buffers, and uncertain-GT medical metrics (some already cited) are related; a slightly more precise novelty statement would avoid over-claim.
  5. Table 1 reports only precision; including IoU/Dice/F1 for the same three methods would make the baseline comparison more informative.
  6. Data licensing and GitHub link are welcome; ensure the repository contains the exact parameter settings used for Table 1 and Fig. 7 so the numbers are fully reproducible.

Circularity Check

2 steps flagged · score 3.0 of 10

Mild circularity only in empirical validation: s and τ are hand-calibrated to the very label-error scales the authors wish to suppress, so the reported recovery of 'true' model performance is partly forced by that choice rather than independently demonstrated.

  1. fitted input called prediction [§5.2 (Table 1 and preceding paragraph)]
    "The parameters for all of the methods are calibrated to the size of the label error (here, the size of the missing forest and cloud cover areas). ... ARLA performs better than the two baseline methods on both of the examples."

    s and τ (and the buffer/soft-label widths) are set by direct inspection of the spatial extent of the artefacts the authors already know they want to suppress. The subsequent claim that ARLA recovers a precision closer to the authors' own post-hoc 'reliable' labels is therefore statistically forced by that calibration rather than an independent test that the residual disagreement equals true model error.

  2. fitted input called prediction [§5.3 (Fig. 7c and surrounding text)]
    "In figure 7c, ARLA has been applied ... using a particularly coarse partitioning resolution with parameters of 1×1 subpatches and sensitivity of 2.5%. ... the model precision subsequently increases from 0.52 to 0.80."

    The coarsest possible aggregation is selected after the cloud-cover holes and noisy auto-labels have already been identified in the same figure. The jump in precision and the reinterpretation of model behaviour (under- vs over-prediction) therefore depend on parameters tuned to erase exactly those artefacts, rendering the 'clearer signal of true performance' circular with the choice of scale.

full rationale

ARLA is a procedural evaluation method, not a first-principles derivation of a physical or statistical quantity. There is no self-definitional equation, no uniqueness theorem imported from the authors, no self-citation that carries the central claim, and no renaming of a known result. The aggregation operator itself is self-contained and transparent. The only circularity risk is the classic 'fitted input called prediction' pattern that appears in the two quantitative examples: parameters are chosen after inspecting the spatial scale of the known artefacts (forest holes, cloud cover), the 'reliable' labels used as ground truth in Table 1 are themselves post-hoc constructions by the authors, and the resulting metric improvements are then presented as evidence that ARLA retains true model error. The paper itself warns against treating the parameters as optimisable hyper-parameters, which keeps the circularity mild and non-load-bearing for the method definition. No stronger circularity patterns are present; an honest non-finding of deep circularity is therefore appropriate, with the residual score reflecting only the post-hoc calibration of the illustrative results.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

ARLA is a lightweight post-processing method. Its load-bearing choices are two free parameters chosen by the user and a small set of domain assumptions about spatial structure of labels; no new physical entities are postulated.

free parameters (2)
  • s (number of subpatches / aggregation resolution)
    Chosen by hand to match expected spatial scale of label error; different values produce different reported metrics (Figs. 2, 6).
  • τ (sensitivity threshold, percent positive pixels)
    Chosen by hand to control how aggressively sparse positive labels are filled or noisy positives are suppressed; again user-selected rather than derived.
assumptions (3)
  • domain assumption Label noise and true model error occupy different characteristic spatial scales, so a single aggregation resolution can suppress the former while retaining the latter.
    Stated as the design goal in §4 and §6; never proved, only illustrated on the flood examples.
  • domain assumption Both the ground-truth label and the model prediction share the same spatial domain and can be partitioned identically.
    Required by the formal definition of V(Sa,b,f) in §4.
  • standard math Standard binary or multi-class segmentation metrics remain meaningful after hard aggregation of both maps.
    Implicit throughout §5; no new metric theory is derived.
invented entities (1)
  • ARLA aggregation operator V(Sa,b,f)
    purpose: Maps an arbitrary-resolution binary (or multi-class) mask to a coarser binary decision grid controlled by s and τ.
    The operator is defined in §4; it is a procedural invention, not a physical entity, and has no independent existence outside the evaluation pipeline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Novel Method to Evaluate Models on Unreliable, Noisy and Inconsistent Labels: Adaptive Resolution Label Aggregation (ARLA)." pith.science (2026). https://pith.science/paper/3OD5OCYP

@misc{pith2026260711214,
  author       = {Pith},
  title        = {Pith review of: A Novel Method to Evaluate Models on Unreliable, Noisy and Inconsistent Labels: Adaptive Resolution Label Aggregation (ARLA)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3OD5OCYP}},
  note         = {Machine review of arXiv:2607.11214}
}
read the original abstract

Labels are critical for both training and evaluating deep learning segmentation models, but are often inconsistent, noisy, or ambiguous at class boundaries. Many approaches have been developed to support training models on weak labels, but few to none currently exist to facilitate evaluating models on unreliable labels. We therefore introduce a method called "Adaptive Resolution Label Aggregation", or "ARLA", which dynamically adapts the resolution of both the label and the model prediction at inference time before the evaluation metrics are computed. We demonstrate how ARLA can be used to better analyse model behaviour with a practical application to a real flood prediction model, where ARLA was able to overcome issues with inconsistent labelling of forested areas and errors in labels within regions of heavy cloud cover. Our work presents a new approach to evaluating segmentation models, with adjustable parameters to adapt the aggregated resolution to the precision of the label or the level of label noise. Fundamentally, ARLA exploits the information encapsulated by a label but minimises the label error, extracting from the noise a clearer signal of a model's true performance.

Figures

Figures reproduced from arXiv: 2607.11214 by the authors.

Figure 1
Figure 1. Examples of unreliable, inconsistent, ambiguous and noisy labels in a real flood mapping dataset. The blue [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The ARLA method is applied to an example label that depicts areas of flooding (blue: 1) and non-flooding [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The ARLA method applied to an example multi-class label from the Cityscapes dataset [ [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Two examples that compare a label, model prediction, and prediction error, both before and after ARLA has [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The ‘Buffer’, ‘Soft Label’, and ARLA methods are applied to two examples of unreliable labels, where blue [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: F1 scores for the flood prediction model under a variety of combinations of ARLA sensitivity and subpatches [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Unreliable labels of a real flood event (a). The performance of a flood prediction model is compared before [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 14 canonical work pages

  1. [1]

    Guidelines to compare semantic segmentation maps at different resolutions.IEEE Transactions on Geoscience and Remote Sensing, 62:1–16, 2024

    Christian Ayala, Carlos Aranda, and Mikel Galar. Guidelines to compare semantic segmentation maps at different resolutions.IEEE Transactions on Geoscience and Remote Sensing, 62:1–16, 2024. doi: 10.1109/TGRS.2024.3369310

    work page doi:10.1109/tgrs.2024.3369310 2024

  2. [2]

    Active label cleaning for improved dataset quality under resource constraints.Nature communications, 13(1):1161, 2022

    Mélanie Bernhardt, Daniel C Castro, Ryutaro Tanno, Anton Schwaighofer, Kerem C Tezcan, Miguel Monteiro, Shruthi Bannur, Matthew P Lungren, Aditya Nori, Ben Glocker, et al. Active label cleaning for improved dataset quality under resource constraints.Nature communications, 13(1):1161, 2022. doi: 10.1038/s41467-022-28818-3

    doi:10.1038/s41467-022-28818-3 2022

  3. [3]

    Boundary iou: Improving object-centric image segmentation evaluation

    Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15334–15342, 2021. doi: 10.1109/cvpr46437.2021.01508

    doi:10.1109/cvpr46437.2021.01508 2021

  4. [4]

    Weighted intersection over union (wiou) for evaluating image segmentation.Pattern Recognition Letters, 185:101–107, 2024

    Yeong-Jun Cho. Weighted intersection over union (wiou) for evaluating image segmentation.Pattern Recognition Letters, 185:101–107, 2024. doi: 10.1016/j.patrec.2024.07.011

    work page doi:10.1016/j.patrec.2024.07.011 2024

  5. [5]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InProc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. doi: 10.1109/CVPR.2016.350

    doi:10.1109/cvpr.2016.350 2016

  6. [6]

    Copernicus emergency management service on-demand mapping, 2026

    Directorate Space, Security and Migration, European Commission Joint Research Centre (EC JRC). Copernicus emergency management service on-demand mapping, 2026. URLhttps://mapping.emergency.copernicus. eu/

    2026

  7. [7]

    Data labeling: An empirical investigation into industrial challenges and mitigation strategies

    Teodor Fredriksson, David Issa Mattos, Jan Bosch, and Helena Holmström Olsson. Data labeling: An empirical investigation into industrial challenges and mitigation strategies. InInternational conference on product-focused software process improvement, pages 202–216. Springer, 2020. doi: 10.1007/978-3-030-64148-1_13

    work page doi:10.1007/978-3-030-64148-1_13 2020

  8. [8]

    Learning and evaluation in presence of non-iid label noise

    Nico Görnitz, Anne Porbadnigk, Alexander Binder, Claudia Sannelli, Mikio Braun, Klaus-Robert Müller, and Marius Kloft. Learning and evaluation in presence of non-iid label noise. InArtificial Intelligence and Statistics, pages 293–302. PMLR, 2014. URLhttps://proceedings.mlr.press/v33/gornitz14.html

    2014

Show all 27 references
  1. [9]

    Evaluating classification systems against soft labels with fuzzy precision and recall.arXiv preprint arXiv:2309.13938, 2023

    Manu Harju and Annamaria Mesaros. Evaluating classification systems against soft labels with fuzzy precision and recall.arXiv preprint arXiv:2309.13938, 2023. doi: 10.48550/arXiv.2309.13938. 10 Adaptive Resolution Label Aggregation (ARLA)PREPRINT

    2023 doi arXiv

  2. [10]

    Perfect labelling: A review and outlook of label optimization techniques in dynamic earth observation.Remote Sensing, 17(7):1246, 2025

    Sarah Hauser, Lena Augner, and Andreas Schmitt. Perfect labelling: A review and outlook of label optimization techniques in dynamic earth observation.Remote Sensing, 17(7):1246, 2025. doi: 10.3390/rs17071246

    2025 doi

  3. [11]

    Soft labels for training and evaluating semantic segmentation models

    Nima Jamali. Soft labels for training and evaluating semantic segmentation models. Master of mathematics thesis, University of Waterloo, 2026. URLhttps://hdl.handle.net/10012/23010

    2026

  4. [12]

    Deep learning with noisy labels: Ex- ploring techniques and remedies in medical image analysis.Medical image analysis, 65:101759, 2020

    Davood Karimi, Haoran Dou, Simon K Warfield, and Ali Gholipour. Deep learning with noisy labels: Ex- ploring techniques and remedies in medical image analysis.Medical image analysis, 65:101759, 2020. doi: 10.1016/j.media.2020.101759

    2020 doi

  5. [13]

    Noisy annotations in semantic segmentation.arXiv preprint arXiv:2406.10891, 2024

    Moshe Kimhi, Omer Kerem, Eden Grad, Ehud Rivlin, and Chaim Baskin. Noisy annotations in semantic segmentation.arXiv preprint arXiv:2406.10891, 2024. doi: 10.48550/arXiv.2406.10891

    2024 doi arXiv

  6. [14]

    Evaluating segmentation error without ground truth

    Timo Kohlberger, Vivek Singh, Chris Alvino, Claus Bahlmann, and Leo Grady. Evaluating segmentation error without ground truth. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 528–536. Springer, 2012. doi: 10.1007/978-3-642-33415-3_65

    2012 doi

  7. [15]

    Evaluating classifiers by means of test data with noisy labels

    Chuck P Lam and David G Stork. Evaluating classifiers by means of test data with noisy labels. InIJCAI, volume 3, pages 513–518, 2003. URLhttps://www.ijcai.org/Proceedings/03/Papers/076.pdf

    2003

  8. [16]

    Computing precision and recall with missing or uncertain ground truth

    Bart Lamiroy and Tao Sun. Computing precision and recall with missing or uncertain ground truth. InInternational Workshop on Graphics Recognition, pages 149–162. Springer, 2011. doi: 10.1007/978-3-642-36824-0_15

    2011 doi

  9. [17]

    Marcus Nordström, Henrik Hult, Fredrik Löfman, and Jonas Söderberg. On image segmentation with noisy labels: Characterization and volume properties of the optimal solutions to accuracy and dice.Advances in Neural Information Processing Systems, 35:34321–34333, 2022. doi: 10.52...

    2022 doi

  10. [18]

    Confident learning: Estimating uncertainty in dataset labels.Journal of Artificial Intelligence Research, 70:1373–1411, 2021

    Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels.Journal of Artificial Intelligence Research, 70:1373–1411, 2021. doi: 10.1613/jair.1.12125

    2021 doi

  11. [19]

    Pervasive label errors in test sets destabilize machine learning benchmarks.arXiv preprint arXiv:2103.14749, 2021

    Curtis G Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks.arXiv preprint arXiv:2103.14749, 2021. doi: 10.48550/arXiv.2103.14749

    2021 doi arXiv

  12. [20]

    Sophie Ostmeier, Brian Axelrod, Fabian Isensee, Jeroen Bertels, Michael Mlynash, Soren Christensen, Maarten G Lansberg, Gregory W Albers, Rajen Sheth, Benjamin FJ Verhaaren, et al. Use-evaluator: Performance metrics for medical image segmentation models supervised by uncertain...

    2023 doi

  13. [21]

    Data programming: Creating large training sets, quickly.Advances in neural information processing systems, 29, 2016

    Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly.Advances in neural information processing systems, 29, 2016. doi: 10.5555/3157382.3157497

    2016 doi

  14. [22]

    Deep learning is robust to massive label noise

    David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint arXiv:1705.10694, 2017. doi: 10.48550/arXiv.1705.10694

    2017 doi arXiv

  15. [23]

    Learning from noisy labels with deep neural networks: A survey.IEEE transactions on neural networks and learning systems, 34(11):8135–8153, 2022

    Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey.IEEE transactions on neural networks and learning systems, 34(11):8135–8153, 2022. doi: 10.1109/TNNLS.2022.3152527

    2022 doi

  16. [24]

    Evaluating medical ai systems in dermatology under uncertain ground truth.Medical Image Analysis, 103:103556, 2025

    David Stutz, Ali Taylan Cemgil, Abhijit Guha Roy, Tatiana Matejovicova, Melih Barsbey, Patricia Strachan, Mike Schaekermann, Jan Freyberg, Rajeev Rikhye, Beverly Freeman, et al. Evaluating medical ai systems in dermatology under uncertain ground truth.Medical Image Analysis, 1...

    2025 doi

  17. [25]

    Are ground truth labels reproducible? an empirical study

    Ka Wong, Praveen Paritosh, and Kurt Bollacker. Are ground truth labels reproducible? an empirical study. In Proceedings of ML Evaluation Standards Workshop at ICLR, pages 25–29, 2022. URL https://ml-eval. github.io/assets/pdf/GroundTruthReproducibilityICLRSubmitted.pdf

    2022

  18. [26]

    Assessing inter-annotator agreement for medical image segmentation.IEEe Access, 11:21300–21312, 2023

    Feng Yang, Ghada Zamzmi, Sandeep Angara, Sivaramakrishnan Rajaraman, André Aquilina, Zhiyun Xue, Stefan Jaeger, Emmanouil Papagiannakis, and Sameer K Antani. Assessing inter-annotator agreement for medical image segmentation.IEEe Access, 11:21300–21312, 2023. doi: 10.1109/ACCE...

    2023 doi

  19. [27]

    Uanet: An uncertainty-aware network for feature calibration and boundary refinement in medical image segmentation.Biomedical Signal Processing and Control, 112:108413, 2026

    Qian Zhang, Junxia Wang, Wei Wu, and Yuanjie Zheng. Uanet: An uncertainty-aware network for feature calibration and boundary refinement in medical image segmentation.Biomedical Signal Processing and Control, 112:108413, 2026. doi: 10.1016/j.bspc.2025.108413. 11

    2026 doi

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.