Pith. sign in

REVIEW 7 cited by

Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.12088 v2 pith:JPYOWAKM submitted 2021-10-22 cs.LG stat.ML

Jiaheng Wei , Zhaowei Zhu , Hao Cheng , Tongliang Liu , Gang Niu , Yang Liu This is my paper
classification cs.LGstat.ML
keywords noisenoisyreal-worlddatasetslabellabelssyntheticefforts
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Existing research on learning with noisy labels mainly focuses on synthetic label noise. Synthetic noise, though has clean structures which greatly enabled statistical analyses, often fails to model real-world noise patterns. The recent literature has observed several efforts to offer real-world noisy datasets, yet the existing efforts suffer from two caveats: (1) The lack of ground-truth verification makes it hard to theoretically study the property and treatment of real-world label noise; (2) These efforts are often of large scales, which may result in unfair comparisons of robust methods within reasonable and accessible computation power. To better understand real-world label noise, it is crucial to build controllable and moderate-sized real-world noisy datasets with both ground-truth and noisy labels. This work presents two new benchmark datasets CIFAR-10N, CIFAR-100N, equipping the training datasets of CIFAR-10, CIFAR-100 with human-annotated real-world noisy labels we collected from Amazon Mechanical Turk. We quantitatively and qualitatively show that real-world noisy labels follow an instance-dependent pattern rather than the classically assumed and adopted ones (e.g., class-dependent label noise). We then initiate an effort to benchmarking a subset of the existing solutions using CIFAR-10N and CIFAR-100N. We further proceed to study the memorization of correct and wrong predictions, which further illustrates the difference between human noise and class-dependent synthetic noise. We show indeed the real-world noise patterns impose new and outstanding challenges as compared to synthetic label noise. These observations require us to rethink the treatment of noisy labels, and we hope the availability of these two datasets would facilitate the development and evaluation of future learning with noisy label solutions. Datasets and leaderboards are available at http://noisylabels.com.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Delving into Instance-Dependent Label Noise in Graph Data: A Comprehensive Study and Benchmark

    cs.LG 2025-06 conditional novelty 7.0 of 10

    BeGIN provides 10 graph datasets with six label-noise types and a broad evaluation, showing that LLM-simulated noise hurts GNNs more than uniform or pairwise noise.

  2. CLID-MU: Cross-Layer Information Divergence Based Meta Update Strategy for Learning with Noisy Labels

    cs.LG 2025-07 conditional novelty 6.0 of 10

    CLID-MU replaces the clean meta-dataset in meta-learning with an unsupervised cross-layer divergence metric, improving noisy-label and semi-supervised results on several benchmarks.

  3. Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement

    cs.CL 2025-05 conditional novelty 6.0 of 10

    SiDyP improves classifiers trained on LLM-generated noisy labels by retrieving likely true labels from embedding-space neighbors and iteratively refining them with a simplex diffusion model, reporting average gains of...

  4. Laplace Sample Information: Data Informativeness Through a Bayesian Lens

    cs.LG 2025-05 conditional novelty 6.0 of 10

    LSI ranks training samples by informativeness using the KL divergence between Laplace-approximated posteriors with and without each sample, and the ordering transfers from a small probe to larger models.

  5. Uncertainty-Driven Reliability: Selective Prediction and Trustworthy Deployment in Modern Machine Learning

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A training-dynamics abstention method matches deep ensembles at a fraction of the training cost, and a five-term error budget explains why selective classifiers still fall short of the oracle.

  6. When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification

    cs.CV 2025-05 conditional novelty 5.0 of 10

    REVEAL ensembles four VLMs and label-noise detectors to detect and correct noisy and missing labels in six image classification test sets, reporting high agreement with human annotations.

  7. Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A dynamic pruning method scores each sample by combining task loss with CLIP image-text similarity and selects samples near the median score each epoch.

Pith tools