Pith. sign in

REVIEW 4 major objections 4 minor 57 references

When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VLM ensemble renovates labels on six benchmark test sets.

desk verdict Useful framework for missing-label renovation, but the validation doesn't back the strong claims; deserves a serious referee but with heavy revision. read the letter →

arxiv 2505.16149 v1 pith:TKNOFFJZ submitted 2025-05-22 cs.CV cs.AIcs.CL

Zirui Pang , Haosheng Tan , Yuhan Pu , Zhijie Deng , Zhouan Shen , Keyu Hu , Jiaheng Wei This is my paper
classification cs.CVcs.AIcs.CL
keywords testsetrenovationmissinglabeldetectionnoisevision-languagemodelsensemblelabelingsoftlabelsbenchmarkevaluationdatasetcuration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Widely used image-classification test sets still contain two kinds of mistakes: labels that are simply wrong, and labels that are incomplete because the image shows more than one valid class while only one is recorded. The paper argues that this second kind, missing labels, is common and has been largely ignored, so benchmark comparisons based on these test sets can mislead. To address it, REVEAL asks several pre-trained vision-language models and two statistical label-auditing methods to propose labels for every image, weights each method by its estimated accuracy, and converts surviving labels into soft labels with likelihoods. On six datasets, the paper reports that these renovated labels agree with human crowd judgments more often than any single vision-language model on five of the six datasets, and it concludes that test-set renovation is a viable model-centric alternative to full manual re-annotation.

What carries the argument

The load-bearing machinery is a two-stage voting scheme. In stage one, the first 100 images of each dataset are used to build pseudo ground-truth labels by majority voting: a class is accepted into the pseudo ground truth if at least k of the m methods predicted it. Each method's estimated accuracy is then the overlap of its predictions with that pseudo ground truth, multiplied by a regularization factor that penalizes predicting too many labels. In stage two, each method's votes on the full test set are weighted by its estimated accuracy to form a support score per image and class; a threshold and a top-n constraint filter the candidates, and softmax turns the surviving scores into label likelihoods. The estimated-accuracy weighting is what makes the ensemble more than a popularity contest, and the regularization term is what keeps verbose models from dominating.

What would settle it

Take a random sample of images from the six test sets, have independent annotators list every class they can see without restricting them to the candidate labels flagged by any single tool, and compare those exhaustive annotations with REVEAL's soft labels and with the pseudo ground-truth labels from the first 100 images. If the ensemble does not agree with the exhaustive annotations better than its best individual member, or if the accuracy estimates computed from the 100-image pseudo ground truth diverge from accuracy measured on the independent annotations, the circularity in the weighting scheme is doing real damage.

Watch

Extended reading notes

Core claim

The paper's central claim is that an ensemble of vision-language models and label-diagnosis tools can detect both noisy and missing labels in existing test sets and that the ensembled soft labels align with human judgment better than the original labels or any single model's labels. The mechanism is a weighted vote: for each image, every method casts a vote for each class it predicts, and each vote is weighted by an estimate of that method's accuracy; labels whose weighted support passes a threshold and a top-n cut are kept and then softmax-normalized into likelihoods. Against crowd annotations the paper reports agreement rates of 0.976 on CIFAR-10, 0.880 on CIFAR-100, 0.946 on Caltech256, 0.776 on MNIST, 0.630 on QuickDraw, and 0.569 on ImageNet, with the ensemble beating every individual model on five of the six datasets; ImageNet is the exception, and the paper argues the crowd protocol there is too restrictive to capture missing labels. The paper also documents ten failure modes of individual VLMs, such as fine-grained class collapse, repetitive generation, associative hallucination, and refusal to answer 'None'.

Load-bearing premise

The weights that decide how much each model's vote counts are computed from pseudo ground-truth labels that come from majority voting among the same models, so the whole pipeline assumes that the consensus of those models on the first 100 images is a trustworthy stand-in for true labels.

Editorial extensions

If this is right

  • The paper counts 552 noisy or missing-label images in CIFAR-100 and 6,546 in ImageNet under its configuration; if those counts hold, benchmark rankings based on original one-hot labels are built on a non-negligible amount of incorrect supervision.
  • REVEAL returns soft labels with likelihoods, so downstream evaluation can measure whether a classifier put probability mass on all valid classes rather than only matching the one recorded label.
  • The ten documented VLM failure modes imply that any automatic label-cleaning effort built on a single VLM will inherit hallucination, repetition, and abstention failures; consensus filtering is a necessary corrective.
  • The paper's agreement results suggest that human evaluation restricted to a single guessed label per image systematically undercounts missing-label performance in large-label-space datasets such as ImageNet and QuickDraw.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If REVEAL's soft labels are adopted as ground truth, published accuracy numbers for classifiers on these benchmarks would shift, and models that correctly predicted a second object in an image would be rewarded rather than penalized; the relative ordering of models could change.
  • A clean ablation test of the circularity worry would be to recompute REVEAL's per-method accuracy weights using independent human annotations on a hold-out subset and see whether the ensemble's agreement with humans improves, stays flat, or drops.
  • The same weighted-voting-with-regularization recipe transfers to image captioning and object detection, where one ground-truth annotation is known to undercount valid content; single-reference metrics would benefit from soft, multi-reference evaluation.
  • Because the paper's agreement metric is defined only over the two labels available in the prior crowd protocol, REVEAL's true agreement on missing labels is probably understated; an open-ended annotation study would clarify this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes REVEAL, a pipeline that combines four vision-language models (BLIP, LLaVA, Janus, Qwen) with automated label-diagnosis tools (Cleanlab, Docta) to renovate the test sets of six image-classification benchmarks by detecting both noisy labels and missing (omitted) labels. The method estimates per-model accuracies on the first 100 images of each dataset, using pseudo ground-truth labels formed by majority voting among the same models, and then uses those accuracy estimates as weights in a voting ensemble. Thresholding and top-K selection produce final soft labels. The paper reports counts of detected noisy and missing labels, agreement rates against MTurk annotations taken from prior work, and ten qualitative observations about VLM labeling behavior.

Significance. The problem is relevant: benchmark test sets contain both wrong and missing labels, and the missing-label phenomenon is indeed under-studied. The paper usefully draws attention to VLM failure modes such as hallucination, repetition, and refusal to abstain, and the multi-dataset scope is a strength. However, the headline claim of 'significant improvement' and 'high alignment with human judgments' is not currently established, because the accuracy estimates used for ensembling are computed against pseudo labels generated by the same models, and the only human comparison is restricted to a Cleanlab-flagged subset with a protocol that cannot detect missing labels. With additional non-circular validation and a representative human study, the framework could be a valuable contribution; in its present form the evidence is not load-bearing enough for the abstract's claims.

major comments (4)
  1. [Section 4.1-4.3, Eq. (1) and Eq. (2)] The estimated accuracy that supplies the weights in the aggregation is computed against pseudo ground-truth labels obtained by majority voting of the same VLM set (the voting equation in Section 4.1). This creates a self-referential loop: the ensemble is weighted by accuracy scores measured on a consensus of the very models being aggregated, so REVEAL's improvement over individual VLMs can be an artifact of the estimation procedure. Please validate the estimated accuracies against held-out human labels or another independent oracle, and show that the full-set results are stable under different calibration subsets rather than only the first 100 images.
  2. [Section 4.6 / Appendix E.1] The only external human evaluation uses MTurk annotations from Northcutt et al. (2021), which, as the paper itself states, were collected only for images that Cleanlab flagged as likely mislabeled. This subset is enriched for label noise and cannot validate performance on randomly selected images or on the missing-label discovery that is the paper's central novelty. The four-outcome protocol (given, guessed, both, neither) cannot detect omitted valid labels on a general sample, and the paper's own Observation 9 dismisses the ImageNet and QuickDraw discrepancies as artifacts of the MTurk protocol. The abstract's claim of 'highly aligning to human judgments' therefore rests on a non-representative, label-noise-enriched sample. A human study on a random sample with an open-label annotation protocol is needed.
  3. [Appendix E.2, Table 3] The Agreement Rate is a membership condition that rewards broad prediction sets. For MTurk = both, agreement requires both the given and the guessed label to appear in the model's predicted set, so a method that simply predicts a superset of those labels is counted as agreeing. For MTurk = neither, a method that predicts nothing agrees. REVEAL's high agreement rates in Table 3 may therefore reflect label-set cardinality rather than label precision. Please report precision and recall against MTurk labels, or compare against a trivial baseline that always predicts the top-K labels, and control for predicted-set size.
  4. [Section 4.4, Table 2] The headline counts of noisy and missing labels are produced entirely by the pipeline with per-dataset thresholds tau, top-K limits, vote thresholds, and batch sizes, but the paper provides no sensitivity analysis for these free parameters and no principled calibration beyond a heuristic trade-off in Figure 4. Because Table 2 is one of the main advertised outputs, the paper should show how the detected counts vary with tau and top-K and justify the chosen operating points; otherwise the counts are not readily falsifiable.
minor comments (4)
  1. [Section 3.3] Observations 1 and 2 are each presented twice: once as an enumerated observation and again immediately afterward as an example block with the same observation number. Please renumber or restructure these example blocks to avoid duplicate numbering.
  2. [Appendix A.1] The heading 'Border Impacts' should be 'Broader Impacts'.
  3. [Abstract] The abstract has a subject-verb agreement error: 'datasets ... serves' should be 'datasets ... serve'.
  4. [Table 2] The table caption uses 'Threshold/Full score' without explaining the 'Full score' concept in the caption; since 'Full Score' is defined only in Table 1, please clarify the column or split it into two columns.

Circularity Check

2 steps flagged · score 6.0 of 10

REVEAL's accuracy weighting is a self-referential majority-vote loop, and its human-validation target is seeded by Cleanlab, one of the method's own inputs.

  1. self definitional [Section 4.1, Eq. (1); Section 4.2, Eq. (2); Section 4.3, Eq. (3); Table 1 caption]
    "The aggregated pseudo ground-truth label set for image x^(j) is defined as: ~y^(j)_ground_truth = { c in C | vote^(j)(c) >= k }. ... Given the pseudo ground-truth labels ~y_ground_truth, we assess the estimated accuracy of each model on this subset. ... Following the estimated accuracy, we perform prediction aggregation using the estimated accuracy scores {Acc_1,...,Acc_m} as model weights. (Table 1 caption: "Origin represents the dataset's original label estimated accuracy evaluated on pseudo ground truth.")"

    The accuracy weights Acc_i are computed by comparing each model to pseudo ground-truth labels that are themselves the majority vote of exactly the same model outputs (Eq. 1). Eq. (2) then scores each model by overlap with that vote, and Eq. (3) re-weights the same outputs by those scores. A model that agrees with the majority is certified accurate, so the ensemble is a monotone aggregate of the very votes used to define correctness. The phrase "manually verified" in the text is not reflected in the equations or the Table 1 caption, which explicitly says "evaluated on pseudo ground truth." Thus the weighted soft labels are partly self-confirming rather than externally grounded.

  2. other [Section 4.6, Appendix E.1, Appendix E.2; Table 3]
    "In [33], the MTurk results are based on the Cleanlab framework. In their study, only those instances where Cleanlab identified a disagreement with the original label were submitted to MTurk for human evaluation. ... Case 2 (MTurk = guessed): Agreement is counted if s_i in R_i. ... Case 3 (MTurk = both): Agreement is counted if both g_i in R_i and s_i in R_i."

    The human-judgment target is not an independent sample of full test-set quality: the evaluated images are exactly those flagged by Cleanlab, and the "guessed" label s_i in every agreement condition is Cleanlab's predicted label. Cleanlab is itself one of the m methods whose outputs enter REVEAL's weighted aggregation (Table 1 lists Cleanlab accuracy; Eq. (3) sums over all m models). REVEAL's agreement with "guessed" and "both" cases is therefore partly agreement with one of its own inputs. Moreover, the paper concedes in E.1 that this protocol "fails to adequately capture missing label errors," which is the paper's central novelty. The abstract's claim of "highly aligning to human judgments" is thus not validated on the full test set or for the missing-label phenomenon.

full rationale

REVEAL's central soft-label construction is only partially self-contained. The majority-vote pseudo-ground-truth of Eq. (1) is built from the same predictors whose "expertise" Eq. (2) then measures; using those accuracies as weights in Eq. (3) makes the ensemble a self-referential smoothing of its own votes. This does not make the final labels fully tautological, because thresholding and top-K selection can exclude some majority labels, but it means the reported accuracy scores and "Full Score" in Table 1 are not independent evidence of label quality. The MTurk comparison is real human data and therefore genuine evidence in principle, but the paper itself restricts it to Cleanlab-flagged images and to Cleanlab's guessed label, and Cleanlab is an input to REVEAL; the agreement metric is a membership test on that two-label target, so the "human alignment" result is partially seeded by the method's own inputs and cannot validate the full-test-set or missing-label claims. No load-bearing self-citation chain was found: references such as [44] support terminology, and [33] is external. The limitation passages in Appendix E.1 and Observation 9 honestly acknowledge the MTurk protocol's blind spot, but they undercut rather than rescue the abstract's strongest claim. Overall score 6 reflects the partial circularity in the weighting loop and the contaminated human-evaluation target, while acknowledging that the final labels still depend on non-tautological thresholds and on human MTurk judgments.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central method depends on several per-dataset free parameters (batch size, thresholds, top-K) and on the assumption that VLM consensus can define pseudo ground truth. No new physical or theoretical entities are introduced.

free parameters (5)
  • Label batch size per dataset = 10 (CIFAR-10, MNIST), 20 (CIFAR-100), 50 (Caltech256), 67 (ImageNet), 60 (QuickDraw)
    Chosen via recall/runtime tradeoff on the first 100 images (Figure 4, Table 7). This is a per-dataset tuning choice.
  • Score threshold tau = 0.900 (CIFAR-10), 0.830 (CIFAR-100), 0.037 (Caltech), 0.018 (QuickDraw), 0.300 (ImageNet), 0.950 (MNIST)
    Used in post-aggregation filtering (Section 4.4). Selection criteria are not specified; values vary by dataset and appear tuned to each benchmark.
  • Top-K max labels per image = 3 (CIFAR-10, MNIST), 5 (CIFAR-100, QuickDraw), 7 (Caltech), 10 (ImageNet)
    Maximum number of labels retained per image (Table 2). Chosen by the authors per dataset with no stated justification.
  • Vote threshold k = Not specified
    Minimum number of votes for a label to enter the pseudo ground-truth set in Eq. (1). A free parameter that affects the accuracy estimates and final weights.
  • BLIP threshold and top-alpha = e.g., (0.15,3) for CIFAR-10, (0.015,5) for CIFAR-100, (0.006,5) for Caltech, (0.00015,20) for ImageNet, (0.004,5) for…
    Per-dataset hyperparameters for BLIP ITM confidence filtering (Table 7). Chosen without a formal selection procedure.
assumptions (3)
  • domain assumption Vision-language models produce sufficiently accurate multi-label predictions to serve as a basis for test-set renovation.
    The entire pipeline relies on VLM outputs being informative about both noisy and missing labels; the paper provides observations of failure modes but no independent proof of sufficiency.
  • ad hoc to paper Pseudo ground-truth labels constructed by majority voting of the same VLMs are a reliable proxy for true labels.
    Section 4.1 builds pseudo ground truth from the very models whose accuracy is then estimated (Section 4.2), making the accuracy estimates self-referential.
  • domain assumption MTurk annotations from Cleanlab-flagged images are a valid external benchmark for evaluating renovation quality.
    The MTurk data (from Northcutt et al. 2021) only covers images where Cleanlab found a potential error, not a random sample; the authors acknowledge this limits ImageNet and QuickDraw comparisons (Observation 9).

how reviews work

0 comments
Cite this review

Pith. "Pith review of When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification." pith.science (2026). https://pith.science/paper/TKNOFFJZ

@misc{pith2026250516149,
  author       = {Pith},
  title        = {Pith review of: When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TKNOFFJZ}},
  note         = {Machine review of arXiv:2505.16149}
}
read the original abstract

Image classification benchmark datasets such as CIFAR, MNIST, and ImageNet serve as critical tools for model evaluation. However, despite the cleaning efforts, these datasets still suffer from pervasive noisy labels and often contain missing labels due to the co-existing image pattern where multiple classes appear in an image sample. This results in misleading model comparisons and unfair evaluations. Existing label cleaning methods focus primarily on noisy labels, but the issue of missing labels remains largely overlooked. Motivated by these challenges, we present a comprehensive framework named REVEAL, integrating state-of-the-art pre-trained vision-language models (e.g., LLaVA, BLIP, Janus, Qwen) with advanced machine/human label curation methods (e.g., Docta, Cleanlab, MTurk), to systematically address both noisy labels and missing label detection in widely-used image classification test sets. REVEAL detects potential noisy labels and omissions, aggregates predictions from various methods, and refines label accuracy through confidence-informed predictions and consensus-based filtering. Additionally, we provide a thorough analysis of state-of-the-art vision-language models and pre-trained image classifiers, highlighting their strengths and limitations within the context of dataset renovation by revealing 10 observations. Our method effectively reveals missing labels from public datasets and provides soft-labeled results with likelihoods. Through human verifications, REVEAL significantly improves the quality of 6 benchmark test sets, highly aligning to human judgments and enabling more accurate and meaningful comparisons in image classification.

Figures

Figures reproduced from arXiv: 2505.16149 by the authors.

Figure 1
Figure 1. Exemplary CIFAR-100 test set with noisy labels. The text below each picture denotes the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Exemplary CIFAR-100 training images with multiple labels. The text below each picture [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. REVEAL renovation pipeline. Both the VLM-based and human-annotated methods first assign labels to each image independently. These preliminary labels are then aggregated using a weighted voting ensembling strategy. To refine the results, a score threshold is applied to filter the aggregated labels, followed by a softmax operation to compute the corresponding likelihoods. This process ultimately yields a soft-labeled … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Evaluation on different settings of prompt. Results shown from left to right are from [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Models Pairs Comparison. These three sub-figures illustrate confusion matrix of renovation [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Count of VLMs’ output on CIFAR￾10. The label distribution is presented in [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Label correction visualization across datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Examples comparing REVEAL results with human annotations and single VLM results. With respect to the comparison against MTurk human annotations, we give: Observation 8: High Alignment with Human Annotations Across most datasets, REVEAL method demonstrates a high degree…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 28 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning.arXiv preprint arXiv:2204.14198, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Federico Lucarella, and et al. Flamingo: a visual language model for few-shot learning.arXiv preprint arXiv:2204.14198, 2022

    arXiv 2022

  2. [2]

    Robust bi-tempered logistic loss based on Bregman divergences.Advances in Neural Information Processing Systems, 32, 2019

    Ehsan Amid, Manfred K Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on Bregman divergences.Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  3. [3]

    Openflamingo: An open-source framework for multimodal few-shot learning.arXiv preprint arXiv:2308.01390, 2023

    Hossam Awadalla, Jiahui Yu, Jack Clark, and et al. Openflamingo: An open-source framework for multimodal few-shot learning.arXiv preprint arXiv:2308.01390, 2023

    arXiv 2023

  4. [4]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  5. [5]

    Multi-label classification with partial annotations using class-aware selective loss

    Emanuel Ben-Baruch, Tal Ridnik, Itamar Friedman, Avi Ben-Cohen, Nadav Zamir, Asaf Noy, and Lihi Zelnik-Manor. Multi-label classification with partial annotations using class-aware selective loss. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4764–4772, 2022

    2022

  6. [6]

    Revolt: Collaborative crowdsourcing for labeling machine learning datasets

    Joseph Chee Chang, Saleema Amershi, and Ece Kamar. Revolt: Collaborative crowdsourcing for labeling machine learning datasets. InProceedings of the 2017 CHI conference on human factors in computing systems, pages 2334–2346, 2017

    work page 2017

  7. [7]

    Understanding and utilizing deep neural networks trained with noisy labels

    Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. InInternational conference on machine learning, pages 1062–1070. PMLR, 2019

    work page 2019

  8. [8]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    arXiv 2025

Show all 57 references
  1. [9]

    Learning with instance-dependent label noise: A sample sieve approach

    Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. InInternational Conference on Learning Representations, 2021

    2021

  2. [10]

    Instructblip: Towards general-purpose vision- language models with instruction tuning.arXiv preprint arXiv:2305.06500, 2023

    Xue Dai, Junnan Li, Zhengyuan Dai, and et al. Instructblip: Towards general-purpose vision- language models with instruction tuning.arXiv preprint arXiv:2305.06500, 2023

    2023 arXiv

  3. [11]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  4. [12]

    Learning a deep convnet for multi-label classification with partial labels

    Thibaut Durand, Nazanin Mehrasa, and Greg Mori. Learning a deep convnet for multi-label classification with partial labels. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 647–657, 2019

    2019

  5. [13]

    Training deep neural-networks using a noise adapta- tion layer

    Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using a noise adapta- tion layer. InInternational conference on learning representations, 2017

    2017

  6. [14]

    Caltech-256 object category dataset

    Gregory Griffin, Alex Holub, Pietro Perona, et al. Caltech-256 object category dataset. Technical report, Technical Report 7694, California Institute of Technology Pasadena, 2007

    2007

  7. [15]

    Using trusted data to train deep networks on labels corrupted by severe noise.Advances in neural information processing systems, 31, 2018

    Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise.Advances in neural information processing systems, 31, 2018

    2018

  8. [16]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InInternational conference on machine learning, pages 4904–4916. ...

    2021

  9. [17]

    Detecting and preventing confused labels in crowdsourced data.Proceedings of the VLDB Endowment, 13(12):2522–2535, 2020

    Evgeny Krivosheev, Siarhei Bykau, Fabio Casati, and Sunil Prabhakar. Detecting and preventing confused labels in crowdsourced data.Proceedings of the VLDB Endowment, 13(12):2522–2535, 2020

    2020

  10. [18]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009

    2009

  11. [19]

    Constrained instance and class reweighting for robust learning under label noise.arXiv preprint arXiv:2111.05428, 2021

    Abhishek Kumar and Ehsan Amid. Constrained instance and class reweighting for robust learning under label noise.arXiv preprint arXiv:2111.05428, 2021

    2021 arXiv

  12. [20]

    Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 1998

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  13. [21]

    Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  14. [22]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven CH Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    2023 arXiv

  15. [23]

    Align before fuse: Vision and language representation learning with momentum distillation.Advances in neural information processing systems, 34:9694–9705, 2021

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation.Advances in neural information processing systems, 34:9694–9705, 2021

    2021

  16. [24]

    Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    Haotian Liu, Chunyuan Zhang, Yuheng Xu, Shiqi Chang, Jianwei Zhang, Yizhou Wang, and Lu Yuan. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  17. [25]

    Automatic dataset construction (adc): Sample collection, data curation, and beyond.arXiv preprint arXiv:2408.11338, 2024

    Minghao Liu, Zonglin Di, Jiaheng Wei, Zhongruo Wang, Hengxiang Zhang, Ruixuan Xiao, Haoyu Wang, Jinlong Pang, Hao Chen, Ankit Shah, et al. Automatic dataset construction (adc): Sample collection, data curation, and beyond.arXiv preprint arXiv:2408.11338, 2024

    2024 arXiv

  18. [26]

    Human and ai perceptual differences in image classification errors

    Minghao Liu, Jiaheng Wei, Yang Liu, and James Davis. Human and ai perceptual differences in image classification errors. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 14318–14326, 2025

    2025

  19. [27]

    Classification with noisy labels by importance reweighting

    Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447–461, 2016

    2016

  20. [28]

    Peer loss functions: Learning from noisy labels without knowing noise rates

    Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. InInternational Conference on Machine Learning, pages 6226–6236. PMLR, 2020

    2020

  21. [29]

    Normalized loss functions for deep learning with noisy labels

    Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. InInternational Conference on Machine Learning, pages 6543–6553. PMLR, 2020

    2020

  22. [30]

    Label structure preserv- ing contrastive embedding for multi-label learning with missing labels.arXiv preprint arXiv:2209.01314, 2022

    Zhongchen Ma, Lisha Li, Qirong Mao, and Songcan Chen. Label structure preserv- ing contrastive embedding for multi-label learning with missing labels.arXiv preprint arXiv:2209.01314, 2022

    2022 arXiv

  23. [31]

    Learning with noisy labels

    Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. InAdvances in neural information processing systems, pages 1196–1204, 2013

    2013

  24. [32]

    Confident learning: Estimating uncertainty in dataset labels.Journal of Artificial Intelligence Research, 70:1373–1411, 2021

    Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels.Journal of Artificial Intelligence Research, 70:1373–1411, 2021

    2021

  25. [33]

    Northcutt, Anish Athalye, and Jonas Mueller

    Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. InProceedings of the 35th Conference on Neural Information Processing Systems Track on Datasets and Benchmarks, December 2021

    2021

  26. [34]

    Pervasive label errors in test sets destabilize machine learning benchmarks.arXiv preprint arXiv:2103.14749, 2021

    Curtis G Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks.arXiv preprint arXiv:2103.14749, 2021. 12

    2021 arXiv

  27. [35]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  28. [36]

    Learning from noisy labels by regularized estimation of annotator confusion

    Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C Alexander, and Nathan Silberman. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11244–11253, 2019

    2019

  29. [37]

    Policy learning using weak supervision.Advances in Neural Information Processing Systems, 34, 2021

    Jingkang Wang, Hongyi Guo, Zhaowei Zhu, and Yang Liu. Policy learning using weak supervision.Advances in Neural Information Processing Systems, 34, 2021

    2021

  30. [38]

    Symmetric cross entropy for robust learning with noisy labels

    Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 322–330, 2019

    2019

  31. [39]

    Combating noisy labels by agreement: A joint training method with co-regularization

    Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13726–13735, 2020

    2020

  32. [40]

    Open-set label noise can improve robustness against inherent label noise.Advances in Neural Information Processing Systems, 34, 2021

    Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise.Advances in Neural Information Processing Systems, 34, 2021

    2021

  33. [41]

    To smooth or not? when label smoothing meets noisy labels

    Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. InInternational Conference on Machine Learning, pages 23589–23614. PMLR, 2022

    2022

  34. [42]

    When optimizing f-divergence is robust with label noise.arXiv preprint arXiv:2011.03687, 2020

    Jiaheng Wei and Yang Liu. When optimizing f-divergence is robust with label noise.arXiv preprint arXiv:2011.03687, 2020

    2011 arXiv

  35. [43]

    Distributionally robust post-hoc classifiers under prior shifts

    Jiaheng Wei, Harikrishna Narasimhan, Ehsan Amid, Wen-Sheng Chu, Yang Liu, and Ab- hishek Kumar. Distributionally robust post-hoc classifiers under prior shifts. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  36. [44]

    Learning with noisy labels revisited: A study using real-world human annotations.arXiv preprint arXiv:2110.12088, 2021

    Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations.arXiv preprint arXiv:2110.12088, 2021

    2021 arXiv

  37. [45]

    To aggregate or not? learning with separate noisy labels

    Jiaheng Wei, Zhaowei Zhu, Tianyi Luo, Ehsan Amid, Abhishek Kumar, and Yang Liu. To aggregate or not? learning with separate noisy labels. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2523–2535, 2023

    2023

  38. [46]

    Fairness improves learning from noisily labeled long-tailed data.arXiv preprint arXiv:2303.12291, 2023

    Jiaheng Wei, Zhaowei Zhu, Gang Niu, Tongliang Liu, Sijia Liu, Masashi Sugiyama, and Yang Liu. Fairness improves learning from noisily labeled long-tailed data.arXiv preprint arXiv:2303.12291, 2023

    2023 arXiv

  39. [47]

    Vision- language models are strong noisy label detectors.Advances in Neural Information Processing Systems, 37:58154–58173, 2024

    Tong Wei, Hao-Tian Li, ChunShu Li, Jiang-Xin Shi, Yu-Feng Li, and Min-Ling Zhang. Vision- language models are strong noisy label detectors.Advances in Neural Information Processing Systems, 37:58154–58173, 2024

    2024

  40. [48]

    iclip: Bridging image classification and contrastive language-image pre-training for visual recognition

    Yixuan Wei, Yue Cao, Zheng Zhang, Houwen Peng, Zhuliang Yao, Zhenda Xie, Han Hu, and Baining Guo. iclip: Bridging image classification and contrastive language-image pre-training for visual recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2023

  41. [49]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    arXiv

  42. [50]

    Are anchor points really indispensable in label-noise learning?Advances in Neural Information Processing Systems, 32, 2019

    Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning?Advances in Neural Information Processing Systems, 32, 2019. 13

    2019

  43. [51]

    Filip: Fine-grained interactive language-image pre-training.arXiv preprint arXiv:2111.07783, 2021

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training.arXiv preprint arXiv:2111.07783, 2021

    2021 arXiv

  44. [52]

    How does disagreement help generalization against label corruption? InInternational Conference on Machine Learning, pages 7164–7173

    Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? InInternational Conference on Machine Learning, pages 7164–7173. PMLR, 2019

    2019

  45. [53]

    Lit: Zero-shot transfer with locked-image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18123–18133, 2022

    2022

  46. [54]

    Learning in imperfect environment: Multi-label classification with long-tailed distribution and partial labels

    Wenqiao Zhang, Changshuo Liu, Lingze Zeng, Bengchin Ooi, Siliang Tang, and Yueting Zhuang. Learning in imperfect environment: Multi-label classification with long-tailed distribution and partial labels. InProceedings of the IEEE/CVF International Conference on Computer Vision,...

    2023

  47. [55]

    Conditional prompt learning for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816–16825, 2022

    2022

  48. [56]

    Clusterability as an alternative to anchor points when learning with noisy labels

    Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. InInternational Conference on Machine Learning, pages 12912–12923. PMLR, 2021

    2021

  49. [57]

    Unmasking and improving data credibility: A study with datasets for training harmless language models.arXiv preprint arXiv:2311.11202, 2023

    Zhaowei Zhu, Jialu Wang, Hao Cheng, and Yang Liu. Unmasking and improving data credibility: A study with datasets for training harmless language models.arXiv preprint arXiv:2311.11202, 2023. 14 A Broader Impacts and Limitations In this section, we list some of the broader impa...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.