Pith. sign in

REVIEW 3 major objections 6 minor 57 references

Exploring Aleatoric Uncertainty in Object Detection via Vision Foundation Models

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that the aleatoric uncertainty of each object instance in a detection dataset can be read off from the feature space of a vision foundation model, and that using this score to filter noisy or redundant samples and to…

desk verdict A plausible data-centric trick that delivers small consistent gains, but the core score is misdefined in the written equations and the aleatoric-uncertainty claim outruns the evidence. read the letter →

arxiv 2411.17767 v1 pith:X4AOH547 submitted 2024-11-26 cs.CV cs.LG

Peng Cui , Guande He , Dan Zhang , Zhijie Deng , Yinpeng Dong , Jun Zhu This is my paper · ORCID
classification cs.CVcs.LG
keywords aleatoricuncertaintyobjectdetectionvisionfoundationmodelsSAMMahalanobisdistancesamplefilteringregularizationdata-centriclearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that aleatoric (data) uncertainty in object detection can be quantified per instance by measuring how far an object's pooled feature vector sits from the centroid of its class in the feature space of a vision foundation model. The authors use the Segment Anything Model (SAM) to extract object features, fit a class-conditional Gaussian with a shared covariance, and define a Mahalanobis-distance-based score as the uncertainty measure. They then put this score to use in two plug-and-play ways: discarding high-uncertainty (noisy) and redundant training samples, and weighting an entropy regularizer so that harder samples receive stronger regularization. Across COCO and BDD100K, and across CNN-based and transformer-based detectors, the uncertainty-aware filter and regularizer improve average precision and recall. A sympathetic reader would care because the approach treats uncertainty as an extra annotation computed once offline, so it can benefit any detector without changing the model.

What carries the argument

The central object is the Mahalanobis distance-based uncertainty score computed from SAM's vision encoder: for each labeled object instance, the feature vector $V(z_j)$ is obtained by pooling the SAM feature map inside the ground-truth box, and a class-conditional Gaussian $\mathcal{N}(V(z) \mid \mu_k, \Sigma)$ with a class-shared covariance $\Sigma$ is fit on the training set. The score $M(z_j|c_j) = -\left(V(z_j)-\mu_{c_j}\right)^\top \Sigma^{-1}\left(V(z_j)-\mu_{c_j}\right)$ measures distance to the class centroid, and a per-class log-transform plus min-max normalization turns it into an uncertainty value in $(0,1)$. This score does the argument's work because it is cheap to compute once per dataset, is model-agnostic, and its two downstream uses (quantile-based filtering and uncertainty-weighted entropy regularization) are what produce the reported performance gains.

What would settle it

Take a set of clean COCO objects, corrupt a random subset by shifting or shrinking their boxes or swapping their class labels, and check whether the Mahalanobis-based score ranks the corrupted instances among the top 5–10% of uncertainty scores. If the score does not separate these synthetic noisy samples from hard-but-clean ones, or if removing the top-5% scored samples fails to improve AP relative to removing a random 5%, the central claim that the score identifies noise rather than merely difficulty would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that the Mahalanobis distance of a SAM-pooled object feature to its class-conditional Gaussian centroid is a reliable aleatoric uncertainty score, and that this score captures three meaningful regimes: easy objects with low score, hard objects (occluded, small, or obscure) with medium score, and noisy objects (misleading boxes or wrong labels) with high score. The paper further claims that this score is practically useful: keeping only samples below a quantile of the score abandons harmful noisy instances and improves AP, while binning samples by score and discarding a fraction within each bin removes redundancy more safely than uniform random dropping. Finally, the paper claims that an uncertainty-weighted entropy regularizer, which replaces a constant entropy penalty with one scaled by each sample's uncertainty score, consistently beats both vanilla training and constant-entropy regularization, with larger gains on small-capacity models.

Load-bearing premise

The paper assumes that a single class-conditional Gaussian with a shared covariance matrix, fit to SAM-pooled features of the same training set, accurately describes the feature distribution, so that Mahalanobis distance is a valid proxy for aleatoric uncertainty without any ground-truth uncertainty labels to calibrate against.

Editorial extensions

If this is right

  • Detection training can be made more reliable by treating uncertainty as an offline per-instance annotation, avoiding any extra cost during training.
  • Discarding roughly 5% of the most uncertain samples yields consistent AP gains across YOLOX, FCOS, Deformable DETR, and DINO, suggesting that noisy instances in standard benchmarks actively hurt training.
  • Uncertainty-aware redundant-sample pruning degrades performance far less than uniform random pruning, supporting a data-pruning paradigm guided by feature-space density.
  • Uncertainty-aware entropy regularization improves small-capacity detectors by a larger margin than large ones, which could translate to cheaper deployment of compact models.
  • The approach transfers to a self-driving dataset (BDD100K) with long-tailed class distributions, indicating the score is not tuned to COCO's statistics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never validates the score against ground-truth uncertainty labels, so a natural next test is to corrupt known-clean boxes with synthetic noise and check whether the score ranks them as more uncertain; this would separate genuine noise detection from mere hardness ranking.
  • Because the score only relies on SAM's frozen features and the training labels, it could be extended to other dense-prediction tasks like segmentation or to label-efficient settings where noisy pseudo-labels come from an auto-labeling pipeline.
  • The class-shared covariance assumption could be relaxed to class-specific covariances or a mixture-of-Gaussians with more components; the reported robustness to hyperparameters suggests the score is driven by the centroid distance more than by the covariance shape.
  • The authors' closing remark hints that filtering high-uncertainty COCO-derived instruction data could reduce hallucination in large vision-language models; this is a testable application the paper does not itself evaluate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a method for estimating per-object aleatoric (data) uncertainty in object detection by computing a Mahalanobis-distance-based score in the feature space of SAM. A class-conditional Gaussian with a shared covariance is fit to SAM-pooled object features from the training set, and the resulting score is used in two plug-and-play ways: to filter noisy and redundant training instances, and to define an uncertainty-aware entropy regularizer for the classification loss. Experiments are reported on COCO and BDD100K with YOLOX-S/M, Deformable DETR, FCOS, and DINO, showing consistent AP improvements when the proposed filtering and regularization are applied.

Significance. If the proposed score genuinely quantifies aleatoric uncertainty in object detection, the paper would contribute a practical data-centric tool that is model-agnostic and offline-computable, with potentially broad applicability to data pruning and robust training. The paper's strengths include the use of a large-scale vision foundation model (SAM) for feature extraction, the plug-and-play formulation, and the breadth of detectors and benchmarks considered. However, the current evidence does not establish the central claim: the score is not validated against any ground-truth uncertainty or annotation-noise benchmark, and the formal definition in Eqs. (3)-(4) is internally inconsistent as written. The reported downstream gains are consistent with a weaker interpretation of the score as a generic difficulty/outlier measure.

major comments (3)
  1. [Section 3, Eqs. (3)-(4)] The uncertainty score is undefined as written. Eq. (3) defines M(zj|cj) as the negative of the quadratic form (V(zj) - mu_cj)^T Sigma^{-1} (V(zj) - mu_cj). Since this quadratic form is nonnegative (for positive definite Sigma), M is nonpositive for every object, and log M in Eq. (4) is therefore not a real number for any object off the class centroid (and is -infinity at the centroid). The positive scores shown in Fig. 1 indicate that this is likely a sign typo, but as submitted the method cannot be executed or audited. Please correct the sign and state the exact scoring function used, including how Sigma is estimated and made invertible for the SAM feature dimensionality.
  2. [Section 3 and Sections 5.1-5.2] The empirical validation is indirect and does not support the central claim that the score measures aleatoric (data) uncertainty. The paper evaluates the score only through its downstream effects on filtering and regularization, which the authors themselves call "proxy tasks" (Section 4). These gains are consistent with the score being a generic outlier or difficulty measure, and they do not distinguish aleatoric uncertainty from epistemic uncertainty or distributional atypicality. The qualitative examples in Figs. 1, A2, and A3 are illustrative only. Please add a direct validation: for instance, inject controlled label noise or bounding-box jitter into a subset of COCO and measure the rank correlation between the score and the injected noise level, or compare the score against human difficulty ratings. Without such evidence, the paper's conclusion that it "accurately characterize[s] aleatoric uncertainty" (Section 6) is not established.
  3. [Section 5, Implementation Details and Tables 1, 3, 4, 5] The hyperparameters beta and the filtering quantile p appear to be selected without a stated held-out procedure. Specifically, Section 5 states that beta is set to 0.2 for YOLOX and 0.3 for Deformable DETR, and Tables 3 and 5 use p = 95% and 90% across models. If these values were tuned on the COCO val set, the reported gains may reflect model selection on the evaluation set. The paper should state the selection protocol, report results across a range of p (as is done for beta in Table 6), or use a separate validation split for hyperparameter choice.
minor comments (6)
  1. [Abstract and Section 1] There are typos: "curial" in the abstract should be "crucial", and "aimt" in Section 1 should be "aim".
  2. [Eq. (7)] The notation in Eq. (7) is confusing due to nested braces and superscripts; please rewrite it more clearly, e.g., by defining the per-class quantile threshold first and then the retained set.
  3. [Eq. (8)] The definition of B^{c_j}_m is inconsistent with its use; please define the bin index and class explicitly, e.g., as B^c_m = {j : c_j = c and d(z_j|c_j) in I_m}, and then define D* accordingly.
  4. [Section 5.2, Table 5] Table 5 reports only AP for BDD100K; please also report AP50, AP75, or AR to allow a fuller comparison, or state why these are omitted.
  5. [Section 5.2] The phrase "uniforming data selection" should be "uniform data selection" or "uniform sampling".
  6. [Fig. 2] The histogram in Fig. 2 appears without labeled axes; please add axis labels and a caption explaining the units of the horizontal axis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the uncertainty score is an independently defined data-derived annotation, and its validation rests on external detection benchmarks rather than on the score's own construction.

full rationale

The paper's central derivation is not circular. Object features are pooled from SAM using ground-truth boxes, a class-conditional Gaussian is fit to those training features, and a Mahalanobis-distance score is computed from that fit (Eqs. 1-4). This score is not defined in terms of detection accuracy, and no fitted parameter is renamed as a prediction; the score is an independent annotation of the training data. The empirical validation uses held-out COCO and BDD100K validation sets and compares against vanilla training, constant-entropy regularization, and random filtering. These improvements are not forced by construction: the quantile filter and the uncertainty-weighted regularizer could plausibly fail, and the reported gains are contingent empirical results. The one self-citation, [5], is used only to motivate Gaussian feature-space modeling and is corroborated by external references [34,45]; it is not a uniqueness theorem and does not forbid alternative scoring functions. The paper does contain a formal defect: Eq. (3) defines M as a negative quadratic form, so taking log M in Eq. (4) is mathematically invalid as written, and the central claim that the score measures aleatoric rather than generic atypicality is not validated against ground-truth noise labels. Those are correctness and external-validity concerns, not circularity. No load-bearing step reduces by definition or by self-citation to its own inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim rests on the Gaussianity of SAM feature distributions, the semantic organization of SAM features, the validity of bounding-box pooling, and the correspondence between Mahalanobis distance and annotation noise. The paper provides only qualitative visual evidence and downstream performance for these assumptions.

free parameters (3)
  • p (noisy sample quantile) = 0.95 or 0.90
    Chosen as the fraction of training data retained after discarding high-uncertainty samples; no selection criterion or sensitivity study for p is given in the main text.
  • beta (entropy regularization coefficient) = 0.2 for YOLOX, 0.3 for Deformable DETR
    Set per model in Section 5 Implementation Details; ablation in Table 6 shows UA-entropy robustness, but the values are tuned, not derived.
  • M (number of uncertainty bins for redundant filtering) = not specified
    Equation (8) requires partitioning each class into M bins, but M is never stated in the paper.
assumptions (5)
  • domain assumption SAM's vision encoder feature space is semantically organized so that objects of the same class cluster together.
    Invoked in Section 3 to justify class-conditional Gaussian modeling; evidence is cited from [46] but not validated in this paper.
  • ad hoc to paper A class-conditional Gaussian with a shared covariance matrix adequately models SAM object features.
    Equation (1) postulates this structure; no goodness-of-fit or comparison with other densities is provided.
  • ad hoc to paper Mahalanobis distance to the class centroid is a monotone proxy for aleatoric uncertainty, that is, for noise and hardness.
    This is the central modeling assumption, supported only by qualitative visualizations (Fig 1, A2, A3) and downstream gains.
  • domain assumption Bounding-box pooling of the SAM feature map yields a per-object feature vector that preserves semantic content.
    Algorithm 1 computes V(zj) based on ground-truth boxes or masks; pooling details are unspecified.
  • domain assumption Ground-truth labels are reliable enough to estimate class means, despite also being noisy.
    Equation (2) uses all training labels to compute mu_k; if labels are systematically wrong, the Gaussian centers are corrupted.
invented entities (1)
  • Per-object aleatoric uncertainty score d(zj|cj)
    purpose: Extra annotation used for sample filtering and loss reweighting
    The score is a derived quantity from SAM features and training labels, with no independent human or benchmark ground-truth uncertainty labels; only qualitative visual agreement is shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Aleatoric Uncertainty in Object Detection via Vision Foundation Models." pith.science (2026). https://pith.science/paper/X4AOH547

@misc{pith2026241117767,
  author       = {Pith},
  title        = {Pith review of: Exploring Aleatoric Uncertainty in Object Detection via Vision Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X4AOH547}},
  note         = {Machine review of arXiv:2411.17767}
}
read the original abstract

Datasets collected from the open world unavoidably suffer from various forms of randomness or noiseness, leading to the ubiquity of aleatoric (data) uncertainty. Quantifying such uncertainty is particularly pivotal for object detection, where images contain multi-scale objects with occlusion, obscureness, and even noisy annotations, in contrast to images with centric and similar-scale objects in classification. This paper suggests modeling and exploiting the uncertainty inherent in object detection data with vision foundation models and develops a data-centric reliable training paradigm. Technically, we propose to estimate the data uncertainty of each object instance based on the feature space of vision foundation models, which are trained on ultra-large-scale datasets and able to exhibit universal data representation. In particular, we assume a mixture-of-Gaussian structure of the object features and devise Mahalanobis distance-based measures to quantify the data uncertainty. Furthermore, we suggest two curial and practical usages of the estimated uncertainty: 1) for defining uncertainty-aware sample filter to abandon noisy and redundant instances to avoid over-fitting, and 2) for defining sample adaptive regularizer to balance easy/hard samples for adaptive training. The estimated aleatoric uncertainty serves as an extra level of annotations of the dataset, so it can be utilized in a plug-and-play manner with any model. Extensive empirical studies verify the effectiveness of the proposed aleatoric uncertainty measure on various advanced detection models and challenging benchmarks.

Figures

Figures reproduced from arXiv: 2411.17767 by the authors.

Figure 1
Figure 1. Visualization of scoring objects with corresponding uncertainty scores in training images of MS-COCO [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The histogram of d(zj |cj ) for MS-COCO. samples. Specifically, we can empirically estimate them by µk = 1 Nk X j:cj=k V (zj ), Σ = 1 N X k X j:cj=k (V (zj ) − µk)(V (zj ) − µk) ⊤, (2) where Nk is the number of training samples (i.e., object instances) with the label cj = k. Mahalanobis distance-based uncertainty score. Lever￾aging the class-conditional Gaussian distributions fitted above, we measure the Mahalanobis… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 40 canonical work pages

  1. [31]

    Towards robust adaptive object detection under noisy annotations

    Xinyu Liu, Wuyang Li, Qiushi Yang, Baopu Li, and Yixuan Yuan. Towards robust adaptive object detection under noisy annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14207–14216,

    work page

  2. [1]

    Segment any anomaly without training via hybrid prompt regularization

    Yunkang Cao, Xiaohao Xu, Chen Sun, Yuqi Cheng, Zongwei Du, Liang Gao, and Weiming Shen. Segment any anomaly without training via hybrid prompt regularization. arXiv preprint arXiv:2305.10724, 2023. 3

    arXiv 2023

  3. [2]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 23–28, 2020, Proceedings, Part I 16 , pages 213–229. Springer, 2020. 1, 6

    work page 2020

  4. [3]

    Data uncertainty learning in face recognition

    Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5710–5719, 2020. 1, 3

    work page 2020

  5. [4]

    Confidence- based reliable learning under dual noises

    Peng Cui, Yang Yue, Zhijie Deng, and Jun Zhu. Confidence- based reliable learning under dual noises. In Advances in Neural Information Processing Systems, 2022. 1

    work page 2022

  6. [5]

    Learning sample difficulty from pre-trained models for reliable prediction

    Peng Cui, Dan Zhang, Zhijie Deng, Yinpeng Dong, and Jun Zhu. Learning sample difficulty from pre-trained models for reliable prediction. Advances in Neural Information Process- ing Systems, 36, 2024. 3

    work page 2024

  7. [6]

    Decomposition of uncer- tainty in bayesian deep learning for efficient and risk-sensitive learning

    Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decomposition of uncer- tainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pages 1184–1193. PMLR, 2018. 1, 3

    work page 2018

  8. [7]

    Aleatory or epis- temic? does it matter? Structural safety, 31(2):105–112,

    Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epis- temic? does it matter? Structural safety, 31(2):105–112,

    work page

Show all 57 references
  1. [8]

    Dietterich and Alex Guyer

    Thomas G. Dietterich and Alex Guyer. The familiarity hy- pothesis: Explaining the behavior of deep open set methods. Pattern Recognition, 132:108931, 2022. 3

    2022

  2. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  3. [10]

    Everingham, S

    M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98–136, 2015. 2

    2015

  4. [11]

    Dropout as a Bayesian approximation: Representing model uncertainty in deep learn- ing

    Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learn- ing. In international conference on machine learning, pages 1050–1059, 2016. 3

    2016

  5. [12]

    Yolox: Exceeding yolo series in 2021

    Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, 2021. 2, 6

    2021 arXiv

  6. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 1

    2016

  7. [14]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 1

    2017

  8. [15]

    Open-set image tagging with multi-grained text supervision

    Xinyu Huang, Yi-Jie Huang, Youcai Zhang, Weiwei Tian, Rui Feng, Yuejie Zhang, Yanchun Xie, Yaqian Li, and Lei Zhang. Open-set image tagging with multi-grained text supervision. arXiv e-prints, pages arXiv–2310, 2023. 3

    2023

  9. [16]

    Tag2text: Guiding vision-language model via image tagging

    Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text: Guiding vision-language model via image tagging. arXiv preprint arXiv:2303.05657, 2023. 3

    2023 arXiv

  10. [17]

    Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods

    Eyke H ¨ullermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Mach. Learn., 110(3):457–506, 2021. 1, 3

    2021

  11. [18]

    What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems

    Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017. 1, 3

    2017

  12. [19]

    Why normalizing flows fail to detect out-of-distribution data

    Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of-distribution data. In Advances in Neural Information Processing Systems, pages 20578–20589. Curran Associates, Inc., 2020. 3

    2020

  13. [20]

    Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Visio...

    2023

  14. [21]

    Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. In ICML, 2022. 3

    2022

  15. [22]

    BLIP- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, pages 19730–19742. PMLR, 2023. 3

    2023

  16. [23]

    Clipsam: Clip and sam collaboration for zero-shot anomaly segmentation, 2024

    Shengze Li, Jianjian Cao, Peng Ye, Yuhan Ding, Chongjun Tu, and Tao Chen. Clipsam: Clip and sam collaboration for zero-shot anomaly segmentation, 2024. 3

    2024

  17. [24]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pages 292–305, 2023. 9

    2023

  18. [25]

    Gmm- seg: Gaussian mixture based generative semantic segmenta- tion models

    Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang. Gmm- seg: Gaussian mixture based generative semantic segmenta- tion models. In Advances in Neural Information Processing Systems, 2022. 3

    2022

  19. [26]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence 9 Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedi...

    2014

  20. [27]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. In Pro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 2, 5

    2017

  21. [28]

    Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models

    Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models...

    arXiv

  22. [29]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 8

    2024

  23. [30]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 3, 8

    2023 arXiv

  24. [32]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10012–10022, 2021. 1

    2021

  25. [33]

    Calibrating deep neural networks using focal loss

    Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Infor- mation Processing Systems, 33:15288–15299, 2020. 5

    2020

  26. [34]

    Deep deterministic uncertainty: A new simple baseline

    Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deep deterministic uncertainty: A new simple baseline. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 24384–24394, 2023. 3

    2023

  27. [35]

    Maxime Oquab, Timoth´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicola...

  28. [36]

    Regularizing neural networks by penalizing confident output distributions

    Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017. 2

    2017 arXiv

  29. [37]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , p...

    2021

  30. [38]

    Yolov3: An incremental improvement

    Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018. 6

    2018 arXiv

  31. [39]

    A simple fix to mahalanobis distance for improving near-ood detection

    Jie Jessie Ren, Stanislav Fort, Jeremiah Zhe Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. ArXiv, abs/2106.09022, 2021. 3

    2021 arXiv

  32. [40]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information pro- cessing systems, pages 91–99, 2015. 1

    2015

  33. [41]

    Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024. 3

    2024

  34. [42]

    Segmenter: Transformer for semantic segmentation

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7262–7272, 2021. 1

    2021

  35. [43]

    FCOS: fully convolutional one-stage object detection

    Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: fully convolutional one-stage object detection. In IEEE/CVF International Conference on Computer Vision, ICCV, pages 9626–9635, 2019. 2, 6

    2019

  36. [44]

    Mlp-mixer: An all-mlp architecture for vision

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, An- dreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. In Advances in Neural Inf...

    2021

  37. [45]

    Uncertainty estimation using a single deep de- terministic neural network

    Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep de- terministic neural network. In International conference on machine learning, pages 9690–9700. PMLR, 2020. 3

    2020

  38. [46]

    Segment and Caption Anything

    Huang Xiaoke, Wang Jianfeng, Tang Yansong, Zhang Zheng, Hu Han, Lu Jiwen, Wang Lijuan, and Liu Zicheng. Segment and Caption Anything. In CVPR, 2024. 2, 3

    2024

  39. [47]

    Segformer: Simple and efficient design for semantic segmentation with transform- ers

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transform- ers. Advances in Neural Information Processing Systems, 34: 12077–12090, 2021. 1

    2021

  40. [48]

    BDD100K: A diverse driving dataset for heterogeneous mul- titask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A diverse driving dataset for heterogeneous mul- titask learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seat...

    2020

  41. [49]

    mixup: Beyond empirical risk minimization

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. 6

    2018

  42. [50]

    Dino: Detr with 10 improved denoising anchor boxes for end-to-end object detec- tion

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. Dino: Detr with 10 improved denoising anchor boxes for end-to-end object detec- tion. In The Eleventh International Conference on Learning Representations, 2022. 1, 2, 6

    2022

  43. [51]

    Wang Zhang, Ziwen Martin Ma, Subhro Das, Tsui-Wei Lily Weng, Alexandre Megretski, Luca Daniel, and Lam M. Nguyen. One step closer to unbiased aleatoric uncertainty estimation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(15):16857–16864, 2024. 1, 3

    2024

  44. [52]

    Recognize anything: A strong image tagging model

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. arXiv preprint arXiv:2306.03514, 2023. 3

    2023 arXiv

  45. [53]

    Segment any- thing model for medical image segmentation: Current ap- plications and future directions

    Yichi Zhang, Zhenrong Shen, and Rushi Jiao. Segment any- thing model for medical image segmentation: Current ap- plications and future directions. Computers in Biology and Medicine, 171:108238, 2024. 3

    2024

  46. [54]

    Analyzing and mitigating object hallucination in large vision- language models

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision- language models. In The Twelfth International Conference on Learning Representations, 2023. 9

    2023

  47. [55]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representa- tions, 2023. 8

    2023

  48. [56]

    Deformable {detr}: Deformable transformers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable {detr}: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021. 1, 2

    2021

  49. [57]

    dog” and “bird

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021. 6 11 Exploring Aleatoric Uncertainty in Object Detection via Vision ...

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.