{"id":"78632f5a-f871-4fb8-a104-c558e6699627","arxiv_id":"2411.17767","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A Mahalanobis distance computed in SAM feature space is used as an aleatoric uncertainty score for object instances, and filtering and reweighting by this score yields modest AP gains on COCO and BDD100K.","lead":"Using features from the Segment Anything Model, this paper scores how noisy or hard each object in a detection dataset is, then filters and reweights training samples accordingly. It is proposed as a plug-and-play data-centric add-on for any detector, and the authors report small accuracy gains on COCO and BDD100K.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Mahalanobis-distance score is never validated against any ground-truth aleatoric uncertainty, and Eq. (3)–(4) are undefined as written (negative M logged); the downstream gains do not establish the central claim.","rationale":"The reader's weakest assumption identifies the same gap: the paper assumes that a class-conditional Gaussian fit to SAM features makes Mahalanobis distance a valid proxy for aleatoric uncertainty, without testing this against any ground-truth uncertainty labels. My stress-test agrees and sharpens it: the proxy tasks (filtering and regularization) cannot distinguish 'true aleatoric uncertainty' from generic outlier/hard-example reweighting, so the central claim is unsupported even if the equations were typo-free. The sign/log inconsistency in Eq. (3)–(4) is a separate correctness problem that makes the method undefined as written and increases correctness risk. I do not see a reason to move the reader's REJECT verdict; the idea may be salvageable, but the current evidence and formal presentation do not support acceptance.","tokens_in":14890,"tokens_out":5536,"duration_ms":57003,"concrete_test":"Build a controlled validation set from COCO train2017: take a fixed set of clean object instances, compute SAM-pooled features, and fit the Gaussian as in Eq. (2) on clean objects only. Generate corrupted copies by (i) flipping class labels to a confusable class, (ii) shifting or resizing bounding boxes by controlled amounts, and (iii) overlaying occlusion patches. Compute the proposed score for clean and corrupted instances and measure rank separation (e.g., AUROC or Kendall tau) within each class and scale bucket. If corrupted instances are not ranked above clean ones significantly, the claim that this score captures aleatoric uncertainty fails. As a secondary check, recompute Table 3 after correcting the sign in Eq. (3) to confirm the reported gains are reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Mahalanobis distance of SAM-pooled object features to a class-conditional Gaussian centroid is a reliable measure of aleatoric uncertainty (Section 3, Eq. 1–4). For this to hold, the score must track object-level data noise (mislabeling, box jitter, occlusion) rather than generic feature atypicality or class-context rarity. The paper never validates this. The offered support is qualitative images (Fig. 1, A2, A3) and proxy gains from filtering and regularization. Both are consistent with a weaker statement: MD identifies outliers or hard examples, and removing or reweighting them can help training. That does not establish 'aleatoric uncertainty,' and the Gaussian model's core assumption is left untested. In addition, as written the score is undefined: Eq. (3) defines M as a negative quadratic form, so taking log M in Eq. (4) is invalid for every object; the positive scores in Fig. 1 indicate a sign typo, but without code or data the actual scoring function cannot be audited. Because the central claim rests on this unvalidated and internally inconsistent formalization, the empirical results do not support the stated conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for estimating per-object aleatoric (data) uncertainty in object detection by computing a Mahalanobis-distance-based score in the feature space of SAM. A class-conditional Gaussian with a shared covariance is fit to SAM-pooled object features from the training set, and the resulting score is used in two plug-and-play ways: to filter noisy and redundant training instances, and to define an uncertainty-aware entropy regularizer for the classification loss. Experiments are reported on COCO and BDD100K with YOLOX-S/M, Deformable DETR, FCOS, and DINO, showing consistent AP improvements when the proposed filtering and regularization are applied.","tokens_in":15140,"tokens_out":5024,"duration_ms":47338,"significance":"If the proposed score genuinely quantifies aleatoric uncertainty in object detection, the paper would contribute a practical data-centric tool that is model-agnostic and offline-computable, with potentially broad applicability to data pruning and robust training. The paper's strengths include the use of a large-scale vision foundation model (SAM) for feature extraction, the plug-and-play formulation, and the breadth of detectors and benchmarks considered. However, the current evidence does not establish the central claim: the score is not validated against any ground-truth uncertainty or annotation-noise benchmark, and the formal definition in Eqs. (3)-(4) is internally inconsistent as written. The reported downstream gains are consistent with a weaker interpretation of the score as a generic difficulty/outlier measure.","major_comments":[{"comment":"The uncertainty score is undefined as written. Eq. (3) defines M(zj|cj) as the negative of the quadratic form (V(zj) - mu_cj)^T Sigma^{-1} (V(zj) - mu_cj). Since this quadratic form is nonnegative (for positive definite Sigma), M is nonpositive for every object, and log M in Eq. (4) is therefore not a real number for any object off the class centroid (and is -infinity at the centroid). The positive scores shown in Fig. 1 indicate that this is likely a sign typo, but as submitted the method cannot be executed or audited. Please correct the sign and state the exact scoring function used, including how Sigma is estimated and made invertible for the SAM feature dimensionality.","section":"Section 3, Eqs. (3)-(4)"},{"comment":"The empirical validation is indirect and does not support the central claim that the score measures aleatoric (data) uncertainty. The paper evaluates the score only through its downstream effects on filtering and regularization, which the authors themselves call \"proxy tasks\" (Section 4). These gains are consistent with the score being a generic outlier or difficulty measure, and they do not distinguish aleatoric uncertainty from epistemic uncertainty or distributional atypicality. The qualitative examples in Figs. 1, A2, and A3 are illustrative only. Please add a direct validation: for instance, inject controlled label noise or bounding-box jitter into a subset of COCO and measure the rank correlation between the score and the injected noise level, or compare the score against human difficulty ratings. Without such evidence, the paper's conclusion that it \"accurately characterize[s] aleatoric uncertainty\" (Section 6) is not established.","section":"Section 3 and Sections 5.1-5.2"},{"comment":"The hyperparameters beta and the filtering quantile p appear to be selected without a stated held-out procedure. Specifically, Section 5 states that beta is set to 0.2 for YOLOX and 0.3 for Deformable DETR, and Tables 3 and 5 use p = 95% and 90% across models. If these values were tuned on the COCO val set, the reported gains may reflect model selection on the evaluation set. The paper should state the selection protocol, report results across a range of p (as is done for beta in Table 6), or use a separate validation split for hyperparameter choice.","section":"Section 5, Implementation Details and Tables 1, 3, 4, 5"}],"minor_comments":[{"comment":"There are typos: \"curial\" in the abstract should be \"crucial\", and \"aimt\" in Section 1 should be \"aim\".","section":"Abstract and Section 1"},{"comment":"The notation in Eq. (7) is confusing due to nested braces and superscripts; please rewrite it more clearly, e.g., by defining the per-class quantile threshold first and then the retained set.","section":"Eq. (7)"},{"comment":"The definition of B^{c_j}_m is inconsistent with its use; please define the bin index and class explicitly, e.g., as B^c_m = {j : c_j = c and d(z_j|c_j) in I_m}, and then define D* accordingly.","section":"Eq. (8)"},{"comment":"Table 5 reports only AP for BDD100K; please also report AP50, AP75, or AR to allow a fuller comparison, or state why these are omitted.","section":"Section 5.2, Table 5"},{"comment":"The phrase \"uniforming data selection\" should be \"uniform data selection\" or \"uniform sampling\".","section":"Section 5.2"},{"comment":"The histogram in Fig. 2 appears without labeled axes; please add axis labels and a caption explaining the units of the horizontal axis.","section":"Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"The sign error in Eqs. (3)-(4) is likely a typo, but it is a serious presentation flaw that must be corrected. The main substantive risk is the lack of direct validation of the score as aleatoric uncertainty; the downstream proxy tasks are not sufficient. I believe the paper is worth a major revision rather than rejection because the gap is addressable with additional experiments, and the core idea of using SAM features for data-centric training is interesting. I would also encourage the editor to verify that the hyperparameter selection protocol is clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper repurposes a Mahalanobis-distance-in-feature-space idea (their own NeurIPS 2024 paper and the OOD literature) for object detection, using SAM features, and shows that filtering and reweighting by the score gives consistent AP improvements across YOLOX, FCOS, Deformable DETR, and DINO on COCO and BDD100K. That is a real, useful empirical result, and the plug-and-play framing is honest.\n\nWhat is actually new: applying this scoring to box-level SAM features for detection, and the two usages — quantile-based noisy-sample filtering and uncertainty-weighted entropy regularization. The experiments are broad enough to make the point that the score carries signal beyond what uniform filtering or constant entropy regularization provide.\n\nThe soft spots are real. First, Eq. (3) defines M as a negative quadratic form, so log M in Eq. (4) is undefined for every object. The positive scores in the figures make it almost certain this is a sign typo, but without code or data the actual scoring function can't be audited. A reader should not have to guess. Second, the paper calls this 'aleatoric uncertainty' but never validates that label. The score is feature atypicality with respect to a class-conditional Gaussian; it may track occlusion, box jitter, or label noise, but also just rarity or domain shift. The only support is qualitative images and proxy gains from filtering/regularization on the same dataset. That is consistent with a weaker claim: the score identifies hard or atypical instances, and removing or reweighting them helps. Third, the comparisons are against vanilla and constant-entropy baselines, not against the noisy-label detection literature (e.g., their own cited [31]), and hyperparameters p and β are chosen per model without a clear selection rule. No error bars, no code.\n\nThe sign error is easy to fix, but the missing validation is not. As written, the central claim outruns the evidence. Still, the empirical demonstration is valuable enough that a serious editor should send it to review — the method may well work as a data-centric add-on, and the reviewers can push for the corrections and the harder benchmarks.","headline":"A plausible data-centric trick that delivers small consistent gains, but the core score is misdefined in the written equations and the aleatoric-uncertainty claim outruns the evidence.","tokens_in":15684,"tokens_out":2706,"would_cite":false,"duration_ms":26679,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that the aleatoric uncertainty of each object instance in a detection dataset can be read off from the feature space of a vision foundation model, and that using this score to filter noisy or redundant samples and to…","keywords":["aleatoric uncertainty","object detection","vision foundation models","SAM","Mahalanobis distance","sample filtering","uncertainty regularization","data-centric learning"],"falsifier":"Take a set of clean COCO objects, corrupt a random subset by shifting or shrinking their boxes or swapping their class labels, and check whether the Mahalanobis-based score ranks the corrupted instances among the top 5–10% of uncertainty scores. If the score does not separate these synthetic noisy samples from hard-but-clean ones, or if removing the top-5% scored samples fails to improve AP relative to removing a random 5%, the central claim that the score identifies noise rather than merely difficulty would be falsified.","tokens_in":1674,"feed_emoji":"🎯","tokens_out":1927,"duration_ms":34907,"temperature":0.7,"pith_summary":"The paper claims that aleatoric (data) uncertainty in object detection can be quantified per instance by measuring how far an object's pooled feature vector sits from the centroid of its class in the feature space of a vision foundation model. The authors use the Segment Anything Model (SAM) to extract object features, fit a class-conditional Gaussian with a shared covariance, and define a Mahalanobis-distance-based score as the uncertainty measure. They then put this score to use in two plug-and-play ways: discarding high-uncertainty (noisy) and redundant training samples, and weighting an entropy regularizer so that harder samples receive stronger regularization. Across COCO and BDD100K, and across CNN-based and transformer-based detectors, the uncertainty-aware filter and regularizer improve average precision and recall. A sympathetic reader would care because the approach treats uncertainty as an extra annotation computed once offline, so it can benefit any detector without changing the model.","feed_headline":"SAM feature distance flags noisy objects in detection data","feed_subtitle":"Ranking training instances by Mahalanobis distance to class centroids improves detector AP on COCO and BDD100K.","key_machinery":"The central object is the Mahalanobis distance-based uncertainty score computed from SAM's vision encoder: for each labeled object instance, the feature vector $V(z_j)$ is obtained by pooling the SAM feature map inside the ground-truth box, and a class-conditional Gaussian $\\mathcal{N}(V(z) \\mid \\mu_k, \\Sigma)$ with a class-shared covariance $\\Sigma$ is fit on the training set. The score $M(z_j|c_j) = -\\left(V(z_j)-\\mu_{c_j}\\right)^\\top \\Sigma^{-1}\\left(V(z_j)-\\mu_{c_j}\\right)$ measures distance to the class centroid, and a per-class log-transform plus min-max normalization turns it into an uncertainty value in $(0,1)$. This score does the argument's work because it is cheap to compute once per dataset, is model-agnostic, and its two downstream uses (quantile-based filtering and uncertainty-weighted entropy regularization) are what produce the reported performance gains.","core_discovery":"The central claim is that the Mahalanobis distance of a SAM-pooled object feature to its class-conditional Gaussian centroid is a reliable aleatoric uncertainty score, and that this score captures three meaningful regimes: easy objects with low score, hard objects (occluded, small, or obscure) with medium score, and noisy objects (misleading boxes or wrong labels) with high score. The paper further claims that this score is practically useful: keeping only samples below a quantile of the score abandons harmful noisy instances and improves AP, while binning samples by score and discarding a fraction within each bin removes redundancy more safely than uniform random dropping. Finally, the paper claims that an uncertainty-weighted entropy regularizer, which replaces a constant entropy penalty with one scaled by each sample's uncertainty score, consistently beats both vanilla training and constant-entropy regularization, with larger gains on small-capacity models.","pith_inferences":["The paper never validates the score against ground-truth uncertainty labels, so a natural next test is to corrupt known-clean boxes with synthetic noise and check whether the score ranks them as more uncertain; this would separate genuine noise detection from mere hardness ranking.","Because the score only relies on SAM's frozen features and the training labels, it could be extended to other dense-prediction tasks like segmentation or to label-efficient settings where noisy pseudo-labels come from an auto-labeling pipeline.","The class-shared covariance assumption could be relaxed to class-specific covariances or a mixture-of-Gaussians with more components; the reported robustness to hyperparameters suggests the score is driven by the centroid distance more than by the covariance shape.","The authors' closing remark hints that filtering high-uncertainty COCO-derived instruction data could reduce hallucination in large vision-language models; this is a testable application the paper does not itself evaluate."],"forward_implications":["Detection training can be made more reliable by treating uncertainty as an offline per-instance annotation, avoiding any extra cost during training.","Discarding roughly 5% of the most uncertain samples yields consistent AP gains across YOLOX, FCOS, Deformable DETR, and DINO, suggesting that noisy instances in standard benchmarks actively hurt training.","Uncertainty-aware redundant-sample pruning degrades performance far less than uniform random pruning, supporting a data-pruning paradigm guided by feature-space density.","Uncertainty-aware entropy regularization improves small-capacity detectors by a larger margin than large ones, which could translate to cheaper deployment of compact models.","The approach transfers to a self-driving dataset (BDD100K) with long-tailed class distributions, indicating the score is not tuned to COCO's statistics."],"supporting_citations":[{"why":"Supplies the vision foundation model (SAM) whose frozen feature map is used to pool per-object features.","marker":"[20]"},{"why":"Provides evidence that SAM's feature space implicitly encodes semantics, justifying the class-conditional Gaussian assumption.","marker":"[46]"},{"why":"Shows the effectiveness of Gaussian distribution modeling of features for sample difficulty, the methodological precedent for the uncertainty score.","marker":"[5]"},{"why":"Defines focal loss, the baseline hard-sample weighting scheme that the uncertainty-aware regularizer is compared against and improves upon.","marker":"[27]"},{"why":"Establishes that cross-entropy with a maximum-entropy regularizer is a lower bound of focal loss, motivating the uncertainty-weighted entropy regularizer.","marker":"[33]"},{"why":"Provides the MS-COCO benchmark used in the main experiments, including its training and validation splits.","marker":"[26]"},{"why":"Provides the BDD100K self-driving benchmark used to test transfer across domains.","marker":"[48]"}],"fun_headline_variants":["SAM Mahalanobis distance flags noisy object instances","SAM-based aleatoric uncertainty improves detector AP","Uncertainty-aware sample filtering with SAM boosts accuracy","Class-conditional Gaussian on SAM features spots bad data"],"cache_read_input_tokens":17792,"weakest_assumption_plain":"The paper assumes that a single class-conditional Gaussian with a shared covariance matrix, fit to SAM-pooled features of the same training set, accurately describes the feature distribution, so that Mahalanobis distance is a valid proxy for aleatoric uncertainty without any ground-truth uncertainty labels to calibrate against.","fun_headline_variants_meta":{"raw":{"variants":["SAM Mahalanobis distance flags noisy object instances","SAM-based aleatoric uncertainty improves detector AP","Uncertainty-aware sample filtering with SAM boosts accuracy","Class-conditional Gaussian on SAM features spots bad data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000356,"raw_usage":{"total_tokens":1949,"prompt_tokens":977,"completion_tokens":972,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":910}},"tokens_in":593,"tokens_out":972,"duration_ms":8836,"temperature":1.0,"reasoning_tokens":910,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:25:47.049702+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of clean COCO objects, corrupt a random subset by shifting or shrinking their boxes or swapping their class labels, and check whether the Mahalanobis-based score ranks the corrupted instances among the top 5–10% of uncertainty scores. If the score does not separate these synthetic noisy samples from hard-but-clean ones, or if removing the top-5% scored samples fails to improve AP relative to removing a random 5%, the central claim that the score identifies noise rather than merely difficulty would be falsified.","supporting_citations":[{"cited_title":"Segment and Caption Anything","cited_arxiv_id":null,"evidence_quote":"Provides evidence that SAM's feature space implicitly encodes semantics, justifying the class-conditional Gaussian assumption."},{"cited_title":"Learning sample difficulty from pre-trained models for reliable prediction","cited_arxiv_id":null,"evidence_quote":"Shows the effectiveness of Gaussian distribution modeling of features for sample difficulty, the methodological precedent for the uncertainty score."},{"cited_title":"Focal loss for dense object detection","cited_arxiv_id":null,"evidence_quote":"Defines focal loss, the baseline hard-sample weighting scheme that the uncertainty-aware regularizer is compared against and improves upon."},{"cited_title":"Calibrating deep neural networks using focal loss","cited_arxiv_id":null,"evidence_quote":"Establishes that cross-entropy with a maximum-entropy regularizer is a lower bound of focal loss, motivating the uncertainty-weighted entropy regularizer."},{"cited_title":"Microsoft coco: Common objects in context","cited_arxiv_id":null,"evidence_quote":"Provides the MS-COCO benchmark used in the main experiments, including its training and validation splits."},{"cited_title":"BDD100K: A diverse driving dataset for heterogeneous mul- titask learning","cited_arxiv_id":null,"evidence_quote":"Provides the BDD100K self-driving benchmark used to test transfer across domains."}],"review_version":1}