{"id":"aa906e0c-37df-4c76-a9f0-9edf5746b2d9","arxiv_id":"2607.13361","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Open-vocabulary detector confidence stays high when a queried object is occluded because it fires on same-category distractors, so it reports category presence rather than target visibility.","lead":"This paper shows that open-vocabulary detectors keep reporting a named object as present when it is almost fully hidden, because they fire on same-category objects in front of it. The finding matters for robots and vision systems that use detector confidence to decide whether to look again or trust a view.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim presumes an instance-level downstream task, but queries are only category labels; firing on a same-category distractor is then correct category detection, so the decoupling may be a query-task mismatch rather than a detector fault.","rationale":"I read the strongest claim as: detector confidence indicates category presence anywhere in the view, not visibility of a specific target instance, and therefore cannot be used for gating/evaluation in active perception. The most load-bearing condition for this claim is that the downstream system must be committed to a specific instance. The paper sets this up in §1 and §5 by citing active-perception systems that move a camera to reveal a named object and then localize/grasp that specific object, but the empirical audit always uses a category label as the query. A high score on a same-category distractor is exactly what a category-level open-vocabulary detector should produce; the failure only becomes a 'false positive' under an instance-level interpretation. Since the reader's weakest_assumption already identifies this, I agree. I do not see a more serious internal inconsistency: the geometry-oracle methodology is sound, the threshold-robustness analysis in §4.4 supports the flatness of confidence, and the ManiSkill/DAVIS replications strengthen generalizability. The numerical discrepancy between §4.1's 'off-target rises from 15%' and Table 4's 'on-target 29%→0%' is worth a cleanup but does not undermine the qualitative decoupling. The paper's own §6 limitations (no end-to-end grasp success, approximate real-video oracle) are honestly stated and correctly temper the conclusion. My concern therefore keeps the verdict at CONDITIONAL: the central claim is plausible and well-supported for category-level queries used in instance-level manipulation, but it has not been shown to survive instance-specific prompting or same-category-free occlusion. The concrete test would settle this directly, and the recommendation should be conditional on that task framing.","tokens_in":11393,"tokens_out":6470,"duration_ms":76051,"concrete_test":"Re-run the LIBERO frontal-layout audit at k=12 under two manipulations: (a) use an instance-discriminating prompt, e.g. a referring expression with spatial context ('the soup can behind the carton') or a reference image/mask of the specific target, and (b) keep the category prompt but remove all same-category distractors, occluding the target only with dissimilar objects. Measure ρ(logV, c) and on-target rate for Grounding DINO-base. If confidence now tracks visibility (correlation drops or resolved rate falls when V falls), the paper's central failure is an artifact of category-level querying plus same-category clutter, and the recommendation should be narrowed to 'use instance-grounded prompts' rather than 'confidence cannot be used.' If confidence stays flat in both conditions, the claim generalizes beyond category-presence ambiguity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central conclusion in §5—'confidence answers whether something matching the query appears anywhere in the view, not whether the specific target is visible'—is convincing only if the downstream task really requires locating a particular instance. Throughout the audit, the query q is a category name (§3.1: 'a scene contains one target object o* named by a text query q'), and all prompted text in the experiments is a category label ('soup can', 'milk carton', etc.). When the target is occluded, the high-scoring detection lands on another object of the same category (§4.1: 'it is reading a different can'). But a category-level detector is supposed to respond to category presence; scoring a second can when asked for 'soup can' is not a failure of the detector's stated function. The paper's argument that real systems make the category-to-instance inference (§1, §5) rests on citing VISO-Grasp and ActiveGrasp, not on measuring their task specifications or failure modes. If a user or downstream module only asks 'is a can present?', the detector's behavior is arguably correct and the decoupling is not a problem. This is not an internal inconsistency, but it is the load-bearing scope condition: the headline claim—confidence cannot be read as visibility for gating/evaluation—holds for category-level queries used in instance-level tasks, and the paper has not demonstrated that this condition is unavoidable in the cited systems. The paper's own §6 limitation (end-to-end grasp success remains 0%, real-video oracle approximate) further narrows the practical domain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a geometry-oracle audit to test whether open-vocabulary detector confidence tracks the true visibility of a specific target object under occlusion. In rendered manipulation scenes, the oracle counts the target's visible pixels from a ground-truth segmentation, while detectors (Grounding DINO, OWLv2, SAM3) are queried with category labels. The results show that as true target visibility falls from ~100% to ~12% of scenes, detector confidence remains nearly flat (≈0.40) and the resolved rate stays above 90%, with top detections increasingly landing on same-category distractors (up to 88% off-target). This decoupling is shown across nine categories, two simulators, and real video (DAVIS), with some variation in degree (OWLv2 and SAM3 lower their scores under occlusion but still fail to localize the target). The paper quantifies downstream consequences: a confidence-based active-perception benefit metric understates the value of moving the camera by about a factor of ten, and a confidence-based gate fails to signal occlusion. The authors also show that a ring occluder layout does not actually occlude the target, so confidence drops there measure clutter rather than occlusion. They release (claim) a benchmark and recommend target-grounded signals for gating and evaluation.","tokens_in":11674,"tokens_out":7986,"duration_ms":83710,"significance":"The paper provides a well-controlled, detector-free empirical demonstration that open-vocabulary detector confidence is not a reliable proxy for the visibility of a specific target instance in clutter. The geometry-oracle protocol is a strong methodological contribution, and the robustness checks (thresholds, multiple detectors, two simulators, real video) support the core finding. If accepted, the paper would serve as an important caution for the robotics and vision communities: confidence scores should be interpreted as category-presence signals, not instance-visibility signals, in tasks such as active perception, grounding, and retrieval. The paper is also honest about limitations, including the residual localization error and the fact that end-to-end grasping still fails even after active perception. The main weakness is a scope condition: the claim that the detector is 'wrong' depends on an instance-level downstream task, while the queries are category names. For a category-level presence task, firing on another instance of the same category is arguably correct. The paper should more carefully delimit its claims and provide evidence that the cited deployed systems actually mak","major_comments":[{"comment":"The central conclusion—'Detector confidence answers whether something matching the query appears anywhere in the view, not whether the specific target is visible at a specific place'—is conditional on an instance-level downstream task. However, the paper motivates the 'matters' claim by citing VISO-Grasp and ActiveGrasp without verifying that these systems actually prompt with category names and read the score as an instance-visibility signal. A category-level query is expected to fire on a same-category distractor; the paper even shows that the detector is reading a different can. The paper should either (a) provide evidence from the cited systems that they perform the category-to-instance inference, or (b) explicitly scope the conclusion to instance-level tasks and soften the title/abstract to avoid implying that the detector is malfunctioning for category-presence detection. This is l","section":"§1, §5"},{"comment":"The paper repeatedly states that it 'releases the controlled benchmark' but provides no artifact, repository link, or data-availability statement. Without the benchmark, the third contribution is unverifiable and the experiments cannot be exactly reproduced from the description alone. Please add a working link or state that the data will be made available upon acceptance; if no artifact is intended, the release claim should be removed.","section":"Abstract, §1, §4.4"}],"minor_comments":[{"comment":"Typo: 'Thedetectorisalmostblindtothisocclusion' should have spaces. Similar spacing issues appear in §2 ('compareintegrationparadigmsandreport'). A proofread pass is needed.","section":"§4.1"},{"comment":"The term 'resolved' is used to mean that confidence exceeds a threshold, but this could be misread as actual target resolution. Consider using 'detector-resolved' or adding a qualifier to distinguish it from oracle visibility.","section":"§3.1, §4.1"},{"comment":"The phrase 'a confidence gate fires exactly when the object is hidden' is misleading: the gate fails by staying put (always confident), not by triggering at hidden times. Rephrase to 'a confidence gate reports the target as visible and fails to trigger a move when the object is hidden.'","section":"§4.1"},{"comment":"SAM3 is described as a 'detector' throughout, but it is a promptable segmentation model with a presence score. Please clarify this in the text to avoid conflating detection and segmentation.","section":"§4.3, Table 4"},{"comment":"The large gap between GDINO (AUC 0.50) and OWLv2 (AUC 0.76) for confidence-based gating deserves a brief explanation in the text, e.g., linking to the partial correlation of OWLv2 with visibility reported in Table 4.","section":"Table 3"},{"comment":"The downstream grasping experiment shows 0% end-to-end grasp success even after moving the camera. The paper honestly acknowledges this, but the statement in §5 that 'the safe policy is to treat heavy clutter as a reason to look' could imply a validated solution. Consider explicitly stating that active perception improves localizability but not grasp success within the tested pipeline.","section":"§4.5, §6"}],"recommendation":"major_revision","confidential_remarks":"The empirical core is solid and the oracle protocol is a genuine contribution. The main risk is overgeneralization from instance-level failures to detector behavior in general; the author should be asked to tighten the scoping and, if possible, provide evidence about the cited systems' actual use of confidence. Also, the promised benchmark should be made available before acceptance; as it stands, the 'release' claim is unverifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know. First, this is a well-built audit and the central claim is convincing for the tasks it cares about: detector confidence from open-vocabulary models tracks whether anything of that category is in view, not whether the specific target is visible. Second, the one big caveat — that this only breaks if you need a particular instance rather than just the category — is real but sits inside the paper's own framing, so it doesn't sink the argument.\n\nWhat's genuinely new is the geometry-oracle protocol. Pairing every view with a segmentation-based visibility count and then comparing detector confidence against it is the right way to separate occlusion from detector behavior. The evidence is broad: three detectors of different designs, two simulators with different renderers, nine categories, real video, and a threshold-robustness check. The control with the ring layout is a nice catch — confidence drops when the target is merely surrounded, which isolates clutter from occlusion. The consequence results, an 8 vs 88 point benefit under a confidence metric and a gate that fires when the target is hidden, are concrete and useful.\n\nSoft spots, in proportion. The promised benchmark is announced but not shipped; there's no code or data artifact to check. Some statistics are underreported — correlations in the main table have no intervals, and there is a small discrepancy between Tables 2 and 4 at k=12 that should be reconciled. The real-video oracle uses the annotated mask as ground truth, which is reasonable but not as clean as the simulator oracle. And the end-to-end grasp success is zero even after moving, so the paper's practical payoff is diagnostic rather than an immediate fix. The stress-test worry about category-level vs instance-level is a fair flag, but the paper explicitly motivates the instance-level task with grasp systems, and the discrepancy only matters if you believe those systems don't need to identify the specific target. That seems like a stretch given the manipulation setting.\n\nThis paper deserves serious peer review. It's a careful negative result that will be useful to anyone building active-perception or grounding stacks, and it gives other groups a cheap diagnostic to run on their own detectors. I'd send it to a robotics or perception venue with the expectation that the authors add the artifact and sharpen the task-scope discussion.","headline":"A well-designed negative result showing detector confidence tracks category presence, not target visibility, with real consequences for active-perception gating — the instance-level caveat is real but already inside the paper's stated scope.","tokens_in":12203,"tokens_out":2494,"would_cite":true,"duration_ms":25697,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-vocabulary detector confidence signals category presence, not target visibility.","keywords":["open-vocabulary detection","detector confidence","occlusion","active perception","robot manipulation","object presence","calibration","target grounding"],"falsifier":"A factorial experiment would settle it: fix the target's visible pixels at a constant value and vary the number of same-category distractors in view. If confidence tracks category presence, mean confidence should rise monotonically with distractor count while visibility is unchanged; if confidence tracks occlusion, it should remain flat. The paper's ring-layout control (confidence drops while visibility stays constant) and randomized frontal layout (confidence rises while visibility falls) already bracket this, but a direct manipulation of distractor count at constant target visibility would i","tokens_in":1463,"feed_emoji":"🔍","tokens_out":1980,"duration_ms":89988,"temperature":0.7,"pith_summary":"The paper claims that an open-vocabulary detector's confidence score answers whether an object matching a text query appears anywhere in the image, not whether a specific target instance is visible at a specific location. Using a detector-free geometry oracle that counts the target's visible pixels, the paper shows that as true visibility falls from every scene to about one in eight, confidence stays flat near 0.40 and the detector still reports the target present in over nine scenes out of ten, with its detection box landing on a same-category distractor. This decoupling holds across three detectors of different design, nine object categories, two simulators, built and natural occlusion, and real video. Two consequences matter for robotics and perception: a confidence-based evaluation metric understates the value of moving the camera by roughly a factor of ten, and a confidence-based gate that should trigger a look fails precisely when the target is hidden. The paper recommends treating clutter as a reason to look and scoring active perception with target-grounded signals rather than raw category confidence.","feed_headline":"Detector confidence tracks presence, not visibility","feed_subtitle":"In clutter the score stays high on same-category distractors, so gates, thresholds, and metrics misread occlusion.","key_machinery":"The central machinery is the geometry-oracle audit: a detector-free measurement of true visibility V(p), the pixel count of the target's ground-truth mask in the rendered view at pose p, compared against the detector's confidence c(p) and its top-scoring region r(p) on the same view. The audit pairs a fixed default view with the best of 24 candidate views on a hemisphere, and attributes a detection as on-target when its overlap with the target mask exceeds one half. This separation of geometry from appearance is what lets the paper establish that c(p) stays high while V(p) collapses, and that r(p) rests on same-category distractors. The audit is cheap because a simulator provides the segment","core_discovery":"The central discovery is that open-vocabulary detector confidence is a presence signal, not an occlusion or visibility signal. In controlled cluttered scenes, the target's true visible pixel count fell with added distractors from covering every scene to one in eight, while a common detector's mean confidence stayed at 0.39–0.40 and its resolved rate stayed above 0.92; by pixel-level attribution, the fraction of confident detections that actually overlapped the target dropped from near one to near zero, because the detector fired on another object of the same category. On real video the detector reported the occluded target present in 99% of frames and localized to a different instance in 94%","pith_inferences":["If confidence is presence rather than visibility, then referring-expression grounding and instance retrieval that reuse the same score likely inherit the same identity confusion in cluttered scenes, even outside manipulation.","A direct testable extension: hold target visibility fixed and vary the number of same-category distractors; the presence hypothesis predicts confidence rises with distractor count, while the visibility hypothesis predicts it does not.","The instance-level task assumption is load-bearing: if the downstream goal is only to know whether any soup can is present, the detector's distractor responses would be correct rather than erroneous. The paper argues active-perception systems make the category-to-instance inference, but the reported failure is conditional on that premise.","The residual localization error after moving suggests a limit beyond confidence: mask-plus-depth localization is biased because an occluder sits at the target's center pixel, so active perception alone may not reach grasp tolerance without multi-view fusion or contact-rich search."],"forward_implications":["Confidence thresholds used to filter detections will admit false positives whenever another object of the named category is in view, because the score does not fall when the target itself is hidden.","Any evaluation of open-vocabulary detection or active perception that scores by category confidence will understate the benefit of resolving occlusion; in the paper's active-perception setting the confidence-based benefit is 8 points against an 88-point oracle benefit.","A gate that moves the camera only when confidence is low will stay put in heavily occluded scenes, since confidence stays high; the safe policy is to treat heavy clutter as a reason to look.","Benchmarks built with a ring of distractors around the target do not actually occlude it, so results on such layouts may credit active perception with resolving occlusion that was never present.","In a downstream grasping test, a high-confidence fixed view left the target localizable in only 5% of scenes, while moving first made it localizable in every scene; yet even then residual depth bias kept end-to-end grasp success at 0%."],"fun_headline_variants":["Confidence stays high when object is 87% occluded","Object detector fires on decoys as true target disappears","Detector confidence understates occlusion resolution 10x","Confidence-based gates misfire exactly when object is hidden","Detector confidence signals presence, not occlusion"],"cache_read_input_tokens":13440,"weakest_assumption_plain":"The central finding assumes the system's task is to locate a specific target instance while the detector is prompted with only a category name, so firing on a same-category distractor counts as a failure; for a purely category-level question such as 'is any soup can present?', the detector's response would not be an error.","fun_headline_variants_meta":{"raw":{"variants":["Confidence stays high when object is 87% occluded","Object detector fires on decoys as true target disappears","Detector confidence understates occlusion resolution 10x","Confidence-based gates misfire exactly when object is hidden","Detector confidence signals presence, not occlusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1555,"prompt_tokens":848,"completion_tokens":707,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":630}},"tokens_in":592,"tokens_out":707,"duration_ms":7640,"temperature":1.0,"reasoning_tokens":630,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T05:22:21.078863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A factorial experiment would settle it: fix the target's visible pixels at a constant value and vary the number of same-category distractors in view. If confidence tracks category presence, mean confidence should rise monotonically with distractor count while visibility is unchanged; if confidence tracks occlusion, it should remain flat. The paper's ring-layout control (confidence drops while visibility stays constant) and randomized frontal layout (confidence rises while visibility falls) already bracket this, but a direct manipulation of distractor count at constant target visibility would i","supporting_citations":[],"review_version":1}